Use r10k with separate module repositories

Question

I have a working setup of Puppet with Master and Agent. Now I'm trying to use r10k to manage Puppet code. My goal is that on each git pushof a code change the Puppet code below /etc/puppetlabs/code/environments on the Puppet Master gets updated automatically.

While this seems to work for one monolithic control repository containing all modules, I'm not getting it to work if custom modules are in separate Git repositories.

My setup is as follows:

r10k configuration in /etc/puppetlabs/r10k/r10k.yaml:

sources:
  operations:
    remote: '/srv/git/puppet.git'
    basedir: '/etc/puppetlabs/code/environments'

Central control repository /srv/git/puppet.git containing branches production and testing, each with:

• A manifests/site.pp with node configurations, e.g. node default { contain mymodule }
• A Puppetfile with external module references

The Puppetfile for testing would look like e.g.

mod 'mymodule',
  :git => 'file:///srv/git/mymodule.git',
  :ref => 'HEAD'

while for production it would look like e.g.

mod 'mymodule',
  :git => 'file:///srv/git/mymodule.git',
  :ref => 'stable'

(Until this works I only have branch production and reference to HEAD.)

Another Git repository for each custom module, e.g. /srv/git/mymodule.git

Git Hooks to run r10k whenever git push is executed:

• /srv/git/puppet.git/hooks/post-receive

  with content r10k deploy environment -pv

• /srv/git/mymodule.git/hooks/post-receive

  with content r10k deploy module mymodule -v

The hooks get executed, since upon git push I get output like

remote: INFO     -> Deploying environment /etc/puppetlabs/code/environments/production
remote: INFO     -> Environment production is now at 991830eb1561cddd7970be4152748168df52ef79
remote: INFO     -> Deploying Puppetfile content /etc/puppetlabs/code/environments/production/modules/mymodule

and

remote: INFO     -> Deploying module /etc/puppetlabs/code/environments/production/modules/mymodule

My problem is that - despite this output - changes pushed to the module's repository are not reflected by the files below /etc/puppetlabs/code/environments.

---

Edit

Elaborating a bit on my intended workflow:

• My control repo has two branches, production and testing, while the module repository has only one branch master. I thought it would be easier that way, especially to avoid merge issues. But considering that in Git merges can be restricted to fast-forward, and r10k has active support for the tracking of branches, my workflow might actually be easier using the branches production and testing for the module repositories as well. I would then work (develop) on testing and only switch to production to execute something like git merge --ff-only testing.
• My first thought was to create a Git tag stable for the module repository which I move to the current commit every now and then, i.e. staying in the master branch all the time without any merging done at all. But a fast-forward-only merge between branches can also save me from merge conflicts, so there's probably no reason to avoid branches. Also I could get the same workflow on both the control repo and the module repos.
• I'm only starting to learn about Puppet with two virtual machines, one Puppet master and one agent. I'll add another VM agent and then point one to the production environment and the other one to testing so I can verify my workflow while building a usable Puppet configuration for the agents. For now I'll simply set the environment in each agent's puppet.conf.
• Later I'll use real Linux boxes and map one agent to testing and all others to production. The master will be on another box and set up manually.

Answer 1

As per your question, and the comments:

My problem is that - despite this output - changes pushed to the module's repository are not reflected by the files below /etc/puppetlabs/code/environments.

r10k doesn't know about HEAD. HEAD is non deterministic, so it's not possible to use it like this.

If you want to track a branch, use something like:

mod 'mymodule',
  :git => 'file:///srv/git/mymodule.git',
  :ref => 'master'

To keep the setup simple I wanted to avoid branching. Instead I wanted to always use the latest commit of the module repository for testing and use another stable tag for production which I move forward whenever the code is in acceptable shape.

Honestly, I'm unsure how your workflow looks like:

• You've got a control repo with two branches, master and testing. master is the stable, production code, while you're using testing for development?

• You're pushing new code to testing, and once it reaches a state you're comfortable with, you're merging testing into master?

• After this, you would like to tag the current state in master as stable, which your agents should use?

• How does the rest of your infrastructure look like? How many agents are in place? How do you test your code, i.e. how do you tell {one,all} (?) agents: "Please use this code coming out of this environment"?

• Follow up to the above: If you've code in master and testing, how do your agents know which environment to use?

• If you could elaborate on the above points and tell me some more details (and please add this to your question, the comments might be too limited for that), I might able to help and extend this answer.

• Anyway, good luck! puppet and r10k, especially if coupled with git hooks, are an awesome setup! Using this myself, it's just great! :)

Additional info:

• Thanks for elaborating on your workflow.

• If you aren't that versatile yet with git, and especially if you're doing a lot of branching and merging and are afraid of merge conflicts, make sure to read about git rebase, which does

  reapply commits on top of another base tip

• Say you're in the following situation:

  • You're branching of master to feature1.
  • You do some work in file1:3-6 in feature1.
  • You're changing file1:7-10 directly in master to fix a pressing bug.
  • You want to merge feature1 back into master.
  • Normally, at this point, you would get a merge conflict, because the same file was modified in both branches.
  • If you do git rebase master while on feature1 before merging, this will "pull" the latest state / commit(s) into feature1, in this case the bug fix.
  • Now do the merge, without conflicts.
• I wouldn't limit myself to only one branch in the module repositories. How do you handle testing new code inside there? It's not only your control repo which needs to enhance over time.

• To make this comfortable, use another feature of r10k:

  # Track control branch and fall-back to master if no matching branch exists
  mod 'mymodule',
    :git => 'file:///srv/git/mymodule.git',
    :branch => :control_branch,
    :default_branch => 'master'
  

• This will allow you to do the following, for example in a situation where you would like to develop a new feature in your module:

  • In your control repo, branch of master to testing.
  • In your module repo, branch of master to testing.
  • Without changing your Puppetfile, your puppet agent which uses the testing branch of your control repo now also uses the testing branch of your module repo.
  • Develop up until you're satisfied.
  • In your module repo, merge testing into master.
  • The new code you've developed will now go to all your puppet agents which use the production code, i.e. the master branch of your control repo.

• One last note, for now: It sounds like you're the only person developing code, so maybe that's not that important for you, but just to let you know:

  • If two branches are too limited, it's quite possible to use a different environment during a puppet agent run without modifying the config, but via supplying a cli param:

    puppet agent -t --environment ${FEATURE_BRANCH}
    

• That's all for now I can think of. Hope this helps, again, good luck and all the best!