2

My state files are kept along side the terraform files. (I know this is not ideal, but it is what it is at my workplace at the moment. There are plans to move away from this state management model)

The goal of my branch is to create a new ec2 instance (named adhoc-ec2) with various policy attachments. The change is tested so the ec2 instance exists in the development environment.

A the end my branch and its associated PR has gone through a PR review process, the state file has already updated in another branch and merged into the master branch.

The difference between the two versions of the files are so big it is impossible to reconcile manually.

The root cause is of course that the terraform file contains too many resources, but it is too late to undo this now.

Now I have to resolve all the conflicts before I can merge my branch into master.

I tried to resolve the conflict this way:

  1. Rebase my branch to master (such that my terraform file will contain all the latest changes)

  2. Overwrite the stale state file in branch with the latest version in the master

  3. run terraform refresh in an attempt to synchronise the state file and the actual resources of the AWS environment.

However apparently this plan does not work. Because when I run terraform plan immediately afterwards, the plans shows that terrafrom still wants to create the ec2 instance adhoc-ec2 (which is already created as I stated earlier in this question)

Alternatively I can destroy the ec2 instances with all the dependent roles and policies, but it is time consuming.

Is there a easier way to resolve this kind of state file conflict? Why refresh did not work in the first place?

Anthony Kong
  • 2,976
  • 10
  • 53
  • 91

2 Answers2

2

Running terraform refresh will only update resources already in the state file so if your instance doesn't exist yet in the state file, Terraform still wants to create it.

What you should do instead is use terraform import to add the resources to state file based on what the AWS API's return which should then match what is in your manifests.

bodgit
  • 4,661
  • 13
  • 26
2

Terraform's workflow does not support sharing changes to the state through a pull request unless you can guarantee that only one pull request will be pending at a time. Terraform expects that once it has updated the state file that new file will be immediately available to all subsequent Terraform runs.

In order to better approximate a standard Terraform workflow (with remote state), I'd suggest modifying your approach as follows:

  • When you're developing a change, use terraform plan to see what its effect will be but do not apply it yet.
  • Once you're happy with your change, submit your PR including only configuration changes. Since the change isn't applied yet, there is no state snapshot update to include in this PR.
  • Once the change is reviewed and merged, communicate to your colleagues somehow that the master branch is temporarily frozen while you apply your change. (This is a manual version of the locking that Terraform would normally handle automatically using a remote backend.)
  • Run terraform apply against the master branch with your change merged into it, and create a new commit containing the terraform.tfstate update. Push this directly to the master branch, bypassing pull requests, because it's just the effect of your change that was already reviewed. (You must push the new state to the master branch even if the terraform apply didn't succeed, because you'll need to open a new PR to respond to any errors but you need to ensure that anyone else who runs Terraform in the meantime will see the partially-updated state.)
  • Once you've pushed the updated terraform.tfstate to the master branch, tell your coworkers that the master branch is unfrozen again and that they must now rebase any pending branches against master in order to ensure they are testing against the latest state. (This is the manual equivalent of unlocking the state.)

As you know, keeping the Terraform state in version control is not recommended because it prevents automatic locking and creates some uncertainty about whether developers are working against the very latest state snapshot. But if you're careful to preserve all of the assumptions Terraform is making, as I described above, then you should be able to use Terraform relatively safely for now until you're able to transition over to the standard approach of using remote state.

Martin Atkins
  • 2,188
  • 18
  • 19