Terraform: Choosing credentials for a remote state file

Question

I have existing infrastructure in Terraform and have been using it for a while. Recently I had swapped the AWS credentials of my local laptop (the creds stored in ~/.aws/credentials) and it stopped working until I re-set those credentials back.

The problem is that I'm declaring the creds in the Terraform source itself but it doesn't seem to be using them at all.

terraform {
  backend "s3" {
    bucket = "example_tf_states"
    key    = "global/vpc/us_east_1/example_state.tfstate"
    encrypt = true
    region     = "us-east-1"
  }
}

provider "aws" {
  access_key = "${var.access_key}"
  secret_key = "${var.secret_key}"
  region = "${var.region}"
}



variable "access_key" {
  default = "<hidden_for_stack_exchange_post>"
}

variable "secret_key" {
  default = "<hidden_for_stack_exchange_post>"
}

variable "region" {
  default = "us-east-1"
}

The access ID permissions are 100% good. I am using the same account ID and secret key both for the aws configure settings that go into ~/.aws/credentials as I am in the above Terraform variable declarations.

Everything works fine as long as the creds are in ~/.aws/credentials but as soon as the OS level credentials are gone (ie rm ~/.aws/credentials) I get the following when trying to run Terraform operations, such as terraform plan:

Failed to load backend:
Error configuring the backend "s3": No valid credential sources found for AWS Provider.
  Please see https://terraform.io/docs/providers/aws/index.html for more information on
  providing credentials for the AWS Provider

Please update the configuration in your Terraform files to fix this error.
If you'd like to update the configuration interactively without storing
the values in your configuration, run "terraform init".

If I re-populate the ~/.aws/credentials by running aws configure it will work fine again.

I'm not understanding -- if my provider setting is explicitly declaring the credentials to use inside the Terraform source code, why does my OS-level AWS configuration matter at all?

How can I make Terraform only use the creds defined in my Terraform configuration and ignore what's in my OS user profile?

Edit, it's Terraform v0.11.7

Edit: Please note that I'm trying to solve the issue on why the statically declared creds are not being utilized in the provider declaration. Not looking for alternative methods or workarounds. Thanks.

Hrm... just a hunch. Make sure that `AWS_PROFILE` or `AWS_DEFAULT_PROFILE` environment variables are not set as those are a hint to the AWS SDK that it should look in the credentials file. — Erik Osterman, Jul 27 '18 at 05:22
Thanks. Late reply but I'm revisiting this. Unfortunately I've checked the env vars are not present. Really would be nice if it could promp for creds at runtime like other plans that don't use a remote backend. Thanks anyways. — emmdee, Nov 29 '18 at 19:07

score 12 · Accepted Answer · answered Dec 04 '18 at 21:38

Your first question

if my provider setting is explicitly declaring the credentials to use inside the Terraform source code, why does my OS-level AWS configuration matter at all?

The error message "Failed to load backend: Error configuring the backend "s3"" is referring to your Backend S3 configuration.

Look in the file ./.terraform/terraform.tfstate and you will see the S3 Backend configuration.

The Terraform S3 Backend is different than the Terraform AWS Provider. The error message "No valid credential sources found for AWS Provider." is misleading. It implies that the AWS Provider configuration is used, which is false. S3 Backend credentials are configured separately and stored in the terraform.tfstate file.

Your OS-level AWS configuration matters because if no S3 Backend credentials are specified, as documented here https://www.terraform.io/docs/backends/types/s3.html, then Terraform defaults to using the following, in order:

Environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
AWS Shared credentials file, default value is "~/.aws/credentials".

You didn't specify any credentials in your S3 Backend config so terraform is defaulting to the AWS Shared Credentials File.

Your S3 Backend configuration contains no credentials.

terraform {
  backend "s3" {
    bucket = "example_tf_states"
    key    = "global/vpc/us_east_1/example_state.tfstate"
    encrypt = true
    region     = "us-east-1"
  }
}

Your second question,

How can I make Terraform only use the creds defined in my Terraform configuration and ignore what's in my OS user profile?

First, Backends cannot contain interpolation, see https://www.terraform.io/docs/backends/config.html. So you cannot use any variables in the Backend config. e.g. this config is invalid

terraform {
  backend "s3" {
    bucket = "example_tf_states"
    key    = "global/vpc/us_east_1/example_state.tfstate"
    encrypt = true
    region     = "us-east-1"
    access_key = ${var.access_key}
    secret_key = ${var.secret_key}
  }
}

If you want to specify AWS credentials when running terraform init you specify backend configuration as options.

terraform init --backend-config="access_key=your_access_key" --backend-config="secret_key=your_secret_key"

This produces a S3 Backend config that looks like this, stored in the ./.terraform/terraform.tfstate file:

{
    "version": 3,
    "serial": 1,
    "lineage": "bd737d2d-1181-ed64-db57-467d14d2155a",
    "backend": {
        "type": "s3",
        "config": {
            "access_key": "your_access_key",
            "secret_key": "your_secret_key"
        },
        "hash": 9345827190033900985
    },

Again, the S3 Backend credentials are configured separately from your AWS Provider credentials.

Re-run terraform init and specify the credentials on the command line as --backend-config options to fix your error.

Thanks. Very detailed response and I understand the problem now. In my use-case I ended up using the `shared_credentials_file` option that all my backends and providers will harvest and use. Seems to be working nicely and nothing is locked into the OS workstation aws credentials. — emmdee, Dec 10 '18 at 22:55

score 3 · Answer 2 · answered Dec 04 '18 at 11:17

The error you're getting is specifically referring to configuring the S3 backend, which AFAIK does not inherit the settings from the AWS provider configuration; it too has access_key & secret_key configuration options which if you're not using ~/.aws/credentials you will need to explicitly configure.

score 1 · Answer 3 · answered Jun 26 '18 at 22:21

1

You are better off setting up profiles in your ~/.aws/credentials files like

[profile1]
aws_access_key_id = xxxx
aws_secret_access_key = xxxxx
region = us-east-1

[profile2]
aws_access_key_id = xxxx
aws_secret_access_key = xxxx
region = us-west-2

Then in your provider you can tell it which profile to use

provider "aws" {
  profile = "profile2"
  region = "${var.region}"
}

It will keep the keys out of your terraform files which is a good thing if you ever want to put them into source control.

answered Jun 26 '18 at 22:21

Mike

21,910
7
55
79

Thanks for the input. I'm very much aware of the credential profiles, however I'm seeking a solution to the specific question rather than an alternative method or workaround. The creds need to be in variables for the scope of this question. Much appreciated regardless. – emmdee Jun 27 '18 at 20:05

score 0 · Answer 4 · answered Nov 09 '20 at 21:53

I also had the same issue, the one of easiest and the secure way is to fix this issue is that configure the AWS profile. Even if you are properly mentioned the AWS_PROFILE in your project, you have to mention it again in your backend.tf.

my problem was, I have already set up the AWS provider in the project as below and it is working properly.

provider "aws" {
region = "${var.AWS_REGION}"
profile = "${var.AWS_PROFILE}"
}

but end of the project I was trying to configure the S3 backend configuration file. therefore I have run the command terraform init and I also got the same error message.

Error: error configuring S3 Backend: no valid credential sources for S3 Backend found.

Note that is not enough for the terraform backend configuration. you have to mention the AWS_PROFILE in the backend file as well.

Full Solution

I'm using the terraform latest version at this moment. it's v0.13.5. Please note that you can't use variables in the backend file as well.

please see the provider.tf

provider "aws" {
region = "${var.AWS_REGION}"
profile = "${var.AWS_PROFILE}" # lets say profile is my-profile
}

for example your AWS_PROFILE is my-profile then your backend.tf should be as below.

terraform {
    backend "s3" {
    bucket = "my-terraform--bucket"
    encrypt = true
    key = "state.tfstate"
    region = "ap-southeast-2"
    profile = "my-profile" # you have to give the profile name here. not the variable("${var.AWS_PROFILE}")
  }
}

then run the terraform init

Terraform: Choosing credentials for a remote state file

4 Answers4