Can't run AWS CLI from CRON (credentials)

Question

Trying to run a simple AWS CLI backup script. It loops through lines in an include file, backs those paths up to S3, and dumps output to a log file. When I run this command directly, it runs without any error. When I run it through CRON I get an "Unable to locate credentials" error in my output log.

The shell script:

AWS_CONFIG_FILE="~/.aws/config"

while read p; do
 /usr/local/bin/aws s3 cp $p s3://PATH/TO/BUCKET --recursive >> /PATH/TO/LOG 2>&1
done </PATH/TO/INCLUDE/include.txt

I only added the line to the config file after I started seeing the error, thinking this might fix it (even though I'm pretty sure that's where AWS looks by default).

Shell script is running as root. I can see the AWS config file at the specified location. And it all looks good to me (like I said, it runs fine outside of CRON).

Definitely tried that first (was using /root/.aws/config), but jumped back to ~/ after seeing it in some other threads. Same error either way. — binaryorganic, Jul 24 '14 at 18:42
Not a direct answer but a comment about using the API keys: It is better practice (and much easier) to assign roles to your instances, and create policies around those roles, and then you are not required to specify the keys at all, or have them lying around in plaintext on the instance. Unfortunately this can only be specified at instance creation time. As an aside, for copying logfiles (and backups etc) have a look at the s3cmd tools, that provides functionality similar to rsync. — nico, Aug 24 '15 at 11:57

score 45 · Answer 1 · edited May 04 '16 at 13:54

45

When you run a job from crontab, your $HOME environment variable is /

The Amazon client looks for either

~/.aws/config

or

~/.aws/credentials

If $HOME = /, then the client won't find those files

To make it work, update your script so that it exports an actual home directory for $HOME

export HOME=/root

and then put a config or credentials files in

/root/.aws/

edited May 04 '16 at 13:54

chicks

3,639
10
26
36

answered Oct 15 '14 at 14:00

Garreth McDaid

3,399
26
41

1

This helped, along with the following fix from http://stackoverflow.com/a/26480929/354709 which involved adding the absolute path for the aws command - as $PATH wasn't set correctly in the root user. – Dan Smart Jun 24 '16 at 16:17
4

This should be the accepted answer. – Madbreaks Mar 07 '17 at 06:52

chicks · Accepted Answer · 2015-08-10T18:28:28.523

24

If it works when you run it directly but not from cron there is probably something different in the environment. You can save your environment interactively by doing

set | sort > env.interactive

And do the same thing in your script

set | sort > /tmp/env.cron

And then diff /tmp/env.cron env.interactive and see what matters. Things like PATH are the most likely culprits.

edited Aug 10 '15 at 18:28

answered Jul 24 '14 at 22:19

chicks

3,639
10
26
36

4

Thanks! A step toward being able to troubleshoot the problem on my own is basically invaluable. There were definitely several differences in the PATH variable, and I kind of think in this case it was the difference in HOME that was throwing things off. As for my specific issue, I ended up just running this from the user's cron file, instead of /etc/crontab, which solved everything on my end. Thanks again! – binaryorganic Jul 26 '14 at 16:55
1

Right. adding a correct `PATH` variable(`echo $PATH` will tell what it should be) in script **_usually_** solves it. – Fr0zenFyr Nov 23 '16 at 17:50

score 6 · Answer 3 · edited Jul 08 '15 at 19:58

6

I was able to solve this issue through the following:

export AWS_CONFIG_FILE="/root/.aws/config"
export AWS_ACCESS_KEY_ID=XXXX
export AWS_SECRET_ACCESS_KEY=YYYY

edited Jul 08 '15 at 19:58

030

5,731
12
61
107

answered Jul 08 '15 at 17:35

Daniel Tronolone

69
1
1

3

But the whole point of doing `aws configure` is so that you don't have to put credentials in e.g. scripts. See answer posted by @chicks to solve this properly. – Madbreaks Mar 07 '17 at 06:53
5

Don't store `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` values in scripts. The first line should have already provided those values. – AWippler Feb 03 '18 at 14:18

score 3 · Answer 4 · answered May 31 '17 at 20:48

3

Put this code before your command line to be executed into crontab -e

SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

answered May 31 '17 at 20:48

Roberto Carlos Reyes Fernández

131
1

I tried the first solution with the diff but nothing. the trick for me was the PATH variable. – borracciaBlu Oct 28 '17 at 00:19
Adding the PATH variable was a simple solution to this problem for me. – the4thv Aug 26 '20 at 09:29

score 1 · Answer 5 · edited Feb 22 '17 at 14:21

The aws cli tool's binaries are installed under /usr/local/bin/aws.

The error I had is that the cron user could not access /usr/local/bin/aws while running; it can only access /usr/bin/

What I did was to create a link in /usr/bin for aws with the below command.

root@gateway:~# ln -s /usr/local/bin/aws /usr/bin/aws

I also added some changes in my script; here is a sample function:

starter () {
    echo "
    ==================================================

    Starting Instance

    ==================================================
    "

    /usr/bin/aws ec2 start-instances --instance-ids $instance --region us-east-1

    sleep 30

    echo "Assigning IP Address "

    /usr/bin/aws ec2 associate-address --instance-id $instance  --region us-east-1 --public-ip XX.XX.XX.XX

}

And the cron entry:

30 5 * * * sh /usr/local/cron/magentocron.sh

This method worked for me.

Mansur, your answer formatting is completely broken. – Aldekein Sep 27 '16 at 11:59 — Aldekein, Sep 27 '16 at 11:59
using full path `/usr/bin/aws` is key to solution. – Ramratan Gupta Mar 12 '18 at 01:28 — Ramratan Gupta, Mar 12 '18 at 01:28

Peter Gluck · Answer 6 · 2017-10-04T19:32:29.687

This line in the default .bashrc file for the user will prevent non-interactive shells from getting the full user environment (including the PATH variable):

# If not running interactively, don't do anything
[ -z "$PS1" ] && return

Comment the line out to allow $HOME/.bashrc to be executed from a non-interactive context.

I also had to add an explicit source command to my shell script to get the environment set up correctly:

#!/bin/bash
source $HOME/.bashrc

See this answer for additional info.

score 1 · Answer 7 · answered Jan 30 '19 at 16:38

1

We all know that environment path variable $PATH has location of binaries. $PATH of Crontab might not have location awscli.

What you can do is, find the path of awscli binary.

# which aws
/usr/local/bin/aws

and add the path in $PATH of crontab by adding below line in the beginning of your script(after shebang).

PATH=$PATH:/usr/local/bin/

This worked for me!!!

answered Jan 30 '19 at 16:38

Nijil

121
3

Your answer worked for me. Scratching my head for an hour. Thanks, buddy – Hussain7 Jun 25 '19 at 11:50

score 0 · Answer 8 · answered Aug 24 '20 at 01:15

Assuming you are using Ubuntu and trying to add a cronjob for the root user.

Add this to the top of your shell script export HOME=/root
Ensure when you use any aws command in your script you use the full path, ie. /usr/local/bin/aws s3 cp <from> <to>.
sudo mkdir /root/.aws
sudo cp ~/.aws/credentials /root/.aws/

score 0 · Answer 9 · answered Mar 23 '21 at 07:07

If you have attached IAM role for ECS Fargate task role then this solution will work

Add the following line in the entrypoint.sh declare -p | grep -Ev 'BASHOPTS|BASH_VERSINFO|EUID|PPID|SHELLOPTS|UID' > /container.env
Add below line in crontab or cron file SHELL=/bin/bash
BASH_ENV=/container.env

It worked for me.

score 0 · Answer 10 · answered Mar 29 '17 at 07:30

0

I know it's not the perfect solution but that worked for me:

export HOME=/home/user
export AWS_CONFIG_FILE="/home/user/.aws/config"
export AWS_ACCESS_KEY_ID=XXX
export AWS_SECRET_ACCESS_KEY=XXX

answered Mar 29 '17 at 07:30

Gustavo

101
2

score 0 · Answer 11 · edited Jul 24 '17 at 08:49

0

Just to add some value added i was having issue with new bash version while using awscli tool installed via PIP i found that nothing will work with this tool with new bash versions.

I was able to resolve by installing aws-apitools-ec2 this can be install by

yum install -y aws-apitools-ec2

I am attaching its guide for more reference.

http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ec2-clt.pdf

edited Jul 24 '17 at 08:49

Gerald Schneider

19,757
8
52
79

answered Jul 24 '17 at 07:54

Mansur Ul Hasan

264
3
9

on ubuntu 16.04 i couldn't find the package. – borracciaBlu Oct 28 '17 at 00:18

score 0 · Answer 12 · answered Aug 02 '18 at 04:51

I had the same issue, but after removing the stderr redirect from my cron entry (2>@1), I saw aws: command not found in the log.

This is because the AWS cli was installed in the user's home folder and I had added a line to my user's .bash_profile to add the AWS cli path to the $PATH. Oddly, this is in fact the way that the AWS cli install documentation tells you to install it. But the user's .bash_profile doesn't get used when the user's crontab is executed (at least not in my environment anyway).

So all I did to fix this was ensure my crontab script also had the aws cli in its path. So below the shebang of my script, I now have PATH=~/.local/bin:$PATH.

score 0 · Answer 13 · answered Aug 24 '18 at 10:43

For me this did the trick:

#!/bin/bash

HOME=/home/ubuntu
AWS_CONFIG_FILE="/home/ubuntu/.aws/config"

aws ec2 describe-instances #or whatever command you need to use.

The default user in todays EC2 instances is ubuntu, and the root folder is that users home folder. That's where the aws cli exists as well.

score 0 · Answer 14 · answered Oct 23 '18 at 11:48

0

Not the best, but I had to provide the config directly in my shell/bash script before the AWS client commands. like:

#!/bin/bash

export AWS_ACCESS_KEY_ID=<ZZZ>
export AWS_SECRET_ACCESS_KEY=<AAA>
export AWS_DEFAULT_REGION=<BBB>
aws s3 cp ....

answered Oct 23 '18 at 11:48

user1859675

101
1

Can't run AWS CLI from CRON (credentials)

14 Answers14

Linked