1

I am having troubles getting a cronjob on ubuntu 16.04 digital ocean to work.

I programmed a python spider, which i want to run every 5 minutes. In order to run this spider I made a script runmyspider.sh (chmod +x) with the command:

scrapy runspider aspider.py

Now I want to call this script via Cron. 

*/5 * * * * sh /scripts/runmyspider.sh 2>&1 /scripts/spider.log

However, the spider never runs (I can see that no changes in the database have been made, if I execute the file manually the changes happen)

What am I doing wrong here? I have set up a cron multiple times before, but this time I seem to get an error into it...

Thanks for all your advice!

Tom
  • 133
  • 1
  • 1
  • 5
  • What you see in `/var/log/cron.log` and `sysog`? – Alexander Tolkachev Aug 24 '17 at 19:40
  • the spider. log file is empty? – Gnudiff Aug 24 '17 at 20:10
  • Did you try to define the `PATH` environment variable? – dkokmadis Aug 24 '17 at 21:26
  • @AlexanderTolkachev /var/log/cron.log doesnt exists, when i do cat syslog | grep "cron" I get: Aug 24 17:17:01 ubuntu-512mb-fra1-01 CRON[15149]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Aug 24 18:17:01 ubuntu-512mb-fra1-01 CRON[15700]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Aug 24 19:17:01 ubuntu-512mb-fra1-01 CRON[16190]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) – Tom Aug 24 '17 at 21:56
  • @Gnudiff yes the log is empty :( – Tom Aug 24 '17 at 21:56
  • @spilia no i havent, how would i do that though? Add the path variable to the ~./profile file? – Tom Aug 24 '17 at 21:59
  • 1
    Do you add cron record via `crontab -e` command or via creating a new file in `/etc/cron.d/` directory? And check that crond service is running by command `systemctl status crond`. – Mikhail Khirgiy Aug 25 '17 at 04:31
  • @MikhailKhirgiy I added it via crontab -e. service cron status says it is active – Tom Aug 25 '17 at 18:36

1 Answers1

0
  1. Always use absolute paths to avoid problems.

/bin/sh (or /usr/bin/?)

/usr/bin/scrapy (or wherever it is, you can use which scrapy to find the binary)

and most importantly add script path for the aspider.py file for scrapy to use, I guess it could be /scripts/aspider.py ?

My first guess is that cron is running, finds sh, which finds scrapy which doesn't find the py file.

  1. Don't overwrite the logfile, add to it:

*/5 * * * * /bin/sh /scripts/runmyspider.sh >> /scripts/spider.log 2>&1

Gnudiff
  • 533
  • 5
  • 20
  • Alternatively you could do cd /scripts in the runmyspider.sh file, but that is generally not a very good idea. – Gnudiff Aug 25 '17 at 06:58
  • I agree, using an absolute path is good, or you van put /usr/bin/ in the PATH. – Lexib0y Aug 25 '17 at 10:06
  • /usr/bin/ can be expected to be in path, however, it is good practice nevertheless – Gnudiff Aug 25 '17 at 11:04
  • thanks for your time and answer. I update the cronjob via crontab -e to: */5 * * * * /usr/local/bin/scrapy /scripts/crawler.py >> /scripts/crawler.log 2>&1 . still no result. service cron status says cron is running – Tom Aug 25 '17 at 18:35
  • @Tom why is it crawler.py if your initial post said it is aspider.py? – Gnudiff Aug 26 '17 at 20:46
  • Also, do a "ls -l /scripts/" please, and tell under which user you are executing "crontab -e" – Gnudiff Aug 27 '17 at 06:19