0

I am executing python script on linux server, and it works perfectly if started manually. It downloads data from SQL server, adds new columns, saves, and loads. It executes one function after another. Below is the part of code from main.py:

balance_histories_part_1_and_2()
user_shift_actions_part_1()
sex_id_correct()
user_shift_actions_part_2_and_3()
user_shift_actions_part_5_and_6()
user_shift_actions_part_ML()

When it is executed manually, what I see in top -u mluser is:

top -u manual execution

But when I start using crontab, it breaks down all the time (the process is killed):

[Tue Aug  9 08:43:15 2022] Out of memory: Killed process 10806 (python3) total-vm:4445212kB, anon-rss:3906016kB, file-rss:60kB, shmem-rss:0kB, UID:1001 pgtables:7952kB oom_score_adj:0

And there are multiple python processes running:

top-u with while using cron

There are also many python scripts consisting only of one file, and they run in crontab without a problem. What can possibly solve the issue? May be crontab is not good at all for complex scripts and Apache should do better?

We have 16 Gb of Ram on server, but seems the problem is different.

Gerald Schneider
  • 19,757
  • 8
  • 52
  • 79
Nikita Tsekhanovich
  • 1
  • What does your crontab file look like? And Apache is a web server; not a replacement for cron. – vidarlo Aug 09 '22 at 09:35
  • [Please don't repost your question](https://serverfault.com/questions/1107478/why-had-the-process-being-killed-in-cron-but-when-in-works-fine-if-started-manu). – vidarlo Aug 09 '22 at 09:40
  • * */1 * * * /usr/bin/python3 /home/mluser/Model_building/main.py – Nikita Tsekhanovich Aug 09 '22 at 09:49
  • Note that cron is a very simple scheduler and when your batch should be started every hour, cron will start that batch every hour. And it will do so without checking if a previous batch completed. When the first batch is not completed when the next iteration is started you often either get resource deadlocks or contention and both will run more slowly. If both are still running when the third batch is started, three batches will run concurrently, etc. etc. often until the server runs out of resources. - decrease the frequency with which you run your batch of do something even smarter: – HBruijn Aug 09 '22 at 12:13

0 Answers0