0

Database is going to recovery mode whenever the OOM killer is invoked on the postgres process.

/var/log/messages output with respect to the issue

oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-137004.scope,task=postgres,pid=4193383,uid=1001 Mar 25 15:55:21 kernel: Out of memory: Killed process 4193383 (postgres) total-vm:13226668kB, anon-rss:172912kB, file-rss:4kB, shmem-rss:10194068kB, UID:1001

postgres logs output with respect to the issue

postgresql log FATAL error : postgresql-2021-03-25_000000.log:2021-03-25 15:55:26 IST [54678]: [1-1] [unknown] 10.10.10.10. db=dbname ,user=username FATAL: the database system is in recovery mode

sysctl -a command output

vm.drop_caches = 3 vm.extfrag_threshold = 500 vm.hugetlb_shm_group = 0 vm.laptop_mode = 0 vm.legacy_va_layout = 0 vm.lowmem_reserve_ratio = 256 256 32 0 0 vm.max_map_count = 65530 vm.memory_failure_early_kill = 0 vm.memory_failure_recovery = 1 vm.min_free_kbytes = 67584 vm.min_slab_ratio = 5 vm.min_unmapped_ratio = 1 vm.mmap_min_addr = 4096 vm.mmap_rnd_bits = 28 vm.mmap_rnd_compat_bits = 8 vm.nr_hugepages = 0 vm.nr_hugepages_mempolicy = 0 vm.nr_overcommit_hugepages = 0 vm.numa_stat = 1 vm.numa_zonelist_order = Node vm.oom_dump_tasks = 1 vm.oom_kill_allocating_task = 0 vm.overcommit_kbytes = 0 vm.overcommit_memory = 0 vm.overcommit_ratio = 50 vm.page-cluster = 3 vm.panic_on_oom = 0 vm.percpu_pagelist_fraction = 0 vm.stat_interval = 1 vm.swappiness = 30 vm.user_reserve_kbytes = 131072 vm.vfs_cache_pressure = 100 vm.watermark_scale_factor = 10 vm.zone_reclaim_mode = 0

Need to understand what exactly is causing the issue here?

Sandesh Bailur
  • 1
  • 1
    What else are you running? If one process is being killed by OOM it is often the "fatty" one, but it does not always means that it's the problematic one. E.g. quite often the problem in web stacks is with too many workers allocated to e.g. PHP-FPM. – Danila Vershinin Mar 26 '21 at 05:36
  • This is the database server, Postgres is a database process no other process are being killed. By adding kernel parameters should solve the issue? – Sandesh Bailur Mar 26 '21 at 06:07
  • 3
    The problem is not that OOM killer is targeting PostgreSQL, it's that the OOM killer is invoked. Hence Danila's question of what else is running on the system? If PostgreSQL is the only service running on that host, it either needs more RAM or you need to tune PostgreSQL's settings so it uses less RAM. – Ginnungagap Mar 26 '21 at 08:05
  • Thanks for the response. antivirus is running and also the software (a monitoring tool for the server) I will stop these processes and check and also in the meanwhile we can increase the RAM as well. But it is certain that the postgres was using the highest memory out of all. is this because of some heavy queries running on the system? – Sandesh Bailur Mar 26 '21 at 17:58

0 Answers0