I`ve setup a small python program that uses a websocket connection to acquire data from an API and write it to a postgresql database.
There are only two (user) programs running: the websocket connection that receives data and write it to the database, and another program that is basic a while loop that runs every 15 seconds and check if data is being written.
Both programs are daemonized with supervisor and when no data is being written for 15 seconds, supervisor is restarted (to handle dead websocket connection).
Also I was (very) lazy and used Django ORM to do the db connection for me instead of psycopg2 directly.
It works, but I'm having a steady high cpu load on the server. It is a 1CPU 1Gb memory server (AWS micro). Top command outputs the following:
top - 17:10:58 up 19 days, 15:03, 1 user, load average: 1,57, 1,63, 1,58
Tasks: 116 total, 1 running, 115 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0,3 us, 0,3 sy, 0,0 ni, 0,0 id, 99,0 wa, 0,0 hi, 0,0 si, 0,3 st
KiB Mem : 1014552 total, 63440 free, 86572 used, 864540 buff/cache
KiB Swap: 1048572 total, 987380 free, 61192 used. 615096 avail Mem
Well, system is laggish and ocasionally crashes.
I can see that this is caused by a high IO load (99 wa). Lots of sleeping processes. But I`m writing an average of 400MB per day only on the database.
I've tried modding Postgresql config for high writing load following documentation and setting up a swap memory file (1GB), but those didn`t help on easing the load average.
So, for a non-experienced user like me, I don't know if this is the load I should expect. Can I optimize this setup? If I remove django ORM and write the program using only psycopg2, would it improve? Should I make a different check method for dead websocket connections? Is there any server config that can optimize for this need case?
Thanks!