1

I've installed the latest at the moment g-wan web server (4.3.14 x64) on a KVM VPS with Ubuntu 14.04.02 x64 server edition provided by Vultr ( 1GB RAM, 1 CPU core and 2GB swap ).

The VPS setup is absolute barebones - just Ubuntu and g-wan. I've copied g-wan to /opt/gwan/ and started it as a service by using the init scripts provided in the g-wan documentation. I've also disabled the csp folder (by renaming it to _csp) and haven't installed any additional scripting languages to work with it because I need it only as a static file server. Other than that I haven't made any additional customizations (except to enable the access and error logs).

Initially g-wan starts with a very low memory footprint - just a little above 4MB but only after a few hours it grows a lot more (7 hours after the last restart and now it's taking around 360MB) - the more time passes the more it grows and the machine becomes non-responsive (I suppose due to swapping) and the only solution is to reboot it.

Also I must note that this happens while the server is mostly idle as it is not used yet into production - I've seen only a few bot requests in the access and error logs.

I can provide additional info if required, just don't know what else to mention at this point.

//EDIT:

Here are the last 30 lines of the trace log:

Tue, 28 Jul 2015 08:08:55 GMT: start
Tue, 28 Jul 2015 08:08:56 GMT: start
Tue, 28 Jul 2015 08:08:57 GMT: start
Tue, 28 Jul 2015 08:08:58 GMT: start
Tue, 28 Jul 2015 08:09:00 GMT: start
Tue, 28 Jul 2015 08:09:01 GMT: start
Tue, 28 Jul 2015 08:09:02 GMT: start
Tue, 28 Jul 2015 08:09:03 GMT: start
Tue, 28 Jul 2015 08:09:04 GMT: start
Tue, 28 Jul 2015 08:09:06 GMT: start
Tue, 28 Jul 2015 08:09:07 GMT: start
Tue, 28 Jul 2015 08:09:08 GMT: start
Tue, 28 Jul 2015 08:09:09 GMT: start
Tue, 28 Jul 2015 08:09:10 GMT: start
Tue, 28 Jul 2015 08:09:11 GMT: start
Tue, 28 Jul 2015 08:09:13 GMT: start
Tue, 28 Jul 2015 08:09:14 GMT: start
Tue, 28 Jul 2015 08:09:15 GMT: start
Tue, 28 Jul 2015 08:09:16 GMT: start
Tue, 28 Jul 2015 08:09:18 GMT: start
Tue, 28 Jul 2015 08:09:19 GMT: start
Tue, 28 Jul 2015 08:09:20 GMT: start
Tue, 28 Jul 2015 08:09:20 GMT: signal 15: Termination
Tue, 28 Jul 2015 08:09:40 GMT: start
Tue, 28 Jul 2015 14:45:05 GMT: signal 2: Ctrl-C
Tue, 28 Jul 2015 14:45:41 GMT: start
Tue, 28 Jul 2015 17:56:47 GMT: signal 15: Termination
Tue, 28 Jul 2015 17:58:15 GMT: start
Tue, 28 Jul 2015 17:59:00 GMT: signal 15: Termination
Tue, 28 Jul 2015 17:59:11 GMT: start

And some explanation about them:

I've checked the whole log and it's filled only with lines containing the timestamp and start for each second. The first termination is when I stopped the service and started the server manually via ./gwan. The second termination Ctrl-C is for when I stopped the previous command (which didn't showed anything on the terminal while was running) and this time started it with ./gwan -d:www-data:www-data - again no problems whatsoever. I killed it again and started it as a service - again no start entries in the log on each second, though it has been running only for an hour so far.

The gwan.log file also had nothing different than the start/stop information and the memory consumption printed every second like this:

[Tue Jul 28 03:29:32 2015 GMT] Host /opt/gwan/0.0.0.0_8080/#0.0.0.0
[Tue Jul 28 03:29:32 2015 GMT] log files enabled
[Tue Jul 28 03:29:32 2015 GMT] memory footprint: 188.71 MiB.
[Tue Jul 28 03:29:33 2015 GMT] memory footprint: 187.56 MiB.
[Tue Jul 28 03:29:33 2015 GMT] Host /opt/gwan/0.0.0.0_8080/#0.0.0.0
[Tue Jul 28 03:29:33 2015 GMT] log files enabled
[Tue Jul 28 03:29:33 2015 GMT] memory footprint: 189.10 MiB.
[Tue Jul 28 03:29:34 2015 GMT] memory footprint: 187.58 MiB.
[Tue Jul 28 03:29:34 2015 GMT] Host /opt/gwan/0.0.0.0_8080/#0.0.0.0
[Tue Jul 28 03:29:34 2015 GMT] log files enabled
[Tue Jul 28 03:29:34 2015 GMT] memory footprint: 188.72 MiB.
[Tue Jul 28 03:29:35 2015 GMT] memory footprint: 187.61 MiB.
[Tue Jul 28 03:29:35 2015 GMT] Host /opt/gwan/0.0.0.0_8080/#0.0.0.0
[Tue Jul 28 03:29:35 2015 GMT] log files enabled
[Tue Jul 28 03:29:35 2015 GMT] memory footprint: 190.40 MiB.
[Tue Jul 28 03:29:36 2015 GMT] memory footprint: 187.61 MiB.
[Tue Jul 28 03:29:36 2015 GMT] Host /opt/gwan/0.0.0.0_8080/#0.0.0.0
[Tue Jul 28 03:29:36 2015 GMT] log files enabled
[Tue Jul 28 03:29:36 2015 GMT] memory footprint: 188.75 MiB.
[Tue Jul 28 03:29:37 2015 GMT] memory footprint: 187.63 MiB.
[Tue Jul 28 03:29:37 2015 GMT] Host /opt/gwan/0.0.0.0_8080/#0.0.0.0
[Tue Jul 28 03:29:37 2015 GMT] log files enabled
[Tue Jul 28 03:29:37 2015 GMT] memory footprint: 190.32 MiB.
[Tue Jul 28 03:29:38 2015 GMT] memory footprint: 187.66 MiB.
[Tue Jul 28 03:29:38 2015 GMT] Host /opt/gwan/0.0.0.0_8080/#0.0.0.0
[Tue Jul 28 03:29:38 2015 GMT] log files enabled
[Tue Jul 28 03:29:38 2015 GMT] memory footprint: 188.79 MiB.
[Tue Jul 28 03:29:39 2015 GMT] memory footprint: 187.67 MiB.
[Tue Jul 28 03:29:39 2015 GMT] Host /opt/gwan/0.0.0.0_8080/#0.0.0.0
[Tue Jul 28 03:29:39 2015 GMT] log files enabled
[Tue Jul 28 03:29:39 2015 GMT] memory footprint: 190.61 MiB.
[Tue Jul 28 03:29:41 2015 GMT] memory footprint: 187.69 MiB.
[Tue Jul 28 03:29:41 2015 GMT] Host /opt/gwan/0.0.0.0_8080/#0.0.0.0
[Tue Jul 28 03:29:41 2015 GMT] log files enabled
[Tue Jul 28 03:29:41 2015 GMT] memory footprint: 188.83 MiB.
[Tue Jul 28 03:29:42 2015 GMT] memory footprint: 187.70 MiB.
[Tue Jul 28 03:29:42 2015 GMT] Host /opt/gwan/0.0.0.0_8080/#0.0.0.0
[Tue Jul 28 03:29:42 2015 GMT] log files enabled
[Tue Jul 28 03:29:42 2015 GMT] memory footprint: 190.10 MiB.
[Tue Jul 28 03:29:43 2015 GMT] memory footprint: 187.71 MiB.
[Tue Jul 28 03:29:43 2015 GMT] Host /opt/gwan/0.0.0.0_8080/#0.0.0.0
[Tue Jul 28 03:29:43 2015 GMT] log files enabled
[Tue Jul 28 03:29:43 2015 GMT] memory footprint: 188.86 MiB.
[Tue Jul 28 03:29:44 2015 GMT] memory footprint: 187.73 MiB.
[Tue Jul 28 03:29:44 2015 GMT] Host /opt/gwan/0.0.0.0_8080/#0.0.0.0
[Tue Jul 28 03:29:44 2015 GMT] log files enabled
[Tue Jul 28 03:29:44 2015 GMT] memory footprint: 190.13 MiB.
[Tue Jul 28 03:29:45 2015 GMT] memory footprint: 187.76 MiB.
[Tue Jul 28 03:29:45 2015 GMT] Host /opt/gwan/0.0.0.0_8080/#0.0.0.0
[Tue Jul 28 03:29:45 2015 GMT] log files enabled
[Tue Jul 28 03:29:45 2015 GMT] memory footprint: 188.91 MiB.
[Tue Jul 28 03:29:46 2015 GMT] memory footprint: 187.74 MiB.
[Tue Jul 28 03:29:46 2015 GMT] Host /opt/gwan/0.0.0.0_8080/#0.0.0.0
[Tue Jul 28 03:29:46 2015 GMT] log files enabled
[Tue Jul 28 03:29:46 2015 GMT] memory footprint: 188.90 MiB.
[Tue Jul 28 03:29:47 2015 GMT] memory footprint: 187.79 MiB.
[Tue Jul 28 03:29:47 2015 GMT] Host /opt/gwan/0.0.0.0_8080/#0.0.0.0
[Tue Jul 28 03:29:47 2015 GMT] log files enabled
[Tue Jul 28 03:29:47 2015 GMT] memory footprint: 188.92 MiB.
[Tue Jul 28 03:29:48 2015 GMT] memory footprint: 187.79 MiB.
[Tue Jul 28 03:29:48 2015 GMT] Host /opt/gwan/0.0.0.0_8080/#0.0.0.0
[Tue Jul 28 03:29:48 2015 GMT] log files enabled
[Tue Jul 28 03:29:48 2015 GMT] memory footprint: 190.17 MiB.
[Tue Jul 28 03:29:50 2015 GMT] memory footprint: 187.82 MiB.
[Tue Jul 28 03:29:50 2015 GMT] Host /opt/gwan/0.0.0.0_8080/#0.0.0.0
[Tue Jul 28 03:29:50 2015 GMT] log files enabled
[Tue Jul 28 03:29:50 2015 GMT] memory footprint: 188.95 MiB.
[Tue Jul 28 03:29:51 2015 GMT] memory footprint: 187.84 MiB.
[Tue Jul 28 03:29:51 2015 GMT] Host /opt/gwan/0.0.0.0_8080/#0.0.0.0
[Tue Jul 28 03:29:51 2015 GMT] log files enabled
[Tue Jul 28 03:29:51 2015 GMT] memory footprint: 190.21 MiB.
[Tue Jul 28 03:29:52 2015 GMT] memory footprint: 187.84 MiB.
[Tue Jul 28 03:29:52 2015 GMT] Host /opt/gwan/0.0.0.0_8080/#0.0.0.0
[Tue Jul 28 03:29:52 2015 GMT] log files enabled
[Tue Jul 28 03:29:52 2015 GMT] memory footprint: 188.10 MiB.

//EDIT 2:

It seems that for some weird reason it starts logging the weird start events each night exactly at midnight. I started it as a service last night at Tue, 28 Jul 2015 19:31:22 GMT: start and it worked without a problem until midnight. Will test it later with changing the system date while running as a normal app.

Also, forgot to mention that there are no cronjobs defined at all in case any of them might interfere with g-wan.

//EDIT 3:

I've tried manually changing the system date while g-wan was running as a service and as a normal app (also disabled the ntp service before that) but it seems that g-wan doesn't recognize the changes because there is no log rotation in the folder where gwan.log is stored. Also there are no new entries in the trace log.

All of this made me think that the problem might be related to the ntp service somehow but this still doesn't explain why it occurs only at midnight while the ntp service updates the system time more frequently.

//EDIT 4:

Here's the hardware detection part from the last start of the web server (with the IP address and domain name altered). The server was started with service gwan start,though I suppose the log entry would've been the same if I started it with ./gwan

[Wed Jul 29 11:37:56 2015 GMT] ------------------------------------------------
[Wed Jul 29 11:37:56 2015 GMT] G-WAN 4.3.14 64-bit (Mar 14 2013 07:33:12)
[Wed Jul 29 11:37:56 2015 GMT] ------------------------------------------------
[Wed Jul 29 11:37:56 2015 GMT] Local Time: Wed, 29 Jul 2015 11:37:56 GMT+0
[Wed Jul 29 11:37:56 2015 GMT] RAM: (474.11 MiB free + 0 shared + 92.90 MiB buffers) / 993.92 MiB total
[Wed Jul 29 11:37:56 2015 GMT] Physical Pages: 474.11 MiB / 993.92 MiB
[Wed Jul 29 11:37:56 2015 GMT] DISK: 15.65 GiB free / 19.70 GiB total
[Wed Jul 29 11:37:56 2015 GMT]  Filesystem     Type      Size  Used Avail Use% Mounted on
[Wed Jul 29 11:37:56 2015 GMT]  /dev/vda1      ext3       20G  4.1G   15G  22% /
[Wed Jul 29 11:37:56 2015 GMT]  none           tmpfs     4.0K     0  4.0K   0% /sys/fs/cgroup
[Wed Jul 29 11:37:56 2015 GMT]  udev           devtmpfs  487M  4.0K  487M   1% /dev
[Wed Jul 29 11:37:56 2015 GMT]  tmpfs          tmpfs     100M  372K  100M   1% /run
[Wed Jul 29 11:37:56 2015 GMT]  none           tmpfs     5.0M     0  5.0M   0% /run/lock
[Wed Jul 29 11:37:56 2015 GMT]  none           tmpfs     497M     0  497M   0% /run/shm
[Wed Jul 29 11:37:56 2015 GMT]  none           tmpfs     100M     0  100M   0% /run/user
[Wed Jul 29 11:37:56 2015 GMT] 80 processes, including pid:2542 './gwan -d'
[Wed Jul 29 11:37:56 2015 GMT] Page-size:4,096 Child-max:7,780 Stream-max:16
[Wed Jul 29 11:37:56 2015 GMT] CPU: 1x Virtual CPU e7da7129d3ee
[Wed Jul 29 11:37:56 2015 GMT]  0 id: 0     0
[Wed Jul 29 11:37:56 2015 GMT] Cores: possible:0 present:0 online:0
[Wed Jul 29 11:37:56 2015 GMT] L1d cache:   32K line:64     0
[Wed Jul 29 11:37:56 2015 GMT] L1i cache:   32K line:64     0
[Wed Jul 29 11:37:56 2015 GMT] L2  cache: 4096K line:64     0
[Wed Jul 29 11:37:56 2015 GMT] NUMA node #1 0
[Wed Jul 29 11:37:56 2015 GMT] CPU(s):1, Core(s)/CPU:0, Thread(s)/Core:2
[Wed Jul 29 11:37:56 2015 GMT] Bogomips: 4,799.99
[Wed Jul 29 11:37:56 2015 GMT] Hypervisor: KVMKVMKVM
[Wed Jul 29 11:37:56 2015 GMT]   using   1 workers 0[1]0
[Wed Jul 29 11:37:56 2015 GMT]   among   2 threads 0[]1
[Wed Jul 29 11:37:56 2015 GMT] 64-bit little-endian (least significant byte first)
[Wed Jul 29 11:37:56 2015 GMT] Ubuntu 14.04.2 LTS \n \l (3.13.0-58) 64-bit
[Wed Jul 29 11:37:56 2015 GMT] user: root (uid:0), group: root (uid:0)
[Wed Jul 29 11:37:56 2015 GMT] system  fd_max: 1,024
[Wed Jul 29 11:37:56 2015 GMT] program fd_max: 1,024
[Wed Jul 29 11:37:56 2015 GMT] updated fd_max: 500,000
[Wed Jul 29 11:37:56 2015 GMT] Available network interfaces (2):
[Wed Jul 29 11:37:56 2015 GMT] 127.0.0.1
[Wed Jul 29 11:37:56 2015 GMT] 1.2.3.4
[Wed Jul 29 11:37:56 2015 GMT] memory footprint: 1.33 MiB.
[Wed Jul 29 11:37:56 2015 GMT] Host /opt/gwan/0.0.0.0_80/#domain.com
[Wed Jul 29 11:37:56 2015 GMT] log files enabled
[Wed Jul 29 11:37:56 2015 GMT] memory footprint: 3.93 MiB.

//EDIT 5:

Tonight I left it running as a normal process (with ./gwan) and there were no problems at midnight so the problem might be from the daemon mode.

Guess I can run it this way at startup with cron if no fix exists for the service mode.

  • Since you have no script (init, maintenance, servlet, handler) your G-WAN setup is used for static contents... Now, regarding the problem itself, a few things like a dump of the "gwan/logs/gwan.log" and "gwan/trace" files would help to shed some more light on what is happening. Also, running as an application rather than as a service would let you get more information in the parent terminal in case of errors. – Gil Jul 28 '15 at 08:00
  • @Gil - I've added the interesting stuff from trace and gwan.log. I've looked through all the log files since the server was installed and there were no errors at all except one from the first run when I still didn't have gcc installed. Let me know if you need anything else. – Sledge Hammer Jul 28 '15 at 18:56
  • The gwan.log [edited] part that contains your HW detection is of crutial importance... so please publish it. This will explain why the program consumes almost 200 MB of RAM and silently crash. – Gil Jul 29 '15 at 10:50
  • @Gil - The hardware detection is published in my latest edit. I'll add more info if needed. – Sledge Hammer Jul 29 '15 at 11:49

1 Answers1

1
CPU: 1x Virtual CPU e7da7129d3ee
[Wed Jul 29 11:37:56 2015 GMT] Cores: possible:0 present:0 online:0
CPU(s):1, Core(s)/CPU:0, Thread(s)/Core:2
Hypervisor: KVMKVMKVM
using   1 workers 0[1]0
among   2 threads 0[]1

Look no further, one day the KVM hypervisor might get CPU virtualization right by updating the /proc/sys kernel variables correctly...

In the meantime, KVM will break every multicore application, like G-WAN.

The "fix" is to run G-WAN like this: # ./gwan -g -w 1 -d

This will force using 1 CPU Core, despite bogus system information.

Gil
  • 307
  • 3
  • 12
  • Thanks! I've added a new line to the crontab and so far it has worked for more than 24 hours without any problems - `@reboot /opt/gwan/./gwan`. I'll alter it with the parameters you gave me (or edit the init script which might be the better idea) if the problem reappears. – Sledge Hammer Jul 31 '15 at 10:49