After a few reading about jvm memory (here, here, here, others I forgot...), I am expecting the resident set size of my java process to be roughly equal to the current heap space capacity. That's not what the numbers are saying, it seems to be roughly equal to the max heap space capacity:
Resident set size:
# echo 0 $(cat /proc/1/smaps | grep Rss | awk '{print $2}' | sed 's#^#+#') | bc
11507912
# ps -C java -O rss | gawk '{ count ++; sum += $2 }; END {count --; print "Number of processes =",count; print "Memory usage per process =",sum/1024/count, "MB"; print "Total memory usage =", sum/1024, "MB" ;};'
Number of processes = 1
Memory usage per process = 11237.8 MB
Total memory usage = 11237.8 MB
Java heap
# jmap -heap 1
Attaching to process ID 1, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.55-b03
using thread-local object allocation.
Garbage-First (G1) GC with 18 thread(s)
Heap Configuration:
MinHeapFreeRatio = 10
MaxHeapFreeRatio = 20
MaxHeapSize = 10737418240 (10240.0MB)
NewSize = 1363144 (1.2999954223632812MB)
MaxNewSize = 17592186044415 MB
OldSize = 5452592 (5.1999969482421875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 20971520 (20.0MB)
MaxPermSize = 85983232 (82.0MB)
G1HeapRegionSize = 2097152 (2.0MB)
Heap Usage:
G1 Heap:
regions = 2560
capacity = 5368709120 (5120.0MB)
used = 1672045416 (1594.586769104004MB)
free = 3696663704 (3525.413230895996MB)
31.144272834062576% used
G1 Young Generation:
Eden Space:
regions = 627
capacity = 3279945728 (3128.0MB)
used = 1314914304 (1254.0MB)
free = 1965031424 (1874.0MB)
40.089514066496164% used
Survivor Space:
regions = 49
capacity = 102760448 (98.0MB)
used = 102760448 (98.0MB)
free = 0 (0.0MB)
100.0% used
G1 Old Generation:
regions = 147
capacity = 1986002944 (1894.0MB)
used = 252273512 (240.5867691040039MB)
free = 1733729432 (1653.413230895996MB)
12.702574926293766% used
Perm Generation:
capacity = 39845888 (38.0MB)
used = 38884120 (37.082786560058594MB)
free = 961768 (0.9172134399414062MB)
97.58628042120682% used
14654 interned Strings occupying 2188928 bytes.
- Are my expectations wrong? What should I expect?
- I need the heap space to be able to grow during spikes (to avoid very slow Full GC), but I would like to have the resident set size as low as possible the rest of the time, to benefit the other processes running on the server. Is there a better way to achieve that?
Linux 3.13.0-32-generic x86_64
java version "1.7.0_55"
Running in Docker version 1.1.2
Java is running elasticsearch 1.2.0:
/usr/bin/java -Xms5g -Xmx10g -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -Xss256k -Djava.awt.headless=true -XX:+UseG1GC -XX:MaxGCPauseMillis=350 -XX:InitiatingHeapOccupancyPercent=45 -XX:+AggressiveOpts -XX:+UseCompressedOops -XX:-OmitStackTraceInFastThrow -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintClassHistogram -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:/opt/elasticsearch/logs/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt elasticsearch/logs/heapdump.hprof -XX:ErrorFile=/opt/elasticsearch/logs/hs_err.log -Des.logger.port=99999 -Des.logger.host=999.999.999.999 -Delasticsearch -Des.foreground=yes -Des.path.home=/opt/elasticsearch -cp :/opt/elasticsearch/lib/elasticsearch-1.2.0.jar:/opt/elasticsearch/lib/*:/opt/elasticsearch/lib/sigar/* org.elasticsearch.bootstrap.Elasticsearch
There actually are 5 elasticsearch nodes, each in a different docker container. All have about the same memory usage. Some stats about the index:
size: 9.71Gi (19.4Gi)
docs: 3,925,398 (4,052,694)
