We have a 3-box cluster Kafka+Zookeeper, in a virtualized enviroment (in Windows 2019 HyperV).
In the machines' config dynamic memory is disabled, and they have 4G RAM each (I know it's a very few amount for this kind of cluster, but the usage is very light)
- Guest SO is Debian stretch 9.3, kernel 4.9.0-5-amd64
- Kafka is 2.11-2.3.0
- Zookeeper is 3.4.9
Once you start the cluster or restart a node, ram usage it's normal:
# free -m
total used free shared buff/cache available
Mem: 3937 2200 198 40 1538 1462
Swap: 1903 0 1903
After a week ram usage gets normalized (50% ram usage). But after 2 weeks starts to drain ram slowly. Until a few days later when physical free percent reaches 15, that's when it falls sharply (and quickly), starting to eat swap until it gets completely allocated.
# free -m
total used free shared buff/cache available
Mem: 3937 3744 117 3 76 39
Swap: 1903 1197 706
I've installed smem and here you can see the difference between both situations:
(NOTE: right and wrong are different machines, but config and behavior over the time are the same)
RIGHT:
# smem -tw
Area Used Cache Noncache
firmware/hardware 0 0 0
kernel image 0 0 0
kernel dynamic memory 2777144 1983260 793884
userspace memory 1027116 44548 982568
free memory 228016 228016 0
----------------------------------------------------------
4032276 2255824 1776452
WRONG:
# smem -tw
Area Used Cache Noncache
firmware/hardware 0 0 0
kernel image 0 0 0
kernel dynamic memory 2763772 64968 2698804
userspace memory 1138972 35928 1103044
free memory 129532 129532 0
----------------------------------------------------------
4032276 230428 3801848
Most of the memory is in kernel dynamic memory/Noncache, I'm not sure what it implies nor the reason.
Through the monitoring we've seen that (in /etc/meminfo):
- DirectMap4k grows
- MemFree decreases sharply when it starts to eat swap
- MemAvailable decreases slowly
- Most of the other ones decrease
Here's a pic of our monitoring tool:
Adding all processes' Ram/Swap:
RIGHT:
# for file in /proc/*/status ; do awk -F ' ' '/VmRSS/ {print $2}' $file; done | awk '{sum += $1} END {print sum/1024 "Mb"}'
1053.5Mb
# for file in /proc/*/status ; do awk -F ' ' '/VmSwap/ {print $2}' $file; done | awk '{sum += $1} END {print sum/1024 "Mb"}'
0Mb
WRONG:
# for file in /proc/*/status ; do awk -F ' ' '/VmRSS/ {print $2}' $file; done | awk '{sum += $1} END {print sum/1024 "Mb"}'
308.941Mb
# for file in /proc/*/status ; do awk -F ' ' '/VmSwap/ {print $2}' $file; done | awk '{sum += $1} END {print sum/1024 "Mb"}'
1147.54Mb
top -c, sorted by Mem:
RIGHT:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
674 kafka 20 0 17.229g 818152 19956 S 12.1 20.3 3304:07 java -Xmx2048m -Xms512m -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesCo+
734 zookeep+ 20 0 4838708 172708 16736 S 0.0 4.3 81:41.62 java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,CONSOLE -cp /opt/zookeeper/bin/../build/classes:/opt/zooke+
676 Debian-+ 20 0 58368 10636 6236 S 0.0 0.3 16:08.80 /usr/sbin/snmpd -LSwd -Lf /dev/null -u Debian-snmp -g Debian-snmp -p /run/snmpd.pid -f
1 root 20 0 57124 6900 5320 S 0.0 0.2 34:29.32 /sbin/init
116398 root 20 0 95168 6812 5840 S 0.3 0.2 0:00.34 sshd: root@pts/0
WRONG:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2732 kafka 20 0 17.269g 235004 0 S 6.9 5.8 1259:50 java -Xmx2048m -Xms512m -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesCo+
707 root 20 0 1955516 20720 6896 S 2.8 0.5 52:04.15 /usr/share/filebeat/bin/filebeat -environment systemd -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /et+
2537 zookeep+ 20 0 4437272 8944 0 S 0.0 0.2 20:29.38 java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,CONSOLE -cp /opt/zookeeper/bin/../build/classes:/opt/zooke+
1 root 20 0 57292 3688 2704 S 0.0 0.1 29:28.41 /sbin/init
This being said, I would appreciate if anyone could point me how to get more evidences to locate the root cause of this.
Thank you.
Edit (2020/06/26 Add some outputs):
RIGHT:
# cat /proc/meminfo
MemTotal: 4032276 kB
MemFree: 155480 kB
MemAvailable: 1925864 kB
Buffers: 679696 kB
Cached: 1293012 kB
SwapCached: 0 kB
Active: 2013308 kB
Inactive: 985748 kB
Active(anon): 867464 kB
Inactive(anon): 200204 kB
Active(file): 1145844 kB
Inactive(file): 785544 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 1949692 kB
SwapFree: 1949692 kB
Dirty: 160 kB
Writeback: 0 kB
AnonPages: 1026380 kB
Mapped: 49320 kB
Shmem: 41332 kB
Slab: 129624 kB
SReclaimable: 81024 kB
SUnreclaim: 48600 kB
KernelStack: 6592 kB
PageTables: 5936 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3965828 kB
Committed_AS: 1151652 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 1368000 kB
DirectMap2M: 2826240 kB
DirectMap1G: 0 kB
# ps -ef | grep java
kafka 674 1 10 Jun03 ? 2-09:20:08 java -Xmx2048m -Xms512m -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true -Xloggc:/opt/kafka/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/opt/kafka/bin/../logs -Dlog4j.configuration=file:/opt/kafka/bin/../config/log4j.properties -cp /opt/kafka/bin/../libs/activation-1.1.1.jar:/opt/kafka/bin/../libs/aopalliance-repackaged-2.5.0.jar:/opt/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/bin/../libs/connect-api-2.3.0.jar:/opt/kafka/bin/../libs/connect-basic-auth-extension-2.3.0.jar:/opt/kafka/bin/../libs/connect-file-2.3.0.jar:/opt/kafka/bin/../libs/connect-json-2.3.0.jar:/opt/kafka/bin/../libs/connect-runtime-2.3.0.jar:/opt/kafka/bin/../libs/connect-transforms-2.3.0.jar:/opt/kafka/bin/../libs/guava-20.0.jar:/opt/kafka/bin/../libs/hk2-api-2.5.0.jar:/opt/kafka/bin/../libs/hk2-locator-2.5.0.jar:/opt/kafka/bin/../libs/hk2-utils-2.5.0.jar:/opt/kafka/bin/../libs/jackson-annotations-2.9.9.jar:/opt/kafka/bin/../libs/jackson-core-2.9.9.jar:/opt/kafka/bin/../libs/jackson-databind-2.9.9.jar:/opt/kafka/bin/../libs/jackson-dataformat-csv-2.9.9.jar:/opt/kafka/bin/../libs/jackson-datatype-jdk8-2.9.9.jar:/opt/kafka/bin/../libs/jackson-jaxrs-base-2.9.9.jar:/opt/kafka/bin/../libs/jackson-jaxrs-json-provider-2.9.9.jar:/opt/kafka/bin/../libs/jackson-module-jaxb-annotations-2.9.9.jar:/opt/kafka/bin/../libs/jackson-module-paranamer-2.9.9.jar:/opt/kafka/bin/../libs/jackson-module-scala_2.11-2.9.9.jar:/opt/kafka/bin/../libs/jakarta.annotation-api-1.3.4.jar:/opt/kafka/bin/../libs/jakarta.inject-2.5.0.jar:/opt/kafka/bin/../libs/jakarta.ws.rs-api-2.1.5.jar:/opt/kafka/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/bin/../libs/jersey-client-2.28.jar:/opt/kafka/bin/../libs/jersey-common-2.28.jar:/opt/kafka/bin/../libs/jersey-container-servlet-2.28.jar:/opt/kafka/bin/../libs/jersey-container-servlet-core-2.28.jar:/opt/kafka/bin/../libs/jersey-hk2-2.28.jar:/opt/kafka/bin/../libs/jersey-media-jaxb-2.28.jar:/opt/kafka/bin/../libs/jersey-server-2.28.jar:/opt/kafka/bin/../libs/jetty-client-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-continuation-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-http-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-io-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-security-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-server-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-servlet-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-servlets-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-util-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/bin/../libs/jsr305-3.0.2.jar:/opt/kafka/bin/../libs/kafka_2.11-2.3.0.jar:/opt/kafka/bin/../libs/kafka_2.11-2.3.0-sources.jar:/opt/kafka/bin/../libs/kafka-clients-2.3.0.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-2.3.0.jar:/opt/kafka/bin/../libs/kafka-streams-2.3.0.jar:/opt/kafka/bin/../libs/kafka-streams-examples-2.3.0.jar:/opt/kafka/bin/../libs/kafka-streams-scala_2.11-2.3.0.jar:/opt/kafka/bin/../libs/kafka-streams-test-utils-2.3.0.jar:/opt/kafka/bin/../libs/kafka-tools-2.3.0.jar:/opt/kafka/bin/../libs/log4j-1.2.17.jar:/opt/kafka/bin/../libs/lz4-java-1.6.0.jar:/opt/kafka/bin/../libs/maven-artifact-3.6.1.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/bin/../libs/paranamer-2.8.jar:/opt/kafka/bin/../libs/plexus-utils-3.2.0.jar:/opt/kafka/bin/../libs/reflections-0.9.11.jar:/opt/kafka/bin/../libs/rocksdbjni-5.18.3.jar:/opt/kafka/bin/../libs/scala-library-2.11.12.jar:/opt/kafka/bin/../libs/scala-logging_2.11-3.9.0.jar:/opt/kafka/bin/../libs/scala-reflect-2.11.12.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.26.jar:/opt/kafka/bin/../libs/slf4j-log4j12-1.7.26.jar:/opt/kafka/bin/../libs/snappy-java-1.1.7.3.jar:/opt/kafka/bin/../libs/spotbugs-annotations-3.1.9.jar:/opt/kafka/bin/../libs/validation-api-2.0.1.Final.jar:/opt/kafka/bin/../libs/zkclient-0.11.jar:/opt/kafka/bin/../libs/zookeeper-3.4.14.jar:/opt/kafka/bin/../libs/zstd-jni-1.4.0-1.jar kafka.Kafka /opt/kafka/config/server.properties
zookeep+ 734 1 0 Jun03 ? 01:28:30 java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,CONSOLE -cp /opt/zookeeper/bin/../build/classes:/opt/zookeeper/bin/../build/lib/*.jar:/opt/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/opt/zookeeper/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper/bin/../lib/jline-0.9.94.jar:/opt/zookeeper/bin/../zookeeper-3.4.9.jar:/opt/zookeeper/bin/../src/java/lib/*.jar:/opt/zookeeper/bin/../conf: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /opt/zookeeper/conf/zoo.cfg
WRONG:
# cat /proc/meminfo
MemTotal: 4032276 kB
MemFree: 141820 kB
MemAvailable: 170960 kB
Buffers: 104896 kB
Cached: 150488 kB
SwapCached: 129780 kB
Active: 1872244 kB
Inactive: 798372 kB
Active(anon): 1709660 kB
Inactive(anon): 746052 kB
Active(file): 162584 kB
Inactive(file): 52320 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 1949692 kB
SwapFree: 1412724 kB
Dirty: 72 kB
Writeback: 0 kB
AnonPages: 2286048 kB
Mapped: 39720 kB
Shmem: 40468 kB
Slab: 72456 kB
SReclaimable: 31544 kB
SUnreclaim: 40912 kB
KernelStack: 8360 kB
PageTables: 9968 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3965828 kB
Committed_AS: 4008680 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 3798976 kB
DirectMap2M: 395264 kB
DirectMap1G: 0 kB
# ps -ef | grep java
zookeep+ 2537 1 0 Jun08 ? 00:23:35 java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,CONSOLE -cp /opt/zookeeper/bin/../build/classes:/opt/zookeeper/bin/../build/lib/*.jar:/opt/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/opt/zookeeper/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper/bin/../lib/jline-0.9.94.jar:/opt/zookeeper/bin/../zookeeper-3.4.9.jar:/opt/zookeeper/bin/../src/java/lib/*.jar:/opt/zookeeper/bin/../conf: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /opt/zookeeper/conf/zoo.cfg
kafka 2732 1 5 Jun08 ? 22:06:19 java -Xmx2048m -Xms512m -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true -Xloggc:/opt/kafka/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/opt/kafka/bin/../logs -Dlog4j.configuration=file:/opt/kafka/bin/../config/log4j.properties -cp /opt/kafka/bin/../libs/activation-1.1.1.jar:/opt/kafka/bin/../libs/aopalliance-repackaged-2.5.0.jar:/opt/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/bin/../libs/connect-api-2.3.0.jar:/opt/kafka/bin/../libs/connect-basic-auth-extension-2.3.0.jar:/opt/kafka/bin/../libs/connect-file-2.3.0.jar:/opt/kafka/bin/../libs/connect-json-2.3.0.jar:/opt/kafka/bin/../libs/connect-runtime-2.3.0.jar:/opt/kafka/bin/../libs/connect-transforms-2.3.0.jar:/opt/kafka/bin/../libs/guava-20.0.jar:/opt/kafka/bin/../libs/hk2-api-2.5.0.jar:/opt/kafka/bin/../libs/hk2-locator-2.5.0.jar:/opt/kafka/bin/../libs/hk2-utils-2.5.0.jar:/opt/kafka/bin/../libs/jackson-annotations-2.9.9.jar:/opt/kafka/bin/../libs/jackson-core-2.9.9.jar:/opt/kafka/bin/../libs/jackson-databind-2.9.9.jar:/opt/kafka/bin/../libs/jackson-dataformat-csv-2.9.9.jar:/opt/kafka/bin/../libs/jackson-datatype-jdk8-2.9.9.jar:/opt/kafka/bin/../libs/jackson-jaxrs-base-2.9.9.jar:/opt/kafka/bin/../libs/jackson-jaxrs-json-provider-2.9.9.jar:/opt/kafka/bin/../libs/jackson-module-jaxb-annotations-2.9.9.jar:/opt/kafka/bin/../libs/jackson-module-paranamer-2.9.9.jar:/opt/kafka/bin/../libs/jackson-module-scala_2.11-2.9.9.jar:/opt/kafka/bin/../libs/jakarta.annotation-api-1.3.4.jar:/opt/kafka/bin/../libs/jakarta.inject-2.5.0.jar:/opt/kafka/bin/../libs/jakarta.ws.rs-api-2.1.5.jar:/opt/kafka/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/bin/../libs/jersey-client-2.28.jar:/opt/kafka/bin/../libs/jersey-common-2.28.jar:/opt/kafka/bin/../libs/jersey-container-servlet-2.28.jar:/opt/kafka/bin/../libs/jersey-container-servlet-core-2.28.jar:/opt/kafka/bin/../libs/jersey-hk2-2.28.jar:/opt/kafka/bin/../libs/jersey-media-jaxb-2.28.jar:/opt/kafka/bin/../libs/jersey-server-2.28.jar:/opt/kafka/bin/../libs/jetty-client-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-continuation-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-http-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-io-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-security-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-server-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-servlet-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-servlets-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-util-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/bin/../libs/jsr305-3.0.2.jar:/opt/kafka/bin/../libs/kafka_2.11-2.3.0.jar:/opt/kafka/bin/../libs/kafka_2.11-2.3.0-sources.jar:/opt/kafka/bin/../libs/kafka-clients-2.3.0.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-2.3.0.jar:/opt/kafka/bin/../libs/kafka-streams-2.3.0.jar:/opt/kafka/bin/../libs/kafka-streams-examples-2.3.0.jar:/opt/kafka/bin/../libs/kafka-streams-scala_2.11-2.3.0.jar:/opt/kafka/bin/../libs/kafka-streams-test-utils-2.3.0.jar:/opt/kafka/bin/../libs/kafka-tools-2.3.0.jar:/opt/kafka/bin/../libs/log4j-1.2.17.jar:/opt/kafka/bin/../libs/lz4-java-1.6.0.jar:/opt/kafka/bin/../libs/maven-artifact-3.6.1.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/bin/../libs/paranamer-2.8.jar:/opt/kafka/bin/../libs/plexus-utils-3.2.0.jar:/opt/kafka/bin/../libs/reflections-0.9.11.jar:/opt/kafka/bin/../libs/rocksdbjni-5.18.3.jar:/opt/kafka/bin/../libs/scala-library-2.11.12.jar:/opt/kafka/bin/../libs/scala-logging_2.11-3.9.0.jar:/opt/kafka/bin/../libs/scala-reflect-2.11.12.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.26.jar:/opt/kafka/bin/../libs/slf4j-log4j12-1.7.26.jar:/opt/kafka/bin/../libs/snappy-java-1.1.7.3.jar:/opt/kafka/bin/../libs/spotbugs-annotations-3.1.9.jar:/opt/kafka/bin/../libs/validation-api-2.0.1.Final.jar:/opt/kafka/bin/../libs/zkclient-0.11.jar:/opt/kafka/bin/../libs/zookeeper-3.4.14.jar:/opt/kafka/bin/../libs/zstd-jni-1.4.0-1.jar kafka.Kafka /opt/kafka/config/server.properties
For ease of view (kafka's params) :
-Xmx2048m
-Xms512m
-XX:+UseG1GC
-XX:MaxGCPauseMillis=20
-XX:InitiatingHeapOccupancyPercent=35
-XX:+ExplicitGCInvokesConcurrent
-Djava.awt.headless=true
-Xloggc:/opt/kafka/bin/../logs/kafkaServer-gc.log
-verbose:gc
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=100M
I'm not sure if the Java mem's values are optimal for this scenario, but I tried to configure them a bit restrained.
