When setting up Elasticsearch cluster on Kubernetes with 2x data, 2x master and 2x client nodes (each container on sepearete physical node) i get the following error in client-node
:
[2019-01-28T12:25:08,574][WARN ][o.e.d.z.UnicastZenPing ] [elasticsearch-client-5f7759d57c-95wwm] [81] failed send ping to {#zen_unicast_elasticsearch-discovery_0#}{LuD6SiE5RCG7Ol9xb0LBFw}{elasticsearch-discovery}{10.233.12.222:9300}
java.lang.IllegalStateException: handshake failed, mismatched cluster name [Cluster [elasticsearch-default]] - {#zen_unicast_elasticsearch-discovery_0#}{LuD6SiE5RCG7Ol9xb0LBFw}{elasticsearch-discovery}{10.233.12.222:9300}
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:406) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:369) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.discovery.zen.UnicastZenPing$PingingRound.getOrConnect(UnicastZenPing.java:400) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.discovery.zen.UnicastZenPing$3.doRun(UnicastZenPing.java:503) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.3.jar:6.2.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
This error does not have any sense because i dont have cluster name [elasticsearch-default]..
Also, the client node is responding when checking against elasticsearch api endpoint and show correct cluster name that is set in the config:
{
"name" : "elasticsearch-client-5f7759d57c-q2vd5",
"cluster_name" : "elasticsearch-test2",
"cluster_uuid" : "_na_",
"version" : {
"number" : "6.2.3",
"build_hash" : "c59ff00",
"build_date" : "2018-03-13T10:06:29.741383Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
But when trying healthcheck endpoint it responds with:
curl -X GET "10.233.43.73:9200/_cluster/health?pretty"
{
"error" : {
"root_cause" : [
{
"type" : "master_not_discovered_exception",
"reason" : null
}
],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status" : 503
}
Additionally, the master-node
is showing this error in the log:
[WARN ][o.e.t.n.Netty4Transport ] [49d98428-ddef-479e-85b2-e00fb6f0acef] exception caught on transport layer [NettyTcpChannel{localAddress=/10.233.67.119:9300, remoteAddress=/10.233.69.29:35712}], closing connection
io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (ff,f4,ff,fd)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:392) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:359) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.logging.LoggingHandler.channelInactive(LoggingHandler.java:167) [netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1354) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:917) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (ff,f4,ff,fd)
at org.elasticsearch.transport.TcpTransport.validateMessageHeader(TcpTransport.java:1283) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:36) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[?:?]
... 20 more
ES image: txn2/k8s-es:v6.2.3
ES config file:
cluster:
name: ${CLUSTER_NAME}
node:
master: ${NODE_MASTER}
data: ${NODE_DATA}
name: ${NODE_NAME}
ingest: ${NODE_INGEST}
max_local_storage_nodes: ${MAX_LOCAL_STORAGE_NODES}
processors: ${PROCESSORS:1}
network.host: ${NETWORK_HOST}
path:
data: /data/data
logs: /data/log
repo: ${REPO_LOCATIONS}
bootstrap:
memory_lock: ${MEMORY_LOCK}
http:
enabled: ${HTTP_ENABLE}
compression: true
cors:
enabled: true
allow-origin: "*"
discovery:
zen:
ping.unicast.hosts: ${DISCOVERY_SERVICE}
minimum_master_nodes: ${NUMBER_OF_MASTERS}
all env variables are being read correctly
bash-4.4# printenv | grep CLUSTER
CLUSTER_NAME=elasticsearch-test2
the error from master-node
is indicating like there is some network problem like the containers couldnt "talk" to each other, but i even tested if the connection is possible from client-node
container to elasticsearch-discovery service ClusterIP:
sudo nsenter -t 4552 -n telnet 10.233.12.222 9300
Trying 10.233.12.222...
Connected to 10.233.12.222.
Escape character is '^]'.
Kubernetes configuration: 7 node cluster (3 masters + 4 minions)
clientVersion:
buildDate: 2018-04-27T09:10:24Z
compiler: gc
gitCommit: 81753b10df112992bf51bbc2c2f85208aad78335
gitTreeState: clean
gitVersion: v1.10.2
goVersion: go1.9.3
major: "1"
minor: "10"
platform: linux/amd64
serverVersion:
buildDate: 2018-04-27T09:10:24Z
compiler: gc
gitCommit: 81753b10df112992bf51bbc2c2f85208aad78335
gitTreeState: clean
gitVersion: v1.10.2
goVersion: go1.9.3
major: "1"
minor: "10"
platform: linux/amd64
Docker version: 17.3.2
I can also add that i managed to get it to work using exactly the same image on another cluster (2 masters + 2 minions. exaclty the same kubernetes and docker versions).