0

I have a Cassandra cluster set up on 5 EC2 nodes. The nodes can communicate using the private IP addresses just fine, but get connection failures when using the private internal DNS name. Localhost also fails.

I am trying to use the cassandra-stress tool but I get the Failed to connect over JMX; not collecting these stats, and when turning on verbose logging I get the following stack:

java.lang.RuntimeException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: ip-172-1-2-2.eu-west-1.compute.internal; nested exception is:
    java.net.ConnectException: Connection refused (Connection refused)]
    at org.apache.cassandra.stress.util.JmxCollector.connect(JmxCollector.java:99)
    at org.apache.cassandra.stress.util.JmxCollector.<init>(JmxCollector.java:85)
    at org.apache.cassandra.stress.StressMetrics.<init>(StressMetrics.java:62)
    at org.apache.cassandra.stress.StressAction.run(StressAction.java:198)
    at org.apache.cassandra.stress.StressAction.runMulti(StressAction.java:126)
    at org.apache.cassandra.stress.StressAction.run(StressAction.java:72)
    at org.apache.cassandra.stress.Stress.main(Stress.java:109)

When I run cqlsh I have to specify the private IP address (./cqlsh 172.1.2.3) as using the default localhost fails.

Is there some sort of additional configuration that is required to map this internal name to localhost, or to get it to resolve the private IP addresses on each node?

EDIT:

Running host ip-172-1-2-3.eu-west-1.compute.internal gives the correct output:

ip-172-1-2-3.eu-west-1.compute.internal has address 172.1.2.3

So it looks like DNS is working correctly.

For my cassandra.yaml settings, I have listen_address commented out, but am using:

listen_interface: eth0
broadcast_address: 172.1.2.3
rpc_address: 172.1.2.3

Replication works fine with these settings. I'm not sure how to configure JMX to listen explicitly on the private IP address.

MeanwhileInHell
  • 121
  • 5
  • Please check the output of the `host ` command, to see if DNS resolution is working properly. Localhost is a different issue, as it maps to a different address, and server processes can choose to listen on specific addresses even if they are configured on the same host. – b0fh Mar 22 '19 at 13:59
  • please add how the `listen_address` and `rpc_address` are configured. Also, have you configured JMX explicitly to listen on private IP? JMX & `cqlsh` are very different settings... – Alex Ott Mar 25 '19 at 09:13
  • @b0fh I've added the information you asked for into my question. – MeanwhileInHell Mar 25 '19 at 12:11
  • @AlexOtt I've added my cassandra.yaml settings to my question. – MeanwhileInHell Mar 25 '19 at 12:12
  • Those addresses are wrong. It is probably either a typo or you have deliberately put fake information unnecessarily. Try using the correct addresses. – Michael Hampton Mar 25 '19 at 13:45

1 Answers1

0

Your command cqlsh 172.1.2.3 fails because you configured rpc_address to listen only on the given address. If you want to listen for CQL traffic everywhere, you need to set rpc_address to 0.0.0.0, and set broadcast_rpc_address to 172.1.2.3 - it's all described in the cassandra.yaml itself.

Regarding JMX - it's not really required that you had it enabled - it collects some data about garbage collection, etc., that maybe not necessary yet if you're just starting. I recommend to concentrate on testing itself, like, described in this article, for example.

But if you really want to expose JMX to outside, you need to look into cassandra-env.sh - it has detailed comments on what to do. You only need to make sure that you secured JMX, otherwise anybody can do almost anything with your cluster - wipe data, configure it, etc.

Alex Ott
  • 316
  • 1
  • 5