Datastax OpsCenter 3.1.1 Agent JMX Authentication Error

Question

I am using opscenter 3.1.1 and just enabled JMX username/password authentication on my Cassandra cluster. I think I've updated all my opscenter configs correctly to force the agents to use JMX auth, but it is not working.

I've updated the config under /etc/opscenter/Clusters/[cluster-name].conf with the following jmx properties

    [jmx]
    username=username
    password=password
    port=7199

I then restarted opscenter and opscenter agents, but see the following error in the opscenter agent logs:

    DEBUG [Thread-10] 2013-11-15 10:07:28,461 Adding JMX connector to pool (127.0.0.1:7199)
    DEBUG [node-details-1] 2013-11-15 10:07:28,464 Starting real-time collection
    DEBUG [node-details-2] 2013-11-15 10:07:28,466 Starting short-time collection
    DEBUG [node-details-3] 2013-11-15 10:07:28,468 Starting long-time collection
    ERROR [node-details-1] 2013-11-15 10:07:28,474 Error getting realtime node details
            java.lang.SecurityException: Authentication failed! Credentials required
            at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticationFailure(Unknown Source)
            at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticate(Unknown Source)
            at sun.management.jmxremote.ConnectorBootstrap$AccessFileCheckerAuthenticator.authenticate(Unknown Source)
            at javax.management.remote.rmi.RMIServerImpl.doNewClient(Unknown Source)
            at javax.management.remote.rmi.RMIServerImpl.newClient(Unknown Source)
            at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
            at java.lang.reflect.Method.invoke(Unknown Source)
            at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
            at sun.rmi.transport.Transport$1.run(Unknown Source)
            at sun.rmi.transport.Transport$1.run(Unknown Source)
            at java.security.AccessController.doPrivileged(Native Method)
            at sun.rmi.transport.Transport.serviceCall(Unknown Source)
            at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
            at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
            at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)
            at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(Unknown Source)
            at sun.rmi.transport.StreamRemoteCall.executeCall(Unknown Source)
            at sun.rmi.server.UnicastRef.invoke(Unknown Source)
            at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
            at javax.management.remote.rmi.RMIConnector.getConnection(Unknown Source)
            at javax.management.remote.rmi.RMIConnector.connect(Unknown Source)
            at javax.management.remote.rmi.RMIConnector.connect(Unknown Source)
            at opsagent.jmx$create_jmx_pool$wrapper__1044.invoke(jmx.clj:183)
            at opsagent.nodedetails$run_realtime_collection.invoke(nodedetails.clj:201)
            at opsagent.nodedetails$start_pool$fn__3237.invoke(nodedetails.clj:302)
            at clojure.lang.AFn.run(AFn.java:24)

A few lines above this stacktrace, I do see a line saying the agent is retrieving the latest config from opscenter:

    INFO [StompConnection receiver] 2013-11-15 10:07:18,298 Got new config from OpsCenter:

But the only jmx-related property in the data is :jmx_port 7199.

This seems to be a bug in 3.2.2 (http://mail-archives.apache.org/mod_mbox/cassandra-user/201310.mbox/%3CCALHkrw_xjJBy5k-7vCc3NezLy17HLY_HPDCFptG74RqiivcgBg@mail.gmail.com%3E), but I see no indication that it is a bug in 3.1.1 or that it has been fixed in 4.0.

Any ideas on how to fix this?

score 1 · Accepted Answer · answered Nov 18 '13 at 16:26

The bug mentioned in that mailing list thread affects 3.1.1 as well, and has been fixed in 4.0.0. We'll get a mention of that added to the release notes.

For future reference, the internal ticket number is OPSC-2136. And we strip any authentication information from the config logging you mentioned.

Datastax OpsCenter 3.1.1 Agent JMX Authentication Error

1 Answers1