1

I am running into this error on a OpsCenter node with DSE Cassandra installed locally. It is in it's own DC and part of a cluster with three other nodes in another DC.

Authentication is enabled on the cluster with org.apache.cassandra.auth.PasswordAuthenticator and org.apache.cassandra.auth.CassandraAuthorizer. The system_auth table is replicated on all nodes in the cluster with simple replication strategy and replication factor of 4.

[cqlsh 5.0.1 | Cassandra 2.1.5.469 | DSE 4.7.0 | CQL spec 3.2.0 | Native protocol v3]

From Cassandra system.log on the OpsCenter node:

WARN  [MessagingService-Incoming-/192.168.33.152] 2015-05-29 16:52:42,646  IncomingTcpConnection.java:97 - UnknownColumnFamilyException reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Got range slice command for nonexistent table dse_system.real_leaders.  If the table was just created, this is likely due to the schema not being fully propagated.  Please wait for schema agreement on table creation.
        at org.apache.cassandra.db.RangeSliceCommandSerializer.deserialize(RangeSliceCommand.java:193) ~[cassandra-all-2.1.5.469.jar:2.1.5.469]
        at org.apache.cassandra.db.RangeSliceCommandSerializer.deserialize(RangeSliceCommand.java:151) ~[cassandra-all-2.1.5.469.jar:2.1.5.469]
        at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[cassandra-all-2.1.5.469.jar:2.1.5.469]
        at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:188) ~[cassandra-all-2.1.5.469.jar:2.1.5.469]
        at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:170) ~[cassandra-all-2.1.5.469.jar:2.1.5.469]
        at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:88) ~[cassandra-all-2.1.5.469.jar:2.1.5.469]






INFO  [MemtableFlushWriter:3] 2015-05-29 16:53:49,817  Memtable.java:380 - Completed flushing /usr/local/dse-4.7.0/dse-data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-20-Data.db (164 bytes) for commitlog position ReplayPosition(segmentId=1432918359985, position=2480593)
INFO  [CompactionExecutor:3] 2015-05-29 16:53:49,818  CompactionTask.java:140 - Compacting [SSTableReader(path='/usr/local/dse-4.7.0/dse-data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-18-Data.db'), SSTableReader(path='/usr/local/dse-4.7.0/dse-data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-17-Data.db'), SSTableReader(path='/usr/local/dse-4.7.0/dse-data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-19-Data.db'), SSTableReader(path='/usr/local/dse-4.7.0/dse-data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-20-Data.db')]
INFO  [CompactionExecutor:3] 2015-05-29 16:53:49,908  CompactionTask.java:270 - Compacted 4 sstables to [/usr/local/dse-4.7.0/dse-data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-21,].  1,556 bytes to 498 (~32% of original) in 88ms = 0.005397MB/s.  23 total partitions merged to 8.  Partition merge counts were {1:1, 3:6, 4:1, }
WARN  [pool-5-thread-1] 2015-05-29 16:54:39,512  LeaderManagerWatcher.java:379 - OH GOD
java.lang.NullPointerException: null
        at com.datastax.bdp.util.LeaderManagerWatcher.checkNetwork(LeaderManagerWatcher.java:223) ~[dse-4.7.0.jar:4.7.0]
        at com.datastax.bdp.util.LeaderManagerWatcher.getNextView(LeaderManagerWatcher.java:456) ~[dse-4.7.0.jar:4.7.0]
        at com.datastax.bdp.util.LeaderManagerWatcher.run(LeaderManagerWatcher.java:326) ~[dse-4.7.0.jar:4.7.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_67]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_67]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_67]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_67]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_67]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_67]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
WARN  [pool-5-thread-1] 2015-05-29 16:54:44,526  LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up?  - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
INFO  [pool-5-thread-1] 2015-05-29 16:54:44,527  LeaderManagerWatcher.java:290 - /192.168.33.158: Leader HadoopJT.DC_ANALYTICS_1 changed from /192.168.33.156 to <null> [unregistered] [notified 0 listeners]
WARN  [pool-5-thread-1] 2015-05-29 16:54:49,529  LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up?  - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
WARN  [pool-5-thread-1] 2015-05-29 16:54:54,532  LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up?  - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
WARN  [pool-5-thread-1] 2015-05-29 16:54:59,534  LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up?  - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
WARN  [pool-5-thread-1] 2015-05-29 16:55:04,536  LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up?  - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
WARN  [pool-5-thread-1] 2015-05-29 16:55:09,538  LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up?  - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
WARN  [pool-5-thread-1] 2015-05-29 16:55:14,539  LeaderManagerWatcher.java:470 - Local query timed out; is this node under heavy load/starting up?  - org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE

Cluster details:

nodetool status
Datacenter: DC_DATA_1
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens  Owns    Host ID                               Rack
UN  192.168.33.151  854.95 KB  256     ?       2af17bd4-e454-4cd0-8621-f42481f52398  RAC1
UN  192.168.33.153  857.32 KB  256     ?       712a892a-44f0-46fd-b28b-04eae418ebde  RAC1
UN  192.168.33.152  870.35 KB  256     ?       ce08a377-8b71-4789-beae-e80a70b3b685  RAC1
Datacenter: DC_OPSCENTER_1
==========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens  Owns    Host ID                               Rack
UN  192.168.33.158  512.37 KB  256     ?       22468c08-cd3e-4fc2-8ff8-fa4aeff12caa  RAC1

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless




[root@cassandra-opscenter-1 ~]# cqlsh -u cassandra_admin -p password 192.168.33.151 9042
Connected to Cluster1 at 192.168.33.151:9042.
[cqlsh 5.0.1 | Cassandra 2.1.5.469 | DSE 4.7.0 | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
cassandra_admin@cqlsh> select * from system.schema_keyspaces;

 keyspace_name   | durable_writes | strategy_class                                       | strategy_options
-----------------+----------------+------------------------------------------------------+----------------------------
     system_auth |           True |          org.apache.cassandra.locator.SimpleStrategy | {"replication_factor":"4"}
      dse_system |           True |      org.apache.cassandra.locator.EverywhereStrategy |                         {}
       OpsCenter |           True | org.apache.cassandra.locator.NetworkTopologyStrategy |     {"DC_OPSCENTER_1":"1"}
          system |           True |           org.apache.cassandra.locator.LocalStrategy |                         {}
 sample_keyspace |          False |          org.apache.cassandra.locator.SimpleStrategy | {"replication_factor":"2"}
   system_traces |           True |          org.apache.cassandra.locator.SimpleStrategy | {"replication_factor":"2"}

(6 rows)
cassandra_admin@cqlsh> desc keyspaces;

sample_keyspace  system_auth  system  "OpsCenter"  system_traces  dse_system
krafts
  • 11
  • 5
  • you can change the strategy to network topology. For a multi dc deployment like yours, it is a must to have NetworkTopology strategy for what you are trying to achieve. http://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architectureDataDistributeReplication_c.html – skipy Jun 29 '15 at 11:53
  • @skipy I did try that initially and it led to similar errors. I do not have the logs for that at the moment (that setup was destroyed). – krafts Jun 29 '15 at 15:37

0 Answers0