Opscenter 5.2 missing data for 1day and 1week graph scale

Question

I recently update Opscenter from 5.1 to 5.2, and yesterday just discovered that if I select different 1day or 1week graph scales no data will be shown in the graph. I checked the logs and the chrome console and no issue there, the requests just got 200 and the response ha no data in it:

curl 'http://cassandra.mydomain.com:8888/keyspace/new-metrics?metrics=read-repair-attempted&start=1437630985&end=1438250185&step=7200&forecast=0&node_aggregation=1&node_group=*'
response
{"aggregation_function": {"thrift-connections": "Total"}, "columnfamilies": [], "bounds": {"start": 1437624000, "step": 7200, "end": 1438250400}, "metrics": ["thrift-connections"], "nodes": ["10.0.1.178", "10.0.3.148", "10.0.2.145", "10.0.2.172", "10.0.3.86", "10.0.1.29"], "data": {"*": [{"metric": "thrift-connections", "data-points": []}]}}

I checked with cqlsh the Opscenter keyspace ant the data are there, for example:

cqlsh:OpsCenter> SELECT * FROM rollups7200 LIMIT 2;
 key                                                   | column1   | value
-------------------------------------------------------+-----------+----------------------------
 10.0.1.172-keyspace-hostinfo-getWriteCount | 730437247 | 0x3beb2702000000003eb26e06
 10.0.1.172-keyspace-hostinfo-getWriteCount | 730444447 | 0x3bc9ff51000000003eb25444
(2 rows)
cqlsh:OpsCenter> SELECT * FROM rollups86400  LIMIT 2;
 key                                                   | column1   | value
-------------------------------------------------------+-----------+----------------------------
 10.0.1.172-keyspace-hostinfo-getWriteCount | 730437247 | 0x3bb54ce4000000003eb2765d
 10.0.1.29-keyspace-hostinfo-getPendingTasks | 710392447 | 0x3dbbac7e0000000042f80000

* Update *

Here the describe of the Opscenter keyspace:

CREATE KEYSPACE "OpsCenter" WITH replication = {
  'class': 'SimpleStrategy',
  'replication_factor': '2'
};

USE "OpsCenter";

CREATE TABLE backup_reports (
  week text,
  event_time timestamp,
  backup_id text,
  type text,
  destination text,
  deleted_at timestamp,
  full_status text,
  keyspaces text,
  status text,
  PRIMARY KEY ((week), event_time, backup_id, type, destination)
) WITH CLUSTERING ORDER BY (event_time DESC, backup_id ASC, type ASC, destination ASC) AND
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.100000 AND
  gc_grace_seconds=864000 AND
  index_interval=128 AND
  read_repair_chance=0.000000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='99.0PERCENTILE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};

CREATE TABLE bestpractice_results (
  key text,
  column1 varint,
  value blob,
  PRIMARY KEY ((key), column1)
) WITH COMPACT STORAGE AND
  CLUSTERING ORDER BY (column1 DESC) AND
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='{"info": "OpsCenter management data.", "version": [5, 1, 0]}' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=0 AND
  index_interval=128 AND
  read_repair_chance=0.250000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='NONE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};

CREATE TABLE events (
  key text,
  action bigint,
  column_family text,
  level bigint,
  message text,
  success boolean,
  target_node text,
  time bigint,
  PRIMARY KEY ((key))
) WITH COMPACT STORAGE AND
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='{"info": "OpsCenter management data.", "version": [5, 2, 0]}' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=864000 AND
  index_interval=128 AND
  read_repair_chance=0.250000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='NONE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};

CREATE TABLE events_timeline (
  key text,
  column1 bigint,
  value blob,
  PRIMARY KEY ((key), column1)
) WITH COMPACT STORAGE AND
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='{"info": "OpsCenter management data.", "version": [5, 1, 0]}' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=864000 AND
  index_interval=128 AND
  read_repair_chance=0.250000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='NONE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};

CREATE TABLE pdps (
  key text,
  column1 text,
  value blob,
  PRIMARY KEY ((key), column1)
) WITH COMPACT STORAGE AND
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='{"info": "OpsCenter management data.", "version": [5, 1, 0]}' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=0 AND
  index_interval=128 AND
  read_repair_chance=0.250000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='NONE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};

CREATE TABLE rollups300 (
  key text,
  "timestamp" varint,
  value blob,
  PRIMARY KEY ((key), "timestamp")
) WITH COMPACT STORAGE AND
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='{"info": "OpsCenter management data.", "version": [5, 2, 0]}' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=0 AND
  index_interval=128 AND
  read_repair_chance=0.250000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='NONE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};

CREATE TABLE rollups60 (
  key text,
  "timestamp" varint,
  value blob,
  PRIMARY KEY ((key), "timestamp")
) WITH COMPACT STORAGE AND
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='{"info": "OpsCenter management data.", "version": [5, 2, 0]}' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=0 AND
  index_interval=128 AND
  read_repair_chance=0.250000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='NONE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};

CREATE TABLE rollups7200 (
  key text,
  column1 varint,
  value blob,
  PRIMARY KEY ((key), column1)
) WITH COMPACT STORAGE AND
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='{"info": "OpsCenter management data.", "version": [5, 1, 0]}' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=0 AND
  index_interval=128 AND
  read_repair_chance=0.250000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='NONE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};

CREATE TABLE rollups86400 (
  key text,
  column1 varint,
  value blob,
  PRIMARY KEY ((key), column1)
) WITH COMPACT STORAGE AND
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='{"info": "OpsCenter management data.", "version": [5, 1, 0]}' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=0 AND
  index_interval=128 AND
  read_repair_chance=0.250000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='NONE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};

CREATE TABLE settings (
  key blob,
  column1 blob,
  value blob,
  PRIMARY KEY ((key), column1)
) WITH COMPACT STORAGE AND
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='{"info": "OpsCenter management data.", "version": [5, 1, 0]}' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=864000 AND
  index_interval=128 AND
  read_repair_chance=1.000000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='NONE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};

Possibility schema migraiton failed (should be timestamp, not column1). Can you provide output for describe schema on OpsCenter keyspace? Can try bouncing OpsCenter daemon to try again maybe. — Chris Lohfink, Jul 30 '15 at 14:36
Hi Luigi, please accept the answer instead of adding "Resolved" (it may take some time until you can due to your rep). — bummi, Jul 31 '15 at 14:00

score 1 · Answer 1 · answered Jul 30 '15 at 15:34

So looks like some of the schemas failed to upgrade. You can try restarting opscenterd to see if it runs. The rollups7200 table is still running 5.1.0 (see comments in schema). You can manually fix this too:

ALTER TABLE "OpsCenter".rollups7200 RENAME column1 to timestamp

Opscenter 5.2 missing data for 1day and 1week graph scale

1 Answers1