0

I've looked through multiple forum posts, I've turned on debug et al, and I still cannot for the life of me figure out why the disk space isn't working. ALL other information is filled--except for the disk space.

On the Agent logs, I see requests for /os-metric/disk-space, but it always returns an empty value. There are NO errors on start-up, and DEBUG logging doesn't reveal anything. The output in the log is:

INFO [qtp728395819-23] 2014-03-20 16:49:19,636 HTTP: :get /os-metric/disk-space {} - 200

There are, also, no errors on the Opscenter server.

I initially had errors where it could not find the right cassandra.yaml file, but I have corrected these, and there are now zero errors in the startup-logs. This occurs with the latest version of Opscenter 4.1.1, and Cassandra 1.2.15. Are there any more suggestions on what to look for? For example, does it use a non-standard utility (i.e., not df)?

EDIT: Adding output from //nodes; this is common for each node.

{
    "data_held": 18001238572.0,
    "dc": "DC1",
    "devices": {
        "commitlog": "sdb",
        "data": [
            "sdc"
        ],
        "other": [
            "sda1"
        ],
        "saved_caches": "sdb"
    },
    "has_jna": true,
    "last_seen": 0,
    "load": 0.13,
    "mode": "normal",
    "network_interfaces": [
        "lo",
        "eth0",
        "sit0"
    ],
    "node_ip": "{snip}",
    "node_name": "{snip}",
    "node_version": {
        "cassandra": "1.2.15",
        "dse": null,
        "jobtracker": null,
        "search": null,
        "tasktracker": null
    },
    "num_procs": 8,
    "os": "linux",
    "partitions": {
        "commitlog": "/dev/sdc",
        "data": [
            "_"
        ],
        "other": [
            "/dev/sdb",
            "/dev/sda1"
        ],
        "saved_caches": "/dev/sdc"
    },
    "rack": "DEFAULT-RACK",
    "rpc_ip": "{snip}",
    "streaming": {},
    "task_progress": {},
    "token": "-5829041219150380184",
    "vnodes": true
}

cassandra.yaml data directories:

# Directories where Cassandra should store data on disk.  Cassandra
# will spread data evenly across them, subject to the granularity of
# the configured compaction strategy.
data_file_directories:
    - /var/lib/cassandra/data

$ cat /proc/mounts
[snip]
/dev/sdc /var/lib/cassandra/data ext3 rw,data=ordered 0 0
Andrew M.
  • 10,982
  • 2
  • 34
  • 29
  • Can you try restarting opscenterd, and then restarting each of your agents to see if that resolves the issue? There is a known race condition we're working to fix around restarting opscenterd without restarting the agents as well. – mbulman Mar 24 '14 at 16:43
  • I have done this multiple times. In fact, I have taken down the entire Cassandra cluster and started from scratch 2-3 times already. The agents are communicating all the other information just fine--just not the disk-usage metric. – Andrew M. Mar 24 '14 at 16:55
  • Next step is to see if opscenterd is aware of the Cassandra data partitions, but looking at the node information from the API: http://www.datastax.com/documentation/opscenter/4.1/api/docs/cluster_info.html#node-properties , as well as hitting the API route you pasted in the original message on the agent to see if you're getting accurate information back. – mbulman Mar 24 '14 at 23:35
  • Thanks, @mbulman, I went ahead and added the call to /nodes to the API, since this seemed to show the relevant information (i.e., the devices and partitions). – Andrew M. Mar 25 '14 at 02:47
  • 1
    We ran into the same issue. If your backend storage is not a block device then the storage will not show up in OpsCenter. I have been told by Datastax that there is a open issue for this. –  Mar 25 '14 at 14:10
  • The device is an EC2 virtualized drive, though, which _is_ a block storage device. I would assume this simply used `df` (or similar) and interpreted the results... after all, it uses `iostat` for the IO metrics. – Andrew M. Mar 25 '14 at 16:54
  • @mbulman Any more suggestions? It looks like the "data" partition is being set to "_" (underscore). However, I confirmed in `cassandra.yaml` that it's set to a valid value. – Andrew M. Mar 25 '14 at 22:56
  • @AndrewM. mind pasting the output of the /nodes API route from opscenterd and /os-metrics/disk-space API from one of the agents? – mbulman Apr 01 '14 at 14:27
  • @DanDentinger were you given a number or anything else to help reference the ticket? I want to check on the issue for you, as well as find out if Andrew is hitting the same issue – mbulman Apr 01 '14 at 14:27

0 Answers0