I can't create a new cluster using datastax opscenter 5.0.1
I can add the datastax agent to node on a existing cluster but can't create one (ports are opened, ssh connection OK, sudo works for the installation user)
Here is what I do:
- click "create brand new cluster"
- select datastax community 2.0.9 in the drop down
- add IPs of the machines
- click "build cluster"
- wait
- get the error Install Errored: The installed agent doesn't seem to be responding.
SSH-ing to the machine, I can see that nothing was installed/transfered, apparently the "agent_files.tar" has not been scp-ed. No details of what could have failed during transfert.
opscenterd.log extract, loglevel DEBUG (volontarely using only 1 server here to avoid multiple log entries):
2014-11-04 15:48:11+0000 [] INFO: Testing SSH connectivity to 10.133.243.24
2014-11-04 15:48:11+0000 [] INFO: Testing SSH login to 10.133.243.24
2014-11-04 15:48:11+0000 [] DEBUG: performing ssh: ['/usr/bin/ssh', '-l', u'deploy', '-p', '22', '-o', 'LogLevel=Error', u'10.133.243.24', '/usr/bin/test', '0']
2014-11-04 15:48:11+0000 [] INFO: SSH connectivity/login test succeeded
2014-11-04 15:48:11+0000 [] INFO: agent_config items: {'cassandra_log_location': '/var/log/cassandra/system.log', 'thrift_port': 9160, 'jmx_pass': '*****', 'thrift_ssl_truststore': None, 'rollups86400_ttl': -1, 'api_port': '61621', 'use_ssl': 0, 'rollups7200_ttl': 31536000, 'kerberos_debug': False, 'storage_keyspace': 'OpsCenter', 'thrift_user': '', 'provisioning': 0, 'metrics_ignored_column_families': '', 'metrics_ignored_keyspaces': 'system, system_traces, system_auth, dse_auth, OpsCenter', 'jmx_user': '', 'cassandra_install_location': '', 'kerberos_use_keytab': True, 'rollups300_ttl': 2419200, 'thrift_pass': '*****', 'metrics_ignored_solr_cores': '', 'metrics_enabled': 1, 'kerberos_use_ticket_cache': True, 'thrift_ssl_truststore_type': 'JKS', 'rollups60_ttl': 604800, 'ec2_metadata_api_host': '169.254.169.254', 'kerberos_renew_tgt': True, 'thrift_ssl_truststore_password': '*****'}
2014-11-04 15:48:12+0000 [] INFO: Starting provisioning process
2014-11-04 15:48:12+0000 [] DEBUG: Persisting config file /etc/opscenter/clusters/ChallengerDeep.conf
2014-11-04 15:48:12+0000 [] INFO: Starting installation phase of cluster provisioning
2014-11-04 15:48:12+0000 [] DEBUG: performing ssh: ['/usr/bin/ssh', '-l', u'deploy', '-p', '22', '-o', 'LogLevel=Error', u'10.133.243.24', 'echo', '-n', '.$(which apt-get 2> /dev/null) .$(which yum 2> /dev/null)']
2014-11-04 15:48:12+0000 [] DEBUG: Seeing if ip/hostname 10.133.243.24 is an ipv4 address
2014-11-04 15:48:12+0000 [] DEBUG: 10.133.243.24 is an ipv4 address
2014-11-04 15:48:12+0000 [] DEBUG: Performing HTTP request (GET): http://10.133.243.24:61621/alive?, body: None
2014-11-04 15:48:12+0000 [] WARN: HTTP request http://10.133.243.24:61621/alive? failed: Connection was refused by other side: 111: Connection refused.
2014-11-04 15:48:12+0000 [] INFO: Beginning install of OpsCenter agent to 10.133.243.24
2014-11-04 15:48:12+0000 [] DEBUG: Prepping ssh connections
2014-11-04 15:48:12+0000 [] DEBUG: performing scp: ['/usr/bin/scp', '-q', '-P', '22', '/tmp/tmpdJdGZ3', u'deploy@10.133.243.24:/tmp/tmpdJdGZ3']
2014-11-04 15:48:13+0000 [] DEBUG: performing scp: ['/usr/bin/scp', '-q', '-P', '22', './agent_files.tar', u'deploy@10.133.243.24:agent_files.tar']
2014-11-04 15:48:13+0000 [] DEBUG: performing ssh: ['/usr/bin/ssh', '-l', u'deploy', '-p', '22', '-o', 'LogLevel=Error', u'10.133.243.24', 'rm', '-rf', 'datastax-agent-installer', '&&', 'mkdir', 'datastax-agent-installer', '&&', 'cp', 'agent_files.tar', 'datastax-agent-installer/agent_files.tar', '&&', 'cd', 'datastax-agent-installer', '&&', 'tar', 'xvf', 'agent_files.tar', '&&', 'cd', '../', '&&', 'mv', '/tmp/tmpdJdGZ3', 'datastax-agent-installer/pfile', '&&', './datastax-agent-installer/bin/install_agent.sh', '', '10.133.249.88', ';', 'rm', '-rf', 'datastax-agent-installer', 'agent_files.tar']
2014-11-04 15:48:14+0000 [] DEBUG: Performing HTTP request (GET): http://10.133.243.24:61621/alive?, body: None
2014-11-04 15:48:14+0000 [] WARN: HTTP request http://10.133.243.24:61621/alive? failed: Connection was refused by other side: 111: Connection refused.
2014-11-04 15:48:14+0000 [] DEBUG: Agent is still not alive, sleeping 5 seconds...
2014-11-04 15:48:19+0000 [] DEBUG: Average opscenterd CPU usage: 2.24%, memory usage: 43 MB
2014-11-04 15:48:19+0000 [] DEBUG: Performing HTTP request (GET): http://10.133.243.24:61621/alive?, body: None
2014-11-04 15:48:19+0000 [] WARN: HTTP request http://10.133.243.24:61621/alive? failed: Connection was refused by other side: 111: Connection refused.
2014-11-04 15:48:19+0000 [] DEBUG: Agent is still not alive, sleeping 5 seconds...
2014-11-04 15:48:24+0000 [] DEBUG: Performing HTTP request (GET): http://10.133.243.24:61621/alive?, body: None
2014-11-04 15:48:24+0000 [] WARN: HTTP request http://10.133.243.24:61621/alive? failed: Connection was refused by other side: 111: Connection refused.
2014-11-04 15:48:24+0000 [] DEBUG: Agent is still not alive, sleeping 5 seconds...
2014-11-04 15:48:29+0000 [] DEBUG: Performing HTTP request (GET): http://10.133.243.24:61621/alive?, body: None
2014-11-04 15:48:29+0000 [] WARN: HTTP request http://10.133.243.24:61621/alive? failed: Connection was refused by other side: 111: Connection refused.
2014-11-04 15:48:29+0000 [] DEBUG: Agent is still not alive, sleeping 5 seconds...
2014-11-04 15:48:34+0000 [] DEBUG: Performing HTTP request (GET): http://10.133.243.24:61621/alive?, body: None
2014-11-04 15:48:34+0000 [] WARN: HTTP request http://10.133.243.24:61621/alive? failed: Connection was refused by other side: 111: Connection refused.
2014-11-04 15:48:34+0000 [] DEBUG: Agent is still not alive, sleeping 5 seconds...
2014-11-04 15:48:39+0000 [] DEBUG: Performing HTTP request (GET): http://10.133.243.24:61621/alive?, body: None
2014-11-04 15:48:39+0000 [] WARN: HTTP request http://10.133.243.24:61621/alive? failed: Connection was refused by other side: 111: Connection refused.
2014-11-04 15:48:39+0000 [] DEBUG: Agent is still not alive, sleeping 5 seconds...
2014-11-04 15:48:44+0000 [] DEBUG: Performing HTTP request (GET): http://10.133.243.24:61621/alive?, body: None
2014-11-04 15:48:44+0000 [] WARN: HTTP request http://10.133.243.24:61621/alive? failed: Connection was refused by other side: 111: Connection refused.
2014-11-04 15:48:44+0000 [] DEBUG: Agent is still not alive, sleeping 5 seconds...
2014-11-04 15:48:49+0000 [] DEBUG: Performing HTTP request (GET): http://10.133.243.24:61621/alive?, body: None
2014-11-04 15:48:49+0000 [] WARN: HTTP request http://10.133.243.24:61621/alive? failed: Connection was refused by other side: 111: Connection refused.
2014-11-04 15:48:49+0000 [] DEBUG: Agent is still not alive, sleeping 5 seconds...
2014-11-04 15:48:54+0000 [] DEBUG: Performing HTTP request (GET): http://10.133.243.24:61621/alive?, body: None
2014-11-04 15:48:54+0000 [] WARN: HTTP request http://10.133.243.24:61621/alive? failed: Connection was refused by other side: 111: Connection refused.
2014-11-04 15:48:54+0000 [] DEBUG: Agent is still not alive, sleeping 5 seconds...
2014-11-04 15:48:59+0000 [] DEBUG: Performing HTTP request (GET): http://10.133.243.24:61621/alive?, body: None
2014-11-04 15:48:59+0000 [] WARN: HTTP request http://10.133.243.24:61621/alive? failed: Connection was refused by other side: 111: Connection refused.
2014-11-04 15:48:59+0000 [] DEBUG: Agent is still not alive, sleeping 5 seconds...
2014-11-04 15:49:04+0000 [] WARN: Marking request 58fdf092-fc83-4b82-a2be-22b3e63ff795 as failed: The installed agent doesn't seem to be responding.
2014-11-04 15:49:04+0000 [] INFO: Successfully installed agent and dsc on node 10.133.243.24
2014-11-04 15:49:04+0000 [] DEBUG: Subrequests complete for 'install stage' (31cc12a1-5552-443b-8cd5-ec1a91a9191d)
2014-11-04 15:49:04+0000 [] WARN: Marking request 'install stage' (31cc12a1-5552-443b-8cd5-ec1a91a9191d) as failed: The installed agent doesn't seem to be responding.
2014-11-04 15:49:04+0000 [] ERROR: Installation stage failed: The installed agent doesn't seem to be responding.
2014-11-04 15:49:04+0000 [] DEBUG: Subrequest failed (key=install request=RequestCollection[31cc12a1-5552-443b-8cd5-ec1a91a9191d](error, The installed agent doesn't seem to be responding.)): Installation stage failed: The installed agent doesn't seem to be responding.
2014-11-04 15:49:04+0000 [] WARN: Marking request 'provision' (c5243946-3bb6-4eb5-b669-04355c319339) as failed: Installation stage failed: The installed agent doesn't seem to be responding.
2014-11-04 15:49:04+0000 [] ERROR:
2014-11-04 15:49:04+0000 [] ERROR: Cluster provisioning failed: Exception: Installation stage failed: The installed agent doesn't seem to be responding.
2014-11-04 15:49:04+0000 [] DEBUG: Seeing if ip/hostname 10.133.243.24 is an ipv4 address
2014-11-04 15:49:04+0000 [] DEBUG: 10.133.243.24 is an ipv4 address
2014-11-04 15:49:04+0000 [] ERROR: Failed to provision cluster: Cluster provisioning failed: Exception: Installation stage failed: The installed agent doesn't seem to be responding.
