I have an application that uses the PyCassa Cassandra client to open many concurrent connections to a Cassandra server. When I run the client application locally on the database server, it works. However, when i forward the Cassandra port thru an ssh tunnel to the database server and connect the client application to the server thru this tunnel, a few connections get thru but generally i get errors like this:
'2012210105:49:49'|WARNING |_append_historyStorage|75 |Exception when appending to CassandraTimeSeries
Traceback (most recent call last):
File "atr/cassandratimeseries.py", line 140, in append
cf = self._getColumnFamily(duration, 'main')
File "atr/cassandratimeseries.py", line 63, in _getColumnFamily
return ColumnFamily(ConnectionPool(self._keyspace, pool_timeout=100, timeout=15, max_overflow=5), self._columnFamilyName(duration, table_type), **self._column_family_op_options)
File "/usr/local/lib/python2.7/dist-packages/pycassa/pool.py", line 356, in __init__
self.fill()
File "/usr/local/lib/python2.7/dist-packages/pycassa/pool.py", line 415, in fill
conn = self._create_connection()
File "/usr/local/lib/python2.7/dist-packages/pycassa/pool.py", line 403, in _create_connection
(exc.__class__.__name__, exc))
AllServersUnavailable: An attempt was made to connect to each of the servers twice, but none of the attempts succeeded. The last failure was TTransportException: TSocket read 0 bytes
On a terminal where i was ssh'd into the server is printed a line of messages like:
channel 1034: open failed: administratively prohibited: open failed
channel 1035: open failed: administratively prohibited: open failed
channel 1036: open failed: administratively prohibited: open failed
channel 1037: open failed: administratively prohibited: open failed
channel 1038: open failed: administratively prohibited: open failed
channel 1039: open failed: administratively prohibited: open failed
channel 1040: open failed: administratively prohibited: open failed
Interestingly, if i then attempt to ssh to the server, i get:
channel 1023: chan_read_failed for istate 1
channel 1023: chan_write_failed for ostate 3
Shared connection to xx.xxx.xxx.xxx closed.
If i delete the master socket file in /tmp, then i can ssh to it again.
Any ideas what could be going wrong? I already increased the nofile limit via /etc/security/limits.conf for both the client and the server (to ten thousand) and rebooted. The "open failed: administratively prohibited" suggests that there is some administrative limit being hit. What is it and how do i raise it (and how should i have figured this out for myself?)
Are there other ways in which ssh tunnels are more limited than connecting to a local port?
thanks