SSL trouble in Perl's LWP after Debian Wheezy upgrade

Question

I just upgraded to Debian Wheezy from Lenny (long story) and a few of my Perl packages didn't make the upgrade for some reason, among them Crypt::SSLeay (libcrypt-ssleay-perl).

I was unable to connect to my credit card payment gateway using LWP until I installed libcrypt-ssleay-perl, and now it "works", but it takes 15-20 seconds to make the SSL connection due to the very unhelpful error message "error in unknown state":

SSL_connect:before/connect initialization
SSL_connect:unknown state
(waiting 15-20 seconds...) #this comment is from waldo22
SSL_connect:error in unknown state
SSL_connect:before/connect initialization
SSL_connect:SSLv3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A

I have a feeling this has something to do with LWP changing the default behavior for certificate authorities and server certificiate verification:
https://stackoverflow.com/questions/74358/how-can-i-get-lwp-to-validate-ssl-server-certificates#5329129
and possibly:
https://stackoverflow.com/questions/5639803/aws-ses-certificate-verify-failed

My Perl module is using Crypt::SSLeay via LWP::useragent.

Obviously 15-20 seconds is way too long to wait to make an SSL connection, but without a more helpful error message, I don't know what to do.

Does anyone have any suggestions on how to better debug this or to get more verbose output?

Thanks a bunch,

-Wes

Answer 1

Wow, this one was a doozy.

There appears to be a problem with OpenSSL 1.0.1 where attempting to auto-negotiate with TLS1.1 (or 1.2???) with some servers (edit: BigIP servers running firmware < 10.2.4) causes those servers to drop the connection and reject the request. See:

https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/965371

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=665452

I'm trying to connect to Paypal's "Payflow Pro" credit card gateway, and Paypal was one of the main culprits on the list, originally.

According to the bug ticket, they say it is "fixed" for Paypal in 1.0.1b, but I'm running 1.0.1c-3 and still experience the problem. I don't know if that means it's fixed for www.paypal.com but not payflowpro.paypal.com, or what.

WORKAROUND: Setting $ENV{HTTPS_VERSION} = 3 to force SSL3 seems to fix the problem, at least for Crypt::SSLeay.

Presumably this works because it does not attempt to negotiate TLS1.1, and just uses SSL3.

Testing with openssl s_client, it works with the options -ssl3, -tls1, and -no_tls1, so it must be a negotiation problem, in my opinion.

Anyway, that's a workaround, at least.

The actual problem is caused by a bug in the firmware of F5 BigIP load balancers with firmware less than 10.2.4. This causes TLS 1.1 or 1.2 connections to not respond properly to long ClientHello requests, and thus hang.

The real fix is to update the firmware on the BigIP load balancer to >= 10.2.4.

Of course Paypal/Payflow blamed OpenSSL for this...