I dug up some candidates of smoking gun.
I think the latest OpenSSL needs to be made available.
[] Initial lead : GIT_SSL_NO_VERIFY
The following is about the use of git under XP, but it turns out to be
related in a somewhat indirect manner to my problem, too. At least the
use of GIT_SSL_NO_VERIFY is a news to me, and the workaround that uses
GIT_SSL_NO_VERIFY didn't work for me, and so I could home in to other
possible real causes.
https://stackoverflow.com/questions/3777075/https-github-access
I followed the first suggestion from the above URL (use of
GIT_SSL_NO_VERIFY). But it didn't solve the issue. So my next
trail was about maybe the curl library version is outdated?
Quote from the post above:
---- begin quote
The problem is that you do not have any of Certification Authority
certificates installed on your system. And these certs cannot be
installed with cygwin's setup.exe.
There are two solutions: 1. Ignore ssl certificate verification:
$ env GIT_SSL_NO_VERIFY=true git clone https://github...
Actually install root certificates. Curl guys extracted for you certificates from mozilla:
http://curl.haxx.se/docs/caextract.html
cacert.pem file is what you are looking for. This file contains > 250 CA certs (don't know how to trust this number of ppl). You need to download this file, split it to individual certificates put them to /usr/ssl/certs (your CApath) and index them.
*************************8
under solaris: CApath: /opt/csw/ssl/certs
Here is how to do it: With cygwin setup.exe install curl and openssl
packages Execute:
$ cd /usr/ssl/certs
$ curl http://curl.haxx.se/ca/cacert.pem | awk 'split_after==1{n++;split_after=0} /-----END CERTIFICATE-----/ {split_after=1} {print > "cert" n ".pem"}'
$ c_rehash
---- quote end
I tried GIT_SSL_NO_VERIFY=true, but still no dice.
So certificates probably are not the direct cause of the problems.
After searching for clues, I found
GIT_CURL_VERBOSE=1
export GIT_CURL_VERBOSE
produces much output to follow what is going on. I attach the verbose
output for comparison under solaris and linux.
[] CERT availability turned out to be insignificant to my problem.
git using https transport worked under linux.: The distribution I use
(Debian now) installs some public key certs from CAs and that helps
git in determining the validity of bitbuckt cert, etc.
Under solaris10, opencsw repository seems to have installed already
some certs under /opt/csw/ssl/certs.
So having certs didn't seem to be a problem (or is it?).
And the following log produced by GIT_CURL_VERBOSE seemed to suggest
(albeit somewhat in a spartan manner under solaris) that
the cert is verified.
[] libraries and commands mixed up from sfw and opencsw
Then why doesn&t solaris version of git fails using https protocol?
It seems git uses libcurl and friends.
Maybe libcurl from opencsw package is not compiled with "with-ssl"
option (?)
curl-config command, if available, prints the version
and curl --version shows the built-in protocol, etc.
While checking the library version numbers directory by listing
libraries, I noticed a grave confusion on solaris 10. There are two
versions of ssl and libcurl. One is from opencsw, and the other is
from somewhat older sfw (sun freeware) distribution, and I am afraid
that sfw was the preferred choice some years ago. And, to top it off,
curl command itself is from sfw (at least on my solaris x86 (64bits)
incarnation!)
You can see the combination is royally mixed up on my solarix 10 image.
SFW-related:
zephyr@zephyrus:~/repos$ ls /usr/sfw/lib/*ssl*
/usr/sfw/lib/libssl.so@ /usr/sfw/lib/llib-lssl
/usr/sfw/lib/libssl.so.0.9.7* /usr/sfw/lib/llib-lssl.ln
/usr/sfw/lib/libssl_extra.so.0.9.7*
zephyr@zephyrus:~/repos$
ls /opt/sfw/bin/*curl*
/opt/sfw/bin/curl* /opt/sfw/bin/curl-config*
zephyr@zephyrus:~/repos$ which curl
/opt/sfw/bin/curl
CSW-related:
ls -l /opt/csw/lib/*curl*
lrwxrwxrwx 1 root root 16 Mar 18 18:00 /opt/csw/lib/libcurl.so.4 -> libcurl.so.4.2.0*
-rwxr-xr-x 1 root bin 689040 Feb 23 19:38 /opt/csw/lib/libcurl.so.4.2.0*
zephyr@zephyrus:~/repos$ ls -l /opt/csw/lib/*ssl*
lrwxrwxrwx 1 root root 29 Mar 18 17:57 /opt/csw/lib/libgnutls-openssl.so.26 -> libgnutls-openssl.so.26.16.13*
-rwxr-xr-x 1 root bin 70432 Jan 18 2011 /opt/csw/lib/libgnutls-openssl.so.26.16.13*
-rw-r--r-- 1 root bin 527064 Jan 24 06:08 /opt/csw/lib/libssl.a
lrwxrwxrwx 1 root root 15 Mar 27 22:04 /opt/csw/lib/libssl.so -> libssl.so.0.9.8*
-r-xr-xr-x 1 root bin 314056 Jan 3 2008 /opt/csw/lib/libssl.so.0.9.7*
-r-xr-xr-x 1 root bin 388928 Jan 24 06:08 /opt/csw/lib/libssl.so.0.9.8*
zephyr@zephyrus:~/repos$
Ok, which libraries does sfw version of curl is using?
It uses CSW-distribution if available! (I put /opt/csw/lib early in my
LD_LIBRARY_PATH. That is why, I think.)
zephyr@zephyrus:~/repos$ ldd /opt/sfw/bin/curl
libcurl.so.4 => /opt/csw/lib/libcurl.so.4
libssl.so.0.9.7 => /opt/csw/lib/libssl.so.0.9.7
libcrypto.so.0.9.7 => /opt/csw/lib/libcrypto.so.0.9.7
libsocket.so.1 => /lib/libsocket.so.1
libnsl.so.1 => /lib/libnsl.so.1
libdl.so.1 => /lib/libdl.so.1
libz.so.1 => /opt/csw/lib/libz.so.1
libz.so.1 (SUNW_1.1) => (version not found)
libc.so.1 => /lib/libc.so.1
libidn.so.11 => /opt/csw/lib/libidn.so.11
libssl.so.0.9.8 => /opt/csw/lib/libssl.so.0.9.8
libcrypto.so.0.9.8 => /opt/csw/lib/libcrypto.so.0.9.8
libmp.so.2 => /lib/libmp.so.2
libmd.so.1 => /lib/libmd.so.1
libscf.so.1 => /lib/libscf.so.1
libintl.so.8 => /opt/csw/lib/libintl.so.8
libiconv.so.2 => /opt/csw/lib/libiconv.so.2
libdoor.so.1 => /lib/libdoor.so.1
libuutil.so.1 => /lib/libuutil.so.1
libgen.so.1 => /lib/libgen.so.1
libm.so.2 => /lib/libm.so.2
zephyr@zephyrus:~/repos$
[] git-remote-https command.
Digging further, I noticed that there is
git-remote-https command under /opt/csw/libexec/git-core directory.
This seems to be used by git clone https: ... command.
(I found this out Using GIT_TRACE=1 environment variable.)
zephyr@zephyrus:~/repos$ ldd /opt/csw/libexec/git-core/git-remote-https
libpthread.so.1 => /lib/libpthread.so.1
libintl.so.8 => /opt/csw/lib/libintl.so.8
libz.so.1 => /opt/csw/lib/libz.so.1
libiconv.so.2 => /opt/csw/lib/libiconv.so.2
libsocket.so.1 => /lib/libsocket.so.1
libnsl.so.1 => /lib/libnsl.so.1
libresolv.so.2 => /lib/libresolv.so.2
libcrypto.so.0.9.8 => /opt/csw/lib/libcrypto.so.0.9.8
libcurl.so.4 => /opt/csw/lib/libcurl.so.4
libexpat.so.1 => /opt/csw/lib/libexpat.so.1
libc.so.1 => /lib/libc.so.1
libmp.so.2 => /lib/libmp.so.2
libmd.so.1 => /lib/libmd.so.1
libscf.so.1 => /lib/libscf.so.1
libdl.so.1 => /lib/libdl.so.1
libidn.so.11 => /opt/csw/lib/libidn.so.11
libssl.so.0.9.8 => /opt/csw/lib/libssl.so.0.9.8
libdoor.so.1 => /lib/libdoor.so.1
libuutil.so.1 => /lib/libuutil.so.1
libgen.so.1 => /lib/libgen.so.1
libm.so.2 => /lib/libm.so.2
zephyr@zephyrus:~/repos$
So it uses libcurl and libssl from /opt/csw/lib.
This openssl may have an issue about point 2 noted above. Oh well.
[] Comparison of versions : solaris 10 vs linux
On Solaris opencsw version (where git clone https: ... failed)
curl-config --version
libcurl 7.16.4
curl --version
curl 7.16.4 (i386-pc-solaris2.10) libcurl/7.24.0 OpenSSL/0.9.7m zlib/1.2.5 libidn/1.24
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp
Features: IDN IPv6 Largefile NTLM SSL libz
On linux (where success): somehow I could not find curl-config
/usr/bin/curl --version
curl 7.24.0 (i486-pc-linux-gnu) libcurl/7.24.0 OpenSSL/1.0.0h zlib/1.2.6 libidn/1.24 libssh2/1.2.8 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtmp rtsp scp sftp smtp smtps telnet tftp
Features: Debug TrackMemory GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz
zephyrus@debian-vm:~/repos$
Note that OpenSSL is 1.0h under linux while it is at 0.9.7m on solaris
(!). This may well be the problem.
[] Simple curl execution: libcurl operaton failure?
I checked the basic operation of libucurl by running the following
command both under solaris10 (failure) and Debian GNU/Linux (success).
curl -v https://myuserid@bitbucket.org/myuserid/bb101rep.git
The difference I noted was immediate. Verbose output from curl showed
the following near the beginning under linux while it didn't on
solaris. So maybe sslv3 handshake is not woriking due to some reason.
(From linux dump)
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
Maybe the following lines from the output of curl
shows some relaions?
Note the following points.
From Solaris 10 log:
* SSL: couldn't set callback! (<--- ? point 1 ?)
* successfully set certificate verify locations:
* CAfile: none
CApath: /opt/csw/ssl/certs
* WARNING: failed to configure server name indication (SNI) TLS
extension (<===? point 2??)
* SSL connection using AES256-SHA
[] Maybe the smoking gun, finally.
Regarding point 2 in the preceding section, I found a reference to a
bug in openssl:
http://curl.haxx.se/mail/lib-2009-09/0294.html
but it was "the problem was introduced in 0.9.7f and was fixed in
0.9.8k" and so should not be a problem in 0.9.7m which opencsw curl
uses.
But who knows if other bugs in pre 1.0.0 openssl lurk. For example
the following bug with OpenSSL 0.9.8g
http://rt.openssl.org/Ticket/Display.html?id=1629
also referred to in
http://serverfault.com/questions/303090/stunnel-wont-work-with-sslv3-from-some-hosts
may be significant here.
Regarding point 1, I found the following post. This may also
contribute to the problem. It seems that if curl is BUILT against
sfw libssl, it fails in a subtle manner (!).
RE: couldn't set callback! (RESOLVED)
http://curl.haxx.se/mail/archive-2010-04/0093.html
I suspect that the above post may be relevant and points at the
culprit (It is solaris 10 specific!)
In any case, I will post my finding to opencsw and ask for building
openssl 1.0.0h built and rebuilding curl library against it, and then
git as well.
My conclusion is that if openssl based on 1.0.0h is made available,
and curl, libcurl, and git are recompiled, then the problem with
"git clone https:..." I saw may not happen.
Reference Log
[] curl log: failure under Solaris 10
curl -v https://bitbucket.org/userid/bb101repo.git/info/refs?service=git-upload-pack
* About to connect() to bitbucket.org port 443 (#0)
* Trying 207.223.240.182...
* connected
* Connected to bitbucket.org (207.223.240.182) port 443 (#0)
* SSL: couldn't set callback!
* successfully set certificate verify locations:
* CAfile: none
CApath: /opt/csw/ssl/certs
* WARNING: failed to configure server name indication (SNI) TLS extension
??? key exchange is missing here in comparison to linux dump ...???
* SSL connection using AES256-SHA
* Server certificate:
* subject: 2.5.4.15=Private Organization; 1.3.6.1.4.1.311.60.2.1.3=US; 1.3.6.1.4.1.311.60.2.1.2=Delaware; serialNumber=3928449; streetAddress=375 Alabama St Ste 325; postalCode=94110; C=US; ST=CA; L=San Francisco; O=Atlassian, Inc.; CN=bitbucket.org
* start date: 2012-01-03 00:00:00 GMT
* expire date: 2014-03-12 12:00:00 GMT
* subjectAltName: bitbucket.org matched
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert High Assurance EV CA-1
* SSL certificate verify ok.
> GET /userid/bb101repo.git/info/refs?service=git-upload-pack HTTP/1.1
> User-Agent: curl/7.16.4 (i386-pc-solaris2.10) libcurl/7.24.0 OpenSSL/0.9.7m zlib/1.2.5 libidn/1.24
> Host: bitbucket.org
> Accept: */*
>
< HTTP/1.1 404 NOT FOUND
< Server: nginx/1.0.6
< Date: Fri, 30 Mar 2012 18:04:59 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: close
< Vary: Cookie
<
html payload omitted. ...
...
* Closing connection #0
zephyr@zephyrus:~/repos$
[] curl log : success under LINUX
curl -v https://userid@bitbucket.org/userid/bb101repo.git
* About to connect() to bitbucket.org port 443 (#0)
* Trying 207.223.240.182...
* connected
* Connected to bitbucket.org (207.223.240.182) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES256-SHA
* Server certificate:
* subject: businessCategory=Private Organization; 1.3.6.1.4.1.311.60.2.1.3=US; 1.3.6.1.4.1.311.60.2.1.2=Delaware; serialNumber=3928449; street=375 Alabama St Ste 325; postalCode=94110; C=US; ST=CA; L=San Francisco; O=Atlassian, Inc.; CN=bitbucket.org
* start date: 2012-01-03 00:00:00 GMT
* expire date: 2014-03-12 12:00:00 GMT
* subjectAltName: bitbucket.org matched
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert High Assurance EV CA-1
* SSL certificate verify ok.
* Server auth using Basic with user 'userid'
> GET /userid/bb101repo.git HTTP/1.1
> Authorization: Basic emVwaHlydXMwMGpwOg==
> User-Agent: curl/7.24.0 (i486-pc-linux-gnu) libcurl/7.24.0 OpenSSL/1.0.0h zlib/1.2.6 libidn/1.24 libssh2/1.2.8 librtmp/2.3
> Host: bitbucket.org
> Accept: */*
>
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 404 NOT FOUND
< Server: nginx/1.0.6
< Date: Fri, 30 Mar 2012 18:17:29 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: close
< Vary: Cookie
<
... payload omitted ...
* Closing connection #0
remote: Counting objects: 3, done.
remote: Compressing objects: ...
...
Unpacking objects: 100% (3/3), done.
zephyrus@debian-vm:~/repos$ exit
exit
Script done on 2012年03月31日 01時45分21秒
