"Assertion failed: n->len <= 32" in montmul_vt.c part of pkcs11_softtoken_extra

Question

My Java application makes SSL connections to https://domains.live.com:443. When moving my app to Solaris 10 from Solaris 9 those SSL connection attempts caused an assertion error in a Solaris library:

Assertion failed: n->len <= 32, file /on10/build-nd/F10U10B17/usr/src/common/bignum/sun4v/montmul_vt.c, line 437

Turns out this wasn't a Java issue. I managed to solve it but the problem was obscure enough I thought I'd provide the answer here for Google, and the next person's, benefit.

From the core file, the fault lies here:

-----------------  lwp# 1 / thread# 1  --------------------
 ff2ce364 _lwp_kill (6, 0, ff3350c0, ff2ad954, ffffffff, 6) + 8
 ff242910 abort    (ffbfb4d8, 1, 0, edbe0, ff333558, 0) + 110
 ff242b4c _assert  (b34dca18, b34dca28, 1b5, b34f049c, ed8fc, ff257cd4) + 64
 b34d6338 big_mont_mul_yf (ffbfbadc, ffbfbd5c, ffbfc214, 1121754, 49c5d7f1, 40) +                0d10 big_mont_conv (ffbfc19c, ffbfc214, 1121754, 49c5d7f1, d54e0049, ffbfbd5c) + d4
 b34d1e68 big_modexp (ffbfc214, ffbfc214, 112177c, 1121754, ffbfbd5c, 0) + 138
 b34aaa10 soft_rsa_encrypt (112177c, 1236550, 200, ffbfca94, 1121728, 1121148) + 15c
 b34ab6b0 soft_rsa_verify_common (1121140, b538c8, 23, 1236550, 200, 1) + 60
 b3498ab8 C_Verify (1121140, b538c8, 23, 1236550, 200, 7) + 154
 b389a0a4 Java_sun_security_pkcs11_wrapper_PKCS11_C_1Verify (36594, ffbfd2dc,         b36b8ad0, 1121140, ffbfd2d0, ffbfd2cc) + 5c
 f8c0c280 * sun/security/pkcs11/wrapper/PKCS11.C_Verify(J[B[B)V+0

Using "pldd" on the core file and then passing those through "/usr/ccs/bin/nm" the library file was determined to be:

/usr/lib/security/pkcs11_softtoken_extra.so.1

Answer 1

This is indeed a problem in PKCS#11 softtoken library.

The problem was fixed under bug 15707130/7033814 (new/old number).

To get the problem fixed, you need to get a patch which fixes 15707130/7033814. Solaris 10u11 contains the fix already.

Answer 2

Another answer on serverfault (Strong Cryptography in Solaris Zone) suggested disabling 'softtoken_extra' as a solution to an unrelated problem. Disabling it also solved this issue for me:

cryptoadm disable \
    provider=/usr/lib/security/\$ISA/pkcs11_softtoken_extra.so \
    mechanism=all

Turning it back on reproduced the "assertion failed" error in every case