Docker container CMAKE gives crypto/fips/fips.c:153: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE - dracut-fips crypto.fips_enabled = 1

Question

Getting crypto/fips/fips.c:153: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE when

dracut-fips package is installed and sysctl -a shows crypto.fips_enabled = 1

Restriction: Can not disable dracut-fips package or crypto.fips_enable setting in my case.

QUESTION:

How can I get cmake --version working inside the container?

This host is where the issue is coming insider the container but not of the HOST OS.

someLinuxUser@jenkins-project_team_rh ~]$ hostname -f; hostname -i
jenkins-project_team_rh.lewisville.us.company.com
10.20.20.10[someLinuxUser@jenkins-project_team_rh ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)
[someLinuxUser@jenkins-project_team_rh ~]$
[someLinuxUser@jenkins-project_team_rh ~]$ cmake --version
cmake version 3.18.2

CMake suite maintained and supported by Kitware (kitware.com/cmake).

List and Start OpenSuse 15.2 container

[someLinuxUser@jenkins-project_team_rh ~]$ sudo docker image ls
REPOSITORY                    TAG                 IMAGE ID            CREATED             SIZE
opensuse-image                15.2                618840498a55        3 hours ago         2.59GB

Run Docker container

NOW I'm inside the container. Running "cmake --version" here fails with an error.

NOTE: The same docker image will work fine if you have ANY host OS, where "sysctl -a | grep fips" will show "crypto.fips_enabled = 0" in it's output. So, we may need to have this value set as 0.

[someLinuxUser@jenkins-project_team_rh ~]$ sudo docker run -it opensuse-image:15.2 bash
Active Directory Password:
WARNING: IPv4 forwarding is disabled. Networking will not work.
docker_nonroot_user@eaa40032f4d3:~/git>


docker_nonroot_user@eaa40032f4d3:~/git> which cmake; cmake --version
/usr/bin/cmake
crypto/fips/fips.c:153: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE
Aborted (core dumped)
docker_nonroot_user@eaa40032f4d3:~/git>

docker_nonroot_user@3e63938cf7e7:~/git> cat /etc/os-release
NAME="openSUSE Leap"
VERSION="15.2"
ID="opensuse-leap"
ID_LIKE="suse opensuse"
VERSION_ID="15.2"
PRETTY_NAME="openSUSE Leap 15.2"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap:15.2"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"

Either on HOST OS (RH 7.9) --/-- inside the Docker container (OpenSuse 15.2) Running:

[someLinuxUser@jenkins-project_team_rh ~]$ sysctl -a | grep fips_enabled
crypto.fips_enabled = 1

i.e. it shows, fips is enabled on this HOST machine. I think if this (crypto.fips_enabled was set to = 0) then we may not see this issue with cmake, but in my case, I can't disable this setting.

On the Host machine, some related packages are:

[someLinuxUser@jenkins-project_team_rh ~]$ sudo yum list installed | egrep "fips|openssl|dracut"
dracut-fips.x86_64                  033-572.el7           @rhel-x86_64-server-7
fipscheck.x86_64                    1.4.1-6.el7           @anaconda/7.6
fipscheck-lib.x86_64                1.4.1-6.el7           @anaconda/7.6
CentrifyDC-openssl.x86_64           5.7.1-347             installed
openssl.x86_64                      1:1.0.2k-22.el7_9     @q1_rhel-x86_64-server-7
openssl-devel.x86_64                1:1.0.2k-22.el7_9     @q1_rhel-x86_64-server-7
openssl-libs.x86_64                 1:1.0.2k-22.el7_9     @q1_rhel-x86_64-server-7
openssl098e.x86_64                  0.9.8e-29.el7_2.3     @anaconda/7.6
xmlsec1-openssl.x86_64              1.2.20-7.el7_4        @anaconda/7.6
dracut.x86_64                       033-572.el7           @rhel-x86_64-server-7
dracut-config-rescue.x86_64         033-572.el7           @rhel-x86_64-server-7
dracut-network.x86_64               033-572.el7           @rhel-x86_64-server-7
[someLinuxUser@jenkins-project_team_rh ~]$

NOW, Proof that DOCKER image/container used above is not bad.

Using same Docker image on a different machine, same HOST OS.

This another RH 7.9 OS host machine I have, has the same docker image, when I start the same OpenSuse 15.2 image as docker container and run cmake --version, everything works fine and I don't see this error.

Running:

[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$ hostname -f; hostname -i
rh_7_9_os_machine.company.local
10.100.100.10
[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)

[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$ sudo docker image ls
REPOSITORY                     TAG                IMAGE ID            CREATED             SIZE
opensuse-image                 15.2               618840498a55        3 hours ago         2.59GB

This HOST shows:

[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$ sysctl -a 2>/dev/null | grep fips_enabled
crypto.fips_enabled = 0

Running Docker image --> container and cmake --version in it, works!

[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$ sudo docker run -it opensuse-image:15.2 bash
docker_nonroot_user@fb751d198066:~/git>
docker_nonroot_user@fb751d198066:~/git> sysctl -a 2>/dev/null | grep fips
crypto.fips_enabled = 0
docker_nonroot_user@fb751d198066:~/git>
docker_nonroot_user@fb751d198066:~/git> cmake --version
cmake version 3.17.0

CMake suite maintained and supported by Kitware (kitware.com/cmake).
docker_nonroot_user@fb751d198066:~/git> exit

YUM packages on HOST machine are:

[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$ sudo yum list installed |grep fips
fipscheck.x86_64             1.4.1-6.el7        @anaconda/7.4
fipscheck-lib.x86_64         1.4.1-6.el7        @anaconda/7.4
[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$

[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$ sudo yum list installed |grep openssl
openssl.x86_64               1:1.0.2k-22.el7_9   @rhel-7-server-rhui-rpms
openssl-devel.x86_64         1:1.0.2k-22.el7_9   @rhel-7-server-rhui-rpms
openssl-libs.x86_64          1:1.0.2k-22.el7_9   @rhel-7-server-rhui-rpms
openssl11-libs.x86_64        1:1.1.1g-2.el7      @epel
xmlsec1-openssl.x86_64       1.2.20-7.el7_4      @rhui-REGION-rhel-server-releases
[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$

[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$ sudo yum list installed |grep dracut\-fips
[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$
[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$ sudo yum list installed |grep dracut
dracut.x86_64                033-572.el7       @rhel-7-server-rhui-rpms
dracut-config-generic.x86_64 033-572.el7       @rhel-7-server-rhui-rpms
dracut-config-rescue.x86_64  033-572.el7       @rhel-7-server-rhui-rpms
dracut-network.x86_64        033-572.el7       @rhel-7-server-rhui-rpms

As you can see above, there's NO package named dracut-fips installed on this machine, assuming that's why crypto.fips_enabled = 0 and "cmake --version" works on this HOST and within container as well!!

QUESTION:

How can I get cmake --version working inside the container? when I can't uninstall dracut-fips package and disable FIPS by having crypto.fips_enabled = 0 on the first HOST OS.

Answer 1

Found the solution

WHY this issue was coming: Docker build (to create the image) was run on a machine (the other machine, where cmake was working inside the docker container)... i.e. where FIPS was not installed. During the build step, Docker file was running zypper install cmake (or yum install cmake).

As FIPS was disabled on the machine where docker image was built, cmake was installed in docker image, without understanding FIPS being enabled / dracut-fips being installed.

Then, when you copy this image and consume it on a machine where FIPS is actually ENABLED, then cmake was failing with that error mesg: crypto/fips/fips.c:153: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE

Actually there are 2 solutions.

--

Solution #1: Tag your docker image appropriately.

Summary:

PS: If you do install cmake using the above, then it'll only work on target machine where you run container, if that machine's FIPS is disabled or enabled at build time. i.e. if FIPS was enabled, you install cmake and run it on a machine where it's NOT same FIPS setting as the host, where image was built, then you'll be reading this post for help.

Better way in case you want to install cmake using the above package managers, would be, tag your docker image appropriately during image creation time i.e.:

docker build -t <image-name>-fips-enabled ... if FIPS is enabled

and

docker build -t <image-name>-fips-disabled ... if FIPS on that that machine is disabled.

That way, you can pick the correct docker image imagename-fips-enabled vs imagename-fips-disabled acc. to what your target machine FIPS setting is (where you'll actually perform docker run ... using this image).

--

Solution #2: Don't use zypper (OpenSuse) or yum if you have RedHat container. and this solution is flexible in the sense, that it's independent of FIPS setting = 0 / 1 on the host, where image was built.

I didn't use zypper/yum to install cmake inside Dockerfile, but just grabbed cmake-3.18.2-Linux-x86_64.tar.gz bundle file.

Within Dockerfile, I simply extracted this .tar.gz file inside some directory. Also I set export PATH:/path/where/I/installed/cmake-3.18.2../bin:/..some_other_paths:/...:/.... in Dockerfile inside RUN statement.

i.e. RUN export PATH=/path/where/I/installed/cmake-3.18.2../bin:/...... && <more cmds here> && <some other cmds here> && ... etc, so it can find the extracted cmake 3.18.2 for any buil-time (cmake operations) and also SET the same PATH=/... variable as ENV PATH=/.... same value used during RUN for PATH so at runtime, when the container runs, $PATH is all set for finding cmake (3.18.2 version) rather than using any existing /usr/bin/cmake or some other shit).

Dockerfile snapshot:

#    curl -k -sSf -H "X-JFrog-Art-Api:dslfhjlieurqwihlj233lk2l4j6p9usdkajdfasddl809842iijhlkhflhafOHIHFLyeaGoodLuck" \
#    -o /tmp/cmake.tar.gz https://artifactory.company.com/artifactory/some-Local/cmake/cmake-3.18.2-Linux-x86_64.tar.gz && \

and

as my umask settings were set to 022, I didn't have to do any chicken chmod operation post untar:

#    echo -e "\n-- Installing CMake ...\n" && \
#    tar -xvzpf /tmp/cmake.tar.gz -C /home/docker_nonroot_user/tools/ && \

Inside the docker container, as ENV PATH=/... was set in Dockerfile for this target path as well, the correct cmake 3.18.2 was used at docker container runtime action.

cmake installed location in my case was:

/home/docker_nonroot_user/tools/cmake-3.18.2-Linux-x86_64/bin/cmake

PATH inside docker container was:

/home/docker_nonroot_user/tools/cov-analysis/bin:/home/docker_nonroot_user/tools/cmake-3.18.2-Linux-x86_64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

RESULT:

Using a Docker image built using SOLUTION #2 above, i.e. docker image built on a machine where FIPS = 0 aka DISABLED and then later, using this same image to create a container on a totally different target host machine where FIPS = 1 aka ENABLED, I see:

87d8104d8c41:/home/docker_nonroot_user # sysctl -a|grep fips_en
crypto.fips_enabled = 1
87d8104d8c41:/home/docker_nonroot_user #
87d8104d8c41:/home/docker_nonroot_user # which cmake
/usr/bin/cmake
87d8104d8c41:/home/docker_nonroot_user #
87d8104d8c41:/home/docker_nonroot_user # cmake --version
crypto/fips/fips.c:153: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE
Aborted (core dumped)
87d8104d8c41:/home/docker_nonroot_user #

87d8104d8c41:/home/docker_nonroot_user #
87d8104d8c41:/home/docker_nonroot_user # ls -l /home/docker_nonroot_user/tools/cmake-3.18.2-Linux-x86_64/bin
total 75504
-rwxr-xr-x 1 root root 11908568 Aug 20  2020 ccmake
-rwxr-xr-x 1 root root 12096216 Aug 20  2020 cmake
-rwxr-xr-x 1 root root 27476480 Aug 20  2020 cmake-gui
-rwxr-xr-x 1 root root 12398808 Aug 20  2020 cpack
-rwxr-xr-x 1 root root 13318712 Aug 20  2020 ctest
87d8104d8c41:/home/docker_nonroot_user #
87d8104d8c41:/home/docker_nonroot_user # ls -l /home/docker_nonroot_user/tools/cmake-3.18.2-Linux-x86_64/bin/cmake
-rwxr-xr-x 1 root root 12096216 Aug 20  2020 /home/docker_nonroot_user/tools/cmake-3.18.2-Linux-x86_64/bin/cmake
87d8104d8c41:/home/docker_nonroot_user #
87d8104d8c41:/home/docker_nonroot_user # /home/docker_nonroot_user/tools/cmake-3.18.2-Linux-x86_64/bin/cmake --version
cmake version 3.18.2

CMake suite maintained and supported by Kitware (kitware.com/cmake).
87d8104d8c41:/home/docker_nonroot_user # : Merry X-mas Baaaaeeebyyy! - no more FIPS shit error now. Next I'll fix some chown on ~<user> and close my story.

SOLUTION #3: You can disable FIPS but only if you are allowed to do so, then you don't need solution #1 or solution #2.