Running Solr with SystemD: user limitations (ulimit) applied at runtime different from the configured limitations

Question

I'm trying to run solr as a SystemD service. When I start the service I get this warning then the solr server stops.

jun 22 16:20:07 solr_start[1488]: *** [WARN] ***  Your Max Processes Limit is currently 14972.
jun 22 16:20:07 solr_start[1488]: It should be set to 65000 to avoid operational disruption.
jun 22 16:20:07 solr_start[1488]: If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
jun 22 16:20:09 solr_start[1488]: Warning: Available entropy is low. As a result, use of the UUIDField, SSL, or any other features that require
jun 22 16:20:09 solr_start[1488]: RNG might not work properly. To check for the amount of available entropy, use 'cat /proc/sys/kernel/random/entropy_avail'.
jun 22 16:20:14 solr_start[1488]: [146B blob data]
jun 22 16:20:14 solr_start[1488]: Started Solr server on port 8983 (pid=1579). Happy searching!
jun 22 16:20:15 solr_stop[1680]: Sending stop command to Solr running on port 8983 ... waiting up to 180 seconds to allow Jetty process 1579 to stop gracefully.

I checked the code at https://github.com/apache/lucene-solr/blob/master/solr/bin/solr#L1509. I can see that solr runs a check on ulimit -u and ulimit -n.

The strange thing is that I made sure SystemD would run the solr server with the "solr" user.

[Unit]
Description=Apache SOLR
After=syslog.target network.target remote-fs.target nss-lookup.target systemd-journald-dev-log.socket
Before=multi-user.target
[Service]
User=solr
#PIDFile=/mnt/solrdata/solr-8983.pid
Environment=SOLR_INCLUDE=/opt/solr/bin/solr.in.sh
ExecStart=/opt/solr/bin/solr_start
ExecStop=/opt/solr/bin/solr_stop
#Restart=on-failure
[Install]
WantedBy=multi-user.target​

And this "solr" user (that I configured especially to run Solr server) can create an unlimited number of processes and open an unlimited number of files.

[solr@xxx ~]# ulimit -n
unlimited
[root@xxx ~]# ulimit -u
unlimited

When SystemD starts solr I see it uses the correct user (the "solr" user I configured). Yet I see the error above.

When I run solr directly with the "solr" user (no with SystemD), it works.

Why are the user limitations applied at runtime when I run the Solr server myself different from the ones used when run by SystemD ???

Answer 1

The correct way to configure limits for services is to do so within the .service unit file itself. Use parameters LimitNOFILE= and similar.

[Service]
LimitNOFILE=infinity
LimitNPROC=infinity

"User accounts" are a somewhat fuzzy concept in Linux. The full interactive login process consists of several independent steps – setting environment variables, opening a PAM session, changing the UID/GID, starting a shell in login mode and having it run /etc/profile...

Most of those steps do not apply to services. Starting a service doesn't constitute a "login" in Linux, so e.g. PAM is not called. When you specify a User= parameter in service configuration (whether in systemd, or in a daemon's own config), it does only one thing: change the UID/GID to the one you specified.

This means that any limits from /etc/security/limits.conf won't be applied (because that would require PAM), and any commands from /etc/profile won't be run (that'd require a shell).