3

I setup a new VPS with ubuntu 18.04, including virtualmin/usermin. In auth.log I see a lot of

su[12936]: Successful su for domain by root
su[12936]: + ??? root:domain 
systemd-logind[148]: New session c315 of user domain .
su[12936]: pam_unix(su:session): session opened for user domain by (uid=0)
su[12936]: pam_unix(su:session): session closed for user domain

in syslog, I see a lot of

systemd[1]: Started Session c314 of user domain.
systemd[1]: Started Session c315 of user domain.

domain is the user of my virtual server defined in the VPS. c314/c315 increased by 1 each time... It used to appear every 2-3 minutes, now it's every 5 minutes.

Reading on the internet about this, all the "solutions" were how to remove this logging from the log but nothing was explaining what are all those open/close sessions in the first place.

Also, when running loginctl list-sessions those sessions are accumulated in "active=yes" and "state=closing" mode and never disappear from the list. At the moment there are 95 such sessions.

What is happening on my VPS, who is opening/closing sessions so many times and why? Also, why those sessions never disappear from the sessions list?

Thanks

update

loginctl session-status c315
c315 - domain (1000)
           Since: Sat 2020-02-08 20:27:08 UTC; 23h ago
          Leader: 12936
             TTY: ???
          Remote: user root
         Service: su; type tty; class user
           State: closing
            Unit: session-c315.scope

Unit user-1000.slice (/user.slice/user-1000.slice):
└─session-2691929.scope
├─19035 sshd: domain [priv]
├─19051 sshd: domain@pts/0
├─19052 -bash
├─20124 sudo systemd-cgls -u user-1000.slice
├─20125 systemd-cgls -u user-1000.slice
└─20126 pager
Amos
  • 237
  • 1
  • 4
  • 10
  • **pam_unix** sessions exit normally: `who` will show you only a couple of records. **systemd-logind** (through **pam_systemd**) doesn't close its session: you should see `systemd-logind[148]: Session c315 logged out. Waiting for processes to exit.` and `systemd-logind[148]: Removed session c315.` when it does. Add the status of the unclosed sessions: `loginctl session-status c315` to show which processes are still running under it. – Piotr P. Karwasz Feb 09 '20 at 18:31
  • As for who is starting those session, IIRC your [previous question](https://serverfault.com/q/1001236/530633), neither **systemd** nor **cron** can do it, but add the result of `systemctl list-timers` and `cat /etc/cron.d/* /etc/crontab /var/spool/cron/crontabs/* | grep -v '^#'` – Piotr P. Karwasz Feb 09 '20 at 18:39
  • 1
    I updated my question with more info, the open/close is happening every 5 minutes, nothing is scheduled for 5 minutes neither in cron nor timers.... I do wonder what is causing all this but I'm more surprised that no one else encountered it. Those who wrote about it, were satisfied solely by knowing how to remove it from the logging... – Amos Feb 09 '20 at 19:39
  • The session status you added looks like an **ssh** session, although it was opened by `su`. Are all the sessions similar? – Piotr P. Karwasz Feb 09 '20 at 19:59
  • All sessions remain state=closing. I did connect with user domain to ssh (instead of root) so you probably see this but that also means, nothing else is happening with user domain. Maybe this is how ubuntu is handling a user that was sudo'ed? I don't know. – Amos Feb 09 '20 at 20:53
  • One of the advantages of `systemd-logind` is that it isolates every session into its own [cgroup](https://en.wikipedia.org/wiki/Cgroups). The ssh connection, bash and `systemd-cgls` command you see, where launched inside session `c315`. Until they exit, the session will be in a **'closing'** state. – Piotr P. Karwasz Feb 09 '20 at 21:08
  • Perhaps there's a misunderstanding. I selected c315 randomly and ran the slice command on user domain just to see the output so the output of slice has nothing to do with c315 session. If the above does give you an idea on what I can do to dig further, do let me know :) – Amos Feb 09 '20 at 21:18
  • A misunderstanding indeed: usually the output of `loginctl session-status` contains the list of all the processes in the session. Your list is empty, so the mystery of never-closing sessions persists. – Piotr P. Karwasz Feb 09 '20 at 21:36
  • As you can see in auth.log, it is opening and closing immediately. A real mystery. Do you have any idea where I can ask this question? I'm lost... It's over 400 sessions now... I suddenly found this: https://github.com/systemd/systemd/issues/8598 does it make any sense? – Amos Feb 09 '20 at 21:41
  • FWIW: in /etc/systemd/logind.conf everything is in comment – Amos Feb 10 '20 at 07:34

1 Answers1

1

pam_unix sessions exit normally, as seen in the logs. Those increasing number of sessions are systemd-logind sessions, which for some reason remain open, even when they don't contain any processes.

A workaround you might try would be to force systemd-logind to kill all the session processes, when the session leader exits. You can do it by modifying the KillUserProcesses and KillOnlyUsers setting in /etc/systemd/logind.conf:

KillUserProcesses=yes
KillOnlyUsers=domain

and restarting systemd-logind:

systemctl restart systemd-logind

However, this does not answer the question, why the sessions are not closing by themselves, since the session scopes are empty.

Edit: About the difference between pam_unix and systemd-logind sessions:

  • pam_unix sessions consist in a small record added to or removed from /var/run/utmp. You can list them with w or who,
  • systemd-logind sessions are more heavy, as explained in the manpage of pam_systemd. Dangling systemd-logind consume much more resources. They are listed with loginctl list-sessions

Since you identified a possible culprit (in a comment to this answer), you can apply another workaround: replace

@include common-session

in /etc/pam.d/su with:

@include common-session-noninteractive

which does not contain pam_systemd. When you modify PAM files, the usual precautions apply: keep a root shell active (e.g. sudo -i) until you tested the new config, in case you break something.

Piotr P. Karwasz
  • 5,292
  • 2
  • 9
  • 20
  • Here is the source! https://www.cloudmin.com/node/54674 question remains about why session remain open.... – Amos Feb 10 '20 at 17:22
  • w/who show only my own session but loginctl list-sessions shows 652 (and counting) sessions... what are those sessions in loginctl list-sessions? – Amos Feb 10 '20 at 17:35
  • oh, one more thing: I connected as root using ssh, exited and re-connected with root and waited. Both sessions were in loginctl list-sessions.... – Amos Feb 10 '20 at 17:58
  • what do you mean by "keep a root shell active"? I understand that the change simply won't list those sessions, it won't really "fix" the problem itself, right? – Amos Feb 10 '20 at 19:48
  • I mean: keep a shell running as **root** open, not just a shell that can `sudo` or `su` to root, because breaking `PAM` will stop you from becoming root again. The second workaround will stop the creation of **systemd-logind** sessions, it doesn't fix what's wrong with those already created. – Piotr P. Karwasz Feb 10 '20 at 19:52
  • Virtualmin has a reason for opening those sessions and I prefer not to break them without knowing what I'm doing. I posted a question in virtualmin forum, but it's the community so I hope I will get some answers – Amos Feb 10 '20 at 20:21
  • Virtualmin just wants to execute a command, (`openssl` according to your link), it doesn't need any kind of session. A session is just a way to keep common processes together. – Piotr P. Karwasz Feb 10 '20 at 21:36
  • I admit that I'm a bit afraid of doing that, that's a production server... – Amos Feb 10 '20 at 22:50
  • I got a test vps, installed centos7 + virtualmin and I see 9 sessions, 7 of them are of the domain user, 2 of root. There is no increase over time but those 9 are there with no intention to go away. I see a session created for root (not for the domain user!) every 5 minutes but it’s not on the list. I wonder if it’s a bug with ubuntu 18.04.4 or a switch I forgot to turn on or off. ssh sessions also remain in the list for good. I setup this vps quickly to test it and not the comprehensive setup I did with my real VPS. – Amos Feb 10 '20 at 23:08
  • After several hours I have 14 sessions in the list (5 new are of root and not the domain user). I remind you that this is one difference from ubuntu, the open/close was of root and not of the domain user. Looks like the issue is happening also with centos but with much less effect. The question remains whether it's an ubuntu problem or a setting of some kind. – Amos Feb 11 '20 at 06:57
  • The question is whether those sessions contain processes or are empty. – Piotr P. Karwasz Feb 11 '20 at 07:00
  • I'm back to my server, will a pstree -p output help here? – Amos Feb 11 '20 at 18:38