21

I'd like to know what are the best approaches for tracking superuser activities on a Linux environment.

Specifically, I'm looking for these features:

  • A) Logging keystrokes to a secured syslog server
  • B) Ability to replay shell sessions (something like scriptreplay)
  • C) Ideally, this should be something impossible (or quite difficult) to circumvent without having physical access to the server.

Think about this from a security / auditing perspective, in a environment where different sysadmins (or even third parties) need to be allowed to perform privileged operations on a server.

Every administrator would have his o her own nominal account, and every interactive session should be fully logged, with the possibility of replaying it if necessary (for example, if someone used mc to delete or alter critical files, it wouldn't be enough to know that that person issued the mc command; there must be a way to see exactly what was done after launching mc).

Additional notes:

  1. As womble has pointed out, may be the best option would be not having people logging in with root privileges to perform changes on servers, but instead doing that through a configuration management system. So let's assume a situation where we don't have such a system and we need to grant root level access to different people over the same server.
  2. I'm not interested at all in doing this surreptitiously: every person logging in to a server with root privileges would be fully aware that the session will be recorded (in the same way that, for example, call center operators know that their conversations are being recorded)
  3. No one would be using a generic superuser account ("root")
  4. I am aware of ttyrpld and it seems to do what I'm looking for. But before going that way, I'd like to know if this can be solved by using an unmodified kernel. I want to know if there are any tools for Debian in particular (or Linux in general) that allow full auditing of superuser accounts without patching the shell or the kernel.
mfriedman
  • 1,959
  • 1
  • 13
  • 14
  • 2
    (grabs chair and popcorn) this should be good... – Avery Payne Aug 06 '09 at 01:20
  • +1 ...was thinking the exact same thing. LOL – KPWINC Aug 06 '09 at 02:03
  • Also note this related question: http://serverfault.com/questions/46614/how-can-i-prevent-that-a-server-admin-can-see-data – sleske Aug 06 '09 at 08:15
  • I still think you should use a configuration management system. (puppet/cfengine/chef/systemimager/chef/etc...) – KevinRae Aug 23 '09 at 05:58
  • Kevin, I agree with you. See for example my comment to womble's answer: http://serverfault.com/questions/50710/how-to-track-superuser-activities/50713#50713. Unfortunately, that is not an option on this environment, and that's why I asked to assume a scenario where a configuration management system isn't avaible. Anyway, I'd like to thank you for your feedback on this topic. – mfriedman Aug 23 '09 at 11:31

10 Answers10

8

For environments with multiple admins just don't use root - ever if possible.

Use sudo for everything - sudo is extremely configurable and easily logable.

Log any / all logins or su's to root & investigate them as someone is then going around your established rules.

DisabledLeopard
  • 802
  • 6
  • 9
  • 3
    Yeah, sudo's got great logging -- all those "womble ran /bin/sh as root" entries are *real* helpful. Without config management, people are always going to be becoming root to do admin tasks, and someone who wanted to do something nefarious could just do their thing in the same root session as performing a valid task. The perfect cover. – womble Aug 06 '09 at 02:08
  • Policy has to discourage simply shelling out as root for this to be any good of course, and you won't be able to know what they did once they went off the reservation, but it narrows the list of suspects... – dmckee --- ex-moderator kitten Aug 06 '09 at 02:37
  • 2
    Policy: "sudo /bin/sh" = fired/investigated. Pretty clear, pretty easy solution. – Karl Katzke Aug 06 '09 at 03:24
  • 5
    there are so many ways of getting a shell from programs that people will legitimately need to run (e.g. from sudo vi) that there's little point in just blocking 'sudo /bin/sh'...unless you can be certain that you've blocked every possible method, you'd just be issuing a challenge to find more obscure ways. in any case: a) sometimes sudo /bin/sh is necessary, and b) it's a management problem, not tech. – cas Aug 06 '09 at 04:46
  • Chris makes a great point: Management problem, not tech problem. – Karl Katzke Aug 07 '09 at 04:17
2

For one, what type of root user access are you looking to monitor for? Stupid admin mistakes or malicious insider? The former -- you'll want a good configuration management solution, as has already been suggested. The latter -- if they know what they're doing, you can only hope to catch enough to indicate something happened worth investigating. You just want to know that some form of unauthorized activity started, and be alerted to that fact. If they're smart, they'll disable most of the logging you build in (by changing the server state or by bringing in their own tools) but hopefully you can catch the beginnings of the incident.

That being said, I suggest a couple of tools you can use. First, start with a good sudo policy (which has been suggested already). Second, check out sudoshell if you have need to give those admins root shell access. Third, probably your best bet (though most intensive), look into linux kernel auditing.

romandas
  • 3,242
  • 8
  • 37
  • 44
  • +1 Thank you for suggesting sudoshell, and specially for metioning Linux kernel's audit system -- that could be an excellent complement for what I'm trying to achieve. – mfriedman Aug 06 '09 at 04:35
2

What you might could do is use this library for sudo, give everyone their own useraccount and put sudo -i in everyones profile. That way they have instant root access and every command they use is being logged.

blauwblaatje
  • 953
  • 1
  • 6
  • 19
1

They've got root. The best you can hope for is to at least see when they decided to break out of your little monitoring utopia, but beyond that what they did is anyone's guess.

The "best" option I can think of is to mandate the use of pervasive configuration automation and management, and manage your manifests using a revision control system and deploy updates through that. Then prevent actual root logins to the servers. (Emergency "oh noes I broke something" access can be provided by a not-distributed-and-changed-after-each-use password or SSH key, and everyone gets to watch the sysadmin who screwed up to make sure they don't change anything).

Yes, this is going to be inconvenient and annoying, but if you're paranoid enough to want to monitor everyone's actions to this degree, I'm guessing you're in an environment which is inconvenient and annoying enough in other ways that this won't seem like a big problem.

womble
  • 95,029
  • 29
  • 173
  • 228
  • I have to agree with you. The best option would be not having people logging in with root privileges to perform changes on servers, but instead doing that through a configuration management system. I find your comments helpful for refining and clarifying my question. – mfriedman Aug 06 '09 at 01:47
1

As others have said there's pretty much no way to log users with full root access in a way they can't disable, but if you're running debian/ubuntu take a look at snoopy, which comes pretty close to what you want

snoopy is merely a shared library that is used as a wrapper to the execve() function provided by libc as to log every call to syslog (authpriv). system administrators may find snoopy useful in tasks such as light/heavy system monitoring, tracking other administrator's actions as well as getting a good 'feel' of what's going on in the system (for example apache running cgi scripts).

theotherreceive
  • 8,235
  • 1
  • 30
  • 44
0

I agree with disabledleopard's comments about using sudo for everything. It certainly makes things a bit easier to log.

I'd also add backing up the bash history file periodically. Seemingly often overlooked but can sometimes be a great source of information... just ask Goldman Sachs. ;-)

http://www.guardian.co.uk/business/2009/jul/12/goldman-sachs-sergey-aleynikov

KPWINC
  • 11,274
  • 3
  • 36
  • 44
  • 2
    i have a .bash_logout script that makes a time-stamped copy of the history to /var/lib/history/$user.$tty-or-IP.$yymmddhhss if i cared more, i'd set up process accounting or proper auditing tools...but it's not really for security, it's so i can find out who did something dumb and tell them a) not to do it again, and b) how to do it properly. increasing the clue level of juniors is much more of an issue here than trust. – cas Aug 06 '09 at 04:56
  • 1
    Reminds me of the story where a junior sales guy blows the million dollar deal. He expects the boss to fire him and the boss says, "Hell no! It just cost me a million dollars to TRAIN YOU!" I can feel the "clue level" of the juniors increasing as we speak. ;-) – KPWINC Aug 06 '09 at 05:30
0

That would be difficult ...

root can run carefully well tested scripts that can breach all security measures (kill monitoring processes), shred log files / trim them etc ... But still ...

Assuming multiple admins given root privilege are working as a team. And root can kill any monitoring process as well. And unfortunately, that login/password becomes public. Or they get unwanted company.

Creating Multiple root accounts with UID 0 though not recommended, might be applicable here.

In /etc/ssh/sshd_config Changing the line to: PermitRootLogin no

is recommended. So that, Here, a user logs in using his/her normal account (datetime stamp is logged alongwith (maybe spoofed IP Address)) Then switches to root. using su command

And Direct logging in as root is prevented like this.

We have to think, what root cant do here.

sudo should be good. Backing up /etc directory Configuration Files should be good. /var/ directory Log files should be emailed periodically or stored on separate NFS.

How about Writing Scripts that integrate APIs from Mobile Gateway companies that group SMS all root user's mobiles, that one of them is out of home to work. I know that would be irritating, but still.

Breaking SSH is mostly out of question.

0

We have the following setup on a customer's site:

  • Likewise-open to authenticate with kerberos on AD (personal accounts)
  • Authorization only to certain AD groups of Unix admins
  • sudoers group == AD group
  • OSSEC HIDS agents on every server and a manager on a hardened server
  • OSSEC Web UI
  • Splunk 3 with Splunk-for-OSSEC

It will log every sudo use on the servers and also track any changes to files, installation of packages, suspicious processes, etc.

chmeee
  • 7,270
  • 3
  • 29
  • 43
0

We have several terminal servers to access to all our equipment, that means one can log to anything either from terminal server or if one has physical access.

Sshd on terminal servers is patched with http://www.kdvelectronics.eu/ssh-logging/ssh-logging.html , works fine, but wasn't updated for a long time. I've modified it a bit to work on openssh 4.7, but failed to do that with 5.1 . Patched sshd segfaults, and as long i don't have enough time to fix that, i've almost switched to ttyrpld.

Dima Medvedev
  • 346
  • 1
  • 7
0

So far, this is what I have:

  • sudosh : seems to support A and B (not completely sure about A though)
  • Sudoscript : seems to support B (Sudoscript has a component called sudoshell, and if that's what romandas suggested, thanks for the tip)
  • Snoopy Logger or sudo_exetrace : not exactly what I'm looking for, but could be a good complement (thanks to theotherreceive and blauwblaatje for those links)

Do you know any other similar tools that don't involve patching the kernel or other system components?

mfriedman
  • 1,959
  • 1
  • 13
  • 14