26

The idea would be to prevent an attacker who has stolen a root/admin account or escalated to clear his own activities or even read the traces of what he is doing. Let's assume we are under Linux, we log with auditd, have centralized logs, and we can use MAC with SELinux. But I am interested also for answers under Windows.

One solution would be to forbid all root accounts to access the logs. Logs are managed only by authorized processes on specific servers from logrotate, syslog, and all SIEM stuff. So only the SOC can read and analyse the admins' logs. Only a purge process can delete old logs. Can anyone confirm this is doable?

Is it possible to have something more flexible where admins with their own root privileges could read the logs of other root accounts?

schroeder
  • 123,438
  • 55
  • 284
  • 319
lalebarde
  • 587
  • 1
  • 5
  • 13
  • 14
    The generalized version of the answers: *any log should happen on a location which is unreachable for the logged entity*. In the case of a linux roots, it means mainly remote logging to other servers. So he will be able to turn off the log, but not to hide its trace before that (including the logging turnoff), except if he is doing it very smartly. – peterh Sep 21 '18 at 21:39
  • 3
    `*.* /dev/lp0`. – R.. GitHub STOP HELPING ICE Sep 22 '18 at 20:33
  • Windows does have a version of RBAC. I'm not a windows guy, so you will have to research the details yourself, but it might be worth checking this as an alternative. – Tom Sep 24 '18 at 08:31
  • I agree logs shall be centralized. Regarding my _flexible_ request, what about retrieving or copying each admin event provided by auditd in a log file dedicated to this admin account and manage access to these log files with a policy preventing one admin to access his logs with help of SELinux? Retreiving or copying would be performed by say a 1' cron job. – lalebarde Sep 25 '18 at 09:42
  • With your addition of the fact that you already have centralised logging, the rest of the details do not make as much sense. – schroeder Sep 25 '18 at 15:51
  • Why? Admins normally can access the centralized log server. Preventing an admin to access or modify his own traces is not obvious even with ACL and SELinux. Your answer §2 looks the right way. A detailed example would be welcome. – lalebarde Sep 25 '18 at 16:20
  • An admin of a server will not automatically have access to anything on the log server. If you are talking about restricting the access of the log server's root user on the log server, then your description of the fact that your logs are centralised is moot. And even my proposed answer below is useless. It really seems like you changed the fundamental aspects of your question after you received answers. – schroeder Sep 25 '18 at 19:17
  • No, it was obvious for me in the initial description that the logs was centralized : _"Logs are managed ..... on specific servers"_. But it looks like I was wrong. I propose to remove my EDIT, select one of the answer, and open a new question targetted on the log server. What do you think is the best ? – lalebarde Sep 26 '18 at 10:08

7 Answers7

44

The accepted solution to this is to not store the logs locally, but on a log server. Once the logs are there, you can restrict or limit access as you see fit.

In some log server/aggregator solutions, you can limit a user from seeing entries that contain references to certain data (like their user accounts or machine IPs). This means that you can enable admins to see other admin activity, but not their own.

You typically want to place alerts within the log server/aggregator to trigger if logs from any one machine stop coming in or are reduced below certain thresholds, which helps to detect if a local admin has prevented the logs from being shipped to the log server/aggregator.

Syslog servers, SIEMs, log aggregators, ELK stacks etc. There are numerous options for you to explore.

schroeder
  • 123,438
  • 55
  • 284
  • 319
  • 1
    My favorite old-school solution. Fill up the disk on which logging is done. Unless that disk is several tens of TB I'll have it filled with dummy logs in short order. – Joshua Sep 22 '18 at 02:47
  • 25
    I thought the old school solution for secure logs was a line printer... – Paul Sep 22 '18 at 03:49
  • Of course, this just raises the question "How do I partition privileges in a distributed system?" which is a far more interesting one IMHO. – Kevin Sep 23 '18 at 16:50
  • 1
    @Joshua Logging software often throttles excessive triggered logs and prioritizes emergency or alert-level messages. It'd be hard to fill up the disk just by spamming some action that creates a log (e.g. writing to a deprecated procfs entry), and unless you have root, you won't be able to fill up the entire disk anyway, since some extra space is reserved for daemons (like logging daemons) running as root, at least on certain filesystems. – forest Sep 24 '18 at 02:42
  • @forest: The trick is well-targeted for taking out remote-logging systems. It works less well on local ones because you really do need root to forge log entries with local loggers. – Joshua Sep 24 '18 at 02:55
  • 2
    @Paul although [Cliff Stoll had to make do with character printers and terminals](https://en.wikipedia.org/wiki/The_Cuckoo%27s_Egg) – dave_thompson_085 Sep 24 '18 at 03:20
  • @schroeder a concrete example of your §2 with some standard log server/aggregator solution would be welcome please – lalebarde Sep 24 '18 at 13:12
  • @lalebarde Not wanting to promote any particular product, Splunk can do this. I know that the logic can also be used by other products. If the product can mask things like credit card numbers or national ID numbers, it can block user access to specific data (like usernames). – schroeder Sep 24 '18 at 13:25
12

Any logs on a compromised host are suspect. You need a centralized logging platform, either a central syslog server/ splunk / logrhythm / whatever. Keep a different set of administrators and accounts. That's the whole idea.

Once you get a platform in place you can delegate the rights to view their actions, either their own or other admins - can be performed. We had rights read specific log sources and hosts delegated out.

Tim Brigham
  • 3,762
  • 3
  • 29
  • 35
2

If an attacker acquired high privileges on a machine, then the whole machine is untrusted anymore, let alone log controllers.

A remote logging server is the only option here, though details may vary. You'll be able to handle logs and manage access controls in a safer way than storing the logs locally.

iBug
  • 1,378
  • 1
  • 9
  • 12
1

First step is to send the logs to a different server, so that after the first server is compromised (and thus their logs could be altered), that log server would have those log entries. And this server can (should) be protected more tightly.

Nevertheless, someone must be able to administrate that server, if only so that the server can be updated!

Ways you can additionally protect the logs in that server:

  • Require multiple admins for any (non-trivial) action (eg. commands introduced require the confirmation from N admins)
  • The logs can be stored linking to earlier ones in a Merkle tree, so that removal or modification of an entry would affect any later ones¹
  • The hash of the current log could be sent to an external party (getting from a time stamping server, sending a partner company, or simply sending to a twitter feed²).

¹ When arguing to use this it may sell better to state that this uses blockchain technology.

² Or if the logs itself are not confidential, you could even push some log server events in the clear: "jdoe logged into the log server and run rm -rf /"

Ángel
  • 17,578
  • 3
  • 25
  • 60
  • 1
    An often cited method to publish the log hash nowadays is to publish the hash in an existing public blockchain that allows arbitrary message like Bitcoin. That'll make the log as tamper proof as the blockchain itself. Though do note this still only prevent undetectable tampering, but it wouldn't prevent log destruction. For that, you'll need proper backup. – Lie Ryan Sep 23 '18 at 02:48
  • What are the concrete tools please behind your solutions, especially the multiple admins and Merkle tree ones ? – lalebarde Sep 24 '18 at 13:49
1

Some possible ways to protect logs from the administrator:

  • remote logging to a machine that's not controlled by the administrator in question
  • write the log to a write-once media (e.g. Bluray Recordable), note you may also need a strategy to prevent overfilling the media, such as disconnecting all admins if the logs are about to get full
  • publish the hash of the logs to a public blockchain, this detects tampering, though not log destruction
  • force access to the machine via an mitm server, admins can only connect to the administered machine through the mitm server, which will record all key/mouse inputs and screen/terminal output (using either ssh mitm or vnc/rdp mitm)
Lie Ryan
  • 31,089
  • 6
  • 68
  • 93
  • Write-once media is a logistics nightmare, but a simple and effective solution for preventing modification of captured logs (but not access to the logs). Privileged Access Servers are an awesome solution for admin access but does not help if high-priv access is gained to the target through low-priv access (exploits). – schroeder Sep 23 '18 at 13:05
  • Wouldn't deploying a mitm server increase too much the attack surface? Is it easyto secure? – lalebarde Sep 24 '18 at 13:19
0

One solution is to use a data diode to protect access to the centralized log collector server, which is then only accessible physically, what is easy to control. In addition, specific individual user accounts can be created on it for better protection.

lalebarde
  • 587
  • 1
  • 5
  • 13
  • Why would a data diode be a solution? What's the problem with 2-way communication with the server if you have authentication processes on the server? A data diode is an over-engineered solution for this use case. – schroeder Sep 25 '18 at 15:46
  • Indeed, it is a bit expensive – lalebarde Sep 25 '18 at 16:22
0

Filter auditd logs with rsyslog per uid to have a file per uid. Then SELinux can be applyed to implement any policy, especially to prevent one admin to access his own logs. The proposed solution here avoids to require an independant team of admins to access the logs on the log servers when you have a restrictive policy.

As said by others, logs shall be centralized. In addition, access to these centralized log servers shall be performed with local accounts only.

lalebarde
  • 587
  • 1
  • 5
  • 13