Putting a whole linux server under source control (git)

Question

I am thinking about putting my whole linux server under version control using git. The reason behind it being that that might be the easiest way to detect malicious modifications/rootkits. All I would naively think is necessary to check the integrity of the system: Mount the linux partition every week or so using a rescue system, check if the git repository is still untempered and then issue a git status to detect any changes made to the system.

Apart from the obvious waste in disk space, are there any other negative side-effects?

Is it a totally crazy idea?

Is it even a secure way to check against rootkits since I most likely would have to at least exclude /dev and /proc ?

Answer 1

That's a "Bad Idea" (tm). Aside from all else, your repository will run slow as all heck, and get worse as every revision is kept.

Try centralised management, like puppet / cfengine / chef. That'll keep things as you expect, and revert unexpected changes.

Combine that with something like iwatch to get emails of unauthorised file alterations.

Combine that further with rpm/deb files if needed to roll out custom applications.

Throw in something like rkhunter or chkrootkit now and then for kicks and you should be good to go.

Job done.

Answer 2

Another alternative is to set up tripwire, which is GPL'ed software that spiders through all the important files on your system and determines which have changed in ways you have defined as unacceptable. Change can be defined as simply as mtime, through inode number, all the way to cryptographically-strong checksums.

It takes some setting up and tuning if you don't want to get a whole lot of reports every night about changed files in /var/run, changes in DHCP client files in /etc, and the like, but if you do go to that trouble, it can be very helpful indeed.

The database of file properties is signed with a key not known to the machine, which helps you have confidence that no tool has maliciously changed the database or the tripwire binaries. For complete certainty you can burn a copy of the tripwire tools and databases to a read-only medium, which can be mounted on the server and used to verify all changes since the disc was burned, if a complete forensic analysis is needed.

If you're going to do this, it's quite important to get tripwire set up and running before the machine is deployed into production, or you can never be completely sure that some malicious user didn't have a chance to infect the machine before it was tripwired.

Answer 3

I don't think this is likely to work, but as an experiment I'd like to see what happens if you do this with just the /etc folder. That's where most of the configuration information is kept.

Answer 4

@Sirex provided a very good answer already, but if you want to make a step further with security, the best way to deal with it is first prevention then detection.

Try setting up a system with / filesystem mounted read-only. Make /tmp a separate ramfs mounted with noexec,nodev option. For the system to work, you really only need /var to be mounted read-write. So under /var mount an fs with rw,noexec,nodev and remove write permissions to /var/tmp (afaik, it is rarely needed by daemons and that should be configurable). Also use a security patch for your kernel to further limit access to resources by users, try grsec for example. Use a firewall with the most restrictive rules possible.

Some distributions provide extensive documentation on system hardening. For example:

• Securing Debian Manual
• Hardening Debian Wiki

Answer 5

I think it's a good idea to analyse the changes a tool makes in your system:

1. install a bare Linux in a VM
2. initialise the root git
3. install the tool you want to analyse
4. see all changes the tool made in your system

... Delete the VM

You would have to add a lot of folders to the .gitignore file though like proc, etc.

Answer 6

For the situations where you are just interested in keeping certain folders across the entire filesystem under version control, the following approach might work:

First, create a Git repository at the / level:

$ cd /
# git init

Then create a /.gitignore that whitelists only certain folders, for example to whitelist only /path/to/versioned/config/folder/ (based on https://stackoverflow.com/a/11018557/320594):

/*
!/path/
/path/*
!/path/to/
/path/to/*
!/path/to/versioned/
/path/to/versioned/*
!/path/to/versioned/config/
/path/to/versioned/config/*
!/path/to/versioned/config/folder/
!/.gitignore

Then create a first commit:

# git add -A
# git commit -m "Initial commit"

And then add the additional folders that you want under version control on demand.

PS:

In addition to the previous method, if you need to keep /etc/ under version control, you might prefer to use etckeeper (https://etckeeper.branchable.com/) for versioning that specific folder as it is more specialized for that purpose (e.g. it commits automatically after installing packages).