In the UK, the company TalkTalk was recently hacked.
It was later discovered, after 'investigation' that the hack was not as serious as it could have been (and less than expected).
I'm wondering: How do organizations (not necessarily TalkTalk -- that's just what prompted me to ask) check what has been hacked? I'm sure there are many ways; but what are the 'main' ones?
In a word: Forensics.
Computer forensics is the art of examining a system and determining what happened upon it previously. The examination of file and memory artifacts, especially file timelines, can paint a very clear picture of what the attacker did, when they did it, and what they took.
Just as an example - given a memory dump of a Windows system, it is possible to extract not only the command lines typed by an attacker, but also the output that they saw as a result of running those commands. Pretty useful in determining impact, eh?
Depending on the freshness of the compromise, it's possible to tell quite a lot about what happened.
@AleksandrDubinsky suggested that it would be useful to outline the various Computer Forensic fields and techniques, which I'm happy to do for you. They include, but are not limited to, the following (I'm going to use my rough terms; they aren't official or comprehensive):
Log/Monitor Forensics: The use of 'external' data such as centralized logs, firewall logs, packet captures, and IDS hits straddles the line between "Detection" and "Forensics". Some of the data, like logs (even centralized) cannot be trusted to be complete or even truthful, as attackers can filter it or inject it once they have control of the system. Off-the-system packet logging tools like the firewall, IDS, or packet recorders (such as (formerly)NetWitness) are unlikely to be tampered with, but contain only a limited amount information; usually just a record of IP conversations and sometimes signatures (such as HTTP URLs) associated with malicious activity.
Unless an unencrypted network connection was used in the compromise, these tools are rarely able to detail the activity during a compromise, and so (going back to the original question) don't "check what has been hacked." On the other hand, if an unencrypted connection (ftp) was used to exfiltrate data out, and full packets were recorded, then it's possible to know exactly what data the attacker ran away with.
Live Forensics: More properly part of Incident Response, so-called "Live" forensics involves logging into the system and looking around. Investigators may enumerate processes, look in applications (e.g. browser history), and explore the file system looking for indications of what happened. Again, this is usually designed to verify a compromise happened, and not to determine the extent of a compromise, since a compromised system is capable of hiding files, processes, network connections, really anything it wants to. The plus side is that it allows access to memory-resident things (processes, open network connections) that aren't available once you shut the system down to image the disk (but, again, the system may be lying, if it's compromised!)
Filesystem Forensics: Once a copy of the disk has been made, it can be mounted on a clean system and explored, removing any chance of the the compromised operating system "lying" about what files are in place. Tools exist to build timelines using the variety of timestamps and other metadata available, incorporating file data, registry data (on Windows), even application data (e.g. browser history). It's not just file write times that are used; file read times can indicate which files have been viewed and also which programs have been executed (and when). Suspicious files can be run through clean antivirus checkers, signatures made and submitted, executables loaded into debuggers and explored, "strings" used to search for keywords. On Unix, "history" files - if not turned off by the attacker - may detail the commands the attacker entered. On Windows, the Shadow Copy Service may provide snapshots of past activity that has since been cleaned up.
This is the step most commonly used to determine the scope and extent of a compromise, and to determine what data may or may not have been accessed and/or exfiltrated. This is the best answer to how we "check what has been hacked."
Disk Forensics: Deleted files disappear from the filesystem, but not from the disk. Also, clever attackers may hide their files in the "slack space" of existing files. These things can only be found by examining the raw disk bytes, and tools like The Sleuth Kit exist to rip that raw data apart and determine what it means about the past. If there was a .bash_history file, and the attacker deleted it as his last step before logging out, then that file may still exist as disk blocks, and be recoverable. Likewise, attack tools that were downloaded and temporary data files that were exfiltrated can help determine the extent of the compromise.
Memory Forensics: If the investigator can get there soon enough, and get a snapshot of the memory of the system involved, then they can thoroughly plumb the depths of the compromise. An attacker must compromise the kernel to hide from programs, but there's no way to hide what's been done if a true image of kernel memory can be made. And just as the disk blocks contain data that has since been removed from the filesystem, the memory dump contains data that was in memory in the past (such as command line history and output!) that has not yet been overwritten... and most data is not overwritten quickly.
Want more? There are training programs and certifications to learn Forensics; probably the most common publicly available training is from SANS. I hold their GCFA ("Forensic Analyst") certification. You can also review the Challenges from the Honeynet Project, in which parties compete to unravel the cases given disk images, memory dumps, and malware samples from actual compromises - they submit their reports of what they found, so you can see the sorts of tools used and findings made in this field.
Anti-Forensics is a hot topic in the comments - basically, "can't the attacker just hide their traces?" There are ways to chip away at it - see this list of techniques - but it's far from perfect and not what I'd call reliable. When you consider that "the 'God' of cyberespionage" went so far as to occupy the firmware of the hard drive to maintain access to a system without leaving a trace on the system itself - and still got detected - it seems clear that, in the end, it's easier to detect evidence than it is to erase it.
Working within the technology sector, how I would do it would be as follows:
Find the hack, the hole, the weak spot.
Check the syslog, dmesg, access.log and error.log to confirm the thesis. (when hacking in to a network you normally have the break the system in the case of TalkTalk it was a MySQL injection, doing this would have left errors in the mysql.err log as to break in you have to break it.)
Replicate the hack, replicating the hack will allow you to see how deep the hack can go and allow you to estimate the damages.
If you find the hole, you can replicate the attack, replicate the attack and you will see what the attacker saw.
In addition to the excellent answer on Forensics (and comments) some organisations use intrusion detection systems such as 'Snort' (network intrusion detection) and OSSEC (host based detection system) as well as many others by vendors such as Cisco etc. which record various aspects of unusual or unauthorised behaviour.
For example, host based intrusion detection systems often log file changes on servers so a forensic investigation can show which files have been changed within a given period and a forensic investigator can use this to piece together activities of some users or attackers.
Attackers will also leave traces in logs all over the infrastructure from firewall logs, to server logs, service logs (web, smtp etc.) and these can provide vital clues. However, it is possible for attackers to change these logs unless they have been designed to be secure and unalterable e.g. by centralising logs on a write-only medium.
Once an attacker gains privileged access the gloves are off and logs can be tampered with or deleted.
In the end it will rest on readiness - whether the organisation has prepared for intrusion and the forensic exercise that will follow. This should be part of every company's security planning.
I haven't seen this mentioned yet and it is likley a major resource for such an investigation, especially one revolving around data access:
RDBMS auditing can be a major resource for determining exactly what data has been accessed, when, and by whom. This of course depends on how, or if, it has been configured.
For example, SQL Server Audit allows retrieval of user IDs, time stamps, and the actual queries submitted to the server thus allowing complete re-construction of the data set stolen.
The audit systems should have redundancies built in to prevent tampering with the audit trail. Some of this may be stored in the RDBMS, and some may be written to external sources. SQL Server can log audits to the Windows Security log.
gowenfawr mentioned system forensics but identifying the compromised asset is almost always done via logs. Logs /should/ always indicate the compromised asset if you search hard enough. Whether you have an IDS that detects external logins/bruteforce attempts/SQL injection attempts or if you manually search logs for what I just mentioned, there are several ways to bring a breach to your attention.
Once the asset has been identified, you can begin forensic examination of the asset. Mostly for additional information that logs may not report, especially if it looks like a file has been dropped to maintain access for the attacker. Memory scraping and analyzing what the dropped file does, step-by-step, will help the security team/IR team to gain an understanding of what's going on, how the compromise initially occurred, the vulnerability utilized and potentially the source of the attack (the real source).
They can check several things that I can mention randomly:
If someone broke into your house using a stolen key, and did not steal anything, how would you know? Often, you would. Think about how the ways in which you'd detect the occurrence. It could be as silly as a letter open on your kitchen worktop that you don't remember getting around to reading. There is basically no distinction here. It helps if you find scratch marks on your lock from the first ten entry attempts before the burglar finally determined which key was actually yours (read: security logs).
In terms of what has been taken, rather than how, bigger enterprises and agencies utilise deep packet inspection to record what has been transferred in/out of their network.
Devices record all traffic and can utilise IDS for alerting. After an incident occurs they can analyse the magnitude of the attack.
Some common industry standard tools for this are:
In the case of Talk Talk, we know that the stolen data is being used to target people.
There is a LOT of monitoring going to by the UK banks etc to see what the impact is, this monitoring will pick up enough cases, along with reported cases by bank customers, to give a good ideal of the impact by now.
