73

Yesterday, Twitter anounced that they recently identified a bug that stored passwords unmasked in an internal log. Recently, Github also had a similar bug. In both cases, they claim that nobody had access to these files.

Twitter:

We have fixed the bug, and our investigation shows no indication of breach or misuse by anyone.

Again, although we have no reason to believe password information ever left Twitter’s systems or was misused by anyone, there are a few steps you can take to help us keep your account safe:

GitHub:

The company says that the plaintext passwords have only been exposed to a small number of GitHub employees with access to those logs. No other GitHub users have seen users' plaintext passwords, the company said.

GitHub said it discovered its error during a routine audit and made it clear its servers weren't hacked.

To note, GitHub has not been hacked or compromised in any way.

How can Twitter and GitHub be sure that they have not been hacked or compromised in any way? Would someone who is compromising a server and read/copy a file always leave traces?

Tom K.
  • 7,913
  • 3
  • 30
  • 53
Kepotx
  • 822
  • 1
  • 8
  • 16
  • 63
    Twitter's statement is actually "shows no indication of breach" which could mean that if someone was there, there was no sign of it (this could be concluded from a bunch of different log sources that look at connections to and from said machine, which users had access to the machine in cause, etc). – sir_k May 04 '18 at 08:18
  • 42
    They did not state they are sure they haven't been hacked. They stated that they have no evidence of being hacked, but the absence of evidence is not proof of not being hacked - which they very wisely didn't claim it to be. – Adwaenyth May 04 '18 at 10:03
  • 4
    A recent article on this issue: ["It’s Impossible to Prove Your Laptop Hasn’t Been Hacked."](https://theintercept.com/2018/04/28/computer-malware-tampering/) – André Paramés May 04 '18 at 12:40
  • 8
    "Hacked" isn't even the major category of risk here. Certain Twitter employees with legitimate access could steal these without any unauthorized penetration at all. In fact, that's almost certainly how this was discovered in the first place. – Beanluc May 04 '18 at 17:33
  • Could you point out which parts of those messages indicate, to you, that they are "**sure**" that the passwords were not compromised? – Bakuriu May 04 '18 at 19:17
  • @Bakuriu as Anders said in his good answer, GitHub have quite categorical words, but Twitter is much less sure. maybe I should not have emphasized it so much on the "**sure**" – Kepotx May 04 '18 at 19:20
  • 3
    log file contents may travel unexpectedly. I once observed log file snippets exchanged between two supporters in the ticket system in a ticket I (external user) opened. The snippets were around the time of my problem, but contained many log entries concerning other users, including personal identification, cleartext passwords, cleartext security questions + answers, as well as relations between users (is assistant of). With the information, I could have dialed a phone number, ask for a specific name and tell them what their boss's favourite movie was ... So much for only exposed to a few – Hagen von Eitzen May 04 '18 at 21:19
  • 2
    Skeptics.SE answer : this is all meaningless syntactic BS. – Mazura May 05 '18 at 00:11
  • 3
    *"How can Twitter and GitHub be sure that they haven't been hacked?"* I always find these questions fascinating. @Kepotx, how can you be sure that you're not in a dream right now? – user541686 May 05 '18 at 02:01
  • @AndréParamés Every issue they mentioned in there can be mitigated with proper use of a TPM (e.g. as Qubes' Anti-Evil Maid does). – forest May 05 '18 at 02:21
  • Your question comes from a false premise. Even if they were sure that they *have* been hacked, they wouldn't openly admit it. –  May 05 '18 at 13:03
  • As they will not reveal their interna we can only guess. But for a hypothetical scenario: Assume you leak password data to a log file on a system which is able to intercept the passwords anyway. Any attacker on the system would have already hacked the system without the leak, so the leak is ugly but does not add much danger there as long as there is no attacker and if there is one they have too much access anyway. – allo May 07 '18 at 07:49
  • 3
    @MaskedMan - I would strongly disagree with your assertion. While some other organisations may try to hide a known hack (*cough* Uber *cough*), both Twitter and Github are well aware of the importance of responsible disclosure. You can be pretty sure that they've been honest here: Yes, they have both found a problem; no, they don't believe any data has leaked; buy no, they can't be certain of that so you should change your passwords. Non-disclosure almost always comes back to bite you in the end (and with GDPR coming into force, it will bite very hard indeed anyone caught covering up a hack). – Spudley May 07 '18 at 11:08
  • @Spudley It sounds like you are *agreeing* with my assertion here, even though you say otherwise. You seem to be suggesting that non-disclosure will bite them only if they get caught, and if either the probability of that happening is minuscule or if it will take ages before they are caught, then not disclosing that they have been hacked actually makes good business sense. It is all about risk versus reward. –  May 07 '18 at 14:39
  • 1
    @MaskedMan No, I'm really not agreeing with you. The risk/reward argument fails because the risk is huge (the fines possible under GDPR are eye watering), and on-going (by covering up a hack, you give yourself a permanent problem that it could be discovered at any time). You're also wide open to blackmail from anyone who knows, which means the bad guys who hacked you now have an additional avenue of attack. Anyone who thinks it through properly will understand that the only real option after being hacked is full disclosure. It may be painful and embarrassing but the alternatives are much worse – Spudley May 08 '18 at 21:55
  • @Spudley You are confusing risk with reward. The huge fine you are talking about is the "reward" (albeit it is negative in this case). Even if the "reward" is a fine of 100 billion dollars (or some such huge number), you only need to pay it if you get caught. If the probability of getting caught is 0.00000000001 ("risk"), then it makes no sense to admit you have been hacked. (That's setting aside the fact that GDPR doesn't apply in USA). –  May 09 '18 at 02:26
  • The so-called blackmailing is also nothing to worry about. If the hackers call you saying "if you don't pay up, we will disclose the hack to the public", you can just ask them to carry out their threat. Then if they are foolish enough to actually do so, you can easily play the victim and claim the moral high ground. In other words, you can use the "hiding the hack" to *improve* your reputation. –  May 09 '18 at 02:30
  • @MaskedMan GDPR *does* apply in the US, if your system holds the details of any European citizens. It may be less enforceable, but it still applies. You're also failing to grasp how quickly these things can spiral out of control, and how easily. In any case, from the sound of it, we're going to have to just agree to disagree; I don't have time to continue arguing this. – Spudley May 09 '18 at 06:17
  • @Spudley GPDR is not really relevant to the point I was making. Anyway, let's say I am a hacker who now says I hacked Facebook in 2015. Where do you think the general public sympathy will lie? Also, another effective trick is if you have been hacked in Manner A, you just own up to being hacked in Manner B. That earns you some brownie points and nobody will bother investigating if something else is also going on. That is close to what I suspect Twitter and Github have done here. –  May 09 '18 at 06:24
  • You are also making this hack sound more dramatic than necessary. Public attention span is getting shorter by the day, and in a couple of months (or even weeks), nobody will care about this hack, and they will safely escape the radar. A huge song and dance was made when the Heartbleed exploit was discovered a few years ago. If someone comes up with some additional details today, how many people will care? It makes perfect business sense to not reveal anything more than necessary. Ruining your business today for fear of a future trouble, which is uncertain and unlikely, is just foolishness. –  May 09 '18 at 06:28

8 Answers8

118

They can't be sure. In fact, you can never be sure you haven't been hacked. But a thorough examination can make you conclude that it is more or less likely.

The Twitter statements only says that there is no indication of a hack. That doesn't exclude the possibility that they were hacked, and in urging their users to change their passwords they implicitly admits that.

As for GitHub, the wording is a bit more categorical. But I think forcing a password reset shows that they understand the risks involved.

Anders
  • 64,406
  • 24
  • 178
  • 215
  • 4
    "you can never be sure you haven't been hacked" - I cannot agree less. – Pedro Lobito May 05 '18 at 10:47
  • 34
    @PedroLobito Care to develop? – Anders May 05 '18 at 11:45
  • 1
    you can build your own system on an air-gaped machine, but most folks don't bother. – dandavis May 05 '18 at 13:04
  • 14
    Merely air-gaped won't guarantee no hacks at all. Unless he means that for literally every component, for instance an independent power source instead of mains. Maybe even an air-gap wouldn't be sufficient, perhaps a vacuum.... But there is always user error, e.g. I'm aware for instance of air-gapped systems being phished per se with USB drives seeded in the parking lot. There is also the consideration that an air-gapped machine wouldn't be useful as a system users need to authenticate into and actually use.... – ttbek May 05 '18 at 14:00
  • 14
    You can be convinced you haven't been hacked, and eventually you will be wrong. – Theraot May 06 '18 at 16:43
  • 6
    Anyone is capable of building a system *they* can't hack. A corollary to that is: anyone can build a system where they can't detect that a hack has occurred. They are doing the best they can to examine their own environments for evidence of hacking. That they found no such evidence only proves that: *they* could find no evidence. It's up to you to determine whether or not their statement satisfies your own requirements for security. – Christopher Schultz May 06 '18 at 20:32
  • 3
    @dandavis Airgapping a system isn't enough, as ttbek mentioned power sources can be an info leak, but I'll take this one step further: EM shielding is a bare-minimum requirement for air-gapping. If you use unshielded (or poorly shielded) cables for your monitor, they leak EM radiation that can be picked up and reconstructed (assuming a copy-protection protocol isn't being used) from over 20 feet away with pretty rudimentary SDR equipment. I'm not speaking a cyberpunk fantasy; I've done it myself. There are people who have put great thought into this: http://cm.bell-labs.com/who/ken/trust.html – CassOnMars May 06 '18 at 23:30
  • 2
    The Iranian centrifuges have been air-gapped. Stuxnet still killed them. – Christian May 07 '18 at 11:45
  • Well, for example, I built a texting device that uses ~100ma@5v, from a power bank or wall usb adapter, a nodeMCU, a 1.8" LCD driven over SPI with about a 1" cable, with a PS2 keyboard for input, a HWRNG to generate keys, and a snap together connector allowing identical units to share keys. I wrote the firmware (~1500 LOC), and there is no OS. It only knows how to do what it needs and doesn't allow remote updates, so I don't see how it's hackable or _can_ even leak, given the tiny EMI/RFI of such a device and the physical possession required to alter functionality. not for all, but fun for me. – dandavis May 07 '18 at 14:07
  • 2
    Food for thought: if there is no way to detect that you have bee hacked, have you been really hacked?? – user541686 May 07 '18 at 19:08
  • 2
    @ttbek [Mind the Gap: This Researcher Steals Data With Noise, Light, and Magnets](https://www.wired.com/story/air-gap-researcher-mordechai-guri/). People have exfiltrated data using room temperature, too, heating the room using the CPU. – mbomb007 May 07 '18 at 20:31
  • 1
    @drheart your link is not working – msb May 07 '18 at 23:57
20

One more thing to note is that in both cases, the leak was in a purely internal logging system. There is no indication that 3rd party users ever had access to this system. Internal logging systems are rarely exposed externally, and only consulted internally when a system needs troubleshooting. That's also probably the reason why this bug went unnoticed for months: singular log entries somewhere in what's probably a gigantic amount of other statements usually don't get noticed unless they happen to be right next to or in the middle of statements that are needed to debug other entries.

Twitter also only recently found out about the bug themselves, which means it's unlikely that people from outside the company were aware of this bug before Twitter was, let alone figured out and executed an attack to retrieve them.

Nzall
  • 7,313
  • 6
  • 29
  • 45
10

It's hard to prove a negative.

So how do you prove a positive? In this case: how do you prove an attack from the outside? Typically there are several systems in place to monitor different forms of attacks, breaches or access. These can be firewalls, intrusion detection systems, SIEMs and a variety of monitoring and logging systems. In today's networks each component either has some form of monitoring or is allowing monitoring through third party tools like Check_MK.

So each step of the way - from the border of the corporate network to the machine that held the valuable information itself - is in some shape or form monitored. These logs are, depending on the network and corporate policies, regularly analyzed. The analyzing systems can distinguish between expected and unexpected traffic or behaviour. Un/Expected behaviour is for instance file access.

Internal log files are typically considered confidential data, so file access is probably monitored as well. If someone that is not part of a certain user group tries to copy/access an internal log file, that would've probably been logged as unexpected or even forbidden behaviour. If a possible adversary was able to impersonate someone with the rights to access this file, it would've been logged as well, but as expected behaviour.

In theory it is possible that an attacker is able to overcome all security controls, exploit 0day vulnerabilities, leave no trace in every log on every component, the IDS, the SIEM and so on, copy the internal log file and smuggle it outside, but it is very unlikely.

My guess is, that after the log file was discovered, all these logs were thoroughly analyzed to try to prove if there was an attack from the outside. The analysts did not find any suspicious data and therefore concluded that with almost absolute certainty there was no attack from the outside. And this actually what you see in Twitter's press release (see Florin Coada's comment). Again, my guess: GitHub's press release had a more strict language to stop speculations if there was a hack. (Didn't really work out. ;)

Of course it's also possible that Twitter and GitHub have no such security controls in place, but I really hope not.

Tom K.
  • 7,913
  • 3
  • 30
  • 53
6

I assume they have access logs for just about anything happening on their servers, they can check whether someone accessed the file, when that was, what alias they logged in with, their source IP address, etc. If they can prove to themselves that all access was by legitimate employees they can confidently say they have not been hacked. Note that this does not actually guarantee them not being hacked, just that all evidence points in that direction.

kevin
  • 161
  • 1
1

The answer is very simple. Nowhere do they state they are sure. In fact, they implicitly tell you that there actually could have been a breach in two ways:

  1. They say that there is "no reason to believe there were" or "no evidence of" a breach. Lack of evidence could simply mean that the intruder(s) covered their tracks.
  2. They implore you to change your password, just to be on the safe side. This implies they cannot guarantee that no breach has occurred.
MahNas92
  • 111
  • 1
0

My interpretation of, especially of the more bold GitHub statement, is that they want to say that the fact that the passwords were put in the logfile is not the result of a hacking attack, but the result of a developer introducing debugging code by accident into production. This is relevant since an attacker might have introduced this functionality in order to collect passwords for grabbing them later.

There is no guarantee that they never have been hacked and won't ever be, but this incident is independent from hacking attempts.

johannes
  • 101
  • 1
0

Big companies like this should have lots of servers and therefore all outside access is routed through a bastion host (outside meaning not from within the actual server cage/room). The bastion might have say logging set up in a way since it relays all commands from the outside network into the operational servers. The commands if logged could easily be used to tell if someone opened a file with vim for example and that would solve the question of if a user had been hacked. SSH access is also logged so a known "good" IP/s can be filtered out for a user and any suspicious or odd entries could be examined by IT and if an entry has not been able to be explained then that would constitute a breach. Otherwise the server would be "safe" and there would not be as much of a need to worry regarding this matter.

0

What they are actually saying is that they are 100% sure that there was indeed a security breach. Accidentally. Probably. And by themselves. The rest is gloss.

They may view the individual differently, as an employee, but I don't. A good hacker works from the inside and gains trust. NSA? FSB?

They should never have plain text access to our passwords. That is not how password access works. The implications are that it is now up to you to change that password everywhere you ever use it. Assume the password is now known by everyone.

Sentinel
  • 188
  • 5
  • 4
    If having one password compromised means you have to go change it in a lot of places, that's your own fault. You shouldn't be doing that in the first place. – barbecue May 06 '18 at 06:11
  • @barbecue Also correct. This advise is for the mainstream. Everywhere can mean 0 to many places. – Sentinel May 06 '18 at 19:03