GitLab account hacked and repo wiped

Question

I was working on a project, a private repo, and suddenly all the commits disappeared and were replaced with a single text file saying

To recover your lost code and avoid leaking it: Send us 0.1 Bitcoin (BTC) to our Bitcoin address 1ES14c7qLb5CYhLMUekctxLgc1FV2Ti9DA and contact us by Email at admin@gitsbackup.com with your Git login and a Proof of Payment. If you are unsure if we have your data, contact us and we will send you a proof. Your code is downloaded and backed up on our servers. If we dont receive your payment in the next 10 Days, we will make your code public or use them otherwise.

At the time of this happening, Google search didn't show up anything, but in an hour or so this started coming up.

I am using SourceTree (always up-to-date) but somehow I doubt that SourceTree is the issue, or that my system (Windows 10) was compromised. I'm not saying it's not that, it's just that I doubt it.

This happened only to one of my repositories (all of them private) and all the others were left untouched. I changed my password, enabled 2 factor authentication, removed one access token that I wasn't using for years and wrote an email to GitLab in the hopes that they could tell me something about where/who the attacker got in.

My password was a weak one that could've been relatively easily cracked via brute-force (it's not a common one but starts with "a" and has only a-z characters in it) and it could be that they just automatically checked if they can access the account and then ran some git commands. It is also possible that my email address and that particular password are on a list of leaked accounts. One might argue that if this is how they got in, they would've simply changed the account credentials but searching the Internet revealed that in these cases GitLab/GitHub will simply restore the credentials for you, and so I assume this is why they didn't do it this way.

Could've also been that old access token, I can't remember what and where I used it for in the past - most likely generated for use on a computer I previously owned, so I doubt that that was the issue.

There are also 4 developers working on it, all having full access to the repository, so their accounts being compromised is also a possibility.

I've scanned my computer with BitDefender and couldn't find anything but I am not doing shady things on the internet so I don't think that me being infected with a malware/trojan is what caused this.

I am waiting for an answer from GitLab and maybe they can shed some light on this. I have the code base on my local Git, so that is not an issue, but I am not pushing the code back to the repository just yet. Also, just in case the code gets published somewhere, I will change any passwords that are to be found in the source (databases, IMAP accounts)

UPDATE

I found out that the code isn't gone. I tried accessing a commit's hash and it worked. So the code is there but there's something wrong with the HEAD. My knowledge on this is very limited but

git reflog

shows all my commits.

What this means to me is that the attackers most likely didn't clone the repositories (would be a logistical nightmare to do this for all the victims, anyway) and that the chances for them going over the source code looking for sensitive data, or of making the code public are low. It also means to me that is not a targeted attack but a random, bulk attack, carried out by a script. I really hope this is the case for our own sake!

UPDATE 2

So, if you do

git checkout origin/master

you will see the attacker's commit

git checkout master

you will see all your files

git checkout origin/master
git reflog # take the SHA of the last commit of yours
git reset [SHA]

will fix your origin/master...but

git status

now will say

HEAD detached from origin/master

still searching for a fix on this

UPDATE 3

If you have the files locally, running

git push origin HEAD:master --force

will fix everything. See Peter's comment

So, the question is what commands will get my repository back to the previously working state assuming you don't have the repo locally, as for how the attacked got in, I am hoping that the answer from GitLab (if any) will help us more.

There is a discussion going on here

The attack targets GitHub, BitBucket and GitLab accounts. Here's the magnitude on GitHub's public repos

Answer 1

You can use git reflog in a clone and checkout the last commit before this happened.

It happened because .git/config on your webserver (in the directory of the cloned repo) includes the remote URLs and people added username:password in it which should never be the case - people should use SSH, deploy keys or authenticate on each pull. Never store your credentials in a config file. Use the credential helper(s).

Source: https://www.reddit.com/r/git/comments/bk1eco/comment/emg3cxg

hello, it is me , the guy with your backups ..

i will reveal your sins

Here is an article from 2015, its more detailed, https://en.internetwache.org/dont-publicly-expose-git-or-how-we-downloaded-your-websites-sourcecode-an-analysis-of-alexas-1m-28-07-2015/

Article by Internetwache about this: https://en.internetwache.org/dont-publicly-expose-git-or-how-we-downloaded-your-websites-sourcecode-an-analysis-of-alexas-1m-28-07-2015/

To prevent this either block access to directories starting with a dot, see https://github.com/h5bp/html5-boilerplate/blob/master/dist/.htaccess#L528-L551

# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

# Block access to all hidden files and directories with the exception of
# the visible content from within the `/.well-known/` hidden directory.
#
# These types of files usually contain user preferences or the preserved
# state of an utility, and can include rather private places like, for
# example, the `.git` or `.svn` directories.
#
# The `/.well-known/` directory represents the standard (RFC 5785) path
# prefix for "well-known locations" (e.g.: `/.well-known/manifest.json`,
# `/.well-known/keybase.txt`), and therefore, access to its visible
# content should not be blocked.
#
# https://www.mnot.net/blog/2010/04/07/well-known
# https://tools.ietf.org/html/rfc5785

<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteCond %{REQUEST_URI} "!(^|/)\.well-known/([^./]+./?)+$" [NC]
    RewriteCond %{SCRIPT_FILENAME} -d [OR]
    RewriteCond %{SCRIPT_FILENAME} -f
    RewriteRule "(^|/)\." - [F]
</IfModule>

Or separate the .git directory and the data using --separate-git-dir.

--separate-git-dir=<git dir>
Instead of initializing the repository as a directory to either $GIT_DIR or ./.git/, create a text file there containing the path to the actual repository. This file acts as filesystem-agnostic Git symbolic link to the repository.

If this is reinitialization, the repository will be moved to the specified path.

But the best is to rm -rf .git after a deployment - which should just copy a build artefact to the destination using rsync.

https://git-scm.com/docs/git-init#Documentation/git-init.txt---separate-git-dirltgitdirgt

--separate-git-dir=<git dir>
Instead of placing the cloned repository where it is supposed to be, place the cloned repository at the specified directory, then make a filesystem-agnostic Git symbolic link to there. The result is Git repository can be separated from working tree.

https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---separate-git-dirltgitdirgt

https://stackoverflow.com/a/8603156/753676

Information about deploy keys and the credential helpers:

https://developer.github.com/v3/guides/managing-deploy-keys/

Deploy keys are read-only by default, but you can give them write access when adding them to a repository.

https://gist.github.com/zhujunsan/a0becf82ade50ed06115

https://help.github.com/en/articles/caching-your-github-password-in-git

Use git push -u origin master -f && git push --tags -f from your local clone to push all references for master, tags and so on to the remote and then enable 2FA in your account.

If more branches are affected use git push -u --all -f

Also please enable 2FA to decrease the possibility of such attacks.

Please do not forget to change all compromised logins / passwords and revoke any unknown sessions.

Answer 2

I doubt that the hackers pushed a "delete all" commit, or else you could simply revert the last commit. Rather, they force-pushed a different commit with the note to the HEAD of the master branch, making it look like your entire commit history is gone.

As others have pointed out, you can easily use a local repo to force push the correct code to the server. Due to the distributed nature of Git, this always works whether or not the server was wiped since every local repo has a complete clone of the server, including both commits and code. Of course, you should make sure the server has been secured first before attempting recovery efforts. :-)

If you don't have a local repo that includes the most recent commit, the commit history (and all associated files) will still exist on the server for a while. However, the server will eventually run git gc, which will clean up those unreachable commits. As of 2013, GitHub said they will run git gc at most once per day but it can also be triggered manually, while BitBucket will run it as needed, or perhaps after each push. GitLab runs it after 200 pushes by default, or it can be triggered manually.

However, even if all of the commits and files are still on the server, you would need to find the hash of the commit so you can restore it. Without a local repo with a reflog, it's hard to find the correct commit to restore. Some ideas that you could try:

• If you are using GitHub, check the Events API for your repo at https://api.github.com/repos/:user/:repo/events to see if you can locate the proper commit. I don't know if other services offer a similar feature.
• Pull requests are typically kept forever, so you should be able to look at the most recent pull request merged into the master branch. Just make sure to pick the hash of the merge commit, not the hash of the branch. (GitHub has a green check mark next to the merge commit hash, GitLab shows "merged into master with", not sure about BitBucket).
• If you have a build server, see what the most recent build of the master branch was (perhaps in the build log?)
• You may want to check forks of your repo as well. GitHub allows you to see them either in Forks or Network views.

Once you find the correct hash for master, you can restore your server using the following commands (assuming you have a Git remote called 'origin').

git fetch origin <hash>
git checkout master
git reset --hard <hash>
git push --force origin master:master

Note that you should never use git push --force unless you intend to overwrite someone's work.

Answer 3

If more branches are affected, you may need to checkout all branches first with the following command before performing git push -u --all -f

for branch in `git branch -a | grep remotes | grep -v HEAD | grep -v master `; do
   git branch --track ${branch#remotes/origin/} $branch
done

https://gist.github.com/octasimo/66f3cc230725d1cf1421

Answer 4

I guess you already know the more obvious, but nevertheless:

1. Going forward, set up SSH for communicating with GitLab (and any other remote that supports it, for that matter) instead of username+password, as @Daniel Ruf has advised.

2. Configure a very strong password (on the order of 16+ randomly-generated characters) for your GitLab account, and use a password manager to manage it.

3. Make sure your computer is not compromised. I would go one step further and change passwords for all of my online accounts just in case.

Now to address another pressing matter:

What this means to me is that the attackers most likely didn't clone the repositories (would be a logistical nightmare to do this for all the victims, anyway) _{(assumption #1)}
(...)
and that the chances for them going over the source code looking for sensitive data, or of making the code public are low (...) _{(assumption #2)}

It also means to me that is not a targeted attack but a random, bulk attack, carried out by a script (...) _{(assumption #3)}

Assumptions #1 and #3 may or may not be true (I personally don't think it's a logistical nightmare at all to clone repos when your plan is to deface them for ransom - the attacker may well have a server dedicated for that task, configured through a VPN or the sort. And it might be that you were targeted). But they're not very crucial.

However, assumption #2 is one you can't afford to make right now.

If the code or repo history contained private information or any sort of trade secret, begin taking contingency steps immediately.

To quote part of their message:

If we dont receive your payment in the next 10 Days, we will make your code public or use them otherwise.

I'm afraid it's safe for you to assume they will do that whether you pay the ransom or not. Specially the "use them otherwise" bit.