What is the purpose of the "secret" that is displayed when you take the picture of the QR code to set up 2 Factor Authentication on a non-Google site?
Why would you need it?
If you save it, doesn't that defeat the whole purpose of 2 Factor Auth because anyone with the secret can get a login code?
Google Authenticator supports the TOTP and HOTP algorithms.
In both algorithms, a secret needs to be shared between the server and the client to successfully generate the one time passwords. I suggest you read the respective RFCs of the algorithms (or my answer here) to understand it better.
Yes, if the attacker manages to get hold of the shared secret the algorithm is defeated. However, this isn't very practical. The secret is only displayed in the form of a QR-code once when setting up the algorithm. Presumably you will be sitting in front of the screen when this happens and will take notice if someone suspicious walks up and starts scanning the QR code with his phone. There are plenty of physical measures you can employ if this is the case.
Once the secret is shared, it is stored on the server and the client (in this case your phone). At least in the case of Android, the secret is stored unencrypted in a SQLi database file. However, this file can only be accessed by the Authethenticator app unless your device is rooted so it isn't easy to get your hands on it. It should be just as difficult if not harder on an iOS device. You have much bigger worries if an attacker manages to compromise the server.
So yes, the algorithm is defeated if the secret is leaked. But the secret leaking is not an easy thing to happen. In any case, it's TWO-factor authentication. You still have the other factor to protect you.
The secret (or the seed) is the value upon which the One-time Password OTP. It's essential for you (or rather, your password generator, and in your case your Google Authenticator) to know that secret value. It's essential that both you and server know that value.
The QR code you're scanning is the secret value, just encoded in a way that your camera understands it to make it easier for you to input it in your phone. If your phone doesn't have a camera, or isn't able to scan QR codes, then you must input the secret value manually. That's why they show you the value. They show it only to you, and you should make sure that nobody is around when they show it to you.
Quibble: you are not really setting up two-factor authentication here, you are setting one authentication factor, of the “what you have” type. This is meant to be combined with a password, which is a “what you know” authentication factor, giving you two-factor authentication.
“What you have” is intended to be a mobile device. The way a user is authenticated as having that mobile device is to request that the user enter a code displayed by that device. In order for this code to authenticate the device, it must depend on some secret value that is only present on the device, the authentication server (it's a shared secret) and nowhere else.
The secret value is generated on the server and needs to be transmitted to the mobile device. The QR code contains that secret value.
The secret value is only displayed when you set up this device-based authentication. You are supposed to do this in a secure environment, e.g. at home, where nobody can shoulder-surf and nab the secret. Anyone with the secret can get a login code, but only the legitimate user is supposed to have the secret.
TOTP authentication (like that used by Google Authenticator) works by converting the timestamp to a 6-digit number using the pre-shared secret that only exists on your phone and the the authentication server. To express this simply, f(timestamp, secret) = 6-digit code. If the secret is known to a 3rd party, the yes, the purpose of 2-factor authentication is defeated, but obtaining this secret from a portable phone is non-trivial.
