First of all, any time you install any software of any variety, you're trusting the author of that software. There's no way around that. You're trusting that the author isn't malicious, that he isn't sloppy, that he knows what he's doing, that his implementation is correct, and a dozen other points as well.
Likewise you're trusting the site owner in a lot of the same ways. Beyond that, it's just math. You're trusting the math.
Google Authenticator is arguably as trustworthy as is possible as the company is well-established and has built a reputation on its tools and services being bulletproof. There is literally no incentive for them to cheat, and a strong incentive (and ample resources) to create a well-designed solution.
But they make some security decisions that a lot of people find inconvenient (like disallowing export of keys). So that could be reason enough to use an alternative.
I would hasten to point out that if at all possible, use the U2F token. I personally use it and am very satisfied with the experience. While TOTP/HOTP is really good, U2F takes it a step further providing absolute protection against phishing -- something no other product I'm aware of can claim. U2F was designed specifically for this purpose, and it really does make a difference. Just 2 weeks ago I encountered a report from a victim who was using TOTP but was tricked into typing in his authenticator token in a phishing attack, and lost control of his account. U2F would have protected him.