A blog post I read had a step-by-step guide for bruteforcing a base32 encoded shared secret. The technique used only needed two OTP codes + time stamps, but I don't remember the details or if the codes had to be for consecutive time periods. I can't seem to find the post now, and I don't have it bookmarked - if anyone knows the post I'm talking about, please share!
Summarizing from memory, you need to enumerate (via brute force) many possible shared secrets that could have been used to generate each code. Comparing the two lists, you will see only one shared secret that could have generated both codes.
It's not possible to brute force a 128-bit secret, but (again, from memory) ambiguity in the standard and encoding quirks of base32 values means that most secrets are actually much shorter - enough to brute force. A base32-encoded secret with 16 characters, for example, is only 80 bits when decoded.
There may have been other optimizations to make the bruteforcing faster. I think I recall something about working backwards from the OTP codes to find portions of the secret's hash, and then using a Hashcat mask to quickly discard any candidate hash not matching the pattern. Or maybe rainbow tables were used. A lot of the finer details escape me, so I'd really like that link if anyone has it...