The only thing that consistently makes pgbench slower when you increase shared_buffers is a debugging option called assertion checking. If you do this in psql:
show debug_assertions;
And it's on, your build has this problem, and the results you're seeing are expected. You need a new install that's not enabled for assertion debugging to make that go away.
Otherwise, if you didn't do all this more than once, I wouldn't be so sure the cause here was the shared_buffers change. As one example, if autovacuum happens to run at the point your large scale database is running, it will slow test results. A checkpoint starting as the test begins will too. Turn off autovacuum to eliminate it from the test, and turn on log_checkpoints to see if it's interfering.
A third possibility, and this I'm almost sure you're suffering from to some degree, is that things moving around on disk can cause a 50% slowdown in results. Disks are about twice as fast on their early parts as their later ones, and as you run pgbench repeatedly the data will move to the slower parts as you go. I end up rebuilding the whole filesystem in order to get repeatable results, only way to be sure you're starting at the same point on the drive. This doesn't impact the results at the smaller scales because those all fit in RAM.
When I see a performance change after touching a configuration parameter, I always try the original configuration again to make sure that was the cause. I'm often surprised to discover that the test just gets slower each time you run it, with this disk location speed difference being one source for that issue.