The difference is in how HTTP/1 and HTTP/2 set up sockets for networking.
In HTTP/1, the browser makes a request for a single asset and then waits for that asset to download before making the next request. This is very slow, so to get around that, browsers started multiplexing their requests and making 6-8 requests at a time, splitting the bandwidth between those 6-8 requests.
In HTTP/2, the browser opens a single socket, makes all of the requests and then waits for all of the assets to be streamed at the highest available bandwidth.
The reason that you're supposed to see a high TTFB is because in HTTP/1, each request is a separate socket, but in HTTP/2, all requests download through the single stream. You may have made the requests, but you have to wait for the server to respond with all of the data that you've requested.
In theory, HTTP/2 is supposed to be faster because it's more efficient - you don't have the overhead of opening a new socket for each request. You're streaming all of the data down a single data pipe and you can prioritize requests based on how the page should load.
In practice, this depends upon the browser making prioritized requests properly and the network allowing full bandwidth for the single socket stream and not limiting requests artificially.
Additionally, since you've stated that you're doing unencrypted HTTP/1 and encrypted HTTP/2, encryption will cause additional overhead and performance loss. You might try doing encrypted HTTP/1 for a better comparison.
One hypothesis is that IIS10 and Windows Server 2016 are not finished yet and there are problems with HTTP/2 performance but I could not find anyone else complaining about it – Daniel Wardin – 2015-06-23T10:04:01.977
I am using unencrypted HTTP/1.1 against encrypted HTTP/2. Do you think that makes a difference, should unencrypted HTTP/1.1 outperform encrypted HTTP/2? – Daniel Wardin – 2015-06-23T14:50:08.737