I don't think we have a generic answer specific to this question but a lot of "my server is slow" questions get closed as a duplicate of this question. The reason is that capacity planning is nearly always about performance. When you run out of capacity, the normal effect is that the website starts going slow.
Your case is not running out of capacity, however the question is far from unique on ServerFault.
The most likely cause is a cache, or possibly lots of different caches, all of which have expired when you make that first request. You have the DNS cache, filesystem cache, PHP opcode cache, application server pools, the database cache, the query cache, API response caches and probably many more. Since you're running on a VPS, some of your memory may have been swapped out to disk while you were idle, giving the more active users of the physical hardware more RAM to play with and this takes time to swap back in.
Some applications and CMS manage their own cron-like system where page requests from normal users can trigger off scheduled jobs. After a long period with no requests, the first request is guaranteed to trigger or these scheduled jobs, possibly causing that page load to be slow.
What you need to do to find the problem is performance profiling. I often fall back on strace because it's always available and I know it well, but XDebug (for PHP) or dtrace are also good options. Profile the first request and compare it to the profile of a later request. The differences between the two profiles will be where the time is being spent.
There are also a lot of guesses in the answers to the questions I linked above. Some of them were correct for the askers of those questions and may be for you too.
