PHP isn’t a compiled language, you upload your script to the server and that’s what it uses, no compiling required. This means every time it wants to run your code though, it has to compile it down to the required bytecode and run it. With WordPress that can be a lot of code to compile. One way of alleviating this is using something called an OpCache which stores the compiled bytecode, usually in memory, ready for the next time it is needed.
This is what we were seeing with the quickest requests, the page request was running from the OpCache and the slower ones were being compiled, stored and run and the OpCache actually adds some overhead for code that isn’t in the cache as well.
As this was only a staging website, the only traffic at all to it was us testing it, no search engines crawling it, no bots looking at it for vulnerabilities etc. and this meant that the idle timeouts on the parent PHP process were expiring and removing the parent process and the OpCache compiled code with it. We saw this across the cPanel and Plesk servers.
Problem #1 solved – increase the timeouts, make the OpCache live longer in memory and look at enabling file backed cache for first startup of the OpCache, now we’re down to all loads being the faster of the two times.