I don't think Google penalizes speed improvement. They penalize use of JavaScript for html rendering. Google bot probably cannot render all page content with JS, I'm sure in most cases it does not make all requests for static assets, it just makes GET to your url and parses html that is in response. If your initial response to first HTTP GET contains blank <body> that is filled dynamically by JavaScript Google bot will only see this blank body.
I'm pretty sure it would be completely inefficient for Google to measure page load times by making hundreds and hundreds of requsts to create full DOM representation.
Comments
I don't think Google penalizes speed improvement. They penalize use of JavaScript for html rendering. Google bot probably cannot render all page content with JS, I'm sure in most cases it does not make all requests for static assets, it just makes GET to your url and parses html that is in response. If your initial response to first HTTP GET contains blank <body> that is filled dynamically by JavaScript Google bot will only see this blank body.
I'm pretty sure it would be completely inefficient for Google to measure page load times by making hundreds and hundreds of requsts to create full DOM representation.