The other side of the moon

We frequently see reports of website usage going up as load time goes down, or vice-versa. It seems logical. Users use a site much more if it's fast, and less if it's slow. However, consider the converse too. Is it possible that a site merely appears to be faster because users are using it more, and therefore have more of it cached? I've seen sites where the server-side cache-hit ratio is much higher when usage is high resulting in lower latency. At this point I haven't seen any data that can convince me one way or the other. Do you?