Assume the memory is instant. We still need to communicate with it across physical distance. How far away the memories are in space is way more critical than the speed of any one element in isolation.
Why are we constrained to such a relatively small amount of L1 cache? What would stop us from extending this arbitrarily?
HP was the only CPU vendor I recall that went for massive L1 with their PA-RISC chips, some with 1-2MB of L1. I'm going to say a large L1 is ~1MB for this comment.
There are power, speed, and complexity trade-offs in cache design. Here were a few of them:
Direct-mapping is the simplest approach, and means a given address can only exist in one location, but a problem occurs when two addresses map to the same cache line - one is evicted even if there's plenty of space in the cache elsewhere.
What if we built an associative cache, where every line had an address indicator? Then we can fully use the cache space. But it's far more complicated to search: a miss requires checking every cacheline. If it's fast, so does a hit.
Many systems today use a mix. Smaller caches are often direct mapped. Larger caches tend to use a combination of 2-8 direct-mapped caches where an address can be searched in at the same time, or within a few cycles of each other.
Another problem is evictions becoming a future cache miss. With only a large L1, a cacheline was either in fast L1 or in DRAM. There's often a write buffer or victim cache between them to try to hide dirty eviction latency, but a subsequent fetch will pay the DRAM access cost. As we scale the cache size up, L1 access speed becomes challenging and eventually it's more effective to limit the L1, and build a even larger, slower L2, and then we get the advantage that L2 is still faster than DRAM and we can prefetch into it.
This cache hierarchy tends to fit typical access patterns better as well - for many workloads most accesses will tend to be clumped together. For streaming workloads like video processing that won't fit in a L1 cache anyway, the algorithms are usually aware of row/column/striding impacts on cache utilization already.
PA-RISC clock speeds were relatively low for the time of their introduction. For example, the PA-8900 was launched in May 2005. The low end Athlon 64 of May 2005 was clocked at 2.2GHz (and eventually hit 3.2GHz) versus the PA-RISC top clock of 1.1GHz. You can always build a larger L1 cache in the same process tech if you're willing to sacrifice clock speed sufficiently.
Access time is a a function of areal density, not unlike hard drives. Plus access logic increases at least logarithmic to the number of lines in the cache, so you’re probably looking at in the neighborhood of 50% slower for every doubling of L1 cache size.
Now as to why cache sizes didn’t increase much once clock speeds stagnated and feature size continued to decline, I couldn’t say. But L3 caches didn’t used to exist, and L2 has gotten bigger.
How far away the memories are in space is way more critical than the speed of any one element in isolation.
Correct me if I'm wrong, but I don't think this is true. If you put instant memory a whole meter away it'll still only have a round trip of 6.6ns at the speed of light, which is approximately the latency of L2. Given how close L2 is, I don't think distance is a large factor of its latency.
Comments
Assume the memory is instant. We still need to communicate with it across physical distance. How far away the memories are in space is way more critical than the speed of any one element in isolation.
Why are we constrained to such a relatively small amount of L1 cache? What would stop us from extending this arbitrarily?
HP was the only CPU vendor I recall that went for massive L1 with their PA-RISC chips, some with 1-2MB of L1. I'm going to say a large L1 is ~1MB for this comment.
There are power, speed, and complexity trade-offs in cache design. Here were a few of them:
Direct-mapping is the simplest approach, and means a given address can only exist in one location, but a problem occurs when two addresses map to the same cache line - one is evicted even if there's plenty of space in the cache elsewhere.
What if we built an associative cache, where every line had an address indicator? Then we can fully use the cache space. But it's far more complicated to search: a miss requires checking every cacheline. If it's fast, so does a hit.
Many systems today use a mix. Smaller caches are often direct mapped. Larger caches tend to use a combination of 2-8 direct-mapped caches where an address can be searched in at the same time, or within a few cycles of each other.
Another problem is evictions becoming a future cache miss. With only a large L1, a cacheline was either in fast L1 or in DRAM. There's often a write buffer or victim cache between them to try to hide dirty eviction latency, but a subsequent fetch will pay the DRAM access cost. As we scale the cache size up, L1 access speed becomes challenging and eventually it's more effective to limit the L1, and build a even larger, slower L2, and then we get the advantage that L2 is still faster than DRAM and we can prefetch into it.
This cache hierarchy tends to fit typical access patterns better as well - for many workloads most accesses will tend to be clumped together. For streaming workloads like video processing that won't fit in a L1 cache anyway, the algorithms are usually aware of row/column/striding impacts on cache utilization already.
There's probably more to consider, like SMP.
PA-RISC clock speeds were relatively low for the time of their introduction. For example, the PA-8900 was launched in May 2005. The low end Athlon 64 of May 2005 was clocked at 2.2GHz (and eventually hit 3.2GHz) versus the PA-RISC top clock of 1.1GHz. You can always build a larger L1 cache in the same process tech if you're willing to sacrifice clock speed sufficiently.
Access time is a a function of areal density, not unlike hard drives. Plus access logic increases at least logarithmic to the number of lines in the cache, so you’re probably looking at in the neighborhood of 50% slower for every doubling of L1 cache size.
Now as to why cache sizes didn’t increase much once clock speeds stagnated and feature size continued to decline, I couldn’t say. But L3 caches didn’t used to exist, and L2 has gotten bigger.
Correct me if I'm wrong, but I don't think this is true. If you put instant memory a whole meter away it'll still only have a round trip of 6.6ns at the speed of light, which is approximately the latency of L2. Given how close L2 is, I don't think distance is a large factor of its latency.
The problem is resistance and parasitic capacitance scales with wire length, e^(-t/RC) is going to limit your max frequency.
L1 cache has fast access because it is small. If you make it larger, you necessarily make it slower to access.