Memory Access Performance and Cache

Here's an inner loop that does something funny--it jumps around (using "loc") inside an array called "buf", but only within the bounds established by "mask".  Like any loop, each iteration takes some amount of time; but what's suprising is that there's a very strong dependence of the speed on the value of "mask", which establishes the size of the array we're jumping around in.

 for (i=0;i<max;i++) { /* jump around in buffer, incrementing as we go */
	sum+=buf[loc&mask]++;
	loc+=del;
	del=del+sum;
 }
(Try this in NetRun now!)

Here's the performance of this loop, in nanoseconds per iteration, as a function of the array size (as determined by "mask").

I claim each performance plateau corresponds to a chunk of hardware.  Note that there are three jumps in the timings:

• "L1" cache is the fastest (like 1-5ns) but tiny (100KB or less).
• "L2" cache is 2x slower but much bigger (up to a few megs).
• RAM is painfully slow but huge (gigs).  

Memory Speed

In general, memory accesses have performance that's:

• Good *if* you're accessing a small amount of memory--small enough to stay in cache.  The pieces of memory you use often automatically get copied into fast cache memory.  For example, the top of the stack, recent stuff from the heap, and commonly used globals are almost always in cache, and hence really fast.  Machines only have a couple of megs of cache nowdays.
• Good *if* you're accessing memory sequentially--if you access a[i], you then access a[i+1].  This "streaming" access is fast because memory chips don't have to change rows. 
• Terrible *if* you're accessing a big buffer (too big to fit in cache) in a nonsequential way.  Cached accesses are on the order of 1ns.  Streaming access is also a few ns.  Random access that isn't in the cache can be as much as 100ns!

So if you're getting bad performance, you can either:

• Make things smaller, or re-use them more often, to fit in the cache.
• Make your accesses more regular, to get streaming access.

These are actually two aspects of "locality", the similarity between the values you access.  The cache lets you reuse stuff you've used recently in time (temporal locality); streaming access is about touching stuff nearby in space (spatial locality).

There are lots of different ways to improve locality:

• Change your algorithm to improve locality.  Think about how your loops access memory.  Often it's trivial to reorganize your loops so you touch the same data 5 times before hitting the next piece of data (temporal locality); or so you access arrays sequentially (spatial locality).  Sometimes an algorithm that's theoretically more efficient will be slower than another algorithm with more instructions but better locality!
• Change your data structure to improve locality.  For example, you might change linked lists into arrays.  The trouble with linked lists is that links are allocated at random separate locations; but an array will be contiguous in memory (spatial locality).
• Change your input data to improve locality.  For example, processing data in small pieces (1KB) will usually have better temporal locality than huge chunks (1GB).  Of course, if your pieces are too small, sometimes your program will slow down because of the per-piece overhead!