There is a race here, where more then one thread could be trying to
remove entries.  This could result in too many entries being removed
from the cache.  This is considered OK as the size of the cache
should be rather large, and we would rather have performance over
keeping the cache size exactly at the maximum.