It is my experience that it only gets slow when the harddrive(s) goes to sleep.
My powersaving options let my harddrives go to sleep if they aren't accessed for a few hours.
The delay for the toolbar pop up appears to be the 15-20 seconds it takes for the harddrive to wind-up again. If you catch it around the 20 second period, you can actually see it re-reading each individual thumbnail from the drive, as if it is scanning individual executables for their icons. I don't see any caching occurring except what it pulls into active memory.
I haven't tested it yet, but it may actually need to wake up each individual drive that any menu shortcuts are pointing to. Once I can read one toolbar menu, the others only have a 1-2 second delay. After that, they all respond in less than a second.
Seems like this would be a major oversight for any OS GUI, but then again Windows has proven they don't design things for continued usage. They still don't have a simple icon for "Upgrade your harddrive".