You can verify if GC is the issue by using Performance Monitor and the '.NET Memory\% Time in GC' performance counter. If you only have one .NET process on the server, you can just use the _total instance. Otherwise you'll have to find the instance that has a matching process ID and watch that one (though be aware that the instance name for you application can change on the fly if any apps start up or shut down).
If spikes in this counter correspond to the CPU spikes, garbage collection is your issue--you will need to look for leaks, allocate fewer objects, keep things small enough to keep them out of the LOH, keep them around less time, reuse them, and/or eliminate destructors. Each of these things will reduce time spent locked up in GC. Ironically, too much caching can make your site inconsistently unresponsive, as cached items eventually end up in heap 2, and request processing pauses while the GC sweeps through every item in heap 2. As memory pressure increases, the frequency of these lockouts increases until eventually your requests get completely starved out.