The problem you are describing has much less to do with mongoose, mongodb and node and is more a manifestation of a problem commonly referred to as a "cache stampede" or "dog pile".
As the name implies, a cache stampede happens when a whole bunch of things try to refresh the cache at once. In your case, this happen when the cache expires or during the initial load of data into the cache. Suddenly, lots of requests come in and apply lots of read load onto your database which 1) causes the cache to be slow to be refreshed and 2) causes even more requests to stack up waiting for the cache. This basically leads to the behavior you saw, where things just crash
This Wikipedia page describes the problem fairly clearly and how one could solve it using a separate process or locks. Since node doesn't have locks or threads, thats probably not a solution. Also, while separate process would work, it has a lot more complex.
One technique that I have used in the past is to use two expiring cache keys, one key is used only indicate when the cache should be refreshed, while the other holds the actual data.
To illustrate, let's assume I have an object foo
that I want to cache and have expire every hour. I can create another key foo_refresh
that I expire 1 minute before the foo
key.
When the foo_refresh
key expires, one worker/request immediately replaces the foo_refresh
key and ignores the cache and instead pulls the data from the database, refreshing the foo
key when finished (also resetting the expiry time). Using a mechanism like this, we obtain a sort of "lock" on refreshing the cache, meaning no more than one worker will ever be doing the expensive read.
Assuming that the cache refresh takes less than 1 minute, the foo
object never expires, instead, it gets refreshed via the expiration of the foo_refresh
key.
Hopefully that helps!