Today Eric upgraded our development servers at work to the new Rails 0.14.3, also known as Release Candidate 4 for the mythical 1.0 release. As expected, everything promptly broke. All of our sites stopped working. Eric worked some magic and brought the sites back up. He claimed the he didn't do anything more than restart apache, but I have a hard time believing that because I had restarted the server, twice, only moments before.
As I scanned the query logs I noticed that our ActiveRecord caching scheme was suddenly broken. This left me in a bit of a panic. We have developed a clever little caching layer, using the amazing memcached service along with an intra-page cache, that has successfully reduced the number of queries to the database by somewhere between 1/3 to 1/2. Performance would really suffer if we couldn't get the caching to work again.
The caching layer works by decorating ActiveRecord's find, update, destroy, and reload methods. The local cache and memcache are checked for objects before going to the database., while the caches are cleared after an update or destroy and before a reload. Optimistic locking and a reasonable TTL provide a safety net in case anything should go wrong.
I tried going through the Rails source code and change lists to uncover what might have caused the caches to be ignored, but this is a pretty grungy and hard to decipher part of the framework and I made little progress on that front. After some more experimentation, I realized that has_one and belongs_to relationship lookups were no longer calling find but instead were going directly to find_by_sql.
The solution? Another wrapper around find_by_sql that checks for queries of the form "SELECT * FROM table WHERE (table.id = ###) LIMIT 1" and call the cache-enabled find method instead of going to the database. The trick, however, is that if the objects are not in cache then find will call back into find_by_sql and we are then stuck in a loop. A class variable guards against by skipping the sql check in find_by_sql and going directly to the database if it is called as part of the processing of a simple find request.
I fear this caching layer is getting more fragile and more complicated with every revision of Rails. This latest patch is far from polished. It isn't thread safe--that's not a concern for us--and now that I think about it, isn't exception safe either. But at least the caches are working again.
