Recently in Ruby on Rails Category

I was up until 2:00 am last night working on the database. I wanted to wait until the regular 1:00 am full backup completed so there would be something to go back to if I really screwed something up badly.

Using my training in the scientific method that I gained as a physics student at one of the top research universities in the country, I decided that my strategy would be to blindly poke at things and see if anything happened. And I would have to poke hard. If a little is a good thing then surely a lot must be even better. I made some adjustments to three or four settings and restarted the database. Things seemed to be working better than ever. Of course, it's hard to judge because the site always performs well under the light night traffic.

I went to sleep confident that I had accomplished something good. The sound of the pager at 4 am suggested otherwise. With a strange calmness I was able to form a bit of a theory as to why the database was freaking out and back out those changes. The clues were there earlier but at that time I wasn't aware of what they meant.

I was perhaps a little more conservative than necessary, but I waas able to leave in a change that I think will actually make the biggest difference. And with that, I've consumed pretty much all of the collective knowledge I could glean off the internet for tuning MySQL databases.

The only thing left that I can think would make a difference is to sacrifice some up to the second synchronization gurantees for reduced disk access. Dr. Brain may disagree, but I'm not convinced that an fsync() after each and every transaction is really necessary. But even this will become less of an issue when our new database server with its faster striped disks comes online.

With things working once again, I headed back to bed, only to wake up too soon to the ringing of the alarm clock. I pressed snooze for as long as I could, but I needed to be awake and ready to meet Andrej for breakfast this morning. And so I'm not sure whether I will have enough energy to carry me through the new year or whether I will just crash sometime later this evening.

A little more digging into the memcache code revealed some interesting details. It looks like the root of the problem is due to socket options in the server. To get the maximum network performance the server tries to disable the default Nagle packet buffering algorithm. On systems that support the TCP_NOPUSH socket option, the server will bracket network writes within a no push section and then let the operating system send back the result as soon as all the data has been written. If the system doesn't support TCP_NOPUSH, memcache will instead fall back to TCP_NODELAY.

It looks like FreeBSD supports the TCP_NOPUSH option but it doesn't seem to work exactly the way you would want it to. Reading up on the newsgroups, it looks like there have been some proposed kernel patches to bring FreeBSD's handling more in line with what is found on Linux. I didn't really want to mess with the kernel, so I simply recompiled the memcache server to use TCP_NODELAY.

Initial testing looks good. The 100 millisecond response is now processed by the Ruby client in just over one millisecond. This is definitely much better than 100 milliseconds. I'll let the new server run on our staging machines for a while before trying to push it out to the live site.

My natural distrust of the Ruby programming language might have caused me to miss something important. On 43 Things we rely heavily on memcache to offload database reads. After a bit of work simplifying and tuning the networking code, we were able to get very good response times for data lookups. Occassionally, however, the timings drifted from their usual submillisecond range to close to 100 milliseconds. I just assumed that either Ruby networking code occassionally freaked out or that we were caching complicated data structures that could take a while to parse.

Recently I've been working on some alternative algorithms to help solve some of the performance problems we've been seeing on a few of the more intensive pages. These solutions make extensive use of caching. While a single 100 ms lookup on a page might slip through without much notice, a handful of them will easily kill page serve times.

I dug into the code a little deeper and added some additional logging statements around cache access. Data marshalling costs of even the most complicated structures could account for no more than about 2 ms of the 100 ms. Something was wrong. Then I noticed that while most of the entries we read and write are fairly small--often only several hundred bytes--one entry that seemed to be performing consistently poorly was a larger 22K. Now 22K isn't that big, but it was a clue.

Last night I had downloaded the C language libmemcache client to think about whether it might make sense to ditch Ruby for a few resource intensive computations. The unit tests in the package include a benchmark app that can repeatedly make requests of a certain size to the cache server. With some trial and error I found that reading entries sized 14,304 bytes completed in about 130 microseconds, while reading entries 14,305 bytes or larger required 100 milliseconds. This is pure, untainted, wonderful C code so there's no way I can blame Ruby for these strange results.

Something strange is afoot at the memcache server....

Today I spent more time--by several orders of magnitude--reading LiveJournal entries than I have in the last five years put together. And all this without actually even going to the LiveJournal site. I've been working on a Ruby module to parse Six Apart's update stream service for use in an upcoming site feature, and spent a good chunk of the day watching as entries scrolled across the terminal.

I was a bit worried about how I was going to parse the never ending stream of RSS entries. Most examples of XML parsing assume a (relatively short) document that is parsed into a DOM tree. This clearly isn't an option. Looking through the REXML source files I found a couple attempts at stream parsers. A search on Google found an article demonstrating the use of REXML::Parsers::StreamParser.

The StreamParser example looked good, but it still suffered from the problem that it assumed you would be passing in a string or a file. What I had, however, was a socket. I didn't think it would work, but I just handed the parser my socket and it happily started firing events. With a little more fussing around with code, I was able to set up a simple state machine to decipher the data. Now all I have to do is figure out what to do with it....

Today Eric upgraded our development servers at work to the new Rails 0.14.3, also known as Release Candidate 4 for the mythical 1.0 release. As expected, everything promptly broke. All of our sites stopped working. Eric worked some magic and brought the sites back up. He claimed the he didn't do anything more than restart apache, but I have a hard time believing that because I had restarted the server, twice, only moments before.

As I scanned the query logs I noticed that our ActiveRecord caching scheme was suddenly broken. This left me in a bit of a panic. We have developed a clever little caching layer, using the amazing memcached service along with an intra-page cache, that has successfully reduced the number of queries to the database by somewhere between 1/3 to 1/2. Performance would really suffer if we couldn't get the caching to work again.

The caching layer works by decorating ActiveRecord's find, update, destroy, and reload methods. The local cache and memcache are checked for objects before going to the database., while the caches are cleared after an update or destroy and before a reload. Optimistic locking and a reasonable TTL provide a safety net in case anything should go wrong.

I tried going through the Rails source code and change lists to uncover what might have caused the caches to be ignored, but this is a pretty grungy and hard to decipher part of the framework and I made little progress on that front. After some more experimentation, I realized that has_one and belongs_to relationship lookups were no longer calling find but instead were going directly to find_by_sql.

The solution? Another wrapper around find_by_sql that checks for queries of the form "SELECT * FROM table WHERE (table.id = ###) LIMIT 1" and call the cache-enabled find method instead of going to the database. The trick, however, is that if the objects are not in cache then find will call back into find_by_sql and we are then stuck in a loop. A class variable guards against by skipping the sql check in find_by_sql and going directly to the database if it is called as part of the processing of a simple find request.

I fear this caching layer is getting more fragile and more complicated with every revision of Rails. This latest patch is far from polished. It isn't thread safe--that's not a concern for us--and now that I think about it, isn't exception safe either. But at least the caches are working again.

I got a lot of attention from the Ruby community when I posted a Ruby haiku on my last blog.

I really had planned to give a more technical focus to this blog, but that hasn't yet come to pass. So, to tide over any Ruby holdouts, here is another haiku:

the perfect framework
for website development
is ruby on rails

do not be fooled by
the many immitators
they are all lacking

the magic of rails
the elegance of ruby
a powerful team

with joy and less code
and no xml sit-ups
whatever that means

Sorry about that one. I really will try harder to write some interesting technical entries. Really, I mean it this time.

UPDATE: Ruby guru Eric Hodel, known to the world as the enigmatic Dr. Brain, informed me that to be a true Ruby haiku, the poem must parse as valid Ruby code. I will have to be content, then, with writing haikus about Ruby instead of writing Ruby haikus.