A little more digging into the memcache code revealed some interesting details. It looks like the root of the problem is due to socket options in the server. To get the maximum network performance the server tries to disable the default Nagle packet buffering algorithm. On systems that support the TCP_NOPUSH socket option, the server will bracket network writes within a no push section and then let the operating system send back the result as soon as all the data has been written. If the system doesn't support TCP_NOPUSH, memcache will instead fall back to TCP_NODELAY.
It looks like FreeBSD supports the TCP_NOPUSH option but it doesn't seem to work exactly the way you would want it to. Reading up on the newsgroups, it looks like there have been some proposed kernel patches to bring FreeBSD's handling more in line with what is found on Linux. I didn't really want to mess with the kernel, so I simply recompiled the memcache server to use TCP_NODELAY.
Initial testing looks good. The 100 millisecond response is now processed by the Ruby client in just over one millisecond. This is definitely much better than 100 milliseconds. I'll let the new server run on our staging machines for a while before trying to push it out to the live site.
I've always been suspicious of TCP_NOPUSH myself... thanks for proving its flakiness once and for all. Usher in the days of TCP_NODELAY with much fanfare in the air (as well as an additional 99 milliseconds with which we can do anything we wish).
Sorry, I think I'm the author of that code (in memcached, not FreeBSD). I only did what my FreeBSD friends told me was correct, and only really test that it builds right, not runs right.
I'll open a ticket about this, so future versions perhaps can change the behavior at runtim.
- Brad