December 2005 Archives

I was up until 2:00 am last night working on the database. I wanted to wait until the regular 1:00 am full backup completed so there would be something to go back to if I really screwed something up badly.

Using my training in the scientific method that I gained as a physics student at one of the top research universities in the country, I decided that my strategy would be to blindly poke at things and see if anything happened. And I would have to poke hard. If a little is a good thing then surely a lot must be even better. I made some adjustments to three or four settings and restarted the database. Things seemed to be working better than ever. Of course, it's hard to judge because the site always performs well under the light night traffic.

I went to sleep confident that I had accomplished something good. The sound of the pager at 4 am suggested otherwise. With a strange calmness I was able to form a bit of a theory as to why the database was freaking out and back out those changes. The clues were there earlier but at that time I wasn't aware of what they meant.

I was perhaps a little more conservative than necessary, but I waas able to leave in a change that I think will actually make the biggest difference. And with that, I've consumed pretty much all of the collective knowledge I could glean off the internet for tuning MySQL databases.

The only thing left that I can think would make a difference is to sacrifice some up to the second synchronization gurantees for reduced disk access. Dr. Brain may disagree, but I'm not convinced that an fsync() after each and every transaction is really necessary. But even this will become less of an issue when our new database server with its faster striped disks comes online.

With things working once again, I headed back to bed, only to wake up too soon to the ringing of the alarm clock. I pressed snooze for as long as I could, but I needed to be awake and ready to meet Andrej for breakfast this morning. And so I'm not sure whether I will have enough energy to carry me through the new year or whether I will just crash sometime later this evening.

A little more digging into the memcache code revealed some interesting details. It looks like the root of the problem is due to socket options in the server. To get the maximum network performance the server tries to disable the default Nagle packet buffering algorithm. On systems that support the TCP_NOPUSH socket option, the server will bracket network writes within a no push section and then let the operating system send back the result as soon as all the data has been written. If the system doesn't support TCP_NOPUSH, memcache will instead fall back to TCP_NODELAY.

It looks like FreeBSD supports the TCP_NOPUSH option but it doesn't seem to work exactly the way you would want it to. Reading up on the newsgroups, it looks like there have been some proposed kernel patches to bring FreeBSD's handling more in line with what is found on Linux. I didn't really want to mess with the kernel, so I simply recompiled the memcache server to use TCP_NODELAY.

Initial testing looks good. The 100 millisecond response is now processed by the Ruby client in just over one millisecond. This is definitely much better than 100 milliseconds. I'll let the new server run on our staging machines for a while before trying to push it out to the live site.

My natural distrust of the Ruby programming language might have caused me to miss something important. On 43 Things we rely heavily on memcache to offload database reads. After a bit of work simplifying and tuning the networking code, we were able to get very good response times for data lookups. Occassionally, however, the timings drifted from their usual submillisecond range to close to 100 milliseconds. I just assumed that either Ruby networking code occassionally freaked out or that we were caching complicated data structures that could take a while to parse.

Recently I've been working on some alternative algorithms to help solve some of the performance problems we've been seeing on a few of the more intensive pages. These solutions make extensive use of caching. While a single 100 ms lookup on a page might slip through without much notice, a handful of them will easily kill page serve times.

I dug into the code a little deeper and added some additional logging statements around cache access. Data marshalling costs of even the most complicated structures could account for no more than about 2 ms of the 100 ms. Something was wrong. Then I noticed that while most of the entries we read and write are fairly small--often only several hundred bytes--one entry that seemed to be performing consistently poorly was a larger 22K. Now 22K isn't that big, but it was a clue.

Last night I had downloaded the C language libmemcache client to think about whether it might make sense to ditch Ruby for a few resource intensive computations. The unit tests in the package include a benchmark app that can repeatedly make requests of a certain size to the cache server. With some trial and error I found that reading entries sized 14,304 bytes completed in about 130 microseconds, while reading entries 14,305 bytes or larger required 100 milliseconds. This is pure, untainted, wonderful C code so there's no way I can blame Ruby for these strange results.

Something strange is afoot at the memcache server....

One of my favorite pieces of equipment in my building's fitness center is the Nautilus Gravitron. It uses a bit of anti-gravity magic to reduce your to make it easier to do pull ups or chin ups. I've never been able to do a full pull up before, but thanks to a 230 lb reduction in weight (the maximum the machine allows), I am able to easily lift myself. I can pretend, if only for a while, that I am getting ripped like Dr. Brain, training to become one of his henchmen. The only problem I have is that I must now, unfortunately, reduce the counterweight by 10 lbs to remain at the level I started.

There was a time a few years ago that I would walk close to 60 miles per week. The effect on my body (and shoes) was quite striking. It's hard to keep up that kind of activity--it just takes too much time to walk that much--and this past year my average dropped closer to 15 miles per week. I've actually been doing better than that for the last few weeks as I've consistently tried to walk all the way to work each morning instead of completed that last half of the journey on the bus.

Yesterday I decided it was time to kick things up a notch once again. I left my apartment at about 11:30 and started walking toward the University District. That's a respectable walk in and of itself and I wouldn't have felt bad about jumping on a bus right then and returning home. Instead I decided to join the Burke-Gillman trail for a while. This was a bold move. Normally I try to walk close to major bus routes. There are more shops along the way in case I need to stop for water or use the restroom, and it's easy to end the walk early for whatever reason. Walking along the trail, however, trades these conveniences for a more enjoyable walking experience, a relatively flat path sheltered from the noise of street traffic.

When it felt like I had walked far enough I turned around. I'm not sure how far I went, because I never looked at my watch throughout the whole trip. Instead I had only vague estimates based on how far I had gotten along my iPod playlist. For example, it took about the lenght of The Wren's "The Meadowlands" for me to return from where I turned around on the trail to reach the southernmost part of the University bridge.

Instead of returning home, I continued walking to downtown and looked for an interesting movie to watch that would give my poor feet a few hours rest. Only when I reached the theater did I look at my watch. It was 4:00 and I had been walking without stopping for four and a half hours. I bought tickets for the next showing of Syriana and then headed to the drugstore to get some water. I hadn't realized just how dehydrated you can get in the cold winter air.

I learned last night that you can take a really bad picture--I mean a truly horrible, dark, out of focus, and grainy 0.3 megapixel, cheap cameraphone picture--post it on Flickr and have a ton of people view it by simply tagging the photo "erikbenson". What I need to do now is figure out other ways to take pictues that people will want to look at. Forget artistry or artistic integrity, I want popularity.

Black Thursday
Originally uploaded by rgcottrell.

I saw this shirt in a store on the way in to work on day and felt compelled to buy it. The pros and cons of the recently enacted smoking ban had been debated at the office and on 43 Things for a while, and this felt like a silly way to commemorate them.

I was going to use this as a prop in some elaborate presentation that I had been imagining in my head. Of course I ultimately realized that I wouldn't be as clever, charming, and witty in person as I pictured myself and with the unexpected change of venue last night I just didn't feel in the mood.

The deadline for the ban has come and gone now. I really don't feel much about the ban as I've never even taken a drag off of a cigarette. Still, I didn't follow through on my plan to take up smoking before our freedoms were taken away in the slow but steady march toward a totalitarian regime and it's too late now. My smoking experiences will have to consist of things might or might not have happened when I was last in Amsterdam.

I'm not sure what I'm going to do with this shirt now. I didn't get it in my size so I can't even wear it myself.

At lunch today (err, yesterday) we were trying to decide where to eat. Eric suggest Tacos Guaymas and although I am not a big fan of that particular restaurant, we hadn't been there for a while and so I agreed to go. As we walked down the hall, I starting reciting lines from an episode of Beavis & Butt-head. "Tacos...burritos...spaghetti". Erik latched on to the end and started talking about spaghetti burritos.

During Office Hours he still seemed obsessed with the concept of spaghetti burritos and asked just about everyone around what they thought should go into a spaghetti burrito. At first I dismissed this as silly, but after thinking about it for a while, I'm not so sure. Is spaghetti any worse a source of starch as the rice that you would find in a traditional burrito?

I may have to think about a good combination of ingredients and try to create a spaghetti burrito this weekend. I think I have the culinary skills necessary to boil a pot of water to cook the spaghetti. The hardest part, I think, would be clearing enough space around the range to keep the apartment from going up in flames when I turn on the gas.

As the displaced cast of Office Hours regular slowly arrived at our temporary location, a slew of Dodgeball check-ins took to the ether. I sent in my check-in as well, not so much because I needed to inform anyone about where I was but because I was curious to see if I would receive any friend-of-a-friend messages.

A few seconds later I received a notice that Angela had just checked in as well. I knew that, of course, as she was sitting just two seats down from me. After reading the brief introduction and scrolling past the rather large profile image (more than a screenful on my phone), I noticed that the message seemed to be coming from one of those carrier email gateway addresses. I confirmed with Angela that I had in fact just received her phone number. This seems a bit strange to me.

I think your contact information is private even from your friends but it is just given away to friends-of-friends. Is this the intended functionality? Of course, I'm too much of a chicken to ever call Angela, but I think I'll keep her photo and phone number just the same....

Tonight Office Hours faced its strongest challenge yet and survived remarkably well I think. Erik and I left work and made our way to the Hideout, only to find that it was closed for the night for a private benefit. We wandered around the neighborhood for a while but didn't find a suitable replacement. I'm no help with suggestions at all and Erik was just about ready to give up and cancel the night when he received a call from Sami suggesting an alternate venue. Through a series of Dodgeball messages and phone calls, Erik was able to alert enough a surprising number of people to our new location. Being at my core a relatiively antisocial person, I enjoyed the smaller turn out. Still, I think I've been making some progress interacting with others and managed to engage in a few conversations at least.

Just after midnight we arrived at the War Room for some late night I don't know what. It was mostly a DJ playing music although there was a guitar player who would pick out a few odd notes every now and then. Later in the night a singer took the stage and made the music a little more interesting. As the rest of the group took to the floor, I reluctantly joined them. I hope there weren't any cameras or other recording devices present at the time to record my inept attempts at dancing. I started off trying to sway to the music but that pretty much left me missing every single beat. My next attempt was more like jogging in place. This was a little more successful at keepintg me in the rhythm but made me look ever more foolish.

As the night ended I split a cab with Erik back to Belltown. It was just a little too cold for me to make the longer walk home. I'm actually surprised I survived the night as well as I did. I was certain that the 20 minutes on the elliptical machine in the morning or the three times I walked around the park in the afternoon would have left me too tired to make it through the night. But at 2 am I feel almost as alive and awake as I have all day. I guess my new exercise routines have been making a difference after all.

Amazon may have made a mistake offering me a free three month trial of their "Prime" free second day shipping program. If they were hoping to increase unit sales this holiday season, they certainly succeeded. I suspect that next quarter's financial statement will include a line item recognizing all the purchases I've made. But free expedited shipping on $10 orders hardly seems like an economically viable situation.

Normally I don't preorder items, but since there was no longer an extra cost for ordering single items, I went to the music store and put in preorders for a bunch of upcoming CDs. One of these was the new Lindsay Lohan album that was just released yesterday. I went Amazon to check on the status of my order and found that the price had dropped by another dollar since I ordered it. Fortunately, with the exception of the Harry Potter books, I've found that Amazon can take a little time shipping new product on release day, so there was still time to cancel the order and reorder at the lower price.

Why would I buy the new Lindsay Lohan CD? Well, quite simply Lindsay Lohan rocks. If you don't believe me you can just check out the URLs for her website (http://www.llrocks.com) or her MySpace page (http://www.myspace.com/lindsaylohanrocks). I was on the MySpace page this morning as they were updating it. And so I became the very first person to play the newly uploaded track on MySpace. I feel so special.

There's been a lot of talk around work and among friends about self improvement. Erik has been particularly expressive of his plans in a series of posts on 43 Things. I think I've fallen into a bad crowd and feel myself succumbing to some implicit peer pressure.

As I was wasting away Thanksgiving day with nothing to do until Erik's party later that evening, I decided to take advantage of the relative emptiness of the building to spend some time doing some simple exercises while no one was around to mock my wimpy attempts. My first session reminded me of a scene out of Beavis and Butt-head when the coach orders the boys into the gym to build up some muscle. Their time there is a disaster. Butt-head lies down on the bench and tries to press the bare barbell with no additional weight on it but can't even manage that. The bar falls on his neck and starts to choke him. As Butt-head starts to turn blue, Beavis rushes over to help but even the combined strength of the two of them is not enough to lift the bar. I did a little better, but still I was lifting the barbell with no weights, struggling to maintain balance.

The party that evening lasted well into night and toward the end I felt my arms getting really quite sore. I did my best to stay curled up in the corner of the sofa because I nearly lacked the strength to prop myself up. My muscles stayed sore throughout the rest of the weekend. It wasn't until early the next week that I noticed the absence of the constant pain I had been living with for the past several days.

The worst part of exercising is that the effects are only temporary. The same weights and repetitions no longer tire my muscles like they used to. And so I am going to have to increase the weights to compensate for new strength. I find that I am now walking faster and I can usually get to the office in the morning in less time when I walk than when I waited for the bus. And this is starting to disturb my sleeping habbits as well. I'm staying up longer and have more energy throughout the day. I'm feeling less of an urge to just sit passively in front of the television .What am I supposed to do with all the extra time? I don't like it at all.

Ever since I finally got the server's email system properly configured, I began receiving nightly summaries from the LogWatch script. Part of the report details system logins and attempted logins. There are usually a handful of failed login attempts each day, but yesterday was the worst I've seen. There were over 2,000 login attempts coming from three separate hosts. A MaxMind GeoIP lookup of these IP addresses showed that most came from an ISP based out of New York while others came from a Danish ISP and a handful of others came from our good friends at Brasil Telecom.

SInce I've been going out quite a bit more than usual these last few weeks, my television watching has suffered significantly. I've fallen behind on multiple shows. I think "Lost" is coming close to being a lost cause--the back episodes are just about to slide off the TiVo's keep up to five episodes setting.

After getting home from a day at my sister's, I needed some quality time with the television to properly unwind. I started playing last week's episode of "Grey's Anatomy". I was kind of freaked out by the scene at the beginning in which Christina walked through Burke's empty apartment and I recognized my counter stools and dining chairs as well as the refrigerator and dining table that closely lack the look and style of mine. The general state and cleanliness of my aparment, however, more closely match Christina's than Burke's.

The scales that I ordered from Amazon that haven't shipped yet arrived on Friday and were waiting for me when I returned home on Saturday afternoon. The two forty page manuals looked impressive, although in reality, each had only half a dozen pages of instructions that were then repeated in seven Western European languages. The scale itself, however, was similar enough to my old model that I was able to quickly set it up without needing to consult the instructions.

It was late in the day when I took my first reading so the readings will be a bit off from subsequent readings which I intend to make every morning just after waking up. I weighed myself twice and got matching weights both times (which also matched the weight on my old scales) so I will considered basic weight measurements fairly accurate. There was a swing of a few percentage points in the fat and water percentages, so I won't consider any of the other measurements authoritative but watching the trends should be fun. My old scale also measured fat percentage and itsmy measurement was significantly higher than the new scale's, increasing my suspicions of the accuracy of these secondary measurements.

And so I have my starting point for measuring my body's composition in nine different areas. It appears that I have the metabolism of a senior citizen and so should probably start increasing my activity levels. And unlike Cartman's assertion that "I'm not fat, I'm just big boned", I have evidence that I am both fat and big boned.

On Tuesday I ordered one of those fancy bathroom scales from Amazon. Not only is it quite stylish, but it can apparently slice up your weight into percentage fat, water, muscle, and bone. I thought I might use this as a data source for an Erik Benson style Health-O-Meter as I begin another try at getting healthier and losing some weight. There was a floor display at Bed, Bath & Beyond that I look at last weekend but unfortunately they didn't seem to have any in stock, so I went to Amazon to fill the order.

It was suppose to ship within 24 hours by second day air (thank you Amaozn Prime) but when I checked my order today I noticed that it still hadn't shipped. Not only that, but the price had dropped by $31 and the order was too far along in the process that I could no longer cancel the order. I didn't have high hopes, but I filled out a customer service form asking for a credit for the difference so I wouldn't have to return it and then reorder it. Of course, I was bluffing--I'm way too lazy to actually go to the additional effort this would require.

I received a response a few hours later confirming that the price had in fact dropped and that the difference would be credited back to me. The email went on to apologize that it was too late for them to try to upgrade the shipping to get it to me faster. So thank you "Amber", if that is indeed your name, for once again exceeding my expectations.

Today is Britney Spears birthday. I got to the office quite a bit earlier than normal--it's amazing how much faster it is to drive to work than to walk--and played through her last CD twice before anyone else showed up for work.

Oh, and it's probably my birthday too, although that one is much less important.

I think I might have been just a little too hasty when I put my last iPod to the hammer. I think I just assumed that when it refused to turn on that the battery had just died. It was an old iPod and had lived a very long and fulfilling life. Surely its time had just come.

Tonight, however, I am having second thoughts about that theory. When I plugged my brand new 60 GB video iPod into the computer, it suddenly locked, stuck on the "do not disconnect" screen. It wouldn't power down. It wouldn't power up. And the computer refused to recognize its existence.

I could just feel my anger rising. Had I just destroyed another brand new and very expensive iPod? Would I have to spend hours waiting for a spot at the genius bar to to convince them to replace the player? Or maybe I should just put the thing out of its misery. I starting trying to remember where I left the hammer. For a moment, I even considered going down to my storage unit to fetch the power drill.

Instead, I decided that I would simply go for a walk. I grabbed my coat and headed down to Pioneer Square. I haven't been to that part of town for almost a year now. I took some time to walk by some of the office buildings where I used to work. I returned nearly two hours later no less furious but at least too tired to do something I might regret.

I did a little Googling for iPod factory reset codes and after spinning around the Apple troubleshooting documents for a bit found the magic combination of buttons to press. And with that, my iPod returned back to life, good as new.

I'm sure my brother will be thrilled when I tell him that the iPod that I had promised to give him was probably still working just fine and that I smashed it to bits for no reason at all.