If you’ve read a couple of previous posts on this subject, or attempted to access the Columbia Missourian Web site recently, you might have noticed it’s been a bit quirky.

So, to sum up:

Monday, we got hit with a large load of traffic, mostly from the Drudge Report, that knocked both our Web servers offline. Thecampus load balancer (not controlled by us) apparently stopped searching for either of our web servers once both went down, so when the servers came back up and were ready to go, it wouldn’t allow either to serve up content. A frustrating day ensued … everything was running fine as far as we could tell, but we couldn’t track it down to the load balancer until close to 4 p.m. That’s now been fixed. Monday traffic was close to 29,000 visitors in 8 hours of uptime (a normal 24-hour day has maybe 10,000-11,000).

Tuesday, more traffic and more slowness. Our programmer spent most of the day restructuring our database to help with load times, tweaking caches, and the normal sort of things you do. Down to about 18,500 visitors.

Wednesday, we spent a couple hours in the late afternoon testing a new host location for the Missourian’s mySQL database, with help from a mySQL expert (neither my expertise nor our programmer’s). While the site was offline and the new database was being tested, our programmer made a few tweaks to firewall and cache settings and did some upgrades to help with speed. Only about 7,500 visitors made it through.

Today, we moved the SQL database to a new location that lets us basically store the whole database in RAM, which speeds up the site considerably. Hoping that will lead to visitor numbers coming back to what they used to be … and that we can get a couple of lingering Apache programming issues dealt with. More updates TK.

So Google Analytics says at least a couple hundred people read this post about what happened to the Missourian Web site on Monday.

The 200 of you who represent a record day for my blog (thanks, all!) are about 28,800 fewer visitors than the Missourian got on Monday, in the 8 hours or so that the site was live. We normally get about 10,000 to 12,000 visits per day, so that was clearly a bit of an epic day for us. On Tuesday, our numbers slowed a bit to about 18,500 visits, and they seem to be slowing a bit more today. Of course, analytics code only counts visitors who can make it through to load a page, and so clearly those numbers are undercounting actual visitors.

That, in a nutshell, is what’s still up with the site. We’re getting more visits than our servers can handle (the greatest strain seems to be on our database server). We’re also getting offers for help from several people on campus, and taking them up on it when we can. Our programmer is working to restructure our database to more quickly load content, and is also working overtime clearing out dead connections, rebooting Apache when needed, and so on. Appreciate the patience as we work through it.

More updates as needed.

Long day yesterday with the Columbia Missourian site up and down, but mostly down.

Here’s the basic explanation of what happened from our programmer, Noah Medling:

Our website runs on four physical servers: one for media (photos, videos, etc.), one for the database (articles, user permissions, almost everything textual in nature), and two webservers that take that information and assemble it into what you see in the browser.

There’s a load balancer that sits between you and the two webservers that alternately sends requests to one or the other, spreading the work across both as evenly as it can. If one of those servers stops functioning, it stops directing traffic to it and we run off of just one webserver until the other comes back online.

The load balancer was using HTTP response time to determine if a server was working or not. Our website is, as you probably know, not very fast, but they’re usually not so bad that the load balancer complains. This morning, around 8:00, one of the webservers stopped being responsive. The load balancer steered all of our traffic to the other one, which was now handling everything rather than just half, and we managed to get record amounts of traffic at the same time. This slowed down the remaining server to unacceptable levels and the load balancer took it offline, leaving us with no website.

I got here at 8:30 and worked on the problem constantly from then until it was finally resolved around 4:00. I never did figure out what happened to the first webserver, but I rewrote a sizable chunk of configuration code in resolving other issues, so it’s likely that whatever caused that problem in the first place no longer exists. Because the load balancer is outside of my control, it took a long time to discover that it was part of the problem. It has now been reconfigured to no longer take servers offline for being slow, and Rob and I now have contact information for a number of people in networking and CSG who have access to the parts that I don’t, and can help in getting this sort of thing resolved much faster in the future.

On top of that, we had a couple of other issues once it came back late yesterday:

Because the site was down pretty much all day, all its caches expired, so it took forever to re-cache items, leading to further slowness.

We were also being linked to from the Drudge Report, which was quite possibly the source of some of the heavy traffic from Monday morning (the story about cotton balls being dropped at the Black Culture Center on the MU campus).

At around 4:30 on Monday, for example, we had about 10,000 concurrent connections (our previous record was 7,000) to the site, which were a combination of Drudge traffic, search engines/bots and regular connections.

The good news, I suppose, is that our site was handling the traffic without completely crumbling. The bad news, of course, is that it was running ridiculously slowly in the meantime.

Anyhow, we’re slowly recovering and will hopefully have things running more speedily as time goes on. Thanks for being patient in the meantime.