I hope everyone is enjoying their Q4 right now. I know all of our servers are appreciating the extra love and attention from the holiday shoppers.

While everyone is frantically fulfilling orders, and maybe placing a few of their own, I felt this would be the perfect time to talk a bit about how all of that data is stored in our system. Fascinating holiday reading I know, so I’ll try my best to keep this entertaining and stick to the high level points of why your business should care.

Data is King
In my last 2 posts, I’ve focused on how all the “work” in our system gets distributed by bank style queues and self-heals from “evil monkey” errors through rollbacks and retries. Now it’s time to focus on the data itself. Every single scrap of information that powers your business: from the titles of your inventory to the buyers who purchase your products to the pagination setting you use in your views, all of it is stored in a database. Specifically at ChannelAdvisor, it’s a Microsoft SQL Server 2000 database (yeah yeah, we know we need to upgrade to 2005, but we are always conservative in upgrading our servers until several Microsoft patches have burned in first). If all of the application servers that processed the “work” were the linemen, the database would be the quarterback. The database holds all of the cards – it knows the score, the play, and where to pass the ball. Data is king!

As Spiderman would say, with great power comes great responsibility (although I still think Batman could whoop him). With such crucial information stored in the database, many steps must be taken to make sure that data stays safe. The first of these steps we take at CA is to distribute this information across many servers – last I counted I believe we had 21 quad-CPU, 32 gig Dell database servers actively serving the Merchant and Pro products, for which this article is geared. These are not your typical MP3 servers, these suckers have some honkin’ horsepower available to service our customers’ needs. (more…)

Share This

Grocery shopping…not a big fan. Cut-throat parking, endless waiting at the deli counter, squeaky shopping carts. No thanks. The checkout line is especially fun - I like to play a little game I call “find the fastest line”. The rules are simple: pick the one line that will get you the heck out in the least amount of time. Novice players oversimplify: find the shortest line and get in it. Amateurs. The pros know better, there are a dozen factors to consider: How full are the shopping carts? Any produce? How quick is the clerk? (bonus points for managers). Is there a bagger? Is anyone paying by check? And beware the ultimate evil: the customer with a stack of coupons, there will *always* be a dispute.

As amusing as this game is, it pains me to have to play it. Banks have figured this out years ago: one line per clerk is wasteful; one line feeding all clerks is fair to all. If little Jimmy is depositing $50 all in pennies, no problem – you skip his teller and move to the next. No games, everyone gets a fair shot.

Years ago at CA we made a fundamental design decision in our software that has paid off tenfold: use centralized “bank style” queuing to scale out our load efficiently. Back in the dawn of CA history (circa ~2000) our very own product manager Rick Watson, then a “light side” disciple of our engineering group (bet you didn’t know that, huh?), introduced an architecture we refer to internally as “Generic Services”. What the name lacks in flair it more than makes up for in capability. Generic Services was our own homegrown version of Grid Computing, before the phrase “Grid Computing” got hip. There are many overly complex definitions of what Grid Computing is, but I’ll oversimplify with this one statement: “Grid computing lets you spread lots of work across lots of machines in automated manner that uses all of these machines as efficiently as possible.” The enemy of a grid is an idle server – every minute your CPU sits at zero you’re wasting capacity. If you buy new servers while others are sitting idle, you’re wasting money.

(more…)

Share This

When I was a kid, I remember watching the movie Christine, based on a lesser known Stephen King novel by the same name. It’s been a few years, but I vaguely remember the plot went something like this: nerdy loser guy fixes up classic car, evil punk kids trash car (because, well, they’re evil), car gets possessed by evil spirits/magic lightning/whatever, car repairs itself and starts killing evil punk kids. The movie was mediocre at best, and is sure to have not aged well, but one part that has always stuck in my memory was how cool it was to see the car automatically repair itself each time the punks bashed it in. Metal untwists, windows pop back into place, floor mats straighten out, paint buffs itself, life is good.

I’m not sure why this (minus the evil Satan car killing people part) appealed to me so much. Maybe it was the cool special effects (it was 1983 after all) or maybe it was some personal rebellion against all those long, cold nights holding a flashlight in the garage while my father fixed some obscure part of the family car (”the fan belt combobulator is engaging the brake line manifold…”). I think one thing this did teach me (other than a total lack of interest in repairing cars), was a complete disdain for fixing the same problem again and again. A car that can fix itself, now that’s pretty darn cool.

Of course we don’t have self repairing cars or even self-fixing computer hardware yet (and no, exploiting Jed the intern doesn’t count), but in the world of “self-healing software”, we’re further along than you may think. Sure it’s not quite as cool as watching a punctured tire reinflate itself, but it’s still pretty darn neat and saves just as much time.

Coding for Failure
Let’s first take a step back. When an aspiring software developer takes a programming course or reads through a programming book, odds are they will learn what I call “coding for success”. In other words, you have a problem to solve, and you write code to provide a solution. Make me an application that can calculate how many bananas you can carry in a wheelbarrow (the monkeys are hungry). If you run out of bananas, flash up an error dialog and ask the user to provide more bananas. Every line of code is written with the assumption that the machine is up and running and working just fine.

In this wacky world 24/7 e-commerce, though, the rules are not so simple. Say I want to push 3,000 listings over to eBay in five minutes or less. No problem, ebay has an API, I’ll just write some code to push the data. But wait, even if my code is as fast as possible, I still can’t possible push that much data in that little time, the computer is just not that fast. Alright, so I’ll change my code so it can run in parallel on the same machine (think 1 person in a harness pushing 2 wheelbarrows side by side, a lot tougher but still possible). Hmm, but that’s still not fast enough, and there’s now smoke shooting out of the back of my machine. Okay, so maybe I need more than one machine to do the work, but how does that work? Somehow I need to orchestrate the work so it can run on many machines (e.g. 5 people pushing 10 wheelbarrows). Okay, so now I figured that out, and the boss comes back saying “by next year I want to push 30,000 listings out in 5 minutes”. Guess I need to buy more servers…

As you grow your software to run on more and more machines, handling more and more work and running longer and longer between breaks, an interesting phenomenon occurs: things that should always “just work”, don’t always “just work”. The database sitting on the Mac-Daddy “Hal-9000 would be jealous” hardware sometimes says “you know, I’m kindof busy right now. Would you mind coming back a bit later?”. Or maybe you’re connecting to eBay, but eBay’s network says “gosh, there sure are a lot of people coming in right now, I’m going to have to ask you to come back later.”. Wait a minute, this isn’t supposed to happen!

With this level of scale and interdependency, your approach to software must become more evolved. No longer can you “code for success”, but rather now you must “code for failure”. Each and every line of software must be written with the challenge “If this line failed, how would the system recover?”. If the Hal-9000 database stopped working (”What are you doing…Mark?”), or your Internet connection decided to “take 5″, or evil, crazed monkeys started shoving bananas into the server, how would the system recover without losing the work?

These are lessons they don’t teach you in college, but are oh so critical to keeping software running 24/7/365 across hundreds of servers. Remember, even if a failure only happens once in a million times, if you run over a million transactions each day, you’ll get a failure each and every day. Imagine if you walked up to an ATM and tried to withdraw $500, but the ATM software failed after the $500 debit from your account was made, but before the cash was dispensed into your eager hand. Yikes, you’re out $500, not cool! And a response from the bank of “oh that only happens once in a million times. Sorry, better luck next time!”, doesn’t really fly very well. Release the monkeys!

(more…)

Share This

At CA, we design a lot of UIs. Pretty much every feature requires one sort of UI or another. That’s why this article caught my eye. Seomoz links to an interesting article in UxMatters on the placement of labels on forms you fill out on the web.

As Seomoz mentions, having text on the left, and left-aligned, always seemed like the way to do things. Even text on the left that is right-aligned to the box received much better test results from their eye-tracking study..

Another interesting finding: if you put a drop-down list on a form, users will immediately fixate on it first, regardless of if it’s first. Then later when they go down through the form, they will re-check it.

I wonder what Jakob Nielsen would say about these… ChannelAdvisor Merchant generally left aligns field labels on the left side. If this is true, then not only would simply right-aligning them be better, but it would also be better to put the input right above the field. Looks like we are the worst of all worlds here.

WordPress itself, which is the blog software I’m using right now, can’t seem to decide. This post box has the label at the top, but the image upload right below this box right-aligns labels on the left. What to do?
Might be an interesting thing for us to try out….

Share This