Greetings! This is my second post in a series I recently started discussing the Software Development Lifecycle we use here at ChannelAdvisor. If you have not yet read my prior posting on this topic, you may want to do so now. It’s a fascinating tale of pirates, danger, monkeys, and death defying software development on the high seas!

Previously I covered the concept of Software as a Service and then delved into the different system “environments” we utilize to deploy our software. In this post I will describe the concept of a “Change Request” and how those are categorized, ticketed and tracked. In upcoming posts I’ll focus on how Change Requests are deployed (”Short Term” and “Long Term” releases), and will delve into our movement towards Agile Development.

Change Requests
Within the company, every time a defect is found in our software (ahem, never), every time someone mentions “wouldn’t it be cool if the product could do X” (ahem, always), and every time a service request is issued to the engineering group, a “Change Request” (CR) ticket is opened to record and track the request. We literally log hundreds and thousands of these requests against ourselves, no matter how large or small in scope.

With all of those tickets opened, the use of Change Management software is critical to providing us order from the chaos. Currently we use a product called Synergy Change from Telelogic. It’s a solid product and serves our needs sufficiently, but could definitely be a little cheaper (just in case someone from Telelogic is reading this heehee).

(more…)

Share This

Well we’re definitely in the home stretch of the e-commerce super-bowl season right now. It’s starting to become a well known e-commerce fact that Monday’s are typically the busiest e-commerce day of the week. An interesting commentary on society I’m sure - rather than facing the doldrums of another week at work, we go to the office, log on and…start shopping! This year is especially interesting, as the last Monday of the season falls a full week away from Christmas - just enough time to get that Playstation 3 by ground shipping. It’ll be interesting to see if consumers take that risk on Monday and break some e-commerce records.

Anyway, as promised in my last post, today I thought I’d talk a little about security, specifically as it applies to our Merchant and Pro products. I’m not going to pull a CNN here and broadcast exact specifics (”as you can see here, the 3rd cavalry is going to launch their surprise attack on this position in exactly 10 minutes…”), but there is a high level discussion we can have here that doesn’t give away the secret sauce.

First of all, let me stress that security of our data is something we take very seriously here at CA. Is any system foolproof? Of course not, we’d be fools to think otherwise. We do however use well established industry practices to secure your data, and have subjected ourselves to numerous internal and external security audits to validate our practices. With that said, we are always working to get even better.

Physical Security
First and foremost, there are the servers themselves. Our servers aren’t hanging out in a garage or under Marshall Smith’s desk, they live in a dedicated data center. If you’ve never seen a datacenter, they look a lot like this. If there’s a part a part of reality that’s straight out of a sci-fi movie, it’s a data center. Walking into one you could easily convince yourself that computers have taken over the world and you are the last human on earth. Time to wakeup and feed your master.

Here’s a picture of a Google Data Center I found on, well Google:

(more…)

Share This

Grocery shopping…not a big fan. Cut-throat parking, endless waiting at the deli counter, squeaky shopping carts. No thanks. The checkout line is especially fun - I like to play a little game I call “find the fastest line”. The rules are simple: pick the one line that will get you the heck out in the least amount of time. Novice players oversimplify: find the shortest line and get in it. Amateurs. The pros know better, there are a dozen factors to consider: How full are the shopping carts? Any produce? How quick is the clerk? (bonus points for managers). Is there a bagger? Is anyone paying by check? And beware the ultimate evil: the customer with a stack of coupons, there will *always* be a dispute.

As amusing as this game is, it pains me to have to play it. Banks have figured this out years ago: one line per clerk is wasteful; one line feeding all clerks is fair to all. If little Jimmy is depositing $50 all in pennies, no problem – you skip his teller and move to the next. No games, everyone gets a fair shot.

Years ago at CA we made a fundamental design decision in our software that has paid off tenfold: use centralized “bank style” queuing to scale out our load efficiently. Back in the dawn of CA history (circa ~2000) our very own product manager Rick Watson, then a “light side” disciple of our engineering group (bet you didn’t know that, huh?), introduced an architecture we refer to internally as “Generic Services”. What the name lacks in flair it more than makes up for in capability. Generic Services was our own homegrown version of Grid Computing, before the phrase “Grid Computing” got hip. There are many overly complex definitions of what Grid Computing is, but I’ll oversimplify with this one statement: “Grid computing lets you spread lots of work across lots of machines in automated manner that uses all of these machines as efficiently as possible.” The enemy of a grid is an idle server – every minute your CPU sits at zero you’re wasting capacity. If you buy new servers while others are sitting idle, you’re wasting money.

(more…)

Share This

When I was a kid, I remember watching the movie Christine, based on a lesser known Stephen King novel by the same name. It’s been a few years, but I vaguely remember the plot went something like this: nerdy loser guy fixes up classic car, evil punk kids trash car (because, well, they’re evil), car gets possessed by evil spirits/magic lightning/whatever, car repairs itself and starts killing evil punk kids. The movie was mediocre at best, and is sure to have not aged well, but one part that has always stuck in my memory was how cool it was to see the car automatically repair itself each time the punks bashed it in. Metal untwists, windows pop back into place, floor mats straighten out, paint buffs itself, life is good.

I’m not sure why this (minus the evil Satan car killing people part) appealed to me so much. Maybe it was the cool special effects (it was 1983 after all) or maybe it was some personal rebellion against all those long, cold nights holding a flashlight in the garage while my father fixed some obscure part of the family car (”the fan belt combobulator is engaging the brake line manifold…”). I think one thing this did teach me (other than a total lack of interest in repairing cars), was a complete disdain for fixing the same problem again and again. A car that can fix itself, now that’s pretty darn cool.

Of course we don’t have self repairing cars or even self-fixing computer hardware yet (and no, exploiting Jed the intern doesn’t count), but in the world of “self-healing software”, we’re further along than you may think. Sure it’s not quite as cool as watching a punctured tire reinflate itself, but it’s still pretty darn neat and saves just as much time.

Coding for Failure
Let’s first take a step back. When an aspiring software developer takes a programming course or reads through a programming book, odds are they will learn what I call “coding for success”. In other words, you have a problem to solve, and you write code to provide a solution. Make me an application that can calculate how many bananas you can carry in a wheelbarrow (the monkeys are hungry). If you run out of bananas, flash up an error dialog and ask the user to provide more bananas. Every line of code is written with the assumption that the machine is up and running and working just fine.

In this wacky world 24/7 e-commerce, though, the rules are not so simple. Say I want to push 3,000 listings over to eBay in five minutes or less. No problem, ebay has an API, I’ll just write some code to push the data. But wait, even if my code is as fast as possible, I still can’t possible push that much data in that little time, the computer is just not that fast. Alright, so I’ll change my code so it can run in parallel on the same machine (think 1 person in a harness pushing 2 wheelbarrows side by side, a lot tougher but still possible). Hmm, but that’s still not fast enough, and there’s now smoke shooting out of the back of my machine. Okay, so maybe I need more than one machine to do the work, but how does that work? Somehow I need to orchestrate the work so it can run on many machines (e.g. 5 people pushing 10 wheelbarrows). Okay, so now I figured that out, and the boss comes back saying “by next year I want to push 30,000 listings out in 5 minutes”. Guess I need to buy more servers…

As you grow your software to run on more and more machines, handling more and more work and running longer and longer between breaks, an interesting phenomenon occurs: things that should always “just work”, don’t always “just work”. The database sitting on the Mac-Daddy “Hal-9000 would be jealous” hardware sometimes says “you know, I’m kindof busy right now. Would you mind coming back a bit later?”. Or maybe you’re connecting to eBay, but eBay’s network says “gosh, there sure are a lot of people coming in right now, I’m going to have to ask you to come back later.”. Wait a minute, this isn’t supposed to happen!

With this level of scale and interdependency, your approach to software must become more evolved. No longer can you “code for success”, but rather now you must “code for failure”. Each and every line of software must be written with the challenge “If this line failed, how would the system recover?”. If the Hal-9000 database stopped working (”What are you doing…Mark?”), or your Internet connection decided to “take 5″, or evil, crazed monkeys started shoving bananas into the server, how would the system recover without losing the work?

These are lessons they don’t teach you in college, but are oh so critical to keeping software running 24/7/365 across hundreds of servers. Remember, even if a failure only happens once in a million times, if you run over a million transactions each day, you’ll get a failure each and every day. Imagine if you walked up to an ATM and tried to withdraw $500, but the ATM software failed after the $500 debit from your account was made, but before the cash was dispensed into your eager hand. Yikes, you’re out $500, not cool! And a response from the bank of “oh that only happens once in a million times. Sorry, better luck next time!”, doesn’t really fly very well. Release the monkeys!

(more…)

Share This