When Shopify (SHOP 0.14%) founder Tobi Lutke started building what would become the biggest provider of e-commerce storefronts for small business in 2004, he built the core using Ruby on Rails, a programming language that helped define much of the wiring of the world's internet-based software -- from Twitter to privately-held Basecamp.
Web programming has changed a lot in the 15 years since. Languages such as Django and Python and different flavors of Javascript, such as Node.js, have grown popular for wiring apps quickly. Shopify still uses Rails at the core, but it has significantly upped its game in the way it handles data and keeps over 820,000 online shops up and running. I'm convinced the company's approach gives it a competitive edge -- here's how.
Shopify is scaling up in a big way, just as Amazon did once. Image source: Shopify.com.
Built to scale
Like most tech companies -- and frankly, most of us humans -- it took a mistake for Shopify to learn how to grow at a massive scale.
By 2014, Shopify had so much data that the company's MySQL "instance" had become overwhelmed. The problem? Shopify's core database was, effectively, one giant warehouse stuffed to overflowing. To solve the problem, the company split up its database into partitions, like a series of storage spaces rather than one large warehouse.
Unfortunately, as Shopify Production Engineer Kir Shatrov writes at Stackshare, solving the problem introduced an entirely new problem. Instead of one MySQL database making requests of the rest of the infrastructure, now dozens of mini databases were. That proved to be too much to bear.
"All those shards were still using a single Redis [database for handling queries]," Shatrov writes. "At one point, the outage of that Redis took down all of Shopify, causing a major disruption we later called 'Redismageddon.' This taught us an important lesson to avoid any resources that are shared across all of Shopify."
Today, the company uses containers from Docker and Kubernetes to pull together "pods" of compute, storage, memory, and more. These chunks of code represent a "full instance of Shopify with its own datastores," Shatrov writes. More importantly, the company uses the Google Kubernetes Engine to create and orchestrate its pods. Microsoft (MSFT -1.84%) recently made a similar capability available in embracing Kubernetes for its Azure cloud-computing platform.
How we'll know if Shopify's advantage is fading
Shopify has plenty of skeptics, including, most recently, notorious short-seller Citron Research, which doubts Shopify's ability to keep attracting merchants amid increased competition from Instagram and others.
Maybe Citron is right. We'll know if store expansion slows significantly. On Black Friday weekend in 2017, Shopify set new records by serving over a half-million merchants. Today, less than two years later, over 820,000 are on the platform. If Instagram truly is eating into its business, 2019 won't be the year Shopify hits 1 million merchants served.
I think the safer bet is that Shopify will pass not only 1 million but also 1.1 million, and maybe more. Thanks to careful investments in its tech infrastructure over the last five years, Shopify has positioned itself as the one small business e-commerce vendor that can scale up with successful businesses as they grow. Amazon birthed AWS for similar reasons.
That's an important corollary. Because if history is at all instructive, here -- and I believe it is -- investing in scale tends to produce it, as you inevitably draw in the customers that need your services most. In the world of small business, there remain a great many that need the sort of scale that -- so far -- only Shopify can offer.
