Five Lessons from Deploying 25 Sites across 18 Metros in 14 Months...During a Pandemic!
Building anything physical during the last year and a half has been a challenge, but looking on the bright side, the challenges we faced made us a better and more creative team.
When Packet was acquired in March 2020, we had key network points of presence in Equinix data centers, but the majority of our footprint was located in a variety of wholesale facilities. From Interxion and Cologix to DataBank, Cyrus, Cyxtera and Sabey -- we had a little bit of everything!
As part of Equinix we suddenly had a very new and compelling opportunity: to build our entire platform smack in the middle of the most desirable, interconnection-rich data center campuses in the world. As a longtime cloud builder, this was a dream come true.
Together with our colleagues across Equinix we decided upon the initial hitlist: 18 metros that included the obvious (Amsterdam, Washington DC, Silicon Valley, Singapore, etc) and a few fresh faces (Madrid, Seoul, we’re looking at you). By May we started to buy and deploy.
It took us 7 months to complete the first five builds (in DC, SV, NY, LA, SG). but in the past 7 months — as the pandemic and supply chain competed for “best performance in a dramatic feature film” — we’ve knocked out 20 more builds across 13 metros (we often build into more than one data center within a metro, that’s why we have more builds than metros).
While my team and I have built a lot of clouds in a lot of places over the years, this time was different. Looking back, here’s what I’d share with anyone interested in doing the same.
#5 - Supply chain expertise will make you or break you
It sounds simple, but you can’t build with things that you don’t have! Not only do you have to be up to 9 months ahead of your game for key components (we aim to stock a year’s worth of switches), but having a proactive stance on things like logistics, shipping, and importing as well as management of vendor ready stock relationships is critical.
As the supply chain tightened this year, we were fortunate to already be closer to the front of the line versus the back of it, and that has really proven invaluable in getting our new sites live on time.
#4 - Staging components makes everything go faster
The funny thing about really valuable data centers is that most of the space is taken up by rows and rows of tightly packed cabinets. As such, there is very little on-site storage for the equipment needed to build or expand an environment.
To mitigate this challenge we stage a lot of our components in regional hubs (you could also leverage an integrator) and then rely upon a “just in time” delivery strategy to get things to our space-constrained environments at just the right time.
#3 - Automation and program management are like peanut butter and chocolate
There were times over the past year when we were turning up 3-4 new sites simultaneously while also adding substantial inventory to existing sites. With everything competing for priority, it was a combination of automation and good old-fashioned program management that saved the day.
To get from zero to 18 metros we built and refined a runbook to hone our “standard operating procedures.” Not only did this allow our teams to stay aligned but we were able to improve and refine the process as we hit new challenges. Alongside this we partnered with our engineering team to build tools that sped up switch setup, automated server enrollment, kicked off burn in / testing runs, and identified human errors like cable swaps.
#2 - Open19 is going to be a game changer
To make this year a bit more exciting we ramped up our investment in Open19-based hardware, which now makes up more than 30% of our fleet. Since Open19 is still pretty new when it comes to deploying in the field, we hit plenty of “gotchas” that took time away from other efforts. However, looking back it was worth it — we’re making investments in efficiency of both financial and human capital that will help us get to the next level.
Deploying traditional 19” racks either using “rack and roll” or onsite build models is both expensive and time consuming, requiring capital and expertise at the same time. With Open19, we’re able to disaggregate those two things and build significant amounts of low cost core infrastructure (rack, cable and network) separate from dropping in our (expensive and often changing) compute infrastructure. If you’re looking to build at even moderate scale, I would encourage you to check it out and get involved. You can see most of our nodes over at AsRockRack, an OEM partner.
#1 Building digital infrastructure is a team sport
I like to compare what we do to fielding a professional football squad or performing a Beethoven symphony. It takes a wide variety of individual experts working together as one coordinated team to do it right. Of course, you can muscle your way through a build here and there, but to do something that lasts — and to do it globally, consistently and at scale — takes building a great team and culture that can perform in the face of unexpected challenges.
From having the right “boots on the ground” who understand your design and process to partnering with network engineers, SRE’s, and procurement, it really does take a village.
Bonus: Murphy’s Law really is a law
If it can go wrong, it will. Cables will go missing, people will get sick, shipments will get delayed, and bugs will be found. Do what you can to prepare for the unexpected, build relationships with your key suppliers and partners, and manage expectations up the chain.
You’ll be glad that you did.