How did our society’s need to 'Netflix and chill' lead to a revolution in hardware networks? Eliminating the need to drive down to your local Blockbuster Video just to rent a movie took a miracle of hardware and convoluted networking…and a few motivated geniuses to figure it all out.
In episode 3 of the Traceroute Podcast, Dave Temkin, Ingrid Burrington, Jack Waters, and Andrew Blum join us to discuss how the physical internet works. Detailing the hidden infrastructure involved in getting computers connected around the world, they shed some light on the fact that, contrary to what digital natives might think, your connection to the World Wide Web isn't 100% wireless.
What was most important to me was knowing that when you wanted to stream stranger things that you could just sit down press play and it would work. You didn’t need to be a computer genius to fix it. What was important was to have big fat pipes to deliver a lot of streaming video.
the question of where the internet lives is such a hard one to answer in so far as the internet is always moving, that’s the whole point. It’s disbribution is the entire purpose]
This is Traceroute [rowt]...a podcast about the inner workings of our digital world - all the PHYSICAL STUFF that most of us never have to think about. In a world that is increasingly defined by digital…. We look at the real people and services building, maintaining, and scaling the internet.
I’m your host, Grace Ewura-Esi. a technical storyteller and product evangelist at Equinix. In this episode: NETWORKS.
[Act I: NETFLIX]
when I got there, our primary business was shipping DVDs. You know, we were shipping millions of DVDs at that point. // And we always knew that streaming was going to be the future// It's not a coincidence that the company was called Netflix. the intention was always to deliver it over the network. We just needed to feel that the network was ready.
That’s Dave Temkin, who recently left Netflix after over a decade at the company. His last job was vice president of networking systems - And he built one of the biggest content delivery networks in the world - serving over 200 million customers.
The Netflix that we use today - the one that we expect to provide us with hours of uninterrupted streaming content, day in and day out - took years to build. The content is important - but it wouldn’t have happened without strong networks. 10, 12 years ago, we're like, all right, the network is clearly advancing// to a point where we think we could have a business by delivering over the Internet now one. And so as we looked at our infrastructure//we've got servers sitting in a data center in Sunnyvale and that's running that whole DVD shipping business.//Looked at this and said, hey, we need to figure out how to build an infrastructure that's scalable that we know can scale to the point of, you know, 200 million plus and, you know, eventually 500 million or however many customers without breaking the Internet Who knew that the servers sitting in one data center running their DVD business would become a massive global network. Netflix - and others - succeeded because they solved a major infrastructure challenge. And the key to streaming entertainment as we know it today - in all its binge-watching glory - is networks.
[ACT II What is a Network]
A network in its broadest sense is a group of overlapping and interconnecting things. In the case of the internet... Temkin: You know, we could be talking about a virtual network that's not actually physically tied together. We could be talking about a large physical network that's tied together via submarine cables around the world. Or we could be talking about, you know, your home Wi-Fi. Most of us ignore how networks are built, or at least don’t understand the process. Temkin thinks about that a lot.
Here in Hoboken, New Jersey, for example, I can look up and all of our utilities are aerial//and I try to figure out what's where, you know, power is obvious. And then I look down and. You see, OK, there's a cable line, and then I look and I see there's Fios and then I look and I see there's a bunch more fiber optic cabling out there. And I realize that, you know, the Internet, this decentralized giant network, literally important parts of it, traverse directly in front of my house. [00:52:57][42.0] [00:52:58] And no one really thinks about that. No one thinks about how ubiquitous this network is. They just expect it to work.
And like most kids that I've interacted with still think the Internet travels through space. Right. They think it's satellites because they've. Wi-Fi their whole lives, right, and the idea of it being wired totally blows their minds] Ingrid Burrington is a journalist who writes about tech, data centers and networks. She grew up in Silicon Valley. And one of her pet projects is to look deeper into the infrastructure that makes up the internet - and find a way to make that information meaningful to people outside of tech.
there is also an element of it, //like it's kind of missing the kind of central draw or appeal of the technology, right? Because like, the thing that's cool about the Internet is that it like kills space and time. Right. //The whole point is you don't want to know about all that space and what's going on in between it. Like, that's not why you're here.
Burrington thinks it’s super important that more of us understand how the internet works so we can recognize it as the public resource it is
Burrington [the absence sort of understanding, the understanding of how a lot of this stuff works makes it a lot harder for people to conceive of it as potentially being a public resource or asset or is something that sort of has a responsibility to be part of the Internet. Like, I think the fact that we even talk about it as infrastructure is very telling. Like we don't call it public works. Right. It's not like the Hoover Dam or whatever. It's it is a it is an asset that's like infrastructure is this weird, fuzzy word that that's sort of come to mean a lot of different things]
In the case of network infrastructure….
We’re talking about data centers, towers - and the wires, cables, and fiber that connect them.
After the government relaxed regulations in the 90s - there was a big wave of infrastructure development. Commercial investment wasn’t just about websites - but all the other things you needed to make the internet run.
There was a big push by a lot of people to jump into the network market
[00:22:55] And you saw a lot of people either with connections to or ownership of right of way getting into it. I think my favorite one of these is the Williams companies, which is an oil and gas company. They started building out fiber networks basically using unused no longer operational oil and gas pipeline routes because they'd already negotiated like, oh, they own the land and had gotten all the leases to run that cable. So it was really easy to run a bunch of straight lines across parts of the Central West and stuff. And you had a lot of people doing that]
Developers came in and built tons of fiber networks - actually too many. Demand wouldn’t catch up for a while.
A lot of the network infrastructure built at that time - especially the fiber - is still in use today. But the only network that most of us give any thought to is the one that we use in our homes - our personal wifi networks.
but the router network that's in your house is connected to like the larger networks of the Internet just through a cable. Right. So the cable that like kind of leaves your house and depending on where you live is going to maybe be going across the telephone line or it's going to be like going underground like that eventually. All kind of ends up in some exchange point] To get the internet to our computers and phones - it’s not just one network, but dozens, if not hundreds. That actually builds in durability.
there is a resiliency kind of built into the way that Internet networks function in that it's not just like one single cable that gets cut and everyone loses their Internet access.
That resiliency of the internet isn’t just because of the way it was built - It’s also because multiple companies are involved in the infrastructure space, increasing access for more and more people. More on what it takes to create those kind of resilient networks - when we come back.
[Act III Level 3 Levels Up]
So let’s dive deeper into what exactly it takes to build a network - in this case, from SCRATCH.
My name is Jack Waters, I have been hanging around the Internet since the mid 80s, which brings us a long time ago. I was a college intern at one of the first sites on the Internet in 1985, to be exact] Waters felt the internet could open up the world, if it were built the right way.
that notion of boundaryless communications was something that struck me as, wow, if this really takes hold, it's going to change the way we communicate.
Waters was interested in networks and how to build them - which is what drew him to the telecommunications company Level 3.
the plumbers like me, they were the folks that that went and dug up the ground and through conduits in the ground and through fiber in the conduits and hooked up optical equipment to the fiber and hooked up routers to the optical equipment and all that, to most people that are even technologists in the Internet today is transparent to them. They don't understand that. For the people and the companies building the networks of the internet, they had to know about a lot more than code.
there are very physical things. There are fibers and and switches and routers and those things, and they get interconnected. But the the technologies that sit inside that equipment, they have evolved over time. And there were there were huge debates early on in the 80s and 90s whether, you know, the Internet technology could become as scalable as the traditional phone network, which is laughable when we think about it today] At this point many experts were still skeptical about the long-term profitability of the internet. But Level 3’s founder Walter Scott got serious about building networks after he attended an exclusive meeting in Ireland in 1995. WATERS there was a meeting with Bill Gates, Warren Buffett, Walter Scott.//And and Gates got up and talked about, if you aren't investing in Internet infrastructure, you've missed the boat] After that, Level 3 started working towards a clear goal. WATERS [we wanted to build something from scratch. We wanted to go out and construct a network. We had lots of folks who had construction backgrounds. So actually go and dig up the streets and sides of railroad beds and and lay fiber and conduit and the ground, the whole idea of this notion of continuously upgradable network. So we would get up in the morning and go to sleep at night thinking about how do we build something that can be continuously upgraded] And building infrastructure really means thinking about the long term. [dream voice] WATERS [if you dig up the ground once, don't just put a cable in it. Maybe you put pipes in the ground and then you can pull cable through the pipes. That's one very simple, simplistic view of continuously upgradable. So we didn't have to go back and construct again] But Level 3 still felt pressure to build smart - AND build fast. - it wasn’t clear how long the buzz and the money would last. [host voice]
We started in and I started in November of 97 and our network was up and running 32 months later or something. So so we really moved quickly. We built 16 and a half thousand miles of network in the United States and 3500 miles of network and in Europe in 30 months] Before that point, the internet was largely running on legacy phone networks.
Well, they're the only networks that have been built were the original AT&T network, which had been built in the 70s, maybe a little bit in the 80s. Sprint and MCI built fiber networks in the in the 80s. But then there had been huge advancements in fiber technology that the actual glass had improved quite a bit. [00:29:56] you go back and look at Sprint's commercial, the pin drop// the pin drop talked about having a fiber network and how great the voice quality was on. It was so good you could hear a pin drop, but they just didn't foresee the Internet and data communications and what could be done]
Today, bragging about the clarity of a voice over a connection seems basic given how much more we can now do… But at the time...
WATERS [the thought of carrying a television show over. [00:30:23] Internet technology never happened. [00:30:27] I sat with many people in the in the 90s and even in the 2000s, it said it was impossible, would never happen. And I know I would always say, all right, well, I don't think I believe that and certainly wasn't the case.
Driving hard to build Level 3’s network - to use all that cash floating around - turned out to be a good move. Because the bubble did eventually burst. [serious host voice]
we were all just trying to stay in business and stay afloat and we had lots of cash, which turned out to be great. [00:34:48] And and that was a time when other companies struggled. And so at the beginning, we started buying companies. // [00:35:07] And, you know, it was it was probably that we were ahead of our skis a little bit when it comes to demand and when the demand was going to show up. You know, I mentioned that the the the smartphone you know, the smartphone happened in 2007, 2008. You know, we had built a lot of network that was waiting for that kind of demand for five, six, seven years. And we certainly probably missed a little bit on timing. We were right in the long run.
Over time, Level 3 grew and consolidated. By 2017, it was acquired and merged into another company. But its legacy remains - it helped play a huge role in building out internet infrastructure, in creating powerful new networks that we use day in and day out. [hosty-dreamy] The internet has changed dramatically - cloud is now all the rage. And the smartphone helped reshape the landscape. [hopeful hosty]
I mean, the whole idea of having this device that sits in your hand and has the compute power in the screen and the ergonomics and the usability, I mean, it just has changed everything.// I do think it is probably one of the most important things that's ever been developed for mankind. And I think it's changed everyone's lives for the better, even though there are many downsides that that we're navigating through
Waters says that as the industry morphs, there will always be power struggles and concerns over consolidation and equal access. Growth will come in fits and starts, bubbles might burst. [regular host]
WATERS [But take a step back and remind yourself, how old is the smartphone? [00:46:54] Fourteen years. I mean, it's staggering. [00:46:58]
It really is unbelievable when you think about it's only 14 years old. Competitive telecom is only 35 years old, so, I mean, we're I know we we love to say how old stuff is and think that it's been around forever. But I I believe that these types of changes that for society are actually really new. And we're going to fumble through how to deal with many things for our families and our business and all that. //[00:47:29] But these are the beginning and not the end, in my opinion.// [00:51:42]
And we’re also at the beginning when it comes to investing in infrastructure. WATERS even if we stay at the current growth rates and there are many arguments that could go well beyond the current growth rates, the amount of infrastructure needed is measured in decades of investment. It's not measured in a couple of years. It's not measured in the next 10 years. It's measured in the next 50 years. // So, you know, there's a lot a lot of cool things that can happen. And that's what I'm very excited about.
[Act IV Connecting the Tubes]
I think it's fair to say that we are constantly connected to networks and sort of every moment of our contemporary lives, //despite the fact that the Internet all comes out of the same screen, every time we look at that screen, there are layers and layers of networks serving what we're seeing and hearing and experiencing. Andrew Blum is a journalist who wrote the book “Tubes: A Journey to the Center of the Internet.” BLUM: the metaphor I like to use most is of a kind of pile of fishing nets. So you have many networks around the world and it's not as if they're sort of independent in place the way a road network might be. But in fact, the networks of the Internet are often piled on top of each other where one network rides upon another network. And the result is confusing. There are many layers of ownership. There are many technical layers of how networks fit together. But fundamentally, a network means that we can connect to each other across distance and we can expand those connections constantly] 8 And we should be clear here about what we mean - the specific physical building blocks we are talking about - when it comes to internet networks.
There are the data centers where data is stored and processed. There are the Internet exchange points where networks physically connect to each other. And then there are kind of the lines in between, whether underneath the ocean or in cities or across the United States that connect all of the other pieces and most importantly, connect them to us.
So far, we’ve talked a lot about those lines. But the points of interconnection - the internet exchange points - also have to be built and operated the right way for the internet to continue working.
an Internet exchange point is a building, a place where networks physically connect to each other with the networks of the Internet, with often with a yellow fiber-optic jumper. Cable are connected from one physical router inside a steel cage to another physical router inside another steel cage somewhere else in the building. They are kind of groups of networks that have banded together to share a physical location in order to make it as easy as possible for the many networks of the Internet to physically connect to each other.[00:08:07]//Internet exchange point is a very particular sort of special flavor of data center, Infrastructure companies had to spend a lot of money to build - and many of them didn’t make it.
when you were digging a trench, you know, across the west, the expensive part was the trench. The fiber was cheap enough that you could put a lot of it in the ground. And what that meant was that you had this incredible glut of capacity that really kind of fixed this infrastructure for for at least a decade. //[00:15:36] even though there was this, you know, incredible infrastructure boom, it was a horrible business. They all went bust.
The early cash influx into the internet - despite the crash that followed - allowed for infrastructure development that we still use today.
BLUM backhoes and and bulldozers and hundreds of millions of dollars to build the networks. But once those networks were built and once you kind of paid to access the Internet, then it appeared free. It seemed limitless. And I think that, you know, that had a huge impact on kind of laying the groundwork for the way that we use the Internet today [00:20:06] And those initial infrastructure investments - paved the way for the kinds of services we now use with the cloud and on our smartphones.
BLUM [there was really a kind of more capacity in the network than we knew what to do with at the time. And it was that kind of access that allowed, you know, for the kinds of companies we saw emerge in the in the mid 2000s, you know, like like YouTube, like like Netflix with
with streaming, like the, you know, Apple iTunes store. And without that sort of initial infrastructure push it's hard to imagine that all of the consumer pieces of the Internet that we now live with and beyond would have been able to to to to rise because of that excess capacity] Like others, Blum worries a bit about the consolidation of infrastructure ownership in today’s internet environment. As more and more people use networks, eating up more bandwidth - the industry has shifted.
One of the one of the really one of the remarkable things the last 10 years is the way in which the kind of wonderful agglomeration of tens of thousands of Internet networks that all rely on each other and interconnect with each other has slowly been replaced and kind of replaced from the inside by the internets biggest companies / [00:25:12] So you now have, you know, by some accounts, essentially two thirds of the Internet belonging to Facebook, Google, Amazon, Apple, Microsoft with and all of that kind of the the agglomeration that used to allow for so much innovation and spontaneity is now kind of pushed to one corner of the Internet. And you see that now in the physical networks as well.there are, of course, advantages to the way in which, you know, Google's network and Facebook's network and Amazon's network all work so smoothly. But when you compare it to the kind of early, earlier era of the Internet, it is clear that the scale of these companies really changes and limits the ability of others to interconnect with each other. [00:26:25][35.9
About six years ago, companies like Google, Amazon, and Facebook began to invest more in their own international networks.
BLUM: just as Amazon has decided to buy airplanes to move its packages around the country, it has made more sense to Google and Facebook to build and operate their own global networks rather than riding on top of the existing telecom networks.// [00:40:41]
As a result….. BLUM when the Internet grows now, it grows according to the terms that the giants of the Internet set for it] Pressure has only grown as the cloud has exploded. [dream host]
what we've seen is the growth of these kind of massive cloud companies.when you have that kind of global network, you then need a place to connect to all of the other regional and local networks. And so// you end up again with this pile of fishing nets where you have Google's network, Google's network and Amazon network.And they have this glorious moment where they, you know, exchange data faster than you can ever imagine because they are connected in the kind of backplane of the Internet] So yeah, there’s the benefit of increased speed and efficiency, says Blum.
If you're a kind of small company that's coming up with a new idea and don't have to build your own data center, you can spin it up quickly on on Amazon's cloud or on Microsoft's cloud. But what it has meant for the Internet more broadly is this sort of take you take over from the inside of the global networks themselves. Blum argues that one thing that counteracts that is the existence of neutral exchange points.
the role of Internet exchanges//is to provide that physical place for the regional networks to connect to the giants of the Internet for the, you know, academic networks, to connect to the corporate networks, for the telephone companies to connect to the data center companies. And we still need those physical connections.
It's hard to imagine networks connecting at Google's house. You want a kind of neutral exchange point where these connections can happen,
Because I need your traffic, you need my traffic, and we need a physical place for that connection to happen. [00:34:19]//[00:53:00] interconnection points in such a crucial role in allowing small networks to be on the Internet, you know, to sort of exist and exist independently and not have to rely entirely on the services that the Internet giants provide.
How does an interconnected and open internet affect us, the users? Blum gives the example of Otter - a transcription service.
when you want to upload something, an audio file to order to be transcribed by their I you can sort of just upload it in
the browser. But if it's already on your Dropbox, you tell order to grab it from your Dropbox and a huge file instantly transfers over. And that's this kind of amazing example where the interconnection between Otter's servers and Dropbox servers, which in fact may live on Amazon's cloud or Google's cloud, allows for the sort of transport of huge data, not through your computer, but within the clouds, within among different clouds that are directly connected to each other Which brings us back to the kinds of companies we interact with every day.
fundamentally, the growth of the Internet, you know, in the in the era up until 2015 was all about the tens of thousands of networks interconnecting with each other. And as we have demanded more of the Internet, whether watching gigabit for 4K video streams on Netflix or using Google Docs or Zoom, you know that you know, that require very stable, very low latency bandwidth the companies that are best at delivering it have done the best. Companies… like Netflix.
One of the reasons that Netflix has done so well is because people like Dave Temkin and others were so incredibly successful at giving high quality video streams in a way that just works. And there's a whole lot of magic going on behind the scenes to make that not stutter. I mean, there was an era I remember I wrote about it 2009, 2010, 2011, when you couldn't watch Netflix on a Saturday night because the network in my neighborhood couldn't handle it. That is rarely the case. Now, we've kind of solved that problem. But in the process of solving that problem, we shouldn't lose sight of the fact that we have replaced an Internet of many tens of thousands of networks with the Internet that is unequivocally dominated by a handful of networks
[ACT V THE IMPORTANCE OF NETWORKS INTO THE FUTURE]
So where does the internet - and the growth of networks - go from here? Dave Temkin, the former Netflix VP, says he’s a firm believer in a decentralized internet...
it's very rare that we hear about one company buying another company in the network or data center space and go, oh, that's amazing. Thank thank heavens that happened. It's usually like, oh, are we going to get the worst of both of these or are we going to, you know, maybe maybe one good one bought one, not a great one./ [00:08:11] But in general, the decentralization and the fact that there are multiple players here allows for technical innovation that we otherwise would not have seen. It allows for for price flexibility we may not otherwise see. competition in any of these spaces is good, you know, for the Internet. It's good for customers. It's good for everyone. [00:08:35][24.2] For huge data streaming services, decentralization of internet networks can be really helpful.
when you're delivering hundreds of terabytes of data, terabytes per second of data? Having all that delivered from one central place is completely untenable, if not impossible. But if you decentralized that in such a way where you're deploying that from hundreds of thousands of places, you have this extremely complex system that breaks in, you know, in every possible dimension.
Temkin remembers the challenge for Netflix when it wanted to shift away from shipping DVDs - to streaming movies and tv shows over internet connections. They did it with the help of neutral exchanges.
It was, hey, how do we get this? How do we get a proof of concept? Stood up to say, hey, we can build this global delivery network and how do we get users migrated to it? And I remember, you know, when the hardware team delivered our first server to go into an X location to putting that in the trunk of my car and driving it to Equinix, we won. And plugging it into a router that had just been waiting there for it, and I remember like getting the first bit of traffic onto it and just the elation of that and going, OK, great, we got that working. Now, how do we scale this?]
To prevent their sites from crashing, content providers like Netflix need reliable, strong networks. That means they can’t rely on just one provider. They need to work with other tech companies, as well as traditional cable companies and more, to deliver the highest quality streaming to their customers. Finding the right colocation points is also important.
temkin[00:19:49] we're looking for a place where we can interconnect with dozens or hundreds of networks in a single location. Where can we find the best power density or where can we find the most space is where can we find the most networks? And then how can we shoehorn our infrastructure into that? As we’ve mentioned before - a lot of the technology was already there.
we've seen tremendous growth. You know, how much data we can pack on a single strand of fiber. What I always think about what always gets me is that we have fiber that was lead 20 plus years ago in the ground that is still heavily in use today, that we can still squeeze more and more data out of. Show me some other piece of technology aside from maybe copper wire that we've been able to do that with without digging up the ground]
For businesses like Netflix to grow - innovation was born from the need of efficiency...
Temkin //complexity is money. We we you know, the more employees we have to maintain more physical locations, it's complexity. The more complex our network is, the more likely it is to break in, the more support we need for it.
And it was also about managing relationships and strengthening partnerships.
what was most important to me was knowing that when you wanted to stream stranger things, that you could just sit down, press play and it would just work and you didn't have to be a computer genius to fix it//[00:29:14] the thing that I really thought, you know, was was super important was that you could be on optimum, you could be on or whoever they are now, Altis, you could be on Comcast, you could be on Fayose, and the service should just
work/[00:29:34]//we went through a lot of growing pains// mostly, though, from a relationship perspective, the infrastructure was almost always stable. we continued to grow it horizontally and vertically. But we definitely, you know, felt like from the very beginning we built this scalable, stable architecture that that we could support, and as we needed to reach new markets or new ISPs, we could literally just, you know, press, you know, copy and paste to build build that infrastructure over again in another metropolitan area.
Managing those relationships meant keeping access open - keeping the pipes flowing.
that was a lot of the battle that we faced from day one of fighting upstream on how can we do this? How can we encourage networks to not have caps that will put us in a position where users won't want to use our service because they can't without it costing them a fortune? And how do we encourage ISPs to deploy better, more efficient infrastructure?
As Temkin moves on to his next role, he reflects about the impact he and his team have had on the evolution of Netflix - and the industry.
TEMKIN In the last quarter of twenty twenty, we shipped from if you measured the height of the number of servers that we shipped ISPs to enable the service, it was to Statue of Liberty is tall we shipped enough of those to do that.//we shipped I think it was one hundred Teslas worth of weight of servers. TEMKIN so, as I think about all these awesome things that the team has done and all these technical contributions that we've made to make the Internet better, /I remember when we were at a million customers and when ten million seemed heady. And I remember 2010 when we were 10 million and 20 million seemed ridiculous. And now here they are, you know, 20 times that and it's not stopping. [00:51:47][79.3] As the internet continues to grow, Temkin says it’s important for people to not forget all the building blocks that make it work. The physical investments that make the network run.
Technology infrastructure is something that has been silently overlooked since the dawn of communications. No one you know, if I ask my parents when they pick up their phone, whether it's a landline or the cell phone, what happens when you text someone? What happens when you call someone they couldn't tell you? [00:43:41] And arguably, if you talked to a developer whose building games or even building a streaming service and you asked the same question quite often, they can't answer it either.// People like the allure of things they can touch. And this infrastructure is not stuff that you can touch. It's not an app. It's not, you know, a widget.We will continue to see this underpin all of the technological advances that we continue to make in the world. Everything the fact that we've got, you know, a Covid vaccine that, you know, in reality took years to design, but only weeks to produce is because we have this giant computer infrastructure that they were able to leverage to model this and to make it work and then to collaborate with their partners all over the world in getting this out. And if we didn't have that infrastructure, it would be like, you know, we would be years and not measured in weeks, months or days. //I think we will just continue to shorten the innovation cycle as that happens And it's just give me a net good for society as a whole.
Next time on Traceroute...
Thanks for listening to Traceroute, an Equinix. Our theme music is by Ty Gibbons. Be sure to subscribe on Apple Podcasts, Spotify, or wherever you get your podcasts. You can learn more by heading to metal dot equinix dot com slash traceroute. Want to get in touch? Reach out to us at [metal dot equinix dot com]. And make sure to leave us a review and tell us what you think.