The year 2020 was always going to be a big one for Disney’s Direct-To-Consumer and International team.
At just two years old, the organization — also known as DTCI — encompasses the entertainment giant’s streaming services and international media businesses, employing nearly 3,500 technologists, developers, engineers, data scientists and designers in more than 20 locations worldwide. The team spent the early months of this year planning for some major milestones, like the launch of Disney+, its flagship streaming service, across western Europe in March and the livestreamed broadcast of the NFL Draft in April.
But the world changed. COVID-19 spread across the globe, sending entire societies into lockdown and forcing many organizations, including teams across DTCI, to pivot from carefully laid plans.
Fast forward to today, and the company is celebrating the success of those two projects, both of which were orchestrated under unprecedented conditions by almost entirely remote teams. Disney+ recently surpassed 54 million subscribers, while the NFL Draft drew in more than 55 million sports-starved viewers over its three-day broadcast.
To learn more about what it took to achieve these milestones, Built In caught up with the chief technology officers behind them. They walked us through the techniques and technologies their teams used to overcome extraordinary circumstances.
Swimming Upstream
After launching in the United States, Canada and the Netherlands in November 2019, plans to roll out Disney+ throughout western Europe in March were interrupted by the coronavirus pandemic. DTCI could no longer send technologists to Europe to oversee the launch as originally planned, opting instead to hire and onboard locally based QA teams on extremely short notice.
But mobilizing a local team wasn’t the only challenge they faced. Internet usage spiked as people across Europe sheltered at home, and introducing a streaming service as anticipated as Disney+ threatened to topple an already stressed broadband system.
“As stay-at-home orders went rapidly into effect, governments across Europe grew concerned about how the expected high consumer demand for Disney+ would impact the broadband infrastructure already seeing a surge in demand with video conferencing, gaming and distance learning,” said Joe Inzerillo, the CTO of Disney Streaming Services. “In some markets there was concern that launching Disney+ could actually break the internet.”
In a detailed interview with Built In, Inzerillo described the rollout as a brand new challenge for his veteran team, which has previously launched ESPN+, along with streaming services for HBO, MLB and WWE.
What did your original plans for the launch of Disney+ in Europe look like, and how did the pandemic affect them?
We spent a significant amount of time setting up the local infrastructure required to deliver a high-quality experience for consumers as we launched Disney+ across Europe, including establishing local co-location facilities and ramping up data centers to cache content regionally.
As part of our planning, we also intended to send a large cohort of technologists to Europe to support the launch, as a lot can happen in a live environment. Having boots on the ground to QA in real time as we’re deploying the app across platforms is a critical component of a launch. However, the pandemic made travel impossible and going in without our folks on the ground posed the biggest threat to our launch.
How did you work around not having QA on the ground?
We are fortunate to be a part of a global organization with The Walt Disney Company, and we’re able to tap into local resources in the markets to pivot quickly, train local employees and onboard new partners in record time to serve as our remote ears and eyes during deployment.
We have an incredible team that pulled together this entirely new virtual QA program on the fly and delivered through adversity to solve the problem quickly, so we were able to move on to the next challenge.
“We were able... to instate market-to-market bandwidth reductions and successfully mitigate any disturbance to critical services.”
What experience does your team have in building streaming services?
We built and launched the ESPN+ streaming service in 2018, which marked the first scaled direct-to-consumer streaming service for The Walt Disney Company. ESPN+ runs on many of the same technology underpinnings powering Disney+ today, so it provided an opportunity for us to iterate on our tech stack and product features in a live environment, including our personalization engine.
Prior to that, many of us on the tech team came over to Disney through the acquisition of Bamtech, where we were early pioneers in streaming, having supported our first livestream of an MLB game in 2002 and going on to build many early streaming services with MLB.TV, Eurosport Player, PlayStation Vue, HBO Now and the WWE Network.
Aside from the restrictions on travel, what other challenges did you encounter?
The pandemic created another late-breaking challenge for us just days before the launch of Disney+ in Europe that no amount of prep could have prevented. As stay-at-home orders went rapidly into effect, governments across Europe grew concerned about how the expected high consumer demand for Disney+ would impact the broadband infrastructure already seeing a surge in demand with video conferencing, gaming and distance learning.
In some markets there was concern that launching Disney+ could actually break the internet, so we proactively instituted measures to lower our overall bandwidth utilization by at least 25 percent ahead of our March 24th launch, and delayed our launch in France by two weeks.
To do so, we needed to re-engineer the product quickly in a way that wouldn’t adversely impact new subscribers experiencing Disney+ for the first time. Through hard work and ingenuity, we were able to work with our content delivery network partners to instate market-to-market bandwidth reductions and successfully mitigate any disturbance to critical services. On April 8th, we announced the service had crossed over 50 million paid subscribers, a truly humbling result for our team.
Disney Gets Drafted
With coaches, players, managers and executives all stuck at home through April, the organizers behind the NFL Draft made a big decision: conduct the entire event virtually. But what could have devolved into a video conferencing fiasco became the most-watched NFL Draft in history, with the event’s broadcasters — a joint effort between ESPN, ABC, the NFL and DTCI Technology — committed to telling the stories and capturing the reactions and celebrations fans have come to expect.
The contingency plan involved shipping out cameras, tripods and DIY production equipment to more than 100 households across the country, with instructions for player prospects, coaches and officials to set up their own home “war rooms.” Behind the scenes, DTCI Technology was aggregating incoming video streams, conditioning media to provide consistent video and audio quality and delivering the product to network broadcasters and digital platforms.
DTCI Chief Technology Officer Aaron LaBerge said organizing these disparate video and audio streams was the most difficult piece of the entire operation.
“We had to develop workflows to manage more than 180 different video feeds from more than 100 locations, and do it with the vast majority of our team working remotely,” he said. “In the first week of March, we were planning for a remote production in Las Vegas. Six weeks later, we were delivering one of the most complex — and successful — events we’ve ever had.”
LaBerge, who stepped in as DTCI’s first CTO in 2018 after years leading technology efforts at ESPN, leads a global team responsible for the design and development of 200-plus consumer-facing digital products across Disney’s brands, which reach more than 350 million users monthly. The organization also handles worldwide digital and linear media distribution — including the digital supply chain feeding content to Disney+, ESPN+ and 350 television channels globally — as well as advertising technology, data platforms and broadcast engineering and operations outside the United States.
LaBerge explained how his technologists were the linchpin in the plan to broadcast the first-ever all-remote NFL Draft.
Walk us through the challenges your team faced in helping organize this year’s NFL Draft. What was it like?
The NFL Draft is one of the most dynamic sporting events every year because of the complexity of producing a live event, with reactions from players and families, the often-humorous die-hard fans in attendance, the constant highlights and analysis of each pick. This year, we had to adapt because an in-person event was impossible, but we wanted to deliver a similar personal experience for players, fans and teams.
Our team partnered with ESPN and the NFL to rethink the production of the NFL Draft to deliver “live sports” again for the first time in weeks. To do that, we had to develop workflows to manage more than 180 different video feeds from more than 100 locations, and do it with the vast majority of our team working remotely. Plus, rethinking the entire process for managing an event like this had to take place in a matter of weeks. In the first week of March, we were planning for a remote production in Las Vegas. Six weeks later, we were delivering one of the most complex — and successful — events we’ve ever had.
Not to be forgotten, our team then had to deliver the show to all of our ESPN and ABC distributors, to NFL Network — which simulcast the ESPN coverage — and to the ESPN App and ESPN.com for those fans that were streaming it via TV Everywhere authentication.
“The 2020 NFL Draft was a success in every level — a one-of-a-kind event and a historic broadcast. I’m proud of our ability to deliver sports to fans with innovative technology solutions to complex problems.”
How did you go about building an entertainment experience that went beyond a massive virtual call?
DTCI Technology provided the tools, video management and distribution for the ESPN production and teams, and worked hand-in-hand throughout the event to execute the best possible product. Despite the new challenges and numerous complications, there was no shortage of material or video to enable ESPN to tell the best stories throughout the Draft.
DTCI Technology’s Transmission team in Bristol, Connecticut served as the central aggregator and manager for more than 180 distinct feeds coming in from locations all over the country. That was all done with a small team on-site, while introducing new and necessary social distancing precautions. We leveraged a combination of DTCI and NFL Network workflows across multiple technologies, including fiber, satellite, AWS, Secure Reliable Transport (SRT) and IP technology. Our Transmission team was responsible for acquiring the source feeds and conditioning them to provide consistent sound and video quality to enable ESPN’s production and operations teams to create a never-before-done kind of show.
On top of all of that, a different part of our team, also working from home, was hard at work making sure that the digital experience around the NFL Draft — which included our live Draftcast tracker, video clip packages of top prospects, editorial coverage, authenticated streaming, a new Fantasy Pick’Em game and more were flawless for those who weren’t watching the TV telecast but wanted to follow the event.
How did you communicate with coaches, players, managers and executives to prevent glitches?
The most complex problem to solve for this “virtual” Draft was capturing the remote feeds from player prospects, NFL coaches and team personnel. The NFL distributed more than 100 at-home production kits with professional lighting, HD camera devices and necessary software to securely stream into our Transmission workflow. Each prospect’s home “war room” had an ‘always on’ wide-shot camera that created a virtual green room feed, along with a separate camera set up for a tighter interview shot. The combination allowed production to drop in live to capture real-time reactions and conduct interviews.
The live three-day event was a symphony of DTCI Technology, ESPN Operations and ESPN/NFL Production working together closely while in constant communication. The teams leveraged RTS intercoms and the Unity Intercom app to create a proxy for our on-site communication systems that ensured close connection between Technology, Production and Operations. We also used some consumer technologies like Skype and FaceTime for communication between talent, producers and board operators.
Talk to us about some of the technical challenges your team encountered.
The biggest challenge was just the complexity of developing workflows that needed to be put in place under tight timelines. This was a technical masterpiece of execution with the integration and deployment of new workflows, communications methods and new workplace environment configurations.
Our delivery leveraged multiple workflows, using multiple types of technology and multiple levels of aggregation of incoming feeds and returns. Interview feeds were transmitted from the at-home kits to one of three NFL Network vendor sites and our partner DTAGS in Tulsa. From there, they moved over managed fiber to Bristol. Meanwhile, the ‘always on’ wide shots went directly into DTCI’s Bristol Transmission hub via AWS using an SRT protocol. To make that possible, a significant reconfiguration of our IP Gateway in Bristol was required of the technology teams.
To ensure social distancing, production and operations teams were spread out across four control rooms; two for ESPN and two for ABC. Technology personnel on site at our facility represented a fraction of what it would have been normally, but we had more than 60 team members working remotely, helping to manage the Draft as it unfolded. That meant enabling remote access into the tools and systems for managing transmission was critical.
However, part of why this was possible at all is that we were already solving for that. That was a process that began on day one of the COVID-19 era, when we had to support the 24/7 operation of networks and shows in a largely remote environment. The NFL Draft, though, was taking that to a much greater scale and complexity.
What did you learn from this experience that your team might apply to future drafts and similar events?
The 2020 NFL Draft was a success in every level — a one-of-a-kind event and a historic broadcast. I’m proud of our ability to deliver sports to fans with innovative technology solutions to complex problems.
One of the things that we definitely learned is that the investments in technological infrastructure and expertise that our company has made give us enormous flexibility and capability. If we can dream up a new idea, we have a pretty good chance to be able to pull it off.
As a fan, I also think that it helped make the NFL Draft this year a little more personal and intimate. I love the spectacle of the big event, and I can’t wait for that aspect to be back as well. But I think what we did this year gives our producers and storytellers another way to think about parts of how to connect the world to that unique event — and to know that it’s a tool that is now in their technology-enabled toolbox.
