Netflix moving to Amazon Cloud by Hien Luu - A Review.

It is not often that you get to listen to a story that describes how a massively scalable application — as of October 2010, Netflix has nearly 17 million paid subscribers for their streaming solution — has been developed for the cloud, in this case the Amazon cloud, known as Amazon Web Services (AWS). Hien Luu described such a story. He was part of the implementation team at Netflix, and is presently with LinkedIn.
The event was conducted by Software and Architecture Modeling (SAM) Special Interest Group of the SDForum.

Netflix Moving To Cloud

View more presentations from hluu.

He made the presentation livelier by sprinkling interesting questions about Netflix throughout his presentation, and awarding a copy of the book titled Host Your Web Site in the Cloud to whoever in the audience answered a question with the greatest accuracy¹.

Hien's presentation involved four sections (Slide #2):

• Why Amazon cloud? The answer, as you would expect, is that "Netflix is not in the business of creating and operating data centers", even though they currently have 2 data centers for their private data, and the Amazon cloud proved to be an effective platform for hosting the Netflix streaming solution.
• Tour of AWS. This tour was a very useful introduction to AWS, particularly since the later portion of the talk makes use of several of the AWS components in creating the streaming solution.
• How Netflix leverages [the cloud]. Netflix uses ~4000 EC2² instances during encoding movies into digital format for streaming, typically a 15-day exercise, while ~1000 EC2 are normally deployed for servicing subscribers. The storage is of the order of petabytes. The storage for movies, built on AWS S3, has a rather large latency — 300 mS — and hence the S3 is front-ended by Memcached, and the resulting latency is brought down to about 5 mS. (Slide #30). Slide #32 highlights the storage usage: e.g., ~800M objects in SimpleDB, ~17M objects in S3. Slide #33 brings out the importance of some key features that would be desired in a cloud, but were missing from AWS, as Netflix prepared their implementation; Memcached is one of the missing services.
• Best practices. In this section, Hien highlighted the need for automation — Can you imagine setting up 4000 machines by hand? —, minimizing latency, ensuring storage persistence, instrumenting every step of the call, sharding, etc.

Overall, a lively evening, as you got to know how Netflix has prepared itself for paid subscriber growth for the streaming solution.

There were several announcements made at the beginning of the session:

• SAM SIG & Java SIG will be combined beginning January 2011
• Jian Zhang of GridX announced they have openings for software engineers — Hadoop, Java, Front-end UI programmers — in building a cloud based financial application for the Smart Grid.

---

¹One example question was "How many paid subscribers does Netflix currently have?". The answer is 17 million. There were more books than there were questions, and some lucky folks in the audience got the rest of them in the end.

²Elastic Compute Cloud.