The event was conducted by Software and Architecture Modeling (SAM) Special Interest Group of the SDForum.
He made the presentation livelier by sprinkling interesting questions about Netflix throughout his presentation, and awarding a copy of the book titled Host Your Web Site in the Cloud to whoever in the audience answered a question with the greatest accuracy1.
Hien's presentation involved four sections (Slide #2):
- Why Amazon cloud? The answer, as you would expect, is that "Netflix is not in the business of creating and operating data centers", even though they currently have 2 data centers for their private data, and the Amazon cloud proved to be an effective platform for hosting the Netflix streaming solution.
- Tour of AWS. This tour was a very useful introduction to AWS, particularly since the later portion of the talk makes use of several of the AWS components in creating the streaming solution.
- How Netflix leverages [the cloud]. Netflix uses ~4000 EC22 instances during encoding movies into digital format for streaming, typically a 15-day exercise, while ~1000 EC2 are normally deployed for servicing subscribers. The storage is of the order of petabytes. The storage for movies, built on AWS S3, has a rather large latency — 300 mS — and hence the S3 is front-ended by Memcached, and the resulting latency is brought down to about 5 mS. (Slide #30). Slide #32 highlights the storage usage: e.g., ~800M objects in SimpleDB, ~17M objects in S3. Slide #33 brings out the importance of some key features that would be desired in a cloud, but were missing from AWS, as Netflix prepared their implementation; Memcached is one of the missing services.
- Best practices. In this section, Hien highlighted the need for automation — Can you imagine setting up 4000 machines by hand? —, minimizing latency, ensuring storage persistence, instrumenting every step of the call, sharding, etc.
There were several announcements made at the beginning of the session:
- SAM SIG & Java SIG will be combined beginning January 2011
- Jian Zhang of GridX announced they have openings for software engineers — Hadoop, Java, Front-end UI programmers — in building a cloud based financial application for the Smart Grid.