So, I just got back from QCon SF 2010 last night. All in all, a very good conference. Rather than write up any kind of extensive summary, I’ll offer up my rapid digest of the major themes from the sessions I attended. Without further ado, here it is in outline form:
- Dealing with data at large scale
- OLTP
- Those who can get away with it are using systems that have more flexible consistency models than traditional RDBMS (CAP theorem trade-offs)
- Most using some form of eventual consistency
- Many sites implementing their own Read-Your-Own-Writes consistency on top of more general storage systems
- These systems must deal with data growth (partitioning data across nodes)
- Must deal with hot spots (redistributing / caching hot data across many nodes)
- Must deal with multiple data centers (some are simply punting on this)
- Twitter and Facebook both built their own key-value stores on top of MySql, Memcache
- Twitter’s solution seemed a little cleaner, Facebook’s a little more crusty
- Amazon S3: also key value store with own caching, replication, consistency models
- This one had the most sophisticated seeming solution for dealing with hot spots
- Those who can get away with it are using systems that have more flexible consistency models than traditional RDBMS (CAP theorem trade-offs)
- OLAP
- Lots of people using Hadoop to crunch offline data
- Good tools for workflow of jobs, dependency management, monitoring are essential
- Quantcast found that EC2 was not adequate for their needs in terms of throughput compared to an owned, highly-tuned cluster, though it has improved over time
- still good to have on hand for surge capability
- Lots of people using Hadoop to crunch offline data
- Operating on the public cloud
- Increased demand for monitoring — and most monitoring tools not built for cloud instances that wink in and out of existence
- Increased demand for fault-tolerance — latency can vary more widely, hardware failures happen out of your control
- Increased demand for sophisticated deployment automation
- Motivation is that you want to use a cloud, not build one
- Capacity planning is difficult when you’re in a huge growth scenario
- Leverage the staffing and expertise of the public cloud companies (Amazon, Gigaspaces, etc)
- Data center is a large, inflexible capital commitment
- Traditional CDNs are still necessary and useful for low-latency, high bandwidth media delivery
- PCI compliant storage in the cloud is not a solved problem
- Serious interest in alternative languages, both on and off the JVM
- There are lots of serious choices available in this sphere (scala, jruby, javascript -> node.js, erlang, clojure)
- Lots of enthusiasm for JVM, less enthusiasm for oracle’s ability or intention to be good stewards of it
- Though there were many very good sessions, especially in the Architectures You Always Wondered About track, in terms of sheer rock-star appeal these two presentations appeared to be the standouts that had everyone talking:
- OLTP