On the Road Again with Microsoft Research

Microsoft Research had its annual "Roadshow" event to demo all their new ideas. After the usual introduction in the traditional Klingon style (how big they are, how many battles they've won and enemies they've killed, and so on), Jim Gray said a few words about "TerraServer".

TerraServer deals with the issue of large datasets. It is a 20 Terabyte data source (of USGS images) handling 10 million hits per day as a web service. On the insides, it uses commodity X86 servers (surprise - and everyone in 1989 said "port to Alpha, not the X86", but AlphaBSD didn't sound as good as 386BSD). Jim's also looking at astronomical images, which are also pretty big.

But where do I see the big growth? Easy, right in your own digital camera folder.

Digital photos and digital video will easily surpass in aggragate the large complex datasets of TerraServer, but it will be everywhere - not in one place. As Jim speaks of a "management challenge" of a centralized dataset, I ponder what we will do when everyone gets into the act.

Video archives alone in the personal digital closet can easily run to 100 GBytes for an average library of movies and stills, and the GBytes grow with the addition of each personal video.

According to Jim, CERN moves 1 GByte/sec on an OC-192 (9.9 Gbps) link - which means they're moving about 900 MBytes per second. This is like a typical full DVD movie shipped in a few seconds.

EE Times editor homed in on the concept presented of the "modular datacenter" with "virtualized storage systems". I'm an advocate of this - ardently - as I've written about datacenter operations and management and know how difficult it is. So I listened carefully as the very knowledgable editor (and nobody's fool) focussed no the issue of redundant fileservers, with all the consequent locking problems and global state consistency difficulties. In other words, he wanted a "demo". Of course, there wasn't one.