Hypertable
KFS at Quantcast
Filed in archive Scalability stories by Mateusz Berezecki on December 24, 2008
KFS at Quantcast
With people claiming Quantcast is using Hadoop, and not giving a credit to the KFS it is necessary to give more details about the story, and not so surprisingly, the details are available instantly, and here they are (these details refer to Quantcast KFS deployment)


Two deployments:
-  130 node cluster hosting log data
- ~2M files; 70TB of data; WORM system
- Metaserver uses ~2GB RAM
- ~1TB of data copied in during a week
- Used for daily jobs in read mode

Plan is to use both KFS and HDFS
-  For job output, backup from KFS to HDFS using Hadoop's distcp


For some more insight and technical details about moving petabyte data centers, hadoop and KFS and they uses in practice please see these:


Permalink: KFS at Quantcast
Tags: KosmosFS  CloudStore  Hadoop  quantcast  hadoop  source+alternatives  open+source  scalability+stories 
Trackback: http://publish.creative-weblogging.com/publish/mt-tb.pl/140125
img Addthis img Ask img Blinklist img del.icio.us img Digg img Fark img Facebook img Google img Lycos img Ma.gnolia Add this page to Mister Wong Mr Wong img Netscape img Netvousz img Newsvine img Reddit img StumbleUpon img Slashdot img Tailrank img Technorati img Wink img Yahoo

Vote for KFS at Quantcast:

  • Currently 9.00/10
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
Rating: 9.00 out of 5 vote(s) cast.
 
Subscribe
Share It
Popular posts from our other blogs:
RSSrss
See all blog subscribe options
Google google
What is RSS?
Yahoo! yahoo
Addthis Subscribe using any feed reader!
Bloglines Bloglines
Newsletter

TwitterFollow us on Twitter!