KFS at Quantcast

This entry was posted on Tuesday, December 23rd, 2008 at 9:58 pm and is filed under Scalability stories . You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

KFS at Quantcast

With people claiming Quantcast is using Hadoop, and not giving a credit to the KFS it is necessary to give more details about the story, and not so surprisingly, the details are available instantly, and here they are (these details refer to Quantcast KFS deployment)

Two deployments:
-  130 node cluster hosting log data
- ~2M files; 70TB of data; WORM system
- Metaserver uses ~2GB RAM
- ~1TB of data copied in during a week
- Used for daily jobs in read mode

Plan is to use both KFS and HDFS
-  For job output, backup from KFS to HDFS using Hadoop's distcp

For some more insight and technical details about moving petabyte data centers, hadoop and KFS and they uses in practice please see these:



2 Responses to “ KFS at Quantcast ”

  1. neon tabela Says:
    October 7th, 2009 at 5:57 am

    Your site is very easy in terms of expression and open. I think everyone who enters your site is very gratifying, but also sharing a very nice opportunity to give …

  2. lisa Says:
    October 23rd, 2009 at 1:38 am

    You can also try http://www.estimix.com – a free tool that provides a nice summary of the website performance. The estimation provided by estimix is the result of a complex analysis based on factors like: the age of the website, the demographic structure of the traffic, the countries where the website is popular and sources of the traffic

Leave a Reply