KFS at Quantcast
Filed in archive Scalability stories on December 24, 2008
With people claiming Quantcast is using Hadoop, and not giving a credit to the KFS it is necessary to give more details about the story, and not so surprisingly, the details are available instantly, and here they are (these details refer to Quantcast KFS deployment)
Two deployments:
-  130 node cluster hosting log data
- ~2M files; 70TB of data; WORM system
- Metaserver uses ~2GB RAM
- ~1TB of data copied in during a week
- Used for daily jobs in read mode
Plan is to use both KFS and HDFS
-  For job output, backup from KFS to HDFS using Hadoop's distcp
For some more insight and technical details about moving petabyte data centers, hadoop and KFS and they uses in practice please see these:
- Kosmos FS - presentation slides
- Hadoop User Group video lecture with Sriram Rao
Two deployments:
-  130 node cluster hosting log data
- ~2M files; 70TB of data; WORM system
- Metaserver uses ~2GB RAM
- ~1TB of data copied in during a week
- Used for daily jobs in read mode
Plan is to use both KFS and HDFS
-  For job output, backup from KFS to HDFS using Hadoop's distcp
Permalink: KFS at Quantcast
Tags: KosmosFS CloudStore Hadoop quantcast hadoop source+alternatives open+source scalability+stories
Trackback: http://www.creative-weblogging.com/publish/mt-tb.pl/140125
Addthis
Ask
Blinklist
del.icio.us
Digg
Fark
Facebook
Google
Lycos
Ma.gnolia
Mr Wong
Netscape
Netvousz
Newsvine
Reddit
StumbleUpon
Slashdot
Tailrank
Technorati
Wink
Yahoo
Vote for KFS at Quantcast:
|
Rating: 9.00 out of 5 vote(s) cast.
|
Response from:
neon tabela
(10/07/09 11:57am)
Your site is very easy in terms of expression and open. I think everyone who enters your site is very gratifying, but also sharing a very nice opportunity to give …
Response from:
lisa
(10/23/09 7:38am)
You can also try http://www.estimix.com
– a free tool that provides a nice summary of the website performance. The estimation provided by estimix is the result of a complex analysis based on factors like: the age of the website, the demographic structure of the traffic, the countries where the website is popular and sources of the traffic
– a free tool that provides a nice summary of the website performance. The estimation provided by estimix is the result of a complex analysis based on factors like: the age of the website, the demographic structure of the traffic, the countries where the website is popular and sources of the traffic
| RSS |  See all blog subscribe options |
 What is RSS? | |
| Yahoo! |
|
| Addthis |
|
| Bloglines |
|
| Follow us on Twitter! |
