{"id":102,"date":"2007-11-16T11:49:12","date_gmt":"2007-11-16T16:49:12","guid":{"rendered":"http:\/\/otac.isa-geek.net\/blog\/2007\/11\/16\/102-how-to-open-source-distributed-computing"},"modified":"2007-11-16T11:51:33","modified_gmt":"2007-11-16T16:51:33","slug":"how-to-open-source-distributed-computing","status":"publish","type":"post","link":"https:\/\/otac.isa-geek.net\/blog\/?p=102","title":{"rendered":"How-to: Open Source distributed computing"},"content":{"rendered":"

As some of you who know me are aware, I am a big fan of commodity hardware-based distributed computing as a mechanism for solving high-order problems. I first became aware of this type of solution when I read Google’s Map-Reduce presentation<\/a>, that started much of this ball rolling. The use of cheap PC’s in large (> 1000) clusters to solve data-intensive processing problems, gave Google a significant competitive advantage, particularly in their earlier years. I think it’s noteworthy that Yahoo and Microsoft both elected to start using huge server farms for their search efforts in response to this innovation from Google; Microsoft using Windows server deployed over ‘000s of PC’s<\/a>, while Yahoo has decided to support an Open Source project from Apache<\/a> called Hadoop<\/a>. Hadoop is pretty cool, in that it builds out much of the functionality seen in Google’s proprietary Map-Reduce implementation, but using all open-source. The infrastructure stack is LAMP, while the actual code is written in Java, giving Hadoop a pretty significant cross-platform compatibility metric. Yahoo’s Web 2.0 maven Jeremy Zawodny<\/a> has been blogging about Hadoop use in Yahoo<\/a>, and the technobloggers have been taking note (this is a recent one from Tim O’Reilly<\/a>).<\/p>\n

So what does this mean for me? Not sure yet, but it is clear that my initial experiments in distributed computing using ClusterKnoppix<\/a> (a Linux Debian derivative) and Mosix<\/a>, are really just a small part of a total research effort. Some of my questions are:<\/p>\n