Satnam Alag's Blog

“Collective Intelligence” book review

Taran Rampersad wrote a good review for the book at http://www.knowprose.com/node/18880.

I have copied the last three paragraphs from Taran's review

"Part three brings it all together in a way such that it is greater than the sum of the two other parts. Code, diagrams and the author's voice are clear and left me wondering what else is going to go into this book this August.

Overall, I am very impressed with this work and even excited about the book - something which may something about myself as much as the book. It is technical, it is theoretical - but most importantly, it is practical and focused throughout a diverse group of disciplines that, through the author, are integrated very well.

Since it is an unedited draft, I will not give the book a formal review at this point. But I will say that it is what many people have been waiting for - an explanation for how these things work. Great stuff. Collective Intelligence in Action is worth an early look - or waiting for".

Thanks Taran for taking the time to look at the book and writing the review.

Differences from other books

The book is really meant for developers (basic level of Java understanding helps) who are looking to add intelligence to their applications, especially user-centric Web 2.0 applications. A lot of work has been done by the open-source community in Java in the areas of text processing and search (Lucene), data mining (WEKA), web crawling (Nutch), and data mining standards (JDM). This book leverages these frameworks; presents examples and develops code that you can directly use in your Java application.  

This is a practical book and I present a holistic view on things required to apply these techniques in the real-world. Consequently, the book discusses the architectures for implementing intelligence – you will find lots of diagrams, especially UML diagrams, lots of screen shots from well-known sites,  in addition to code listings, and even database schema designs.

There are lots of examples. Typically, concepts and the underlying math for algorithms is explained via examples with detailed step-by-step analysis. Accompanying the examples is Java code that demonstrates the concepts by implementing the concept and/or using open-source frameworks.

There are a number of exciting topics that you will find interesting and are typically not covered by other books: harvesting information from the blogosphere, analyzing content – especially user-generated content, intelligent web crawling, intelligent search, building recommendation systems. In the last chapter, I also cover three real-world examples of personalization by Amazon, Google News, and Netflix – the BellKor solution from the Netflix competition is also covered. At the end of this you should be familiar with text analysis using Lucene, web crawling using Nutch, building content-based and collaborative-based recommendation engines, and data mining using WEKA and JDM.

The book is on track to be released in August. In the meantime you can get a copy from the <a href="http://www.manning.com/affiliate/idevaffiliate.php?id=891_113_1_23> Manning website </a> through their early access program.

 

Bestselling Early Access Titles

I happened to check Manning's homepage and was happy to see that my book Collective Intelligence in Action made it to the second spot in the bestselling early access titles.

Thanks for all the positive feedback and for buying the book. I am working with Manning editors to finishing the book. Hopefully, we will get done in the next couple of weeks.

Data Search Engine

On February 28,  2008, Bloomberg.com reported that George Church, a Professor of Genetics at Harvard Medical School, plans to spend S1 billion to create a database to find new drugs by correlating each person’s personal health history to their DNA related information.  Church, who’s PhD in 1984 led to the first direct genomic sequencing method and also helped initiate the Human Genome Project is backed by Google and OrbiMed Advisors LLC. Church plans to decode the DNA of 100,000 people in world’s biggest gene sequencing project.

The entire human genome has over 3 billion DNA base pairs. Humans have 24 unique chromosomes and an estimated 20,000 – 25, 000 unique genes. A gene is a portion of the genomic sequence that encodes proteins – building blocks of cells and tissues. Variations in genes and other parts of the DNA have been linked to various types of diseases. DNA chips or microarrays enable researchers to generate large amounts of DNA related data. Computational biology or bioinformatics is a very active area of research that deals with deriving intelligence from biological data.

Companies such as 23andme.com, navigenics.com, and decodeme.com provide consumers a service by which Single Neucleotide Polymorphisms (SNP) – tiny sequence variations – in their DNA is detected, with the promise that it can be used to calculate the levels of risks associated with various diseases. You can see a sample of results provided by 23andme here Above are just some of the examples of the growing trends in the life sciences area. Data is being generated at a fast pace within this field. Searching through large amounts of experimental data where you normalize the data, use the actual experimental values of the data along with any meta-text or annotations associated with the experiments to discover new relationships is data-based search. Such search engines typically also leverage an ontology and complex biological relationships to guide their searches.

One such company that I am associated with is NextBio, a vertical search engine for life sciences . At NextBio, we leverage all publically available life sciences related data, along with user-contributed data to provide a platform for life scientists to discover new relationships, perhaps between genes, diseases and treatments.

Going back to the original example of Church’s plan to decode DNA of 100,000 people I do hope that all that data will be publically available in the future for search engines, such as NextBio to use and help discover new drugs for diseases.

Collective Intelligence Book with Examples in Java

Last Wednesday, an early  version  of my book "Collective Intelligence in Action" was made available by Manning. This is a part of their early access program where an unedited copy is made available to interested readers.

You will find this book useful if you are looking to add intelligence to your application, especially Web 2.0 applications. All the code examples in the book are in Java. In addition I do cover the ideal system architecture required for embedding intelligence along with some of the database design required to build the features.

The book is divided into three parts. The first part deals with gathering information -- from both within and outside your application. It covers converting user interaction into intelligence, the role of tagging and building tag clouds, extracting intelligence from user-generated content. Next a toolkit is developed to harvest information from the blogosphere as well as to intelligently crawl the web. You will also learn how to leverage the open-source crawler Nutch.

The second part of the book deals with discovering intelligence. It  provides  an overview of the data mining process and introduces you to  open-source  machine learning toolkit WEKA. It also provides you with an overview of Java Data Mining (JDM) standard. Next, you will develop a text-analysis toolkit and learn about clustering data along with making predictions.

The third part of the book deals with applying intelligence to your application. It begins with an overview of intelligent search followed by developing a recommendation engine using both content-based and collaborative-based techniques.

The book has a plethora of examples to explain the concepts. There are Java implementations for understanding and implementing the key algorithms. This is typically followed by the use of open-source tools to solve the problem.

This book will provide you with a good overview of -- Lucene, Nutch, WEKA, JDM, tagging and harvesting the blogosphere.

The first chapter of the book is freely available. I look forward to your suggestions to improve the book.