Satnam Alag's Blog

Bestselling Early Access Titles

I happened to check Manning's homepage and was happy to see that my book Collective Intelligence in Action made it to the second spot in the bestselling early access titles.

Thanks for all the positive feedback and for buying the book. I am working with Manning editors to finishing the book. Hopefully, we will get done in the next couple of weeks.

Data Search Engine

On February 28,  2008, Bloomberg.com reported that George Church, a Professor of Genetics at Harvard Medical School, plans to spend S1 billion to create a database to find new drugs by correlating each person’s personal health history to their DNA related information.  Church, who’s PhD in 1984 led to the first direct genomic sequencing method and also helped initiate the Human Genome Project is backed by Google and OrbiMed Advisors LLC. Church plans to decode the DNA of 100,000 people in world’s biggest gene sequencing project.

The entire human genome has over 3 billion DNA base pairs. Humans have 24 unique chromosomes and an estimated 20,000 – 25, 000 unique genes. A gene is a portion of the genomic sequence that encodes proteins – building blocks of cells and tissues. Variations in genes and other parts of the DNA have been linked to various types of diseases. DNA chips or microarrays enable researchers to generate large amounts of DNA related data. Computational biology or bioinformatics is a very active area of research that deals with deriving intelligence from biological data.

Companies such as 23andme.com, navigenics.com, and decodeme.com provide consumers a service by which Single Neucleotide Polymorphisms (SNP) – tiny sequence variations – in their DNA is detected, with the promise that it can be used to calculate the levels of risks associated with various diseases. You can see a sample of results provided by 23andme here Above are just some of the examples of the growing trends in the life sciences area. Data is being generated at a fast pace within this field. Searching through large amounts of experimental data where you normalize the data, use the actual experimental values of the data along with any meta-text or annotations associated with the experiments to discover new relationships is data-based search. Such search engines typically also leverage an ontology and complex biological relationships to guide their searches.

One such company that I am associated with is NextBio, a vertical search engine for life sciences . At NextBio, we leverage all publically available life sciences related data, along with user-contributed data to provide a platform for life scientists to discover new relationships, perhaps between genes, diseases and treatments.

Going back to the original example of Church’s plan to decode DNA of 100,000 people I do hope that all that data will be publically available in the future for search engines, such as NextBio to use and help discover new drugs for diseases.

Collective Intelligence Book with Examples in Java

Last Wednesday, an early  version  of my book "Collective Intelligence in Action" was made available by Manning. This is a part of their early access program where an unedited copy is made available to interested readers.

You will find this book useful if you are looking to add intelligence to your application, especially Web 2.0 applications. All the code examples in the book are in Java. In addition I do cover the ideal system architecture required for embedding intelligence along with some of the database design required to build the features.

The book is divided into three parts. The first part deals with gathering information -- from both within and outside your application. It covers converting user interaction into intelligence, the role of tagging and building tag clouds, extracting intelligence from user-generated content. Next a toolkit is developed to harvest information from the blogosphere as well as to intelligently crawl the web. You will also learn how to leverage the open-source crawler Nutch.

The second part of the book deals with discovering intelligence. It  provides  an overview of the data mining process and introduces you to  open-source  machine learning toolkit WEKA. It also provides you with an overview of Java Data Mining (JDM) standard. Next, you will develop a text-analysis toolkit and learn about clustering data along with making predictions.

The third part of the book deals with applying intelligence to your application. It begins with an overview of intelligent search followed by developing a recommendation engine using both content-based and collaborative-based techniques.

The book has a plethora of examples to explain the concepts. There are Java implementations for understanding and implementing the key algorithms. This is typically followed by the use of open-source tools to solve the problem.

This book will provide you with a good overview of -- Lucene, Nutch, WEKA, JDM, tagging and harvesting the blogosphere.

The first chapter of the book is freely available. I look forward to your suggestions to improve the book.