Interview at Facebook

Took photo as a "proof of concept" or rather as  evidence that can be discovered :)

Visiting Bay Area Hadoop Meetup

http://www.meetup.com/hadoop/events/16805556/

Look at their numbers! 2235 total, and 291 attending.

Houston should beat that! (So far we have 23 total and 5 attending).

Cheers

History in FreeEed

Open source eDiscovery project, FreeEed, got a concept of history. History is shown in a separate window that can be open or closed it will. You can all erase it and start with a clean slate.

History is real-time, and it frees up the user interface to do other things while the software is crunching the information.

The code is currently in "Branch 2" and will be part of the 2.5 release.

A use-case for FreeEed: When your business model involves litigation

Imagine that you are in the business of buying other businesses. With the business you may acquire its assets, its debts, and its legal obligations. Litigation becomes a way of life, part of regular business.

The big part of litigation, of course, is eDiscovery. Now imagine that you have that part completely under your control. No payments to the third-party vendors, no timing limitations on their side. You can experiment with your litigation strategies at will, both for plaintiff and defendant situations.

That is where FreeEed comes it. You can run your eDiscovery and be in control of it. The FreeEed team, in turn, will do everything in its power to help you advance your goal, win or settle the cases, and grow your business.

Art: Fernando Botero - Man Reading a Paper

SQL pagination with complex query

It is well known that SQL has an OFFSET parameter, so if you need pages, you could do something like the following

SELECT * from TABLE1 LIMIT 100 OFFSET 100

The problem is, this query is very inefficient. Since many databases today are big (believe it or not, I saw a 2008 blog post saying "since many databases today are small"), inefficiency of OFFSET, which results in fetching all rows before it, is unacceptable.

The next step is to go specific and use a unique index in the TABLE1 (hope you have it), so you can sort and offset on this index, which will be efficient. You can then do something like the following

SELECT * from TABLE1 where TABLE1.id > 2500 order by TABLE1.id LIMIT 100,

assuming that the last id value you saw was 2500.

What do you do when you do not have an index that you can use, but instead are doing some join with multiple indexes? For example,

SELECT * from TABLE1 JOIN TABLE2 ORDER BY TABLE1.ID1, TABLE2.ID2 LIMIT 100

The query above is actually already half the solution. You only need to add this condition

WHERE
TABLE1.ID1 > last_value_id1 OR
(TABLE1.ID1 = last_value_id1 AND TABLE1.ID2 > last_value_id2).

You can continue and do this with as many indexes as you would like. This question and the solution came up in the  case of 4 indexes, and it worked great and very efficiently.

Art: Leonardo Da Vinci - Crossbow Machine

Installing Hadoop on Natty Narwhal

Here are the notes that will save you some trouble

To install Java on Natty, use this link
http://www.multimediaboom.com/how-to-install-java-in-ubuntu-11-04-natty-narwhal-ppa/

That is because you need a screen to accept Java license.

Start by installing Cloudera Hadoop in pseudo-distributed mode for Ubuntu lucid, not natty.

Formatting the node must be done as user hdfs, or else permissions won't work.

FreeEed gets graphical user interface

Version 2.0 of FreeEed is out!

It sports a clear GUI (graphical user interface) where the user can set parameters and run the processing. The GUI is intentionally a thin wrapper around the command-line utility, to keep the tool simple and efficient.

For all the rest,
Let Lion, Moonshine, Wall, and lovers twain
At large discourse, while here they do remain.

(Shakespeare, Midsummer Night's Dream)