Metadata according to Judge Shira Sheindlin

In her recent decision, Judge Shira Sheindlin did one more service for eDiscovery - she outlined the standard metadata fields to be produced. That was a joy - these fields went straight into FreeEed.

Now it has standard fields from this list, and all the non-standard, or application-defined fields, following the standard ones.

I had a spare hour (not really, I should have been working on the book), but okay, and I added these fields. It never stops surprising me how fast you can add features if only you use the right tools. Look up the code :), it is committed. Now I only need to fill the fields out, but that too should be a breeze.

The first Houston Hadoop Meetup

It was definitely a success, all due to the wonderful and colorful people who came.

Nick Popov described how using a super-size database like HBase could have helped his previous company, a mortgage conglomerate, that needed to process thousands of events per second - not foreclosures:) as one of those present suggested, but all sorts of event.

Edward Wiener wanted to know what Hadoop and BigData is all about.

Helen Jiang sees definite opportunities for BigData at her current company, Structure Group, and is looking for ways to deepen her knowledge and also to introduce those ideas to the management.

Hal Martin came all the way from Clear Lake, and found that Montrose Library is a good compromise location for everybody. He is an experienced software contractor, and wants to expand his horizons.

Ron Chichester's appearance was a pleasant and welcome surprise: Ron is a lawyer, a legal expert, a forensics expert, an open source expert, and a software expert. Oof! Ron's opinions on politics and education (he also teaches law at the UH) were so fascinating that the group kept asking him and listening when we were forced out of the meeting room by the next group - for another half hour.

Pat Kerr is a pleasant and overall good fellow, and he will go anywhere where his friends' interests are.

The remaining members of the group - we missed you and hope to see you next time.

Oh, and what was the meeting about? We discussed what BigData is - each guest giving his or her example, how Hadoop and his friends can help, and were entertained by some anecdotal evidence about the BigData explosion.

Art: Waltner Charles Albert - The Pickwick Club

Cancer center builds Texas-sized cloud

As researchers at The University of Texas MD Anderson Cancer Center work at "making cancer history," they're doing so with the help of compute power and storage capacity from a private cloud.

But this is no ordinary cloud.

After all, when you're researching something as complex as the human genome you tend to think big, and MD Anderson's cloud reflects that type of ambition and scale. We're talking 8,000 processors and a half-dozen shared "large memory machines" with hundreds of terabytes of data storage attached, says Lynn Vogel, vice president and CIO of MD Anderson, in Houston.

Continue reading...

Open source eDiscovery - preparing for beta release

We had good interest and feedback so far (thank you, everybody!), and we are preparing for a beta release, where the software actually does some useful work.

I have put a number of documentation pages on the project's wiki. I have also added the basic culling on the text of the documents.

The two enhancements that remain for beta are

• organizing metadata into standard fields (that is in addition to the fact that all metadata, standard or not, is extracted); and
• processing emails from a PST file.

I have taken the decision to stop putting the documentation into the text-based files in the project itself, and put it on the GitHub's wiki instead. This means being integrated into GitHub and relying on it, but what of it? - it is very convenient for editing.

How important is blogging for a lawyer?

Not important at all, but really, very important. How can both be true? One asks a trusted friend for a lawyer, and the friend gives him the reference... based on that lawyer's blog. Continue reading...

FreeEed is reading processing parameters from a file

FreeEed now reads processing parameters from a file, has a number of default settings, and saves each run into a dated parameter file. More precisely,

• All defaults are read from default.freeeed.properties, which is provided with the distribution;
• They can be overwritten by any parameter file (such as my.freeeed.properties) mentioned with the option -param_file "filename"
• Command-line parameters take precedence over all.

That is simple and powerful: you don't need to know about parameters at all, and it will give you all the defaults. Or you can change every one of them.

In addition, when the program is run, it saves the parameters from the current run into a file that has a date in its name, such as freeeed.parameters.110323_175926.properties.

Plans for open source eDiscovery FreeEed

Here is what is need to make the package more competitive

• Add culling, PST (email) processing, deduping, and PDF/TIFF creation;
• Process all of Enron email (about 150 mailboxes) and announce the first, the 10, the 100, and all;
• Add advanced text processing, similar to predictive coding, which is the current direction of eDiscovery.

That will take a few months, but gradually the package will be become more and more useful.