Open Source E-Discovery Finds Scalability

In his front-page article in LTN, Evan Koblentz writes

"Open-source electronic discovery products, which appeared on the market in 2011, are quickly expanding in the new year. There is an evolving system for email archiving and searching, along with a cloud-based processing engine, both of which are on the cusp of large scalability, allowing them to handle growing workloads."

The three products discussed are

• Enkive from LinuxBox, for email storage, compliance, and eDiscovery;
• MailArchiva from Stimulus Software; and
• FreeEed from SHMsoft

Here is what Evan writes about FreeEed

On the processing side, SHMsoft's FreeEed project is now in its third generation. Previous versions could be clustered -- the concept of sharing large projects across many servers, resulting in faster jobs and better reliability -- by using the Hadoop system when hosted locally. But now there is a version called EC2Eed running atop the Amazon EC2 cloud, which will get clustering in two months, project leader Mark Kerzner said.

SHMsoft has also hired its first business development manager to help the project grow beyond word-of-mouth, Kerzner said.

...

The articles concludes with a prediction:

"Someone's got to take the plunge," David Horrigan, a 451 Group analyst, added, in Boston. "Invariably, as always, someone will. Because it's a huge potential market."

SHMsoft is Putting the “e” in Discovery with Hadoop Big Data Platforms

Press Release

SHMsoft, a worldwide leader in providing innovative solutions to high-tech problems based upon disruptive technologies, is pleased to announce new releases of FreeEed†™, and EC2Eed†™, open-source platforms for processing e-discovery.

FOR IMMEDIATE RELEASE

PRLog (Press Release) - Jan 12, 2012 -
“FreeEed and EC2Eed provide fully scalable, fault-tolerant ways for processing data and can cope with any size of e-discovery job, from gigabytes to petabytes,” said Mark Kerzner, President of SHMsoft of Houston.

“The processing engine in FreeEed and EC2Eed is organized by the Hadoop framework which supports data-intensive distributed applications under a free license.  I further developed the Hadoop platform using HDFS, HBase/SimpleDB, Tika, Lucene, and Solr, cutting-edge technologies that enable FreeEed and EC2Eed to handle all types of structured, unstructured and semi-structured data in a fast and efficient manner,” Kerzner said.

●   FreeEed Release 3.1.4 provides the litigation support/IT professional an option of running Windows to process data on their personal workstations.  Each project will create its own Lucene index for later searches.  Metadata results are output as a CSV file, while the native files and the extracted text are stored in a zip file(s). The end results can be used for culling and producing native files for legal review.  An impending release of FreeEed planned for February 2012 will create PDFs and/or TIF files for loading in your litigation review platform.

●   EC2Eed, a premium edition of FreeEed, processes e-discovery on the Amazon cloud.  The litigation support or IT professional can log into their AWS account and use EC2Eed as a platform-as-a-service for all their processing needs at a cost of 96 cents per hour per server (prices vary depending on the choice of server).  Data can either be stored on the Amazon S3 cloud or remain in the organization’s own data center.

SHMsoft will provide support and training for both implementations.  SHMsoft’s e-discovery processing platforms provide a much needed solution to address the high costs of e-discovery.

†† FreeEed™ and EC2Eed™ are the registered trademarks of SHMsoft, Inc.


For More Information Contact:
Julie Wade
Director of Business Development
SHMsoft, Inc.
(713) 204-2565
julie.wade@shmsoft.com

# # #

Software development company located in Houston, Texas. Specializing in Hadoop clusters and cloud implementations.

Search news in FreeEed

We've added the automatic build time to the About screen. We are going to have many version of FreeEed floating around, and this will be useful.

Today LinkedIn announced that they open-sources a number of search technologies. It is mind-boggling to think what kind of searches we can add to FreeEed very soon:

https://engineering.linkedin.com/open-source/indextank-now-open-source
http://sna-projects.com/bobo/
http://javasoze.github.com/zoie/
http://sna-projects.com/cleo/

And the big year in Solr/Lucene:
http://blog.sematext.com/2011/12/21/lucene-solr-year-2011-in-review/

How to process Microsoft Outlook .PST files

Here is an efficient way that FreeEed uses:

• Convert PST to MBOX formats. Use readpst in Linux and JPST in Windows. Before I used individual EML emails, but this is not so efficient, since there are too many of them. Dealing with MBOX files that correspond to top-level PST folders fits much better with the overall Hadoop processing;
• Use javamail in conjunction with mstor local access provider to process these MBOX files. This approach is great because it allows to use standard components of high quality. It also gives full access to attachments, CC, BCC, etc.

Now this approach is something I feel very good about, because it combines the best practices with overall efficiency. 

Adding image creation to FreeEed

By "image creation" in eDiscovery we mean making the PDF or TIFF images of the originals. Having these is convenient for review, because it eliminates the need for the various applications required to open the native file formats, and is useful for redacting.

In the last three weeks I was starting a new assignment that has to deal with text analytics and understanding in the context of Big Data, which is great, because the deeper knowledge of it will help me create open source tools for automated document review later on. But it also meant that I only had a couple hours to work on FreeEed in the evening, and that only for two evenings.

Nevertheless, this was enough. OpenOffice/LibreOffice are open source free applications that allow printing MS office documents to PDF, and JodConverter is a bridge that allows the code to talk to it. Altogether, printing is done with five lines of code. Here they are:

OfficeManager officeManager = new DefaultOfficeManagerConfiguration().buildOfficeManager();
officeManager.start();
OfficeDocumentConverter converter = new OfficeDocumentConverter(officeManager);
converter.convert(new File("test.odt"), new File("test.pdf");
officeManager.stop();

Taking out the start/stop code, you have just one line:

converter.convert(new File("test.odt"), new File("test.pdf");

That's it! One line of code (and lots of computing power) to convert all MS Office file formats to PDF. Isn't this amazing? Anytime you need more computing power, you get it from the cloud on the cheap, so FreeEed begins to really shine, because it is designed for parallel processing in the cloud.

Sometimes I wish that I would have more time for FreeEed, perhaps even doing it full-time. But then again, since I can do so much with the great open source tools, then maybe it is not even necessary.

Evan Koblentz on eDiscovery pricing

"The other extreme is products and services that are cost-free. The open-source FreeEED project may get traction now that it's available for Microsoft Windows. In addition, FreeEDD will soon tackle automated document review, project leader Mark Kerzner said. Open source is sometimes controversial for developers' philosophical approach, and for what closed-sourced vendors allege are hidden costs of implementation and support. But at least you know what the base software costs -- nothing."

Full article

Open source eDiscovery (FreeEed) for Windows is released

All paralegals, lawyers, and do-it-yourselfers!

We have released the version of FreeEed, which runs in Windows. It also runs on a Hadoop cluster, is scalable, and is free. You can find it here, http://freeeed.org.

Thank you. Sincerely,
FreeEed Team

Art: Bruegel, Pieter the Elder - Dance