Search in eDiscovery

After you process all the files you have collected, your next step is searching through the data and slicing and dicing it in every way possible. This may be done at different times by both the defendant and the plaintiff. Case analysts need to try and answer different "what if" questions and scenarios. But with millions of files, SQL solutions become terribly slow. Anybody who saw that please raise your hand.

So here is how I formulated this questions on HBase and Cassandra user groups - because I needed it for FreeEed, my open source eDiscovery tool:

Imagine I need to store, say, 10M-100M documents, with each document having say 100 fields, like author, creation date, access date, etc., and then I want to ask questions like "give me all documents whose author is like abc**, and creation date any time in 2010 and access date in 2010-2011, and so on, perhaps 10-20 conditions, matching a list of some keywords."

What's best, Lucene, Katta, HBase CF with secondary indices, or plain scan and compare of every record?

Well, the very nice and knowledgeable people on these groups came with a number of solutions:

• I can't claim its the best, but I'd say solar or katta (Joey Echeverria)
• 'Add search to HBase' - HBASE-3529 is in development. (Jason Rutherglen)
• I, for one, am interested in learning more about elasticsearch with HBase after reading the article over at StumbleUpon (Matt Davies)
• I think this's the key line: "ElasticSearch is the search analogue to HBase that frees us from some restrictions that Solr imposes".  It is quite true however if search is inside of HBase ones gets the same thing.  Solr does have serious limitations in terms of scaling etc.  I think ES has done a great job there, though this could have been done with Solr just as easily, eg, upgrade Solr with the same functionality and remove the need for schemas.  Solr does allow schema-less, with eg, dynamic fields. (Jason Rutherglen)
• I'd say give Lily a spin. Currently, we rely on Solr for search. In the next few months, we'll take a good look at "HBase-native" secondary indexes as well (Steven Noels)
• There's also DrawnToScale (M. C. Srivas)
• I store over 500M documents in HBase, and index using Solr with dynamic fields.  This gives you tremendous flexibility to do the type of queries you are looking for -- and to make them simple and intuitive via a faceted interface. However, there was quite a bit of software that we had to write to get things going, and I can neither release all of it open source, or support other people using it.  If I had to start again, I would seriously look at solutions like elastic search and lily (David Buttler)
• Check out Solandra (Jake Luciani)

Thank you, all! Now I need to investigate all suggestions and select the one most fitting for eDiscovery.

An open-source Hadoop alternative from LexisNexis and FreeEed

As if by coincidence, on the next day after the LTN article by Evan Koblents which mentioned FreeEed, LexisNexis announced that it will open-source its Hadoop alternative for handling Big Data:

"LexisNexis announced today that it will open-source its High Performance Computing Cluster (HPCC) technology, as well as offer an enterprise version with commercial support. The company is positioning HPCC Systems, developed internally by its Risk Solutions unit, as an alternative to Apache Hadoop. A virtual machine for testing purposes will be available soon, and code will be available in a few weeks." For fuller announcement, go here.

As a first impression, what are the major comparison points?

• LexisNexis has been using its technology for a while and has a marketing clout to match, but it announced only plans to make the VM "available soon" and code "in a few weeks." One wonders if this is a reaction to the momentum that FreeEed has been gaining. On the other hand, FreeEed is already out on GitHub;
• LexisNexis is essentially a closed-source company, so one wonders how really open-sourced the offering is going to be. But they may be successful - look at Microsoft open-source contributions. In LexisNexis own words, "Only the core technology is being released, LexisNexis' own data linking techniques aren't being released, nor are its data sources." In contrast, FreeEed is pure open source (with commercial support options), and people are already investigating using it in ways beyond eDiscovery. This illustrates the flexibility of an open source offering.
• LexisNexis has Roxie, a system for query and data warehousing, but FreeEed will have the same based on Cassandra.
• LexisNexis sports ECL (Enterprise Control Language), but Cassandra has CQL (Cassandra Query Language).
• LexisNexis's "HPCC team has been working with Amazon Web Services to make sure the product work well on AWS servers," but FreeEed team has planned on the use of EC2 from the start and is actively working on it now.

The two are not exactly competitors at this point: LexisNexis releases the technology for high performance cluster computing and its risk handling applications, but they are close in their approach to open source and to handling Big Data, so it is worth watching.

"Open source could change the future of eDiscovery"

FreeEed is being profiled on LAW.COM

Implementing design patterns with Scala: Loan Pattern

Programmers are familiar with design patterns - these are best practice solutions to common problems. Then one often adds, almost apologetically, "you know, the code is different in every case, but you get to re-use the idea!"

Well, why would the code have to be different, and why can't you re-use even the code, and not only the idea? That's because in Java you cannot easily pass functions around. However, in Scala you can. Take, for example, the "loan pattern." It allows to borrow the resource and make sure you close it later. Here is what I mean. You may have thousands of little functions that open the SQL connection, use it, then close it, like in this picture

This screen image is to show off the syntax coloring of Scala code in NetBeans, but here is the same code in plain words:

  def doSqlStuff: Unit = {
    var conn: Connection = null
    try {    
      val url = "jdbc:mysql://localhost:3306/";
      val dbName = "beren";
      val driver = "com.mysql.jdbc.Driver";
      val userName = "beren";
      val password = "beren";          
      Class.forName(driver).newInstance();
      conn = DriverManager.getConnection(url + dbName, userName, password)
      val stmt = conn.prepareStatement("insert into stuff (id) values (?) ")
      stmt.setString(1, "" + new Date())
      stmt.executeUpdate
    } catch {
      case e: SQLException => {        
          e.printStackTrace
          println("SQLException: " + e.getMessage)
        }
      case ex: Exception => {        
          println("Exception: " + ex.getMessage)              
        }  
    } finally {
      conn.close
    }    
  }

But, only three lines are really important:


val stmt = conn.prepareStatement("insert into stuff (id) values (?) ")
stmt.setString(1, "" + new Date())
stmt.executeUpdate

Nevertheless, because of those three lines I am forced to replicate the boilerplate over and over. If I could somehow pass these lines to my boilerplate code, I would have it so much nicer!

OK, and indeed, it's better when you do it like this. Here is the boiler plate. Now you promise to pass it the function to do the real work.

def doSqlStuffBoilerplate (sqlDoer: Connection => Int): Unit = {
  var conn: Connection = null
  try {      
    val url = "jdbc:mysql://localhost:3306/";
    val dbName = "beren";
    val driver = "com.mysql.jdbc.Driver";
    val userName = "beren";
    val password = "beren";            
    Class.forName(driver).newInstance();
    conn = DriverManager.getConnection(url + dbName, userName, password)        
    
    sqlDoer(conn)
    
  } catch {
    case e: SQLException => {          
        e.printStackTrace
        println("SQLException: " + e.getMessage)   
      }
    case ex: Exception => {          
        println("Exception: " + ex.getMessage)                
    }    
  } finally {
    conn.close
  }        
}

Here are the three lines that you will pass as a function,

def sqlDoer(conn: Connection): Int = {

  val stmt = conn.prepareStatement("insert into stuff (id) values (?) ")

  stmt.setString(1, "" + new Date())

  stmt.executeUpdate

}

and this is how to call the method:

def doSqlStuffBetter(): Unit = {

  doSqlStuffBoilerplate (sqlDoer)

}

We tried this idea in Java, but the code is unreadable, and we went back to the old way of copying and pasting. With Scala, you can abstract and CODE the design pattern. (Even better is to in-line the function when passing, but I will do it later).

EddUpdate talks about FreeEed, the first open source tool for eDiscovery

Mark Kerzner of FreeEed (https://github.com/markkerzner/FreeEed) released version 1 of a free open source e-discovery processing tool today. Aimed at the do-it-yourself crowd, law firms looking to bring data processing in-house, and vendors seeking a way to process client data license-free, it provides an egalitarian alternative to the various solutions already on the market.

Continue reading...

Houston Hadoop Meetup #3

Vikram Oberoi, a Big Data engineer from Cloudera, presented Hadoop use cases, explaining when and why you would use Hadoop. Here is Vikram's presentation, which he graciously provided for us - an note that he gave it a Houston-specific spin for the energy and medical applications.

Thanks to Erin Mynatt, also of Cloudera, for bringing Vikram from California and for coming from Denver, as well as for sponsoring the pizza & beer afterwards.

Thanks also to John Bland II for joining the group just in time. By the way, John is doing some awesome graphics work CNN, Fox, etc., so check him out!

FreeEed processes its first Enron PST file

Version 1.0.2 fixes processing of PST and loose EML files. Thanks to the latest fixes by the Tika community, unusual email formats are processed unimpeded. When processing emails, Tika extracts the text from all the attachments, thus, one line of code does complete email processing.

These are the benefits of building on top of other open source tools and communities.

FreeEed is an open source tool for eDiscovery that can be downloaded on GitHub here.