Tuesday, September 12, 2017

Does FreeEed search for numbers? - Yes, it does!

This question was asked by one of the users, can he find numbers in the text that FreeEed indexes. I got curious myself and checked.

The reason that this is an important question is that I remember Craig Ball mentioning that in one of the requirements for good eDiscovery software. So OK, I ran a few searches and found out that out-of-the-box FreeEed does index all numbers. That felt good, and I am attaching the screenshots of the experiment.

Of course, that is not a special property of FreeEed but of Tika, Lucene, and SOLR. It's these components that are responsible for what FreeEed indexes.

Had this not been the case, I would tweak the use of the components, but luckily this was the way FreeEed already uses them. The advantage of passing through to these libraries is that the users can rely on the well-known Lucene syntax to do their searches.

Monday, July 31, 2017

Couchbase at Houston Hadoop & Spark Meetup

Justin Tuggle presented the well-justified reasons why today only NoSQL databases are up to par, to provide customer engagement and means for business survival, and of them, why Couchbase is to be preferred.

Here is the link to the materials.

Thursday, July 20, 2017

An easy way to run FreeEed on Amazon

Running FreeEed on Amazon is very easy and offers some substantial benefits.

  1. You can get a fully provisioned server in a minute
  2. You can get any size of hard drive and a large number of CPU
  3. It is as easy as using your desktop.
To start the server, find this AMI in the Oregon region on EC2: ami-e6acbf9f.

After you start the service, open the assigned IP in any browser. You will see a screen like the following below

Click on the 'vnc.html'. You will see the login screen

After you log in, you will see a full Ubuntu desktop, where you can do any work. FreeEed is already installed.


Sunday, July 9, 2017

eDisco and Open Source Software

Today I am starting a series of blog posts on how to do eDiscovery with open source software. I will base it initially on a wonderful book "Project Management in Electronic Discovery". The advice that I will give will not be limited to FreeEed, but it will draw on the complete range of Open Source, Data Science, etc.

Every eDiscovery person has her or his own set of tools, and I hope that these articles will add to your library. Let's organize those docs!

(Image source: Pexels.com)

Saturday, July 8, 2017

New use cases for FreeEed

Today we release early preview of FreeEed with the following use cases
For the plaintiff.

If you ask for the eDiscovery documents, you might eventually get them. Now, what do you do with them? 

The answer that FreeEed gives you is "Use the load file as the data source." That is, FreeEed allows you to load the documents you were sent and start reviewing them. 

For the researcher

Perhaps not directly related to eDiscovery, but people do you FreeEed for various research purposes. For example, at DARPA they loaded the court documents obtained from the NY Court of Appeals website and added some annotations (tags). Now, to do data analytics on the set, they need to export the documents back, with the new tags. This is provided in the option "Export the load file," which will export either the full set, with the annotations, or the current search results.

For the techie

Sometimes your eDiscovery or other data is in the form of a JSON file. JSON format is popular because it is flexible and allows to define your fields. In fact, you can change the fields from record to record.

This is provided now with selecting "JSON" as an input format, with the option "Use the load file as the data source." 

Likewise, you can import any CSV file.

Other improvements include

* Implement extensive continuous testing with Jenkins (http://freeeed.from-tx.com:8000/)
* Review - quick preview now working

Friday, May 26, 2017

Sub-second SQL queries with LLAP from Hortonworks

Houston Hadoop & Spark Meetup in April was graced by the presentation from Ravi Mutyala of Hortonworks. Here are the slides, https://www.slideshare.net/HadoopSummit/llap-subsecond-analytical-queries-in-hive-63959757. Please refer to Ravi for further questions.

Wednesday, March 22, 2017

How to create an IntelliJ shortcut on Ubuntu

I always dread IntelliJ upgrade, because I don't remember how to update my Ubuntu shortcut. So here it is, for me and other souls.

1. Unzip and run from the command line, just you do all versions. No problem here.
2. Choose menu Tools, then Create Desktop Entry.
3. This will create an entry in ./local. Copy it to your desktop:

cp  ./.local/share/applications/jetbrains-idea.desktop ~/Desktop/

Optional: drag it to the toolbar

Happy traveling! 逍遙遊

Sunday, March 12, 2017

What I saw in Bentonville, AR

Recently I was in Bentonville, taught Big Data, and visited the Crystal Bridges museum there. I share this amazing experience in my blog post here.

Thursday, February 9, 2017

FreeEed for eDiscovery response and for general research


We have re-visited loading the eDiscovery production results for review,  and added loading the DAT file. This is available in version 8.1, due to be released soon. We will write another blog post and add the new instruction.

Thank you

FreeEed is a popular open source eDiscovery tool. It boasts over 1,000 users, has active projects in major consulting companies and is popular with researchers. However, it often needs to be used upside down. Here is what I mean.

In regular eDiscovery, you input directories, and FreeEed processes them, giving you these outputs

  1. "Load file," or a CSV file with the metadata, one line per document or email.
  2. "Output file," a zip file containing native documents, extracted text, PDF images of all files, and exceptions, each in its folder.
  3. Case for review, loaded into FreeEedUI review tool. It is put into SOLR as a back end, but for review, one uses the FreeEedUI.
However, there are two use cases that would require the opposite: reviewing the eDiscovery response, and using FreeEed for research.

Reviewing the eDiscovery response

If you send an eDiscovery request, you may get back the load file and the documents. In essence, you are getting the data in the same format that FreeEed outputs it. What you would like then is to reverse the process, to make the load file the input, and to index the documents for search. This is now implemented in FreeEed.

When you select the input, you see a "Data Source" panel. If you choose eDiscovery, FreeEed will work as before, that is, accepting your custodians' files as input.

If you choose the "Load file" radio button as a data source, the program will do the following
  • Read each line of the load file
  • For each line, use the given fields as metadata
  • Make the metadata and the extracted file text searchable and create a case in FreeEed for review
  • Available in FreeEed V 7.3
This use case lends itself very nicely to parallelization, and can, therefore, be processed on a Hadoop cluster, to accommodate large volumes.

Using FreeEed as a research tool

Often, researchers already have the metadata extracted. For example, in our Memex court document investigation, we already have elaborate parsing code that extracts metadata from the court documents. In this case, we want to be able to load the metadata and the file text into FreeEedUI for research. We should be able to answer questions like
  • How many times was a given crime mentioned?
  • Repeat the question above for the particular judge and in a specific time range (this questions will search metadata in a structured way, as well as text).
Clearly, this is the same use case as above. The only difference is that we need a different set of metadata fields than the one used in FreeEed by default. Technically, this amounts to programmatically changing the schema in SOLR, and this will be done in the next update, V 7.4.

Monday, January 23, 2017

Healthcare and Machine Learning

I have written about the current state of Machine Learning in Healthcare and about the practical steps that the healthcare professionals can take today.

The major points are
  • Quick Overview of Machine Learning
  • What can Machine Learning do for healthcare - overview of current use cases
  • What steps one can take today while waiting for big developments to come through

The blog post is on the Elephant Scale blog, http://elephantscale.com/2017/01/healthcare-machine-learning-practical-approach/ so you can continue there.