How to do Early Case Assessment with FreeEed

Sometimes, you have a lot of data to process for eDiscovery. So, you go to your favorite eDiscovery provider and ask them to process your data and then host it for your review. But there's the rub: processing costs X number of dollars per gigabyte, and usually, you don't want to host all the data. 

Here is how you can solve this problem with FreeEED and save oodles of money in the process. First, I will explain the harder way, using the review. Then I will show how to go straight to the results, once you are more trusting the results.

Way 1 - with the review

 Download and start FreeEED

Select your projects and add files to your project

Stage, Process, and Go to Review

In the review, find all responsive documents. 

Now, simply click on "Export as Natives"

Here, you got want you wanted! You know now what documents you will deal with. Read them, analyze them. 

Put them into your favorite review platform, like Relativity. From there, you will be able to do production and share the documents with others who need them. And by the way, we can set you up and help with Relativity as well. 

Way 2 - go straight to the results

Start as above, by downloading FreeEED. But, instead of going all the way with the review, simply use culling

Enter your search string (I entered 'matt' but it accepts complete Lucene syntax, with metadata names and ranges), and click on process. When done, send the production results. Or, be more formal and go to Relativity, as above.

Cheers!

Searching Blockchain with FreeEed

The Blockchain is composed of multiple blocks that can contain any information. However, it is not a database in a traditional sense: it is not fast, and it does not answer queries.

For example, the writing speed is one block every 10 minutes for Bitcoin and about one block every seven seconds for Ethereum. Queries, as such, do not exist at all: neither SQL nor NoSQL-type language is not provided.

Meanwhile, the information stored in Blockchain often needs to be searched. Here is a design pattern from CSIRO.

As of today, such a tool exists. FreeEed has been used by lawyers to do eDiscovery, legal reviewers, and researchers for all kinds of investigations. It allows you to give any data as input (see here) and indexes that data for searches. The data can be open Office files, PST mailboxes, a "load file" produced to lawyers due to an eDiscovery request, and Blockchain.

We are actively working on FreeEed all the time, adding input formats, processing capabilities, and machine learning. The tool is open source and welcomes new additions. The review part is called "FreeEed Review" and works through the browser.

The back end used to implement text search is Elasticsearch. This means that you can also look at the processed data through the mighty ELK (Elasticsearch, Logstash, Kibana), which is also open source.

Happy searching!

FreeEed with Elasticsearch (7.7.2 release)

Improvements in this version (7.7.2):

• Elasticsearch integration. Now the users get more open source tools to work with FreeEed: Elasticsearch, Logstash, and Kibana. 
• Bug fixes, code refactoring.
• Go here http://freeeed.org/

Does FreeEed search for numbers? - Yes, it does!

This question was asked by one of the users, can he find numbers in the text that FreeEed indexes. I got curious myself and checked.

The reason that this is an important question is that I remember Craig Ball mentioning that in one of the requirements for good eDiscovery software. So OK, I ran a few searches and found out that out-of-the-box FreeEed does index all numbers. That felt good, and I am attaching the screenshots of the experiment.

Of course, that is not a special property of FreeEed but of Tika, Lucene, and SOLR. It's these components that are responsible for what FreeEed indexes.

Had this not been the case, I would tweak the use of the components, but luckily this was the way FreeEed already uses them. The advantage of passing through to these libraries is that the users can rely on the well-known Lucene syntax to do their searches.

An easy way to run FreeEed on Amazon

Running FreeEed on Amazon is very easy and offers some substantial benefits.

1. You can get a fully provisioned server in a minute
2. You can get any size of hard drive and a large number of CPU
3. It is as easy as using your desktop.

To start the server, find this AMI in the Oregon region on EC2: ami-e6acbf9f.

After you start the service, open the assigned IP in any browser. You will see a screen like the following below

Click on the 'vnc.html'. You will see the login screen

After you log in, you will see a full Ubuntu desktop, where you can do any work. FreeEed is already installed.

Enjoy!

eDisco and Open Source Software

Today I am starting a series of blog posts on how to do eDiscovery with open source software. I will base it initially on a wonderful book "Project Management in Electronic Discovery". The advice that I will give will not be limited to FreeEed, but it will draw on the complete range of Open Source, Data Science, etc.

Every eDiscovery person has her or his own set of tools, and I hope that these articles will add to your library. Let's organize those docs!

(Image source: Pexels.com)

New use cases for FreeEed

Today we release early preview of FreeEed with the following use cases

For the plaintiff.

If you ask for the eDiscovery documents, you might eventually get them. Now, what do you do with them? 

The answer that FreeEed gives you is "Use the load file as the data source." That is, FreeEed allows you to load the documents you were sent and start reviewing them. 

For the researcher

Perhaps not directly related to eDiscovery, but people do you FreeEed for various research purposes. For example, at DARPA they loaded the court documents obtained from the NY Court of Appeals website and added some annotations (tags). Now, to do data analytics on the set, they need to export the documents back, with the new tags. This is provided in the option "Export the load file," which will export either the full set, with the annotations, or the current search results.

For the techie

Sometimes your eDiscovery or other data is in the form of a JSON file. JSON format is popular because it is flexible and allows to define your fields. In fact, you can change the fields from record to record.

This is provided now with selecting "JSON" as an input format, with the option "Use the load file as the data source." 

Likewise, you can import any CSV file.

Other improvements include

* Implement extensive continuous testing with Jenkins (http://freeeed.from-tx.com:8000/)

* Review - quick preview now working

FreeEed for eDiscovery response and for general research

Update:

We have re-visited loading the eDiscovery production results for review,  and added loading the DAT file. This is available in version 8.1, due to be released soon. We will write another blog post and add the new instruction.

Thank you

FreeEed is a popular open source eDiscovery tool. It boasts over 1,000 users, has active projects in major consulting companies and is popular with researchers. However, it often needs to be used upside down. Here is what I mean.

In regular eDiscovery, you input directories, and FreeEed processes them, giving you these outputs

1. "Load file," or a CSV file with the metadata, one line per document or email.
2. "Output file," a zip file containing native documents, extracted text, PDF images of all files, and exceptions, each in its folder.
3. Case for review, loaded into FreeEedUI review tool. It is put into SOLR as a back end, but for review, one uses the FreeEedUI.

However, there are two use cases that would require the opposite: reviewing the eDiscovery response, and using FreeEed for research.

Reviewing the eDiscovery response

If you send an eDiscovery request, you may get back the load file and the documents. In essence, you are getting the data in the same format that FreeEed outputs it. What you would like then is to reverse the process, to make the load file the input, and to index the documents for search. This is now implemented in FreeEed.

When you select the input, you see a "Data Source" panel. If you choose eDiscovery, FreeEed will work as before, that is, accepting your custodians' files as input.

If you choose the "Load file" radio button as a data source, the program will do the following

• Read each line of the load file
• For each line, use the given fields as metadata
• Make the metadata and the extracted file text searchable and create a case in FreeEed for review
• Available in FreeEed V 7.3

This use case lends itself very nicely to parallelization, and can, therefore, be processed on a Hadoop cluster, to accommodate large volumes.

Using FreeEed as a research tool

Often, researchers already have the metadata extracted. For example, in our Memex court document investigation, we already have elaborate parsing code that extracts metadata from the court documents. In this case, we want to be able to load the metadata and the file text into FreeEedUI for research. We should be able to answer questions like

• How many times was a given crime mentioned?
• Repeat the question above for the particular judge and in a specific time range (this questions will search metadata in a structured way, as well as text).

Clearly, this is the same use case as above. The only difference is that we need a different set of metadata fields than the one used in FreeEed by default. Technically, this amounts to programmatically changing the schema in SOLR, and this will be done in the next update, V 7.4.

Using FreeEed in the Memex program for investigations

A common problem in investigations is that the authors of the research software, which is being produced in the course of the Memex problem, are themselves not authorized to see the data that the investigation agencies deal with.

To address this problem, we added hash search to FreeEed. First, we have added the metadata screen display (which was not previously available), and users can see the metadata.

This screenshot presents the view of the metadata table. Metadata, of course, is "data about data." It shows all the fields collected from the documents being searched, together with their "a.k.a" or synonyms. For example, in this screenshot, you can see that field 22 can be called "From, but it can also be called "Author" or "Message-From." You can see now that there is a new field, called "Hash."

Next, the file hash is added to the metadata fields settings. Users have requested this feature prior, and now it is available. For emails, the hash is defined using the popular email fields. In FreeEed, this is configurable through the database.

This hash is shown in the screenshot on the left, which represents the 'load file' output by FreeEed. There it is seen with other popular metadata fields, which were recently added by request, such as Message-ID.

The investigating agency can simply compute the hashes of the objects, such as texts, phones, images, or anything else that they are looking for, and search for these, without revealing what they are searching for, to the authors of the software or the processors. Entities other than investigating agencies may find this feature useful as well.


Now, this shows in the processing results but is it searchable? For that, Hash has been added to the schema in the FreeEedUI search engine (which is SOLR). Now Hash shows up as one of the fields for each document, as the screenshot shows.

The last question, can one search having just the hash value? The answer is yes, you can search on the hash alone. To verify this, pick up one of the hashes that you saw in the documents and try to search for this value. You will find this one document - as is to be expected, since all hashes, MD5 and SHA-1, are designed to be unique per document.  The last screenshot illustrates this.

Additionally, FreeEed can provide the results sorted by user-defined "document significance," using the user-provided functions. Such functions are supplied by the Memex groups.

Word clouds in FreeEed

Word clouds have been added to FreeEed as an early release. To try, download the jar from here, https://s3.amazonaws.com/shmsoft/releases/freeeed-processing-1.0-SNAPSHOT-jar-with-dependencies.jar and replace the jar by the same name in your install. Then run freeeed_player.sh (.bat) as usual.

Here is an example of a word cloud and a screenshot of the Analytics menu, which features word clouds.

The word cloud is from project included with FreeEed, which is just a collection of unconnected documents, so the cloud is not very meaningful. You should get something related to your use cases and more useful.

Your feedback will be very much appreciated.

FreeEed eDiscovery, AI, Machine Learning, and Social Media

In the V7.0.0 release of FreeEed, we are highlighting text analytics and social media. 

Please read more about "Adding Text Analytics to FreeEed" and "Using FreeEed for Social Media Discovery."

You might also find interesting the articles that Mark Kerzner, the author of FreeEed, wrote recently on Bloomberg Law.

The source code

Our open source code collection is growing, and we have combined it all in one place: the SHMsoft company page on GitHub.

With gratitude and acknowledgment: this work is funded in part by the DARPA/Memex program, here is a Forbes article about our team.

Next: FreeEed as a service in the Amazon AWS cloud.

Cheers, 

FreeEed - eDiscovery easy as popcorn.   

Using FreeEed for social media discovery

One of the areas that the Memex/DARPA teams excel in is crawling. FreeEed and the people behind it are part of the Memex, so it was quite natural to integrate discovery of crawl results into FreeEed processing and review.

Here is a recent Forbes article about the team.

Searching the websites and social media has been added to FreeEed starting from version 7. The common format to store crawl results is JSON. Each JSON description corresponds to a website page, user post, or a similar item.

Each JSON search entry is represented by a one-line in the archive file. The archive is given the extension *.jl, which stands for "JSON line".

FreeEed understands the *.jl extension, parses the JSON content of every line in the *.jl file, and finds indexes such fields as text, authors, etc., and makes them searchable in the FreeEed Review tool.

Below is a screenshot of FreeEeedUI review, illustrating searches in  a collection from an escort services website.

How to create your crawler? You can use the crawler from Scraping Hub, also a member of the Memex team. Or you can use the trusted friend, Apache Nutch. Nutch has been around for more than ten years, and it is the beginning of Hadoop.

By the way, we provide training in all these technologies.

Adding text analytics to FreeEed

Many documents in eDiscovery can be understood on a much deeper level than keyword search. Since groups of documents often have a similar structure, one can configure the software to extract additional fields from such documents.

Case study

We have collected all appeal documents from the NY Court of Appeals. For that, we crawled the court website and collected approximately 100,000 documents.

We have then configured the GATE (General Architecture for Text Engineering) tool to extract the information of interest from every document.

Here is the screenshot of GATE screen configured to extract information. It takes a few minutes to extract this information from 100,000 appeal cases, and the output is a CSV file which can be opened as a spreadsheet.

The verify the quality of the information extraction, we watch the statistics. Below is an example of the statistics from one of the latest runs. It shows the percentages of the information being reflected in the case document and successfully extracted by the software.

Files in dir: 111018
Docs processed : 100.0%
Case number: 100.0%
Metadata extracted: 100.0%
Civil: 71.0%
Criminal: 29.0%
Court: 94.7%
Gap days: 92.7%
First date: 92.8%
Appeal date: 100.0%
Judge: 85.8%
Other judges present: 98.4%
District attorney: 61.3%
Assistant district attorney: 100.0%
Crimes: 37.7%
County: 91.7%
Mode of conviction: 53.9%
Keywords: 93.3%
Interest of justice: 4.9%
References to cases: 19.9%
Number of output files: 12
Runtime: 2086 seconds

Our verification assured us that the rate of successful extraction (when the information is actually present) is high.

Below is an example screenshot of the information obtained. The output for all documents (25 MB) can be downloaded from here.

Adding this information to eDiscovery

There are two ways how you can add this information to FreeEed.

1. The metadata fields can be added to the documents, and FreeEed configured to add them to the review; or
2. The GATE workflow can be compiled and run directly within FreeEed.

Conclusions

The configuration of the GATE tool is an acquired skill, but even out-of-the-box extractors provide useful information. This work was done as part of Memex DARPA project, and the researchers found the extracted information extremely useful.

By the way, we provide training in all these technologies.

The power of text analytics at DARPA/Memex

One of the things we are doing in the DARPA Memex program is text analytics. One of the outcomes of it is an open source project called MemexGATE.

By itself, GATE stands for Generic Architecture for Text Engineering, and it is a mature and widely-used tool. It is up to you to create something useful with GATE, and MemexGATE is our first step. This is an application configured to understand court documents. It will detect people mentioned in the documents, dates, places, and many more characteristics that take you beyond plain key word searches.

To achieve this, GATE combines processing pipelines (such as sentence splitter, language-specific word tokenizer, part of speech tagger, etc) with gazetteers. Now, what is a gazetteer? -- It is a list of people, places etc. that can occur in your documents. MemexGATE includes scripts that collect all US judges, for example, so that they can be detected, when found in a document.

But MemexGATE does more: it is scalable. Building on the Behemot framework, it can parallelise processing for the Hadoop cluster, thus putting no limit on the size of the corpus. MemexGATE was designed and implemented by Jet Propulsion Lab team, and the project committer is Lewis McGibbney.

The picture shown above gives an example of a processed document (from NY court of appeals), with specific finds color-coded. In this way, we process more than 100,000 documents. Why is this useful for us at Memex? - Because we are trying to find and parse court documents related to labor trafficking, so that we can analyze them and better understand the indicators of labor trafficking.

It is very exciting to work on the Memex program. Our team is called "Hyperion Gray" and has been featured in Forbes lately.

What's next? One of the plans is to add understanding of documents to FreeEed, the open source eDiscovery. Instead of just doing keyword searches through the document, the lawyers will be able, by the addition of text analytics, make more sense of the documents: detect people, dates, organizations, etc. This will in turn help create the picture of the case in an automated way.

Disclaimer: we are not official speakers for Memex.
.

FreeEed technologies led to DARPA project

FreeEed technologies impressed the DARPA team and led to a contract to fight human trafficking. The full press release by the main contractor, Hyperion Gray, is quoted below. While FreeEed and Elephant Scale can't have their own press release, their involvement is fully explained.

FreeEed does Concordance (R)

The latest release of FreeEed (V4.4) allows import into Concordance (R) eDiscovery management software. Here are the instructions. It also contains a number of fixes. You can use FreeEed in so many ways: Start a FreeEed server on Amazon, no hardware needed; Download a virtual machine to your workstations; Install in Windows, Linux, or Mac. Download page: here. And all of the popcorn advantages still apply. PS. Sneak preview: we are working on a document processing engine for today's 3V's - volume, velocity, variety. It is 10-100 times faster, and allows dynamic data sources.

Run FreeEed in the cloud, no downloads or hardware needed

Hi, all,

now we have another option to run FreeEed: on Amazon AWS cloud. There are three steps: (1) start the server, (2) connect to it with X2GO, (3) download and unzip the latest software. The rest of the environment is already prepared for you. Check it out here.

We are planning regular webinars teaching this setup, please write indicating your interest.

Cheers,
FreeEed team

We will be presenting FreeEed at a SNIA conference here, so we just updated our slide presentation. We though you would enjoy the more generic form of it, without too much technical detail, so here is the link.

Announcing FreeEed VM eDiscovery appliance

Hi, friends,

 we have packaged all the goodies of FreeEed into a VirtualBox machine, so no more install hassles. This includes the new release 4.2.0 with all the bug fixes and enhancements. Future plans? Adding data collection and advanced analytics tools.

 Cheers, all, and write back!

 Oh, and the download is here, http://freeeed.org/index.php/download

FreeEed survey results

Hi, all friends of open source eDiscovery project FreeEed, we got great feedback from our users, and here is what they want

• Easier to use search features
• Email threading
• Maintained archive of processed files (especially PST) for repeated searches.
• Social media analytics
• iCONECT integration - export to iCONECT XML to simplify loading into iCONECTnXT and XERA

This is awesome list, and we will be working on it.

Cheers,

FreeEed Team