Processing Enron data on a 49-node cluster

By now, all our Hadoop clusters, regardless of size, take about 5 minutes to come up. For the HALSM presentation in Houston yesterday we took the Enron data residing in our S3 Amazon account, and duplicated some of that. The total volume was about 50 GB zipped, so it was over 100 GB unzipped.

Then we used the SHMcloud(TM) Player to start the cluster and run the processing. It took slightly over one hour. Here are some screenshots.

It was fun to show processing live, and to poke around the servers for the audience.

The next goal is to reduce this time to 30 minutes, by better load balancing.

Big clusters for eDiscovery

Every programmer knows that special pleasure and satisfaction when his or her code works right, and with more and more testing, and with more and more data. The special joy of clusters is when it works with any size of cluster.

The SHMcloud (TM) player is now able to start and configure all the machines in a Hadoop cluster at once. This means that a cluster of 1 machine takes five minutes, the cluster of 20 machines takes five minute, and the cluster of 50 or 100 machines also takes five minutes - the latter when Amazon approves my request for more instances :)

Update: got my limit raised to 50!

Then you can verify this in the AWS console.

And, don't forget to shut them down!














Update 2: the nice folks at Amazon gave me 50 machines the next day. Now the cluster looks like this:


-rw-r--r--   1 ubuntu supergroup          0 2012-06-14 21:33 /test-output/_SUCCE
SS
drwxr-xr-x   - ubuntu supergroup          0 2012-06-14 21:32 /test-output/_logs
-rw-r--r--   1 ubuntu supergroup        172 2012-06-14 21:33 /test-output/part-0
0000

12-06-14 16:33:30   Cluster testing and verification is complete
setInitializedState for cluster of 49
12-06-14 16:33:33   Running instances: 49
12-06-14 16:33:33   Completely initialized: 49
setInitializedState for cluster of 49

Gioia gioia mille anni!

47 working nodes (49 total - memory master - work master) working together!

Houston Hadoop Meetup June - Hands-On!

A sign of a good meetup is when people don't want to leave. This happened again now: the librarian wanted us out, because the library was closing, and we, being good compassionate human beings, preempted her a by a little. Even so, people were sighted talking next to the closed library entrance for the next half hour at least.

Why the excitement? Developers love challenges, and here they were challenged enough: the hands-on format had something for everyone, and if you already completed your assignment, you were given the next level of complexity.

We found our next presenter, Dennis, who will be talking about his genome project, and how they use Hadoop at the Medical Center - details to follow soon. After that, the proposed format is to take one chapter in the Hadoop Illuminated book, and combine the presentation based on that chapter with the exercises found at the end of the chapter. This will also give me a push to complete the book. And the companion project, of course!

A la prochaine!

Art:  Jean Charles Meissonier F- Two Men Talking In A Tavern

SHMsoft and FreeEed in the news

LTN's Evan Koblenz writes this 

SHMsoft Tests Open Source E-Discovery App in Cloud

SHMsoft, which makes the FreeEed open-source e-discovery application, is expanding its product lineup into cloud computing and growing its staff.

.....

Matt Toomey, an e-discovery blogger from Albany, New York, recently posted about his tests of the core FreeEed and said he's encouraged about the program's future.

Automated Hadoop Cluster Construction on EC2

Hi, all,

that was the topic of today's presentation, and here are the slides: http://www.slideshare.net/markkerzner/automated-hadoop-cluster-construction-on-ec2

However, it was more than that, because we had a lively discussion, almost an unconference, where people asked questions, gave explanations, and described their business cases.

The case in point: we kept talking next to the library for another hour. To an outsider (and there was one police car watching us), it was a veritable Silicon Valley meeting, what with the startup and technology discussions.

See you next time on Richmond, another hands-on session. However, other presenter ideas are welcome.

The most awesome Houston Hadoop Meetup Ever

Hi, all,

we had the most interesting meeting. It's one thing to sit and listen, and it is completely another to do it yourself. But, paradoxically, this is what interests people the most.

We had a more-than-expected turnout, and everybody was burning to build their own clusters, run them on Amazon, and achieve first-hand experience of what others have only an "I heard the presentation" knowledge.

So, of course, everybody had their own environments, things did not work out at first try, you had to debug and adjust, install and re-install, and it was so much fun that we could not stop until 9 PM, or however late it was that we all had to go home.

Resolutions for the next meetup:

1. We will separate into two groups, beginners and advanced, and we will have two teachers;
2. I will finish publishing whatever chapters I have of Hadoop Illuminated, with all the instructions for Hadoop on EC2;
3. We may need a new venue, at least for this coming May meeting, and we may consider the Microsoft compound nearby.

See you all next time!

If you need to do searches in the FreeEed results...

Then now, with Hive, you can.

Imagine you processed some good amount of eDiscovery data with FreeEed™. Up until now, FreeEed did not give it you capability to search in the load file. You had to either load it in Concordance, or into another tool, open it in Excel (up a certain point), or bring it into a database. And yet, with large data size, all these approaches would eventually either be very slow, or have a limit on the size of the data, like Excel does. Now FreeEed™ gives you the capability to search the results. Here is how.

You select the menu item "Load with Hive." Hive is an open source tool, part of the Hadoop family, which allows you to query the results with a language similar to SQL. Actually, HiveQL, much more powerful than SQL.

Spoiler: in the background, FreeEed™ writes the Hive scripts and loads your load file into Hive, like this (you can see this going on in the History window):


12-03-19 23:41:14   Running command: hive -f /tmp/hive_load_table.sql
12-03-19 23:41:21   Hive history file=/tmp/mark/hive_job_log_mark_201203192341_371336996.txt
12-03-19 23:41:21   Copying data from file:/home/mark/projects/FreeEed/freeeed-output/0009/output/run-120319-233739/results/metadata.txt
12-03-19 23:41:21   Loading data to table default.load_file
12-03-19 23:41:21   OK
12-03-19 23:41:21   Time taken: 5.078 seconds
12-03-19 23:41:21   Running command: xterm -e hive


Now you can ask the data anything you want to know. For example, in response to your question, who is talking about energy at Enron, it will come back, after a few seconds, with the UPI (Unique Production Identifier) of the documents you are interested in. Why a few seconds? Because Hive uses the same MapReduce technology as FreeEed™ and runs on the same cluster. Therefore, it can handle any amount of data, but it has a small overhead to start the Hadoop job.


This feature is included in the RC release on our site, and (sorry, Windows fans), this feature does require running on Linux, with Hadoop in pseudo-distributed mode and Hive installed. If that install is intimidating, do not despair! Soon we will offer even better features in the SHMcloud ™ premium platform, whether in the cloud or inside your computing centers.

A la prochaine! (French for see you soon), from my favorite book, "French for Cats, All The French Your Cat Will Ever Need".