A week at SHMsoft - 8/10

Here are some of the projects that we are involved with at the moment, as well as a comprehensive view on what is happening internally at SHMsoft.

• We are now officially partnered with Hortonworks, a leader in Hadoop distributions. You can find us on their website here and here.
• We proposed the usage of our SHMcloud software to the Center for Computer Assisted Legal Instruction www.cali.org and presented a case for integration in their curriculum for e-Discovery. We are also currently negotiating with another big name in the "law-technology" field, who found our current offerings to be of great value for their business.  Check back for more information on the subject!
• Realizing the potential and flexibility of our software offerings, a large local retailer has reached out to us in order to engage in an innovative marketing project based on big data. It is a whole new approach that has the potential to introduce new ways of approaching big data and expand our product offerings.
• Our entry for the Goradia Innovation Prize attracted the attention of a large local chemical firm, and we have already scheduled a meeting to help them understand how our software can add value to specific aspects of their business.
• Our expertise base is growing! The quality and potential of the SHMsoft product has caught the attention of leading industry figures and experts.  Our latest additions include individuals with many years of experience in the legal industry and the e-Discovery field. As a result, management and marketing of our SHMcloud platform is now under a newly formed team that includes key company employees as well as external advisers, all working towards the same goal - to make our offering the top solution in the field.
• On the same topic, a new developer with extensive experience in Hadoop and cloud administration is about to join the SHMsoft team!
• We have also been busy presenting to more and more potential investors that are fascinated by our work. First round of financing is under way. Stay tuned for more good news!
• It has just been confirmed that our CEO will be attending the Hadoop World Conference from October 24-25 in New York.
• On the software side, we never stop working to make our products better. Our SHMcloud Player now has improved interface and numerous fixes in its version 4.1.1 RC (Release Candidate).

August Houston Hadoop Meetup - Hands-on with Hortonworks

The slides for this presentation are found here. My goal was to play with the Hortonworks distribution of Hadoop, called called HDP, which stands for Hortonworks Data Platform. I also wanted to see what would be involved in porting my eDiscovery (legal) application, called FreeEed, which uses generic Hadoop and runs on EC2, to HDP.

(I had to learn to spell HDP, because my fingers tend to type HPD, which stands of Houston Police Department. I used search for HPD, and fixed it).

There were many things on the Hortonworks web site that I liked, in particular, the short videos. I also understood the positioning of Hortonworks, "We are new as a company, but we are the birthplace of Hadoop, and these same Yahoo people are now working at Hortonworks. We are also all completely open source." I liked the Talend Open Studio, featured on the site.

Then I started the real work. HDP works only on RedHat or CentOS. That was not a problem, since on EC2 I could get any flavor of it. I chose the basic. Note that it comes with only 6 GB of hard drive. For a quick test, I kept this, but watched the harddrive with df through the installs. For more serious use, I would recommend resizing the drive. Here is an instruction on how to do it: http://labs.thredup.com/resizing-the-root-disk-on-a-running-ebs-boot. I've used it successfully many times before. A little setup of your dev machine is required, but that's only done once.

Here is an important note: if you are installing remotely, like I did on the EC2 machine, you don't need to have GUI access to that machine. Just start installing the hmc, or the configuration manager, and it will start a web server. Then you will be able to point your browser to this remote web server, and continue the install.

You get a choice of many Hadoop services that you can install. As you can see from my slides, something did not work when installing HBase. I did not bother fixing that (I am sure it was possible), but instead chose to re-install without HBase. It worked. This was more important for me - I wanted to know if the install can be repeated, and yes, it can.

See the rest of the setup on the slides' screenshots.

My next two questions were, is there an instruction to doing all this automation manually, and how do I re-create my own Hadoop install Java-based software. Rohit of Hortonworks answered both: they are writing an instruction for manual setup - for those people who have their own puppet and chef and other automated environments. With my eDiscovery setup, I will have to wait until there is an instruction, and then repeat Cloudera steps, but with HDP.

The meetup itself was great as always: some very deeply technical people who work with Hadoop or related technology, a very tech savvy investment manager, and some new people who want to jump into the sea of Big Data. Thank you, all!

July Houston Hadoop Meetup - Genomic data analysis with Hadoop

Dianhui (Dennis) Zhu  presented "Genomic data analysis with hadoop".  He talked about using Hadoop framework to do pattern search in genomic sequence datasets. This is based on his three-year project at Baylor, which started using Hadoop a year ago. Dennis is Senior Scientific Programmer at HGSC.

Dianhui told us about the following issues

1. Setup a Hadoop test cluster with 4 nodes.
2. Code walk through and unit testing with Mokito and MRUnit
3. Live demo: running our Hadoop application on the  4-node cluster.

The interesting technical problem that Dennis showed was to break sequence into chunks, before it gets to the Mapper - which is usually trivial in the regular applications, but is quite hard with unlimited unstructured data of the genome. The audience analyzed the actual code, asked many questions, and wanted to compare to the existing open source projects.

Indeed, here is an article on the Cloudera blog, http://www.cloudera.com/blog/2009/10/analyzing-human-genomes-with-hadoop/, and it refers to the Crossbow open source project, http://bowtie-bio.sourceforge.net/crossbow/index.shtml. It will interesting to see how that compares to Dennis's work.

FreeEed™ is now in the cloud!

SHMcloud™ Press Release 7/9/12

FreeEed™ is now in the cloud!

eDiscovery processing: text extraction, culling, and native/text and metadata csv delivery.

Special introductory offer until August 15: $1 per machine-hour. How fast is that? At a recent show we processed 100 GB of Enron data in 1 hour for under $100, as seen here.

How can you get started?
⇢ Just go to here,
⇢ download the SHMcloud(TM) Player,
⇢ and start!

Email: info@shmsoft.com http://www.shmsoft.com Phone: 713-568-9753

Processing Enron data on a 49-node cluster

By now, all our Hadoop clusters, regardless of size, take about 5 minutes to come up. For the HALSM presentation in Houston yesterday we took the Enron data residing in our S3 Amazon account, and duplicated some of that. The total volume was about 50 GB zipped, so it was over 100 GB unzipped.

Then we used the SHMcloud(TM) Player to start the cluster and run the processing. It took slightly over one hour. Here are some screenshots.

It was fun to show processing live, and to poke around the servers for the audience.

The next goal is to reduce this time to 30 minutes, by better load balancing.

Big clusters for eDiscovery

Every programmer knows that special pleasure and satisfaction when his or her code works right, and with more and more testing, and with more and more data. The special joy of clusters is when it works with any size of cluster.

The SHMcloud (TM) player is now able to start and configure all the machines in a Hadoop cluster at once. This means that a cluster of 1 machine takes five minutes, the cluster of 20 machines takes five minute, and the cluster of 50 or 100 machines also takes five minutes - the latter when Amazon approves my request for more instances :)

Update: got my limit raised to 50!

Then you can verify this in the AWS console.

And, don't forget to shut them down!














Update 2: the nice folks at Amazon gave me 50 machines the next day. Now the cluster looks like this:


-rw-r--r--   1 ubuntu supergroup          0 2012-06-14 21:33 /test-output/_SUCCE
SS
drwxr-xr-x   - ubuntu supergroup          0 2012-06-14 21:32 /test-output/_logs
-rw-r--r--   1 ubuntu supergroup        172 2012-06-14 21:33 /test-output/part-0
0000

12-06-14 16:33:30   Cluster testing and verification is complete
setInitializedState for cluster of 49
12-06-14 16:33:33   Running instances: 49
12-06-14 16:33:33   Completely initialized: 49
setInitializedState for cluster of 49

Gioia gioia mille anni!

47 working nodes (49 total - memory master - work master) working together!

Houston Hadoop Meetup June - Hands-On!

A sign of a good meetup is when people don't want to leave. This happened again now: the librarian wanted us out, because the library was closing, and we, being good compassionate human beings, preempted her a by a little. Even so, people were sighted talking next to the closed library entrance for the next half hour at least.

Why the excitement? Developers love challenges, and here they were challenged enough: the hands-on format had something for everyone, and if you already completed your assignment, you were given the next level of complexity.

We found our next presenter, Dennis, who will be talking about his genome project, and how they use Hadoop at the Medical Center - details to follow soon. After that, the proposed format is to take one chapter in the Hadoop Illuminated book, and combine the presentation based on that chapter with the exercises found at the end of the chapter. This will also give me a push to complete the book. And the companion project, of course!

A la prochaine!

Art:  Jean Charles Meissonier F- Two Men Talking In A Tavern