Houston Hadoop Meetup, September 2012, guest post from the presenter, Dianhui Zhu

This is what Dianhui (a.k.a Dennis) reported.

Dianhui (Dennis) Zhu presented his work on “Genomic Data Analysis with Hadoop”. Some of the Hadoopers who didn’t attend his first presentation were very interested in his work and requested this 2nd presentation.

The presentation was very successful. It started with a short description of the challenges that the researchers are facing in the field of genomic data analysis. It was then followed by a use case study on the use of Hadoop framework to detect genomic sequence patterns.

Dianhui actually walked Houston Hadoopers through the code and showed how to test it.. In the last 20 minutes, Dianhui demonstrated how to setup a 4-node Hadoop cluster from scratch and then did a live demo on the cluster. A performance comparison was conducted based on a real dataset. The attendants saw that the speed on the 4-node cluster was roughly three times faster than on a single node.

The meetup was held for the first time at the SHMsoft's new office location. Hadoopers were energized by pizza (thanks to the hospitality of SHMsoft). The presentation was accompanied by a lively interactive session with excellent questions and answers. It was finished with a perfect live demo.

See y'all there next time!

Art: Vassily Kandinsky - Untitled First Abstract Watercolor

SHMsoft Week in Review - September 02, 2012

Let's take a quick look at what happened at SHMsoft last week

• SHMcloud V4.1.6 has been released, and now it is using a better way of distributing our software, called Elastic Block Storage, or EBS, which is a prerequisite for being listed on AWS Marketplace.
• OCR integrated and tested, locally and on the cloud.
• Formulated a product development roadmap, based on market research, expert input and projected funding.
• Through our partnership with Hortonworks, we have established a relationship with a  NY law firm interested in Hadoop/Big Data eDiscovery.
• We are working with a big Legal Process Outsourcing (LPO) firm, which is looking forward to testing and suggesting improvements in the SHMcloud software for their needs.
• Got a new office at Caroline Collective at 4820 Caroline, Houston, Suite 103, TX 77004.
• Got a new adviser, Big Data architect Amit Rathore.
• Started working on search engine optimization (SEO), to drive traffic to our website.
• Started working on the web site itself, to better explain our offerings.

FreeEed gets a home

FreeEed got a home, and SHMsoft got an office - to better support our users. The place is called Caroline Collective, and the feeling is just like in the Silicon Valley. Plus, it's in the Museum district, so in that sense it is like Cision in Chicago, where you can walk to the museum (with the famous lions).

SHMsoft / FreeEed / SHMcloud weekly news, 8/26/12

• SHMcloud V4.1.4 has been released; GUI bugs fixed.
• OCR tested, adding it to SHMcloud install on Amazon.
• Started on specifications for search in Early Case Assessment (ECA).
• Started on phase 1 of Big Data social media marketing project for a large local retailer.
• Staff: an experienced data scientist leads this Big Data social media project.
• Having been accepted as an APN Amazon partner, SHMsoft is in the final stages of getting listed on AWS Marketplace with our SHMcloud(TM) product.
• A large LPO firm in India started working on SHMcloud, to make sure that the product satisfies their needs for scalable processing.
• A new financial consultant started working on SHMsoft’s business and financial plan.

SHMsoft / FreeEed / SHMcloud weekly news, 8/19/12

• SHMcloud V4.1.2 has been released; it features all processing parameters arranged in tabbed dialog for the convenience of the operator, and more parameters have been added
• OCR capability is in testing, expected to be released soon
• Staff additions: a software developer, a Hadoop administrator, an eDiscovery project manager, and an eDiscovery business advisor all started this past week
• We are in meetings with two energy companies, who are planning to use SHMcloud as part of their internal eDiscovery handling
• A New York law firm working with Hortonworks saw our Hadoop-based eDiscovery and asked for a presentation next week
• SHMsoft has become an official member of the Amazon Partner Network (APN), of Rice Alliance for Technology and Entrepreneurship, and was selected to present to the judges of the Goradia startup competition

A week at SHMsoft - 8/10

Here are some of the projects that we are involved with at the moment, as well as a comprehensive view on what is happening internally at SHMsoft.

• We are now officially partnered with Hortonworks, a leader in Hadoop distributions. You can find us on their website here and here.
• We proposed the usage of our SHMcloud software to the Center for Computer Assisted Legal Instruction www.cali.org and presented a case for integration in their curriculum for e-Discovery. We are also currently negotiating with another big name in the "law-technology" field, who found our current offerings to be of great value for their business.  Check back for more information on the subject!
• Realizing the potential and flexibility of our software offerings, a large local retailer has reached out to us in order to engage in an innovative marketing project based on big data. It is a whole new approach that has the potential to introduce new ways of approaching big data and expand our product offerings.
• Our entry for the Goradia Innovation Prize attracted the attention of a large local chemical firm, and we have already scheduled a meeting to help them understand how our software can add value to specific aspects of their business.
• Our expertise base is growing! The quality and potential of the SHMsoft product has caught the attention of leading industry figures and experts.  Our latest additions include individuals with many years of experience in the legal industry and the e-Discovery field. As a result, management and marketing of our SHMcloud platform is now under a newly formed team that includes key company employees as well as external advisers, all working towards the same goal - to make our offering the top solution in the field.
• On the same topic, a new developer with extensive experience in Hadoop and cloud administration is about to join the SHMsoft team!
• We have also been busy presenting to more and more potential investors that are fascinated by our work. First round of financing is under way. Stay tuned for more good news!
• It has just been confirmed that our CEO will be attending the Hadoop World Conference from October 24-25 in New York.
• On the software side, we never stop working to make our products better. Our SHMcloud Player now has improved interface and numerous fixes in its version 4.1.1 RC (Release Candidate).

August Houston Hadoop Meetup - Hands-on with Hortonworks

The slides for this presentation are found here. My goal was to play with the Hortonworks distribution of Hadoop, called called HDP, which stands for Hortonworks Data Platform. I also wanted to see what would be involved in porting my eDiscovery (legal) application, called FreeEed, which uses generic Hadoop and runs on EC2, to HDP.

(I had to learn to spell HDP, because my fingers tend to type HPD, which stands of Houston Police Department. I used search for HPD, and fixed it).

There were many things on the Hortonworks web site that I liked, in particular, the short videos. I also understood the positioning of Hortonworks, "We are new as a company, but we are the birthplace of Hadoop, and these same Yahoo people are now working at Hortonworks. We are also all completely open source." I liked the Talend Open Studio, featured on the site.

Then I started the real work. HDP works only on RedHat or CentOS. That was not a problem, since on EC2 I could get any flavor of it. I chose the basic. Note that it comes with only 6 GB of hard drive. For a quick test, I kept this, but watched the harddrive with df through the installs. For more serious use, I would recommend resizing the drive. Here is an instruction on how to do it: http://labs.thredup.com/resizing-the-root-disk-on-a-running-ebs-boot. I've used it successfully many times before. A little setup of your dev machine is required, but that's only done once.

Here is an important note: if you are installing remotely, like I did on the EC2 machine, you don't need to have GUI access to that machine. Just start installing the hmc, or the configuration manager, and it will start a web server. Then you will be able to point your browser to this remote web server, and continue the install.

You get a choice of many Hadoop services that you can install. As you can see from my slides, something did not work when installing HBase. I did not bother fixing that (I am sure it was possible), but instead chose to re-install without HBase. It worked. This was more important for me - I wanted to know if the install can be repeated, and yes, it can.

See the rest of the setup on the slides' screenshots.

My next two questions were, is there an instruction to doing all this automation manually, and how do I re-create my own Hadoop install Java-based software. Rohit of Hortonworks answered both: they are writing an instruction for manual setup - for those people who have their own puppet and chef and other automated environments. With my eDiscovery setup, I will have to wait until there is an instruction, and then repeat Cloudera steps, but with HDP.

The meetup itself was great as always: some very deeply technical people who work with Hadoop or related technology, a very tech savvy investment manager, and some new people who want to jump into the sea of Big Data. Thank you, all!