Tuesday, April 26, 2016

Big Data architecture for O&G

Houston Hadoop Meetup has grown to over 800 members by now. It is lavishly hosted by the Slalom consultants in the Galleria area, and beer, wine and food are provided by Slalom.

The presenter, Dmitry Kniazev gave an overview of the Proof-Of-Concept solution created for a major Oil & Gas company. He gave a brief overview of the WITSML standard that exists in the industry to share the sensor data among different operators, and described how they tapped to it to build the near real-time alerting application that streams data into Kafka queue and processes it using Spark Streaming.

Dmitry Kniazev is as a Solutions Architect, Data Analytics at EPAM Systems (NYSE: EPAM). EPAM is a solutions integrator that outsources solutions implementation to various locations, primarily Eastern Europe. Dmitry has been working with one of the major Oil & Gas companies here in Houston for almost 4 years and participated in various Data Analytics related projects.

The slides are found here. Again, thank you for hosting, presenting, and coming to the meeting.

Sunday, March 6, 2016

Hadoop as a service at Houston Hadoop Meetup

Hadoop as a service was presented by Ajay Jha, of Altiscale. Here are the slides.

As has become customary, our host, Slalom, provided parking ticket validation, pizza, beer and wine.

This location is in a fashionable Galleria area, where downstairs the geeks can continue Caracol restaurant - mexican coastal cuisine.

Thursday, February 25, 2016

Big Data Cartoons - Paris, Jerusalem, Istanbul, Singapore, where next?

In teaching Big Data, we often travel. Lately, in our view, Big Data is picking up the world over, not only in the US. Israeli Spark meetup are just as advanced as the ones in California. So we asked our artist to show  all the places where we have been. That was too hard though, so we just used travel pointers. But the elephant is real.

(In fact, this post is written on a Turkish Airlines plane - thanks to very good WiFi).

Big Data Cartoon - Elephant Scale enters Google doodle competition

Google 4 doodle competitions will be announced in March, https://www.google.com/doodle4google/ but we are already thinking, what's next? So we asked our artist for an official entry. Our artist is so great that there is a virtual guarantee to win - if  it is not disqualified because our subtle brand promotion. So here is our spoiler.

Monday, February 15, 2016

Stoppable Hadoop cluster

The title of this post was inspired by the following lines

They dined on mince, and slices of quince,
   Which they ate with a runcible spoon

from the poem by Edward Lear, "The Owl and the Pussy-Cat", which the reader is invited to ponder at leisure.

Meanwhile, as a Big Data trainer, I often need to create what I would call a "stoppable cluster" on AWS, one that I can "pause," or put to sleep for a while. The most obvious use of it is saving money while the students are away, so that instead of $100/day, I would pay $33 per day. That would be reason enough. However, at times, as a developer, I want to stop the cluster that I am running.

If you look in the literature, it will cite two obstacles:
  1. Ephemeral nodes disappear on stop/start on AWS; and
  2. IP assignments change.
You can fix both by (1) using root drive and EBS drives; and (2) assigning elastic IPs. Amazon will not let you use more than 5 elastic IPs, but you can call them and ask nicely, and they will give you 10. 

Next, Hortonworks Ambari will check the cluster IP assignment and refuse to use the external IPs, even though Amazon promises you the right resolution:

So I use Cloudera manager, stop the cluster, stop the instance, and restart it.

Now, I try to start the Managing services again and....

alas, CM has resolved the IPs to the old internal ones!! And used that in the configuration.

Next installment - constructing proper clusters in the VPC and controlling the internal IP assignments.

Tuesday, February 2, 2016

Big Data Cartoon - Portraits of Famous People - Cos

Today we inaugurate the series of blog posts called "Big Data Famous People."

This here is Konstantin Boudnik. A graduate of St. Petersburg University, Math Department, and with a PhD in distributed systems, Cos holds strong opinions in many areas of programming languages, philosophy and economics. You can ask him on his site here, or on the restricted version here.

Cos is the 16th most prolific committer to Hadoop. He helped incubate Ignite, Groovy, and a bunch of other projects.

Tuesday, January 26, 2016

Illustrated story of Houston Hadoop Meetup

 Five years ago we came up with this logo for our book, Hadoop Illuminated. The Meetup that I started in 2011 was then meeting in Houston Public Library, and numbered in the tens, with 3-5 showing up. When we wanted to do some Amazon clusters, we found out that the library blocked port 22. We started looking for other quarters.

 Time flew, Meetup grew, and we came to our first Hadoop bootcamp. We spent more money on going to Pappadeaux restaurant than than we made on the bootcamp, but fun was had by all.

Today, the meetup is close to 700, and the current event has about 70 registrants. We have two goals.

1. In 2011, Bay area Hadoop meetup had 3,000 members, and 300 would come to the Yahoo headquarters to hear about Hadoop HA. I was green with envy. Now we have a chance to growing more than they - let's try it.
2. There is still no significant Big Data in Houston. But there are real signs that this is going to change this year. You finally hear about Big Data projects in retails, energy, power, etc.

Guys, let's continue making Houston a Big Data capital. Cheers!