Thursday, February 25, 2016

Big Data Cartoons - Paris, Jerusalem, Istanbul, Singapore, where next?

In teaching Big Data, we often travel. Lately, in our view, Big Data is picking up the world over, not only in the US. Israeli Spark meetup are just as advanced as the ones in California. So we asked our artist to show  all the places where we have been. That was too hard though, so we just used travel pointers. But the elephant is real.

(In fact, this post is written on a Turkish Airlines plane - thanks to very good WiFi).

Big Data Cartoon - Elephant Scale enters Google doodle competition

Google 4 doodle competitions will be announced in March, but we are already thinking, what's next? So we asked our artist for an official entry. Our artist is so great that there is a virtual guarantee to win - if  it is not disqualified because our subtle brand promotion. So here is our spoiler.

Monday, February 15, 2016

Stoppable Hadoop cluster

The title of this post was inspired by the following lines

They dined on mince, and slices of quince,
   Which they ate with a runcible spoon

from the poem by Edward Lear, "The Owl and the Pussy-Cat", which the reader is invited to ponder at leisure.

Meanwhile, as a Big Data trainer, I often need to create what I would call a "stoppable cluster" on AWS, one that I can "pause," or put to sleep for a while. The most obvious use of it is saving money while the students are away, so that instead of $100/day, I would pay $33 per day. That would be reason enough. However, at times, as a developer, I want to stop the cluster that I am running.

If you look in the literature, it will cite two obstacles:
  1. Ephemeral nodes disappear on stop/start on AWS; and
  2. IP assignments change.
You can fix both by (1) using root drive and EBS drives; and (2) assigning elastic IPs. Amazon will not let you use more than 5 elastic IPs, but you can call them and ask nicely, and they will give you 10. 

Next, Hortonworks Ambari will check the cluster IP assignment and refuse to use the external IPs, even though Amazon promises you the right resolution:

So I use Cloudera manager, stop the cluster, stop the instance, and restart it.

Now, I try to start the Managing services again and....

alas, CM has resolved the IPs to the old internal ones!! And used that in the configuration.

Next installment - constructing proper clusters in the VPC and controlling the internal IP assignments.

Tuesday, February 2, 2016

Big Data Cartoon - Portraits of Famous People - Cos

Today we inaugurate the series of blog posts called "Big Data Famous People."

This here is Konstantin Boudnik. A graduate of St. Petersburg University, Math Department, and with a PhD in distributed systems, Cos holds strong opinions in many areas of programming languages, philosophy and economics. You can ask him on his site here, or on the restricted version here.

Cos is the 16th most prolific committer to Hadoop. He helped incubate Ignite, Groovy, and a bunch of other projects.