At the recent meeting, David Ramirez of Informatica presented the platform and explained how it allows to save time in data modeling when it comes to Big Data. His slides are found here, and they also contain a link to a demo on YouTube.
Keep in mind that David is working with Oil & Gas accounts in Houston - so you may get some interesting information from the slides.
Pizza was provided by Elephant Scale, and the meeting hosted by Microsoft. Thanks to Tiru for helping with hosting.
Wednesday, September 16, 2015
Thursday, September 3, 2015
Big Data Cartoon - Hadoop is quite mature
Signs of Hadoop maturity:
- There is a competitor, Spark
- There is consolidation: Cloudera, Hortonworks, MapR dominate
- Promising startups are snatched: Hortonwork acquired Onyara, the maker of NiFi
- The elephant himself grew up - see this picture :)
Thursday, August 20, 2015
Mike Drob presents Cloudera Search at Houston Hadoop Meetup
Cloudera's value add an open source, feature rich search engine plus all the integrations with a data management (also open source) ecosystem, to streamline multi-workload search, or search and other workloads of the same data, without moving it around between systems. Cloudera also provides production tooling, audit, and security.
In addition, open source buffs can use the implementation described in these slides, to glean the best practices to use in their own solutions.
Very good, clear discussion - thank you, Mike!
Thanks again, Microsoft, for hosting the meetup at MS Campus.
Wednesday, July 22, 2015
Review of “Monitoring Hadoop” by Gurmukh Singh
This book is recently published, April 2015, and it covers Nagios, Ganglia, Hadoop monitoring and monitoring best practices.
The first part is rightfully devoted to Nagios. Nagios is covered quite in depth: install, verification and configuration. It gives you the right balance: it does not say everything that there is in a Nagios manual, but tells you sufficient information to install Nagios and prepare it to monitor specific Hadoop daemons, ports, and hardware.
The same goes for Ganglia: it is covered in sufficient detail for one to be able to install and run, with enough attention to Hadoop specifics.
What I did not find in the book, and what could be useful... to read further
Review of “Hadoop in Action,” second edition
Four years have passed since the first publication, and as Russians say, “A lot of water has passed (under the bridge) since then,” so let’s look at what’s new in this edition.
Tuesday, July 7, 2015
The power of text analytics at DARPA/Memex
One of the things we are doing in the DARPA Memex program is text analytics. One of the outcomes of it is an open source project called MemexGATE.
By itself, GATE stands for Generic Architecture for Text Engineering, and it is a mature and widely-used tool. It is up to you to create something useful with GATE, and MemexGATE is our first step. This is an application configured to understand court documents. It will detect people mentioned in the documents, dates, places, and many more characteristics that take you beyond plain key word searches.
To achieve this, GATE combines processing pipelines (such as sentence splitter, language-specific word tokenizer, part of speech tagger, etc) with gazetteers. Now, what is a gazetteer? -- It is a list of people, places etc. that can occur in your documents. MemexGATE includes scripts that collect all US judges, for example, so that they can be detected, when found in a document.
But MemexGATE does more: it is scalable. Building on the Behemot framework, it can parallelise processing for the Hadoop cluster, thus putting no limit on the size of the corpus. MemexGATE was designed and implemented by Jet Propulsion Lab team, and the project committer is Lewis McGibbney.
The picture shown above gives an example of a processed document (from NY court of appeals), with specific finds color-coded. In this way, we process more than 100,000 documents. Why is this useful for us at Memex? - Because we are trying to find and parse court documents related to labor trafficking, so that we can analyze them and better understand the indicators of labor trafficking.
It is very exciting to work on the Memex program. Our team is called "Hyperion Gray" and has been featured in Forbes lately.
What's next? One of the plans is to add understanding of documents to FreeEed, the open source eDiscovery. Instead of just doing keyword searches through the document, the lawyers will be able, by the addition of text analytics, make more sense of the documents: detect people, dates, organizations, etc. This will in turn help create the picture of the case in an automated way.
Disclaimer: we are not official speakers for Memex.
.
By itself, GATE stands for Generic Architecture for Text Engineering, and it is a mature and widely-used tool. It is up to you to create something useful with GATE, and MemexGATE is our first step. This is an application configured to understand court documents. It will detect people mentioned in the documents, dates, places, and many more characteristics that take you beyond plain key word searches.To achieve this, GATE combines processing pipelines (such as sentence splitter, language-specific word tokenizer, part of speech tagger, etc) with gazetteers. Now, what is a gazetteer? -- It is a list of people, places etc. that can occur in your documents. MemexGATE includes scripts that collect all US judges, for example, so that they can be detected, when found in a document.
But MemexGATE does more: it is scalable. Building on the Behemot framework, it can parallelise processing for the Hadoop cluster, thus putting no limit on the size of the corpus. MemexGATE was designed and implemented by Jet Propulsion Lab team, and the project committer is Lewis McGibbney.
The picture shown above gives an example of a processed document (from NY court of appeals), with specific finds color-coded. In this way, we process more than 100,000 documents. Why is this useful for us at Memex? - Because we are trying to find and parse court documents related to labor trafficking, so that we can analyze them and better understand the indicators of labor trafficking.
It is very exciting to work on the Memex program. Our team is called "Hyperion Gray" and has been featured in Forbes lately.
What's next? One of the plans is to add understanding of documents to FreeEed, the open source eDiscovery. Instead of just doing keyword searches through the document, the lawyers will be able, by the addition of text analytics, make more sense of the documents: detect people, dates, organizations, etc. This will in turn help create the picture of the case in an automated way.
Disclaimer: we are not official speakers for Memex.
.
Friday, July 3, 2015
Big Data Cartoons - Summer of Big Data
Subscribe to:
Posts (Atom)







