This is Joe on Data.

Author Archives: Joe Monti

Memory Management in Hadoop MapReduce

If you ever have to write MapReduce jobs or custom UDF or SerDe classes for Hive in Java, you will want to re-use memory as much as possible, meaning as few object and array allocations as possible, while also taking care not to inadvertently use/re-use data that is invalid or corrupted. This is an important practice […]

Amazon ElasticMapReduce and the Screen Command

How many times have you SSH’d into a remote GNU/Linux server, ran a long-running command, only to come back to a stuck or disconnected SSH session? GNU Screen to the rescue! GNU Screen is perhaps one of the most subtle, useful, and difficult to understand commands in the GNU/Linux command arsenal. The important thing it does for […]

Shell Analytics: command line tools for data analysis

Shell Analytics is when you use command line tools to analyse or manipulate data. Command line tools are an invaluable tool for working with data, specifically files or command line programs which output useful data. Not only can you quickly preview the first or last few lines of a file quickly and easily (head/tail), you […]

Playing With D3.js

D3.js is a “JavaScript library for manipulating documents based on data.” D3 works a little like jQuery, but is focused on providing a set of tools for binding data to visual components and a rich set of plugins for many visualization features beyond simple line, bar, or pie charts. Take a look at the gallery […]

Big Data is CRAP

I was watching an interesting “Leaders in Big Data” panel and one of the panelists, Charles Fan from VMWare, had a great name for Big Data (or what you do with it): Create Replicate Append Process Very fitting. If you are interested you can find the video here.

Picking the Right Database for Your Application

One of the first things you need to decide when building a new application or major feature is how you are going to store and process the data for it, which means picking the right database for the job. This is, of course, assuming you will need to store and retrieve data and your storage […]

Intel Launches Hadoop Distribution and Project Rhino

Intel apparently is launching it’s own distribution of Hadoop as well as Project Rhino. Project Rhino is an “open-source effort to enhance security in Hadoop,” which makes Hadoop a more viable option for highly sensitive data. The Intel Hadoop distribution aims to optimize Hadoop for Intel Xenon platforms. I’m not convinced we need another distribution […]

Welcome

Welcome to my new blog, Joe on Data. Hopefully the title of the blog is explanatory enough for what it is about. In it I will be writing about data; how to manage it, how to store it, how to process it, and what you can do with it. Data is everywhere and data is […]