Author Archives: Joe Monti
If you ever have to write MapReduce jobs or custom UDF or SerDe classes for Hive in Java, you will want to re-use memory as much as possible, meaning as few object and array allocations as possible, while also taking care not to inadvertently use/re-use data that is invalid or corrupted. This is an important practice […]
How many times have you SSH’d into a remote GNU/Linux server, ran a long-running command, only to come back to a stuck or disconnected SSH session? GNU Screen to the rescue! GNU Screen is perhaps one of the most subtle, useful, and difficult to understand commands in the GNU/Linux command arsenal. The important thing it does for […]
Shell Analytics is when you use command line tools to analyse or manipulate data. Command line tools are an invaluable tool for working with data, specifically files or command line programs which output useful data. Not only can you quickly preview the first or last few lines of a file quickly and easily (head/tail), you […]
I was watching an interesting “Leaders in Big Data” panel and one of the panelists, Charles Fan from VMWare, had a great name for Big Data (or what you do with it): Create Replicate Append Process Very fitting. If you are interested you can find the video here.
One of the first things you need to decide when building a new application or major feature is how you are going to store and process the data for it, which means picking the right database for the job. This is, of course, assuming you will need to store and retrieve data and your storage […]
Intel apparently is launching it’s own distribution of Hadoop as well as Project Rhino. Project Rhino is an “open-source effort to enhance security in Hadoop,” which makes Hadoop a more viable option for highly sensitive data. The Intel Hadoop distribution aims to optimize Hadoop for Intel Xenon platforms. I’m not convinced we need another distribution […]
Welcome to my new blog, Joe on Data. Hopefully the title of the blog is explanatory enough for what it is about. In it I will be writing about data; how to manage it, how to store it, how to process it, and what you can do with it. Data is everywhere and data is […]