demonCoreData

Photo by r2hox (Flickr/Creative Commons) In a recent blog post on Northern Ireland’s Renewal Heat Incentive (RHI) scandal [ here ] I spent quite a bit of time recording all of the changes, tweaks, and decisions I had to make to get the data into a usable format. With any dataset it is important to understand the transformations that went into bringing it to its final form. If other researchers are unable to follow your process and consistently achieve the same results from the same dataset it brings your analysis into question. Beyond that, it brings the whole endeavour of data science and data analysis into disrepute. If you can’t rely on the figures to tell a consistent story, you can’t make consistent decisions, and you can’t gain reliable insights. You certainly can’t trust the folks who are furnishing you this flawed and unreliable nonsense. If you can’t rely on the information you’re seeing on your dashboard, what is it other than a collection of interesting, but meaningles...

Screenshot of the Tableau Dashboard. Available [ here ] and at the end of this post. (Updated: see notes at end) After much legal wrangling and foot-dragging, the Northern Ireland Department of the Economy have finally published a partial list of recipients of money from the botched Renewable Heat Initiative scheme . At present only limited companies and limited liability partnerships who received in excess of £5,000 (cumulative) are listed. The data runs from the start of the scheme to 28 February 2017. After the list was published (16 March 2017) a number of people complained that they should treated as individuals, and not as limited companies. These corrections were made and a second list was issued the same afternoon. The dataset used here is based on this second list. The first thing I want to note about the document made available by the Department of the Economy is that it is presented as a PDF. This is data! To analyse data you need it in a suitable format su...