Process for Publishing Real-Time Data via DCC IS

This document outlines how to put in place a process for publishing real-time data via Dublin City Council (DCC) Information Services (IS) on the Smart Dublin Open Data Portal data.smartdublin.ie. Such a process is currently being used for the automated publication of DCC Public Bin Location data, as can be seen at https://data.smartdublin.ie/dataset/dcc-public-bin-locations

The publication process is depicted in the figure below.

1. The DCC Bin Location data is collated and managed by IS Services on a dedicated waste management system. 2. An FME Export process is created to select only the relevant properties to be published, namely: The Bin ID, the electoral area, the bin type, and the x and y coordinates. The FME process creates a CSV file with these properties. 3. A scripted FTP process is created to transfer this CSV file to the Smart Dublin AWS EC2 File Server. This transfer runs automatically on a monthly basis. 4. The transferred Bin Location CSV is updated to the Smart Dublin Open Data Portal on a monthly basis, so that the most recent data is available to the public. The dataset is available at https://data.smartdublin.ie/dataset/dcc-public-bin-locations

This process can be replicated for the publication of other data that is stored and managed by DCC IS. It can scale to large datasets, and also manage frequently updated and real-time data.