How to Open, Convert and Use Data across Borders:

Open | Use | Convert
How to Open, Convert and Use Data across Borders:
A Citadel on the Move Guideline
“Open data is data that can be freely used, reused and redistributed by anyone – subject
only, at most, to the requirement to attribute and share alike.” OpenDefinition.org
It seems that everyone these days is talking about Open Data and its
value to cities and citizens alike, particularly in relation to the
creation of innovative new mobile apps that enhance city living and
promote tourism.
Open data has the power to generate powerful economic,
environmental, and social value. As the Open Data Institute argues,
Open Data can help ‘unlock supply, generate demand, and create
and disseminate knowledge to address local and global issues.’1 In
addition to releasing social and commercial value, as the Open
Knowledge Foundation further argues, Open Data can also help to
promote transparency and trust in government as well as increased
citizen participation and engagement.2
Despite all the buzz about Open Data these days, however, one hears very little about the
practical steps that a city administrator needs to take to open local government data and
unleash innovation. Too often Open Data advocates, for all their commitment and passion,
forget that for many cities and towns, especially smaller ones with limited resources, opening
and using data is easier said than done.
Citadel on the Move is a project funded by the European Commission that aims to address this
challenge by providing cities and towns with simple and easy to use templates and tools to help
them open their data and create new applications that can be used and shared across Europe
and even around the world.
1
2
http://theodi.org/about-us
http://okfn.org/opendata/
© Citadel on the Move 2014
1
Open | Use | Convert
Citadel Approach
The following guidelines are designed to help local government actors use Citadel on the Move
tools and templates to kick start their Open Data efforts in 3 easy steps: 1) open up data by
making it available for access by third-party applications; 2) convert the data into a format that
makes easier to create applications; and 3) use the Citadel App Generator to generate value
from data by creating innovative applications that do not as yet exist.
The Citadel website provides more detailed instructions for troubleshooting issues as they may
arise.3 Getting started, however, can be as simple as:
Step 1: Open4
Citadel recommends that cities follow three key principles when embarking on their Open Data
journey:
1. Start Small: You don’t need to open all your data at once. Far better to see what works
and doesn’t work than to not start at all.
2. Accept Mistakes: No dataset is perfect. Opening up data will inevitably reveal
inaccuracies. The good news is that it will also help to correct them.
3. Engage Citizens: Remember – your data is only valuable if it is used. Tell your citizens
that you are making data available and encourage them to use it.
Citadel recommends that cities take the following steps when opening their data:
1. Select the Data Set: Remember – simple and small can be good! The key thing is that the
data is publically available and has no IPR restrictions on the way it can be used, reused
or redistributed.
3
For more detailed instructions, or trouble shooting, see: www.citadelonthemove.eu/appgenerator and
www.citadelonthemove.eu/converter
4
Citadel on the Move has devised these recommendations in partnership with the Open Data Institute and the Open
Knowledge Foundation.
© Citadel on the Move 2014
2
Open | Use | Convert
2. Publish the Data Set: For ease of use with Citadel tools and templates, use CSV – a very
simple form that can be saved from Excel, Open Office and most spreadsheet
programmes.5 Try to use non-proprietary formats (such as CSV instead of Excel) and
structured formats (e.g. not scanned images, PDF etc). But most important of all, put
your data out there. If someone wants to use it they will find a way!
3. Publicise the Data Set: Data needs to be discovered and discoverable to be used. Make
sure to post you data on your website or a central catalogue, and tell your citizens it is
there!
4. Keep your Data Up-to-Date: Expectations for current, if not real-time, data are growing.
Remember to publish new files as amended or updated data becomes available.
There will always be certain circumstances, of course, in which it is simply not feasible to
freely open all data. Wherever possible, though, Citadel on the Move strongly recommends
adherence to the norms of the Open Data Commons wherein the users and providers of
data agree to share and share alike.6 By freely opening and sharing data in this way, local
governments can play a vital role in helping to build strong data user communities that, in
turn, create a virtuous circle of openness in which the innovative use and reuse of data
stimulates the further unleashing of more and more free and open data. The first step is to
open up that initial data set, however small, and let your citizens know!
Step 2: Convert
NOTE – these instructions assume that a) the reader has a basic level of technical knowledge in
handling simple datasets and 2) the reader is able to access their dataset in CSV format.
Citadel-on-the-Move is not just about helping to open data, it is also about making sure that,
once open, the data is used. To support local administrators aiming at this objective, Citadel has
created a new tool to ‘convert’ data released as a CSV file into a format (JSON7) that is
compatible with Citadel’s App Generator – which has been specifically designed to make it easy
for just about anyone with basic computer skills, even a nine year old, to create a mobile app
that works not only in their own city but also across borders.8
5
NB: Citadel on the Move is not arguing that CSV is the preferred approach for all open data initiatives, not least of all
because it only works when the dataset is small, static and simply formed. In its simplicity, however, Citadel believes that
CSV is a good place way for municipalities with limited resources and expertise to get started. Cities that are further along
in their Open Data efforts should examine the Open Data recommendations of W3C - http://www.w3.org/TR/gov-data/ as well as new Linked Data initiatives e.g. LOD2 project http://stack.lod2.eu/blog/.
6
http://opendatacommons.org/norms/odc-by-sa/
7
Currently Citadel templates use JSON files. As a result, the starting version of the Citadel Converter converts CSV datasets
to JSON files. To continue to make it as easy as possible to use Citadel tools and templates, Citadel will shortly release a
new Converter for JAVA. Citadel envisions the future development of additional Converters for other formats, as well as
new templates that work with additional files.
8
Different cities often publish the same type of data in different ways. These differences create a large interoperability
challenge because for a mobile app to work across city borders because (which is a core goal for Citadel) the datasets must
be compatible. To overcome this challenge, Citadel-on-the-Move conversion to a common format that is compatible with
the Citadel mobile app templates is required.
© Citadel on the Move 2014
3
Open | Use | Convert
To use the Citadel Converter, follow these basic steps:
1. Make sure your dataset is clean and well structured, and saved as a CSV file
2. Access the Citadel Converter: http://www.rbox.tv/citadel/converter/php/
3. Upload the CSV file
4. Review the preview that pops up (the first 5 rows of datasets) to check that everything
appears correctly
5. If required, refine the following CSV presets and repeat the preview until the dataset
displays correctly:
a. Delimiter
b. Enclosure
c. Escape Character
6. Follow the on-screen instructions to manually map the headings of your dataset to the
values of the Citadel Dataset Template
7. Select the Latitude and Longitude fields that enable data to be referenced on a map9
8. Complete the necessary metadata fields when prompted
9. Click convert and wait for the JSON file to be produced
10. Download JSON file to your computer
Given its capacity to fuel innovation and enhance overall quality of life, Citadel on the Move
believes that it is important to think of data as a shared public good. For this reason, we
recommend uploading JSON to your own publicly accessible servers and, ideally, create a
reference to this location in the Open Data Catalogue on the Citadel Hub:
http://www.citadelonthemove.eu/en-us/thehub/opendata.aspx. Referencing your data in the
Citadel Hub will help in the creation of a common public space in which everyone is free to
access data and applications on behalf of the public good.
9
If your dataset does not contain the Latitude and Longitude values, these can be simply obtained from the following URL
http://www.mapcoordinates.net/en
© Citadel on the Move 2014
4
Open | Use | Convert
Step 3: Use
To use your data in Citadel template applications10, follow these simple steps:
1. Access the Citadel App Generator:
http://demos.citadelonthemove.eu/app-generator/appForm.php
2. Select the Cities you want the app to work in
3. Select the relevant Dataset(s) by either of the following:
a. Uploading the JSON files directly from your computer
b. Copying the dataset URL into the box provided
4. Select the colour theme and name of the application
5. Access your application hosted on the Citadel Hub and freely share it via its URL
There are a wide variety of potential technical procedures and norms
that local governments can follow as they start to open their data.
Citadel on the Move believes, however, that regardless of the tools and
templates used, the key to success lies in getting started – one data set
at a time, and getting users engaged – one application at a time. Data is
only as valuable as its use. The more people who use data the more
value is created, and the greater the incentive to open data becomes.
Citadel partners are already using Citadel templates and the App
Generator to create new apps of their own! Thanks to Citadel, tourists
to Gent can easily find the city’s wide array of art galleries:
10
The Citadel App Generator is intended to make it quick and easy for anyone with basic computer skills to turn data into a
new mobile app. Existing Citadel templates represent a starting set which can be extended in the future.
© Citadel on the Move 2014
5
Open | Use | Convert
Conclusion
The Citadel approach is quite different to the standard way of publishing Open Data - which
normally consists of publishing somewhat complex datasets as web services and then releasing
them through an API. Our goal here is to simplify this process by helping to quickly get online
the bulk of public data that is currently sitting as Excel files. In so doing, we aim to significantly
de-professionalise the Open Data process in order to extend it to broader communities of users
beginning with the public sector.
This White Paper has explained HOW local administrators can publish data more easily and help
make sure that it is used. We hope we have demonstrated just how simple the process can be.
The reason WHY is just as simple: Opening data in a reusable format gives citizens and local
enterprises the raw material they need to build new applications that take advantage of all the
possibilities that modern technologies now offer to improve public services and ultimately make
our cities and towns better places to work and live.
www.citadelonthemove.eu
Citadel on the Move
@Citadel_eu
Citadel on the Move is a 3 year project part funded by the European Commission’s ICT Policy
Support Programme (ICT PSP) under the Competiveness and Innovation Programme (CIP)
Citadel on the Move is led by the Flemish eGovernment Authority (CORVE) supported by:
© Citadel on the Move 2014
6