COMP0233: Research Software Engineering With Python


Structured Data

Structured data

CSV files can only model data where each record has several fields, and each field is a simple datatype, a string or number.

We often want to store data which is more complicated than this, with nested structures of lists and dictionaries. Structured data formats like JSON, YAML, and XML are designed for this.


JSON is a very common open-standard data format that is used to store structured data in a human-readable way.

This allows us to represent data which is combinations of lists and dictionaries as a text file which looks a bit like a Javascript (or Python) data literal.

In [1]:
import json

Any nested group of dictionaries and lists can be saved:

In [2]:
mydata = {'key': ['value1', 'value2'], 
          'key2': {'key4':'value3'}}
In [3]:
'{"key": ["value1", "value2"], "key2": {"key4": "value3"}}'

If you would like a more readable output, you can use the indent argument.

In [4]:
print(json.dumps(mydata, indent=4))
    "key": [
    "key2": {
        "key4": "value3"

Loading data is also really easy:

In [5]:
%%writefile myfile.json
    "somekey": ["a list", "with values"]
Writing myfile.json
In [6]:
with open('myfile.json', 'r') as json_file:
    my_data_as_string = json_file.read()
In [7]:
'{\n    "somekey": ["a list", "with values"]\n}\n'
In [8]:
mydata = json.loads(my_data_as_string)
In [9]:
['a list', 'with values']

This is a very nice solution for loading and saving Python data structures.

It's a very common way of transferring data on the internet, and of saving datasets to disk.

There's good support in most languages, so it's a nice inter-language file interchange format.


YAML is a very similar data format to JSON, with some nice additions:

  • You don't need to quote strings if they don't have funny characters in
  • You can have comment lines, beginning with a #
  • You can write dictionaries without the curly brackets: it just notices the colons.
  • You can write lists like this:
In [10]:
%%writefile myfile.yaml
    - a list # Look, this is a list
    - with values
Writing myfile.yaml
In [11]:
import yaml  # This may need installed as pyyaml
In [12]:
with open('myfile.yaml') as yaml_file:
    my_data = yaml.safe_load(yaml_file)
{'somekey': ['a list', 'with values']}

YAML is a popular format for ad-hoc data files, but the library doesn't ship with default Python (though it is part of Anaconda and Canopy), so some people still prefer JSON for its universality.

Because YAML gives the option of serialising a list either as newlines with dashes or with square brackets, you can control this choice:

In [13]:
- a list
- with values

In [14]:
print(yaml.safe_dump(mydata, default_flow_style=True))
{somekey: [a list, with values]}

default_flow_style=False (the default) uses a "block style" (rather than an "inline" or "flow style") to delineate data structures. See the YAML docs for more details.


Supplementary material: XML is another popular choice when saving nested data structures. It's very careful, but verbose. If your field uses XML data, you'll need to learn a python XML parser (there are a few), and about how XML works.

Exercise: Saving and loading data

Use YAML or JSON to save your maze data structure to disk and load it again.