Structured Data¶

Structured data¶

CSV files can only model data where each record has several fields, and each field is a simple datatype, a string or number.

We often want to store data which is more complicated than this, with nested structures of lists and dictionaries. Structured data formats like JSON, YAML, and XML are designed for this.

JSON¶

JSON is a very common open-standard data format that is used to store structured data in a human-readable way.

This allows us to represent data which is combinations of lists and dictionaries as a text file which looks a bit like a Javascript (or Python) data literal.

In [1]:

import json

Any nested group of dictionaries and lists can be saved:

In [2]:

mydata = {'key': ['value1', 'value2'], 
          'key2': {'key4':'value3'}}

In [3]:

json.dumps(mydata)

Out[3]:

'{"key": ["value1", "value2"], "key2": {"key4": "value3"}}'

If you would like a more readable output, you can use the indent argument.

In [4]:

print(json.dumps(mydata, indent=4))

{
    "key": [
        "value1",
        "value2"
    ],
    "key2": {
        "key4": "value3"
    }
}

Loading data is also really easy:

In [5]:

%%writefile myfile.json
{
    "somekey": ["a list", "with values"]
}

Writing myfile.json

In [6]:

with open('myfile.json', 'r') as json_file:
    my_data_as_string = json_file.read()

In [7]:

my_data_as_string

Out[7]:

'{\n    "somekey": ["a list", "with values"]\n}\n'

In [8]:

mydata = json.loads(my_data_as_string)

In [9]:

mydata['somekey']

Out[9]:

['a list', 'with values']

This is a very nice solution for loading and saving Python data structures.

It's a very common way of transferring data on the internet, and of saving datasets to disk.

There's good support in most languages, so it's a nice inter-language file interchange format.

YAML¶

YAML is a very similar data format to JSON, with some nice additions:

You don't need to quote strings if they don't have funny characters in
You can have comment lines, beginning with a #
You can write dictionaries without the curly brackets: it just notices the colons.
You can write lists like this:

In [10]:

%%writefile myfile.yaml
somekey:
    - a list # Look, this is a list
    - with values

Writing myfile.yaml

In [11]:

import yaml  # This may need installed as pyyaml

In [12]:

with open('myfile.yaml') as yaml_file:
    my_data = yaml.safe_load(yaml_file)
print(mydata)

{'somekey': ['a list', 'with values']}

YAML is a popular format for ad-hoc data files, but the library doesn't ship with default Python (though it is part of Anaconda and Canopy), so some people still prefer JSON for its universality.

Because YAML gives the option of serialising a list either as newlines with dashes or with square brackets, you can control this choice:

In [13]:

print(yaml.safe_dump(mydata))

somekey:
- a list
- with values

In [14]:

print(yaml.safe_dump(mydata, default_flow_style=True))

{somekey: [a list, with values]}

default_flow_style=False (the default) uses a "block style" (rather than an "inline" or "flow style") to delineate data structures. See the YAML docs for more details.

XML¶

Supplementary material: XML is another popular choice when saving nested data structures. It's very careful, but verbose. If your field uses XML data, you'll need to learn a python XML parser (there are a few), and about how XML works.

Exercise: Saving and loading data¶

Use YAML or JSON to save your maze data structure to disk and load it again.