Close
Packaging¶
Once we've made a working program, we'd like to be able to share it with others.
A good cross-platform build tool is the most important thing: you can always have collaborators build from source.
Distribution tools¶
Laying out a project¶
When planning to package a project for distribution, defining a suitable project layout is essential. A typical layout might look like this:
repository_name
|-- module_name
| |-- __init__.py
| |-- python_file.py
| |-- another_python_file.py
| `-- test
| |-- fixtures
| | `-- fixture_file.yaml
| |-- __init__.py
| `-- test_python_file.py
|-- LICENSE.md
|-- CITATION.md
|-- README.md
`-- setup.py
To achieve this for our greetings.py file from the previous session, we can use the commands shown below. We can start by making our directory structure. You can create many nested directories at once using the -p switch on mkdir.
%%bash
mkdir -p greetings_repo/greetings/test/fixtures
For this notebook, since we are going to be modifying the files bit by bit, we are going to use the autoreload ipython magic so that we don't need to restart the kernel.
%load_ext autoreload
%autoreload 2
Using setuptools¶
To make python code into a package, we need to write a setup.py file. For now we are adding only the name of the package and its version number.
%%writefile greetings_repo/setup.py
from setuptools import setup, find_packages
setup(
name="Greetings",
version="0.1.0",
packages=find_packages(),
)
We can now install this "package" with pip:
%%bash
cd greetings_repo
pip install .
And the package will be then available to use everywhere on the system. But so far this package doesn't contain anythin and there's nothing we can run! We need to add some files first.
To create a regular package, we needed to have __init__.py files on each subdirectory that we want to be able to import. This is, since version 3.3 and the introduction of Implicit Namespaces Packages, not needed anymore. However, if you want to use relative imports and pytest, then you still need to have these files.
The __init__.py files can contain any initialisation code you want to run when the (sub)module is imported.
For this example, and because we are using relative imports in the tests, we are creating the needed __init__.py files.
%%bash
touch greetings_repo/greetings/__init__.py
And we can copy the greet function from the previous section in the greeter.py file.
%%writefile greetings_repo/greetings/greeter.py
def greet(personal, family, title="", polite=False):
greeting = "How do you do, " if polite else "Hey, "
if title:
greeting += f"{title} "
greeting += f"{personal} {family}."
return greeting
For the changes to take effect, we need to reinstall the library:
%%bash
cd greetings_repo
pip install .
And now we are able to import it and use it:
from greetings.greeter import greet
greet("Terry","Gilliam")
Convert the script to a module¶
Of course, there's more to do when taking code from a quick script and turning it into a proper module:
We need to add docstrings to our functions, so people can know how to use them.
%%writefile greetings_repo/greetings/greeter.py
def greet(personal, family, title="", polite=False):
""" Generate a greeting string for a person.
Parameters
----------
personal: str
A given name, such as Will or Jean-Luc
family: str
A family name, such as Riker or Picard
title: str
An optional title, such as Captain or Reverend
polite: bool
True for a formal greeting, False for informal.
Returns
-------
string
An appropriate greeting
Examples
--------
>>> from greetings.greeter import greet
>>> greet("Terry", "Jones")
'Hey, Terry Jones.
"""
greeting = "How do you do, " if polite else "Hey, "
if title:
greeting += f"{title} "
greeting += f"{personal} {family}."
return greeting
We can see the documentation using help.
help(greet)
The documentation string explains how to use the function; don't worry about this for now, we'll consider this on the next section (notebook version).
Write an executable script¶
We can create an executable script, command.py that uses our greeting functionality and the process function we created in the previous section.
Note how we are importing greet using relative imports, where .greeter means to look for a greeter module within the same directory.
%%writefile greetings_repo/greetings/command.py
from argparse import ArgumentParser
from .greeter import greet
def process():
parser = ArgumentParser(description="Generate appropriate greetings")
parser.add_argument('--title', '-t')
parser.add_argument('--polite', '-p', action="store_true")
parser.add_argument('personal')
parser.add_argument('family')
arguments = parser.parse_args()
print(greet(arguments.personal, arguments.family,
arguments.title, arguments.polite))
if __name__ == "__main__":
process()
Specify entry point¶
This allows us to create a command to execute part of our library. In this case when we execute greet on the terminal, we will be calling the process function under greetings/command.py.
%%writefile greetings_repo/setup.py
from setuptools import setup, find_packages
setup(
name="Greetings",
version="0.1.0",
packages=find_packages(),
entry_points={
'console_scripts': [
'greet = greetings.command:process'
]})
%%bash
cd greetings_repo
pip install -e .
And the scripts are now available as command line commands, so the following commands can now be run:
%%bash
greet --help
%%bash
greet Terry Gilliam
greet --polite Terry Gilliam
greet Terry Gilliam --title Cartoonist
Specify dependencies¶
Let's give some live to our output using ascii art
%%writefile greetings_repo/greetings/command.py
from argparse import ArgumentParser
from art import art
from .greeter import greet
def process():
parser = ArgumentParser(description="Generate appropriate greetings")
parser.add_argument('--title', '-t')
parser.add_argument('--polite', '-p', action="store_true")
parser.add_argument('personal')
parser.add_argument('family')
arguments = parser.parse_args()
message = greet(arguments.personal, arguments.family,
arguments.title, arguments.polite)
print(art("cute face"), message)
if __name__ == "__main__":
process()
We use the setup.py file to specify the packages we depend on using install_requires:
%%writefile greetings_repo/setup.py
from setuptools import setup, find_packages
setup(
name="Greetings",
version="0.1.0",
packages=find_packages(),
install_requires=['art', 'pyyaml'],
entry_points={
'console_scripts': [
'greet = greetings.command:process'
]}
)
When installing the package now, pip will also install the dependencies automatically.
%%bash
cd greetings_repo
pip install -e .
%%bash
greet Terry Gilliam
Installing from GitHub¶
We could now submit "greeter" to PyPI for approval, so everyone could pip install it.
However, when using git, we don't even need to do that: we can install directly from any git URL:
pip install git+git://github.com/ucl-rits/greeter
$ greet Lancelot the-Brave --title Sir
Hey, Sir Lancelot the-Brave.
There are a few additional text files that are important to add to a package: a readme file, a licence file and a citation file.
Write a readme file¶
The readme file might look like this:
%%writefile greetings_repo/README.md
# Greetings!
This is a very simple example package used as part of the UCL
[Research Software Engineering with Python](development.rc.ucl.ac.uk/training/engineering) course.
## Installation
```bash
pip install git+git://github.com/ucl-rits/greeter
```
## Usage
Invoke the tool with `greet <FirstName> <Secondname>` or use it on your own library:
```python
from greeting import greeter
greeter.greet(user.name, user.lastname)
```
Write a license file¶
We will discus more about licensing in a later section. For now let's assume we want to release this package into the public domain:
%%writefile greetings_repo/LICENSE.md
(C) University College London 2014
This "greetings" example package is granted into the public domain.
Write a citation file¶
A citation file will inform our users how we would like to be cited when refering to our software:
%%writefile greetings_repo/CITATION.md
If you wish to refer to this course, please cite the URL
http://github-pages.ucl.ac.uk/rsd-engineeringcourse/
Portions of the material are taken from [Software Carpentry](http://software-carpentry.org/)
You may well want to formalise this using the codemeta.json standard or the citation file format - these don't have wide adoption yet, but we recommend it.
Define packages and executables¶
We need to create __init__ files for the source and the tests.
touch greetings/greetings/test/__init__.py
touch greetings/greetings/__init__.py
Write some unit tests¶
We can now write some tests to our library.
Remember, that we need to create the empty __init__.py files so that pytest can follow the relative imports.
%%bash
touch greetings_repo/greetings/test/__init__.py
Separating the script from the logical module made this possible.
%%writefile greetings_repo/greetings/test/test_greeter.py
import os
import yaml
from ..greeter import greet
def test_greet():
with open(os.path.join(os.path.dirname(__file__),
'fixtures',
'samples.yaml')) as fixtures_file:
fixtures = yaml.safe_load(fixtures_file)
for fixture in fixtures:
answer = fixture.pop('answer')
assert greet(**fixture) == answer
Add a fixtures file:
%%writefile greetings_repo/greetings/test/fixtures/samples.yaml
- personal: Eric
family: Idle
answer: "Hey, Eric Idle."
- personal: Graham
family: Chapman
polite: True
answer: "How do you do, Graahm Chapman."
- personal: Michael
family: Palin
title: CBE
answer: "Hey, CBE Mike Palin."
We can now run pytest
%%bash --no-raise-error
cd greetings_repo
pytest
However, this hasn't told us that also the third test is wrong too! A better aproach is to parametrize the testfile greetings_repo/greetings/test/test_greeter.py as follows:
%%writefile greetings_repo/greetings/test/test_greeter.py
import os
import pytest
import yaml
from ..greeter import greet
def read_fixture():
with open(os.path.join(os.path.dirname(__file__),
'fixtures',
'samples.yaml')) as fixtures_file:
fixtures = yaml.safe_load(fixtures_file)
return fixtures
@pytest.mark.parametrize("fixture", read_fixture())
def test_greeter(fixture):
answer = fixture.pop('answer')
assert greet(**fixture) == answer
Now when we run pytest, we get a failure per element in our fixture and we know all that fails.
%%bash --no-raise-error
cd greetings_repo
pytest
We can also make pytest to check whether the docstrings are correct by adding the --doctest-modules flag. We run pytest --doctest-modules and obtain the following output:
%%bash --no-raise-error
cd greetings_repo
pytest --doctest-modules
Finally, if we don't want to include the tests when we distribute our software for our users, you can include that using the exclude option on find_packages on setup.py.
%%writefile greetings_repo/setup.py
from setuptools import setup, find_packages
setup(
name="Greetings",
version="0.1.0",
packages=find_packages(exclude=['*.test']),
install_requires=['art', 'pyyaml'],
entry_points={
'console_scripts': [
'greet = greetings.command:process'
]}
)
Developer Install¶
If you modify your source files, you would now find it appeared as if the program doesn't change.
That's because pip install copies the files.
If you want to install a package, but keep working on it, you can do:
pip install --editable .
or, its shorter version:
pip install -e .
Distributing compiled code¶
If you're working in C++ or Fortran, there is no language specific repository. You'll need to write platform installers for as many platforms as you want to support.
Typically:
dpkgforapt-geton Ubuntu and Debianrpmforyum/dnfon Redhat and Fedorahomebrewon OSX (Possiblymacportsas well)- An executable
msiinstaller for Windows.
Homebrew¶
Homebrew: A ruby DSL, you host off your own webpage
See an installer for the cppcourse example
If you're on OSX, do:
brew tap jamespjh/homebrew-reactor
brew install reactor