Installing distributions

Where things are installed

User site and virtual environments

By default, distil installs in the user site (see PEP 370 for more information). This means that you don’t need to e.g. use sudo to install packages for your personal use. If you specify a virtual environment using the -e flag, installation is done into that environment.

System site-packages

Because of its alpha status, it is not advisable to use distil to install into a system-wide location. However, distil does support this if you use the --system flag when installing, for test purposes. You will typically need administrative privileges (via sudo on POSIX systems) to do this.

Arbitrary directories

You can specify an arbitrary directory to install to through a --target argument passed to the install command. If this is specified, only the library components are installed to that directory: headers, scripts, out-of-package data and package metadata are not installed. This means that such installations cannot be uninstalled, as there is no metadata storing a list of what files were installed.

Installing new distributions

Before installing a distribution, we can check to see what’s currently installed (see Getting information about distributions for more information about the list command):

$ distil -e e2 list
pip   1.3.1

Currently, only pip (and setuptools, which doesn’t show up in the list) are installed in this venv.

Pure-python distributions

Let’s try installing a pure-Python distribution:

$ distil -e e2 install sarge
Checking requirements for sarge (0.1) ... done.
The following new packages will be downloaded and installed:
    sarge (0.1)
Downloading sarge-0.1.tar.gz to /tmp/tmpG5RzKC
    35KB @ 175 KB/s 100 % Done: 00:00:00
Unpacking ... done.
Building sarge (0.1) ...
    Running check ...
    Running build_ext ...
    Running build_py ...
    Build completed.
Installing sarge (0.1) ...
    Running install_headers ...
    Running install_lib ...
    Running install_distinfo ...
    Installation completed.

As you can see from the above, distil employs a set of steps when installing which should be familiar to users of distribute and pip. However, the implementation is different – there is no use of command classes, for example.

If we again list the installed distributions, we get the expected result:

$ distil -e e2 list
sarge 0.1
pip   1.3.1

Bootstrapping pip

You can use distil to bootstrap pip.

Let’s see it bootstrapping pip on a vanilla machine which doesn’t have pip installed. First we’ll check for pip being available already:

vinay@nadia-cinnamon ~ $ which pip
vinay@nadia-cinnamon ~ $

So, there’s no pip on this machine. I have a PEP 370 user site set up at ~/.local, and it’s on my PATH:

vinay@nadia-cinnamon ~ $ echo $PATH
/home/vinay/.local/bin:/home/vinay/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

Now let’s install pip using distil:

vinay@nadia-cinnamon ~ $ distil install pip
Checking requirements for pip (1.3.1) ... done.
The following new packages will be downloaded and installed:
    pip (1.3.1)
Downloading pip-1.3.1.tar.gz to /tmp/tmpSdyftQ
    241KB @  45 KB/s 100 % Done: 00:00:05
Unpacking ... done.
Building pip (1.3.1) ...
    Running check ...
    Running build_ext ...
    Running build_py ...
    Build completed.
Installing pip (1.3.1) ...
    Running install_headers ...
    Running install_lib ...
    Running install_scripts ...
    Running install_distinfo ...
    Installation completed.
vinay@nadia-cinnamon ~ $ which pip
/home/vinay/.local/bin/pip
vinay@nadia-cinnamon ~ $ pip --version
pip 1.3.1 from /home/vinay/.local/lib/python2.7/site-packages (python 2.7)
vinay@nadia-cinnamon ~ $

Now, let’s install using Python 3.2:

$ distil -p python3.2 install pip
Checking requirements for pip (1.3.1) ... done.
The following new packages will be downloaded and installed:
    pip (1.3.1)
Downloading pip-1.3.1.tar.gz to /tmp/tmpc6nk63
    241KB @  14 KB/s 100 % Done: 00:00:17
Unpacking ... done.
Building pip (1.3.1) ...
    Running check ...
    Running build_ext ...
    Running build_py ...
    Build completed.
Installing pip (1.3.1) ...
    Running install_headers ...
    Running install_lib ...
    Running install_scripts ...
    Running install_distinfo ...
    Installation completed.
vinay@nadia-cinnamon ~ $ pip --version
pip 1.3.1 from /home/vinay/.local/lib/python3.2/site-packages (python 3.2)
vinay@nadia-cinnamon ~ $

Distributions which include C extensions

Let’s try installing a distribution which contains a C extension:

$ distil -e e2 install simplejson
Checking requirements for simplejson (3.1.0) ... done.
The following new packages will be downloaded and installed:
    simplejson (3.1.0)
Downloading simplejson-3.1.0.tar.gz to /tmp/tmp2aRbHT
    63KB @ 166 KB/s 100 % Done: 00:00:00
Unpacking ... done.
Building simplejson (3.1.0) ...
    Running check ...
    Running build_ext ...
Building 'simplejson._speedups' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -fPIC -I/tmp/tmp2aRbHT/simplejson-3.1.0 -I/usr/include/python2.7 -c -o /tmp/tmp2aRbHT/simplejson-3.1.0/build/temp.linux-x86_64-2.7/simplejson/_speedups.o /tmp/tmp2aRbHT/simplejson-3.1.0/simplejson/_speedups.c
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro /tmp/tmp2aRbHT/simplejson-3.1.0/build/temp.linux-x86_64-2.7/simplejson/_speedups.o -L/tmp/tmp2aRbHT/simplejson-3.1.0/build/temp.linux-x86_64-2.7 -L/usr/lib -o /tmp/tmp2aRbHT/simplejson-3.1.0/build/lib.linux-x86_64-2.7/simplejson/_speedups.so
    Running build_py ...
    Build completed.
Installing simplejson (3.1.0) ...
    Running install_headers ...
    Running install_lib ...
    Running install_distinfo ...
    Installation completed.

Notice that distil invoked the C compiler to build simplejson‘s C extension. This also works on Windows - simply run the command in a “Visual Studio Command Prompt” window:

http://i.imgur.com/MiudqKK.png

Upgrading existing installed distributions

By default. distil only upgrades distributions if you ask it to. To illustrate, let’s install a distribution which has a newer version:

$ reuse e2
$ distil -e e2 install "Jinja2 (2.5.5)"
Checking requirements for Jinja2 (2.5.5) ... done.
The following new packages will be downloaded and installed:
    Jinja2 (2.5.5)
Downloading Jinja2-2.5.5.tar.gz to /tmp/tmpLBegg4
    428KB @  28 KB/s 100 % Done: 00:00:15
Unpacking ... done.
Building Jinja2 (2.5.5) ...
    Running check ...
    Running build_ext ...
    Running build_py ...
    Build completed.
Installing Jinja2 (2.5.5) ...
    Running install_headers ...
    Running install_lib ...
    Running install_distinfo ...
    Installation completed.

Now, let’s not specify the version:

$ distil -e e2 install Jinja2
Checking requirements for Jinja2 (2.6) ... done.
The following installed packages will be upgraded:
    Jinja2 (2.5.5) -> Jinja2 (2.6)
Proceed? (y/n)

In this case, distil assumes you want to upgrade the existing Jinja2 from 2.5.5 to 2.6. If you elect to proceed, the upgrade will be carried out. Instead, let’s try installing a distribution which requires Jinja2 and can work with version 2.5.5:

$ distil -e e2 install Flask
Checking requirements for Flask (0.9) ... done.
The following new packages will be downloaded and installed:
    Werkzeug (0.8.3) [for Flask]
    Flask (0.9)
Proceed? (y/n)

As you can see, distil only offers to install the additional dependency – Werkzeug – and to use the existing Jinja2 installation. If you want to upgrade to the latest version of Jinja2, you can invoke:

$ distil -e e2 install --upgrade Flask
Checking requirements for Flask (0.9) ... done.
The following new packages will be downloaded and installed:
    Werkzeug (0.8.3) [for Flask]
    Flask (0.9)
The following installed packages will be upgraded:
    Jinja2 (2.5.5) -> Jinja2 (2.6) [for Flask]
Proceed? (y/n)

Now, distil offers to upgrade Jinja2 to the latest version, as well as downloading and installing Werkzeug. If we confirm that we want to proceed, distil does the expected thing:

Downloading Werkzeug-0.8.3.tar.gz to /tmp/tmpWZ7TB6 [for Flask]
    1082KB @  56 KB/s 100 % Done: 00:00:19
Unpacking ... done.
Downloading Jinja2-2.6.tar.gz to /tmp/tmp1SMzzt [for Flask]
    380KB @ 131 KB/s 100 % Done: 00:00:02
Unpacking ... done.
Downloading Flask-0.9.tar.gz to /tmp/tmpuhUOcp
    470KB @  52 KB/s 100 % Done: 00:00:09
Unpacking ... done.
Building Werkzeug (0.8.3) ...
    Running check ...
    Running build_ext ...
    Running build_py ...
    Build completed.
Building Jinja2 (2.6) ...
    Running check ...
    Running build_ext ...
    Running build_py ...
    Build completed.
Building Flask (0.9) ...
    Running check ...
    Running build_ext ...
    Running build_py ...
    Build completed.
Installing Werkzeug (0.8.3) ...
    Running install_headers ...
    Running install_lib ...
    Running install_distinfo ...
    Installation completed.
Installing Jinja2 (2.6) ...
    Running install_headers ...
    Running install_lib ...
    Running install_distinfo ...
    Installation completed.
Installing Flask (0.9) ...
    Running install_headers ...
    Running install_lib ...
    Running install_distinfo ...
    Installation completed.

Listing the distributions shows the expected results:

$ distil -e e2 list
Werkzeug 0.8.3
Flask    0.9
Jinja2   2.6
pip      1.3.1

Dependency handling

As you see from the above examples, distil automatically detects a distribution’s dependencies and downloads and installs them, after getting confirmation from the user. (No confirmation is requested if only a single distribution – the one requested by the user – would be downloaded and installed.)

Installing from wheels

To install from a wheel, you just specify the path of the wheel when you invoke the install command:

$ distil -e e2 install /tmp/Flask-0.9-py27-none-any.whl
Checking requirements for Flask (0.9) ... done.
The following new packages will be downloaded and installed:
    Jinja2 (2.6) [for Flask]
    Werkzeug (0.8.3) [for Flask]
The following new packages will be installed from a local location:
    Flask (0.9)
Proceed? (y/n)

As with distributions installed by name, any dependencies declared in a wheel will be downloaded and installed.

See Binary distributions for information on how to build wheels.

Using requirements files

As well as specifying individual distributions on the command line, you can also specify requirements files, which have the requirements:

$ cat reqts.txt
Flask
Sphinx
$ distil -e e2 install -r reqts.txt
Checking requirements for Flask (0.9) ... done.
Checking requirements for Sphinx (1.1.3) ... done.
The following new packages will be downloaded and installed:
    Jinja2 (2.6) [for Sphinx, Flask]
    Pygments (1.6) [for Sphinx]
    docutils (0.10) [for Sphinx]
    Werkzeug (0.8.3) [for Flask]
    Sphinx (1.1.3)
    Flask (0.9)
Proceed? (y/n)

Installing just library components

You can install to an arbitrary directory rather than a site-packages location by specifying --target:

$ distil install --target /tmp/scratch sarge

The directory must exist. If given, only library components are written to it: headers, scripts, out-of-package data and package metadata are not written. If you use this option, you will not be able to uninstall from the target location (since no metadata is written indicating what was installed).

Finding pre-releases during dependency resolution

As per PEP 426, distil will return pre-releases if they are all that is available. To specify that pre-releases are included when doing dependency resolution (i.e. treated like normal releases), you can specify --prereleases. This flag works with all commands which do dependency resolution and location of distributions.

Installing from version control systems

Currently, distil does not support installing from VCS repositories. This is not hard to do – you just check out an appropriate revision to a temporary directory, and install from there – but installation requires a package.json to be present (to indicate source/data files, build options etc.). Most projects will not have this file (yet), making the install-from-VCS feature one of limited usefulness.

Command line reference – distil install

Here is the complete help for distil‘s install command:

$ distil help install
usage: distil install [-h] [-y] [-u] [--prereleases] [-s] [-t DESTDIR]
                      [-r REQTFILE [REQTFILE ...]]
                      [REQT [REQT ...]]

Install one or more distributions.

positional arguments:
  REQT                  A requirement using a distribution on PyPI, or the
                        path of a wheel file.

optional arguments:
  -h, --help            show this help message and exit
  -y, --yes             Don't ask for confirmation before installing.
  -u, --upgrade         Upgrade dependencies if possible. The default
                        behaviour is to upgrade dependencies only if
                        necessary.
  --prereleases         Include pre-releases when installing. By default, pre-
                        releases are skipped, unless they are all that is
                        available.
  -s, --system          Install to system Python (may require sufficient
                        privileges). By default, installations are written to
                        the user site (~/.local), or to a virtual environment
                        specified with -e.
  -t DESTDIR, --target DESTDIR
                        Install modules and packages to the specified
                        directory. Scripts, headers, non-package data and
                        package metadata are not installed. This means that
                        you can't uninstall anything installed in this way.
  -r REQTFILE [REQTFILE ...]
                        Get requirements from specified file(s)