Running your own PyPi repository

If you are heavily into Python development then hosting your own Python Package Index repository will give you some nice benefits.

  • local package caching
  • hosting private packages
  • testing package uploads

One of the options out there is DevPi which allows you to set up a working repository in minutes.

DevPi simple but powerful enough to handle common needs when it comes to Python repositories, and is open source which makes it easy to extend for your own purposes.

Start of with creating a virtual environment and installing the relevant DevPi packages. 

python3 -m venv /tmp/devpi
. /tmp/devpi/bin/activate
pip install devpi-web devpi-client

For this exercise I use /tmp/devpi as working directory.

The DevPi server require a directory to store data, and this must be an absolute path, not a relative one.
It’s not necessary to use an environment variable, but it’ll save some typing. 

export DATA=/tmp/devpi/data

Initialize the directory, and set the root user password.

devpi-init --serverdir $DATA
devpi-passwd --serverdir $DATA root

DevPi can also generate configuration files that makes it easier to run it as a service, check the documentation for the devpi-gen-config command.

Start the DevPi server and it automatically begins to synchronize with the offical PyPi repository. 

devpi-server --serverdir $DATA --restrict-modify root

By default DevPi starts a web server on http://localhost:3141, and unless the synchronization have completed there’ll be a large red warning. Just ignore it for a while, it should disappear once the process is completed.

Administrating the DevPi server is done with the command devpi. Begin with configuring it to use your instance. 

devpi use http://localhost:3141
devpi login root

Create a user named packages that’ll be used for deploying our own packages.

devpi user -c packages email=packages@example.org password=packages

Note: The password is sent in clear text, make sure you use SSL if not running locally.

The default package index is root/pypi which acts as a mirror and cache for the official PyPi repository index. But it is also unmodifiable so you can’t add your own packages to it.

The solution is to create your own package index that inherits from the root index, that way you have access to your own packages plus the official ones.

devpi index -c packages/stable bases=root/pypi volatile=False

The volatile flag indicates the index can not be modified, i.e. once a package is uploaded it can not be updated nor can the index be deleted.
For testing purposes let’s have an index which is non-volatile as well.

devpi index -c packages/staging bases=packages/stable volatile=True

The staging index will in turn inherit from our stable index, but allow for modifications. Useful for testing package deployments.

For pip to be aware of your new PyPi repository some additions must be made to pip.config (usually found in $HOME/.config/pip/). 

[global]
trusted-host = localhost
index-url = http://localhost:3141/packages/staging/+simple/
[search]
index = http://localhost:3141/packages/staging/

In the above example I chose to use the staging index, as it will contain the experimental uploads and pre-releases of packages.

Note: The trusted-host option is required for hosts not using SSL.

Adding your packages to the index is simple. First add the lines below to your $HOME/.pypirc file. 

[distutils]
index-servers =
 stable
 staging

[stable]
repository = http://localhost:3141/packages/stable/
username = packages

[staging]
repository = http://localhost:3141/packages/staging/
username = packages

Uploading packages is done using twine as usual, but the index must be specified using the --repository option.

Example below on uploading to staging.

cd ~/src/mypythonproject
python setup.py sdist bdist_wheel
twine upload --repository staging dist/*

That is all.

I noticed that sometimes DevPi have issues to download packages from the official repository, likely due to load balancing or issues with the CDN.

The simplest solution is to wait 3-5 minutes and try again with pip, but you can also use the server option --replica-max-retries to set the number of retries the server should do.
Just be careful not to hammer the PyPi servers, or you could end up being throttled not to mention blocked.

By Andreas • Updated at Sat, 30 May 2020