Maintaining a Set of Private Python Packages with Nix

Digging into Nix packaging, and I thought I’d share what I found about maintaining Python packages outside of the core NixPkgs repository

up

2015-07-13

I’m very new to http://nixos.org/[NixOS]. I’ve been following it for a couple years on and off, and recently I’ve had a bigger need to do better packaging, so I’m taking a deeper look at Nix.

I very well may be completey wrong. If you do find a mistake, I would really love to correct it, so please PM me (see the footer of this page)!

Overview

My first goal was to setup a configuration such that a Python (2.7) package didn’t have to be in the public package definition in order to be used. For now, while I’m still experimenting and exploring, I’ve decided to, basically, extend the core expression(s) in NixPkgs like so:

# private-python27.nix
with import <nixpkgs> {};
let
    pythonPackages = pkgs.python27Packages;
    buildPythonPackage = pythonPackages.buildPythonPackage;

    privatePythonPackages = {
        inherit pythonPackages;

        mypackage = buildPythonPackage rec {
            name = "mypackage-${version}";
            version = "0.0.1";

            src = pkgs.fetchurl {
                url = "https://example.server/${name}-py2-none-any.whl";
                sha256 = "0da519rqvail7fwbjsq1hnx0b3368df6hvqj0a6j7xvfafgaeb43";
            };

            meta = {
                homepage = "https://example.server/";
                description = "Example Package";
                license = "GPLv2";
            };
        };
    };
in privatePythonPackages

The above nix expression defines the nix package mypackage for python 2.7 (full name: python27-mypackage-0.0.1).

It needs to be used with the ‘–file’ (or ‘-f’) parameter in order to be evaluated, like so:

$ nix-env -f private-python27.nix -qa python2.7-mypackage

For the time being, this gives me a nice local place to tweak and explore the Nix expression language, without the need for any additional infrastructure. I may not proceed down this path, but it certainly is teaching me a lot!

Deeper Dive

Let’s take a look at each line in this definition:

with import <nixpkgs> {};

This pulls everything from the nixpkgs expression into the current expressions scope.

let
    privatePythonPackages = {
        # ...
    };
in privatePythonPackages

This creates the expression privatePythonPackages which is defined in the let-in block.

    pythonPackages = pkgs.python27Packages;
    buildPythonPackage = pythonPackages.buildPythonPackage;

These define values to use else where in the current scope. The pkgs.python27Packages value describes some handy python related things that are applicable to my example package.

        mypackage = buildPythonPackage rec {
        };

This is the actual package definition – in Nix expressions, it can be referred to by mypackage. The rec means that the following set (sets are defined by {}) is “recursive” and individual elements of the set can refer to each other. buildPythonPackage is one of those things that is conveniently defined in the nixpkgs import. This is the nix expression that defines buildPythonPackage.

What does it do exactly? Well, it’s just kind of a helper for generating the build that gets created for a package in it’s /nix/store/ directory, and works in conjuction with a few other parts of nix.

            name = "mypackage-${version}";
            version = "0.0.1";

Since the set is identified as rec, the name value can include a reference to version. The name attribute is required and will be part of the name that can be used when interacting with a channel – IE nix-env -qa .\*mypackage.*.

            src = pkgs.fetchurl {
                url = "https://example.server/${name}-py2-none-any.whl";
                sha256 = "0da519rqvail7fwbjsq1hnx0b3368df6hvqj0a6j7xvfafgaeb43";
            };

The pkg.fetchurl is a convenience function that, suffice to say, will pull down the url that you would like, and then verifies it’s sha256 hash matches that which is given. Since Nix is lazily evaluated, the fetch isn’t performed until src is used, and src will only be used when the package is to be installed.

Interestingly enough, you can use the nix-prefetch-url to figure out it’s sha256 hash and cache it at the same time. This command basically fetches the file and puts it in the /nix/store, and part of it’s output is giving the sha256 that it generates for the file. Check out the wiki page on it for more info!

            meta = {
                homepage = "https://example.server/";
                description = "Example Package";
                license = "GPLv2";
            };

And finally, this is a set containing some meta information about the package, which is generally just extra information that isn’t actually required for a package to be installed.