Package versioning

This post descibes my approach to versioning packaged applications, using version numbers that look like this: 3.2.0-2109.12272.

The first part (3.2.0) is a manually incremented version number. It’s incremented for changes that need to be communicated and talked about - “our new feature will be in version 4, which will be deployed next week”.

The second part (2109.12272) is an automatically incremented release number. It’s entirely managed by the CI/CD process that packages the software. In my case, that’s done using GitLab, and the number is composed from the pipeline and job identifiers.

Both of these numbers increase monotonically, so that a each time the CI/CD process runs it builds a package with a higher version and release number without a human needing to remember to manually increase it. The automatic release number also ensures that you never build a package with the same version number as an existing one, and so as long as you keep older packages around you can always downgrade to a previous version.

The approach works very neatly with packaging tools like RPM, which have seperate version and release fields. My build process uses a version.txt and release.txt file, the latter of which is created at build time. Both are included in the package name and in the package itself, allowing the application to read them at runtime to display it’s own version number.

You could quite easily remove the manually incremented version number and loose no technical capability. However, I find simple, easy to remember numbers much easier to communicate than the long digit sequences an automatic release number will provide. Asking someone to check if they are running “version 4 or above” is much simpler than asking if the version number is “12034 or above” - people remember small numbers much more easily. You can also still use familiar SemVer like semantics to describe the scale of change - “we’re upgrading from version 9034 to 9725” says very little about the changes involved, whereas “we’re upgading from version 4.1 to 5.3” implies multiple large changes, or “we’re upgading from version 4.1.0 to 4.1.1” implies a small fix. I find this is especially useful when communicating with non-technical people who use the application.

However, a downside to this is that humans will likely end up being very lax about changing the version number. This is fine for me - I only use it to commuicate major changes and not small patches - but is unlikely to work for all cases. It’s particually unsuited for libraries or APIs, where tracking API changes is important and strictly following SemVer may be important.

Managing version numbers in Python

The rest of this post goes describes an implementation of the above approach that I use for Python packages.

A release.txt file includes the manually updated version number, and an optional release.txt file includes the CI/CD pipeline and job number. The __version__ attribute in my Python module is set by reading from these files instead of being statically defined.

__init.py__
"""An example package."""

import example.version

__author__ = 'Sam Clements'
__version__ = example.version.read_version(__file__)

version.py
"""Read version numbers from an embedded text file."""

import pathlib


def read_version(relative_path):
    version_path = pathlib.Path(relative_path).with_name('version.txt')
    release_path = pathlib.Path(relative_path).with_name('release.txt')

    version = version_path.read_text().strip()

    if release_path.exists():
        release = release_path.read_text().strip()
        version = "{}-{}".format(version, release)

    return version

This is included in the Python package metadata by using the attr: directive in the setuptools setup.cfg file (a relatively recent feature that allows package metadata to be set by reading values from a Python module). For this to work properly, the __init__ file has to import as little as possible, so that it’s not including dependencies that may not be installed when you first build or install the package. For my projects, the __init__.py file normally only includes metadata and absolute essentials (e.g. exit handlers with no dependencies).

[metadata]
name = example
version = attr: example.__version__
...

With everything setup to build the application as a Python package, I also use this in a Makefile that builds an RPM with the Python package using fpm.

CI_PIPELINE_ID?=0
CI_JOB_ID?=0

NAME=$(shell python setup.py --name)
VERSION=$(shell cat odyssey/version.txt)
RELEASE=$(CI_PIPELINE_ID).$(CI_JOB_ID)

example/release.txt:
	echo "$(RELEASE)" > "$@"
	
$(NAME)-$(VERSION)-$(RELEASE).x86_64.rpm:
	fpm -s dir -t rpm \
	--name "$(NAME)" \
	--depends "python" \
	--version "$(VERSION)" \
	--iteration "$(RELEASE)" \
	--maintainer "Sam Clements" \
	.

Merging a subdirectory of a separate Git repository

In the repository being merged:

git checkout -b temp
git filter-branch --prune-empty --subdirectory-filter <subdirectory>/ temp

In the repository the subdirectory is being merged into:

git remote add other ../<other-repository>
git fetch other
git merge -s ours --no-commit --allow-unrelated-histories other/temp
git read-tree --prefix=<path> -u other/temp
git commit

Done!

The output of the merge should look something like this:

$ cd other-repository
$ git checkout -b temp
Switched to a new branch 'temp'
$ git filter-branch --prune-empty --subdirectory-filter other/ temp
Rewrite 82e8de6f33e9e4e797e0331aac1fc602f68f6bfa (13/13) (0 seconds passed, remaining 0 predicted)    
Ref 'refs/heads/temp' was rewritten
$ cd ../repository
$ git merge -s ours --no-commit --allow-unrelated-histories other/temp
Automatic merge went well; stopped before committing as requested
$ git read-tree --prefix=other -u other/temp
$ git commit
[feature/merge bc00b33] Merge remote-tracking branch 'other/temp' into feature/FTD-768-build

Interesting papers (2016)

Interesting papers (2015)

  • Holistic Configuration Management at Facebook

    How Facebook manages and distributes configuration - from small JSON files to huge machine-learning datasets - at a massive scale.

  • The impact of syntax colouring on program comprehension

    Advait Sarkar, 2015.

    Study measuring the effects of syntax highlighting - notably that it is useful, and programmers spend less time focusing on keywords when using it.

  • Google Votes: A Liquid Democracy Experiment on a Corporate Social Network

    Hardt, Steve and Lopes, Lia C. R., 2015.

    An experimental voting system built and used inside Google that uses software to mix direct and representative democracy by allowing users to vote directly or entrust their vote to another user.

  • Scaling Agile at Spotify

    Henrik Kniberg & Anders Ivarsson (2012).

    Describes how Spotify has previously organised their engineers into tribes, squads, chapters and guilds; aiming to avoid layers of bureaucracy and other problems often associated with large numbers of employees.

Error handling in Rust

A short cheatsheet for dealing with Return values in Rust.

When a Result is Ok<T>:

let result: Result<&str, &str> = Ok("Succeeded");

result.is_ok() == true;
result.is_err() == false;

result.ok() == Some("Succeeded");
result.err() == None;

let other_result: Result<&str, &str> = Ok("Other result");
result.and(other_result) == Ok("Other result");
result.or(other_result) == Ok("Succeeded");

fn example(string: &str) -> Result<&str, &str> { Ok("Example") }
result.and_then(example) == Ok("Example");
result.or_else(example) == Ok("Succeeded");

When a Result is Err<E>:

let result: Result<&str, &str> = Err("Failed");

result.is_ok() == false;
result.is_err() == true;

result.ok() == None;
result.err() == Some("Failed");

let other_result: Result<&str, &str> = Ok("Other result");
result.and(other_result) == Err("Failed");
result.or(other_result) == Ok("Other result");

fn example(string: &str) -> Result<&str, &str> { Ok("Example") }
result.and_then(example) == Err("Failed");
result.or_else(example) == Ok("Example");

When an Option is Some<T>:

let option: Option<&str> = Some("Example");

option.is_some() == true;
option.is_none() == false;

option.unwrap() == "Example";
option.unwrap_or("Other") == "Example";
option.expect("the world is ending") == "Example";

When an Option is None:

option.is_some() == false;
option.is_none() == true;

option.unwrap(); // panic!()
option.unwrap_or("Other") == "Other";
option.expect("the world is ending"); // panic!("the world is ending")