Go 1.11 introduced a new concept of Modules which brings first class support for managing dependency versions and enabling reproducible builds. Go previously had no notion of dependency versions, and it has been a long and arduous road to get where we are now. Modules do not just copy the style of other programming language’s dependency tools, rather it introduces a few slightly different concepts intended to enable programming in the large. These require some thought in order to fully utilize the features provided by these new concepts.

What is a Go Module

A Module is a way to group together a set of packages and give it a version number to mark it’s existence (state) at a specific point in time. Modules have versions and the version number is meaningful. That is, a developer can expect a certain amount of predictability from a library’s features based on it’s version number.

Go Modules use Semantic Versioning for their numbering scheme.

Modules are supported by a set of commands in the go tool and expected semantics that offer a granular approach for developers in how the go tools decides which dependencies to use during the build step.

This gets at the heart of the reason for Modules. They are intended to make developers lives easier by reducing unexpected behavior projects, and the client libraries they use, grow and evolve. Software is living after all.

The idea of Modules is built into the go tool, and is (or will soon be) in other ancillary tools in the Golang toolbox. This first class support for modules is something no other dependency management solution previously offered. Though gb took the same philosophy that dependency management needed to be part of the core tools.

Why do we need Modules

The Module is about programming in the large. It is a solution to managing unexpected changes in upstream libraries. From the Go Wiki Module page:

Modules record precise dependency requirements and create reproducible builds.

Modules are a versioned grouping of Go packages. A Module is a set of packages that is your application, the main.go and packages that can not stand alone. A Module would also be the libraries that you use in your main application.

Imagine a blogging application with packages for managing it’s HTML templating, document indexing, taging, and other domain specific functionality. These are functionality that do not stand alone as useful libraries. This project also uses Blackfriday for HTML rendering. Blackfriday is able to stand alone as a useful library and as such would be a Module in it’s own right.

No more GOPATH

Modules allow for the deprecation of the GOPATH. There is no longer a need to set it explicitly as a go.mod files defines the root of a Module, and allows the Go toolchain to know where everything is that it needs to work with. This was the purpose of GOPATH. Internally to the go tool GOPATH is still used for a few things, but this not something we need to be concerned with.

Minimum Version Selection

Minimum Version Selection, or MVS, is a strategy of deciding which version of a library to use given the constraints that a developer has specified in the go.mod file, and the constraints of all the other dependent libraries used.

In short MVS wants to use the the oldest known “good” version of a library that works with a given project, where known “good” is a version that a human specified. If the go.mod file specifies a need for version v1.2 then Go will use version v1.2 even if v1.3 is available; this is given that no other dependencies have a need on this library.

If our project’s go.mod file specifies that it needs v1.2 of a library, and one of it’s other dependencies specifies that it needs v1.3 of the same library then Go will use v1.3, as it is the lowest version (oldest) that satifies all stated version needs.

This is in stark contrast to most other dependency tools that other programming languages use. Most other languages (heck maybe all other dependency tools) will use a set of complex notation that allows a developer to let the tool decide, as it will, to use a newer version if a newer version is available, and that Semantic Versioning indicates that nothing should break.

One of Russ’s complaints (dislikes) of other dependency tools is that this complexity introduces the chance of unpredictability. Today I might build the project with v1.2 when it is the latest version available, and tomorrow my co-maintainer might build the project with v1.3, it having been released that morning. Each of us would be unaware that each other was using a different version. In this example my co-maintainer builds from source and because the dependency tools allow for using newer libraries, that ostensibly break nothing, it will not throw an error, warning, or otherwise inform my colleague that they are getting a different build than I got.

Now we have two different builds that could have subtly different behavior.

MVS ensure that the library versions I compile with today will be the same versions that my co-maintainer compiles with tomorrow. Unless a change has been introduced and a human person has updated the go.mod file, or updated a dependency that itself has a greater specified version.

There is a side effect to this; I have yet to decide if I like it or not. If there are updates to the client library that fix a bug or security issue, and it really does not change the behavior of the library, then that update has to be manually specified, by a human, before it will be included in the project.

Other programming language dependency tools will update to the newer version, and those languages see this as a feature of the tool; quick and easy propagation of bug and security fixes. Go, too, sees MVS as a feature. That is to say Go, and the Go Team, prefer to have less unexpected changes instead of the bug and security updates, and this feature is in service to programming in the large.

Semantic Import Versioning

Previously an import path was an import path and developers used the version of the library that lived on the master branch, or a tool like dep was used to specify which branch/tag/hash of a VCS repo to use.

Semantic Import Version, or SIV, specifies that major versions have their version number in the import path. So github.com/russross/blackfriday will become github.com/russross/blackfriday/v2 for version two, and github.com/russross/blackfriday/v3 for version three.

This enables different major versions of a library to be used in a single main application. This, again, is a need for programming in the large.

Don’t worry, there is no need to update older import paths as sub-1.0 and 1.0 versions are allowed to use import paths without a version.

Vendoring

Go stores downloaded dependencies somewhere. We don’t know where exactly, and it really doesn’t matter. Once Go downloads the dependencies, and makes them available to your code it don’t matter where they live. You don’t need to know about the libraries internals, or the drama going on between it’s maintainers; all you care is that the code compiles with the versions expected and the program runs as you intended.

This is all good and fine, but there are times when having more control over dependencies is useful, or required. This is where the go mod vendor command comes in. It allows you to store your dependencies in the vendor directory inside of the project directory.

Vendoring dependencies is not a new idea. It has been in Go since 1.5 as an experiment, and 1.6 enabled by default.

There is a difference between the current vendoring and the previous. When vendoring was solidified in the Go toolchain it was to enable repeatable builds. To allow projects more control over their dependencies than having to use go get every time and hope the dependent library still existed as a project, that it hadn’t been deleted from github, and that no breaking changes had been pushed to master. In this way the the vendor directory is a poor devs version control. This is how the early dependency management tools worked, they downloaded the dependent library to the vendor directory and used that project’s VCS to checkout the version that was specified in the tool’s configuration.

Modules are the future

Modules give more control to you, the programmer, by enabling more configuration options. This control comes with complexity and responsibility; seen in Minimal Version Selection and Semantic Version Importing. The Go Team sees that front loaded complexity as a good trade off to reap the benefit of reliability and consistency through the life of a project. This complexity is considered minimal to the actually benefits to programming in the large, it brings reduced pain over long lived projects that have many many developers working in tandem.

Modules are a way to manage the versioning of dependencies that the Go community has been longing for. It is a core feature of the Go language and tooling, a single way for the community as a whole to utilize each other’s libraries and fill a feature hole that has desired since Go was announced to the public.

The road to Modules has not always been easy, and was never quick. What has, in recent months and years, been a hotly debated topic will soon fade to the background as we all focus again on our projects. This will be the mark of Modules as a successful tool; one that gets out of our way.