“Yellow and orange shipping containers stacked on top of each other” by frank mckenna on Unsplash
“Yellow and orange shipping containers stacked on top of each other” by frank mckenna on Unsplash

A container image in 60* lines of Go

*Not really 60 lines

At Ravelin we build Go binaries, package them into scratch Docker containers and upload them to Google Cloud Registry so we can use them in a GKE cluster. We develop on Mac laptops, so we use Docker for Mac for this. But building and pushing the container images is very slow, particularly if we want to build 44 at once. Given we just want to package up single binaries the process seemed a bit overblown and complicated.

So I thought I’d work out what a docker image actually is, and see if I could build one myself and upload it directly. I wanted a small Go program with few dependencies that could build a very simple container image and upload it.

So what is a container image?

I started off thinking a container image is a file that you would construct and upload. This is wrong. A container image is a set of files and a manifest file. For my simple image there are 3 files we need.

  1. An image layer. This is a tar file containing the files I want in the container. If I had a more complex container it would have multiple layers (tar files) that would be applied one on top of another. But for my simple scratch image a single tar file will do.
  2. An image config. This basically corresponds to the bits of your Dockerfile that aren’t concerned with which files are in your layers. So things like labels, environment variables, volumes, etc. It also contains references to the layers in the image. You can find the definitions I used for this here.
  3. A manifest. This references the layer file and image config file, tying everything together.

The good thing about standards is there’s so many of them.

There are several standards for these things. I began by looking at the Open Container Initiative standards, and that’s nearly where I left it. The OCI documents are extremely difficult to understand, and after quite a bit of work I found they’re not supported by the Docker repository.

So I swapped to the definitions used by Docker, which seem to be documented by code and a string of issues. I decided to copy them out of the docker codebase into my own code to avoid some fairly hefty dependencies, so you can find the definitions here.

So what do you actually do?

  1. Create a tar file of the contents of your container. Yep, just an ordinary tar file.
  2. Run this tar file through sha256 to get a digest. There’s a nice library to help you do this.
  3. Upload the tar file to the repo as a blob. The layer blob is named after its digest.
  4. Build an image config JSON.
  5. Get the sha256 digest of the JSON.
  6. Upload the JSON to the repo as a blob. The image config blob is named after its digest.
  7. Build the manifest JSON. This references the layer & image config via their digests.
  8. Upload the manifest to the repo. The URL for the manifest includes the image name and tag. If you want multiple tags just upload the manifest again with a different URL.
{
  "architecture": "amd64",
  "os": "linux",
  "config": {
    "Entrypoint": [
      "/app"
    ]
  },
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:d950580d13e7b6fcbffbbe90129536e1acbf4be04badb50dcc4307c10b4672c7"
    ]
  }
}

A very simple image config

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 184,
    "digest": "sha256:51f7917e0550525eda6b4656a3bdf8ddbd084664edb1dc372dd63f55ed52c565"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 729097,
      "digest": "sha256:08c20a47894d73e1f4f672b56ef19ae3f836b799a582732bf995439adf492167"
    }
  ]
}

A very simple manifest

How do you upload a blob?

A blob on Docker Hub has a URL like https://index.docker.io/v2/<image name>/blobs/<digest>. So before you upload a blob you can check whether it’s already present by sending a HEAD request for this URL. If you get a 200 OK, the blob is already uploaded.

If you do need to upload the blob, you first create a new upload with a POST to https://index.docker.io/v2/<image name>/blobs/uploads. This will return a 202 ACCEPTED response with a Location header containing the URL to upload the blob to.

Next you upload the blob with a PUT to the URL you’ve received. Add digest=<digest> as a URL query parameter, set the Content-Length, and set the Content-Type to application/octet-stream.

Both the layer tar file and the JSON-encoded image config should be uploaded as blobs.

How do you upload the manifest?

The JSON-encoded manifest should be PUT to https://index.docker.io/v2/<image name>/manifests/<tag>. This time the Content-Type should be application/vnd.docker.distribution.manifest.v2+json

What about auth?

Any of these requests could return a 401 Unauthorised response with a Www-Authenticate header that, for the Docker registry, looks something like the following.

Bearer realm=”https://auth.docker.io/token",service="registry.docker.io",scope="repository:philpearl/test:pull,push"

This is telling you that you can get a Bearer token by sending a GET to https://auth.docker.io/token?service=registry.docker.io&scope=repository:philpearl/test:pull,push. In the case of Docker you need to send your Docker Hub username and password using Basic Auth.

The response looks something like the following. I’ve edited the tokens in this response.

{"token":"eyJ..snip..","access_token":"eyJ..snap...","expires_in":300,"issued_at":"2018-04-29T14:50:42.311414998Z"}

The token field is a Bearer token that can then be used to authorise your requests by adding a header like the following.

Authorization: Bearer eyJ..snip..

If you don’t want to wait for the challenge, you can start by sending a GET to https://index.docker.io/v2/. This will return the WWW-Authenticate header, so you can get a token before attempting any uploads.

So where’s that 60 lines of Go?

Ah, well, it’s more like 540 lines, and they’re here.

Prior Art

If what I’ve put together here is too simple for your needs there are quite a few other container image builders that avoid Docker, but allow you to build complex multi-layered images directly from Dockerfiles. Some of the tools below came out very recently, and were basically what started me thinking about this.