The module cache is already a proxy

In one line

Go modules have two ways of dealing with an air-gapped network: standing up a module proxy as files (GOPROXY=file://) and putting the dependencies into the repository (vendor). Either way, go.sum acts as the import manifest, and a single environment variable line (GOFLAGS) changes which way you go.

Why this was needed

Run go build on an air-gapped build server and it ends with Get "https://proxy.golang.org/...": dial tcp .... By default Go downloads modules from GOPROXY=https://proxy.golang.org,direct and asks sum.golang.org for the hash of a module it sees for the first time. Neither is reachable in an air-gapped network. Some people copy $GOPATH/pkg/mod wholesale from a connected PC, but that directory is unpacked source with read-only permissions, so even copying and deleting it causes trouble, and there is no list of what was taken.

How it works

The syntax of GOPROXY. According to the Go modules documentation, GOPROXY is a list joined by commas or pipes. After a comma it moves on only on 404 or 410, and after a pipe it moves on for any error, including a timeout. off downloads from nowhere, and direct downloads straight from the version control repository. The scheme of an address can be https, http, or file.

The module cache is a proxy. The documentation takes GOPROXY=file://$(go env GOMODCACHE)/cache/download as an example and says the module cache can be used as a file proxy as it is. Under cache/download, each module has <버전>.mod, .zip, and .ziphash (the placeholder is the version), and for modules whose versions were queried, @v/list and .info also pile up, at the same paths as the proxy protocol. If go.mod and go.sum name the version, a build downloads only the .mod and .zip. In this lab, golang.org/x/text, which came along as a pseudo-version, had no .info, and the build worked anyway. So if you download once on the connected side into a new module cache (GOMODCACHE) and then move that directory, you get the import list and the proxy in one go. You download into a new cache so that other modules fetched earlier do not get mixed in.

go.sum is the import manifest. According to the documentation, the go command asks the checksum database only when go.sum has no hash for that file. If go.sum has it, the command compares against the downloaded file and stops with a security error if they differ. So if go.sum is complete, there is no need to reach sum.golang.org in an air-gapped network. It becomes a problem only when you try to add a new dependency inside the air-gapped network. The knobs for that are GOSUMDB=off (nothing is verified except what is already in go.sum) and GONOSUMDB and GOPRIVATE (only modules matching the pattern skip the database). GOINSECURE only allows http for direct downloads and does not turn off checksum verification. GONOPROXY is the pattern of modules to download directly without going through the proxy, and its default is GOPRIVATE.

vendor and -mod. go mod vendor copies the dependency source into vendor/ and creates vendor/modules.txt. The build flag -mod=vendor uses neither the network nor the module cache and looks only at vendor. -mod=mod ignores vendor and edits go.mod if needed, and -mod=readonly ignores vendor and raises an error if go.mod would need editing. If the go version in go.mod is 1.14 or later and a vendor directory exists, the default behaves like -mod=vendor. The default flags are given through the GOFLAGS environment variable.

What it looks like in the field

There is one trap I measured in this lab image. The image's environment has GOFLAGS=-mod=mod baked in (a value put there so that other labs can download modules freely). In that state, when I created vendor and built with GOPROXY=off, it failed with module lookup disabled by GOPROXY=off even though vendor was there. That is because -mod=mod makes vendor ignored. Giving GOFLAGS=-mod=vendor or emptying GOFLAGS made the same build succeed. That is why you first check whether GOFLAGS is hiding in the build server's shell profile or CI variables.

I also measured the scale. Downloading rsc.io/quote v1.5.2 brings in rsc.io/sampler and golang.org/x/text (the 2017 version), and cache/download in the module cache was 4.9MB. Of the three zips, x/text is 4.8MB, nearly all of it. The list to submit for import review is the lines in go.sum that do not have /go.mod attached.

You can also run a module proxy server inside the company. Nexus supports Go as proxy and group, and the documentation says hosted is supported from 3.93. Either way, writing the internal address in GOPROXY is the same.

What you will do in the next lab

You download rsc.io/quote v1.5.2 into a new module cache to create go.sum, move that cache's cache/download to /srv/goproxy, and point to it with GOPROXY=file://. The grader builds again with a new cache and the outside blocked. You create vendor, experience the image's GOFLAGS trap yourself and record it, and then build with -mod=vendor. Finally, you write an import record that checks go.sum against the proxy's .ziphash.

Reference documents: Go Modules Reference — GOPROXY protocol · Environment variables · Vendoring · Authenticating modules