You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
garble/main.go

2061 lines
60 KiB
Go

// Copyright (c) 2019, The Garble Authors.
// See LICENSE for licensing information.
5 years ago
package main
import (
"bytes"
"crypto/rand"
"encoding/base64"
"encoding/binary"
"encoding/gob"
"encoding/json"
fail if we are unexpectedly overwriting files (#418) While investigating a bug report, I noticed that garble was writing to the same temp file twice. At best, writing to the same path on disk twice is wasteful, as the design is careful to be deterministic and use unique paths. At worst, the two writes could cause races at the filesystem level. To prevent either of those situations, we now create files with os.OpenFile and os.O_EXCL, meaning that we will error if the file already exists. That change uncovered a number of such unintended cases. First, transformAsm would write obfuscated Go files twice. This is because the Go toolchain actually runs: [...]/asm -gensymabis [...] foo.s bar.s [...]/asm [...] foo.s bar.s That is, the first run is only meant to generate symbol ABIs, which are then used by the compiler. We need to obfuscate at that first stage, because the symbol ABI descriptions need to use obfuscated names. However, having already obfuscated the assembly on the first stage, there is no need to do so again on the second stage. If we detect gensymabis is missing, we simply reuse the previous files. This first situation doesn't seem racy, but obfuscating the Go assembly files twice is certainly unnecessary. Second, saveKnownReflectAPIs wrote a gob file to the build cache. Since the build cache can be kept between builds, and since the build cache uses reproducible paths for each build, running the same "garble build" twice could overwrite those files. This could actually cause races at the filesystem level; if two concurrent builds write to the same gob file on disk, one of them could end up using a partially-written file. Note that this is the only of the three cases not using temporary files. As such, it is expected that the file may already exist. In such a case, we simply avoid overwriting it rather than failing. Third, when "garble build -a" was used, and when we needed an export file not listed in importcfg, we would end up calling roughly: go list -export -toolexec=garble -a <dependency> This meant we would re-build and re-obfuscate those packages. Which is unfortunate, because the parent process already did via: go build -toolexec=garble -a <main> The repeated dependency builds tripped the new os.O_EXCL check, as we would try to overwrite the same obfuscated Go files. Beyond being wasteful, this could again cause subtle filesystem races. To fix the problem, avoid passing flags like "-a" to nested go commands. Overall, we should likely be using safer ways to write to disk, be it via either atomic writes or locked files. However, for now, catching duplicate writes is a big step. I have left a self-assigned TODO for further improvements. CI on the pull request found a failure on test-gotip. The failure reproduces on master, so it seems to be related to gotip, and not a regression introduced by this change. For now, disable test-gotip until we can investigate.
3 years ago
"errors"
5 years ago
"flag"
"fmt"
"go/ast"
"go/importer"
"go/parser"
"go/token"
"go/types"
"io"
fail if we are unexpectedly overwriting files (#418) While investigating a bug report, I noticed that garble was writing to the same temp file twice. At best, writing to the same path on disk twice is wasteful, as the design is careful to be deterministic and use unique paths. At worst, the two writes could cause races at the filesystem level. To prevent either of those situations, we now create files with os.OpenFile and os.O_EXCL, meaning that we will error if the file already exists. That change uncovered a number of such unintended cases. First, transformAsm would write obfuscated Go files twice. This is because the Go toolchain actually runs: [...]/asm -gensymabis [...] foo.s bar.s [...]/asm [...] foo.s bar.s That is, the first run is only meant to generate symbol ABIs, which are then used by the compiler. We need to obfuscate at that first stage, because the symbol ABI descriptions need to use obfuscated names. However, having already obfuscated the assembly on the first stage, there is no need to do so again on the second stage. If we detect gensymabis is missing, we simply reuse the previous files. This first situation doesn't seem racy, but obfuscating the Go assembly files twice is certainly unnecessary. Second, saveKnownReflectAPIs wrote a gob file to the build cache. Since the build cache can be kept between builds, and since the build cache uses reproducible paths for each build, running the same "garble build" twice could overwrite those files. This could actually cause races at the filesystem level; if two concurrent builds write to the same gob file on disk, one of them could end up using a partially-written file. Note that this is the only of the three cases not using temporary files. As such, it is expected that the file may already exist. In such a case, we simply avoid overwriting it rather than failing. Third, when "garble build -a" was used, and when we needed an export file not listed in importcfg, we would end up calling roughly: go list -export -toolexec=garble -a <dependency> This meant we would re-build and re-obfuscate those packages. Which is unfortunate, because the parent process already did via: go build -toolexec=garble -a <main> The repeated dependency builds tripped the new os.O_EXCL check, as we would try to overwrite the same obfuscated Go files. Beyond being wasteful, this could again cause subtle filesystem races. To fix the problem, avoid passing flags like "-a" to nested go commands. Overall, we should likely be using safer ways to write to disk, be it via either atomic writes or locked files. However, for now, catching duplicate writes is a big step. I have left a self-assigned TODO for further improvements. CI on the pull request found a failure on test-gotip. The failure reproduces on master, so it seems to be related to gotip, and not a regression introduced by this change. For now, disable test-gotip until we can investigate.
3 years ago
"io/fs"
"log"
mathrand "math/rand"
5 years ago
"os"
"os/exec"
"path/filepath"
"regexp"
"runtime"
"runtime/debug"
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
"strconv"
"strings"
"time"
"unicode"
"unicode/utf8"
"golang.org/x/mod/modfile"
"golang.org/x/mod/semver"
"golang.org/x/tools/go/ast/astutil"
"mvdan.cc/garble/internal/literals"
5 years ago
)
var (
flagSet = flag.NewFlagSet("garble", flag.ContinueOnError)
version = "(devel)" // to match the default from runtime/debug
)
5 years ago
var (
flagLiterals bool
flagTiny bool
flagDebug bool
flagDebugDir string
flagSeed seedFlag
)
func init() {
flagSet.Usage = usage
flagSet.BoolVar(&flagLiterals, "literals", false, "Obfuscate literals such as strings")
flagSet.BoolVar(&flagTiny, "tiny", false, "Optimize for binary size, losing some ability to reverse the process")
flagSet.BoolVar(&flagDebug, "debug", false, "Print debug logs to stderr")
flagSet.StringVar(&flagDebugDir, "debugdir", "", "Write the obfuscated source to a directory, e.g. -debugdir=out")
flagSet.Var(&flagSeed, "seed", "Provide a base64-encoded seed, e.g. -seed=o9WDTZ4CN4w\nFor a random seed, provide -seed=random")
}
var rxGarbleFlag = regexp.MustCompile(`-(literals|tiny|debug|debugdir|seed)($|=)`)
type seedFlag struct {
random bool
bytes []byte
}
func (f seedFlag) present() bool { return len(f.bytes) > 0 }
func (f seedFlag) String() string {
return base64.RawStdEncoding.EncodeToString(f.bytes)
}
func (f *seedFlag) Set(s string) error {
if s == "random" {
f.bytes = make([]byte, 16) // random 128 bit seed
if _, err := rand.Read(f.bytes); err != nil {
return fmt.Errorf("error generating random seed: %v", err)
}
} else {
// We expect unpadded base64, but to be nice, accept padded
// strings too.
s = strings.TrimRight(s, "=")
seed, err := base64.RawStdEncoding.DecodeString(s)
if err != nil {
return fmt.Errorf("error decoding seed: %v", err)
}
if len(seed) < 8 {
return fmt.Errorf("-seed needs at least 8 bytes, have %d", len(seed))
}
f.bytes = seed
}
return nil
}
5 years ago
func usage() {
fmt.Fprintf(os.Stderr, `
Garble obfuscates Go code by wrapping the Go toolchain.
5 years ago
garble [garble flags] command [go flags] [go arguments]
For example, to build an obfuscated program:
garble build ./cmd/foo
Similarly, to combine garble flags and Go build flags:
garble -literals build -tags=purego ./cmd/foo
The following commands are supported:
build replace "go build"
test replace "go test"
version print Garble version
reverse de-obfuscate output such as stack traces
To learn more about a command, run "garble help <command>".
garble accepts the following flags before a command:
5 years ago
`[1:])
flagSet.PrintDefaults()
fmt.Fprintf(os.Stderr, `
For more information, see https://github.com/burrowers/garble.
`[1:])
5 years ago
}
func main() { os.Exit(main1()) }
var (
fset = token.NewFileSet()
sharedTempDir = os.Getenv("GARBLE_SHARED")
parentWorkDir = os.Getenv("GARBLE_PARENT_WORK")
// origImporter is a go/types importer which uses the original versions
// of packages, without any obfuscation. This is helpful to make
// decisions on how to obfuscate our input code.
wrap types.Importer to canonicalize import paths The docs for go/importer.ForCompiler say: The lookup function is called each time the resulting importer needs to resolve an import path. In this mode the importer can only be invoked with canonical import paths (not relative or absolute ones); it is assumed that the translation to canonical import paths is being done by the client of the importer. We use a lookup func for two reasons: first, to support modules, and second, to be able to use our information from "go list -json -export". However, go/types does not canonicalize import paths before calling ImportFrom. This is somewhat understandable; it doesn't know whether an importer was created with a lookup func, and ImportFrom only requires the input path to be canonicalized in that scenario. When the lookup func is nil, the importer canonicalizes by itself via go/build.Import. Before this change, the added crossbuild test would fail: > garble build net/http [stderr] # vendor/golang.org/x/crypto/chacha20 typecheck error: /usr/lib/go/src/vendor/golang.org/x/crypto/chacha20/chacha_generic.go:10:2: could not import crypto/cipher (can't find import: "crypto/cipher") # vendor/golang.org/x/text/secure/bidirule typecheck error: /usr/lib/go/src/vendor/golang.org/x/text/secure/bidirule/bidirule.go:12:2: could not import errors (can't find import: "errors") # vendor/golang.org/x/crypto/cryptobyte typecheck error: /usr/lib/go/src/vendor/golang.org/x/crypto/cryptobyte/asn1.go:8:16: could not import encoding/asn1 (can't find import: "encoding/asn1") # vendor/golang.org/x/text/unicode/norm typecheck error: /usr/lib/go/src/vendor/golang.org/x/text/unicode/norm/composition.go:7:8: could not import unicode/utf8 (can't find import: "unicode/utf8") This is because we'd fall back to importer.Default, which only knows how to find packages in $GOROOT/pkg. Those are missing for cross-builds, unsurprisingly, as those built archives end up in the build cache. After this change, we properly support importing std-vendored packages, so we can get rid of the importer.Default workaround. And, by extension, cross-builds now work as well. Note that, in the added test script, the full build of the binary fails, as there seems to be some sort of linker problem: > garble build [stderr] # test/main d9rqJyxo.uoqIiDs5: relocation target runtime.os9A16A3 not defined We leave that as a TODO for now, as this change is subtle enough as it is.
3 years ago
origImporter = importerWithMap(importer.ForCompiler(fset, "gc", func(path string) (io.ReadCloser, error) {
pkg, err := listPackage(path)
if err != nil {
return nil, err
}
return os.Open(pkg.Export)
wrap types.Importer to canonicalize import paths The docs for go/importer.ForCompiler say: The lookup function is called each time the resulting importer needs to resolve an import path. In this mode the importer can only be invoked with canonical import paths (not relative or absolute ones); it is assumed that the translation to canonical import paths is being done by the client of the importer. We use a lookup func for two reasons: first, to support modules, and second, to be able to use our information from "go list -json -export". However, go/types does not canonicalize import paths before calling ImportFrom. This is somewhat understandable; it doesn't know whether an importer was created with a lookup func, and ImportFrom only requires the input path to be canonicalized in that scenario. When the lookup func is nil, the importer canonicalizes by itself via go/build.Import. Before this change, the added crossbuild test would fail: > garble build net/http [stderr] # vendor/golang.org/x/crypto/chacha20 typecheck error: /usr/lib/go/src/vendor/golang.org/x/crypto/chacha20/chacha_generic.go:10:2: could not import crypto/cipher (can't find import: "crypto/cipher") # vendor/golang.org/x/text/secure/bidirule typecheck error: /usr/lib/go/src/vendor/golang.org/x/text/secure/bidirule/bidirule.go:12:2: could not import errors (can't find import: "errors") # vendor/golang.org/x/crypto/cryptobyte typecheck error: /usr/lib/go/src/vendor/golang.org/x/crypto/cryptobyte/asn1.go:8:16: could not import encoding/asn1 (can't find import: "encoding/asn1") # vendor/golang.org/x/text/unicode/norm typecheck error: /usr/lib/go/src/vendor/golang.org/x/text/unicode/norm/composition.go:7:8: could not import unicode/utf8 (can't find import: "unicode/utf8") This is because we'd fall back to importer.Default, which only knows how to find packages in $GOROOT/pkg. Those are missing for cross-builds, unsurprisingly, as those built archives end up in the build cache. After this change, we properly support importing std-vendored packages, so we can get rid of the importer.Default workaround. And, by extension, cross-builds now work as well. Note that, in the added test script, the full build of the binary fails, as there seems to be some sort of linker problem: > garble build [stderr] # test/main d9rqJyxo.uoqIiDs5: relocation target runtime.os9A16A3 not defined We leave that as a TODO for now, as this change is subtle enough as it is.
3 years ago
}).(types.ImporterFrom).ImportFrom)
start using original action IDs (#251) When we obfuscate a name, what we do is hash the name with the action ID of the package that contains the name. To ensure that the hash changes if the garble tool changes, we used the action ID of the obfuscated build, which is different than the original action ID, as we include garble's own content ID in "go tool compile -V=full" via -toolexec. Let's call that the "obfuscated action ID". Remember that a content ID is roughly the hash of a binary or object file, and an action ID contains the hash of a package's source code plus the content IDs of its dependencies. This had the advantage that it did what we wanted. However, it had one massive drawback: when we compile a package, we only have the obfuscated action IDs of its dependencies. This is because one can't have the content ID of dependent packages before they are built. Usually, this is not a problem, because hashing a foreign name means it comes from a dependency, where we already have the obfuscated action ID. However, that's not always the case. First, go:linkname directives can point to any symbol that ends up in the binary, even if the package is not a dependency. So garble could only support linkname targets belonging to dependencies. This is at the root of why we could not obfuscate the runtime; it contains linkname directives targeting the net package, for example, which depends on runtime. Second, some other places did not have an easy access to obfuscated action IDs, like transformAsm, which had to recover it from a temporary file stored by transformCompile. Plus, this was all pretty expensive, as each toolexec sub-process had to make repeated calls to buildidOf with the object files of dependencies. We even had to use extra calls to "go list" in the case of indirect dependencies, as their export files do not appear in importcfg files. All in all, the old method was complex and expensive. A better mechanism is to use the original action IDs directly, as listed by "go list" without garble in the picture. This would mean that the hashing does not change if garble changes, meaning weaker obfuscation. To regain that property, we define the "garble action ID", which is just the original action ID hashed together with garble's own content ID. This is practically the same as the obfuscated build ID we used before, but since it doesn't go through "go tool compile -V=full" and the obfuscated build itself, we can work out *all* the garble action IDs upfront, before the obfuscated build even starts. This fixes all of our problems. Now we know all garble build IDs upfront, so a bunch of hacks can be entirely removed. Plus, since we know them upfront, we can also cache them and avoid repeated calls to "go tool buildid". While at it, make use of the new BuildID field in Go 1.16's "list -json -export". This avoids the vast majority of "go tool buildid" calls, as the only ones that remain are 2 on the garble binary itself. The numbers for Go 1.16 look very good: name old time/op new time/op delta Build-8 146ms ± 4% 101ms ± 1% -31.01% (p=0.002 n=6+6) name old bin-B new bin-B delta Build-8 6.61M ± 0% 6.60M ± 0% -0.09% (p=0.002 n=6+6) name old sys-time/op new sys-time/op delta Build-8 321ms ± 7% 202ms ± 6% -37.11% (p=0.002 n=6+6) name old user-time/op new user-time/op delta Build-8 538ms ± 4% 414ms ± 4% -23.12% (p=0.002 n=6+6)
3 years ago
// Basic information about the package being currently compiled or linked.
curPkg *listedPackage
)
wrap types.Importer to canonicalize import paths The docs for go/importer.ForCompiler say: The lookup function is called each time the resulting importer needs to resolve an import path. In this mode the importer can only be invoked with canonical import paths (not relative or absolute ones); it is assumed that the translation to canonical import paths is being done by the client of the importer. We use a lookup func for two reasons: first, to support modules, and second, to be able to use our information from "go list -json -export". However, go/types does not canonicalize import paths before calling ImportFrom. This is somewhat understandable; it doesn't know whether an importer was created with a lookup func, and ImportFrom only requires the input path to be canonicalized in that scenario. When the lookup func is nil, the importer canonicalizes by itself via go/build.Import. Before this change, the added crossbuild test would fail: > garble build net/http [stderr] # vendor/golang.org/x/crypto/chacha20 typecheck error: /usr/lib/go/src/vendor/golang.org/x/crypto/chacha20/chacha_generic.go:10:2: could not import crypto/cipher (can't find import: "crypto/cipher") # vendor/golang.org/x/text/secure/bidirule typecheck error: /usr/lib/go/src/vendor/golang.org/x/text/secure/bidirule/bidirule.go:12:2: could not import errors (can't find import: "errors") # vendor/golang.org/x/crypto/cryptobyte typecheck error: /usr/lib/go/src/vendor/golang.org/x/crypto/cryptobyte/asn1.go:8:16: could not import encoding/asn1 (can't find import: "encoding/asn1") # vendor/golang.org/x/text/unicode/norm typecheck error: /usr/lib/go/src/vendor/golang.org/x/text/unicode/norm/composition.go:7:8: could not import unicode/utf8 (can't find import: "unicode/utf8") This is because we'd fall back to importer.Default, which only knows how to find packages in $GOROOT/pkg. Those are missing for cross-builds, unsurprisingly, as those built archives end up in the build cache. After this change, we properly support importing std-vendored packages, so we can get rid of the importer.Default workaround. And, by extension, cross-builds now work as well. Note that, in the added test script, the full build of the binary fails, as there seems to be some sort of linker problem: > garble build [stderr] # test/main d9rqJyxo.uoqIiDs5: relocation target runtime.os9A16A3 not defined We leave that as a TODO for now, as this change is subtle enough as it is.
3 years ago
type importerWithMap func(path, dir string, mode types.ImportMode) (*types.Package, error)
func (fn importerWithMap) Import(path string) (*types.Package, error) {
panic("should never be called")
}
func (fn importerWithMap) ImportFrom(path, dir string, mode types.ImportMode) (*types.Package, error) {
if path2 := curPkg.ImportMap[path]; path2 != "" {
path = path2
}
return fn(path, dir, mode)
}
// uniqueLineWriter sits underneath log.SetOutput to deduplicate log lines.
// We log bits of useful information for debugging,
// and logging the same detail twice is not going to help the user.
// Duplicates are relatively normal, given that names tend to repeat.
type uniqueLineWriter struct {
out io.Writer
seen map[string]bool
}
func (w *uniqueLineWriter) Write(p []byte) (n int, err error) {
if !flagDebug {
panic("unexpected use of uniqueLineWriter with -debug unset")
}
if bytes.Count(p, []byte("\n")) != 1 {
panic(fmt.Sprintf("log write wasn't just one line: %q", p))
}
if w.seen[string(p)] {
return len(p), nil
}
if w.seen == nil {
w.seen = make(map[string]bool)
}
w.seen[string(p)] = true
return w.out.Write(p)
}
// debugf is like log.Printf, but it is a no-op by default.
// TODO(mvdan): use our own debug logger instead of hijacking the global one,
// and drop the "flagDebug" no-op check now that we require Go 1.18 or later.
func debugf(format string, args ...any) {
if !flagDebug {
return
}
log.Printf(format, args...)
}
// debugSince is like time.Since but resulting in shorter output.
// A build process takes at least hundreds of milliseconds,
// so extra decimal points in the order of microseconds aren't meaningful.
func debugSince(start time.Time) time.Duration {
return time.Since(start).Truncate(10 * time.Microsecond)
}
5 years ago
func main1() int {
defer func() {
if os.Getenv("GARBLE_WRITE_ALLOCS") != "true" {
return
}
var memStats runtime.MemStats
runtime.ReadMemStats(&memStats)
fmt.Fprintf(os.Stderr, "garble allocs: %d\n", memStats.Mallocs)
}()
5 years ago
if err := flagSet.Parse(os.Args[1:]); err != nil {
return 2
}
log.SetPrefix("[garble] ")
log.SetFlags(0) // no timestamps, as they aren't very useful
if flagDebug {
// TODO: cover this in the tests.
log.SetOutput(&uniqueLineWriter{out: os.Stderr})
} else {
log.SetOutput(io.Discard)
}
5 years ago
args := flagSet.Args()
if len(args) < 1 {
usage()
return 2
5 years ago
}
if err := mainErr(args); err != nil {
if code, ok := err.(errJustExit); ok {
return int(code)
}
fmt.Fprintln(os.Stderr, err)
// If the build failed and a random seed was used,
// the failure might not reproduce with a different seed.
// Print it before we exit.
if flagSeed.random {
fmt.Fprintf(os.Stderr, "random seed: %s\n", base64.RawStdEncoding.EncodeToString(flagSeed.bytes))
}
return 1
}
return 0
}
type errJustExit int
func (e errJustExit) Error() string { return fmt.Sprintf("exit: %d", e) }
// toolchainVersionSemver is a semver-compatible version of the Go toolchain currently
// being used, as reported by "go env GOVERSION".
// Note that the version of Go that built the garble binary might be newer.
var toolchainVersionSemver string
func goVersionOK() bool {
const (
minGoVersionSemver = "v1.18.0"
suggestedGoVersion = "1.18.x"
)
// rxVersion looks for a version like "go1.2" or "go1.2.3"
rxVersion := regexp.MustCompile(`go\d+\.\d+(\.\d+)?`)
toolchainVersionFull := cache.GoEnv.GOVERSION
toolchainVersion := rxVersion.FindString(cache.GoEnv.GOVERSION)
if toolchainVersion == "" {
// Go 1.15.x and older do not have GOVERSION yet.
// We could go the extra mile and fetch it via 'go toolchainVersion',
// but we'd have to error anyway.
fmt.Fprintf(os.Stderr, "Go version is too old; please upgrade to Go %s or a newer devel version\n", suggestedGoVersion)
return false
}
toolchainVersionSemver = "v" + strings.TrimPrefix(toolchainVersion, "go")
if semver.Compare(toolchainVersionSemver, minGoVersionSemver) < 0 {
fmt.Fprintf(os.Stderr, "Go version %q is too old; please upgrade to Go %s\n", toolchainVersionFull, suggestedGoVersion)
return false
}
// Ensure that the version of Go that built the garble binary is equal or
// newer than toolchainVersionSemver.
builtVersionFull := os.Getenv("GARBLE_TEST_GOVERSION")
if builtVersionFull == "" {
builtVersionFull = runtime.Version()
}
builtVersion := rxVersion.FindString(builtVersionFull)
if builtVersion == "" {
// If garble built itself, we don't know what Go version was used.
// Fall back to not performing the check against the toolchain version.
return true
}
builtVersionSemver := "v" + strings.TrimPrefix(builtVersion, "go")
if semver.Compare(builtVersionSemver, toolchainVersionSemver) < 0 {
fmt.Fprintf(os.Stderr, "garble was built with %q and is being used with %q; please rebuild garble with the newer version\n",
builtVersionFull, toolchainVersionFull)
return false
}
return true
}
func mainErr(args []string) error {
// If we recognize an argument, we're not running within -toolexec.
switch command, args := args[0], args[1:]; command {
case "help":
if hasHelpFlag(args) || len(args) > 1 {
fmt.Fprintf(os.Stderr, "usage: garble help [command]\n")
return errJustExit(2)
}
if len(args) == 1 {
return mainErr([]string{args[0], "-h"})
}
usage()
return errJustExit(2)
case "version":
if hasHelpFlag(args) || len(args) > 0 {
fmt.Fprintf(os.Stderr, "usage: garble version\n")
return errJustExit(2)
}
// don't overwrite the version if it was set by -ldflags=-X
if info, ok := debug.ReadBuildInfo(); ok && version == "(devel)" {
mod := &info.Main
if mod.Replace != nil {
mod = mod.Replace
}
version = mod.Version
}
fmt.Println(version)
return nil
case "reverse":
return commandReverse(args)
case "build", "test":
cmd, err := toolexecCmd(command, args)
if err != nil {
return err
}
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
debugf("calling via toolexec: %s", cmd)
return cmd.Run()
}
if !filepath.IsAbs(args[0]) {
// -toolexec gives us an absolute path to the tool binary to
// run, so this is most likely misuse of garble by a user.
return fmt.Errorf("unknown command: %q", args[0])
}
// We're in a toolexec sub-process, not directly called by the user.
// Load the shared data and wrap the tool, like the compiler or linker.
start using original action IDs (#251) When we obfuscate a name, what we do is hash the name with the action ID of the package that contains the name. To ensure that the hash changes if the garble tool changes, we used the action ID of the obfuscated build, which is different than the original action ID, as we include garble's own content ID in "go tool compile -V=full" via -toolexec. Let's call that the "obfuscated action ID". Remember that a content ID is roughly the hash of a binary or object file, and an action ID contains the hash of a package's source code plus the content IDs of its dependencies. This had the advantage that it did what we wanted. However, it had one massive drawback: when we compile a package, we only have the obfuscated action IDs of its dependencies. This is because one can't have the content ID of dependent packages before they are built. Usually, this is not a problem, because hashing a foreign name means it comes from a dependency, where we already have the obfuscated action ID. However, that's not always the case. First, go:linkname directives can point to any symbol that ends up in the binary, even if the package is not a dependency. So garble could only support linkname targets belonging to dependencies. This is at the root of why we could not obfuscate the runtime; it contains linkname directives targeting the net package, for example, which depends on runtime. Second, some other places did not have an easy access to obfuscated action IDs, like transformAsm, which had to recover it from a temporary file stored by transformCompile. Plus, this was all pretty expensive, as each toolexec sub-process had to make repeated calls to buildidOf with the object files of dependencies. We even had to use extra calls to "go list" in the case of indirect dependencies, as their export files do not appear in importcfg files. All in all, the old method was complex and expensive. A better mechanism is to use the original action IDs directly, as listed by "go list" without garble in the picture. This would mean that the hashing does not change if garble changes, meaning weaker obfuscation. To regain that property, we define the "garble action ID", which is just the original action ID hashed together with garble's own content ID. This is practically the same as the obfuscated build ID we used before, but since it doesn't go through "go tool compile -V=full" and the obfuscated build itself, we can work out *all* the garble action IDs upfront, before the obfuscated build even starts. This fixes all of our problems. Now we know all garble build IDs upfront, so a bunch of hacks can be entirely removed. Plus, since we know them upfront, we can also cache them and avoid repeated calls to "go tool buildid". While at it, make use of the new BuildID field in Go 1.16's "list -json -export". This avoids the vast majority of "go tool buildid" calls, as the only ones that remain are 2 on the garble binary itself. The numbers for Go 1.16 look very good: name old time/op new time/op delta Build-8 146ms ± 4% 101ms ± 1% -31.01% (p=0.002 n=6+6) name old bin-B new bin-B delta Build-8 6.61M ± 0% 6.60M ± 0% -0.09% (p=0.002 n=6+6) name old sys-time/op new sys-time/op delta Build-8 321ms ± 7% 202ms ± 6% -37.11% (p=0.002 n=6+6) name old user-time/op new user-time/op delta Build-8 538ms ± 4% 414ms ± 4% -23.12% (p=0.002 n=6+6)
3 years ago
if err := loadSharedCache(); err != nil {
return err
}
5 years ago
_, tool := filepath.Split(args[0])
if runtime.GOOS == "windows" {
tool = strings.TrimSuffix(tool, ".exe")
}
if len(args) == 2 && args[1] == "-V=full" {
return alterToolVersion(tool, args)
}
refactor "current package" with TOOLEXEC_IMPORTPATH (#266) Now that we've dropped support for Go 1.15.x, we can finally rely on this environment variable for toolexec calls, present in Go 1.16. Before, we had hacky ways of trying to figure out the current package's import path, mostly from the -p flag. The biggest rough edge there was that, for main packages, that was simply the package name, and not its full import path. To work around that, we had a restriction on a single main package, so we could work around that issue. That restriction is now gone. The new code is simpler, especially because we can set curPkg in a single place for all toolexec transform funcs. Since we can always rely on curPkg not being nil now, we can also start reusing listedPackage.Private and avoid the majority of repeated calls to isPrivate. The function is cheap, but still not free. isPrivate itself can also get simpler. We no longer have to worry about the "main" edge case. Plus, the sanity check for invalid package paths is now unnecessary; we only got malformed paths from goobj2, and we now require exact matches with the ImportPath field from "go list -json". Another effect of clearing up the "main" edge case is that -debugdir now uses the right directory for main packages. We also start using consistent debugdir paths in the tests, for the sake of being easier to read and maintain. Finally, note that commandReverse did not need the extra call to "go list -toolexec", as the "shared" call stored in the cache is enough. We still call toolexecCmd to get said cache, which should probably be simplified in a future PR. While at it, replace the use of the "-std" compiler flag with the Standard field from "go list -json".
3 years ago
toolexecImportPath := os.Getenv("TOOLEXEC_IMPORTPATH")
refactor "current package" with TOOLEXEC_IMPORTPATH (#266) Now that we've dropped support for Go 1.15.x, we can finally rely on this environment variable for toolexec calls, present in Go 1.16. Before, we had hacky ways of trying to figure out the current package's import path, mostly from the -p flag. The biggest rough edge there was that, for main packages, that was simply the package name, and not its full import path. To work around that, we had a restriction on a single main package, so we could work around that issue. That restriction is now gone. The new code is simpler, especially because we can set curPkg in a single place for all toolexec transform funcs. Since we can always rely on curPkg not being nil now, we can also start reusing listedPackage.Private and avoid the majority of repeated calls to isPrivate. The function is cheap, but still not free. isPrivate itself can also get simpler. We no longer have to worry about the "main" edge case. Plus, the sanity check for invalid package paths is now unnecessary; we only got malformed paths from goobj2, and we now require exact matches with the ImportPath field from "go list -json". Another effect of clearing up the "main" edge case is that -debugdir now uses the right directory for main packages. We also start using consistent debugdir paths in the tests, for the sake of being easier to read and maintain. Finally, note that commandReverse did not need the extra call to "go list -toolexec", as the "shared" call stored in the cache is enough. We still call toolexecCmd to get said cache, which should probably be simplified in a future PR. While at it, replace the use of the "-std" compiler flag with the Standard field from "go list -json".
3 years ago
curPkg = cache.ListedPackages[toolexecImportPath]
if curPkg == nil {
return fmt.Errorf("TOOLEXEC_IMPORTPATH not found in listed packages: %s", toolexecImportPath)
}
transform := transformFuncs[tool]
5 years ago
transformed := args[1:]
if transform != nil {
startTime := time.Now()
debugf("transforming %s with args: %s", tool, strings.Join(transformed, " "))
var err error
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
if transformed, err = transform(transformed); err != nil {
return err
5 years ago
}
debugf("transformed args for %s in %s: %s", tool, debugSince(startTime), strings.Join(transformed, " "))
} else {
debugf("skipping transform on %s with args: %s", tool, strings.Join(transformed, " "))
5 years ago
}
cmd := exec.Command(args[0], transformed...)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
if err := cmd.Run(); err != nil {
return err
5 years ago
}
return nil
5 years ago
}
func hasHelpFlag(flags []string) bool {
for _, f := range flags {
switch f {
case "-h", "-help", "--help":
return true
}
}
return false
}
// toolexecCmd builds an *exec.Cmd which is set up for running "go <command>"
// with -toolexec=garble and the supplied arguments.
//
// Note that it uses and modifies global state; in general, it should only be
// called once from mainErr in the top-level garble process.
func toolexecCmd(command string, args []string) (*exec.Cmd, error) {
// Split the flags from the package arguments, since we'll need
// to run 'go list' on the same set of packages.
flags, args := splitFlagsFromArgs(args)
if hasHelpFlag(flags) {
out, _ := exec.Command("go", command, "-h").CombinedOutput()
fmt.Fprintf(os.Stderr, `
usage: garble [garble flags] %s [arguments]
This command wraps "go %s". Below is its help:
%s`[1:], command, command, out)
return nil, errJustExit(2)
}
for _, flag := range flags {
if rxGarbleFlag.MatchString(flag) {
return nil, fmt.Errorf("garble flags must precede command, like: garble %s build ./pkg", flag)
}
}
start using original action IDs (#251) When we obfuscate a name, what we do is hash the name with the action ID of the package that contains the name. To ensure that the hash changes if the garble tool changes, we used the action ID of the obfuscated build, which is different than the original action ID, as we include garble's own content ID in "go tool compile -V=full" via -toolexec. Let's call that the "obfuscated action ID". Remember that a content ID is roughly the hash of a binary or object file, and an action ID contains the hash of a package's source code plus the content IDs of its dependencies. This had the advantage that it did what we wanted. However, it had one massive drawback: when we compile a package, we only have the obfuscated action IDs of its dependencies. This is because one can't have the content ID of dependent packages before they are built. Usually, this is not a problem, because hashing a foreign name means it comes from a dependency, where we already have the obfuscated action ID. However, that's not always the case. First, go:linkname directives can point to any symbol that ends up in the binary, even if the package is not a dependency. So garble could only support linkname targets belonging to dependencies. This is at the root of why we could not obfuscate the runtime; it contains linkname directives targeting the net package, for example, which depends on runtime. Second, some other places did not have an easy access to obfuscated action IDs, like transformAsm, which had to recover it from a temporary file stored by transformCompile. Plus, this was all pretty expensive, as each toolexec sub-process had to make repeated calls to buildidOf with the object files of dependencies. We even had to use extra calls to "go list" in the case of indirect dependencies, as their export files do not appear in importcfg files. All in all, the old method was complex and expensive. A better mechanism is to use the original action IDs directly, as listed by "go list" without garble in the picture. This would mean that the hashing does not change if garble changes, meaning weaker obfuscation. To regain that property, we define the "garble action ID", which is just the original action ID hashed together with garble's own content ID. This is practically the same as the obfuscated build ID we used before, but since it doesn't go through "go tool compile -V=full" and the obfuscated build itself, we can work out *all* the garble action IDs upfront, before the obfuscated build even starts. This fixes all of our problems. Now we know all garble build IDs upfront, so a bunch of hacks can be entirely removed. Plus, since we know them upfront, we can also cache them and avoid repeated calls to "go tool buildid". While at it, make use of the new BuildID field in Go 1.16's "list -json -export". This avoids the vast majority of "go tool buildid" calls, as the only ones that remain are 2 on the garble binary itself. The numbers for Go 1.16 look very good: name old time/op new time/op delta Build-8 146ms ± 4% 101ms ± 1% -31.01% (p=0.002 n=6+6) name old bin-B new bin-B delta Build-8 6.61M ± 0% 6.60M ± 0% -0.09% (p=0.002 n=6+6) name old sys-time/op new sys-time/op delta Build-8 321ms ± 7% 202ms ± 6% -37.11% (p=0.002 n=6+6) name old user-time/op new user-time/op delta Build-8 538ms ± 4% 414ms ± 4% -23.12% (p=0.002 n=6+6)
3 years ago
// Here is the only place we initialize the cache.
// The sub-processes will parse it from a shared gob file.
cache = &sharedCache{}
start using original action IDs (#251) When we obfuscate a name, what we do is hash the name with the action ID of the package that contains the name. To ensure that the hash changes if the garble tool changes, we used the action ID of the obfuscated build, which is different than the original action ID, as we include garble's own content ID in "go tool compile -V=full" via -toolexec. Let's call that the "obfuscated action ID". Remember that a content ID is roughly the hash of a binary or object file, and an action ID contains the hash of a package's source code plus the content IDs of its dependencies. This had the advantage that it did what we wanted. However, it had one massive drawback: when we compile a package, we only have the obfuscated action IDs of its dependencies. This is because one can't have the content ID of dependent packages before they are built. Usually, this is not a problem, because hashing a foreign name means it comes from a dependency, where we already have the obfuscated action ID. However, that's not always the case. First, go:linkname directives can point to any symbol that ends up in the binary, even if the package is not a dependency. So garble could only support linkname targets belonging to dependencies. This is at the root of why we could not obfuscate the runtime; it contains linkname directives targeting the net package, for example, which depends on runtime. Second, some other places did not have an easy access to obfuscated action IDs, like transformAsm, which had to recover it from a temporary file stored by transformCompile. Plus, this was all pretty expensive, as each toolexec sub-process had to make repeated calls to buildidOf with the object files of dependencies. We even had to use extra calls to "go list" in the case of indirect dependencies, as their export files do not appear in importcfg files. All in all, the old method was complex and expensive. A better mechanism is to use the original action IDs directly, as listed by "go list" without garble in the picture. This would mean that the hashing does not change if garble changes, meaning weaker obfuscation. To regain that property, we define the "garble action ID", which is just the original action ID hashed together with garble's own content ID. This is practically the same as the obfuscated build ID we used before, but since it doesn't go through "go tool compile -V=full" and the obfuscated build itself, we can work out *all* the garble action IDs upfront, before the obfuscated build even starts. This fixes all of our problems. Now we know all garble build IDs upfront, so a bunch of hacks can be entirely removed. Plus, since we know them upfront, we can also cache them and avoid repeated calls to "go tool buildid". While at it, make use of the new BuildID field in Go 1.16's "list -json -export". This avoids the vast majority of "go tool buildid" calls, as the only ones that remain are 2 on the garble binary itself. The numbers for Go 1.16 look very good: name old time/op new time/op delta Build-8 146ms ± 4% 101ms ± 1% -31.01% (p=0.002 n=6+6) name old bin-B new bin-B delta Build-8 6.61M ± 0% 6.60M ± 0% -0.09% (p=0.002 n=6+6) name old sys-time/op new sys-time/op delta Build-8 321ms ± 7% 202ms ± 6% -37.11% (p=0.002 n=6+6) name old user-time/op new user-time/op delta Build-8 538ms ± 4% 414ms ± 4% -23.12% (p=0.002 n=6+6)
3 years ago
// Note that we also need to pass build flags to 'go list', such
// as -tags.
fail if we are unexpectedly overwriting files (#418) While investigating a bug report, I noticed that garble was writing to the same temp file twice. At best, writing to the same path on disk twice is wasteful, as the design is careful to be deterministic and use unique paths. At worst, the two writes could cause races at the filesystem level. To prevent either of those situations, we now create files with os.OpenFile and os.O_EXCL, meaning that we will error if the file already exists. That change uncovered a number of such unintended cases. First, transformAsm would write obfuscated Go files twice. This is because the Go toolchain actually runs: [...]/asm -gensymabis [...] foo.s bar.s [...]/asm [...] foo.s bar.s That is, the first run is only meant to generate symbol ABIs, which are then used by the compiler. We need to obfuscate at that first stage, because the symbol ABI descriptions need to use obfuscated names. However, having already obfuscated the assembly on the first stage, there is no need to do so again on the second stage. If we detect gensymabis is missing, we simply reuse the previous files. This first situation doesn't seem racy, but obfuscating the Go assembly files twice is certainly unnecessary. Second, saveKnownReflectAPIs wrote a gob file to the build cache. Since the build cache can be kept between builds, and since the build cache uses reproducible paths for each build, running the same "garble build" twice could overwrite those files. This could actually cause races at the filesystem level; if two concurrent builds write to the same gob file on disk, one of them could end up using a partially-written file. Note that this is the only of the three cases not using temporary files. As such, it is expected that the file may already exist. In such a case, we simply avoid overwriting it rather than failing. Third, when "garble build -a" was used, and when we needed an export file not listed in importcfg, we would end up calling roughly: go list -export -toolexec=garble -a <dependency> This meant we would re-build and re-obfuscate those packages. Which is unfortunate, because the parent process already did via: go build -toolexec=garble -a <main> The repeated dependency builds tripped the new os.O_EXCL check, as we would try to overwrite the same obfuscated Go files. Beyond being wasteful, this could again cause subtle filesystem races. To fix the problem, avoid passing flags like "-a" to nested go commands. Overall, we should likely be using safer ways to write to disk, be it via either atomic writes or locked files. However, for now, catching duplicate writes is a big step. I have left a self-assigned TODO for further improvements. CI on the pull request found a failure on test-gotip. The failure reproduces on master, so it seems to be related to gotip, and not a regression introduced by this change. For now, disable test-gotip until we can investigate.
3 years ago
cache.ForwardBuildFlags, _ = filterForwardBuildFlags(flags)
if command == "test" {
fail if we are unexpectedly overwriting files (#418) While investigating a bug report, I noticed that garble was writing to the same temp file twice. At best, writing to the same path on disk twice is wasteful, as the design is careful to be deterministic and use unique paths. At worst, the two writes could cause races at the filesystem level. To prevent either of those situations, we now create files with os.OpenFile and os.O_EXCL, meaning that we will error if the file already exists. That change uncovered a number of such unintended cases. First, transformAsm would write obfuscated Go files twice. This is because the Go toolchain actually runs: [...]/asm -gensymabis [...] foo.s bar.s [...]/asm [...] foo.s bar.s That is, the first run is only meant to generate symbol ABIs, which are then used by the compiler. We need to obfuscate at that first stage, because the symbol ABI descriptions need to use obfuscated names. However, having already obfuscated the assembly on the first stage, there is no need to do so again on the second stage. If we detect gensymabis is missing, we simply reuse the previous files. This first situation doesn't seem racy, but obfuscating the Go assembly files twice is certainly unnecessary. Second, saveKnownReflectAPIs wrote a gob file to the build cache. Since the build cache can be kept between builds, and since the build cache uses reproducible paths for each build, running the same "garble build" twice could overwrite those files. This could actually cause races at the filesystem level; if two concurrent builds write to the same gob file on disk, one of them could end up using a partially-written file. Note that this is the only of the three cases not using temporary files. As such, it is expected that the file may already exist. In such a case, we simply avoid overwriting it rather than failing. Third, when "garble build -a" was used, and when we needed an export file not listed in importcfg, we would end up calling roughly: go list -export -toolexec=garble -a <dependency> This meant we would re-build and re-obfuscate those packages. Which is unfortunate, because the parent process already did via: go build -toolexec=garble -a <main> The repeated dependency builds tripped the new os.O_EXCL check, as we would try to overwrite the same obfuscated Go files. Beyond being wasteful, this could again cause subtle filesystem races. To fix the problem, avoid passing flags like "-a" to nested go commands. Overall, we should likely be using safer ways to write to disk, be it via either atomic writes or locked files. However, for now, catching duplicate writes is a big step. I have left a self-assigned TODO for further improvements. CI on the pull request found a failure on test-gotip. The failure reproduces on master, so it seems to be related to gotip, and not a regression introduced by this change. For now, disable test-gotip until we can investigate.
3 years ago
cache.ForwardBuildFlags = append(cache.ForwardBuildFlags, "-test")
}
if err := fetchGoEnv(); err != nil {
return nil, err
}
if !goVersionOK() {
return nil, errJustExit(1)
}
var err error
cache.ExecPath, err = os.Executable()
if err != nil {
return nil, err
}
ensure the runtime is built in a reproducible way We went to great lengths to ensure garble builds are reproducible. This includes how the tool itself works, as its behavior should be the same given the same inputs. However, we made one crucial mistake with the runtime package. It has go:linkname directives pointing at other packages, and some of those pointed packages aren't its dependencies. Imagine two scenarios where garble builds the runtime package: 1) We run "garble build runtime". The way we handle linkname directives calls listPackage on the target package, to obfuscate the target's import path and object name. However, since we only obtained build info of runtime and its deps, calls for some linknames such as listPackage("sync/atomic") will fail. The linkname directive will leave its target untouched. 2) We run "garble build std". Unlike the first scenario, all listPackage calls issued by runtime's linkname directives will succeed, so its linkname directive targets will be obfuscated. At best, this can result in inconsistent builds, depending on how the runtime package was built. At worst, the mismatching object names can result in errors at link time, if the target packages are actually used. The modified test reproduces the worst case scenario reliably, when the fix is reverted: > env GOCACHE=${WORK}/gocache-empty > garble build -a runtime > garble build -o=out_rebuild ./stdimporter [stderr] # test/main/stdimporter JZzQivnl.NtQJu0H3: relocation target JZzQivnl.iioHinYT not defined JZzQivnl.NtQJu0H3.func9: relocation target JZzQivnl.yz5z0NaH not defined JZzQivnl.(*ypvqhKiQ).String: relocation target JZzQivnl.eVciBQeI not defined JZzQivnl.(*ypvqhKiQ).PkgPath: relocation target JZzQivnl.eVciBQeI not defined [...] The fix consists of two steps. First, if we're building the runtime and listPackage fails on a package, that means we ran into scenario 1 above. To avoid the inconsistency, we fill ListedPackages with "go list [...] std". This means we'll always build runtime as described in scenario 2 above. Second, when building packages other than the runtime, we only allow listPackage to succeed if we're listing a dependency of the current package. This ensures we won't run into similar reproducibility bugs in the future. Finally, re-enable test-gotip on CI since this was the last test flake.
3 years ago
binaryBuildID, err := buildidOf(cache.ExecPath)
if err != nil {
return nil, err
}
cache.BinaryContentID = decodeHash(splitContentID(binaryBuildID))
if err := appendListedPackages(args, true); err != nil {
return nil, err
}
start using original action IDs (#251) When we obfuscate a name, what we do is hash the name with the action ID of the package that contains the name. To ensure that the hash changes if the garble tool changes, we used the action ID of the obfuscated build, which is different than the original action ID, as we include garble's own content ID in "go tool compile -V=full" via -toolexec. Let's call that the "obfuscated action ID". Remember that a content ID is roughly the hash of a binary or object file, and an action ID contains the hash of a package's source code plus the content IDs of its dependencies. This had the advantage that it did what we wanted. However, it had one massive drawback: when we compile a package, we only have the obfuscated action IDs of its dependencies. This is because one can't have the content ID of dependent packages before they are built. Usually, this is not a problem, because hashing a foreign name means it comes from a dependency, where we already have the obfuscated action ID. However, that's not always the case. First, go:linkname directives can point to any symbol that ends up in the binary, even if the package is not a dependency. So garble could only support linkname targets belonging to dependencies. This is at the root of why we could not obfuscate the runtime; it contains linkname directives targeting the net package, for example, which depends on runtime. Second, some other places did not have an easy access to obfuscated action IDs, like transformAsm, which had to recover it from a temporary file stored by transformCompile. Plus, this was all pretty expensive, as each toolexec sub-process had to make repeated calls to buildidOf with the object files of dependencies. We even had to use extra calls to "go list" in the case of indirect dependencies, as their export files do not appear in importcfg files. All in all, the old method was complex and expensive. A better mechanism is to use the original action IDs directly, as listed by "go list" without garble in the picture. This would mean that the hashing does not change if garble changes, meaning weaker obfuscation. To regain that property, we define the "garble action ID", which is just the original action ID hashed together with garble's own content ID. This is practically the same as the obfuscated build ID we used before, but since it doesn't go through "go tool compile -V=full" and the obfuscated build itself, we can work out *all* the garble action IDs upfront, before the obfuscated build even starts. This fixes all of our problems. Now we know all garble build IDs upfront, so a bunch of hacks can be entirely removed. Plus, since we know them upfront, we can also cache them and avoid repeated calls to "go tool buildid". While at it, make use of the new BuildID field in Go 1.16's "list -json -export". This avoids the vast majority of "go tool buildid" calls, as the only ones that remain are 2 on the garble binary itself. The numbers for Go 1.16 look very good: name old time/op new time/op delta Build-8 146ms ± 4% 101ms ± 1% -31.01% (p=0.002 n=6+6) name old bin-B new bin-B delta Build-8 6.61M ± 0% 6.60M ± 0% -0.09% (p=0.002 n=6+6) name old sys-time/op new sys-time/op delta Build-8 321ms ± 7% 202ms ± 6% -37.11% (p=0.002 n=6+6) name old user-time/op new user-time/op delta Build-8 538ms ± 4% 414ms ± 4% -23.12% (p=0.002 n=6+6)
3 years ago
sharedTempDir, err = saveSharedCache()
if err != nil {
return nil, err
}
os.Setenv("GARBLE_SHARED", sharedTempDir)
defer os.Remove(sharedTempDir)
wd, err := os.Getwd()
if err != nil {
return nil, err
}
os.Setenv("GARBLE_PARENT_WORK", wd)
if flagDebugDir != "" {
if !filepath.IsAbs(flagDebugDir) {
flagDebugDir = filepath.Join(wd, flagDebugDir)
}
if err := os.RemoveAll(flagDebugDir); err != nil {
return nil, fmt.Errorf("could not empty debugdir: %v", err)
}
if err := os.MkdirAll(flagDebugDir, 0o755); err != nil {
return nil, err
}
}
goArgs := []string{
command,
"-trimpath",
"-buildvcs=false",
}
// Pass the garble flags down to each toolexec invocation.
// This way, all garble processes see the same flag values.
var toolexecFlag strings.Builder
toolexecFlag.WriteString("-toolexec=")
toolexecFlag.WriteString(cache.ExecPath)
appendFlags(&toolexecFlag, false)
goArgs = append(goArgs, toolexecFlag.String())
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
if flagDebugDir != "" {
// In case the user deletes the debug directory,
// and a previous build is cached,
// rebuild all packages to re-fill the debug dir.
goArgs = append(goArgs, "-a")
}
if command == "test" {
// vet is generally not useful on obfuscated code; keep it
// disabled by default.
goArgs = append(goArgs, "-vet=off")
}
goArgs = append(goArgs, flags...)
goArgs = append(goArgs, args...)
return exec.Command("go", goArgs...), nil
}
var transformFuncs = map[string]func([]string) ([]string, error){
"asm": transformAsm,
"compile": transformCompile,
"link": transformLink,
}
func transformAsm(args []string) ([]string, error) {
if !curPkg.ToObfuscate {
return args, nil // we're not obfuscating this package
}
flags, paths := splitFlagsFromFiles(args, ".s")
// When assembling, the import path can make its way into the output object file.
if curPkg.Name != "main" {
refactor "current package" with TOOLEXEC_IMPORTPATH (#266) Now that we've dropped support for Go 1.15.x, we can finally rely on this environment variable for toolexec calls, present in Go 1.16. Before, we had hacky ways of trying to figure out the current package's import path, mostly from the -p flag. The biggest rough edge there was that, for main packages, that was simply the package name, and not its full import path. To work around that, we had a restriction on a single main package, so we could work around that issue. That restriction is now gone. The new code is simpler, especially because we can set curPkg in a single place for all toolexec transform funcs. Since we can always rely on curPkg not being nil now, we can also start reusing listedPackage.Private and avoid the majority of repeated calls to isPrivate. The function is cheap, but still not free. isPrivate itself can also get simpler. We no longer have to worry about the "main" edge case. Plus, the sanity check for invalid package paths is now unnecessary; we only got malformed paths from goobj2, and we now require exact matches with the ImportPath field from "go list -json". Another effect of clearing up the "main" edge case is that -debugdir now uses the right directory for main packages. We also start using consistent debugdir paths in the tests, for the sake of being easier to read and maintain. Finally, note that commandReverse did not need the extra call to "go list -toolexec", as the "shared" call stored in the cache is enough. We still call toolexecCmd to get said cache, which should probably be simplified in a future PR. While at it, replace the use of the "-std" compiler flag with the Standard field from "go list -json".
3 years ago
flags = flagSetValue(flags, "-p", curPkg.obfuscatedImportPath())
}
flags = alterTrimpath(flags)
avoid reproducibility issues with full rebuilds We were using temporary filenames for modified Go and assembly files. For example, an obfuscated "encoding/json/encode.go" would end up as: /tmp/garble-shared123/encode.go.456.go where "123" and "456" are random numbers, usually longer. This was usually fine for two reasons: 1) We would add "/tmp/garble-shared123/" to -trimpath, so the temporary directory and its random number would be invisible. 2) We would add "//line" directives to the source files, replacing the filename with obfuscated versions excluding any random number. Unfortunately, this broke in multiple ways. Most notably, assembly files do not have any line directives, and it's not clear that there's any support for them. So the random number in their basename could end up in the binary, breaking reproducibility. Another issue is that the -trimpath addition described above was only done for cmd/compile, not cmd/asm, so assembly filenames included the randomized temporary directory. To fix the issues above, the same "encoding/json/encode.go" would now end up as: /tmp/garble-shared123/encoding/json/encode.go Such a path is still unique even though the "456" random number is gone, as import paths are unique within a single build. This fixes issues with the base name of each file, so we no longer rely on line directives as the only way to remove the second original random number. We still rely on -trimpath to get rid of the temporary directory in filenames. To fix its problem with assembly files, also amend the -trimpath flag when running the assembler tool. Finally, add a test that reproducible builds still work when a full rebuild is done. We choose goprivate.txt for such a test as its stdimporter package imports a number of std packages, including uses of assembly and cgo. For the time being, we don't use such a "full rebuild" reproducibility test in other test scripts, as this step is expensive, rebuilding many packages from scratch. This issue went unnoticed for over a year because such random numbers "123" and "456" were created when a package was obfuscated, and that only happened once per package version as long as the build cache was kept intact. When clearing the build cache, or forcing a rebuild with -a, one gets new random numbers, and thus a different binary resulting from the same build input. That's not something that most users would do regularly, and our tests did not cover that edge case either, until now. Fixes #328.
3 years ago
fail if we are unexpectedly overwriting files (#418) While investigating a bug report, I noticed that garble was writing to the same temp file twice. At best, writing to the same path on disk twice is wasteful, as the design is careful to be deterministic and use unique paths. At worst, the two writes could cause races at the filesystem level. To prevent either of those situations, we now create files with os.OpenFile and os.O_EXCL, meaning that we will error if the file already exists. That change uncovered a number of such unintended cases. First, transformAsm would write obfuscated Go files twice. This is because the Go toolchain actually runs: [...]/asm -gensymabis [...] foo.s bar.s [...]/asm [...] foo.s bar.s That is, the first run is only meant to generate symbol ABIs, which are then used by the compiler. We need to obfuscate at that first stage, because the symbol ABI descriptions need to use obfuscated names. However, having already obfuscated the assembly on the first stage, there is no need to do so again on the second stage. If we detect gensymabis is missing, we simply reuse the previous files. This first situation doesn't seem racy, but obfuscating the Go assembly files twice is certainly unnecessary. Second, saveKnownReflectAPIs wrote a gob file to the build cache. Since the build cache can be kept between builds, and since the build cache uses reproducible paths for each build, running the same "garble build" twice could overwrite those files. This could actually cause races at the filesystem level; if two concurrent builds write to the same gob file on disk, one of them could end up using a partially-written file. Note that this is the only of the three cases not using temporary files. As such, it is expected that the file may already exist. In such a case, we simply avoid overwriting it rather than failing. Third, when "garble build -a" was used, and when we needed an export file not listed in importcfg, we would end up calling roughly: go list -export -toolexec=garble -a <dependency> This meant we would re-build and re-obfuscate those packages. Which is unfortunate, because the parent process already did via: go build -toolexec=garble -a <main> The repeated dependency builds tripped the new os.O_EXCL check, as we would try to overwrite the same obfuscated Go files. Beyond being wasteful, this could again cause subtle filesystem races. To fix the problem, avoid passing flags like "-a" to nested go commands. Overall, we should likely be using safer ways to write to disk, be it via either atomic writes or locked files. However, for now, catching duplicate writes is a big step. I have left a self-assigned TODO for further improvements. CI on the pull request found a failure on test-gotip. The failure reproduces on master, so it seems to be related to gotip, and not a regression introduced by this change. For now, disable test-gotip until we can investigate.
3 years ago
// If the assembler is running just for -gensymabis,
// don't obfuscate the source, as we are not assembling yet.
// The assembler will run again later; obfuscating twice is just wasteful.
symabis := false
for _, arg := range args {
if arg == "-gensymabis" {
symabis = true
break
}
}
newPaths := make([]string, 0, len(paths))
if !symabis {
var newPaths []string
for _, path := range paths {
name := filepath.Base(path)
pkgDir := filepath.Join(sharedTempDir, filepath.FromSlash(curPkg.ImportPath))
newPath := filepath.Join(pkgDir, name)
newPaths = append(newPaths, newPath)
}
return append(flags, newPaths...), nil
}
// We need to replace all function references with their obfuscated name
// counterparts.
// Luckily, all func names in Go assembly files are immediately followed
// by the unicode "middle dot", like:
//
// TEXT ·privateAdd(SB),$0-24
const middleDot = '·'
middleDotLen := utf8.RuneLen(middleDot)
for _, path := range paths {
// Read the entire file into memory.
// If we find issues with large files, we can use bufio.
content, err := os.ReadFile(path)
if err != nil {
return nil, err
}
// Find all middle-dot names, and replace them.
remaining := content
var buf bytes.Buffer
for {
i := bytes.IndexRune(remaining, middleDot)
if i < 0 {
buf.Write(remaining)
remaining = nil
break
}
// We want to replace "OP ·foo" and "OP $·foo",
// but not "OP somepkg·foo" just yet.
// "somepkg" is often runtime, syscall, etc.
// We don't obfuscate any of those for now.
//
// TODO: we'll likely need to deal with this
// when we start obfuscating the runtime.
// When we do, note that we can't hash with curPkg.
localName := false
if i >= 0 {
switch remaining[i-1] {
case ' ', '\t', '$':
localName = true
}
}
i += middleDotLen
buf.Write(remaining[:i])
remaining = remaining[i:]
// The name ends at the first rune which cannot be part
// of a Go identifier, such as a comma or space.
nameEnd := 0
for nameEnd < len(remaining) {
c, size := utf8.DecodeRune(remaining[nameEnd:])
if !unicode.IsLetter(c) && c != '_' && !unicode.IsDigit(c) {
break
}
nameEnd += size
}
name := string(remaining[:nameEnd])
remaining = remaining[nameEnd:]
if !localName {
buf.WriteString(name)
continue
}
newName := hashWithPackage(curPkg, name)
debugf("asm name %q hashed with %x to %q", name, curPkg.GarbleActionID, newName)
buf.WriteString(newName)
}
// Uncomment for some quick debugging. Do not delete.
// if curPkg.ToObfuscate {
// fmt.Fprintf(os.Stderr, "\n-- %s --\n%s", path, buf.Bytes())
// }
avoid reproducibility issues with full rebuilds We were using temporary filenames for modified Go and assembly files. For example, an obfuscated "encoding/json/encode.go" would end up as: /tmp/garble-shared123/encode.go.456.go where "123" and "456" are random numbers, usually longer. This was usually fine for two reasons: 1) We would add "/tmp/garble-shared123/" to -trimpath, so the temporary directory and its random number would be invisible. 2) We would add "//line" directives to the source files, replacing the filename with obfuscated versions excluding any random number. Unfortunately, this broke in multiple ways. Most notably, assembly files do not have any line directives, and it's not clear that there's any support for them. So the random number in their basename could end up in the binary, breaking reproducibility. Another issue is that the -trimpath addition described above was only done for cmd/compile, not cmd/asm, so assembly filenames included the randomized temporary directory. To fix the issues above, the same "encoding/json/encode.go" would now end up as: /tmp/garble-shared123/encoding/json/encode.go Such a path is still unique even though the "456" random number is gone, as import paths are unique within a single build. This fixes issues with the base name of each file, so we no longer rely on line directives as the only way to remove the second original random number. We still rely on -trimpath to get rid of the temporary directory in filenames. To fix its problem with assembly files, also amend the -trimpath flag when running the assembler tool. Finally, add a test that reproducible builds still work when a full rebuild is done. We choose goprivate.txt for such a test as its stdimporter package imports a number of std packages, including uses of assembly and cgo. For the time being, we don't use such a "full rebuild" reproducibility test in other test scripts, as this step is expensive, rebuilding many packages from scratch. This issue went unnoticed for over a year because such random numbers "123" and "456" were created when a package was obfuscated, and that only happened once per package version as long as the build cache was kept intact. When clearing the build cache, or forcing a rebuild with -a, one gets new random numbers, and thus a different binary resulting from the same build input. That's not something that most users would do regularly, and our tests did not cover that edge case either, until now. Fixes #328.
3 years ago
name := filepath.Base(path)
if path, err := writeTemp(name, buf.Bytes()); err != nil {
return nil, err
} else {
newPaths = append(newPaths, path)
}
}
return append(flags, newPaths...), nil
}
// writeTemp is a mix between os.CreateTemp and os.WriteFile, as it writes a
avoid reproducibility issues with full rebuilds We were using temporary filenames for modified Go and assembly files. For example, an obfuscated "encoding/json/encode.go" would end up as: /tmp/garble-shared123/encode.go.456.go where "123" and "456" are random numbers, usually longer. This was usually fine for two reasons: 1) We would add "/tmp/garble-shared123/" to -trimpath, so the temporary directory and its random number would be invisible. 2) We would add "//line" directives to the source files, replacing the filename with obfuscated versions excluding any random number. Unfortunately, this broke in multiple ways. Most notably, assembly files do not have any line directives, and it's not clear that there's any support for them. So the random number in their basename could end up in the binary, breaking reproducibility. Another issue is that the -trimpath addition described above was only done for cmd/compile, not cmd/asm, so assembly filenames included the randomized temporary directory. To fix the issues above, the same "encoding/json/encode.go" would now end up as: /tmp/garble-shared123/encoding/json/encode.go Such a path is still unique even though the "456" random number is gone, as import paths are unique within a single build. This fixes issues with the base name of each file, so we no longer rely on line directives as the only way to remove the second original random number. We still rely on -trimpath to get rid of the temporary directory in filenames. To fix its problem with assembly files, also amend the -trimpath flag when running the assembler tool. Finally, add a test that reproducible builds still work when a full rebuild is done. We choose goprivate.txt for such a test as its stdimporter package imports a number of std packages, including uses of assembly and cgo. For the time being, we don't use such a "full rebuild" reproducibility test in other test scripts, as this step is expensive, rebuilding many packages from scratch. This issue went unnoticed for over a year because such random numbers "123" and "456" were created when a package was obfuscated, and that only happened once per package version as long as the build cache was kept intact. When clearing the build cache, or forcing a rebuild with -a, one gets new random numbers, and thus a different binary resulting from the same build input. That's not something that most users would do regularly, and our tests did not cover that edge case either, until now. Fixes #328.
3 years ago
// named source file in sharedTempDir given an input buffer.
//
avoid reproducibility issues with full rebuilds We were using temporary filenames for modified Go and assembly files. For example, an obfuscated "encoding/json/encode.go" would end up as: /tmp/garble-shared123/encode.go.456.go where "123" and "456" are random numbers, usually longer. This was usually fine for two reasons: 1) We would add "/tmp/garble-shared123/" to -trimpath, so the temporary directory and its random number would be invisible. 2) We would add "//line" directives to the source files, replacing the filename with obfuscated versions excluding any random number. Unfortunately, this broke in multiple ways. Most notably, assembly files do not have any line directives, and it's not clear that there's any support for them. So the random number in their basename could end up in the binary, breaking reproducibility. Another issue is that the -trimpath addition described above was only done for cmd/compile, not cmd/asm, so assembly filenames included the randomized temporary directory. To fix the issues above, the same "encoding/json/encode.go" would now end up as: /tmp/garble-shared123/encoding/json/encode.go Such a path is still unique even though the "456" random number is gone, as import paths are unique within a single build. This fixes issues with the base name of each file, so we no longer rely on line directives as the only way to remove the second original random number. We still rely on -trimpath to get rid of the temporary directory in filenames. To fix its problem with assembly files, also amend the -trimpath flag when running the assembler tool. Finally, add a test that reproducible builds still work when a full rebuild is done. We choose goprivate.txt for such a test as its stdimporter package imports a number of std packages, including uses of assembly and cgo. For the time being, we don't use such a "full rebuild" reproducibility test in other test scripts, as this step is expensive, rebuilding many packages from scratch. This issue went unnoticed for over a year because such random numbers "123" and "456" were created when a package was obfuscated, and that only happened once per package version as long as the build cache was kept intact. When clearing the build cache, or forcing a rebuild with -a, one gets new random numbers, and thus a different binary resulting from the same build input. That's not something that most users would do regularly, and our tests did not cover that edge case either, until now. Fixes #328.
3 years ago
// Note that the file is created under a directory tree following curPkg's
// import path, mimicking how files are laid out in modules and GOROOT.
func writeTemp(name string, content []byte) (string, error) {
avoid reproducibility issues with full rebuilds We were using temporary filenames for modified Go and assembly files. For example, an obfuscated "encoding/json/encode.go" would end up as: /tmp/garble-shared123/encode.go.456.go where "123" and "456" are random numbers, usually longer. This was usually fine for two reasons: 1) We would add "/tmp/garble-shared123/" to -trimpath, so the temporary directory and its random number would be invisible. 2) We would add "//line" directives to the source files, replacing the filename with obfuscated versions excluding any random number. Unfortunately, this broke in multiple ways. Most notably, assembly files do not have any line directives, and it's not clear that there's any support for them. So the random number in their basename could end up in the binary, breaking reproducibility. Another issue is that the -trimpath addition described above was only done for cmd/compile, not cmd/asm, so assembly filenames included the randomized temporary directory. To fix the issues above, the same "encoding/json/encode.go" would now end up as: /tmp/garble-shared123/encoding/json/encode.go Such a path is still unique even though the "456" random number is gone, as import paths are unique within a single build. This fixes issues with the base name of each file, so we no longer rely on line directives as the only way to remove the second original random number. We still rely on -trimpath to get rid of the temporary directory in filenames. To fix its problem with assembly files, also amend the -trimpath flag when running the assembler tool. Finally, add a test that reproducible builds still work when a full rebuild is done. We choose goprivate.txt for such a test as its stdimporter package imports a number of std packages, including uses of assembly and cgo. For the time being, we don't use such a "full rebuild" reproducibility test in other test scripts, as this step is expensive, rebuilding many packages from scratch. This issue went unnoticed for over a year because such random numbers "123" and "456" were created when a package was obfuscated, and that only happened once per package version as long as the build cache was kept intact. When clearing the build cache, or forcing a rebuild with -a, one gets new random numbers, and thus a different binary resulting from the same build input. That's not something that most users would do regularly, and our tests did not cover that edge case either, until now. Fixes #328.
3 years ago
pkgDir := filepath.Join(sharedTempDir, filepath.FromSlash(curPkg.ImportPath))
if err := os.MkdirAll(pkgDir, 0o777); err != nil {
return "", err
}
avoid reproducibility issues with full rebuilds We were using temporary filenames for modified Go and assembly files. For example, an obfuscated "encoding/json/encode.go" would end up as: /tmp/garble-shared123/encode.go.456.go where "123" and "456" are random numbers, usually longer. This was usually fine for two reasons: 1) We would add "/tmp/garble-shared123/" to -trimpath, so the temporary directory and its random number would be invisible. 2) We would add "//line" directives to the source files, replacing the filename with obfuscated versions excluding any random number. Unfortunately, this broke in multiple ways. Most notably, assembly files do not have any line directives, and it's not clear that there's any support for them. So the random number in their basename could end up in the binary, breaking reproducibility. Another issue is that the -trimpath addition described above was only done for cmd/compile, not cmd/asm, so assembly filenames included the randomized temporary directory. To fix the issues above, the same "encoding/json/encode.go" would now end up as: /tmp/garble-shared123/encoding/json/encode.go Such a path is still unique even though the "456" random number is gone, as import paths are unique within a single build. This fixes issues with the base name of each file, so we no longer rely on line directives as the only way to remove the second original random number. We still rely on -trimpath to get rid of the temporary directory in filenames. To fix its problem with assembly files, also amend the -trimpath flag when running the assembler tool. Finally, add a test that reproducible builds still work when a full rebuild is done. We choose goprivate.txt for such a test as its stdimporter package imports a number of std packages, including uses of assembly and cgo. For the time being, we don't use such a "full rebuild" reproducibility test in other test scripts, as this step is expensive, rebuilding many packages from scratch. This issue went unnoticed for over a year because such random numbers "123" and "456" were created when a package was obfuscated, and that only happened once per package version as long as the build cache was kept intact. When clearing the build cache, or forcing a rebuild with -a, one gets new random numbers, and thus a different binary resulting from the same build input. That's not something that most users would do regularly, and our tests did not cover that edge case either, until now. Fixes #328.
3 years ago
dstPath := filepath.Join(pkgDir, name)
fail if we are unexpectedly overwriting files (#418) While investigating a bug report, I noticed that garble was writing to the same temp file twice. At best, writing to the same path on disk twice is wasteful, as the design is careful to be deterministic and use unique paths. At worst, the two writes could cause races at the filesystem level. To prevent either of those situations, we now create files with os.OpenFile and os.O_EXCL, meaning that we will error if the file already exists. That change uncovered a number of such unintended cases. First, transformAsm would write obfuscated Go files twice. This is because the Go toolchain actually runs: [...]/asm -gensymabis [...] foo.s bar.s [...]/asm [...] foo.s bar.s That is, the first run is only meant to generate symbol ABIs, which are then used by the compiler. We need to obfuscate at that first stage, because the symbol ABI descriptions need to use obfuscated names. However, having already obfuscated the assembly on the first stage, there is no need to do so again on the second stage. If we detect gensymabis is missing, we simply reuse the previous files. This first situation doesn't seem racy, but obfuscating the Go assembly files twice is certainly unnecessary. Second, saveKnownReflectAPIs wrote a gob file to the build cache. Since the build cache can be kept between builds, and since the build cache uses reproducible paths for each build, running the same "garble build" twice could overwrite those files. This could actually cause races at the filesystem level; if two concurrent builds write to the same gob file on disk, one of them could end up using a partially-written file. Note that this is the only of the three cases not using temporary files. As such, it is expected that the file may already exist. In such a case, we simply avoid overwriting it rather than failing. Third, when "garble build -a" was used, and when we needed an export file not listed in importcfg, we would end up calling roughly: go list -export -toolexec=garble -a <dependency> This meant we would re-build and re-obfuscate those packages. Which is unfortunate, because the parent process already did via: go build -toolexec=garble -a <main> The repeated dependency builds tripped the new os.O_EXCL check, as we would try to overwrite the same obfuscated Go files. Beyond being wasteful, this could again cause subtle filesystem races. To fix the problem, avoid passing flags like "-a" to nested go commands. Overall, we should likely be using safer ways to write to disk, be it via either atomic writes or locked files. However, for now, catching duplicate writes is a big step. I have left a self-assigned TODO for further improvements. CI on the pull request found a failure on test-gotip. The failure reproduces on master, so it seems to be related to gotip, and not a regression introduced by this change. For now, disable test-gotip until we can investigate.
3 years ago
if err := writeFileExclusive(dstPath, content); err != nil {
return "", err
}
avoid reproducibility issues with full rebuilds We were using temporary filenames for modified Go and assembly files. For example, an obfuscated "encoding/json/encode.go" would end up as: /tmp/garble-shared123/encode.go.456.go where "123" and "456" are random numbers, usually longer. This was usually fine for two reasons: 1) We would add "/tmp/garble-shared123/" to -trimpath, so the temporary directory and its random number would be invisible. 2) We would add "//line" directives to the source files, replacing the filename with obfuscated versions excluding any random number. Unfortunately, this broke in multiple ways. Most notably, assembly files do not have any line directives, and it's not clear that there's any support for them. So the random number in their basename could end up in the binary, breaking reproducibility. Another issue is that the -trimpath addition described above was only done for cmd/compile, not cmd/asm, so assembly filenames included the randomized temporary directory. To fix the issues above, the same "encoding/json/encode.go" would now end up as: /tmp/garble-shared123/encoding/json/encode.go Such a path is still unique even though the "456" random number is gone, as import paths are unique within a single build. This fixes issues with the base name of each file, so we no longer rely on line directives as the only way to remove the second original random number. We still rely on -trimpath to get rid of the temporary directory in filenames. To fix its problem with assembly files, also amend the -trimpath flag when running the assembler tool. Finally, add a test that reproducible builds still work when a full rebuild is done. We choose goprivate.txt for such a test as its stdimporter package imports a number of std packages, including uses of assembly and cgo. For the time being, we don't use such a "full rebuild" reproducibility test in other test scripts, as this step is expensive, rebuilding many packages from scratch. This issue went unnoticed for over a year because such random numbers "123" and "456" were created when a package was obfuscated, and that only happened once per package version as long as the build cache was kept intact. When clearing the build cache, or forcing a rebuild with -a, one gets new random numbers, and thus a different binary resulting from the same build input. That's not something that most users would do regularly, and our tests did not cover that edge case either, until now. Fixes #328.
3 years ago
return dstPath, nil
}
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
func transformCompile(args []string) ([]string, error) {
var err error
flags, paths := splitFlagsFromFiles(args, ".go")
// We will force the linker to drop DWARF via -w, so don't spend time
// generating it.
flags = append(flags, "-dwarf=false")
for i, path := range paths {
if filepath.Base(path) == "_gomod_.go" {
// never include module info
// TODO: this seems to no longer trigger for our tests?
paths = append(paths[:i], paths[i+1:]...)
break
}
}
var files []*ast.File
for _, path := range paths {
file, err := parser.ParseFile(fset, path, nil, parser.ParseComments)
if err != nil {
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
return nil, err
}
files = append(files, file)
}
tf := newTransformer()
if err := tf.typecheck(files); err != nil {
return nil, err
}
flags = alterTrimpath(flags)
fail if we are unexpectedly overwriting files (#418) While investigating a bug report, I noticed that garble was writing to the same temp file twice. At best, writing to the same path on disk twice is wasteful, as the design is careful to be deterministic and use unique paths. At worst, the two writes could cause races at the filesystem level. To prevent either of those situations, we now create files with os.OpenFile and os.O_EXCL, meaning that we will error if the file already exists. That change uncovered a number of such unintended cases. First, transformAsm would write obfuscated Go files twice. This is because the Go toolchain actually runs: [...]/asm -gensymabis [...] foo.s bar.s [...]/asm [...] foo.s bar.s That is, the first run is only meant to generate symbol ABIs, which are then used by the compiler. We need to obfuscate at that first stage, because the symbol ABI descriptions need to use obfuscated names. However, having already obfuscated the assembly on the first stage, there is no need to do so again on the second stage. If we detect gensymabis is missing, we simply reuse the previous files. This first situation doesn't seem racy, but obfuscating the Go assembly files twice is certainly unnecessary. Second, saveKnownReflectAPIs wrote a gob file to the build cache. Since the build cache can be kept between builds, and since the build cache uses reproducible paths for each build, running the same "garble build" twice could overwrite those files. This could actually cause races at the filesystem level; if two concurrent builds write to the same gob file on disk, one of them could end up using a partially-written file. Note that this is the only of the three cases not using temporary files. As such, it is expected that the file may already exist. In such a case, we simply avoid overwriting it rather than failing. Third, when "garble build -a" was used, and when we needed an export file not listed in importcfg, we would end up calling roughly: go list -export -toolexec=garble -a <dependency> This meant we would re-build and re-obfuscate those packages. Which is unfortunate, because the parent process already did via: go build -toolexec=garble -a <main> The repeated dependency builds tripped the new os.O_EXCL check, as we would try to overwrite the same obfuscated Go files. Beyond being wasteful, this could again cause subtle filesystem races. To fix the problem, avoid passing flags like "-a" to nested go commands. Overall, we should likely be using safer ways to write to disk, be it via either atomic writes or locked files. However, for now, catching duplicate writes is a big step. I have left a self-assigned TODO for further improvements. CI on the pull request found a failure on test-gotip. The failure reproduces on master, so it seems to be related to gotip, and not a regression introduced by this change. For now, disable test-gotip until we can investigate.
3 years ago
// Note that if the file already exists in the cache from another build,
// we don't need to write to it again thanks to the hash.
// TODO: as an optimization, just load that one gob file.
if err := loadCachedOutputs(); err != nil {
return nil, err
}
tf.findReflectFunctions(files)
newImportCfg, err := processImportCfg(flags)
if err != nil {
return nil, err
}
// Literal obfuscation uses math/rand, so seed it deterministically.
randSeed := curPkg.GarbleActionID
if flagSeed.present() {
randSeed = flagSeed.bytes
}
// debugf("seeding math/rand with %x\n", randSeed)
mathrand.Seed(int64(binary.BigEndian.Uint64(randSeed)))
if err := tf.prefillObjectMaps(files); err != nil {
return nil, err
}
// If this is a package to obfuscate, swap the -p flag with the new package path.
// We don't if it's the main package, as that just uses "-p main".
// We only set newPkgPath if we're obfuscating the import path,
// to replace the original package name in the package clause below.
refactor "current package" with TOOLEXEC_IMPORTPATH (#266) Now that we've dropped support for Go 1.15.x, we can finally rely on this environment variable for toolexec calls, present in Go 1.16. Before, we had hacky ways of trying to figure out the current package's import path, mostly from the -p flag. The biggest rough edge there was that, for main packages, that was simply the package name, and not its full import path. To work around that, we had a restriction on a single main package, so we could work around that issue. That restriction is now gone. The new code is simpler, especially because we can set curPkg in a single place for all toolexec transform funcs. Since we can always rely on curPkg not being nil now, we can also start reusing listedPackage.Private and avoid the majority of repeated calls to isPrivate. The function is cheap, but still not free. isPrivate itself can also get simpler. We no longer have to worry about the "main" edge case. Plus, the sanity check for invalid package paths is now unnecessary; we only got malformed paths from goobj2, and we now require exact matches with the ImportPath field from "go list -json". Another effect of clearing up the "main" edge case is that -debugdir now uses the right directory for main packages. We also start using consistent debugdir paths in the tests, for the sake of being easier to read and maintain. Finally, note that commandReverse did not need the extra call to "go list -toolexec", as the "shared" call stored in the cache is enough. We still call toolexecCmd to get said cache, which should probably be simplified in a future PR. While at it, replace the use of the "-std" compiler flag with the Standard field from "go list -json".
3 years ago
newPkgPath := ""
if curPkg.Name != "main" && curPkg.ToObfuscate {
newPkgPath = curPkg.obfuscatedImportPath()
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
flags = flagSetValue(flags, "-p", newPkgPath)
}
newPaths := make([]string, 0, len(files))
for i, file := range files {
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
filename := filepath.Base(paths[i])
debugf("obfuscating %s", filename)
if curPkg.ImportPath == "runtime" && flagTiny {
// strip unneeded runtime code
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
stripRuntime(filename, file)
}
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
tf.handleDirectives(file.Comments)
file = tf.transformGo(file)
refactor "current package" with TOOLEXEC_IMPORTPATH (#266) Now that we've dropped support for Go 1.15.x, we can finally rely on this environment variable for toolexec calls, present in Go 1.16. Before, we had hacky ways of trying to figure out the current package's import path, mostly from the -p flag. The biggest rough edge there was that, for main packages, that was simply the package name, and not its full import path. To work around that, we had a restriction on a single main package, so we could work around that issue. That restriction is now gone. The new code is simpler, especially because we can set curPkg in a single place for all toolexec transform funcs. Since we can always rely on curPkg not being nil now, we can also start reusing listedPackage.Private and avoid the majority of repeated calls to isPrivate. The function is cheap, but still not free. isPrivate itself can also get simpler. We no longer have to worry about the "main" edge case. Plus, the sanity check for invalid package paths is now unnecessary; we only got malformed paths from goobj2, and we now require exact matches with the ImportPath field from "go list -json". Another effect of clearing up the "main" edge case is that -debugdir now uses the right directory for main packages. We also start using consistent debugdir paths in the tests, for the sake of being easier to read and maintain. Finally, note that commandReverse did not need the extra call to "go list -toolexec", as the "shared" call stored in the cache is enough. We still call toolexecCmd to get said cache, which should probably be simplified in a future PR. While at it, replace the use of the "-std" compiler flag with the Standard field from "go list -json".
3 years ago
if newPkgPath != "" {
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
file.Name.Name = newPkgPath
}
src, err := printFile(file)
if err != nil {
return nil, err
}
// It is possible to end up in an edge case where two instances of the
// same package have different Action IDs, but their obfuscation and
// builds produce exactly the same results.
// In such an edge case, Go's build cache is smart enough for the second
// instance to reuse the first's build artifact.
// However, garble's caching via garbleExportFile is not as smart,
// as we base the location of these files purely based on Action IDs.
// Thus, the incremental build can fail to find garble's cached file.
// To sidestep this bug entirely, ensure that different action IDs never
// produce the same cached output when building with garble.
// Note that this edge case tends to happen when a -seed is provided,
// as then a package's Action ID is not used as an obfuscation seed.
// TODO(mvdan): replace this workaround with an actual fix if we can.
// This workaround is presumably worse on the build cache,
// as we end up with extra near-duplicate cached artifacts.
if i == 0 {
src = append(src, fmt.Sprintf(
"\nvar garbleActionID = %q\n", hashToString(curPkg.GarbleActionID),
)...)
}
// Uncomment for some quick debugging. Do not delete.
// if curPkg.ToObfuscate {
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
// fmt.Fprintf(os.Stderr, "\n-- %s/%s --\n%s", curPkg.ImportPath, filename, src)
// }
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
if path, err := writeTemp(filename, src); err != nil {
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
return nil, err
} else {
newPaths = append(newPaths, path)
}
if flagDebugDir != "" {
refactor "current package" with TOOLEXEC_IMPORTPATH (#266) Now that we've dropped support for Go 1.15.x, we can finally rely on this environment variable for toolexec calls, present in Go 1.16. Before, we had hacky ways of trying to figure out the current package's import path, mostly from the -p flag. The biggest rough edge there was that, for main packages, that was simply the package name, and not its full import path. To work around that, we had a restriction on a single main package, so we could work around that issue. That restriction is now gone. The new code is simpler, especially because we can set curPkg in a single place for all toolexec transform funcs. Since we can always rely on curPkg not being nil now, we can also start reusing listedPackage.Private and avoid the majority of repeated calls to isPrivate. The function is cheap, but still not free. isPrivate itself can also get simpler. We no longer have to worry about the "main" edge case. Plus, the sanity check for invalid package paths is now unnecessary; we only got malformed paths from goobj2, and we now require exact matches with the ImportPath field from "go list -json". Another effect of clearing up the "main" edge case is that -debugdir now uses the right directory for main packages. We also start using consistent debugdir paths in the tests, for the sake of being easier to read and maintain. Finally, note that commandReverse did not need the extra call to "go list -toolexec", as the "shared" call stored in the cache is enough. We still call toolexecCmd to get said cache, which should probably be simplified in a future PR. While at it, replace the use of the "-std" compiler flag with the Standard field from "go list -json".
3 years ago
osPkgPath := filepath.FromSlash(curPkg.ImportPath)
pkgDebugDir := filepath.Join(flagDebugDir, osPkgPath)
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
if err := os.MkdirAll(pkgDebugDir, 0o755); err != nil {
return nil, err
}
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
debugFilePath := filepath.Join(pkgDebugDir, filename)
if err := os.WriteFile(debugFilePath, src, 0o666); err != nil {
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
return nil, err
}
}
}
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
flags = flagSetValue(flags, "-importcfg", newImportCfg)
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
if err := writeGobExclusive(
garbleExportFile(curPkg),
cachedOutput,
); err != nil && !errors.Is(err, fs.ErrExist) {
return nil, err
}
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
return append(flags, newPaths...), nil
}
// handleDirectives looks at all the comments in a file containing build
// directives, and does the necessary for the obfuscation process to work.
//
// Right now, this means recording what local names are used with go:linkname,
// and rewriting those directives to use obfuscated name from other packages.
func (tf *transformer) handleDirectives(comments []*ast.CommentGroup) {
for _, group := range comments {
for _, comment := range group.List {
if !strings.HasPrefix(comment.Text, "//go:linkname ") {
continue
}
fields := strings.Fields(comment.Text)
if len(fields) != 3 {
// TODO: the 2nd argument is optional, handle when it's not present
continue
}
// This directive has two arguments: "go:linkname localName newName"
// obfuscate the local name, if the current package is obfuscated
if curPkg.ToObfuscate {
fields[1] = hashWithPackage(curPkg, fields[1])
}
// If the new name is of the form "pkgpath.Name", and
// we've obfuscated "Name" in that package, rewrite the
// directive to use the obfuscated name.
newName := fields[2]
dotCnt := strings.Count(newName, ".")
if dotCnt < 1 {
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
// cgo-generated code uses linknames to made up symbol names,
// which do not have a package path at all.
// Replace the comment in case the local name was obfuscated.
comment.Text = strings.Join(fields, " ")
continue
}
ensure the runtime is built in a reproducible way We went to great lengths to ensure garble builds are reproducible. This includes how the tool itself works, as its behavior should be the same given the same inputs. However, we made one crucial mistake with the runtime package. It has go:linkname directives pointing at other packages, and some of those pointed packages aren't its dependencies. Imagine two scenarios where garble builds the runtime package: 1) We run "garble build runtime". The way we handle linkname directives calls listPackage on the target package, to obfuscate the target's import path and object name. However, since we only obtained build info of runtime and its deps, calls for some linknames such as listPackage("sync/atomic") will fail. The linkname directive will leave its target untouched. 2) We run "garble build std". Unlike the first scenario, all listPackage calls issued by runtime's linkname directives will succeed, so its linkname directive targets will be obfuscated. At best, this can result in inconsistent builds, depending on how the runtime package was built. At worst, the mismatching object names can result in errors at link time, if the target packages are actually used. The modified test reproduces the worst case scenario reliably, when the fix is reverted: > env GOCACHE=${WORK}/gocache-empty > garble build -a runtime > garble build -o=out_rebuild ./stdimporter [stderr] # test/main/stdimporter JZzQivnl.NtQJu0H3: relocation target JZzQivnl.iioHinYT not defined JZzQivnl.NtQJu0H3.func9: relocation target JZzQivnl.yz5z0NaH not defined JZzQivnl.(*ypvqhKiQ).String: relocation target JZzQivnl.eVciBQeI not defined JZzQivnl.(*ypvqhKiQ).PkgPath: relocation target JZzQivnl.eVciBQeI not defined [...] The fix consists of two steps. First, if we're building the runtime and listPackage fails on a package, that means we ran into scenario 1 above. To avoid the inconsistency, we fill ListedPackages with "go list [...] std". This means we'll always build runtime as described in scenario 2 above. Second, when building packages other than the runtime, we only allow listPackage to succeed if we're listing a dependency of the current package. This ensures we won't run into similar reproducibility bugs in the future. Finally, re-enable test-gotip on CI since this was the last test flake.
3 years ago
switch newName {
case "main.main", "main..inittask", "runtime..inittask":
// The runtime uses some special symbols with "..".
// We aren't touching those at the moment.
continue
}
// If the package path has multiple dots, split on the
// last one.
lastDotIdx := strings.LastIndex(newName, ".")
pkgPath, name := newName[:lastDotIdx], newName[lastDotIdx+1:]
lpkg, err := listPackage(pkgPath)
if err != nil {
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
// Probably a made up name like above, but with a dot.
comment.Text = strings.Join(fields, " ")
continue
}
if lpkg.ToObfuscate {
// The name exists and was obfuscated; obfuscate
// the new name.
newName := hashWithPackage(lpkg, name)
newPkgPath := pkgPath
if pkgPath != "main" {
newPkgPath = lpkg.obfuscatedImportPath()
}
fields[2] = newPkgPath + "." + newName
}
comment.Text = strings.Join(fields, " ")
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
}
}
}
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
// processImportCfg parses the importcfg file passed to a compile or link step.
// It also builds a new importcfg file to account for obfuscated import paths.
func processImportCfg(flags []string) (newImportCfg string, _ error) {
start using original action IDs (#251) When we obfuscate a name, what we do is hash the name with the action ID of the package that contains the name. To ensure that the hash changes if the garble tool changes, we used the action ID of the obfuscated build, which is different than the original action ID, as we include garble's own content ID in "go tool compile -V=full" via -toolexec. Let's call that the "obfuscated action ID". Remember that a content ID is roughly the hash of a binary or object file, and an action ID contains the hash of a package's source code plus the content IDs of its dependencies. This had the advantage that it did what we wanted. However, it had one massive drawback: when we compile a package, we only have the obfuscated action IDs of its dependencies. This is because one can't have the content ID of dependent packages before they are built. Usually, this is not a problem, because hashing a foreign name means it comes from a dependency, where we already have the obfuscated action ID. However, that's not always the case. First, go:linkname directives can point to any symbol that ends up in the binary, even if the package is not a dependency. So garble could only support linkname targets belonging to dependencies. This is at the root of why we could not obfuscate the runtime; it contains linkname directives targeting the net package, for example, which depends on runtime. Second, some other places did not have an easy access to obfuscated action IDs, like transformAsm, which had to recover it from a temporary file stored by transformCompile. Plus, this was all pretty expensive, as each toolexec sub-process had to make repeated calls to buildidOf with the object files of dependencies. We even had to use extra calls to "go list" in the case of indirect dependencies, as their export files do not appear in importcfg files. All in all, the old method was complex and expensive. A better mechanism is to use the original action IDs directly, as listed by "go list" without garble in the picture. This would mean that the hashing does not change if garble changes, meaning weaker obfuscation. To regain that property, we define the "garble action ID", which is just the original action ID hashed together with garble's own content ID. This is practically the same as the obfuscated build ID we used before, but since it doesn't go through "go tool compile -V=full" and the obfuscated build itself, we can work out *all* the garble action IDs upfront, before the obfuscated build even starts. This fixes all of our problems. Now we know all garble build IDs upfront, so a bunch of hacks can be entirely removed. Plus, since we know them upfront, we can also cache them and avoid repeated calls to "go tool buildid". While at it, make use of the new BuildID field in Go 1.16's "list -json -export". This avoids the vast majority of "go tool buildid" calls, as the only ones that remain are 2 on the garble binary itself. The numbers for Go 1.16 look very good: name old time/op new time/op delta Build-8 146ms ± 4% 101ms ± 1% -31.01% (p=0.002 n=6+6) name old bin-B new bin-B delta Build-8 6.61M ± 0% 6.60M ± 0% -0.09% (p=0.002 n=6+6) name old sys-time/op new sys-time/op delta Build-8 321ms ± 7% 202ms ± 6% -37.11% (p=0.002 n=6+6) name old user-time/op new user-time/op delta Build-8 538ms ± 4% 414ms ± 4% -23.12% (p=0.002 n=6+6)
3 years ago
importCfg := flagValue(flags, "-importcfg")
if importCfg == "" {
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
return "", fmt.Errorf("could not find -importcfg argument")
}
data, err := os.ReadFile(importCfg)
if err != nil {
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
return "", err
}
var packagefiles, importmaps [][2]string
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
for _, line := range strings.SplitAfter(string(data), "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
i := strings.IndexByte(line, ' ')
if i < 0 {
continue
}
verb := line[:i]
switch verb {
case "importmap":
args := strings.TrimSpace(line[i+1:])
j := strings.IndexByte(args, '=')
if j < 0 {
continue
}
beforePath, afterPath := args[:j], args[j+1:]
importmaps = append(importmaps, [2]string{beforePath, afterPath})
case "packagefile":
args := strings.TrimSpace(line[i+1:])
j := strings.IndexByte(args, '=')
if j < 0 {
continue
}
importPath, objectPath := args[:j], args[j+1:]
packagefiles = append(packagefiles, [2]string{importPath, objectPath})
}
}
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
// Produce the modified importcfg file.
// This is mainly replacing the obfuscated paths.
// Note that we range over maps, so this is non-deterministic, but that
// should not matter as the file is treated like a lookup table.
newCfg, err := os.CreateTemp(sharedTempDir, "importcfg")
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
if err != nil {
return "", err
}
for _, pair := range importmaps {
beforePath, afterPath := pair[0], pair[1]
lpkg, err := listPackage(beforePath)
if err != nil {
panic(err) // shouldn't happen
}
if lpkg.ToObfuscate {
// Note that beforePath is not the canonical path.
// For beforePath="vendor/foo", afterPath and
// lpkg.ImportPath can be just "foo".
// Don't use obfuscatedImportPath here.
beforePath = hashWithPackage(lpkg, beforePath)
afterPath = lpkg.obfuscatedImportPath()
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
}
fmt.Fprintf(newCfg, "importmap %s=%s\n", beforePath, afterPath)
}
for _, pair := range packagefiles {
impPath, pkgfile := pair[0], pair[1]
lpkg, err := listPackage(impPath)
if err != nil {
panic(err) // shouldn't happen
}
if lpkg.Name != "main" {
impPath = lpkg.obfuscatedImportPath()
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
}
fmt.Fprintf(newCfg, "packagefile %s=%s\n", impPath, pkgfile)
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
}
// Uncomment to debug the transformed importcfg. Do not delete.
// newCfg.Seek(0, 0)
// io.Copy(os.Stderr, newCfg)
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
if err := newCfg.Close(); err != nil {
return "", err
}
return newCfg.Name(), nil
}
type (
properly record when type aliases are embedded as fields There are two scenarios when it comes to embedding fields. The first is easy, and we always handled it well: type Named struct { Foo int } type T struct { Named } In this scenario, T ends up with an embedded field named "Named", and a promoted field named "Foo". Then there's the form with a type alias: type Named struct { Foo int } type Alias = Named type T struct { Alias } This case is different: T ends up with an embedded field named "Alias", and a promoted field named "Foo". Note how the field gets its name from the referenced type, even if said type is just an alias to another type. This poses two problems. First, we must obfuscate the field T.Alias as the name "Alias", and not as the name "Named" that the alias points to. Second, we must be careful of cases where Named and Alias are declared in different packages, as they will obfuscate the same name differently. Both of those problems compounded in the reported issue. The actual reason is that quic-go has a type alias in the form of: type ConnectionState = qtls.ConnectionState In other words, the entire problem boils down to a type alias which points to a named type in a different package, where both types share the same name. For example: package parent import "parent/p1" type T struct { p1.SameName } [...] package p1 import "parent/p2" type SameName = p2.SameName [...] package p2 type SameName struct { Foo int } This broke garble because we had a heuristic to detect when an embedded field was a type alias: // Instead, detect such a "foreign alias embed". // If we embed a final named type, // but the field name does not match its name, // then it must have been done via an alias. // We dig out the alias's TypeName via locateForeignAlias. if named.Obj().Name() != node.Name { As the reader can deduce, this heuristic would incorrectly assume that the snippet above does not embed a type alias, when in fact it does. When obfuscating the field T.SameName, which uses a type alias, we would correctly obfuscate the name "SameName", but we would incorrectly obfuscate it with the package p2, not p1. This would then result in build errors. To fix this problem for good, we need to get rid of the heuristic. Instead, we now mimic what was done for KnownCannotObfuscate, but for embedded fields which use type aliases. KnownEmbeddedAliasFields is now filled for each package and stored in the cache as part of cachedOutput. We can then detect the "embedded alias" case reliably, even when the field is declared in an imported package. On the plus side, we get to remove locateForeignAlias. We also add a couple of TODOs to record further improvements. Finally, add a test. Fixes #466.
2 years ago
funcFullName = string // as per go/types.Func.FullName
objectString = string // as per recordedObjectString
reflectParameterPosition = int
properly record when type aliases are embedded as fields There are two scenarios when it comes to embedding fields. The first is easy, and we always handled it well: type Named struct { Foo int } type T struct { Named } In this scenario, T ends up with an embedded field named "Named", and a promoted field named "Foo". Then there's the form with a type alias: type Named struct { Foo int } type Alias = Named type T struct { Alias } This case is different: T ends up with an embedded field named "Alias", and a promoted field named "Foo". Note how the field gets its name from the referenced type, even if said type is just an alias to another type. This poses two problems. First, we must obfuscate the field T.Alias as the name "Alias", and not as the name "Named" that the alias points to. Second, we must be careful of cases where Named and Alias are declared in different packages, as they will obfuscate the same name differently. Both of those problems compounded in the reported issue. The actual reason is that quic-go has a type alias in the form of: type ConnectionState = qtls.ConnectionState In other words, the entire problem boils down to a type alias which points to a named type in a different package, where both types share the same name. For example: package parent import "parent/p1" type T struct { p1.SameName } [...] package p1 import "parent/p2" type SameName = p2.SameName [...] package p2 type SameName struct { Foo int } This broke garble because we had a heuristic to detect when an embedded field was a type alias: // Instead, detect such a "foreign alias embed". // If we embed a final named type, // but the field name does not match its name, // then it must have been done via an alias. // We dig out the alias's TypeName via locateForeignAlias. if named.Obj().Name() != node.Name { As the reader can deduce, this heuristic would incorrectly assume that the snippet above does not embed a type alias, when in fact it does. When obfuscating the field T.SameName, which uses a type alias, we would correctly obfuscate the name "SameName", but we would incorrectly obfuscate it with the package p2, not p1. This would then result in build errors. To fix this problem for good, we need to get rid of the heuristic. Instead, we now mimic what was done for KnownCannotObfuscate, but for embedded fields which use type aliases. KnownEmbeddedAliasFields is now filled for each package and stored in the cache as part of cachedOutput. We can then detect the "embedded alias" case reliably, even when the field is declared in an imported package. On the plus side, we get to remove locateForeignAlias. We also add a couple of TODOs to record further improvements. Finally, add a test. Fixes #466.
2 years ago
typeName struct {
PkgPath, Name string
}
)
// TODO: read-write globals like these should probably be inside transformer
// knownCannotObfuscateUnexported is like KnownCannotObfuscate but for
// unexported names. We don't need to store this in the build cache,
// because these names cannot be referenced by downstream packages.
var knownCannotObfuscateUnexported = map[types.Object]bool{}
// cachedOutput contains information that will be stored as per garbleExportFile.
var cachedOutput = struct {
// KnownReflectAPIs is a static record of what std APIs use reflection on their
// parameters, so we can avoid obfuscating types used with them.
//
// TODO: we're not including fmt.Printf, as it would have many false positives,
// unless we were smart enough to detect which arguments get used as %#v or %T.
KnownReflectAPIs map[funcFullName][]reflectParameterPosition
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
// KnownCannotObfuscate is filled with the fully qualified names from each
// package that we could not obfuscate as per cannotObfuscateNames.
// This record is necessary for knowing what names from imported packages
// weren't obfuscated, so we can obfuscate their local uses accordingly.
properly record when type aliases are embedded as fields There are two scenarios when it comes to embedding fields. The first is easy, and we always handled it well: type Named struct { Foo int } type T struct { Named } In this scenario, T ends up with an embedded field named "Named", and a promoted field named "Foo". Then there's the form with a type alias: type Named struct { Foo int } type Alias = Named type T struct { Alias } This case is different: T ends up with an embedded field named "Alias", and a promoted field named "Foo". Note how the field gets its name from the referenced type, even if said type is just an alias to another type. This poses two problems. First, we must obfuscate the field T.Alias as the name "Alias", and not as the name "Named" that the alias points to. Second, we must be careful of cases where Named and Alias are declared in different packages, as they will obfuscate the same name differently. Both of those problems compounded in the reported issue. The actual reason is that quic-go has a type alias in the form of: type ConnectionState = qtls.ConnectionState In other words, the entire problem boils down to a type alias which points to a named type in a different package, where both types share the same name. For example: package parent import "parent/p1" type T struct { p1.SameName } [...] package p1 import "parent/p2" type SameName = p2.SameName [...] package p2 type SameName struct { Foo int } This broke garble because we had a heuristic to detect when an embedded field was a type alias: // Instead, detect such a "foreign alias embed". // If we embed a final named type, // but the field name does not match its name, // then it must have been done via an alias. // We dig out the alias's TypeName via locateForeignAlias. if named.Obj().Name() != node.Name { As the reader can deduce, this heuristic would incorrectly assume that the snippet above does not embed a type alias, when in fact it does. When obfuscating the field T.SameName, which uses a type alias, we would correctly obfuscate the name "SameName", but we would incorrectly obfuscate it with the package p2, not p1. This would then result in build errors. To fix this problem for good, we need to get rid of the heuristic. Instead, we now mimic what was done for KnownCannotObfuscate, but for embedded fields which use type aliases. KnownEmbeddedAliasFields is now filled for each package and stored in the cache as part of cachedOutput. We can then detect the "embedded alias" case reliably, even when the field is declared in an imported package. On the plus side, we get to remove locateForeignAlias. We also add a couple of TODOs to record further improvements. Finally, add a test. Fixes #466.
2 years ago
KnownCannotObfuscate map[objectString]struct{}
// KnownEmbeddedAliasFields records which embedded fields use a type alias.
// They are the only instance where a type alias matters for obfuscation,
// because the embedded field name is derived from the type alias itself,
// and not the type that the alias points to.
// In that way, the type alias is obfuscated as a form of named type,
// bearing in mind that it may be owned by a different package.
KnownEmbeddedAliasFields map[objectString]typeName
}{
KnownReflectAPIs: map[funcFullName][]reflectParameterPosition{
"reflect.TypeOf": {0},
"reflect.ValueOf": {0},
},
KnownCannotObfuscate: map[objectString]struct{}{},
KnownEmbeddedAliasFields: map[objectString]typeName{},
}
// garbleExportFile returns an absolute path to a build cache entry
// which belongs to garble and corresponds to the given Go package.
//
// Unlike pkg.Export, it is only read and written by garble itself.
// Also unlike pkg.Export, it includes GarbleActionID,
// so its path will change if the obfuscated build changes.
//
// The purpose of such a file is to store garble-specific information
// in the build cache, to be reused at a later time.
// The file should have the same lifetime as pkg.Export,
// as it lives under the same cache directory that gets trimmed automatically.
func garbleExportFile(pkg *listedPackage) string {
trimmed := strings.TrimSuffix(pkg.Export, "-d")
if trimmed == pkg.Export {
panic(fmt.Sprintf("unexpected export path of %s: %q", pkg.ImportPath, pkg.Export))
}
return trimmed + "-garble-" + hashToString(pkg.GarbleActionID) + "-d"
}
func loadCachedOutputs() error {
startTime := time.Now()
loaded := 0
for _, path := range curPkg.Deps {
pkg, err := listPackage(path)
if err != nil {
panic(err) // shouldn't happen
}
if pkg.Export == "" {
continue // nothing to load
}
// this function literal is used for the deferred close
if err := func() error {
filename := garbleExportFile(pkg)
f, err := os.Open(filename)
if err != nil {
return err
}
defer f.Close()
// Decode appends new entries to the existing maps
if err := gob.NewDecoder(f).Decode(&cachedOutput); err != nil {
return fmt.Errorf("gob decode: %w", err)
}
return nil
}(); err != nil {
return fmt.Errorf("cannot load garble export file for %s: %w", path, err)
}
loaded++
}
debugf("%d cached output files loaded in %s", loaded, debugSince(startTime))
return nil
}
func (tf *transformer) findReflectFunctions(files []*ast.File) {
visitReflect := func(node ast.Node) {
funcDecl, ok := node.(*ast.FuncDecl)
if !ok {
return
}
funcObj := tf.info.ObjectOf(funcDecl.Name).(*types.Func)
var paramNames []string
for _, param := range funcDecl.Type.Params.List {
for _, name := range param.Names {
paramNames = append(paramNames, name.Name)
}
}
ast.Inspect(funcDecl, func(node ast.Node) bool {
call, ok := node.(*ast.CallExpr)
if !ok {
return true
}
sel, ok := call.Fun.(*ast.SelectorExpr)
if !ok {
return true
}
fnType, _ := tf.info.ObjectOf(sel.Sel).(*types.Func)
if fnType == nil || fnType.Pkg() == nil {
return true
}
fullName := fnType.FullName()
var identifiers []string
for _, argPos := range cachedOutput.KnownReflectAPIs[fullName] {
arg := call.Args[argPos]
ident, ok := arg.(*ast.Ident)
if !ok {
continue
}
obj := tf.info.ObjectOf(ident)
if obj.Parent() == funcObj.Scope() {
identifiers = append(identifiers, ident.Name)
}
}
if identifiers == nil {
return true
}
var argumentPosReflect []int
for _, ident := range identifiers {
for paramPos, paramName := range paramNames {
if ident == paramName {
argumentPosReflect = append(argumentPosReflect, paramPos)
}
}
}
cachedOutput.KnownReflectAPIs[funcObj.FullName()] = argumentPosReflect
return true
})
}
lenPrevKnownReflectAPIs := len(cachedOutput.KnownReflectAPIs)
for _, file := range files {
for _, decl := range file.Decls {
visitReflect(decl)
}
}
// if a new reflectAPI is found we need to Re-evaluate all functions which might be using that API
if len(cachedOutput.KnownReflectAPIs) > lenPrevKnownReflectAPIs {
tf.findReflectFunctions(files)
}
}
// cmd/bundle will include a go:generate directive in its output by default.
// Ours specifies a version and doesn't assume bundle is in $PATH, so drop it.
//go:generate go run golang.org/x/tools/cmd/bundle@v0.1.9 -o cmdgo_quoted.go -prefix cmdgoQuoted cmd/internal/quoted
//go:generate sed -i /go:generate/d cmdgo_quoted.go
// prefillObjectMaps collects objects which should not be obfuscated,
// such as those used as arguments to reflect.TypeOf or reflect.ValueOf.
// Since we obfuscate one package at a time, we only detect those if the type
// definition and the reflect usage are both in the same package.
func (tf *transformer) prefillObjectMaps(files []*ast.File) error {
tf.linkerVariableStrings = make(map[types.Object]string)
// TODO: this is a linker flag that affects how we obfuscate a package at
// compile time. Note that, if the user changes ldflags, then Go may only
// re-link the final binary, without re-compiling any packages at all.
// It's possible that this could result in:
//
// garble -literals build -ldflags=-X=pkg.name=before # name="before"
// garble -literals build -ldflags=-X=pkg.name=after # name="before" as cached
//
// We haven't been able to reproduce this problem for now,
// but it's worth noting it and keeping an eye out for it in the future.
// If we do confirm this theoretical bug,
// the solution will be to either find a different solution for -literals,
// or to force including -ldflags into the build cache key.
ldflags, err := cmdgoQuotedSplit(flagValue(cache.ForwardBuildFlags, "-ldflags"))
if err != nil {
return err
}
flagValueIter(ldflags, "-X", func(val string) {
// val is in the form of "importpath.name=value".
i := strings.IndexByte(val, '=')
if i < 0 {
return // invalid
}
stringValue := val[i+1:]
val = val[:i] // "importpath.name"
i = strings.LastIndexByte(val, '.')
path, name := val[:i], val[i+1:]
// -X represents the main package as "main", not its import path.
if path != curPkg.ImportPath && !(path == "main" && curPkg.Name == "main") {
return // not the current package
}
obj := tf.pkg.Scope().Lookup(name)
if obj == nil {
return // not found; skip
}
tf.linkerVariableStrings[obj] = stringValue
})
visit := func(node ast.Node) bool {
call, ok := node.(*ast.CallExpr)
if !ok {
return true
}
ident, ok := call.Fun.(*ast.Ident)
if !ok {
sel, ok := call.Fun.(*ast.SelectorExpr)
if !ok {
return true
}
ident = sel.Sel
}
fnType, _ := tf.info.ObjectOf(ident).(*types.Func)
if fnType == nil || fnType.Pkg() == nil {
return true
}
fullName := fnType.FullName()
for _, argPos := range cachedOutput.KnownReflectAPIs[fullName] {
arg := call.Args[argPos]
argType := tf.info.TypeOf(arg)
tf.recursivelyRecordAsNotObfuscated(argType)
}
return true
}
for _, file := range files {
for _, group := range file.Comments {
for _, comment := range group.List {
name := strings.TrimPrefix(comment.Text, "//export ")
if name == comment.Text {
continue
}
name = strings.TrimSpace(name)
obj := tf.pkg.Scope().Lookup(name)
if obj == nil {
continue // not found; skip
}
// TODO(mvdan): it seems like removing this doesn't break any tests.
recordAsNotObfuscated(obj)
}
}
ast.Inspect(file, visit)
}
return nil
}
// transformer holds all the information and state necessary to obfuscate a
// single Go package.
type transformer struct {
// The type-checking results; the package itself, and the Info struct.
pkg *types.Package
info *types.Info
// linkerVariableStrings is also initialized by prefillObjectMaps.
// It records objects for variables used in -ldflags=-X flags,
// as well as the strings the user wants to inject them with.
linkerVariableStrings map[types.Object]string
// recordTypeDone helps avoid cycles in recordType.
recordTypeDone map[types.Type]bool
// fieldToStruct helps locate struct types from any of their field
// objects. Useful when obfuscating field names.
fieldToStruct map[*types.Var]*types.Struct
}
// newTransformer helps initialize some maps.
func newTransformer() *transformer {
return &transformer{
info: &types.Info{
Types: make(map[ast.Expr]types.TypeAndValue),
Defs: make(map[*ast.Ident]types.Object),
Uses: make(map[*ast.Ident]types.Object),
},
recordTypeDone: make(map[types.Type]bool),
fieldToStruct: make(map[*types.Var]*types.Struct),
}
}
func (tf *transformer) typecheck(files []*ast.File) error {
origTypesConfig := types.Config{Importer: origImporter}
pkg, err := origTypesConfig.Check(curPkg.ImportPath, fset, files, tf.info)
if err != nil {
return fmt.Errorf("typecheck error: %v", err)
}
tf.pkg = pkg
// Run recordType on all types reachable via types.Info.
// A bit hacky, but I could not find an easier way to do this.
for _, obj := range tf.info.Defs {
if obj != nil {
tf.recordType(obj.Type())
}
}
for name, obj := range tf.info.Uses {
properly record when type aliases are embedded as fields There are two scenarios when it comes to embedding fields. The first is easy, and we always handled it well: type Named struct { Foo int } type T struct { Named } In this scenario, T ends up with an embedded field named "Named", and a promoted field named "Foo". Then there's the form with a type alias: type Named struct { Foo int } type Alias = Named type T struct { Alias } This case is different: T ends up with an embedded field named "Alias", and a promoted field named "Foo". Note how the field gets its name from the referenced type, even if said type is just an alias to another type. This poses two problems. First, we must obfuscate the field T.Alias as the name "Alias", and not as the name "Named" that the alias points to. Second, we must be careful of cases where Named and Alias are declared in different packages, as they will obfuscate the same name differently. Both of those problems compounded in the reported issue. The actual reason is that quic-go has a type alias in the form of: type ConnectionState = qtls.ConnectionState In other words, the entire problem boils down to a type alias which points to a named type in a different package, where both types share the same name. For example: package parent import "parent/p1" type T struct { p1.SameName } [...] package p1 import "parent/p2" type SameName = p2.SameName [...] package p2 type SameName struct { Foo int } This broke garble because we had a heuristic to detect when an embedded field was a type alias: // Instead, detect such a "foreign alias embed". // If we embed a final named type, // but the field name does not match its name, // then it must have been done via an alias. // We dig out the alias's TypeName via locateForeignAlias. if named.Obj().Name() != node.Name { As the reader can deduce, this heuristic would incorrectly assume that the snippet above does not embed a type alias, when in fact it does. When obfuscating the field T.SameName, which uses a type alias, we would correctly obfuscate the name "SameName", but we would incorrectly obfuscate it with the package p2, not p1. This would then result in build errors. To fix this problem for good, we need to get rid of the heuristic. Instead, we now mimic what was done for KnownCannotObfuscate, but for embedded fields which use type aliases. KnownEmbeddedAliasFields is now filled for each package and stored in the cache as part of cachedOutput. We can then detect the "embedded alias" case reliably, even when the field is declared in an imported package. On the plus side, we get to remove locateForeignAlias. We also add a couple of TODOs to record further improvements. Finally, add a test. Fixes #466.
2 years ago
if obj == nil {
continue
}
properly record when type aliases are embedded as fields There are two scenarios when it comes to embedding fields. The first is easy, and we always handled it well: type Named struct { Foo int } type T struct { Named } In this scenario, T ends up with an embedded field named "Named", and a promoted field named "Foo". Then there's the form with a type alias: type Named struct { Foo int } type Alias = Named type T struct { Alias } This case is different: T ends up with an embedded field named "Alias", and a promoted field named "Foo". Note how the field gets its name from the referenced type, even if said type is just an alias to another type. This poses two problems. First, we must obfuscate the field T.Alias as the name "Alias", and not as the name "Named" that the alias points to. Second, we must be careful of cases where Named and Alias are declared in different packages, as they will obfuscate the same name differently. Both of those problems compounded in the reported issue. The actual reason is that quic-go has a type alias in the form of: type ConnectionState = qtls.ConnectionState In other words, the entire problem boils down to a type alias which points to a named type in a different package, where both types share the same name. For example: package parent import "parent/p1" type T struct { p1.SameName } [...] package p1 import "parent/p2" type SameName = p2.SameName [...] package p2 type SameName struct { Foo int } This broke garble because we had a heuristic to detect when an embedded field was a type alias: // Instead, detect such a "foreign alias embed". // If we embed a final named type, // but the field name does not match its name, // then it must have been done via an alias. // We dig out the alias's TypeName via locateForeignAlias. if named.Obj().Name() != node.Name { As the reader can deduce, this heuristic would incorrectly assume that the snippet above does not embed a type alias, when in fact it does. When obfuscating the field T.SameName, which uses a type alias, we would correctly obfuscate the name "SameName", but we would incorrectly obfuscate it with the package p2, not p1. This would then result in build errors. To fix this problem for good, we need to get rid of the heuristic. Instead, we now mimic what was done for KnownCannotObfuscate, but for embedded fields which use type aliases. KnownEmbeddedAliasFields is now filled for each package and stored in the cache as part of cachedOutput. We can then detect the "embedded alias" case reliably, even when the field is declared in an imported package. On the plus side, we get to remove locateForeignAlias. We also add a couple of TODOs to record further improvements. Finally, add a test. Fixes #466.
2 years ago
tf.recordType(obj.Type())
// Record into KnownEmbeddedAliasFields.
obj, ok := obj.(*types.TypeName)
if !ok || !obj.IsAlias() {
continue
}
vr, _ := tf.info.Defs[name].(*types.Var)
if vr == nil || !vr.Embedded() {
continue
}
vrStr := recordedObjectString(vr)
if vrStr == "" {
continue
}
aliasTypeName := typeName{
PkgPath: obj.Pkg().Path(),
Name: obj.Name(),
}
cachedOutput.KnownEmbeddedAliasFields[vrStr] = aliasTypeName
}
for _, tv := range tf.info.Types {
tf.recordType(tv.Type)
}
return nil
}
// recordType visits every reachable type after typechecking a package.
// Right now, all it does is fill the fieldToStruct field.
// Since types can be recursive, we need a map to avoid cycles.
func (tf *transformer) recordType(t types.Type) {
if tf.recordTypeDone[t] {
return
}
tf.recordTypeDone[t] = true
switch t := t.(type) {
case interface{ Elem() types.Type }:
tf.recordType(t.Elem())
case *types.Named:
tf.recordType(t.Underlying())
}
strct, _ := t.(*types.Struct)
if strct == nil {
return
}
for i := 0; i < strct.NumFields(); i++ {
field := strct.Field(i)
tf.fieldToStruct[field] = strct
if field.Embedded() {
tf.recordType(field.Type())
}
}
}
// TODO: consider caching recordedObjectString via a map,
// if that shows an improvement in our benchmark
properly record when type aliases are embedded as fields There are two scenarios when it comes to embedding fields. The first is easy, and we always handled it well: type Named struct { Foo int } type T struct { Named } In this scenario, T ends up with an embedded field named "Named", and a promoted field named "Foo". Then there's the form with a type alias: type Named struct { Foo int } type Alias = Named type T struct { Alias } This case is different: T ends up with an embedded field named "Alias", and a promoted field named "Foo". Note how the field gets its name from the referenced type, even if said type is just an alias to another type. This poses two problems. First, we must obfuscate the field T.Alias as the name "Alias", and not as the name "Named" that the alias points to. Second, we must be careful of cases where Named and Alias are declared in different packages, as they will obfuscate the same name differently. Both of those problems compounded in the reported issue. The actual reason is that quic-go has a type alias in the form of: type ConnectionState = qtls.ConnectionState In other words, the entire problem boils down to a type alias which points to a named type in a different package, where both types share the same name. For example: package parent import "parent/p1" type T struct { p1.SameName } [...] package p1 import "parent/p2" type SameName = p2.SameName [...] package p2 type SameName struct { Foo int } This broke garble because we had a heuristic to detect when an embedded field was a type alias: // Instead, detect such a "foreign alias embed". // If we embed a final named type, // but the field name does not match its name, // then it must have been done via an alias. // We dig out the alias's TypeName via locateForeignAlias. if named.Obj().Name() != node.Name { As the reader can deduce, this heuristic would incorrectly assume that the snippet above does not embed a type alias, when in fact it does. When obfuscating the field T.SameName, which uses a type alias, we would correctly obfuscate the name "SameName", but we would incorrectly obfuscate it with the package p2, not p1. This would then result in build errors. To fix this problem for good, we need to get rid of the heuristic. Instead, we now mimic what was done for KnownCannotObfuscate, but for embedded fields which use type aliases. KnownEmbeddedAliasFields is now filled for each package and stored in the cache as part of cachedOutput. We can then detect the "embedded alias" case reliably, even when the field is declared in an imported package. On the plus side, we get to remove locateForeignAlias. We also add a couple of TODOs to record further improvements. Finally, add a test. Fixes #466.
2 years ago
func recordedObjectString(obj types.Object) objectString {
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
if obj, ok := obj.(*types.Var); ok && obj.IsField() {
// For exported fields, "pkgpath.Field" is not unique,
// because two exported top-level types could share "Field".
//
// Moreover, note that not all fields belong to named struct types;
// an API could be exposing:
//
// var usedInReflection = struct{Field string}
//
// For now, a hack: assume that packages don't declare the same field
// more than once in the same line. This works in practice, but one
// could craft Go code to break this assumption.
// Also note that the compiler's object files include filenames and line
// numbers, but not column numbers nor byte offsets.
// TODO(mvdan): give this another think, and add tests involving anon types.
pos := fset.Position(obj.Pos())
return fmt.Sprintf("%s.%s - %s:%d", obj.Pkg().Path(), obj.Name(),
filepath.Base(pos.Filename), pos.Line)
}
// Names which are not at the top level cannot be imported,
// so we don't need to record them either.
// Note that this doesn't apply to fields, which are never top-level.
if obj.Pkg().Scope().Lookup(obj.Name()) != obj {
return ""
}
// For top-level exported names, "pkgpath.Name" is unique.
return fmt.Sprintf("%s.%s", obj.Pkg().Path(), obj.Name())
}
// recordAsNotObfuscated records all the objects whose names we cannot obfuscate.
// An object is any named entity, such as a declared variable or type.
//
// So far, it records:
//
// * Types which are used for reflection.
// * Declarations exported via "//export".
// * Types or variables from external packages which were not obfuscated.
func recordAsNotObfuscated(obj types.Object) {
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
if obj.Pkg().Path() != curPkg.ImportPath {
panic("called recordedAsNotObfuscated with a foreign object")
}
if !obj.Exported() {
// Unexported names will never be used by other packages,
// so we don't need to bother recording them in cachedOutput.
knownCannotObfuscateUnexported[obj] = true
return
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
}
objStr := recordedObjectString(obj)
if objStr == "" {
// If the object can't be described via a qualified string,
// then other packages can't use it.
// TODO: should we still record it in knownCannotObfuscateUnexported?
return
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
}
cachedOutput.KnownCannotObfuscate[objStr] = struct{}{}
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
}
func recordedAsNotObfuscated(obj types.Object) bool {
if knownCannotObfuscateUnexported[obj] {
return true
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
}
objStr := recordedObjectString(obj)
if objStr == "" {
return false
}
_, ok := cachedOutput.KnownCannotObfuscate[objStr]
return ok
}
// transformGo obfuscates the provided Go syntax file.
func (tf *transformer) transformGo(file *ast.File) *ast.File {
// Only obfuscate the literals here if the flag is on
// and if the package in question is to be obfuscated.
support GOGARBLE=* with -literals again We recently made an important change when obfuscating the runtime, so that if it's missing any linkname packages in ListedPackages, it does an extra "go list" call to obtain their information. This works very well, but we missed an edge case. In main.go, we disable flagLiterals for the runtime package, but not for other packages like sync/atomic. And, since the runtime's extra "go list" has to compute GarbleActionIDs, it uses the list of garble flags via appendFlags. Unfortunately, it thinks "-literals" isn't set, when it is, and the other packages see it as being set. This discrepancy results in link time errors, as each end of the linkname obfuscates with a different hash: > garble -literals build [stderr] # test/main jccGkbFG.(*yijmzGHo).String: relocation target jccGkbFG.e_77sflf not defined jQg9GEkg.(*NLxfRPAP).pB5p2ZP0: relocation target jQg9GEkg.ce66Fmzl not defined jQg9GEkg.(*NLxfRPAP).pB5p2ZP0: relocation target jQg9GEkg.e5kPa1qY not defined jQg9GEkg.(*NLxfRPAP).pB5p2ZP0: relocation target jQg9GEkg.aQ_3sL3Q not defined jQg9GEkg.(*NLxfRPAP).pB5p2ZP0: relocation target jQg9GEkg.zls3wmws not defined jQg9GEkg.(*NLxfRPAP).pB5p2ZP0: relocation target jQg9GEkg.g69WgKIS not defined To fix the problem, treat flagLiterals as read-only after flag.Parse, just like we already do with the other flags except flagDebugDir. The code that turned flagLiterals to false is no longer needed, as literals.Obfuscate is only called when ToObfuscate is true, and ToObfuscate is false for runtimeAndDeps already.
2 years ago
//
// We can't obfuscate literals in the runtime and its dependencies,
// because obfuscated literals sometimes escape to heap,
// and that's not allowed in the runtime itself.
if flagLiterals && curPkg.ToObfuscate {
file = literals.Obfuscate(file, tf.info, fset, tf.linkerVariableStrings)
// some imported constants might not be needed anymore, remove unnecessary imports
usedImports := make(map[string]bool)
ast.Inspect(file, func(n ast.Node) bool {
node, ok := n.(*ast.Ident)
if !ok {
return true
}
uses, ok := tf.info.Uses[node].(*types.PkgName)
if !ok {
return true
}
usedImports[uses.Imported().Path()] = true
return true
})
for _, imp := range file.Imports {
if imp.Name != nil && (imp.Name.Name == "_" || imp.Name.Name == ".") {
continue
}
path, err := strconv.Unquote(imp.Path.Value)
if err != nil {
panic(err)
}
// The import path can't be used directly here, because the actual
// path resolved via go/types might be different from the naive path.
lpkg, err := listPackage(path)
if err != nil {
panic(err)
}
if usedImports[lpkg.ImportPath] {
continue
}
if !astutil.DeleteImport(fset, file, path) {
panic(fmt.Sprintf("cannot delete unused import: %q", path))
}
}
}
pre := func(cursor *astutil.Cursor) bool {
node, ok := cursor.Node().(*ast.Ident)
if !ok {
return true
}
name := node.Name
if name == "_" {
return true // unnamed remains unnamed
}
obj := tf.info.ObjectOf(node)
if obj == nil {
_, isImplicit := tf.info.Defs[node]
_, parentIsFile := cursor.Parent().(*ast.File)
if !isImplicit || parentIsFile {
// We only care about nil objects in the switch scenario below.
return true
}
// In a type switch like "switch foo := bar.(type) {",
// "foo" is being declared as a symbolic variable,
// as it is only actually declared in each "case SomeType:".
//
// As such, the symbolic "foo" in the syntax tree has no object,
// but it is still recorded under Defs with a nil value.
// We still want to obfuscate that syntax tree identifier,
// so if we detect the case, create a dummy types.Var for it.
//
// Note that "package mypkg" also denotes a nil object in Defs,
// and we don't want to treat that "mypkg" as a variable,
// so avoid that case by checking the type of cursor.Parent.
obj = types.NewVar(node.Pos(), tf.pkg, name, nil)
}
pkg := obj.Pkg()
if vr, ok := obj.(*types.Var); ok && vr.Embedded() {
// The docs for ObjectOf say:
//
// If id is an embedded struct field, ObjectOf returns the
// field (*Var) it defines, not the type (*TypeName) it uses.
//
// If this embedded field is a type alias, we want to
// handle the alias's TypeName instead of treating it as
// the type the alias points to.
//
properly record when type aliases are embedded as fields There are two scenarios when it comes to embedding fields. The first is easy, and we always handled it well: type Named struct { Foo int } type T struct { Named } In this scenario, T ends up with an embedded field named "Named", and a promoted field named "Foo". Then there's the form with a type alias: type Named struct { Foo int } type Alias = Named type T struct { Alias } This case is different: T ends up with an embedded field named "Alias", and a promoted field named "Foo". Note how the field gets its name from the referenced type, even if said type is just an alias to another type. This poses two problems. First, we must obfuscate the field T.Alias as the name "Alias", and not as the name "Named" that the alias points to. Second, we must be careful of cases where Named and Alias are declared in different packages, as they will obfuscate the same name differently. Both of those problems compounded in the reported issue. The actual reason is that quic-go has a type alias in the form of: type ConnectionState = qtls.ConnectionState In other words, the entire problem boils down to a type alias which points to a named type in a different package, where both types share the same name. For example: package parent import "parent/p1" type T struct { p1.SameName } [...] package p1 import "parent/p2" type SameName = p2.SameName [...] package p2 type SameName struct { Foo int } This broke garble because we had a heuristic to detect when an embedded field was a type alias: // Instead, detect such a "foreign alias embed". // If we embed a final named type, // but the field name does not match its name, // then it must have been done via an alias. // We dig out the alias's TypeName via locateForeignAlias. if named.Obj().Name() != node.Name { As the reader can deduce, this heuristic would incorrectly assume that the snippet above does not embed a type alias, when in fact it does. When obfuscating the field T.SameName, which uses a type alias, we would correctly obfuscate the name "SameName", but we would incorrectly obfuscate it with the package p2, not p1. This would then result in build errors. To fix this problem for good, we need to get rid of the heuristic. Instead, we now mimic what was done for KnownCannotObfuscate, but for embedded fields which use type aliases. KnownEmbeddedAliasFields is now filled for each package and stored in the cache as part of cachedOutput. We can then detect the "embedded alias" case reliably, even when the field is declared in an imported package. On the plus side, we get to remove locateForeignAlias. We also add a couple of TODOs to record further improvements. Finally, add a test. Fixes #466.
2 years ago
// Alternatively, if we don't have an alias, we still want to
// use the embedded type, not the field.
properly record when type aliases are embedded as fields There are two scenarios when it comes to embedding fields. The first is easy, and we always handled it well: type Named struct { Foo int } type T struct { Named } In this scenario, T ends up with an embedded field named "Named", and a promoted field named "Foo". Then there's the form with a type alias: type Named struct { Foo int } type Alias = Named type T struct { Alias } This case is different: T ends up with an embedded field named "Alias", and a promoted field named "Foo". Note how the field gets its name from the referenced type, even if said type is just an alias to another type. This poses two problems. First, we must obfuscate the field T.Alias as the name "Alias", and not as the name "Named" that the alias points to. Second, we must be careful of cases where Named and Alias are declared in different packages, as they will obfuscate the same name differently. Both of those problems compounded in the reported issue. The actual reason is that quic-go has a type alias in the form of: type ConnectionState = qtls.ConnectionState In other words, the entire problem boils down to a type alias which points to a named type in a different package, where both types share the same name. For example: package parent import "parent/p1" type T struct { p1.SameName } [...] package p1 import "parent/p2" type SameName = p2.SameName [...] package p2 type SameName struct { Foo int } This broke garble because we had a heuristic to detect when an embedded field was a type alias: // Instead, detect such a "foreign alias embed". // If we embed a final named type, // but the field name does not match its name, // then it must have been done via an alias. // We dig out the alias's TypeName via locateForeignAlias. if named.Obj().Name() != node.Name { As the reader can deduce, this heuristic would incorrectly assume that the snippet above does not embed a type alias, when in fact it does. When obfuscating the field T.SameName, which uses a type alias, we would correctly obfuscate the name "SameName", but we would incorrectly obfuscate it with the package p2, not p1. This would then result in build errors. To fix this problem for good, we need to get rid of the heuristic. Instead, we now mimic what was done for KnownCannotObfuscate, but for embedded fields which use type aliases. KnownEmbeddedAliasFields is now filled for each package and stored in the cache as part of cachedOutput. We can then detect the "embedded alias" case reliably, even when the field is declared in an imported package. On the plus side, we get to remove locateForeignAlias. We also add a couple of TODOs to record further improvements. Finally, add a test. Fixes #466.
2 years ago
vrStr := recordedObjectString(vr)
aliasTypeName, ok := cachedOutput.KnownEmbeddedAliasFields[vrStr]
if ok {
pkg2 := tf.pkg
if path := aliasTypeName.PkgPath; pkg2.Path() != path {
// If the package is a dependency, import it.
// We can't grab the package via tf.pkg.Imports,
// because some of the packages under there are incomplete.
// ImportFrom will cache complete imports, anyway.
properly record when type aliases are embedded as fields There are two scenarios when it comes to embedding fields. The first is easy, and we always handled it well: type Named struct { Foo int } type T struct { Named } In this scenario, T ends up with an embedded field named "Named", and a promoted field named "Foo". Then there's the form with a type alias: type Named struct { Foo int } type Alias = Named type T struct { Alias } This case is different: T ends up with an embedded field named "Alias", and a promoted field named "Foo". Note how the field gets its name from the referenced type, even if said type is just an alias to another type. This poses two problems. First, we must obfuscate the field T.Alias as the name "Alias", and not as the name "Named" that the alias points to. Second, we must be careful of cases where Named and Alias are declared in different packages, as they will obfuscate the same name differently. Both of those problems compounded in the reported issue. The actual reason is that quic-go has a type alias in the form of: type ConnectionState = qtls.ConnectionState In other words, the entire problem boils down to a type alias which points to a named type in a different package, where both types share the same name. For example: package parent import "parent/p1" type T struct { p1.SameName } [...] package p1 import "parent/p2" type SameName = p2.SameName [...] package p2 type SameName struct { Foo int } This broke garble because we had a heuristic to detect when an embedded field was a type alias: // Instead, detect such a "foreign alias embed". // If we embed a final named type, // but the field name does not match its name, // then it must have been done via an alias. // We dig out the alias's TypeName via locateForeignAlias. if named.Obj().Name() != node.Name { As the reader can deduce, this heuristic would incorrectly assume that the snippet above does not embed a type alias, when in fact it does. When obfuscating the field T.SameName, which uses a type alias, we would correctly obfuscate the name "SameName", but we would incorrectly obfuscate it with the package p2, not p1. This would then result in build errors. To fix this problem for good, we need to get rid of the heuristic. Instead, we now mimic what was done for KnownCannotObfuscate, but for embedded fields which use type aliases. KnownEmbeddedAliasFields is now filled for each package and stored in the cache as part of cachedOutput. We can then detect the "embedded alias" case reliably, even when the field is declared in an imported package. On the plus side, we get to remove locateForeignAlias. We also add a couple of TODOs to record further improvements. Finally, add a test. Fixes #466.
2 years ago
var err error
pkg2, err = origImporter.ImportFrom(path, parentWorkDir, 0)
properly record when type aliases are embedded as fields There are two scenarios when it comes to embedding fields. The first is easy, and we always handled it well: type Named struct { Foo int } type T struct { Named } In this scenario, T ends up with an embedded field named "Named", and a promoted field named "Foo". Then there's the form with a type alias: type Named struct { Foo int } type Alias = Named type T struct { Alias } This case is different: T ends up with an embedded field named "Alias", and a promoted field named "Foo". Note how the field gets its name from the referenced type, even if said type is just an alias to another type. This poses two problems. First, we must obfuscate the field T.Alias as the name "Alias", and not as the name "Named" that the alias points to. Second, we must be careful of cases where Named and Alias are declared in different packages, as they will obfuscate the same name differently. Both of those problems compounded in the reported issue. The actual reason is that quic-go has a type alias in the form of: type ConnectionState = qtls.ConnectionState In other words, the entire problem boils down to a type alias which points to a named type in a different package, where both types share the same name. For example: package parent import "parent/p1" type T struct { p1.SameName } [...] package p1 import "parent/p2" type SameName = p2.SameName [...] package p2 type SameName struct { Foo int } This broke garble because we had a heuristic to detect when an embedded field was a type alias: // Instead, detect such a "foreign alias embed". // If we embed a final named type, // but the field name does not match its name, // then it must have been done via an alias. // We dig out the alias's TypeName via locateForeignAlias. if named.Obj().Name() != node.Name { As the reader can deduce, this heuristic would incorrectly assume that the snippet above does not embed a type alias, when in fact it does. When obfuscating the field T.SameName, which uses a type alias, we would correctly obfuscate the name "SameName", but we would incorrectly obfuscate it with the package p2, not p1. This would then result in build errors. To fix this problem for good, we need to get rid of the heuristic. Instead, we now mimic what was done for KnownCannotObfuscate, but for embedded fields which use type aliases. KnownEmbeddedAliasFields is now filled for each package and stored in the cache as part of cachedOutput. We can then detect the "embedded alias" case reliably, even when the field is declared in an imported package. On the plus side, we get to remove locateForeignAlias. We also add a couple of TODOs to record further improvements. Finally, add a test. Fixes #466.
2 years ago
if err != nil {
panic(err)
}
}
tname, ok := pkg2.Scope().Lookup(aliasTypeName.Name).(*types.TypeName)
if !ok || !tname.IsAlias() {
if !ok {
panic(fmt.Sprintf("KnownEmbeddedAliasFields pointed %q to a missing type %q", vrStr, aliasTypeName))
}
panic(fmt.Sprintf("KnownEmbeddedAliasFields pointed %q to a non-alias type %q", vrStr, aliasTypeName))
}
obj = tname
} else {
named := namedType(obj.Type())
if named == nil {
return true // unnamed type (probably a basic type, e.g. int)
}
properly record when type aliases are embedded as fields There are two scenarios when it comes to embedding fields. The first is easy, and we always handled it well: type Named struct { Foo int } type T struct { Named } In this scenario, T ends up with an embedded field named "Named", and a promoted field named "Foo". Then there's the form with a type alias: type Named struct { Foo int } type Alias = Named type T struct { Alias } This case is different: T ends up with an embedded field named "Alias", and a promoted field named "Foo". Note how the field gets its name from the referenced type, even if said type is just an alias to another type. This poses two problems. First, we must obfuscate the field T.Alias as the name "Alias", and not as the name "Named" that the alias points to. Second, we must be careful of cases where Named and Alias are declared in different packages, as they will obfuscate the same name differently. Both of those problems compounded in the reported issue. The actual reason is that quic-go has a type alias in the form of: type ConnectionState = qtls.ConnectionState In other words, the entire problem boils down to a type alias which points to a named type in a different package, where both types share the same name. For example: package parent import "parent/p1" type T struct { p1.SameName } [...] package p1 import "parent/p2" type SameName = p2.SameName [...] package p2 type SameName struct { Foo int } This broke garble because we had a heuristic to detect when an embedded field was a type alias: // Instead, detect such a "foreign alias embed". // If we embed a final named type, // but the field name does not match its name, // then it must have been done via an alias. // We dig out the alias's TypeName via locateForeignAlias. if named.Obj().Name() != node.Name { As the reader can deduce, this heuristic would incorrectly assume that the snippet above does not embed a type alias, when in fact it does. When obfuscating the field T.SameName, which uses a type alias, we would correctly obfuscate the name "SameName", but we would incorrectly obfuscate it with the package p2, not p1. This would then result in build errors. To fix this problem for good, we need to get rid of the heuristic. Instead, we now mimic what was done for KnownCannotObfuscate, but for embedded fields which use type aliases. KnownEmbeddedAliasFields is now filled for each package and stored in the cache as part of cachedOutput. We can then detect the "embedded alias" case reliably, even when the field is declared in an imported package. On the plus side, we get to remove locateForeignAlias. We also add a couple of TODOs to record further improvements. Finally, add a test. Fixes #466.
2 years ago
obj = named.Obj()
}
pkg = obj.Pkg()
}
if pkg == nil {
return true // universe scope
}
if pkg.Path() == "embed" {
// The Go compiler needs to detect types such as embed.FS.
// That will fail if we change the import path or type name.
// Leave it as is.
// Luckily, the embed package just declares the FS type.
return true
}
// The package that declared this object did not obfuscate it.
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
if recordedAsNotObfuscated(obj) {
return true
}
// TODO(mvdan): investigate obfuscating these too.
filename := fset.Position(obj.Pos()).Filename
if strings.HasPrefix(filename, "_cgo_") || strings.Contains(filename, ".cgo1.") {
return true
}
path := pkg.Path()
refactor "current package" with TOOLEXEC_IMPORTPATH (#266) Now that we've dropped support for Go 1.15.x, we can finally rely on this environment variable for toolexec calls, present in Go 1.16. Before, we had hacky ways of trying to figure out the current package's import path, mostly from the -p flag. The biggest rough edge there was that, for main packages, that was simply the package name, and not its full import path. To work around that, we had a restriction on a single main package, so we could work around that issue. That restriction is now gone. The new code is simpler, especially because we can set curPkg in a single place for all toolexec transform funcs. Since we can always rely on curPkg not being nil now, we can also start reusing listedPackage.Private and avoid the majority of repeated calls to isPrivate. The function is cheap, but still not free. isPrivate itself can also get simpler. We no longer have to worry about the "main" edge case. Plus, the sanity check for invalid package paths is now unnecessary; we only got malformed paths from goobj2, and we now require exact matches with the ImportPath field from "go list -json". Another effect of clearing up the "main" edge case is that -debugdir now uses the right directory for main packages. We also start using consistent debugdir paths in the tests, for the sake of being easier to read and maintain. Finally, note that commandReverse did not need the extra call to "go list -toolexec", as the "shared" call stored in the cache is enough. We still call toolexecCmd to get said cache, which should probably be simplified in a future PR. While at it, replace the use of the "-std" compiler flag with the Standard field from "go list -json".
3 years ago
lpkg, err := listPackage(path)
if err != nil {
panic(err) // shouldn't happen
}
if !lpkg.ToObfuscate {
return true // we're not obfuscating this package
}
hashToUse := lpkg.GarbleActionID
debugName := "variable"
// debugf("%s: %#v %T", fset.Position(node.Pos()), node, obj)
switch obj := obj.(type) {
case *types.Var:
fix a number of issues involving types from indirect imports obfuscatedTypesPackage is used to figure out if a name in a dependency package was obfuscated or not. For example, if that package used reflection on a named type, it wasn't obfuscated, so we must have the same information to not obfuscate the same name downstream. obfuscatedTypesPackage could return nil if the package was indirectly imported, though. This can happen if a direct import has a function that returns an indirect type, or if a direct import exposes a name that's a type alias to an indirect type. We sort of dealt with this in two pieces of code by checking for obfPkg!=nil, but a third one did not have this check and caused a panic in the added test case: --- FAIL: TestScripts/reflect (0.81s) testscript.go:397: > env GOPRIVATE=test/main > garble build [stderr] # test/main panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8a5e39] More importantly though, the nil check only avoids panics. It doesn't fix the root cause of the problem: that importcfg does not contain indirectly imported packages. The added test case would still fail, as we would obfuscate a type in the main package, but not in the indirectly imported package where the type is defined. To fix this, resurrect a bit of code from earlier garble versions, which uses "go list -toolexec=garble" to fetch a package's export file. This lets us fill the indirect import gaps in importcfg, working around the problem entirely. This solution is still not particularly great, so we add a TODO about possibly rethinking this in the future. It does add some overhead and complexity, though thankfully indirect imports should be uncommon. This fixes a few panics while building the protobuf module.
3 years ago
if !obj.IsField() {
// Identifiers denoting variables are always obfuscated.
fix a number of issues involving types from indirect imports obfuscatedTypesPackage is used to figure out if a name in a dependency package was obfuscated or not. For example, if that package used reflection on a named type, it wasn't obfuscated, so we must have the same information to not obfuscate the same name downstream. obfuscatedTypesPackage could return nil if the package was indirectly imported, though. This can happen if a direct import has a function that returns an indirect type, or if a direct import exposes a name that's a type alias to an indirect type. We sort of dealt with this in two pieces of code by checking for obfPkg!=nil, but a third one did not have this check and caused a panic in the added test case: --- FAIL: TestScripts/reflect (0.81s) testscript.go:397: > env GOPRIVATE=test/main > garble build [stderr] # test/main panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8a5e39] More importantly though, the nil check only avoids panics. It doesn't fix the root cause of the problem: that importcfg does not contain indirectly imported packages. The added test case would still fail, as we would obfuscate a type in the main package, but not in the indirectly imported package where the type is defined. To fix this, resurrect a bit of code from earlier garble versions, which uses "go list -toolexec=garble" to fetch a package's export file. This lets us fill the indirect import gaps in importcfg, working around the problem entirely. This solution is still not particularly great, so we add a TODO about possibly rethinking this in the future. It does add some overhead and complexity, though thankfully indirect imports should be uncommon. This fixes a few panics while building the protobuf module.
3 years ago
break
}
debugName = "field"
fix a number of issues involving types from indirect imports obfuscatedTypesPackage is used to figure out if a name in a dependency package was obfuscated or not. For example, if that package used reflection on a named type, it wasn't obfuscated, so we must have the same information to not obfuscate the same name downstream. obfuscatedTypesPackage could return nil if the package was indirectly imported, though. This can happen if a direct import has a function that returns an indirect type, or if a direct import exposes a name that's a type alias to an indirect type. We sort of dealt with this in two pieces of code by checking for obfPkg!=nil, but a third one did not have this check and caused a panic in the added test case: --- FAIL: TestScripts/reflect (0.81s) testscript.go:397: > env GOPRIVATE=test/main > garble build [stderr] # test/main panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8a5e39] More importantly though, the nil check only avoids panics. It doesn't fix the root cause of the problem: that importcfg does not contain indirectly imported packages. The added test case would still fail, as we would obfuscate a type in the main package, but not in the indirectly imported package where the type is defined. To fix this, resurrect a bit of code from earlier garble versions, which uses "go list -toolexec=garble" to fetch a package's export file. This lets us fill the indirect import gaps in importcfg, working around the problem entirely. This solution is still not particularly great, so we add a TODO about possibly rethinking this in the future. It does add some overhead and complexity, though thankfully indirect imports should be uncommon. This fixes a few panics while building the protobuf module.
3 years ago
// From this point on, we deal with struct fields.
// Fields don't get hashed with the package's action ID.
// They get hashed with the type of their parent struct.
// This is because one struct can be converted to another,
// as long as the underlying types are identical,
// even if the structs are defined in different packages.
//
// TODO: Consider only doing this for structs where all
// fields are exported. We only need this special case
// for cross-package conversions, which can't work if
// any field is unexported. If that is done, add a test
// that ensures unexported fields from different
// packages result in different obfuscated names.
fix a number of issues involving types from indirect imports obfuscatedTypesPackage is used to figure out if a name in a dependency package was obfuscated or not. For example, if that package used reflection on a named type, it wasn't obfuscated, so we must have the same information to not obfuscate the same name downstream. obfuscatedTypesPackage could return nil if the package was indirectly imported, though. This can happen if a direct import has a function that returns an indirect type, or if a direct import exposes a name that's a type alias to an indirect type. We sort of dealt with this in two pieces of code by checking for obfPkg!=nil, but a third one did not have this check and caused a panic in the added test case: --- FAIL: TestScripts/reflect (0.81s) testscript.go:397: > env GOPRIVATE=test/main > garble build [stderr] # test/main panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8a5e39] More importantly though, the nil check only avoids panics. It doesn't fix the root cause of the problem: that importcfg does not contain indirectly imported packages. The added test case would still fail, as we would obfuscate a type in the main package, but not in the indirectly imported package where the type is defined. To fix this, resurrect a bit of code from earlier garble versions, which uses "go list -toolexec=garble" to fetch a package's export file. This lets us fill the indirect import gaps in importcfg, working around the problem entirely. This solution is still not particularly great, so we add a TODO about possibly rethinking this in the future. It does add some overhead and complexity, though thankfully indirect imports should be uncommon. This fixes a few panics while building the protobuf module.
3 years ago
strct := tf.fieldToStruct[obj]
if strct == nil {
panic("could not find for " + name)
}
node.Name = hashWithStruct(strct, name)
debugf("%s %q hashed with struct fields to %q", debugName, name, node.Name)
return true
case *types.TypeName:
debugName = "type"
case *types.Func:
sign := obj.Type().(*types.Signature)
if sign.Recv() == nil {
debugName = "func"
} else {
debugName = "method"
}
if obj.Exported() && sign.Recv() != nil {
return true // might implement an interface
}
switch name {
case "main", "init", "TestMain":
return true // don't break them
}
if strings.HasPrefix(name, "Test") && isTestSignature(sign) {
return true // don't break tests
}
default:
return true // we only want to rename the above
}
node.Name = hashWithPackage(lpkg, name)
// TODO: probably move the debugf lines inside the hash funcs
debugf("%s %q hashed with %x… to %q", debugName, name, hashToUse[:4], node.Name)
return true
}
post := func(cursor *astutil.Cursor) bool {
imp, ok := cursor.Node().(*ast.ImportSpec)
if !ok {
return true
}
path, err := strconv.Unquote(imp.Path.Value)
if err != nil {
panic(err) // should never happen
}
// We're importing an obfuscated package.
// Replace the import path with its obfuscated version.
// If the import was unnamed, give it the name of the
// original package name, to keep references working.
lpkg, err := listPackage(path)
if err != nil {
panic(err) // should never happen
}
if !lpkg.ToObfuscate {
return true
}
newPath := lpkg.obfuscatedImportPath()
imp.Path.Value = strconv.Quote(newPath)
if imp.Name == nil {
imp.Name = &ast.Ident{Name: lpkg.Name}
}
return true
}
return astutil.Apply(file, pre, post).(*ast.File)
}
// recursivelyRecordAsNotObfuscated calls recordAsNotObfuscated on any named
// types and fields under typ.
//
// Only the names declared in the current package are recorded. This is to ensure
// that reflection detection only happens within the package declaring a type.
// Detecting it in downstream packages could result in inconsistencies.
func (tf *transformer) recursivelyRecordAsNotObfuscated(t types.Type) {
switch t := t.(type) {
case *types.Named:
obj := t.Obj()
if obj.Pkg() == nil || obj.Pkg() != tf.pkg {
return // not from the specified package
}
if recordedAsNotObfuscated(obj) {
return // prevent endless recursion
}
recordAsNotObfuscated(obj)
// Record the underlying type, too.
tf.recursivelyRecordAsNotObfuscated(t.Underlying())
case *types.Struct:
for i := 0; i < t.NumFields(); i++ {
field := t.Field(i)
// This check is similar to the one in *types.Named.
// It's necessary for unnamed struct types,
// as they aren't named but still have named fields.
if field.Pkg() == nil || field.Pkg() != tf.pkg {
return // not from the specified package
}
// Record the field itself, too.
recordAsNotObfuscated(field)
tf.recursivelyRecordAsNotObfuscated(field.Type())
}
case interface{ Elem() types.Type }:
// Get past pointers, slices, etc.
tf.recursivelyRecordAsNotObfuscated(t.Elem())
}
}
// named tries to obtain the *types.Named behind a type, if there is one.
// This is useful to obtain "testing.T" from "*testing.T", or to obtain the type
// declaration object from an embedded field.
func namedType(t types.Type) *types.Named {
switch t := t.(type) {
case *types.Named:
return t
case interface{ Elem() types.Type }:
return namedType(t.Elem())
default:
return nil
}
}
// isTestSignature returns true if the signature matches "func _(*testing.T)".
func isTestSignature(sign *types.Signature) bool {
if sign.Recv() != nil {
return false // test funcs don't have receivers
}
params := sign.Params()
if params.Len() != 1 {
return false // too many parameters for a test func
}
named := namedType(params.At(0).Type())
if named == nil {
return false // the only parameter isn't named, like "string"
}
obj := named.Obj()
return obj != nil && obj.Pkg().Path() == "testing" && obj.Name() == "T"
}
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
func transformLink(args []string) ([]string, error) {
initial support for build caching (#142) As per the discussion in https://github.com/golang/go/issues/41145, it turns out that we don't need special support for build caching in -toolexec. We can simply modify the behavior of "[...]/compile -V=full" and "[...]/link -V=full" so that they include garble's own version and options in the printed build ID. The part of the build ID that matters is the last, since it's the "content ID" which is used to work out whether there is a need to redo the action (build) or not. Since cmd/go parses the last word in the output as "buildID=...", we simply add "+garble buildID=_/_/_/${hash}". The slashes let us imitate a full binary build ID, but we assume that the other components such as the action ID are not necessary, since the only reader here is cmd/go and it only consumes the content ID. The reported content ID includes the tool's original content ID, garble's own content ID from the built binary, and the garble options which modify how we obfuscate code. If any of the three changes, we should use a different build cache key. GOPRIVATE also affects caching, since a different GOPRIVATE value means that we might have to garble a different set of packages. Include tests, which mainly check that 'garble build -v' prints package lines when we expect to always need to rebuild packages, and that it prints nothing when we should be reusing the build cache even when the built binary is missing. After this change, 'go test' on Go 1.15.2 stabilizes at about 8s on my machine, whereas it used to be at around 25s before.
4 years ago
// We can't split by the ".a" extension, because cached object files
// lack any extension.
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
flags, args := splitFlagsFromArgs(args)
newImportCfg, err := processImportCfg(flags)
if err != nil {
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
return nil, err
}
// TODO: unify this logic with the -X handling when using -literals.
// We should be able to handle both cases via the syntax tree.
//
// Make sure -X works with obfuscated identifiers.
// To cover both obfuscated and non-obfuscated names,
// duplicate each flag with a obfuscated version.
flagValueIter(flags, "-X", func(val string) {
// val is in the form of "pkg.name=str"
i := strings.IndexByte(val, '=')
if i <= 0 {
return
}
name := val[:i]
str := val[i+1:]
j := strings.LastIndexByte(name, '.')
if j <= 0 {
return
}
pkg := name[:j]
name = name[j+1:]
refactor "current package" with TOOLEXEC_IMPORTPATH (#266) Now that we've dropped support for Go 1.15.x, we can finally rely on this environment variable for toolexec calls, present in Go 1.16. Before, we had hacky ways of trying to figure out the current package's import path, mostly from the -p flag. The biggest rough edge there was that, for main packages, that was simply the package name, and not its full import path. To work around that, we had a restriction on a single main package, so we could work around that issue. That restriction is now gone. The new code is simpler, especially because we can set curPkg in a single place for all toolexec transform funcs. Since we can always rely on curPkg not being nil now, we can also start reusing listedPackage.Private and avoid the majority of repeated calls to isPrivate. The function is cheap, but still not free. isPrivate itself can also get simpler. We no longer have to worry about the "main" edge case. Plus, the sanity check for invalid package paths is now unnecessary; we only got malformed paths from goobj2, and we now require exact matches with the ImportPath field from "go list -json". Another effect of clearing up the "main" edge case is that -debugdir now uses the right directory for main packages. We also start using consistent debugdir paths in the tests, for the sake of being easier to read and maintain. Finally, note that commandReverse did not need the extra call to "go list -toolexec", as the "shared" call stored in the cache is enough. We still call toolexecCmd to get said cache, which should probably be simplified in a future PR. While at it, replace the use of the "-std" compiler flag with the Standard field from "go list -json".
3 years ago
// If the package path is "main", it's the current top-level
// package we are linking.
// Otherwise, find it in the cache.
lpkg := curPkg
if pkg != "main" {
lpkg = cache.ListedPackages[pkg]
}
if lpkg == nil {
// We couldn't find the package.
// Perhaps a typo, perhaps not part of the build.
// cmd/link ignores those, so we should too.
return
}
refactor "current package" with TOOLEXEC_IMPORTPATH (#266) Now that we've dropped support for Go 1.15.x, we can finally rely on this environment variable for toolexec calls, present in Go 1.16. Before, we had hacky ways of trying to figure out the current package's import path, mostly from the -p flag. The biggest rough edge there was that, for main packages, that was simply the package name, and not its full import path. To work around that, we had a restriction on a single main package, so we could work around that issue. That restriction is now gone. The new code is simpler, especially because we can set curPkg in a single place for all toolexec transform funcs. Since we can always rely on curPkg not being nil now, we can also start reusing listedPackage.Private and avoid the majority of repeated calls to isPrivate. The function is cheap, but still not free. isPrivate itself can also get simpler. We no longer have to worry about the "main" edge case. Plus, the sanity check for invalid package paths is now unnecessary; we only got malformed paths from goobj2, and we now require exact matches with the ImportPath field from "go list -json". Another effect of clearing up the "main" edge case is that -debugdir now uses the right directory for main packages. We also start using consistent debugdir paths in the tests, for the sake of being easier to read and maintain. Finally, note that commandReverse did not need the extra call to "go list -toolexec", as the "shared" call stored in the cache is enough. We still call toolexecCmd to get said cache, which should probably be simplified in a future PR. While at it, replace the use of the "-std" compiler flag with the Standard field from "go list -json".
3 years ago
// As before, the main package must remain as "main".
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
newPkg := pkg
refactor "current package" with TOOLEXEC_IMPORTPATH (#266) Now that we've dropped support for Go 1.15.x, we can finally rely on this environment variable for toolexec calls, present in Go 1.16. Before, we had hacky ways of trying to figure out the current package's import path, mostly from the -p flag. The biggest rough edge there was that, for main packages, that was simply the package name, and not its full import path. To work around that, we had a restriction on a single main package, so we could work around that issue. That restriction is now gone. The new code is simpler, especially because we can set curPkg in a single place for all toolexec transform funcs. Since we can always rely on curPkg not being nil now, we can also start reusing listedPackage.Private and avoid the majority of repeated calls to isPrivate. The function is cheap, but still not free. isPrivate itself can also get simpler. We no longer have to worry about the "main" edge case. Plus, the sanity check for invalid package paths is now unnecessary; we only got malformed paths from goobj2, and we now require exact matches with the ImportPath field from "go list -json". Another effect of clearing up the "main" edge case is that -debugdir now uses the right directory for main packages. We also start using consistent debugdir paths in the tests, for the sake of being easier to read and maintain. Finally, note that commandReverse did not need the extra call to "go list -toolexec", as the "shared" call stored in the cache is enough. We still call toolexecCmd to get said cache, which should probably be simplified in a future PR. While at it, replace the use of the "-std" compiler flag with the Standard field from "go list -json".
3 years ago
if pkg != "main" {
newPkg = lpkg.obfuscatedImportPath()
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
}
newName := hashWithPackage(lpkg, name)
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
flags = append(flags, fmt.Sprintf("-X=%s.%s=%s", newPkg, newName, str))
})
update support for Go 1.17 in time for beta1 Back in early April we added initial support for Go 1.17, working on a commit from master at that time. For that to work, we just needed to add a couple of packages to runtimeRelated and tweak printFile a bit to not break the new "//go:build" directives. A significant amount of changes have landed since, though, and the tests broke in multiple ways. Most notably, the new register ABI is enabled by default for GOOS=amd64. That affected garble indirectly in two ways: there's a new internal package to add to runtimeRelated, and we must make reverse.txt more clever in making its output constant across ABIs. Another noticeable change is that Go 1.17 changes how its own version is injected into the runtime package. It used to be via a constant in runtime/internal/sys, such as: const TheVersion = `devel ...` Since we couldn't override such constants via the linker's -X flag, we had to directly alter the declaration while compiling. Thankfully, Go 1.17 simply uses a "var buildVersion string" in the runtime package, and its value is injected by the linker. This means we can now override it with the linker's -X flag. We make the code to alter TheVersion for Go 1.16 a bit more clever, to not break the package when building with Go 1.17. Finally, our hack to work around ambiguous TOOLEXEC_IMPORTPATH values now only kicks in for non-test packages, since Go 1.17 includes our upstream fix. Otherwise, some tests would end up with the ".test" variant suffix added a second time: test/bar [test/bar.test] [test/bar [test/bar.test].test] All the code to keep compatibility with Go 1.16.x remains in place. We're still leaving TODOs to remind ourselves to remove it or simplify it once we remove support for 1.16.x. The 1.17 development freeze has already been in place for a month, and beta1 is due to come this week, so it's unlikely that Go will change in any considerable way at this point. Hence, we can say that support for 1.17 is done. Fixes #347.
3 years ago
// Starting in Go 1.17, Go's version is implicitly injected by the linker.
// It's the same method as -X, so we can override it with an extra flag.
flags = append(flags, "-X=runtime.buildVersion=unknown")
// Ensure we strip the -buildid flag, to not leak any build IDs for the
// link operation or the main package's compilation.
flags = flagSetValue(flags, "-buildid", "")
// Strip debug information and symbol tables.
flags = append(flags, "-w", "-s")
reimplement import path obfuscation without goobj2 (#242) We used to rely on a parallel implementation of an object file parser and writer to be able to obfuscate import paths. After compiling each package, we would parse the object file, replace the import paths, and write the updated object file in-place. That worked well, in most cases. Unfortunately, it had some flaws: * Complexity. Even when most of the code is maintained in a separate module, the import_obfuscation.go file was still close to a thousand lines of code. * Go compatibility. The object file format changes between Go releases, so we were supporting Go 1.15, but not 1.16. Fixing the object file package to work with 1.16 would probably break 1.15 support. * Bugs. For example, we recently had to add a workaround for #224, since import paths containing dots after the domain would end up escaped. Another example is #190, which seems to be caused by the object file parser or writer corrupting the compiled code and causing segfaults in some rare edge cases. Instead, let's drop that method entirely, and force the compiler and linker to do the work for us. The steps necessary when compiling a package to obfuscate are: 1) Replace its "package foo" lines with the obfuscated package path. No need to separate the package path and name, since the obfuscated path does not contain slashes. 2) Replace the "-p pkg/foo" flag with the obfuscated path. 3) Replace the "import" spec lines with the obfuscated package paths, for those dependencies which were obfuscated. 4) Replace the "-importcfg [...]" file with a version that uses the obfuscated paths instead. The linker also needs that last step, since it also uses an importcfg file to find object files. There are three noteworthy drawbacks to this new method: 1) Since we no longer write object files, we can't use them to store data to be cached. As such, the -debugdir flag goes back to using the "-a" build flag to always rebuild all packages. On the plus side, that caching didn't work very well; see #176. 2) The package name "main" remains in all declarations under it, not just "func main", since we can only rename entire packages. This seems fine, as it gives little information to the end user. 3) The -tiny mode no longer sets all lines to 0, since it did that by modifying object files. As a temporary measure, we instead set all top-level declarations to be on line 1. A TODO is added to hopefully improve this again in the near future. The upside is that we get rid of all the issues mentioned before. Plus, garble now nearly works with Go 1.16, with the exception of two very minor bugs that look fixable. A follow-up PR will take care of that and start testing on 1.16. Fixes #176. Fixes #190.
3 years ago
flags = flagSetValue(flags, "-importcfg", newImportCfg)
return append(flags, args...), nil
}
func splitFlagsFromArgs(all []string) (flags, args []string) {
for i := 0; i < len(all); i++ {
arg := all[i]
if !strings.HasPrefix(arg, "-") {
return all[:i:i], all[i:]
}
if booleanFlags[arg] || strings.Contains(arg, "=") {
// Either "-bool" or "-name=value".
continue
}
// "-name value", so the next arg is part of this flag.
i++
}
return all, nil
}
func alterTrimpath(flags []string) []string {
avoid reproducibility issues with full rebuilds We were using temporary filenames for modified Go and assembly files. For example, an obfuscated "encoding/json/encode.go" would end up as: /tmp/garble-shared123/encode.go.456.go where "123" and "456" are random numbers, usually longer. This was usually fine for two reasons: 1) We would add "/tmp/garble-shared123/" to -trimpath, so the temporary directory and its random number would be invisible. 2) We would add "//line" directives to the source files, replacing the filename with obfuscated versions excluding any random number. Unfortunately, this broke in multiple ways. Most notably, assembly files do not have any line directives, and it's not clear that there's any support for them. So the random number in their basename could end up in the binary, breaking reproducibility. Another issue is that the -trimpath addition described above was only done for cmd/compile, not cmd/asm, so assembly filenames included the randomized temporary directory. To fix the issues above, the same "encoding/json/encode.go" would now end up as: /tmp/garble-shared123/encoding/json/encode.go Such a path is still unique even though the "456" random number is gone, as import paths are unique within a single build. This fixes issues with the base name of each file, so we no longer rely on line directives as the only way to remove the second original random number. We still rely on -trimpath to get rid of the temporary directory in filenames. To fix its problem with assembly files, also amend the -trimpath flag when running the assembler tool. Finally, add a test that reproducible builds still work when a full rebuild is done. We choose goprivate.txt for such a test as its stdimporter package imports a number of std packages, including uses of assembly and cgo. For the time being, we don't use such a "full rebuild" reproducibility test in other test scripts, as this step is expensive, rebuilding many packages from scratch. This issue went unnoticed for over a year because such random numbers "123" and "456" were created when a package was obfuscated, and that only happened once per package version as long as the build cache was kept intact. When clearing the build cache, or forcing a rebuild with -a, one gets new random numbers, and thus a different binary resulting from the same build input. That's not something that most users would do regularly, and our tests did not cover that edge case either, until now. Fixes #328.
3 years ago
// If the value of -trimpath doesn't contain the separator ';', the 'go
// build' command is most likely not using '-trimpath'.
trimpath := flagValue(flags, "-trimpath")
// Add our temporary dir to the beginning of -trimpath, so that we don't
// leak temporary dirs. Needs to be at the beginning, since there may be
// shorter prefixes later in the list, such as $PWD if TMPDIR=$PWD/tmp.
return flagSetValue(flags, "-trimpath", sharedTempDir+"=>;"+trimpath)
avoid reproducibility issues with full rebuilds We were using temporary filenames for modified Go and assembly files. For example, an obfuscated "encoding/json/encode.go" would end up as: /tmp/garble-shared123/encode.go.456.go where "123" and "456" are random numbers, usually longer. This was usually fine for two reasons: 1) We would add "/tmp/garble-shared123/" to -trimpath, so the temporary directory and its random number would be invisible. 2) We would add "//line" directives to the source files, replacing the filename with obfuscated versions excluding any random number. Unfortunately, this broke in multiple ways. Most notably, assembly files do not have any line directives, and it's not clear that there's any support for them. So the random number in their basename could end up in the binary, breaking reproducibility. Another issue is that the -trimpath addition described above was only done for cmd/compile, not cmd/asm, so assembly filenames included the randomized temporary directory. To fix the issues above, the same "encoding/json/encode.go" would now end up as: /tmp/garble-shared123/encoding/json/encode.go Such a path is still unique even though the "456" random number is gone, as import paths are unique within a single build. This fixes issues with the base name of each file, so we no longer rely on line directives as the only way to remove the second original random number. We still rely on -trimpath to get rid of the temporary directory in filenames. To fix its problem with assembly files, also amend the -trimpath flag when running the assembler tool. Finally, add a test that reproducible builds still work when a full rebuild is done. We choose goprivate.txt for such a test as its stdimporter package imports a number of std packages, including uses of assembly and cgo. For the time being, we don't use such a "full rebuild" reproducibility test in other test scripts, as this step is expensive, rebuilding many packages from scratch. This issue went unnoticed for over a year because such random numbers "123" and "456" were created when a package was obfuscated, and that only happened once per package version as long as the build cache was kept intact. When clearing the build cache, or forcing a rebuild with -a, one gets new random numbers, and thus a different binary resulting from the same build input. That's not something that most users would do regularly, and our tests did not cover that edge case either, until now. Fixes #328.
3 years ago
}
// forwardBuildFlags is obtained from 'go help build' as of Go 1.18beta1.
fail if we are unexpectedly overwriting files (#418) While investigating a bug report, I noticed that garble was writing to the same temp file twice. At best, writing to the same path on disk twice is wasteful, as the design is careful to be deterministic and use unique paths. At worst, the two writes could cause races at the filesystem level. To prevent either of those situations, we now create files with os.OpenFile and os.O_EXCL, meaning that we will error if the file already exists. That change uncovered a number of such unintended cases. First, transformAsm would write obfuscated Go files twice. This is because the Go toolchain actually runs: [...]/asm -gensymabis [...] foo.s bar.s [...]/asm [...] foo.s bar.s That is, the first run is only meant to generate symbol ABIs, which are then used by the compiler. We need to obfuscate at that first stage, because the symbol ABI descriptions need to use obfuscated names. However, having already obfuscated the assembly on the first stage, there is no need to do so again on the second stage. If we detect gensymabis is missing, we simply reuse the previous files. This first situation doesn't seem racy, but obfuscating the Go assembly files twice is certainly unnecessary. Second, saveKnownReflectAPIs wrote a gob file to the build cache. Since the build cache can be kept between builds, and since the build cache uses reproducible paths for each build, running the same "garble build" twice could overwrite those files. This could actually cause races at the filesystem level; if two concurrent builds write to the same gob file on disk, one of them could end up using a partially-written file. Note that this is the only of the three cases not using temporary files. As such, it is expected that the file may already exist. In such a case, we simply avoid overwriting it rather than failing. Third, when "garble build -a" was used, and when we needed an export file not listed in importcfg, we would end up calling roughly: go list -export -toolexec=garble -a <dependency> This meant we would re-build and re-obfuscate those packages. Which is unfortunate, because the parent process already did via: go build -toolexec=garble -a <main> The repeated dependency builds tripped the new os.O_EXCL check, as we would try to overwrite the same obfuscated Go files. Beyond being wasteful, this could again cause subtle filesystem races. To fix the problem, avoid passing flags like "-a" to nested go commands. Overall, we should likely be using safer ways to write to disk, be it via either atomic writes or locked files. However, for now, catching duplicate writes is a big step. I have left a self-assigned TODO for further improvements. CI on the pull request found a failure on test-gotip. The failure reproduces on master, so it seems to be related to gotip, and not a regression introduced by this change. For now, disable test-gotip until we can investigate.
3 years ago
var forwardBuildFlags = map[string]bool{
// These shouldn't be used in nested cmd/go calls.
"-a": false,
"-n": false,
"-x": false,
"-v": false,
// These are always set by garble.
"-trimpath": false,
"-toolexec": false,
"-buildvcs": false,
fail if we are unexpectedly overwriting files (#418) While investigating a bug report, I noticed that garble was writing to the same temp file twice. At best, writing to the same path on disk twice is wasteful, as the design is careful to be deterministic and use unique paths. At worst, the two writes could cause races at the filesystem level. To prevent either of those situations, we now create files with os.OpenFile and os.O_EXCL, meaning that we will error if the file already exists. That change uncovered a number of such unintended cases. First, transformAsm would write obfuscated Go files twice. This is because the Go toolchain actually runs: [...]/asm -gensymabis [...] foo.s bar.s [...]/asm [...] foo.s bar.s That is, the first run is only meant to generate symbol ABIs, which are then used by the compiler. We need to obfuscate at that first stage, because the symbol ABI descriptions need to use obfuscated names. However, having already obfuscated the assembly on the first stage, there is no need to do so again on the second stage. If we detect gensymabis is missing, we simply reuse the previous files. This first situation doesn't seem racy, but obfuscating the Go assembly files twice is certainly unnecessary. Second, saveKnownReflectAPIs wrote a gob file to the build cache. Since the build cache can be kept between builds, and since the build cache uses reproducible paths for each build, running the same "garble build" twice could overwrite those files. This could actually cause races at the filesystem level; if two concurrent builds write to the same gob file on disk, one of them could end up using a partially-written file. Note that this is the only of the three cases not using temporary files. As such, it is expected that the file may already exist. In such a case, we simply avoid overwriting it rather than failing. Third, when "garble build -a" was used, and when we needed an export file not listed in importcfg, we would end up calling roughly: go list -export -toolexec=garble -a <dependency> This meant we would re-build and re-obfuscate those packages. Which is unfortunate, because the parent process already did via: go build -toolexec=garble -a <main> The repeated dependency builds tripped the new os.O_EXCL check, as we would try to overwrite the same obfuscated Go files. Beyond being wasteful, this could again cause subtle filesystem races. To fix the problem, avoid passing flags like "-a" to nested go commands. Overall, we should likely be using safer ways to write to disk, be it via either atomic writes or locked files. However, for now, catching duplicate writes is a big step. I have left a self-assigned TODO for further improvements. CI on the pull request found a failure on test-gotip. The failure reproduces on master, so it seems to be related to gotip, and not a regression introduced by this change. For now, disable test-gotip until we can investigate.
3 years ago
"-p": true,
"-race": true,
"-msan": true,
"-asan": true,
"-work": true,
"-asmflags": true,
"-buildmode": true,
"-compiler": true,
"-gccgoflags": true,
"-gcflags": true,
"-installsuffix": true,
"-ldflags": true,
"-linkshared": true,
"-mod": true,
"-modcacherw": true,
"-modfile": true,
"-pkgdir": true,
"-tags": true,
"-workfile": true,
"-overlay": true,
}
// booleanFlags is obtained from 'go help build' and 'go help testflag' as of Go 1.18beta1.
var booleanFlags = map[string]bool{
// Shared build flags.
"-a": true,
"-i": true,
"-n": true,
"-v": true,
"-x": true,
"-race": true,
"-msan": true,
"-asan": true,
"-linkshared": true,
"-modcacherw": true,
"-trimpath": true,
"-buildvcs": true,
// Test flags (TODO: support its special -args flag)
"-c": true,
"-json": true,
"-cover": true,
"-failfast": true,
"-short": true,
"-benchmem": true,
}
fail if we are unexpectedly overwriting files (#418) While investigating a bug report, I noticed that garble was writing to the same temp file twice. At best, writing to the same path on disk twice is wasteful, as the design is careful to be deterministic and use unique paths. At worst, the two writes could cause races at the filesystem level. To prevent either of those situations, we now create files with os.OpenFile and os.O_EXCL, meaning that we will error if the file already exists. That change uncovered a number of such unintended cases. First, transformAsm would write obfuscated Go files twice. This is because the Go toolchain actually runs: [...]/asm -gensymabis [...] foo.s bar.s [...]/asm [...] foo.s bar.s That is, the first run is only meant to generate symbol ABIs, which are then used by the compiler. We need to obfuscate at that first stage, because the symbol ABI descriptions need to use obfuscated names. However, having already obfuscated the assembly on the first stage, there is no need to do so again on the second stage. If we detect gensymabis is missing, we simply reuse the previous files. This first situation doesn't seem racy, but obfuscating the Go assembly files twice is certainly unnecessary. Second, saveKnownReflectAPIs wrote a gob file to the build cache. Since the build cache can be kept between builds, and since the build cache uses reproducible paths for each build, running the same "garble build" twice could overwrite those files. This could actually cause races at the filesystem level; if two concurrent builds write to the same gob file on disk, one of them could end up using a partially-written file. Note that this is the only of the three cases not using temporary files. As such, it is expected that the file may already exist. In such a case, we simply avoid overwriting it rather than failing. Third, when "garble build -a" was used, and when we needed an export file not listed in importcfg, we would end up calling roughly: go list -export -toolexec=garble -a <dependency> This meant we would re-build and re-obfuscate those packages. Which is unfortunate, because the parent process already did via: go build -toolexec=garble -a <main> The repeated dependency builds tripped the new os.O_EXCL check, as we would try to overwrite the same obfuscated Go files. Beyond being wasteful, this could again cause subtle filesystem races. To fix the problem, avoid passing flags like "-a" to nested go commands. Overall, we should likely be using safer ways to write to disk, be it via either atomic writes or locked files. However, for now, catching duplicate writes is a big step. I have left a self-assigned TODO for further improvements. CI on the pull request found a failure on test-gotip. The failure reproduces on master, so it seems to be related to gotip, and not a regression introduced by this change. For now, disable test-gotip until we can investigate.
3 years ago
func filterForwardBuildFlags(flags []string) (filtered []string, firstUnknown string) {
for i := 0; i < len(flags); i++ {
arg := flags[i]
if strings.HasPrefix(arg, "--") {
arg = arg[1:] // "--name" to "-name"; keep the short form
}
name := arg
if i := strings.IndexByte(arg, '='); i > 0 {
name = arg[:i] // "-name=value" to "-name"
}
fail if we are unexpectedly overwriting files (#418) While investigating a bug report, I noticed that garble was writing to the same temp file twice. At best, writing to the same path on disk twice is wasteful, as the design is careful to be deterministic and use unique paths. At worst, the two writes could cause races at the filesystem level. To prevent either of those situations, we now create files with os.OpenFile and os.O_EXCL, meaning that we will error if the file already exists. That change uncovered a number of such unintended cases. First, transformAsm would write obfuscated Go files twice. This is because the Go toolchain actually runs: [...]/asm -gensymabis [...] foo.s bar.s [...]/asm [...] foo.s bar.s That is, the first run is only meant to generate symbol ABIs, which are then used by the compiler. We need to obfuscate at that first stage, because the symbol ABI descriptions need to use obfuscated names. However, having already obfuscated the assembly on the first stage, there is no need to do so again on the second stage. If we detect gensymabis is missing, we simply reuse the previous files. This first situation doesn't seem racy, but obfuscating the Go assembly files twice is certainly unnecessary. Second, saveKnownReflectAPIs wrote a gob file to the build cache. Since the build cache can be kept between builds, and since the build cache uses reproducible paths for each build, running the same "garble build" twice could overwrite those files. This could actually cause races at the filesystem level; if two concurrent builds write to the same gob file on disk, one of them could end up using a partially-written file. Note that this is the only of the three cases not using temporary files. As such, it is expected that the file may already exist. In such a case, we simply avoid overwriting it rather than failing. Third, when "garble build -a" was used, and when we needed an export file not listed in importcfg, we would end up calling roughly: go list -export -toolexec=garble -a <dependency> This meant we would re-build and re-obfuscate those packages. Which is unfortunate, because the parent process already did via: go build -toolexec=garble -a <main> The repeated dependency builds tripped the new os.O_EXCL check, as we would try to overwrite the same obfuscated Go files. Beyond being wasteful, this could again cause subtle filesystem races. To fix the problem, avoid passing flags like "-a" to nested go commands. Overall, we should likely be using safer ways to write to disk, be it via either atomic writes or locked files. However, for now, catching duplicate writes is a big step. I have left a self-assigned TODO for further improvements. CI on the pull request found a failure on test-gotip. The failure reproduces on master, so it seems to be related to gotip, and not a regression introduced by this change. For now, disable test-gotip until we can investigate.
3 years ago
buildFlag := forwardBuildFlags[name]
if buildFlag {
filtered = append(filtered, arg)
} else {
firstUnknown = name
}
if booleanFlags[arg] || strings.Contains(arg, "=") {
// Either "-bool" or "-name=value".
continue
}
// "-name value", so the next arg is part of this flag.
if i++; buildFlag && i < len(flags) {
filtered = append(filtered, flags[i])
}
}
return filtered, firstUnknown
}
// splitFlagsFromFiles splits args into a list of flag and file arguments. Since
// we can't rely on "--" being present, and we don't parse all flags upfront, we
// rely on finding the first argument that doesn't begin with "-" and that has
// the extension we expect for the list of paths.
//
// This function only makes sense for lower-level tool commands, such as
// "compile" or "link", since their arguments are predictable.
func splitFlagsFromFiles(all []string, ext string) (flags, paths []string) {
for i, arg := range all {
if !strings.HasPrefix(arg, "-") && strings.HasSuffix(arg, ext) {
return all[:i:i], all[i:]
}
}
return all, nil
5 years ago
}
// flagValue retrieves the value of a flag such as "-foo", from strings in the
// list of arguments like "-foo=bar" or "-foo" "bar". If the flag is repeated,
// the last value is returned.
func flagValue(flags []string, name string) string {
lastVal := ""
flagValueIter(flags, name, func(val string) {
lastVal = val
})
return lastVal
}
// flagValueIter retrieves all the values for a flag such as "-foo", like
// flagValue. The difference is that it allows handling complex flags, such as
// those whose values compose a list.
func flagValueIter(flags []string, name string, fn func(string)) {
for i, arg := range flags {
if val := strings.TrimPrefix(arg, name+"="); val != arg {
// -name=value
fn(val)
}
if arg == name { // -name ...
if i+1 < len(flags) {
// -name value
fn(flags[i+1])
}
}
}
}
func flagSetValue(flags []string, name, value string) []string {
for i, arg := range flags {
if strings.HasPrefix(arg, name+"=") {
// -name=value
flags[i] = name + "=" + value
return flags
}
if arg == name { // -name ...
if i+1 < len(flags) {
// -name value
flags[i+1] = value
return flags
}
return flags
}
}
return append(flags, name+"="+value)
}
func fetchGoEnv() error {
out, err := exec.Command("go", "env", "-json",
only list missing packages when obfuscating the runtime We were listing all of std, which certainly worked, but was quite slow at over 200 packages. In practice, we can only be missing up to 20-30 packages. It was a good change as it fixed a severe bug, but it also introduced a fairly noticeable slow-down. The numbers are clear; this change shaves off multiple seconds when obfuscating the runtime with a cold cache: name old time/op new time/op delta Build/NoCache-16 5.06s ± 1% 1.94s ± 1% -61.64% (p=0.008 n=5+5) name old bin-B new bin-B delta Build/NoCache-16 6.70M ± 0% 6.71M ± 0% +0.05% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build/NoCache-16 13.4s ± 2% 5.0s ± 2% -62.45% (p=0.008 n=5+5) name old user-time/op new user-time/op delta Build/NoCache-16 60.6s ± 1% 19.8s ± 1% -67.34% (p=0.008 n=5+5) Since we only want to call "go list" one extra time, instead of once for every package we find out we're missing, we want to know what packages we could be missing in advance. Resurrect a smarter version of the runtime-related script. Finally, remove the runtime-related.txt test script, as it has now been superseeded by the sanity checks in listPackage. That is, obfuscating the runtime package will now panic if we are missing any necessary package information. To double check that we get the runtime's linkname edge case right, make gogarble.txt use runtime/debug.WriteHeapDump, which is implemented via a direct runtime linkname. This ensures we don't lose test coverage from runtime-related.txt.
3 years ago
"GOOS", "GOPRIVATE", "GOMOD", "GOVERSION", "GOCACHE",
).CombinedOutput()
if err != nil {
// TODO: cover this in the tests.
fmt.Fprintf(os.Stderr, `Can't find the Go toolchain: %v
This is likely due to Go not being installed/setup correctly.
To install Go, see: https://go.dev/doc/install
`, err)
return errJustExit(1)
}
if err := json.Unmarshal(out, &cache.GoEnv); err != nil {
return err
}
cache.GOGARBLE = os.Getenv("GOGARBLE")
if cache.GOGARBLE != "" {
// GOGARBLE is non-empty; nothing to do.
} else if cache.GoEnv.GOPRIVATE != "" {
// GOGARBLE is empty and GOPRIVATE is non-empty.
// Set GOGARBLE to GOPRIVATE's value.
cache.GOGARBLE = cache.GoEnv.GOPRIVATE
} else {
// If GOPRIVATE isn't set and we're in a module, use its module
// path as a GOPRIVATE default. Include a _test variant too.
// TODO(mvdan): we shouldn't need the _test variant here,
// as the import path should not include it; only the package name.
if mod, err := os.ReadFile(cache.GoEnv.GOMOD); err == nil {
modpath := modfile.ModulePath(mod)
if modpath != "" {
cache.GOGARBLE = modpath + "," + modpath + "_test"
}
}
}
return nil
}