justan0th3rstud3nt
/
garble
mirror of https://github.com/burrowers/garble
1
0
Fork 0
Code Issues 1 Projects Releases Wiki Activity

start using original action IDs (#251)

Browse Source 
When we obfuscate a name, what we do is hash the name with the action ID
of the package that contains the name. To ensure that the hash changes
if the garble tool changes, we used the action ID of the obfuscated
build, which is different than the original action ID, as we include
garble's own content ID in "go tool compile -V=full" via -toolexec.

Let's call that the "obfuscated action ID". Remember that a content ID
is roughly the hash of a binary or object file, and an action ID
contains the hash of a package's source code plus the content IDs of its
dependencies.

This had the advantage that it did what we wanted. However, it had one
massive drawback: when we compile a package, we only have the obfuscated
action IDs of its dependencies. This is because one can't have the
content ID of dependent packages before they are built.

Usually, this is not a problem, because hashing a foreign name means it
comes from a dependency, where we already have the obfuscated action ID.
However, that's not always the case.

First, go:linkname directives can point to any symbol that ends up in
the binary, even if the package is not a dependency. So garble could
only support linkname targets belonging to dependencies. This is at the
root of why we could not obfuscate the runtime; it contains linkname
directives targeting the net package, for example, which depends on runtime.

Second, some other places did not have an easy access to obfuscated
action IDs, like transformAsm, which had to recover it from a temporary
file stored by transformCompile.

Plus, this was all pretty expensive, as each toolexec sub-process had to
make repeated calls to buildidOf with the object files of dependencies.
We even had to use extra calls to "go list" in the case of indirect
dependencies, as their export files do not appear in importcfg files.

All in all, the old method was complex and expensive. A better mechanism
is to use the original action IDs directly, as listed by "go list"
without garble in the picture.

This would mean that the hashing does not change if garble changes,
meaning weaker obfuscation. To regain that property, we define the
"garble action ID", which is just the original action ID hashed together
with garble's own content ID.

This is practically the same as the obfuscated build ID we used before,
but since it doesn't go through "go tool compile -V=full" and the
obfuscated build itself, we can work out *all* the garble action IDs
upfront, before the obfuscated build even starts.

This fixes all of our problems. Now we know all garble build IDs
upfront, so a bunch of hacks can be entirely removed. Plus, since we
know them upfront, we can also cache them and avoid repeated calls to
"go tool buildid".

While at it, make use of the new BuildID field in Go 1.16's "list -json
-export". This avoids the vast majority of "go tool buildid" calls, as
the only ones that remain are 2 on the garble binary itself.

The numbers for Go 1.16 look very good:

	name     old time/op       new time/op       delta
	Build-8        146ms ± 4%        101ms ± 1%  -31.01%  (p=0.002 n=6+6)

	name     old bin-B         new bin-B         delta
	Build-8        6.61M ± 0%        6.60M ± 0%   -0.09%  (p=0.002 n=6+6)

	name     old sys-time/op   new sys-time/op   delta
	Build-8        321ms ± 7%        202ms ± 6%  -37.11%  (p=0.002 n=6+6)

	name     old user-time/op  new user-time/op  delta
	Build-8        538ms ± 4%        414ms ± 4%  -23.12%  (p=0.002 n=6+6)
pull/252/head
Daniel Martí 3 years ago committed by GitHub
parent 09e244986e
commit 6898d61637
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
4 changed files with 182 additions and 236 deletions
Show Stats Download Patch File Download Diff File

41
hash.go
Unescape Escape View File

@ -10,14 +10,13 @@ import (
"fmt" "fmt"
"go/token" "go/token"
"io" "io"
"os"
"os/exec" "os/exec"
"strings" "strings"
) )
const buildIDSeparator = "/" const buildIDSeparator = "/"
// splitActionID returns the action ID half of a build ID, the first element. // splitActionID returns the action ID half of a build ID, the first component.
func splitActionID(buildID string) string { func splitActionID(buildID string) string {
i := strings.Index(buildID, buildIDSeparator) i := strings.Index(buildID, buildIDSeparator)
if i < 0 { if i < 0 {
@ -26,13 +25,13 @@ func splitActionID(buildID string) string {
return buildID[:i] return buildID[:i]
} }
// splitContentID returns the content ID half of a build ID, the last element. // splitContentID returns the content ID half of a build ID, the last component.
func splitContentID(buildID string) string { func splitContentID(buildID string) string {
return buildID[strings.LastIndex(buildID, buildIDSeparator)+1:] return buildID[strings.LastIndex(buildID, buildIDSeparator)+1:]
} }
// decodeHash is the opposite of hashToString, but with a panic for error // decodeHash is the opposite of hashToString, with a panic for error handling
// handling since it should never happen. // since it should never happen.
func decodeHash(str string) []byte { func decodeHash(str string) []byte {
h, err := base64.RawURLEncoding.DecodeString(str) h, err := base64.RawURLEncoding.DecodeString(str)
if err != nil { if err != nil {
@ -61,7 +60,7 @@ func alterToolVersion(tool string, args []string) error {
toolID = decodeHash(splitContentID(f[len(f)-1])) toolID = decodeHash(splitContentID(f[len(f)-1]))
} else { } else {
// For a release, the output is like: "compile version go1.9.1 X:framepointer". // For a release, the output is like: "compile version go1.9.1 X:framepointer".
// Use the whole line. // Use the whole line, as we can assume it's unique.
toolID = []byte(line) toolID = []byte(line)
} }
@ -76,33 +75,29 @@ func alterToolVersion(tool string, args []string) error {
// The slashes let us imitate a full binary build ID, but we assume that // The slashes let us imitate a full binary build ID, but we assume that
// the other components such as the action ID are not necessary, since the // the other components such as the action ID are not necessary, since the
// only reader here is cmd/go and it only consumes the content ID. // only reader here is cmd/go and it only consumes the content ID.
fmt.Printf("%s +garble buildID=_/_/_/%s\n", line, contentID) fmt.Printf("%s +garble buildID=_/_/_/%s\n", line, hashToString(contentID))
return nil return nil
} }
func ownContentID(toolID []byte) (string, error) { func ownContentID(toolID []byte) ([]byte, error) {
// We can't rely on the module version to exist, because it's // We can't rely on the module version to exist, because it's
// missing in local builds without 'go get'. // missing in local builds without 'go get'.
// For now, use 'go tool buildid' on the binary that's running. Just // For now, use 'go tool buildid' on the garble binary.
// like Go's own cache, we use hex-encoded sha256 sums. // Just like Go's own cache, we use hex-encoded sha256 sums.
// Once https://github.com/golang/go/issues/37475 is fixed, we // Once https://github.com/golang/go/issues/37475 is fixed, we
// can likely just use that. // can likely just use that.
path, err := os.Executable() binaryBuildID, err := buildidOf(cache.ExecPath)
if err != nil { if err != nil {
return "", err return nil, err
} }
buildID, err := buildidOf(path) binaryContentID := decodeHash(splitContentID(binaryBuildID))
if err != nil {
return "", err
}
ownID := decodeHash(splitContentID(buildID))
// Join the two content IDs together into a single base64-encoded sha256 // Join the two content IDs together into a single base64-encoded sha256
// sum. This includes the original tool's content ID, and garble's own // sum. This includes the original tool's content ID, and garble's own
// content ID. // content ID.
h := sha256.New() h := sha256.New()
h.Write(toolID) h.Write(toolID)
h.Write(ownID) h.Write(binaryContentID)
// We also need to add the selected options to the full version string, // We also need to add the selected options to the full version string,
// because all of them result in different output. We use spaces to // because all of them result in different output. We use spaces to
@ -120,13 +115,17 @@ func ownContentID(toolID []byte) (string, error) {
fmt.Fprintf(h, " -seed=%x", opts.Seed) fmt.Fprintf(h, " -seed=%x", opts.Seed)
} }
return hashToString(h.Sum(nil)), nil return h.Sum(nil)[:buildIDComponentLength], nil
} }
// buildIDComponentLength is the number of bytes each build ID component takes,
// such as an action ID or a content ID.
const buildIDComponentLength = 15
// hashToString encodes the first 120 bits of a sha256 sum in base64, the same // hashToString encodes the first 120 bits of a sha256 sum in base64, the same
// format used for elements in a build ID. // format used for components in a build ID.
func hashToString(h []byte) string { func hashToString(h []byte) string {
return base64.RawURLEncoding.EncodeToString(h[:15]) return base64.RawURLEncoding.EncodeToString(h[:buildIDComponentLength])
} }
func buildidOf(path string) (string, error) { func buildidOf(path string) (string, error) {

226
main.go
Unescape Escape View File

@ -9,7 +9,6 @@ import (
"compress/gzip" "compress/gzip"
"encoding/base64" "encoding/base64"
"encoding/binary" "encoding/binary"
"encoding/json"
"errors" "errors"
"flag" "flag"
"fmt" "fmt"
@ -100,35 +99,23 @@ var (
return os.Open(pkg.Export) return os.Open(pkg.Export)
}) })
// Basic information about the package being currently compiled or // Basic information about the package being currently compiled or linked.
// linked. These variables are filled in early, and reused later. curPkg *listedPackage
curPkgPath string // note that this isn't filled for the linker yet
curActionID []byte
curImportCfg string
buildInfo = struct { buildInfo = struct {
// TODO: replace part of this with goobj.ParseImportCfg, so that // TODO: do we still need imports?
// we can also reuse it. For now, parsing ourselves is still
// necessary so that we can set firstImport.
imports map[string]importedPkg // parsed importCfg plus cached info imports map[string]importedPkg // parsed importCfg plus cached info
firstImport string // first from -importcfg; the main package when linking
}{imports: make(map[string]importedPkg)} }{imports: make(map[string]importedPkg)}
garbledImporter = importer.ForCompiler(fset, "gc", func(path string) (io.ReadCloser, error) { garbledImporter = importer.ForCompiler(fset, "gc", func(path string) (io.ReadCloser, error) {
return os.Open(buildInfo.imports[path].packagefile) return os.Open(buildInfo.imports[path].packagefile)
}).(types.ImporterFrom) }).(types.ImporterFrom)
opts *options opts *flagOptions
envGoPrivate = os.Getenv("GOPRIVATE") // complemented by 'go env' later envGoPrivate = os.Getenv("GOPRIVATE") // complemented by 'go env' later
) )
const (
// Note that these are capped at 16 bytes.
headerDebugSource = "garble/debugSrc"
)
func garbledImport(path string) (*types.Package, error) { func garbledImport(path string) (*types.Package, error) {
ipkg, ok := buildInfo.imports[path] ipkg, ok := buildInfo.imports[path]
if !ok { if !ok {
@ -150,7 +137,6 @@ func garbledImport(path string) (*types.Package, error) {
type importedPkg struct { type importedPkg struct {
packagefile string packagefile string
actionID []byte
pkg *types.Package pkg *types.Package
} }
@ -284,7 +270,7 @@ func mainErr(args []string) error {
// We're in a toolexec sub-process, not directly called by the user. // We're in a toolexec sub-process, not directly called by the user.
// Load the shared data and wrap the tool, like the compiler or linker. // Load the shared data and wrap the tool, like the compiler or linker.
if err := loadShared(); err != nil { if err := loadSharedCache(); err != nil {
return err return err
} }
opts = &cache.Options opts = &cache.Options
@ -334,10 +320,14 @@ func toolexecCmd(command string, args []string) (*exec.Cmd, error) {
} }
} }
if err := setOptions(); err != nil { if err := setFlagOptions(); err != nil {
return nil, err return nil, err
} }
// Here is the only place we initialize the cache.
// The sub-processes will parse it from a shared gob file.
cache = &sharedCache{Options: *opts}
// Note that we also need to pass build flags to 'go list', such // Note that we also need to pass build flags to 'go list', such
// as -tags. // as -tags.
cache.BuildFlags = filterBuildFlags(flags) cache.BuildFlags = filterBuildFlags(flags)
@ -359,7 +349,7 @@ func toolexecCmd(command string, args []string) (*exec.Cmd, error) {
return nil, err return nil, err
} }
sharedTempDir, err = saveShared() sharedTempDir, err = saveSharedCache()
if err != nil { if err != nil {
return nil, err return nil, err
} }
@ -406,7 +396,7 @@ func transformAsm(args []string) ([]string, error) {
symAbis = true symAbis = true
} }
} }
curPkgPath = flagValue(flags, "-p") curPkgPath := flagValue(flags, "-p")
// If we are generating symbol ABIs, the output does not actually // If we are generating symbol ABIs, the output does not actually
// contain curPkgPath. Exported APIs show up as "".FooBar. // contain curPkgPath. Exported APIs show up as "".FooBar.
@ -415,14 +405,13 @@ func transformAsm(args []string) ([]string, error) {
// To obfuscate the path in the -p flag, we need the current action ID, // To obfuscate the path in the -p flag, we need the current action ID,
// which we recover from the file that transformCompile wrote for us. // which we recover from the file that transformCompile wrote for us.
if !symAbis && curPkgPath != "main" && isPrivate(curPkgPath) { if !symAbis && curPkgPath != "main" && isPrivate(curPkgPath) {
savedActionID := filepath.Join(sharedTempDir, strings.ReplaceAll(curPkgPath, "/", ",")) curPkgPathFull := curPkgPath
var err error if curPkgPathFull == "main" {
curActionID, err = ioutil.ReadFile(savedActionID) // TODO(mvdan): this can go with TOOLEXEC_IMPORTPATH
if err != nil { curPkgPathFull = cache.MainImportPath
return nil, fmt.Errorf("could not read build ID: %v", err)
} }
newPkgPath := hashWith(curActionID, curPkgPath) newPkgPath := hashWith(cache.ListedPackages[curPkgPathFull].GarbleActionID, curPkgPath)
flags = flagSetValue(flags, "-p", newPkgPath) flags = flagSetValue(flags, "-p", newPkgPath)
} }
@ -437,7 +426,7 @@ func transformCompile(args []string) ([]string, error) {
// generating it. // generating it.
flags = append(flags, "-dwarf=false") flags = append(flags, "-dwarf=false")
curPkgPath = flagValue(flags, "-p") curPkgPath := flagValue(flags, "-p")
if (curPkgPath == "runtime" && opts.Tiny) || curPkgPath == "runtime/internal/sys" { if (curPkgPath == "runtime" && opts.Tiny) || curPkgPath == "runtime/internal/sys" {
// Even though these packages aren't private, we will still process // Even though these packages aren't private, we will still process
// them later to remove build information and strip code from the // them later to remove build information and strip code from the
@ -465,27 +454,19 @@ func transformCompile(args []string) ([]string, error) {
if !strings.Contains(trimpath, ";") { if !strings.Contains(trimpath, ";") {
return nil, fmt.Errorf("-toolexec=garble should be used alongside -trimpath") return nil, fmt.Errorf("-toolexec=garble should be used alongside -trimpath")
} }
curPkgPathFull := curPkgPath
if curPkgPathFull == "main" {
// TODO(mvdan): this can go with TOOLEXEC_IMPORTPATH
curPkgPathFull = cache.MainImportPath
}
curPkg = cache.ListedPackages[curPkgPathFull]
newImportCfg, err := fillBuildInfo(flags) newImportCfg, err := fillBuildInfo(flags)
if err != nil { if err != nil {
return nil, err return nil, err
} }
// Tools which run after the main compile run, such as asm, also need
// the current action ID to obfuscate the package path in their -p flag.
// They lack the -buildid flag, so store it in a unique file here to be
// recovered by the other tools later, such as transformAsm.
// Import paths include slashes, which usually cannot be in filenames,
// so replace those with commas, which should be fine and cannot be part
// of an import path.
// We only write each file once, as we compile each package once.
// Each filename is also unique, since import paths are unique.
// TODO: perhaps error if the file already exists, to double check that
// the assumptions above are correct.
savedActionID := filepath.Join(sharedTempDir, strings.ReplaceAll(curPkgPath, "/", ","))
if err := ioutil.WriteFile(savedActionID, curActionID, 0o666); err != nil {
return nil, fmt.Errorf("could not store action ID: %v", err)
}
var files []*ast.File var files []*ast.File
for _, path := range paths { for _, path := range paths {
file, err := parser.ParseFile(fset, path, nil, parser.ParseComments) file, err := parser.ParseFile(fset, path, nil, parser.ParseComments)
@ -497,7 +478,7 @@ func transformCompile(args []string) ([]string, error) {
randSeed := opts.Seed randSeed := opts.Seed
if len(randSeed) == 0 { if len(randSeed) == 0 {
randSeed = curActionID randSeed = curPkg.GarbleActionID
} }
// log.Printf("seeding math/rand with %x\n", randSeed) // log.Printf("seeding math/rand with %x\n", randSeed)
mathrand.Seed(int64(binary.BigEndian.Uint64(randSeed))) mathrand.Seed(int64(binary.BigEndian.Uint64(randSeed)))
@ -535,7 +516,7 @@ func transformCompile(args []string) ([]string, error) {
} }
origTypesConfig := types.Config{Importer: origImporter} origTypesConfig := types.Config{Importer: origImporter}
tf.pkg, err = origTypesConfig.Check(curPkgPath, fset, files, tf.info) tf.pkg, err = origTypesConfig.Check(curPkgPathFull, fset, files, tf.info)
if err != nil { if err != nil {
return nil, fmt.Errorf("typecheck error: %v", err) return nil, fmt.Errorf("typecheck error: %v", err)
} }
@ -575,7 +556,8 @@ func transformCompile(args []string) ([]string, error) {
// package path. // package path.
newPkgPath := curPkgPath newPkgPath := curPkgPath
if curPkgPath != "main" && isPrivate(curPkgPath) { if curPkgPath != "main" && isPrivate(curPkgPath) {
newPkgPath = hashWith(curActionID, curPkgPath) newPkgPath = hashWith(curPkg.GarbleActionID, curPkgPath)
// println("compile -p:", curPkgPath, newPkgPath)
flags = flagSetValue(flags, "-p", newPkgPath) flags = flagSetValue(flags, "-p", newPkgPath)
} }
@ -631,15 +613,14 @@ func transformCompile(args []string) ([]string, error) {
// Replace the import path with its obfuscated version. // Replace the import path with its obfuscated version.
// If the import was unnamed, give it the name of the // If the import was unnamed, give it the name of the
// original package name, to keep references working. // original package name, to keep references working.
actionID := buildInfo.imports[path].actionID lpkg, err := listPackage(path)
newPath := hashWith(actionID, path)
imp.Path.Value = strconv.Quote(newPath)
if imp.Name == nil {
pkg, err := listPackage(path)
if err != nil { if err != nil {
panic(err) // should never happen panic(err) // should never happen
} }
imp.Name = &ast.Ident{Name: pkg.Name} newPath := hashWith(lpkg.GarbleActionID, path)
imp.Path.Value = strconv.Quote(newPath)
if imp.Name == nil {
imp.Name = &ast.Ident{Name: lpkg.Name}
} }
return true return true
}) })
@ -738,30 +719,30 @@ func (tf *transformer) handleDirectives(comments []string) {
if len(target) != 2 { if len(target) != 2 {
continue continue
} }
pkg, name := target[0], target[1] pkgPath, name := target[0], target[1]
if pkg == "runtime" && strings.HasPrefix(name, "cgo") { if pkgPath == "runtime" && strings.HasPrefix(name, "cgo") {
continue // ignore cgo-generated linknames continue // ignore cgo-generated linknames
} }
if !isPrivate(pkg) { if !isPrivate(pkgPath) {
continue // ignore non-private symbols continue // ignore non-private symbols
} }
listedPkg, ok := buildInfo.imports[pkg] lpkg, err := listPackage(pkgPath)
if !ok { if err != nil {
continue // probably a made up symbol name continue // probably a made up symbol name
} }
garbledPkg, _ := garbledImport(pkg) garbledPkg, _ := garbledImport(pkgPath)
if garbledPkg != nil && garbledPkg.Scope().Lookup(name) != nil { if garbledPkg != nil && garbledPkg.Scope().Lookup(name) != nil {
continue // the name exists and was not garbled continue // the name exists and was not garbled
} }
// The name exists and was obfuscated; replace the // The name exists and was obfuscated; replace the
// comment with the obfuscated name. // comment with the obfuscated name.
newName := hashWith(listedPkg.actionID, name) newName := hashWith(lpkg.GarbleActionID, name)
newPkg := pkg newPkgPath := pkgPath
if pkg != "main" { if pkgPath != "main" {
newPkg = hashWith(listedPkg.actionID, pkg) newPkgPath = hashWith(lpkg.GarbleActionID, pkgPath)
} }
fields[2] = newPkg + "." + newName fields[2] = newPkgPath + "." + newName
comments[i] = strings.Join(fields, " ") comments[i] = strings.Join(fields, " ")
} }
} }
@ -878,18 +859,11 @@ func isPrivate(path string) bool {
// and constructs a new importcfg with the obfuscated import paths changed as // and constructs a new importcfg with the obfuscated import paths changed as
// necessary. // necessary.
func fillBuildInfo(flags []string) (newImportCfg string, _ error) { func fillBuildInfo(flags []string) (newImportCfg string, _ error) {
buildID := flagValue(flags, "-buildid") importCfg := flagValue(flags, "-importcfg")
switch buildID { if importCfg == "" {
case "", "true":
return "", fmt.Errorf("could not find -buildid argument")
}
curActionID = decodeHash(splitActionID(buildID))
curImportCfg = flagValue(flags, "-importcfg")
if curImportCfg == "" {
return "", fmt.Errorf("could not find -importcfg argument") return "", fmt.Errorf("could not find -importcfg argument")
} }
data, err := ioutil.ReadFile(curImportCfg) data, err := ioutil.ReadFile(importCfg)
if err != nil { if err != nil {
return "", err return "", err
} }
@ -921,20 +895,8 @@ func fillBuildInfo(flags []string) (newImportCfg string, _ error) {
continue continue
} }
importPath, objectPath := args[:j], args[j+1:] importPath, objectPath := args[:j], args[j+1:]
buildID, err := buildidOf(objectPath)
if err != nil {
return "", err
}
// log.Println("buildid:", buildID)
if len(buildInfo.imports) == 0 { impPkg := importedPkg{packagefile: objectPath}
buildInfo.firstImport = importPath
}
actionID := decodeHash(splitActionID(buildID))
impPkg := importedPkg{
packagefile: objectPath,
actionID: actionID,
}
buildInfo.imports[importPath] = impPkg buildInfo.imports[importPath] = impPkg
if otherPath, ok := importMap[importPath]; ok { if otherPath, ok := importMap[importPath]; ok {
@ -953,16 +915,23 @@ func fillBuildInfo(flags []string) (newImportCfg string, _ error) {
return "", err return "", err
} }
for beforePath, afterPath := range importMap { for beforePath, afterPath := range importMap {
if isPrivate(afterPath) { if isPrivate(beforePath) {
actionID := buildInfo.imports[afterPath].actionID println(beforePath, afterPath)
afterPath = hashWith(actionID, afterPath) pkg, err := listPackage(beforePath)
if err != nil {
panic(err) // shouldn't happen
}
afterPath = hashWith(pkg.GarbleActionID, afterPath)
} }
fmt.Fprintf(newCfg, "importmap %s=%s\n", beforePath, afterPath) fmt.Fprintf(newCfg, "importmap %s=%s\n", beforePath, afterPath)
} }
for impPath, pkg := range buildInfo.imports { for impPath, pkg := range buildInfo.imports {
if isPrivate(impPath) { if isPrivate(impPath) {
actionID := buildInfo.imports[impPath].actionID pkg, err := listPackage(impPath)
impPath = hashWith(actionID, impPath) if err != nil {
panic(err) // shouldn't happen
}
impPath = hashWith(pkg.GarbleActionID, impPath)
} }
fmt.Fprintf(newCfg, "packagefile %s=%s\n", impPath, pkg.packagefile) fmt.Fprintf(newCfg, "packagefile %s=%s\n", impPath, pkg.packagefile)
} }
@ -1172,69 +1141,15 @@ func (tf *transformer) transformGo(file *ast.File) *ast.File {
return true // we only want to rename the above return true // we only want to rename the above
} }
// Handle the case where the name is defined in an indirectly lpkg, err := listPackage(path)
// imported package. Since only direct imports show up in our
// importcfg, buildInfo.imports will not initially contain the
// package path we want.
//
// This edge case can happen, for example, if package A imports
// package B and calls its API, and B's API returns C's struct.
// Suddenly, A can use struct field names defined in C, even
// though A never directly imports C.
//
// For this rare case, for now, do an extra "go list -toolexec"
// call to retrieve its export path.
// TODO: Think about ways to avoid this extra exec call. Perhaps
// add an extra archive header to record all direct and indirect
// importcfg data, like we do with private name maps.
if _, e := buildInfo.imports[path]; !e && path != curPkgPath {
goArgs := []string{
"list",
"-json",
"-export",
"-trimpath",
"-toolexec=" + cache.ExecPath,
}
goArgs = append(goArgs, cache.BuildFlags...)
goArgs = append(goArgs, path)
cmd := exec.Command("go", goArgs...)
cmd.Dir = opts.GarbleDir
out, err := cmd.Output()
if err != nil {
if err := err.(*exec.ExitError); err != nil {
panic(fmt.Sprintf("%v: %s", err, err.Stderr))
}
panic(err)
}
var pkg listedPackage
if err := json.Unmarshal(out, &pkg); err != nil {
panic(err) // shouldn't happen
}
buildID, err := buildidOf(pkg.Export)
if err != nil { if err != nil {
panic(err) // shouldn't happen panic(err) // shouldn't happen
} }
// Adding it to buildInfo.imports allows us to reuse the
// "if" branch below. Plus, if this edge case triggers
// multiple times in a single package compile, we can
// call "go list" once and cache its result.
if pkg.ImportPath != path {
panic(fmt.Sprintf("unexpected path: %q vs %q", pkg.ImportPath, path))
}
buildInfo.imports[path] = importedPkg{
packagefile: pkg.Export,
actionID: decodeHash(splitActionID(buildID)),
}
// log.Printf("fetched indirect dependency %q from: %s", path, pkg.Export)
}
actionID := curActionID
// TODO: Make this check less prone to bugs, like the one we had // TODO: Make this check less prone to bugs, like the one we had
// with indirect dependencies. If "path" is not our current // with indirect dependencies. If "path" is not our current
// package, then it must exist in buildInfo.imports. Otherwise // package, then it must exist in buildInfo.imports. Otherwise
// we should panic. // we should panic.
if id := buildInfo.imports[path].actionID; len(id) > 0 { if buildInfo.imports[path].packagefile != "" {
garbledPkg, err := garbledImport(path) garbledPkg, err := garbledImport(path)
if err != nil { if err != nil {
panic(err) // shouldn't happen panic(err) // shouldn't happen
@ -1245,14 +1160,13 @@ func (tf *transformer) transformGo(file *ast.File) *ast.File {
if o := garbledPkg.Scope().Lookup(obj.Name()); o != nil && reflect.TypeOf(o) == reflect.TypeOf(obj) { if o := garbledPkg.Scope().Lookup(obj.Name()); o != nil && reflect.TypeOf(o) == reflect.TypeOf(obj) {
return true return true
} }
actionID = id
} }
origName := node.Name origName := node.Name
_ = origName // used for debug prints below _ = origName // used for debug prints below
node.Name = hashWith(actionID, node.Name) node.Name = hashWith(lpkg.GarbleActionID, node.Name)
// log.Printf("%q hashed with %x to %q", origName, actionID, node.Name) // log.Printf("%q hashed with %x to %q", origName, lpkg.GarbleActionID, node.Name)
return true return true
} }
return astutil.Apply(file, pre, nil).(*ast.File) return astutil.Apply(file, pre, nil).(*ast.File)
@ -1320,6 +1234,8 @@ func transformLink(args []string) ([]string, error) {
// lack any extension. // lack any extension.
flags, args := splitFlagsFromArgs(args) flags, args := splitFlagsFromArgs(args)
curPkg = cache.ListedPackages[cache.MainImportPath]
newImportCfg, err := fillBuildInfo(flags) newImportCfg, err := fillBuildInfo(flags)
if err != nil { if err != nil {
return nil, err return nil, err
@ -1344,11 +1260,9 @@ func transformLink(args []string) ([]string, error) {
pkgPath := pkg pkgPath := pkg
if pkgPath == "main" { if pkgPath == "main" {
// The main package is known under its import path in pkgPath = cache.MainImportPath
// the import config map.
pkgPath = buildInfo.firstImport
} }
id := buildInfo.imports[pkgPath].actionID id := cache.ListedPackages[pkgPath].GarbleActionID
newName := hashWith(id, name) newName := hashWith(id, name)
newPkg := pkg newPkg := pkg
if pkg != "main" && isPrivate(pkg) { if pkg != "main" && isPrivate(pkg) {

21
reverse.go
Unescape Escape View File

@ -37,6 +37,7 @@ func commandReverse(args []string) error {
if err != nil { if err != nil {
return err return err
} }
curPkg = cache.ListedPackages[cache.MainImportPath]
stdout, err := cmd.StdoutPipe() stdout, err := cmd.StdoutPipe()
if err != nil { if err != nil {
@ -69,16 +70,9 @@ func commandReverse(args []string) error {
if isPrivate(pkg.ImportPath) { if isPrivate(pkg.ImportPath) {
privatePkgPaths = append(privatePkgPaths, pkg.ImportPath) privatePkgPaths = append(privatePkgPaths, pkg.ImportPath)
} }
buildID, err := buildidOf(pkg.Export)
if err != nil {
return err
}
// The action ID, and possibly the export file, will be used // The action ID, and possibly the export file, will be used
// later to reconstruct the mapping of obfuscated names. // later to reconstruct the mapping of obfuscated names.
buildInfo.imports[pkg.ImportPath] = importedPkg{ buildInfo.imports[pkg.ImportPath] = importedPkg{packagefile: pkg.Export}
packagefile: pkg.Export,
actionID: decodeHash(splitActionID(buildID)),
}
} }
if err := cmd.Wait(); err != nil { if err := cmd.Wait(); err != nil {
@ -94,18 +88,17 @@ func commandReverse(args []string) error {
fset := token.NewFileSet() fset := token.NewFileSet()
for _, pkgPath := range privatePkgPaths { for _, pkgPath := range privatePkgPaths {
ipkg := buildInfo.imports[pkgPath] lpkg, err := listPackage(pkgPath)
if err != nil {
return err
}
addReplace := func(str string) { addReplace := func(str string) {
replaces = append(replaces, hashWith(ipkg.actionID, str), str) replaces = append(replaces, hashWith(lpkg.GarbleActionID, str), str)
} }
// Package paths are obfuscated, too. // Package paths are obfuscated, too.
addReplace(pkgPath) addReplace(pkgPath)
lpkg, err := listPackage(pkgPath)
if err != nil {
return err
}
for _, goFile := range lpkg.GoFiles { for _, goFile := range lpkg.GoFiles {
goFile = filepath.Join(lpkg.Dir, goFile) goFile = filepath.Join(lpkg.Dir, goFile)
file, err := parser.ParseFile(fset, goFile, nil, 0) file, err := parser.ParseFile(fset, goFile, nil, 0)

108
shared.go
Unescape Escape View File

@ -3,6 +3,7 @@ package main
import ( import (
"bytes" "bytes"
"crypto/rand" "crypto/rand"
"crypto/sha256"
"encoding/base64" "encoding/base64"
"encoding/gob" "encoding/gob"
"encoding/json" "encoding/json"
@ -14,20 +15,31 @@ import (
"strings" "strings"
) )
// shared this data is shared between the different garble processes // sharedCache this data is sharedCache between the different garble processes.
type shared struct { //
// Note that we fill this cache once from the root process in saveListedPackages,
// store it into a temporary file via gob encoding, and then reuse that file
// in each of the garble toolexec sub-processes.
type sharedCache struct {
ExecPath string // absolute path to the garble binary being used ExecPath string // absolute path to the garble binary being used
BuildFlags []string // build flags fed to the original "garble ..." command BuildFlags []string // build flags fed to the original "garble ..." command
Options options // garble options being used, i.e. our own flags Options flagOptions // garble options being used, i.e. our own flags
ListedPackages listedPackages // non-garbled view of all packages to build
// ListedPackages contains data obtained via 'go list -json -export -deps'. This
// allows us to obtain the non-garbled export data of all dependencies, useful
// for type checking of the packages as we obfuscate them.
ListedPackages map[string]*listedPackage
MainImportPath string // TODO: remove with TOOLEXEC_IMPORTPATH
} }
var cache *shared var cache *sharedCache
// loadShared the shared data passed from the entry garble process // loadSharedCache the shared data passed from the entry garble process
func loadShared() error { func loadSharedCache() error {
if cache == nil { if cache != nil {
panic("shared cache loaded twice?")
}
f, err := os.Open(filepath.Join(sharedTempDir, "main-cache.gob")) f, err := os.Open(filepath.Join(sharedTempDir, "main-cache.gob"))
if err != nil { if err != nil {
return fmt.Errorf(`cannot open shared file, this is most likely due to not running "garble [command]"`) return fmt.Errorf(`cannot open shared file, this is most likely due to not running "garble [command]"`)
@ -36,14 +48,15 @@ func loadShared() error {
if err := gob.NewDecoder(f).Decode(&cache); err != nil { if err := gob.NewDecoder(f).Decode(&cache); err != nil {
return err return err
} }
}
return nil return nil
} }
// saveShared creates a temporary directory to share between garble processes. // saveSharedCache creates a temporary directory to share between garble processes.
// This directory also includes the gob-encoded cache global. // This directory also includes the gob-encoded cache global.
func saveShared() (string, error) { func saveSharedCache() (string, error) {
if cache == nil {
panic("saving a missing cache?")
}
dir, err := ioutil.TempDir("", "garble-shared") dir, err := ioutil.TempDir("", "garble-shared")
if err != nil { if err != nil {
return "", err return "", err
@ -62,8 +75,8 @@ func saveShared() (string, error) {
return dir, nil return dir, nil
} }
// options are derived from the flags // flagOptions are derived from the flags
type options struct { type flagOptions struct {
GarbleLiterals bool GarbleLiterals bool
Tiny bool Tiny bool
GarbleDir string GarbleDir string
@ -72,14 +85,17 @@ type options struct {
Random bool Random bool
} }
// setOptions sets all options from the user supplied flags. // setFlagOptions sets flagOptions from the user supplied flags.
func setOptions() error { func setFlagOptions() error {
wd, err := os.Getwd() wd, err := os.Getwd()
if err != nil { if err != nil {
return err return err
} }
opts = &options{ if cache != nil {
panic("opts set twice?")
}
opts = &flagOptions{
GarbleDir: wd, GarbleDir: wd,
GarbleLiterals: flagGarbleLiterals, GarbleLiterals: flagGarbleLiterals,
Tiny: flagGarbleTiny, Tiny: flagGarbleTiny,
@ -124,31 +140,28 @@ func setOptions() error {
opts.DebugDir = flagDebugDir opts.DebugDir = flagDebugDir
} }
cache = &shared{Options: *opts}
return nil return nil
} }
// listedPackages contains data obtained via 'go list -json -export -deps'. This // listedPackage contains the 'go list -json -export' fields obtained by the
// allows us to obtain the non-garbled export data of all dependencies, useful // root process, shared with all garble sub-processes via a file.
// for type checking of the packages as we obfuscate them.
//
// Note that we obtain this data once in saveListedPackages, store it into a
// temporary file via gob encoding, and then reuse that file in each of the
// garble processes that wrap a package compilation.
type listedPackages map[string]*listedPackage
// listedPackage contains information useful for obfuscating a package
type listedPackage struct { type listedPackage struct {
Name string Name string
ImportPath string ImportPath string
Export string Export string
BuildID string
Deps []string Deps []string
ImportMap map[string]string ImportMap map[string]string
Dir string Dir string
GoFiles []string GoFiles []string
// The fields below are not part of 'go list', but are still reused
// between garble processes. Use "Garble" as a prefix to ensure no
// collisions with the JSON fields from 'go list'.
GarbleActionID []byte
// TODO(mvdan): reuse this field once TOOLEXEC_IMPORTPATH is used // TODO(mvdan): reuse this field once TOOLEXEC_IMPORTPATH is used
private bool private bool
} }
@ -156,7 +169,7 @@ type listedPackage struct {
// setListedPackages gets information about the current package // setListedPackages gets information about the current package
// and all of its dependencies // and all of its dependencies
func setListedPackages(patterns []string) error { func setListedPackages(patterns []string) error {
args := []string{"list", "-json", "-deps", "-export"} args := []string{"list", "-json", "-deps", "-export", "-trimpath"}
args = append(args, cache.BuildFlags...) args = append(args, cache.BuildFlags...)
args = append(args, patterns...) args = append(args, patterns...)
cmd := exec.Command("go", args...) cmd := exec.Command("go", args...)
@ -172,13 +185,42 @@ func setListedPackages(patterns []string) error {
if err := cmd.Start(); err != nil { if err := cmd.Start(); err != nil {
return fmt.Errorf("go list error: %v", err) return fmt.Errorf("go list error: %v", err)
} }
binaryBuildID, err := buildidOf(cache.ExecPath)
if err != nil {
return err
}
binaryContentID := decodeHash(splitContentID(binaryBuildID))
dec := json.NewDecoder(stdout) dec := json.NewDecoder(stdout)
cache.ListedPackages = make(listedPackages) cache.ListedPackages = make(map[string]*listedPackage)
for dec.More() { for dec.More() {
var pkg listedPackage var pkg listedPackage
if err := dec.Decode(&pkg); err != nil { if err := dec.Decode(&pkg); err != nil {
return err return err
} }
if pkg.Export != "" {
buildID := pkg.BuildID
if buildID == "" {
// go list only includes BuildID in 1.16+
buildID, err = buildidOf(pkg.Export)
if err != nil {
panic(err) // shouldn't happen
}
}
actionID := decodeHash(splitActionID(buildID))
h := sha256.New()
h.Write(actionID)
h.Write(binaryContentID)
pkg.GarbleActionID = h.Sum(nil)[:buildIDComponentLength]
}
if pkg.Name == "main" {
if cache.MainImportPath != "" {
return fmt.Errorf("found two main packages: %s %s", cache.MainImportPath, pkg.ImportPath)
}
cache.MainImportPath = pkg.ImportPath
}
cache.ListedPackages[pkg.ImportPath] = &pkg cache.ListedPackages[pkg.ImportPath] = &pkg
} }
@ -217,11 +259,9 @@ func listPackage(path string) (*listedPackage, error) {
// If the path is listed in the top-level ImportMap, use its mapping instead. // If the path is listed in the top-level ImportMap, use its mapping instead.
// This is a common scenario when dealing with vendored packages in GOROOT. // This is a common scenario when dealing with vendored packages in GOROOT.
// The map is flat, so we don't need to recurse. // The map is flat, so we don't need to recurse.
if fromPkg, ok := cache.ListedPackages[curPkgPath]; ok { if path2 := curPkg.ImportMap[path]; path2 != "" {
if path2 := fromPkg.ImportMap[path]; path2 != "" {
path = path2 path = path2
} }
}
pkg, ok := cache.ListedPackages[path] pkg, ok := cache.ListedPackages[path]
if !ok { if !ok {

Write Preview
Loading…
Cancel
Save