printOneCgoTraceback now returns a boolean rather than an int.
Since we need to have different logic based on the Go version,
and toolchainVersionSemver was only set for the main process,
move the string to the shared cache global.
This is a nice thing to do anyway, to reduce the number of globals.
While here, update actions/setup-go to v4, which starts caching
GOMODCACHE and GOCACHE by default now.
Disable it, because it still doesn't help in our case,
and GitHub's Actions caching is still really inefficient.
And update staticcheck too.
The seedFlag.random field had never worked,
as my refactor in December 2021 never set it to true.
Even if the boolean was working, we only printed the random seed
when we failed. It's still useful to see it when a build succeeds,
for example when wanting to reproduce the same binary
or when wanting to reverse a panic from the produced binary.
Add a test this time.
Fixes#696.
Similar to what testscript does, we can reuse the test binary by telling
TestMain to run the main function rather than the Go tests.
This saves a few hundred milliseconds out of each benchmark run.
https://go.dev/cl/466095 lightly refactored the runtime
in a way that broke extractNameOff. In particular, the code
func cfuncname(f funcInfo) *byte {
if !f.valid() || f.nameOff == 0 {
return nil
}
return &f.datap.funcnametab[f.nameOff]
}
func funcname(f funcInfo) string {
return gostringnocopy(cfuncname(f))
}
is now simply
func funcname(f funcInfo) string {
if !f.valid() {
return ""
}
return f.datap.funcName(f.nameOff)
}
Since extractNameOff looked for the func named cfuncname,
and looked for the nameOff selector inside an index expression,
all of that code no longer worked properly.
It all existed to find the name of the field, nameOff,
so that we would automatically adapt if upstream renames it.
Unsurprisingly, the code using the field got refactored first.
It doesn't seem like the extra code on our part is helping us,
and assuming the name of the field works for all Go versions,
so do that instead.
If upstream does rename the field in the future,
the obfuscated Go builds will start failing in an obvious way.
If or when that comes to pass, we can change our constant string.
Added in Go 1.19, types like sync/atomic.Uint64 are handy,
because they ensure proper alignment even on 32-bit GOOSes.
However, this was done via a magic `type align64 struct{}`,
which the compiler spotted by name.
To keep that magic working, do not obfuscate the name.
Neither package path was being obfuscated,
as both packages contain compiler intrinsics already.
Fixes#686.
I mistakenly understood that, when the DepsErrors field has errors,
the Error field would contain an error as well.
That is not always the case; for example,
the imports_missing package in the added test script
had DepsErrors set but Error empty, causing a nil dereference panic.
Make the code more robust, and report both kinds of load errors.
Fixes#694.
We're building the linker binary for the host GOOS,
not the target GOOS that we happen to be building for.
I noticed that, after running `go test`, my garble cache
would contain both link and link.exe, which made no sense
as I run linux and not windows.
`go env` has GOHOSTOS to mirror GOOS, but there is no
GOHOSTEXE to mirror GOEXE, so we reconstruct it from
runtime.GOOS, which is equivalent to GOHOSTOS.
Add a regression test as well.
When we use `go list` on the standard library, we need to be careful
about what flags are passed from the top-level build command,
because some flags are not going to be appropriate.
In particular, GOFLAGS=-modfile=... resulted in a failure,
reproduced via the GOFLAGS variable added to linker.txtar:
go: inconsistent vendoring in /home/mvdan/tip/src:
golang.org/x/crypto@v0.5.1-0.20230203195927-310bfa40f1e4: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
golang.org/x/net@v0.7.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
golang.org/x/sys@v0.5.1-0.20230208141308-4fee21c92339: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
golang.org/x/text@v0.7.1-0.20230207171107-30dadde3188b: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
To ignore the vendor directory, use -mod=readonly or -mod=mod.
To sync the vendor directory, run:
go mod vendor
To work around this problem, reset the -mod and -modfile flags when
calling "go list" on the standard library, as those are the only two
flags which alter how we load the main module in a build.
The code which builds a modified cmd/link has a similar problem;
it already reset GOOS and GOARCH, but it could similarly run into
problems if other env vars like GOFLAGS were set.
To be on the safe side, we also disable GOENV and GOEXPERIMENT,
which we borrow from Go's bootstrapping commands.
`go build ./...` does indeed compile and link main packages,
it just does not move the resulting binaries anywhere permanent
like `go install` does.
As such, the TODO isn't relevant; the fact that we build all packages
inside each module means we are already linking any binaries matched via
`./...` from the module root.
We don't run any of the binaries, which would catch panics at run-time,
but we already have a note at the top about using `garble test`.
The current garble release is able to obfuscate it with Go 1.20.
While here, re-generate all files to use "go 1.20" directives,
and add a TODO about also testing binary builds for each project.
See #600.
Per the inline comment, we want to build every package in the test suite
at least once, otherwise we won't get proper code coverage information.
For example, some code only triggers when obfuscating the runtime,
and if I run "go test" twice in a row, the second might reuse the
first's obfuscation of the runtime and skip the code entirely.
However, per the TODO, we used to solve this in a rather wasteful way,
by making each test use its own separate GOCACHE directory.
Instead, use a new GOCACHE directory, but share it between tests.
This is still enough, as we still build each package at least once.
Running the command below twice in a row with Go 1.20:
go test -cover
took about 1m55s on my laptop before, and now takes about 1m10s.
Not great, but a noticeable improvement.
Both report a total coverage of 88.7%.
While here, do the same for GARBLE_CACHE_DIR,
and use testscript.Env.Setenv for simplicity.
At linker stage, we now encrypt funcInfo.entryoff value with a simple algorithm (1 xor + 1 mul).
This makes it harder to relate function metadata (e.g. name) to function itself in binary, almost without affecting performance.
The test package hack still appears to be needed as of Go 1.20.
Interestingly, the cgo filename check does not trigger at all anymore.
Presumably this means that it's not needed at all, and obfuscating code
generated by cgo appears to be fine. Go with that for now.
Now that we dropped Go 1.19, we can use it again,
since the referenced bug was fixed in Go 1.20.
This is a separate commit, as this change does alter the way we
obfuscate Go builds, so it's not just a cleanup.
It assumes that reflect.Value has a field named "flag",
which wasn't the case with obfuscated builds as we obfuscated it.
We already treated the reflect package as special,
for instance when not obfuscating Method or MethodByName.
In a similar fashion, mark reflect's rtype and Value types to not be
obfuscated alongside their field names. Note that rtype is the
implementation behind the reflect.Type interface.
This fix is fairly manual and repetitive.
transformCompile, transformLinkname, and transformAsm should all
use the same mechanism to tell if names should be obfuscated.
However, they do not do that right now, and that refactor feels too
risky for a bugfix release. We add more TODOs instead.
We're not adding go-spew to scripts/check-third-party.sh since the
project is largely abandoned. It's not even a Go module yet.
The only broken bit from it is what we've added to our tests.
Fixes#676.
Even when setting git's current directory to a temporary directory,
it could find a git repository in a parent directory and still malfunction.
Use the --git-dir flag to ensure that walking doesn't happen at all.
While here, ensure that "git apply" is always applying our patches,
and add a regression test to linker.txtar when not testing with -short.
Go contains such inline comments, like:
crypto/aes/gcm_ppc64x.s:129: VPMSUMD IN, HL, XL // H.lo·H.lo
We didn't notice since these comments were somewhat rare.
While here, "//" does not need to be followed by a space.
The code turns out to be pretty easy with strings.Cut.
Fixes#672.
We were noticing sporadic `go test` failures where running
an obfuscated binary could panic with:
fatal error: invalid function symbol table
It turns out that this could happen when modinfo.txtar runs first;
when it patched the linker with `git -C apply`, its current directory
had a valid git repository initialized, and apparently that somehow
prevents the patches from being applied.
It's unclear whether this is git's fault or our own, but in any case,
using a temporary working directory is easier and fixes the bug.
Do the same for `go build`, just in case.
`go help build` now has -C, -cover, -coverpkg, and -pgo.
There are no new boolean flags, but note that -cover is now a common
build flag rather than just a test flag.
While here, sort the lists so that they are easier to skim for missing
items in the future.
Go master, the upcoming Go 1.21, has had its merge window open for over
two weeks at this point, and it seems calmer at this point.
ALso update staticcheck to its latest release, which supports Go 1.20.
A user correctly points out that some sentences were confusing, like:
It can also make reverse-engineering harder, as an end user could
guess what version of Go or garble you're using.
Rewrite the whole section. It now also explains what I meant by that
sentence with a practical example.
I also merged the bits about "provide your own seed" and "rotating the
seeds", because they're both talking about the same mechanism.
Fixes#666.
This is not common, but it is done by a few projects.
Namely, github.com/goccy/go-json reached into reflect's guts,
which included a number of methods:
internal/runtime/rtype.go
11://go:linkname rtype_Align reflect.(*rtype).Align
19://go:linkname rtype_FieldAlign reflect.(*rtype).FieldAlign
27://go:linkname rtype_Method reflect.(*rtype).Method
35://go:linkname rtype_MethodByName reflect.(*rtype).MethodByName
[...]
Add tests for such go:linkname directives pointing at methods.
Note that there are two possible symbol string variants;
"pkg/path.(*Receiver).method" for methods with pointer receivers,
and "pkg/path.Receiver.method" for the rest.
We can't assume that the presence of two dots means a method either.
For example, a package path may be "pkg/path.with.dots",
and so "pkg/path.with.dots.SomeFunc" is the function "SomeFunc"
rather than the method "SomeFunc" on a type "dots".
To account for this ambiguity, rather than splitting on the last dot
like we used to, try to find a package path prefix by splitting on an
increasing number of first dots.
This can in theory still be ambiguous. For example,
we could have the package "pkg/path" expose the method "foo.bar",
and the package "pkg/path.foo" expose the func "bar".
Then, the symbol string "pkg/path.foo.bar" could mean either of them.
However, this seems extremely unlikely to happen in practice,
and I'm not sure that Go's toolchain would support it either.
I also noticed that goccy/go-json still failed to build after the fix.
The reason was that the type reflect.rtype wasn't being obfuscated.
We could, and likely should, teach our assembly and linkname
transformers about which names we chose not to obfuscate due to the use
of reflection. However, in this particular case, reflect's own types
can be obfuscated safely, so just do that.
Fixes#656.
Declare the ast.GenDecl node once,
which allows us to deduplicate and inline code.
resolveImportSpec doesn't need to be separate either.
We also don't need explicit panics for unexpected nils;
we can rely on nil checks inserted by the compiler.
Finally, move the bulk of the code outside of the scope.Names loop,
which unindents most of the logic.
While here, add a few more inline docs.
garble's -literals flag and its patching of the runtime may leave unused imports.
We used to try to detect those and remove the imports,
but that was still buggy with edge cases like dot imports or renamed imports.
Moreover, it was potentially incorrect.
Completely removing an import from a package means we don't run its init funcs,
which could have side effects changing the behavior of a program.
As an example, database/sql drivers are registered at init time.
Instead, for each import in an obfuscated Go file,
add an unnamed declaration which references the imported package.
This may not be necessary for all imported packages,
as only a minority become unused due to garble,
but it's also relatively harmless to do so.
Fixes#658.
We obfuscate import paths as well as their declared names.
The compiler treats some packages and APIs in special ways,
and the way it detects those is by looking at import paths and names.
In the past, we have avoided obfuscating some names like embed.FS or
reflect.Value.MethodByName for this reason. Otherwise,
go:embed or the linker's deadcode elimination might be broken.
This matching by path and name also happens with compiler intrinsics.
Intrinsics allow the compiler to rewrite some standard library calls
with small and efficient assembly, depending on the target GOARCH.
For example, math/bits.TrailingZeros32 gets replaced with ssa.OpCtz32,
which on amd64 may result in using the TZCNTL instruction.
We never noticed that we were breaking many of these intrinsics.
The intrinsics for funcs declared in the runtime and its dependencies
still worked properly, as we do not obfuscate those packages yet.
However, for other packages like math/bits and sync/atomic,
the intrinsics were being entirely disabled due to obfuscated names.
Skipping intrinsics is particularly bad for performance,
and it also leads to slightly larger binaries:
│ old │ new │
│ bin-B │ bin-B vs base │
Build-16 5.450Mi ± ∞ ¹ 5.333Mi ± ∞ ¹ -2.15% (p=0.029 n=4)
Finally, the main reason we noticed that intrinsics were broken
is that apparently GOARCH=mips fails to link without them,
as some symbols end up being not defined at all.
This patch fixes builds for the MIPS family of architectures.
Rather than building and linking all of std for every GOARCH,
test that intrinsics work by asking the compiler to print which
intrinsics are being applied, and checking that math/bits gets them.
This fix is relatively unfortunate, as it means we stop obfuscating
about 120 function names and a handful of package paths.
However, fixing builds and intrinsics is much more important.
We can figure out better ways to deal with intrinsics in the future.
Fixes#646.
tip is now the start of Go 1.21 as of a couple of days ago.
However, the first week or two is when the biggest changes land,
which means that Go tip is far more prone to bugs and changes that might
break garble.
We'll start tracking tip again in a few weeks, once the dust has settled
and we can look at what changes might have broken garble.
Only string literals over 8 characters in length are now being
obfuscated. This leads to around 20% smaller binaries when building with
-literals.
Fixes#618
Go's package runtime/internal/atomic contains references to functions
which refer to the current package by its package name alone:
src/runtime/internal/atomic/atomic_loong64.s: JMP atomic·Load(SB)
src/runtime/internal/atomic/atomic_loong64.s: JMP atomic·Load64(SB)
src/runtime/internal/atomic/atomic_loong64.s: JMP atomic·Load64(SB)
src/runtime/internal/atomic/atomic_mips64x.s: JMP atomic·Load(SB)
src/runtime/internal/atomic/atomic_mips64x.s: JMP atomic·Load64(SB)
src/runtime/internal/atomic/atomic_mips64x.s: JMP atomic·Load64(SB)
We could only handle unqualified or fully qualified references, like:
JMP ·Load64(SB)
JMP runtime∕internal∕atomic·Load64(SB)
Apparently, all three forms are equally valid.
Add a test case and fix it.
I checked whether referencing an imported package by its name worked;
it does not seem to be the case.
This feature appears to be restricted to the current package alone.
While here, we only need goPkgPath when we need to call listPackage.
Fixes#619.
To detect if an object is global, our previous code did:
obj.Pkg().Scope().Lookup(obj.Name()) == obj
However, that Lookup call is unnecessary.
It is a map lookup, which is cheap, but not free.
Instead, we can compare the scopes directly:
obj.Pkg().Scope() == obj.Parent()
While here, reuse the calls to obj.Pkg(),
and avoid using fmt.Sprintf in this hot path.
Looked here since perf on Linux with a garble build
reports that we spend about 2.6% of CPU time in recordedObjectString.
Overall provides a very minor improvement:
name old time/op new time/op delta
Build-16 21.8s ± 1% 21.7s ± 0% ~ (p=0.200 n=4+4)
name old bin-B new bin-B delta
Build-16 5.67M ± 0% 5.67M ± 0% ~ (all equal)
name old cached-time/op new cached-time/op delta
Build-16 734ms ± 3% 722ms ± 3% ~ (p=0.486 n=4+4)
name old mallocs/op new mallocs/op delta
Build-16 25.0M ± 0% 24.7M ± 0% -1.28% (p=0.029 n=4+4)
name old sys-time/op new sys-time/op delta
Build-16 17.0s ± 1% 16.8s ± 1% ~ (p=0.200 n=4+4)
Daniel Martí
pagran
lu4p