We first called the typecheck method, which starts filling cachedOutput
with information from the current package, and later we would load the
gob files for all dependencies via loadCachedOutputs.
This was a bit confusing; instead, load the cached gob files first,
and then do all the operations which fill information for curPkg.
Similarly, we were waiting until the very end of transformCompile to
write curPkg's cachedOutput gob file to the disk cache.
We can write the file at an earlier point, before we have obfuscated and
re-printed all Go files for the current package.
We can also write the file before other work like processImportCfg.
None of these changes should affect garble's behavior,
but they will make the cache redesign for #708 easier.
To inevstigate #721, I wrote this fuzzer to see if any particular
combination of string literals and literal obfuscators would result
in a broken program.
I didn't find anything, but I reckon this fuzzer can still be useful.
Go 1.21 already breaks one of the three patches.
The complexity of the patches will likely increase,
and we only ever need to actively maintain the highest set of patches,
so splitting by major version feels natural.
See the added comment and test, which failed before the fix when
the encoded certificate was decoded.
It took a while to find the culprit in asn1, but in hindsight it's easy:
case reflect.Slice:
if t.Elem().Kind() == reflect.Uint8 {
return false, TagOctetString, false, true
}
if strings.HasSuffix(t.Name(), "SET") {
return false, TagSet, true, true
}
return false, TagSequence, true, true
Fixes#711.
See the added comment.
An alternative would be to split up the work into more jobs,
but that doesn't feel necessary, and we still have a global limit
on how many free runners we can run jobs on at once.
While here, update the comments, and remove unnecessary step names.
It's perfectly fine for bench_literals.go to be a main package itself,
but the scripts directory doesn't need to be a Go package
This was bothering me at times; for example, "go list ./..." would
show the scripts directory, or "go install mvdan.cc/garble/..." would
build and install a "scripts" binary as well.
The file can still be built or run directly, like either of:
go build bench_literals.go
go run bench_literals.go
The first makes our test scripts more consistent, as all external
program executions happen via "exec" and are not as easily confused
with custom builtin commands like our "generate-literals".
The second catches mistakes if any of our txtar files have duplicate
files, where all but one of the contents would be ignored before.
A couple of new packages in runtimeAndDeps,
and go list's Package.DepsErrors may now include package build errors
which we want to ignore as we would print those as duplicates.
Changes literal obfuscation such that literals of any size will be obfuscated,
but beyond `maxSize` we only use the `simple` obfuscator.
This one seems to apply AND, OR, or XOR operators byte-wise and should be safe to use,
unlike some of the other obfuscators which are quadratic on the literal size or worse.
The test for literals is changed a bit to verify that obfuscation is applied.
The code written to the `extra_literals.go` file by the test helper now ensures
that Go does not optimize the literals away when we build the binary.
We also append a unique string to all literals so that we can test that
an unobfuscated build contains this string while an obfuscated build does not.
printOneCgoTraceback now returns a boolean rather than an int.
Since we need to have different logic based on the Go version,
and toolchainVersionSemver was only set for the main process,
move the string to the shared cache global.
This is a nice thing to do anyway, to reduce the number of globals.
While here, update actions/setup-go to v4, which starts caching
GOMODCACHE and GOCACHE by default now.
Disable it, because it still doesn't help in our case,
and GitHub's Actions caching is still really inefficient.
And update staticcheck too.
The seedFlag.random field had never worked,
as my refactor in December 2021 never set it to true.
Even if the boolean was working, we only printed the random seed
when we failed. It's still useful to see it when a build succeeds,
for example when wanting to reproduce the same binary
or when wanting to reverse a panic from the produced binary.
Add a test this time.
Fixes#696.
Similar to what testscript does, we can reuse the test binary by telling
TestMain to run the main function rather than the Go tests.
This saves a few hundred milliseconds out of each benchmark run.
https://go.dev/cl/466095 lightly refactored the runtime
in a way that broke extractNameOff. In particular, the code
func cfuncname(f funcInfo) *byte {
if !f.valid() || f.nameOff == 0 {
return nil
}
return &f.datap.funcnametab[f.nameOff]
}
func funcname(f funcInfo) string {
return gostringnocopy(cfuncname(f))
}
is now simply
func funcname(f funcInfo) string {
if !f.valid() {
return ""
}
return f.datap.funcName(f.nameOff)
}
Since extractNameOff looked for the func named cfuncname,
and looked for the nameOff selector inside an index expression,
all of that code no longer worked properly.
It all existed to find the name of the field, nameOff,
so that we would automatically adapt if upstream renames it.
Unsurprisingly, the code using the field got refactored first.
It doesn't seem like the extra code on our part is helping us,
and assuming the name of the field works for all Go versions,
so do that instead.
If upstream does rename the field in the future,
the obfuscated Go builds will start failing in an obvious way.
If or when that comes to pass, we can change our constant string.
Added in Go 1.19, types like sync/atomic.Uint64 are handy,
because they ensure proper alignment even on 32-bit GOOSes.
However, this was done via a magic `type align64 struct{}`,
which the compiler spotted by name.
To keep that magic working, do not obfuscate the name.
Neither package path was being obfuscated,
as both packages contain compiler intrinsics already.
Fixes#686.
I mistakenly understood that, when the DepsErrors field has errors,
the Error field would contain an error as well.
That is not always the case; for example,
the imports_missing package in the added test script
had DepsErrors set but Error empty, causing a nil dereference panic.
Make the code more robust, and report both kinds of load errors.
Fixes#694.
We're building the linker binary for the host GOOS,
not the target GOOS that we happen to be building for.
I noticed that, after running `go test`, my garble cache
would contain both link and link.exe, which made no sense
as I run linux and not windows.
`go env` has GOHOSTOS to mirror GOOS, but there is no
GOHOSTEXE to mirror GOEXE, so we reconstruct it from
runtime.GOOS, which is equivalent to GOHOSTOS.
Add a regression test as well.
When we use `go list` on the standard library, we need to be careful
about what flags are passed from the top-level build command,
because some flags are not going to be appropriate.
In particular, GOFLAGS=-modfile=... resulted in a failure,
reproduced via the GOFLAGS variable added to linker.txtar:
go: inconsistent vendoring in /home/mvdan/tip/src:
golang.org/x/crypto@v0.5.1-0.20230203195927-310bfa40f1e4: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
golang.org/x/net@v0.7.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
golang.org/x/sys@v0.5.1-0.20230208141308-4fee21c92339: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
golang.org/x/text@v0.7.1-0.20230207171107-30dadde3188b: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
To ignore the vendor directory, use -mod=readonly or -mod=mod.
To sync the vendor directory, run:
go mod vendor
To work around this problem, reset the -mod and -modfile flags when
calling "go list" on the standard library, as those are the only two
flags which alter how we load the main module in a build.
The code which builds a modified cmd/link has a similar problem;
it already reset GOOS and GOARCH, but it could similarly run into
problems if other env vars like GOFLAGS were set.
To be on the safe side, we also disable GOENV and GOEXPERIMENT,
which we borrow from Go's bootstrapping commands.
`go build ./...` does indeed compile and link main packages,
it just does not move the resulting binaries anywhere permanent
like `go install` does.
As such, the TODO isn't relevant; the fact that we build all packages
inside each module means we are already linking any binaries matched via
`./...` from the module root.
We don't run any of the binaries, which would catch panics at run-time,
but we already have a note at the top about using `garble test`.
The current garble release is able to obfuscate it with Go 1.20.
While here, re-generate all files to use "go 1.20" directives,
and add a TODO about also testing binary builds for each project.
See #600.
Per the inline comment, we want to build every package in the test suite
at least once, otherwise we won't get proper code coverage information.
For example, some code only triggers when obfuscating the runtime,
and if I run "go test" twice in a row, the second might reuse the
first's obfuscation of the runtime and skip the code entirely.
However, per the TODO, we used to solve this in a rather wasteful way,
by making each test use its own separate GOCACHE directory.
Instead, use a new GOCACHE directory, but share it between tests.
This is still enough, as we still build each package at least once.
Running the command below twice in a row with Go 1.20:
go test -cover
took about 1m55s on my laptop before, and now takes about 1m10s.
Not great, but a noticeable improvement.
Both report a total coverage of 88.7%.
While here, do the same for GARBLE_CACHE_DIR,
and use testscript.Env.Setenv for simplicity.
At linker stage, we now encrypt funcInfo.entryoff value with a simple algorithm (1 xor + 1 mul).
This makes it harder to relate function metadata (e.g. name) to function itself in binary, almost without affecting performance.
The test package hack still appears to be needed as of Go 1.20.
Interestingly, the cgo filename check does not trigger at all anymore.
Presumably this means that it's not needed at all, and obfuscating code
generated by cgo appears to be fine. Go with that for now.
Now that we dropped Go 1.19, we can use it again,
since the referenced bug was fixed in Go 1.20.
This is a separate commit, as this change does alter the way we
obfuscate Go builds, so it's not just a cleanup.
It assumes that reflect.Value has a field named "flag",
which wasn't the case with obfuscated builds as we obfuscated it.
We already treated the reflect package as special,
for instance when not obfuscating Method or MethodByName.
In a similar fashion, mark reflect's rtype and Value types to not be
obfuscated alongside their field names. Note that rtype is the
implementation behind the reflect.Type interface.
This fix is fairly manual and repetitive.
transformCompile, transformLinkname, and transformAsm should all
use the same mechanism to tell if names should be obfuscated.
However, they do not do that right now, and that refactor feels too
risky for a bugfix release. We add more TODOs instead.
We're not adding go-spew to scripts/check-third-party.sh since the
project is largely abandoned. It's not even a Go module yet.
The only broken bit from it is what we've added to our tests.
Fixes#676.
Even when setting git's current directory to a temporary directory,
it could find a git repository in a parent directory and still malfunction.
Use the --git-dir flag to ensure that walking doesn't happen at all.
While here, ensure that "git apply" is always applying our patches,
and add a regression test to linker.txtar when not testing with -short.
Go contains such inline comments, like:
crypto/aes/gcm_ppc64x.s:129: VPMSUMD IN, HL, XL // H.lo·H.lo
We didn't notice since these comments were somewhat rare.
While here, "//" does not need to be followed by a space.
The code turns out to be pretty easy with strings.Cut.
Fixes#672.
We were noticing sporadic `go test` failures where running
an obfuscated binary could panic with:
fatal error: invalid function symbol table
It turns out that this could happen when modinfo.txtar runs first;
when it patched the linker with `git -C apply`, its current directory
had a valid git repository initialized, and apparently that somehow
prevents the patches from being applied.
It's unclear whether this is git's fault or our own, but in any case,
using a temporary working directory is easier and fixes the bug.
Do the same for `go build`, just in case.
Daniel Martí
lu4p
Dominic Breuker
pagran