On CI we test on go1.23.x and go1.24.x, so if we always upgrade
to the latest go1.24.x, that will cause garble to complain
when running on go1.23.x:
garble was built with "go1.23.7" and can't be used with the newer "go1.24.1"
Moreover, the test hard-coded go1.24.1, which is currently the latest
go1.24.x but will not be for long, so this test was brittle.
We call `go list` to collect information about all the packages
to obfuscate and build, which is crucial to be able to perform
obfuscation of names used across packages.
However, when GOTOOLCHAIN causes a toolchain upgrade,
we must ensure that we use the upgraded Go tool;
otherwise we are mixing information from different toolchain versions.
Fixes#934.
When we build the patched cmd/link binary for use by garble,
we perform this build in a temporary directory so that the Go module
from the user does not get in the way.
When the user module made us upgrade the toolchain per GOTOOLCHAIN,
leaving that module's directory stops upgrading the toolchain,
so we patch a newer toolchain and build it with an older toolchain.
This is largely harmless, but it makes the newer toolchain think
it is actually an older toolchain, which leads to those pesky
"linker object header mismatch" version errors.
Updates #934.
The diffstat for go_std_tables.go shows that we were missing
more than two dozen new intrinsic functions from Go 1.24,
which could lead to the intrinsification done by the toolchain
to no longer work and leave programs with slower generic functions.
debugdir.txtar also needed tweaking as runtime/map.go is gone
starting in Go 1.24.
Finally, modinfo.txtar needed tweaking since Go 1.24 started stamping
Go binaries with VCS-derived module versions, so we no longer end up
with empty "(devel)" versions.
We only did this for Container in the type switch, but not for Struct.
The added test case panics otherwise.
Just like in the previous case, we still don't need to recurse
into type parameters for fieldToStruct to be filled correctly.
Fixes#899
As spotted by the protobuf package via check-third-party.sh,
the two structs below are identical:
type alias1 = int64
type Struct1 struct { Alias alias1 }
type alias2 = int64
type Struct2 struct { Alias alias2 }
Our previous approach with stripStructTags dealt with struct tags,
but it did not deal with aliases, which are now present in go/types
thanks to the new alias tracking.
The new approach properly ignores struct tags and unaliases any
type aliases, resulting in correct hashes of any type.
See the two "NOTE" comments below. In summary:
* struct field tags are not hashed
* named types are hashed by name rather than by pointer
We are keeping an eye on https://go.dev/issue/69420 as well.
Allows us to remove EmbeddedAliasFields, recordedObjectString,
and all the logic around using them.
Resolves an issue where a user was running into a panic in
our logic to record embedded aliases.
Note that this means we require Go 1.23.5 or later now,
which also meant some changes to goversion.txtar to keep it green.
Fixes#827.
obfuscatedImportPath already handled ToObfuscate, so the callers
don't have to do that as well. Handle main packages too, whose logic
was sprinkled and repeated throughout the project.
Reflection can show package names alone via reflect.Type.String,
and I'm sure they are available in other ways.
We were obfuscating "package p" exactly like the import path "foo.com/p"
which worked OK for the most part, but not for reflection.
This is also a slight improvement to the quality of the obfuscation,
given that package names and import paths will no longer be aligned
when obfuscated.
There's no need to reach for sharedCache.ListedPackages
when the caller is computePkgCache, which already has the lpkg value.
While here, compare *go/types.Package pointers directly
rather than via their import path strings.
This function returns obfuscated names, so use that as its name.
Moreover, some of the callers still called the result "objStr",
which misled me into thinking the string was a unique object path.
Leave a TODO behind about using go/types/objectpath too.
Our own recordedObjectString is sort of similar, but not as principled.
The list flags are entirely unused, so they can be omitted.
The only argument that matters is the package argument to load.
While here, update the TODO, as it no longer applies.
No changes to the values. However, to avoid noise with every bugfix
release changing all the lines, only use go1.X rather than go1.X.Y
to annotate each of the entries in maps and lists.
Now that we only use the list to create a replacer at init time,
we no longer need to spend extra effort sorting by length first.
The benchmark shows no measurable difference in performance.
This way, rather than using a double loop quadratic algorithm
to search for each name to replace in a string,
we can make use of the reasonably efficient generic replacer
which makes use of tries.
Copying some code from the strings package is not ideal,
but it beats having to re-implement such an algorithm ourselves.
Not only is the algorithm much faster, as we are no longer quadratic,
but the replacer also appends into a buffer to avoid repeated string
copies, which means we allocate fewer bytes per operation.
│ old │ new │
│ sec/op │ sec/op vs base │
AbiOriginalNames-8 135708.0n ± 0% 391.1n ± 5% -99.71% (p=0.001 n=7)
│ old │ new │
│ B/s │ B/s vs base │
AbiOriginalNames-8 2.565Mi ± 0% 890.112Mi ± 4% +34597.03% (p=0.001 n=7)
│ old │ new │
│ B/op │ B/op vs base │
AbiOriginalNames-8 5464.0 ± 0% 848.0 ± 0% -84.48% (p=0.001 n=7)
│ old │ new │
│ allocs/op │ allocs/op vs base │
AbiOriginalNames-8 18.00 ± 0% 16.00 ± 0% -11.11% (p=0.001 n=7)
As Paul Scheduikat points out, the loop already does not start
if the length of name is less than minHashLength.
│ old │ new │
│ sec/op │ sec/op vs base │
AbiOriginalNames-8 135.6µ ± 1% 135.5µ ± 0% ~ (p=1.000 n=7)
Use the term "original name" rather than "real name" for the code
as it is clearer that we mean the unobfuscated name.
Update the comments at the top of the file to be clearer with the
explanation of what kinds of inputs we can expect.
While here, use fmt.Appendf to simplify the generation code a bit.
This lets us only check names up to the remaining input string length.
│ old │ new │
│ sec/op │ sec/op vs base │
AbiRealName-8 196.7µ ± 1% 172.3µ ± 1% -12.41% (p=0.001 n=7)
│ old │ new │
│ B/s │ B/s vs base │
AbiRealName-8 1.774Mi ± 1% 2.022Mi ± 0% +13.98% (p=0.001 n=7)
│ old │ new │
│ B/op │ B/op vs base │
AbiRealName-8 5.359Ki ± 0% 5.336Ki ± 0% -0.44% (p=0.001 n=7)
│ old │ new │
│ allocs/op │ allocs/op vs base │
AbiRealName-8 19.00 ± 0% 18.00 ± 0% -5.26% (p=0.001 n=7)
Iterating over a map is much more expensive than iterating over a slice,
given how it needs to work out which keys are present in each bucket
and then randomize the order in which to navigate the keys.
None of this work needs to happen when iterating over a slice.
A map would be nice if we were to actually do map lookups, but we don't.
│ old │ new │
│ sec/op │ sec/op vs base │
AbiRealName-8 707.1µ ± 1% 196.7µ ± 1% -72.17% (p=0.001 n=7)
│ old │ new │
│ B/s │ B/s vs base │
AbiRealName-8 517.6Ki ± 2% 1816.4Ki ± 1% +250.94% (p=0.001 n=7)
│ old │ new │
│ B/op │ B/op vs base │
AbiRealName-8 5.362Ki ± 0% 5.359Ki ± 0% -0.05% (p=0.001 n=7)
│ old │ new │
│ allocs/op │ allocs/op vs base │
AbiRealName-8 19.00 ± 0% 19.00 ± 0% ~ (p=1.000 n=7) ¹
Go code can retrieve and use field and method names via the `reflect` package.
For that reason, historically we did not obfuscate names of fields and methods
underneath types that we detected as used for reflection, via e.g. `reflect.TypeOf`.
However, that caused a number of issues. Since we obfuscate and build one package
at a time, we could only detect when types were used for reflection in their own package
or in upstream packages. Use of reflection in downstream packages would be detected
too late, causing one package to obfuscate the names and the other not to, leading to a build failure.
A different approach is implemented here. All names are obfuscated now, but we collect
those types used for reflection, and at the end of a build in `package main`,
we inject a function into the runtime's `internal/abi` package to reverse the obfuscation
for those names which can be used for reflection.
This does mean that the obfuscation for these names is very weak, as the binary
contains a one-to-one mapping to their original names, but they cannot be obfuscated
without breaking too many Go packages out in the wild. There is also some amount
of overhead in `internal/abi` due to this, but we aim to make the overhead insignificant.
Fixes#884, #799, #817, #881, #858, #843, #842Closes#406
We used it to detect GOOS-specific packages and ignore their load errors
without having to do a substring search.
However, it turns out that repeatedly loading the string slice
from gob files in the cache is rather slow, particularly since many
Go packages have dozens of GOOS-specific files which can be ignored.
│ old │ new │
│ cached-sec/op │ cached-sec/op vs base │
Build-8 340.3m ± 1% 335.8m ± 2% -1.32% (p=0.002 n=10)
│ old │ new │
│ mallocs/op │ mallocs/op vs base │
Build-8 35.73M ± 0% 35.09M ± 0% -1.79% (p=0.000 n=10)
Daniel Martí
Paul Scheduikat
Jamison Lahman