Commit Graph

623 Commits (126618a0d5971456c7bba5a6f4081d432d812083)
 

Author SHA1 Message Date
Daniel Martí 04b4ae390e update build and test flag list for Go 1.16
Go 1.16 just added one flag, -overlay, as documented in
https://golang.org/doc/go1.16#go-command.

We forgot to update these tables for 1.16, though it probably doesn't
matter as this flag is pretty much only used by gopls.
3 years ago
Daniel Martí d338be9941 give a usage example for combining flags
Users are still filing bugs about this, presumably because the docs
aren't clear enough.

Fixes #372.
3 years ago
Daniel Martí c2a75d09d5 test against a newer tip
We're just a couple of weeks away from the final release, so
double-check that we still work well with tip.

Update x/tools as well, as it has had some minor fixes for Go 1.17.
3 years ago
Daniel Martí c9e15e18cf make the positioning of Go flags clearer in the help text
We did say "garble flags before the command" towards the end,
but let's also make that clearer at the top.

For #342.
3 years ago
lu4p 3ab59000f3 Follow up: Obfuscate more byte slice literals 3 years ago
Daniel Martí ec0bdc4012 keep cgo-exported Go names non-obfuscated
Otherwise, the added test case would fail, as we don't modify the C code
and so there would be a name mismatch.

In the far future we might start modifying Go names in C code,
similar to what we did for Go assembly,
but right now that seems out of scope and too complex.

An easier fix is to simply record those (hopefully few) names in ignoreObjects.

While at it, recordReflectArgs started to really outgrow its name, as it
also collected expressions used as constants for literal obfuscation.
Give it a better name.

Fixes #366.
3 years ago
Daniel Martí 5f8bae06b7 small improvements towards obfuscating the runtime
I spent a couple of days trying to obfuscate all of std.
Ultimately I failed at making it fully work,
especially when it comes to the runtime package,
but I did fix a few problems along the way, as seen here.

First, fix the TODO to allow handleDirectives and transformGo to run on
runtime packages as well, if they are considered private. Note that this
is never true right now, but it matters once we remove runtimeRelated.

Second, modify parsedebugvars in a way that doesn't break typechecking.
We can remove AST nodes or even modify them in simple ways,
but if we add new AST nodes after typechecking,
those will lack type information.

We were replacing the entire body, running into that problem.
Instead, carefully cut the body to set some defaults,
but remove everything from the point GODEBUG is read.

Finally, add commented-out debug prints of transformed asm files.

For #193.
3 years ago
Daniel Martí 691a44cecb avoid breaking const declarations using iotas
With the -literals flag, we try to convert some const declarations to
vars, as long as that doesn't break typechecking. We really only do that
for typed constant strings, really.

There was a quirk: if a numerical constant had a type and used iota, we
would not obfuscate its value, but we would still convert the
declaration from const to var. Since iotas only work within const
declarations, that would break compilation:

	> garble -literals build
	[stderr]
	# test/main
	FeWE3zwi.go:19: undefined: iota
	exit status 2

To fix the problem, make the logic more conservative: only obfuscate
constant declarations where the values are typed strings, meaning that
any numerical constants are left entirely untouched.

This fixes the build of google.golang.org/protobuf/runtime/protoiface
with -literals turned on.
3 years ago
lu4p c1672cdc0d Obfuscate more byte slice literals
Slices with hex, octal, binary and rune elements are now obfuscated.
3 years ago
lu4p 552a6bcfb0 Obfuscate literals in string slices and arrays
Fixes #354
3 years ago
Daniel Martí 5e3cdf89a8 update support for Go 1.17 in time for beta1
Back in early April we added initial support for Go 1.17,
working on a commit from master at that time. For that to work, we just
needed to add a couple of packages to runtimeRelated and tweak printFile
a bit to not break the new "//go:build" directives.

A significant amount of changes have landed since, though, and the tests
broke in multiple ways.

Most notably, the new register ABI is enabled by default for GOOS=amd64.
That affected garble indirectly in two ways: there's a new internal
package to add to runtimeRelated, and we must make reverse.txt more
clever in making its output constant across ABIs.

Another noticeable change is that Go 1.17 changes how its own version is
injected into the runtime package. It used to be via a constant in
runtime/internal/sys, such as:

	const TheVersion = `devel ...`

Since we couldn't override such constants via the linker's -X flag,
we had to directly alter the declaration while compiling.

Thankfully, Go 1.17 simply uses a "var buildVersion string" in the
runtime package, and its value is injected by the linker.
This means we can now override it with the linker's -X flag.

We make the code to alter TheVersion for Go 1.16 a bit more clever,
to not break the package when building with Go 1.17.

Finally, our hack to work around ambiguous TOOLEXEC_IMPORTPATH values
now only kicks in for non-test packages, since Go 1.17 includes our
upstream fix. Otherwise, some tests would end up with the ".test"
variant suffix added a second time:

	test/bar [test/bar.test] [test/bar [test/bar.test].test]

All the code to keep compatibility with Go 1.16.x remains in place.
We're still leaving TODOs to remind ourselves to remove it or simplify
it once we remove support for 1.16.x.

The 1.17 development freeze has already been in place for a month,
and beta1 is due to come this week, so it's unlikely that Go will change
in any considerable way at this point. Hence, we can say that support
for 1.17 is done.

Fixes #347.
3 years ago
Daniel Martí ff361f6b15 mod: update direct deps
Via: go get -t -u ./...
3 years ago
Daniel Martí 993c5fbbe8 finalize changelog for v0.3.0 3 years ago
Daniel Martí 1d31a139f5 support aliases as embedded fields in dependencies
Our recent work in fieldToAlias worked well when the embedded field
declaration (using an alias) was in the same package as the use of that
field. We would have the *ast.Ident for the field declaration, so
types.Info.Uses would give us the TypeName for the alias.

Unfortunately, if the declaration was in a dependency package, we did
not have that same *ast.Ident, as we weren't parsing the source code for
dependencies for type-checking. This resulted in us incorrectly
obfuscating the use of such an embedded field:

	> garble build
	[stderr]
	# test/main
	JtzmzxWf.go:4: unknown field 'ExternalForeignAlias' in struct literal of type _BdSNiEL.Vcs_smer

To fix this, look through the direct imports of the package defining the
field to find an alias under the exact same name. Not a foolproof
solution, as there's a TODO, but it should work for most cases.

Fixes the obfuscation of google.golang.org/grpc/internal/status, too.

Updates #349.
3 years ago
Daniel Martí 68b39e8195 fix a link issue when obfuscating complex cgo packages
The added test case, which is obfuscating and linking os/user, would fail
before this fix:

	> garble build
	[stderr]
	# test/main
	/usr/lib/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
	/usr/bin/ld: $WORK/.tmp/go-link-073246656/go.o: in function `Chz0Yfs2._cgo_cmalloc':
	go.go:(.text+0x993cc): undefined reference to `Chz0Yfs2.runtime_throw'
	/usr/bin/ld: $WORK/.tmp/go-link-073246656/go.o: in function `Chz0Yfs2.tDfhQ8uK':
	go.go:(.text+0x99801): undefined reference to `Chz0Yfs2._cgo_runtime_gostring'
	/usr/bin/ld: go.go:(.text+0x9982a): undefined reference to `Chz0Yfs2._cgo_runtime_gostring'
	/usr/bin/ld: go.go:(.text+0x99853): undefined reference to `Chz0Yfs2._cgo_runtime_gostring'
	collect2: error: ld returned 1 exit status

The reason is that we would alter the linkname directives of cgo-generated
code, but we would not obfuscate the code itself at all.

The generated code would end up being transformed into:

	//go:linkname zh_oKZIy runtime.throw
	func runtime_throw(string)

One can clearly see the error there; handleDirectives obfuscated the
local linkname name, but since transformGo didn't run, the actual Go
declaration was not obfuscated in the same way. Thus, the linker fails
to find a function body for runtime_throw, and fails.

The solution is simple: handleDirectives assumes that it's running on
code being obfuscated, so only run it when transformGo is running.

We can also remove the cgo skip check in handleDirectives, as it never
runs on cgo-generated code now.

Fixes a number of build errors that have been noticed since
907aebd770.
3 years ago
Daniel Martí 64317883c9 handle aliases to foreign named types properly
When such an alias name was used to define an embedded field, we handled
that case gracefully via the code using:

	tf.info.Uses[node].(*types.TypeName)

Unfortunately, when the same field name was used elsewhere, such as a
composite literal, tf.Info.Uses gave us a *types.Var, not a
*types.TypeName, meaning we could no longer tell if this was an alias,
or what it pointed to.

Thus, we failed to obfuscate the name properly in the added test case:

	> garble build
	[stderr]
	# test/main/sub
	xxWZf66u.go:36: unknown field 'foreignAlias' in struct literal of type smhWelwn

It doesn't seem like any of the go/types APIs allows us to obtain the
*types.TypeName directly in this scenario. Thus, use a trick that we
used before: after typechecking, but before obfuscating, record all
embedded struct field *types.Var which are aliases via a map, where the
value holds the *types.TypeName for the alias.

Updates #349.
3 years ago
Daniel Martí e2ddce75a7 support embedding via embed.FS
We already added support for "//go:embed" with string and []byte,
by not obfuscating the "embed" import path.

However, embed.FS was still failing:

	> garble build
	[stderr]
	# test/main
	:13: go:embed cannot apply to var of type embed.WtKNvwbN

The compiler detects the type by matching its name to exactly "embed.FS",
so don't obfuscate the name "FS" either.

While at it, ensure that the embed code behaves the same with "go build".

Updates #349.
3 years ago
Daniel Martí 680e5624e9 speed up tests by 20-30% by using GOGC=off
See the added comment for the rationale. For that same reason, I always
build Go itself via "GOGC=off ./make.bash", as it's noticeably faster.

Before this change:

	$ go clean -cache && go test -short
	PASS
	ok      mvdan.cc/garble 35.298s
	$ go test -short
	PASS
	ok      mvdan.cc/garble 2.703s

With the change:

	$ go clean -cache && go test -short
	PASS
	ok      mvdan.cc/garble 25.323s
	$ go test -short
	PASS
	ok      mvdan.cc/garble 2.469s

Incremental test runs with a warm cache are largely unaffected, as those
would run very few of those short-lived and allocation-heavy programs.

However, when the build cache isn't warm (such as when garble itself is
modified), we easily see savings of 20-30%.

We might revisit this in the future if Go's GC gets better in these
situations, which should make "go build" faster. For now, we run our
tests very often, so having them burn a bit less CPU is nice.
3 years ago
Daniel Martí 7fc424ca26 make "garble command -h" give command-specific help
Before, "garble build -h" would print the same as "garble -h", which is
too much and confusing, as it doesn't tell us much about "build".

Now it's far better, and includes the output of "go build -h":

	$ garble build -h
	usage: garble [garble flags] build [arguments]

	This command wraps "go build". Below is its help:

	usage: go build [-o output] [build flags] [packages]
	Run 'go help build' for details.

We do the same for "garble reverse -h", since it doesn't wrap a Go tool
command.
3 years ago
Daniel Martí 750340db5b testdata: deduplicate GODEBUG cleanup line
I added a second one recently without noticing, since the original was
too far below and I had written commands above it.

Deduplicate them, and put the new env line as early as possible, to
prevent further issues and confusion.
3 years ago
Daniel Martí aba3b36218 start writing the upcoming changelog 3 years ago
Daniel Martí 5fa4acf580 testdata: use longer Go filenames for binsubstr
Every now and then, a CI run would fail:

	FAIL: testdata/scripts/reflect.txt:7: unexpected match for ["main.go"] in main

These were rare, and very hard to reproduce or debug.

My best guess is that, since "main.go" is a short string and we use
random eight-character obfuscated filenames ending with ".go", it was
possible that the random filename happened to end in "main" in some
cases.

Given the base64 encoding, the chances of a single suffix collision are
about 0.000006%. Note, however, that a single obfuscated build will most
likely obfuscate many filenames, especially for the tests obfuscating
multiple packages. For a single CI run with many tests across three OSs,
the chances of any collision are likely very low, but realistic.

All this has a simple fix: use longer filenames to match with. We choose
"garble_main.go" since it's long enough, but also because it's still
clear it's a "main" Go file, and it's very unlikely to cause conflicts
with filenames in upstream Go given the "garble_" prefix.
3 years ago
Daniel Martí 8edde922ee remove unused code spotted by -coverprofile
Remove some asthelper APIs that haven't been used for some time.
They can be recovered from the git history if needed again.

One type assertion in the literals package is always true.

Embedded field objects are handled near the top of transformGo, so the
extra !obj.Embedded() check was always true. Remove it.

We always obfuscate standalone funcs now, so the obfuscatedTypesPackage
check is no longer necessary. This was necessary when we used to not
obfuscate func names when they were used in linkname directives.

The workaround for test package imports in obfuscatedTypesPackage I had
to add a few commits ago no longer seems to be necessary. This might be
thanks to the simplification with functions in the paragraph just above.

It's impossible to run garble without -trimpath nowadays, as we error
before the build even starts:

	$ go build -toolexec=garble
	go tool compile: exit status 1
	cannot open shared file, this is most likely due to not running "garble [command]"

When run as "garble build", the trimpath flag is always set. So the
check in alterTrimpath never triggers anymore, and couldn't be tested.

Finally, simplify the handling of comment syntax in printFile, and add a
few TODOs for other code paths not covered by our existing tests.

Total code coverage is up from 90.3% to 91.0%.
3 years ago
Daniel Martí 65ff07875b obfuscate alias names like any other objects
Before this change, we would try to never obfuscate alias names. That
was far from ideal, as they can end up in field names via anonymous
fields.

Even then, we would sometimes still fail to build, because we would
inconsistently obfuscate alias names. For example, in the added test
case:

	--- FAIL: TestScripts/syntax (0.23s)
	    testscript.go:397:
	        > env GOPRIVATE='test/main,private.source'
	        > garble build
	        [stderr]
	        # test/main/sub
	        Lv_a8gRD.go:15: undefined: KCvSpxmQ

To fix this problem, we set obj to be the TypeName corresponding to the
alias when it is used as an embedded field. We can then make the right
choice when obfuscating the name.

Right now, all aliases will be obfuscated. A TODO exists about not
obfuscating alias names when they're used as embedded fields in a struct
type in the same package, and that package is used for reflection -
since then, the alias name ends up as the field name.

With these changes, the protobuf module now builds.
3 years ago
Daniel Martí 68f07389b2 fix a number of issues involving types from indirect imports
obfuscatedTypesPackage is used to figure out if a name in a dependency
package was obfuscated or not. For example, if that package used
reflection on a named type, it wasn't obfuscated, so we must have the
same information to not obfuscate the same name downstream.

obfuscatedTypesPackage could return nil if the package was indirectly
imported, though. This can happen if a direct import has a function that
returns an indirect type, or if a direct import exposes a name that's a
type alias to an indirect type.

We sort of dealt with this in two pieces of code by checking for
obfPkg!=nil, but a third one did not have this check and caused a panic
in the added test case:

	--- FAIL: TestScripts/reflect (0.81s)
	    testscript.go:397:
	        > env GOPRIVATE=test/main
	        > garble build
	        [stderr]
	        # test/main
	        panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	        	panic: runtime error: invalid memory address or nil pointer dereference
	        [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8a5e39]

More importantly though, the nil check only avoids panics. It doesn't
fix the root cause of the problem: that importcfg does not contain
indirectly imported packages. The added test case would still fail, as
we would obfuscate a type in the main package, but not in the indirectly
imported package where the type is defined.

To fix this, resurrect a bit of code from earlier garble versions, which
uses "go list -toolexec=garble" to fetch a package's export file. This
lets us fill the indirect import gaps in importcfg, working around the
problem entirely.

This solution is still not particularly great, so we add a TODO about
possibly rethinking this in the future. It does add some overhead and
complexity, though thankfully indirect imports should be uncommon.

This fixes a few panics while building the protobuf module.
3 years ago
Daniel Martí 654841e1fb skip reflection detection for sibling packages
Our allPkgs boolean logic was wrong, because it could still lead to
garble obfuscating a type when used in the main package, but not in its
defining package. The added test case shows such a case.

To fix that, use a package path to only record the named objects from
the target package, which is a narrower operation without this problem,
but still makes all our tests pass.

This makes the google.golang.org/protobuf/internal/filetype package
start building.
3 years ago
Daniel Martí 4156f35570 remove tinyfmt implementation from a test script
We've had proper build caching for a while now. No need to avoid std
imports like fmt.
3 years ago
Daniel Martí adb4f44fb2 reverse lone filenames as well
I've wanted this to more easily debug build failures.

To not force a build failure in a test script, as that would require
some trickery to remain stable, we use runtime.Caller without printing
the line number. Before this patch, those filenames without line numbers
would not be reversed at all.
3 years ago
Daniel Martí b6dee63b32 improve handling of reflect on foreign unnamed types
If a package A imports package B, and uses reflect.TypeOf on an unnamed
struct type B.T (such as an alias), we don't want to record B.T's fields
as "do not obfuscate". This is for the same reason that we don't if B.T
is a named struct type: the detection only works for the package
defining the type, as otherwise it's inconsistent.

We failed to handle this case well, because we assumed all struct types
would be under some named type. This is not the case for type aliases.

Fortunately, struct fields are named, and as such they are objects.
Check their package too, just like we do for named types.

Fixes another build error when obfuscating the protobuf module.
We add a simplified version of the example above as a test case,
which originated from debugging the protobuf build failure.
3 years ago
Daniel Martí 2ef9386942 use an empty filename when re-parsing source files
In printFile, we print and re-parse the modified AST to be able to have
reliable position information.

The re-parsing step can fail if something goes very wrong, such as a bug
in -literals. It should generally not happen. However, in rare cases it
has happened, and it's confusing for the end user to see syntax errors
pointing at an existing file on disk, when the code doesn't align
- since we're on a modified copy.

To prevent such confusion, use an empty filename. Syntax errors will
still not be terribly helpful, but they should be extremely rare and
promptly fixed, so that's not a huge concern.

For that same reason, we can't really add a good test here. We could
perhaps add a test that forces garble to mess up the src slice in some
way, but that would be a weird test, and not particularly worth it.

Fixes #286.
3 years ago
Daniel Martí 8961e0a39a testdata: split reflection test cases into reflect.txt
The detection of reflection usage is tricky and there are plenty of edge
cases to test for. We definitely want one script for it, rather than
splitting those cases between other scripts like imports.txt and
syntax.txt.

Moreover, those two were rather generic and large, so this helps keep a
balance.
3 years ago
Daniel Martí d8de5a4306 avoid reproducibility issues with full rebuilds
We were using temporary filenames for modified Go and assembly files.
For example, an obfuscated "encoding/json/encode.go" would end up as:

	/tmp/garble-shared123/encode.go.456.go

where "123" and "456" are random numbers, usually longer.

This was usually fine for two reasons:

1) We would add "/tmp/garble-shared123/" to -trimpath, so the temporary
   directory and its random number would be invisible.

2) We would add "//line" directives to the source files, replacing
   the filename with obfuscated versions excluding any random number.

Unfortunately, this broke in multiple ways. Most notably, assembly files
do not have any line directives, and it's not clear that there's any
support for them. So the random number in their basename could end up in
the binary, breaking reproducibility.

Another issue is that the -trimpath addition described above was only
done for cmd/compile, not cmd/asm, so assembly filenames included the
randomized temporary directory.

To fix the issues above, the same "encoding/json/encode.go" would now
end up as:

	/tmp/garble-shared123/encoding/json/encode.go

Such a path is still unique even though the "456" random number is gone,
as import paths are unique within a single build.

This fixes issues with the base name of each file, so we no longer rely
on line directives as the only way to remove the second original random
number.

We still rely on -trimpath to get rid of the temporary directory in
filenames. To fix its problem with assembly files, also amend the
-trimpath flag when running the assembler tool.

Finally, add a test that reproducible builds still work when a full
rebuild is done. We choose goprivate.txt for such a test as its
stdimporter package imports a number of std packages, including uses of
assembly and cgo.

For the time being, we don't use such a "full rebuild" reproducibility
test in other test scripts, as this step is expensive, rebuilding many
packages from scratch.

This issue went unnoticed for over a year because such random numbers
"123" and "456" were created when a package was obfuscated, and that
only happened once per package version as long as the build cache was
kept intact.

When clearing the build cache, or forcing a rebuild with -a, one gets
new random numbers, and thus a different binary resulting from the same
build input. That's not something that most users would do regularly,
and our tests did not cover that edge case either, until now.

Fixes #328.
3 years ago
Daniel Martí 58c15aa680 testdata: scope GODEBUG to a single test case
I've tripped over this GODEBUG env var four or five times already.
Since it affects any Go program by making them print tons of runtime
information, it also affects "go env", breaking garble horribly.

To prevent further issues, unset the env var when done.
3 years ago
Damien Lespiau 5d9506772e
README: fix link to literal obfuscation
I noticed that link was broken - fix it.
3 years ago
Daniel Martí 140de22f87 update direct deps 3 years ago
Daniel Martí d34406d832 clarify the status of the TOOLEXEC_IMPORTPATH hack
Now that upstream has merged our fix, it will ship with 1.17.
We'll be able to remove this entire chunk of code soon enough.
3 years ago
Daniel Martí c9b0b07853 hash field names equally in all packages
Packages P1 and P2 can define identical struct types T1 and T2, and one
can convert from type T1 to T2 or vice versa.

The spec defines two identical struct types as:

	Two struct types are identical if they have the same sequence of
	fields, and if corresponding fields have the same names, and
	identical types, and identical tags. Non-exported field names
	from different packages are always different.

Unfortunately, garble broke this: since we obfuscated field names
differently depending on the package, cross-package conversions like the
case above would result in typechecking errors.

To fix this, implement Joe Tsai's idea: hash struct field names with the
string representation of the entire struct. This way, identical struct
types will have their field names obfuscated in the same way in all
packages across a build.

Note that we had to refactor "reverse" a bit to start using transformer,
since now it needs to keep track of struct types as well.

This failure was affecting the build of google.golang.org/protobuf,
since it makes regular use of cross-package struct conversions.

Note that the protobuf module still fails to build, but for other
reasons. The package that used to fail now succeeds, so the build gets a
bit further than before. #240 tracks adding relevant third-party Go
modules to CI, so we'll track the other remaining failures there.

Fixes #310.
3 years ago
Andrew LeFevre b3db7d6fa7 fix obfuscating linkname directives that where the package name contained a dot 3 years ago
Andrew LeFevre 65b4692dbc hopefully fix position test on Windows by not matching on 'main.go' 3 years ago
Andrew LeFevre 907aebd770 obfuscate local names in linkname directives
Previously, the local name part of linkname directives were never
obfuscated. Now that we can obfuscate function names in Go assembly,
that limitation is required no longer.

Fixes #309
3 years ago
Daniel Martí 596e4d6ef1 README: document the effect of -tiny on reverse
Fixes #292.
3 years ago
Daniel Martí b4fc735a1e fix windows/arm cross-build linking
Obfuscating some std packages for windows/arm triggered a bug; when
encountering a call to runtime·memmove, we'd hash "memmove" with the
current package's action ID.

This is wrong on two levels: First, we aren't obfuscating the runtime
package yet. And second, if we did, we would have to hash the symbol
appropriately, with that package's action ID.

For now, only hashing the local names does the trick. That's all that
the code currently supports, anyway.
3 years ago
Daniel Martí 6ac7dce4a0 remove obsolete TODO in the tiny.txt test
Since 1a8e32227f we have been setting line numbers on call sites, and
that line number is always the minimum value for -tiny: 1.

Still not as zero-overhead as the old mechanism which entirely deleted
line numbers, but a useless and small line number is still pretty good.
3 years ago
Daniel Martí 0150aa8bb0 support reversing field names
THey don't show up in stack traces, and if they show up in regular
program output, we should in theory not obfuscate those names via the
detection of reflection.

However, there's one relatively common scenario where obfuscated field
names can appear: in build error messages, when obfuscation fails a
build.
3 years ago
Daniel Martí 05d35350cf record types into ignoreObjects more reliably
Our previous logic only took care of fairly simple types, such as a
simple struct or a pointer to a struct. If we had a struct embedding
another struct, we'd fail to record the objects for the fields in the
inner struct, and that would lead to miscompilation:

	> garble build
	[stderr]
	# test/main
	LZmt64Nm.go:7: outer.InnerField undefined (type *CcUt1wkQ.EmbeddingOuter has no field or method InnerField)

To fix this issue, make the function that records all objects under a
types.Type smarter. Since it now does more than just dealing with
structs, it's also renamed.

Since the function now walks types properly, we get to remove the extra
ast.Inspect in recordReflectArgs, which is nice.

We also make it a method, to avoid the map parameter. A boolean
parameter is also added, since we need this feature to only look at the
current package when looking at reflect calls.

Finally, we add a test case, a simplified version of the original bug
report.

Fixes #315.
3 years ago
Daniel Martí 2fad0e1583 wrap types.Importer to canonicalize import paths
The docs for go/importer.ForCompiler say:

	The lookup function is called each time the resulting importer
	needs to resolve an import path. In this mode the importer can
	only be invoked with canonical import paths (not relative or
	absolute ones); it is assumed that the translation to canonical
	import paths is being done by the client of the importer.

We use a lookup func for two reasons: first, to support modules, and
second, to be able to use our information from "go list -json -export".

However, go/types does not canonicalize import paths before calling
ImportFrom. This is somewhat understandable; it doesn't know whether an
importer was created with a lookup func, and ImportFrom only requires
the input path to be canonicalized in that scenario. When the lookup
func is nil, the importer canonicalizes by itself via go/build.Import.

Before this change, the added crossbuild test would fail:

	> garble build net/http
	[stderr]
	# vendor/golang.org/x/crypto/chacha20
	typecheck error: /usr/lib/go/src/vendor/golang.org/x/crypto/chacha20/chacha_generic.go:10:2: could not import crypto/cipher (can't find import: "crypto/cipher")
	# vendor/golang.org/x/text/secure/bidirule
	typecheck error: /usr/lib/go/src/vendor/golang.org/x/text/secure/bidirule/bidirule.go:12:2: could not import errors (can't find import: "errors")
	# vendor/golang.org/x/crypto/cryptobyte
	typecheck error: /usr/lib/go/src/vendor/golang.org/x/crypto/cryptobyte/asn1.go:8:16: could not import encoding/asn1 (can't find import: "encoding/asn1")
	# vendor/golang.org/x/text/unicode/norm
	typecheck error: /usr/lib/go/src/vendor/golang.org/x/text/unicode/norm/composition.go:7:8: could not import unicode/utf8 (can't find import: "unicode/utf8")

This is because we'd fall back to importer.Default, which only knows how
to find packages in $GOROOT/pkg. Those are missing for cross-builds,
unsurprisingly, as those built archives end up in the build cache.

After this change, we properly support importing std-vendored packages,
so we can get rid of the importer.Default workaround. And, by extension,
cross-builds now work as well.

Note that, in the added test script, the full build of the binary fails,
as there seems to be some sort of linker problem:

	> garble build
	[stderr]
	# test/main
	d9rqJyxo.uoqIiDs5: relocation target runtime.os9A16A3 not defined

We leave that as a TODO for now, as this change is subtle enough as it
is.
3 years ago
lu4p 1a9fdb4e8e
Fix calls to linkname functions (#314) 3 years ago
Daniel Martí 3afc993266 use "go env -json" to collect env info all at once
In the worst case scenario, when GOPRIVATE isn't set at all, we would
run these three commands:

* "go env GOPRIVATE", to fetch GOPRIVATE itself
* "go list -m", for GOPRIVATE's fallback
* "go version", to check the version of Go being used

Now that we support Go 1.16 and later, all these three can be obtained
via "go env -json":

	$ go env -json GOPRIVATE GOMOD GOVERSION
	{
		"GOMOD": "/home/mvdan/src/garble/go.mod",
		"GOPRIVATE": "",
		"GOVERSION": "go1.16.3"
	}

Note that we don't get the module path directly, but we can use the
x/mod/modfile Go API to parse it from the GOMOD file cheaply.

Notably, this also simplifies our Go version checking logic, as now we
get just the version string without the "go version" prefix and
"GOOS/GOARCH" suffix we don't care about.

This makes our code a bit more maintainable and robust. When running a
short incremental build, we can also see a small speed-up, as saving two
"go" invocations can save a few milliseconds:

	name           old time/op       new time/op       delta
	Build/Cache-8        168ms ± 0%        166ms ± 1%  -1.26%  (p=0.009 n=6+6)

	name           old bin-B         new bin-B         delta
	Build/Cache-8        6.36M ± 0%        6.36M ± 0%  +0.12%  (p=0.002 n=6+6)

	name           old sys-time/op   new sys-time/op   delta
	Build/Cache-8        222ms ± 2%        219ms ± 3%    ~     (p=0.589 n=6+6)

	name           old user-time/op  new user-time/op  delta
	Build/Cache-8        857ms ± 1%        846ms ± 1%  -1.31%  (p=0.041 n=6+6)
3 years ago
Daniel Martí 24d5ff362c fix a regression involving imported linkname funcs
In ce2c45440a, we simplified the code a bit and removed one call to
obfuscatedTypesPackage.

Unfortunately, we introduced a regression; if an exported function is
linknamed to another symbol name, and it's called from an importer
package, we would have a build failure now:

	> garble build
	[stderr]
	# test/main
	ZiOACuw7.go:1: undefined: ODC0xN52.BaDqbhkj

This is because the imported package would not hash the original name,
via its ignoreObjects logic. And, since the importer package has no
access to that knowledge, it would hash the same name, and fail to find
it in the final build.

The regression happened because we used to have a types.Scope Lookup
that saved us in this scenario. Add the test, and re-add the Lookup,
this time only for this particular scenario with function names.

Thanks to Andrew LeFevre for reporting and describing the test case.

While at it, replace more uses of "garbled" to "obfuscated".
3 years ago
Daniel Martí ba4c46eb09 CHANGELOG: finish v0.2.0 draft
Mention two more bugfixes. Not the initial Go 1.17 support just yet,
because coincidentally it broke in the past 24h due to upstream changes.

Also tweak some of the other wording to be clearer.
3 years ago