* Use latest Binject/debug version to support importmap directives in the importcfg file
* Uncomment line in goprivate testscript to test ImportMap
* Fixed issue where a package in specified in importmap would be hashed differently in a package that imported it, due to the mapping of import paths.
Also commented out the 'net' import in the goprivate testscript (again) due to cgo compile errors
We also update the "original types importer" to support ImportMap.
The test now gets further along, no longer getting stuck on "path not
found in listed packages". Instead, we get stuck on:
error parsing importcfg: <...>/importcfg:2: unknown directive "importmap"
This bug has been filed at https://github.com/Binject/debug/issues/17.
Until it's fixed, we can't really proceed on #146, so the net import in
the test file (which triggers this case) is commented out for now.
Updates #146.
That is, a package that is built without obfuscation imports an
obfuscated package. This will result in confusing compilation error
messages, because the importer can't find the exported names from the
imported package by their non-obfuscated names:
> ! garble build ./importer
[stderr]
# test/main/importer
importer/importer.go:5:9: undefined: imported.Name
exit status 2
Instead, detect this bad input case and provide a nice error:
public package "test/main/importer" can't depend on obfuscated package "test/main/imported" (matched via GOPRIVATE="test/main/imported")
For now, this is by design. It also makes little sense for a public
package to import an obfuscated package in general, because the public
package would have to leak details about the private package's API and
behavior.
While at it, fix a quirk where we thought the unsafe package could be
private. It can't be, because the runtime package is always public and
it imports the runtime package:
public package "internal/bytealg" can't depend on obfuscated package "unsafe" (matched via GOPRIVATE="*")
Instead of trying to obfuscate "unsafe" and doing nothing, simply add it
to the neverPrivate list, which is also a better name than
"privateBlacklist" (for #169).
Fixes#164.
Co-authored-by: lu4p <lu4p@pm.me>
Means that we no longer have to pass a dozen parameters around, mainly
to transformGo. We can also start documenting what each of the fields
actually does, and group them better.
While at it, pkgPath and pkgScope can both be replaced by a
*types.Package, since they're both accessible via trivially cheap
methods.
* fix bug where structs would get garbled in some packages but not in others
* only check if struct/field was not defined in current package
* fix a related bug when two objects share the same name in the same package and one is garbled but the other one is not
* renamed parameter for clarity
The test intended to use an extra module to be obfuscated, rsc.io/quote,
which we were bundling in the local proxy as well. Unfortunately, the
use of GOPRIVATE also meant that we did not actually fetch the module
from the proxy, and we would instead do a full roundtrip to the internet
to "git clone" the actual upstream repository.
To prevent that roundtrip, instead use a locally replaced module. This
fits the syntax.txt test too, since it's one more edge case that we want
to make sure works well with garble. Since rsc.io/quote is used in
another test, simply make up our own tiny module.
Reduces a 'go test -run Syntax/syntax' run with warm cache from ~5s to
~0.5s, thanks to removing the multiple roundtrips. A warm 'go test' run
still sits at ~6s, since we still need that much CPU time in total.
While at it, fix a staticcheck warning and fix inconsistent indentation
in a couple of tests.
As per the discussion in https://github.com/golang/go/issues/41145, it
turns out that we don't need special support for build caching in
-toolexec. We can simply modify the behavior of "[...]/compile -V=full"
and "[...]/link -V=full" so that they include garble's own version and
options in the printed build ID.
The part of the build ID that matters is the last, since it's the
"content ID" which is used to work out whether there is a need to redo
the action (build) or not. Since cmd/go parses the last word in the
output as "buildID=...", we simply add "+garble buildID=_/_/_/${hash}".
The slashes let us imitate a full binary build ID, but we assume that
the other components such as the action ID are not necessary, since the
only reader here is cmd/go and it only consumes the content ID.
The reported content ID includes the tool's original content ID,
garble's own content ID from the built binary, and the garble options
which modify how we obfuscate code. If any of the three changes, we
should use a different build cache key. GOPRIVATE also affects caching,
since a different GOPRIVATE value means that we might have to garble a
different set of packages.
Include tests, which mainly check that 'garble build -v' prints package
lines when we expect to always need to rebuild packages, and that it
prints nothing when we should be reusing the build cache even when the
built binary is missing.
After this change, 'go test' on Go 1.15.2 stabilizes at about 8s on my
machine, whereas it used to be at around 25s before.
What obfuscateImports did was valid, but unfortunately made the build
cache redo work. This is because we were modifying object files in-place
in the build cache, meaning that the Go tool would think it had to
re-compile those packages.
Instead, write the modified object files in a temporary directory, and
leave the input object files untouched. We require a bit of extra code
to keep track of this and adjust the link argument as well as its
importcfg file.
The function of obfuscateImports, as well as the reasoning above, is now
summarized in its godoc as well.
This should be the last change in preparation for proper build caching
support. Rebasing the build caching branch on this commit finally makes
caching work reliably every single time.
More correct comments transformation was implemented.
Added processing of //go:linkname localname [importpath.name] directive, now localname is not renamed. This is safe and does not cause a name disclosure because the functions marked //linkname do not have a name in the resulting binary.
Added cgo directives support
Fixed filename leak protection for cgo
Part of #149
Fix for bug when a conflict occurred between generated short names
and local variables/functions/types/structs.
The already existing names are collected and if the generated short name
already exists, the package counter is increased until a free name is found.
Part of #149.
The struct type for buildInfo doesn't need to be named. Plus, the
"packageInfo" name was actually pretty misleading, because buildInfo
contains data from many packages.
Add an importCfg field, so that we don't need to fetch the flag value
many times.
Simplify reading the importCfg file; we used to also write to it, but
that's no longer the case, so we can just use ioutil.ReadFile.
Finally, give the function that fills buildInfo a better name, a godoc,
and fix the origTypesConfig godoc.
We also add a TODO to reuse goobj.ParseImportCfg in the future.
We now store how we obfuscated unexported names in the object file
itself, not a separate file. This means that the data can survive in the
build cache, whereas the separate file was being lost. Luckily, we can
just add an extra header to the archive, and other programs like the Go
linker will just ignore it.
Give the func a name that tells what the return value means.
Add missing newlines to printfs, use consistent quoting, and replace
"%s" with %q.
Document the Go 1.15 date.
Finally, fix the imports via goimports.
Many files were missing copyright, so also add a short script to add the
missing lines with the current year, and run it.
The AUTHORS file is also self-explanatory. Contributors can add
themselves there, or we can simply update it from time to time via
git-shortlog.
Since we have two scripts now, set up a directory for them.
Rework the features section in the README, leaving optional features at
the end of the list. Simplify the caveats list, too; the build cache and
exported field/method bits only need one point each. Overall, the
section was far too wordy for little reason.
Also redo the help text a bit. There's now a line to briefly introduce
the tool, as well as a link to the README with all the details. Finally,
the flags have shorter and more consistent help strings.
While at it, remove two unused global vars as spotted by staticcheck.
Finally, finally this is done. This allows import paths to be obfuscated by modifying
object/archive files and garbling import paths contained within. The bulk of the
code that makes parsing and writing Go object/archive files possible lives at
https://github.com/Binject/debug/tree/master/goobj2, which I wrote as well.
I have tested by garbling and checking for import paths via strings and grep
(in order of difficulty) https://github.com/lu4p/binclude, garble itself, and
https://github.com/dominikh/go-tools/tree/master/cmd/staticcheck.
This only supports object/archive files produced from the Go 1.15 compiler.
The object file format changed at 1.15, and 1.14 and earlier is not supported.
Fixes#13.
Most notably, x/mod now includes the GOPRIVATE pattern-matching API we
were copying before, so we can use it directly.
Also bump the Go version requirement to 1.15, in preparation for the
import path obfuscation PR, and don't let the gotip job fail the entire
workflow.
Before this change, obfuscating any package using unsafe.Pointer and
with GOPRIVATE="*" would result in errors like:
undefined: unsafe.ZrMmYd1lg
This is because the type isn't plain Go; it's rather a special type that
gets special treatment from the typechecker and compiler:
type Pointer *ArbitraryType
So, trying to obfuscate the name "unsafe.Pointer" will never work,
because there isn't a real Go type definition we can obfuscate along
with that.
Updates, but does not yet fully fix, #108.
This is important, because it would mean that we would obfuscate
nothing. At best, it would be confusing; at worst, it could mislead
the user into thinking the binary is obfuscated.
Fixes#20.
Updates #108.
We need to call 'go env GOPRIVATE' instead of just using os.Getenv so
that we pick up the value from the new ${CONFIG}/go/env file, written by
'go env -w'.
However, we were calling 'go env' at every process start, including the
often tens or hundreds of compiler calls to build all the dependencies.
Instead, do that only once on the first 'garble build' process, and use
os.Setenv to pass that along to future garble sub-processes.
name old time/op new time/op delta
Build-8 1.81s ± 0% 1.74s ± 4% -3.78% (p=0.030 n=5+6)
name old sys-time/op new sys-time/op delta
Build-8 1.45s ± 2% 1.22s ± 1% -16.07% (p=0.002 n=6+6)
name old user-time/op new user-time/op delta
Build-8 10.9s ± 1% 10.6s ± 1% -2.82% (p=0.004 n=6+5)
Error strings should never be capitalized.
A binsubstr line in one of the tests was duplicate and thus useless.
Remove duplicate or trailing spaces in test scripts.
Finally, add a TODO for an optimization I just spotted.
Fixes #2.
Line numbers are now obfuscated, via `//line` comments.
Filenames are now obfuscated via `//line` comments, instead of changing the actual filename.
New flag `-tiny` to reduce the binary size, at the cost of reversibility.
Fixes#93.
The second typecheck lead to the creation of different type objects,
which didn't match the types in the blacklist anymore.
It turns out we don't need the second typecheck,
therfore it is now removed.
First, our original append line was completely ineffective; we never
used that "flags" slice again. Second, we only attempted to use the flag
when we obfuscated a package.
In fact, we never care about debugging information here, so for any
package we compile, we can add "-dwarf=false". At the moment, we compile
all packages, even if they aren't to be obfuscated, due to the lack of
access to the build cache.
As such, we save a significant amount of work. The numbers below were
obtained on a quiet machine with "go test -bench=. -benchtime=10x", six
times before and after the change.
name old time/op new time/op delta
Build-8 2.06s ± 4% 1.87s ± 2% -9.21% (p=0.002 n=6+6)
name old sys-time/op new sys-time/op delta
Build-8 1.51s ± 2% 1.46s ± 1% -3.12% (p=0.004 n=6+5)
name old user-time/op new user-time/op delta
Build-8 11.9s ± 2% 10.8s ± 1% -8.71% (p=0.002 n=6+6)
While at it, only do CI builds on pushes and PRs to the master branch,
so that my PRs created from the same repo don't trigger duplicate
builds.
If the flags list included ["-o" "binary"], we would properly skip "-o",
but we wouldn't skip "binary".
Thus, 'go list' would receive "binary" as the first argument, and assume
that's the first parameter and the end of the flags.
And add a unit test case.
Fixes#82, again.
Otherwise any build flags like -tags won't be used, and we might easily
end up with errors or incorrect packages.
The common case with -tags is covered by one of the integration test
scripts. On top of that, we add a table-driven unit test to cover all
edge cases, since there are many we can do quickly in a unit test.
Fixes#82.
We don't really care about tools other than "compile" and "link". Stop
trying to keep a complete list.
Use "if err := f(); err != nil {" where it makes sense.
Simplify some declarations, and use a better variable name than "fW".
Instead of doing a 'go list' call every time we need to fetch a
dependency's export file, we now do a single 'go list' call before the
build begins. With the '-deps' flag, it gives us all the dependency
packages recursively.
We store that data in the gob format in a temporary file, and share it
with the future garble sub-processes via an env var.
This required lazy parsing of flags for the 'build' and 'test' commands,
since now we need to run 'go list' with the same package pattern
arguments.
Fixes#63.
The following identifiers are now skipped,
because they never show up in the binary:
- constant identifiers
- identifiers of local variables
(includes function params and named returns)
- identifiers of local types
Implement a literal obfuscator interface,
to allow the easy addition of new encodings.
Add literal obfuscation for byte literals.
Choose a random obfuscator on literal obfuscation,
useful when multiple obfuscators are implemented.
Fixes#62
Injected functions were mistaken for functions implemented outside go.
Asm functions:
obj.Scope().Pos() == 0
obj.Scope().End() == 0
Injected functions:
obj.Scope().Pos() == 0
obj.Scope().End() == 1
We now check for the End instead of the Pos.
This requires a bit of extra magic to replace one constant in
runtime/internal/sys, but that was simple enough given that we can reuse
a lot of the code to parse the files and write them to a temporary dir.
We can also drop the -X flags, as runtime.buildVersion is based on the
constant that we replace here.
Fixes#44, again.
Generating DWARF in the compiler object files could take as much as 10%
extra CPU time, while we ignore it entirely at the link stage.
Speeds up 'go test -short' from ~19.5s to ~18.5s on my laptop.
Whilst it may not be particularly common, it is legal to embed fields
where the type has universe scope (e.g. int, error, etc). This can
cause a panic in 2 difference places:
- When embedding `error`, a named type is resolved but the package is
nil. The call to `pkg.Name()` results in a panic
- When embedding a basic type such as `int`, no named type is resolved
at all. The call to `namedType(obj.Type()).Obj()` results in a panic
I'm assuming it is OK to return early when a named type cannot be
resolved.. we could let it continue but I think `pkg` should be set to
nil to be correct, so it'd end up returning straight away anyway.
In the added test, the unexported field used to be garbled.
Reflection can only reach exported methods, exported fields, and
unexported fields. Exported methods and fields are currently never
garbled, so unexported fields was the only missing piece.
Avoiding a type switch for the entire node prevents an indentation
level.
We can obtain obj and pkg early, and return early as well if either is
uninteresting. That means less nil checks later on, which means even
less indentation and complexity.
Also remove the -toolexec equivalent, as it's becoming longer now that
we have GARBLE_DIR, and it might become out of date in the future again.
We don't want users to assume it will work forever.
Carefully select a default that will do the right thing when inside a
module, as well as when building ad-hoc packages.
This means we no longer need to look at the compiler's -std flag, which
is nice.
Also replace foo.com/ with test/, as per golang/go#37641.
Fixes#7.
Spotted while trying to link a program using unix.Syscall, since its
implementation is assembly.
Telling if a function couldn't be garbled isn't trivial. If that
function belongs to an imported package, we only load its export data
instead of type-checking from source, so we don't have all the
information needed.
Instead, use the gc export data importer to import two versions of each
dependency: its original version, for the initial type-checking, and its
garbled version, to check if any of its exported names weren't garbled.
Updates #9.
We use 'go list -json -export' to locate required modules. This works
fine to locate direct module dependencies; since we're building in the
current module, we run 'go list' in the correct directory.
However, if we're building one of those module dependencies, and it has
other module dependencies of its own, we would fail with cryptic errors
like:
typecheck error: [...] go list error: updates to go.sum needed, disabled by -mod=readonly
This is because we would try to run 'go list' outside of the main
module, probably inside the module cache. Instead, use a $GARBLE_DIR env
var from the top-level 'garble build' call to always run 'go list' in
the original directory.
We add a few small modules to properly test this.
Updates #9.
In the added test case, we'd see a failure, since we garbled the name of
the "Embedded" type but not its use as an anonymous field. Garble both.
This might possibly break some reflect code, but it doesn't seem like we
have an option. When we garble a type, it's impossible to tell if it's
going to be used as an anonymous field later.
Updates #9.
This is important, because "-std -foo" and "-buildid -foo" are entirely
different cases. The first is equivalent to "-std=true -foo" since the
flag is boolean, but the second is equivalent to "-buildid=-foo" since
the flag isn't boolean.
We can keep track of which of the flags we're interested in are boolean,
which isn't much extra work. Also add unit tests; the build ID is a
hash, so it's very hard to write an end-to-end test that reliably has an
ID starting with a dash.
'go tool compile' receives the package path via the -p flag. This is
better than making up one.
We have to be careful with "-p main" though, as that's not part of the
standard library.
Andrew LeFevre
Daniel Martí
pagran
lu4p
pagran
Zachary Wasserman
Nicholas Jones