Many files were missing copyright, so also add a short script to add the
missing lines with the current year, and run it.
The AUTHORS file is also self-explanatory. Contributors can add
themselves there, or we can simply update it from time to time via
git-shortlog.
Since we have two scripts now, set up a directory for them.
Rework the features section in the README, leaving optional features at
the end of the list. Simplify the caveats list, too; the build cache and
exported field/method bits only need one point each. Overall, the
section was far too wordy for little reason.
Also redo the help text a bit. There's now a line to briefly introduce
the tool, as well as a link to the README with all the details. Finally,
the flags have shorter and more consistent help strings.
While at it, remove two unused global vars as spotted by staticcheck.
Finally, finally this is done. This allows import paths to be obfuscated by modifying
object/archive files and garbling import paths contained within. The bulk of the
code that makes parsing and writing Go object/archive files possible lives at
https://github.com/Binject/debug/tree/master/goobj2, which I wrote as well.
I have tested by garbling and checking for import paths via strings and grep
(in order of difficulty) https://github.com/lu4p/binclude, garble itself, and
https://github.com/dominikh/go-tools/tree/master/cmd/staticcheck.
This only supports object/archive files produced from the Go 1.15 compiler.
The object file format changed at 1.15, and 1.14 and earlier is not supported.
Fixes#13.
Most notably, x/mod now includes the GOPRIVATE pattern-matching API we
were copying before, so we can use it directly.
Also bump the Go version requirement to 1.15, in preparation for the
import path obfuscation PR, and don't let the gotip job fail the entire
workflow.
Before this change, obfuscating any package using unsafe.Pointer and
with GOPRIVATE="*" would result in errors like:
undefined: unsafe.ZrMmYd1lg
This is because the type isn't plain Go; it's rather a special type that
gets special treatment from the typechecker and compiler:
type Pointer *ArbitraryType
So, trying to obfuscate the name "unsafe.Pointer" will never work,
because there isn't a real Go type definition we can obfuscate along
with that.
Updates, but does not yet fully fix, #108.
This is important, because it would mean that we would obfuscate
nothing. At best, it would be confusing; at worst, it could mislead
the user into thinking the binary is obfuscated.
Fixes#20.
Updates #108.
This shouldn't break often, so it doesn't need to be covered by 'go test
-short'. Moreover, it's still a relatively expensive step, since we end
up reaching package compilation.
basic.txt just builds main.go without a module. Similarly, we leave
imports.txt without a GOPRIVATE, to test the 'go list -m' fallback.
For all other tests, explicitly set GOPRIVATE, to avoid two exec calls -
both 'go env GOPRIVATE' as well as 'go list -m'. Each of those calls
takes in the order of 10ms, so saving ~26 exec calls should easily add
to 200-300ms saved from 'go test -short'.
We need to call 'go env GOPRIVATE' instead of just using os.Getenv so
that we pick up the value from the new ${CONFIG}/go/env file, written by
'go env -w'.
However, we were calling 'go env' at every process start, including the
often tens or hundreds of compiler calls to build all the dependencies.
Instead, do that only once on the first 'garble build' process, and use
os.Setenv to pass that along to future garble sub-processes.
name old time/op new time/op delta
Build-8 1.81s ± 0% 1.74s ± 4% -3.78% (p=0.030 n=5+6)
name old sys-time/op new sys-time/op delta
Build-8 1.45s ± 2% 1.22s ± 1% -16.07% (p=0.002 n=6+6)
name old user-time/op new user-time/op delta
Build-8 10.9s ± 1% 10.6s ± 1% -2.82% (p=0.004 n=6+5)
Error strings should never be capitalized.
A binsubstr line in one of the tests was duplicate and thus useless.
Remove duplicate or trailing spaces in test scripts.
Finally, add a TODO for an optimization I just spotted.
Fixes #2.
Line numbers are now obfuscated, via `//line` comments.
Filenames are now obfuscated via `//line` comments, instead of changing the actual filename.
New flag `-tiny` to reduce the binary size, at the cost of reversibility.
Fixes#93.
The second typecheck lead to the creation of different type objects,
which didn't match the types in the blacklist anymore.
It turns out we don't need the second typecheck,
therfore it is now removed.
Like other tests, importing fmt results in quite a lot of extra work,
due to the lack of build caching.
In this particular test, we wanted fmt.Println so that T.String would be
called in an indirect way, without defining or referencing Stringer
interface in the main package.
We can do that by rolling our own "tinyfmt" package in a dozen or so
lines of code.
Below is how 'go test -short -vet=off -run Script/implement' is
affected, measured via benchcmd and benchstat:
name old time/op new time/op delta
GoTestScriptImplement 3.67s ± 9% 2.65s ±11% -27.68% (p=0.008 n=5+5)
name old user-time/op new user-time/op delta
GoTestScriptImplement 8.18s ± 4% 4.55s ± 9% -44.35% (p=0.008 n=5+5)
name old sys-time/op new sys-time/op delta
GoTestScriptImplement 1.27s ±12% 0.71s ±13% -44.07% (p=0.008 n=5+5)
name old peak-RSS-bytes new peak-RSS-bytes delta
GoTestScriptImplement 145MB ± 1% 145MB ± 2% ~ (p=1.000 n=5+5)
All in all, we shave about one full second. It doesn't seem to affect
the total 'go test -short' noticeably, but every little bit counts.
First, our original append line was completely ineffective; we never
used that "flags" slice again. Second, we only attempted to use the flag
when we obfuscated a package.
In fact, we never care about debugging information here, so for any
package we compile, we can add "-dwarf=false". At the moment, we compile
all packages, even if they aren't to be obfuscated, due to the lack of
access to the build cache.
As such, we save a significant amount of work. The numbers below were
obtained on a quiet machine with "go test -bench=. -benchtime=10x", six
times before and after the change.
name old time/op new time/op delta
Build-8 2.06s ± 4% 1.87s ± 2% -9.21% (p=0.002 n=6+6)
name old sys-time/op new sys-time/op delta
Build-8 1.51s ± 2% 1.46s ± 1% -3.12% (p=0.004 n=6+5)
name old user-time/op new user-time/op delta
Build-8 11.9s ± 2% 10.8s ± 1% -8.71% (p=0.002 n=6+6)
While at it, only do CI builds on pushes and PRs to the master branch,
so that my PRs created from the same repo don't trigger duplicate
builds.
If the flags list included ["-o" "binary"], we would properly skip "-o",
but we wouldn't skip "binary".
Thus, 'go list' would receive "binary" as the first argument, and assume
that's the first parameter and the end of the flags.
And add a unit test case.
Fixes#82, again.
The test case we had didn't have a realistic-looking module path with a
dot, so we hadn't noticed the bug with IndexByte.
Fix that. We verified that the new test fails if we undo the fix.
Otherwise any build flags like -tags won't be used, and we might easily
end up with errors or incorrect packages.
The common case with -tags is covered by one of the integration test
scripts. On top of that, we add a table-driven unit test to cover all
edge cases, since there are many we can do quickly in a unit test.
Fixes#82.
We don't really care about tools other than "compile" and "link". Stop
trying to keep a complete list.
Use "if err := f(); err != nil {" where it makes sense.
Simplify some declarations, and use a better variable name than "fW".
Instead of doing a 'go list' call every time we need to fetch a
dependency's export file, we now do a single 'go list' call before the
build begins. With the '-deps' flag, it gives us all the dependency
packages recursively.
We store that data in the gob format in a temporary file, and share it
with the future garble sub-processes via an env var.
This required lazy parsing of flags for the 'build' and 'test' commands,
since now we need to run 'go list' with the same package pattern
arguments.
Fixes#63.
The following identifiers are now skipped,
because they never show up in the binary:
- constant identifiers
- identifiers of local variables
(includes function params and named returns)
- identifiers of local types
First, unindent some of the AST code.
Second, genRandInt is unused; delete it.
Third, genRandIntn is really just mathrand.Intn. Just use it directly.
Fourth, don't use inline comments if they result in super long lines.
Since the new linker was failing on our crypto/aes shenanigans until the
recent commit to remove it for literal obfuscation.
Building Go does take about two minutes on the CI machine, but that's
fast enough. One can see the exact version that was used via the 'go
version' line.
Implement a literal obfuscator interface,
to allow the easy addition of new encodings.
Add literal obfuscation for byte literals.
Choose a random obfuscator on literal obfuscation,
useful when multiple obfuscators are implemented.
Fixes#62
Daniel Martí
lu4p
Andrew LeFevre
pagran
Zachary Wasserman