It's common for asset bundling code generators to produce huge literals,
for example in strings. Our literal obfuscators are meant for relatively
small string-like literals that a human would write, such as URLs, file
paths, and English text.
I ran some quick experiments, and it seems like "garble build -literals"
appears to hang trying to obfuscate literals starting at 5-20KiB. It's
not really hung; it's just doing a lot of busy work obfuscating those
literals. The code it produces is also far from ideal, so it also takes
some time to finally compile.
The generated code also led to crashes. For example, using "garble build
-literals -tiny" on a package containing literals of over a megabyte,
our use of asthelper to remove comments and shuffle line numbers could
run out of stack memory.
This all points in one direction: we never designed "-literals" to deal
with large sizes. Set a source-code-size limit of 2KiB.
We alter the literals.txt test as well, to include a few 128KiB string
literals. Before this fix, "go test" would seemingly hang on that test
for over a minute (I did not wait any longer). With the fix, those large
literals are not obfuscated, so the test ends in its usual 1-3s.
As said in the const comment, I don't believe any of this is a big
problem. Come Go 1.16, most developers should stop using asset-bundling
code generators and use go:embed instead. If we wanted to somehow
obfuscate those, it would be an entirely separate feature.
And, if someone wants to work on obfuscating truly large literals for
any reason, we need good tests and benchmarks to ensure garble does not
consume CPU for minutes or run out of memory.
I also simplified the generate-literals test command. The only argument
that matters to the script is the filename, since it's used later on.
Fixes#178.
In Go 1.15, if a dependency is required but not listed in go.mod/go.sum,
it's resolved and added automatically.
This is changing in 1.16. From that release, one will have to explicitly
update the mod files via 'go mod tidy' or 'go get'.
To get ahead of the curve, start using -mod=readonly to get the same
behavior in 1.15, and fix all existing tests.
The only tests that failed were imports.txt and syntax.txt, the only
ones to require other modules. But since we're here, let's add the 'go'
line to all go.mod files as well.
Use a static main.stderr file, like in the other tests. This means we
don't need to always start the test with a 'go build', and the output is
also obvious by just reading the txtar file.
We can also move generate-literals to a later stage, so that 'go test
-short' needs to do even less work.
'go test -short -run Script/literals' drops from ~0.4s to ~0.2s on my
laptop.
Finally, make the printing of byte lists not use trailing spaces, so
that the txtar file itself doesn't have trailing whitespace in its lines
either.
Fixes#103.
In tiny.txt, we already check line numbers via stderr, so there's no
need to do that via -debugdir.
In syntax.txt, we only really care about what names remain in the
binary, not the names which remain in the source but don't affect the
binary.
These changes are important because -debugdir adds a non-trivial amount
of work, which will impede build caching once that feature lands. We
will likely make -debugdir support build caching eventually, but for
now, this preliminary change will make 'go test' much faster with build
caching.
And of course, the tests get simpler, which is nice.
basic.txt just builds main.go without a module. Similarly, we leave
imports.txt without a GOPRIVATE, to test the 'go list -m' fallback.
For all other tests, explicitly set GOPRIVATE, to avoid two exec calls -
both 'go env GOPRIVATE' as well as 'go list -m'. Each of those calls
takes in the order of 10ms, so saving ~26 exec calls should easily add
to 200-300ms saved from 'go test -short'.
Fixes #2.
Line numbers are now obfuscated, via `//line` comments.
Filenames are now obfuscated via `//line` comments, instead of changing the actual filename.
New flag `-tiny` to reduce the binary size, at the cost of reversibility.
Implement a literal obfuscator interface,
to allow the easy addition of new encodings.
Add literal obfuscation for byte literals.
Choose a random obfuscator on literal obfuscation,
useful when multiple obfuscators are implemented.
Fixes#62
Daniel Martí
lu4p
pagran