You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
garble/testdata/script/cache.txtar

101 lines
1.9 KiB
Plaintext

# Ensure that garble knows how to handle all kinds of initial state scenarios
# when it comes to caching. If any cache file is missing, garble should redo
# the work and write the cache file again. See the docs below.
[short] stop # This step is slow by design, since it starts with an empty cache.
replace our caching inside GOCACHE with GARBLE_CACHE For each Go package we obfuscate, we need to store information about how we obfuscated it, which is needed when obfuscating its dependents. For example, if A depends on B to use the type B.Foo, A needs to know whether or not B.Foo was obfuscated; it depends on B's use of reflect. We record this information in a gob file, which is cached on disk. To avoid rolling our own custom cache, and since garble is so closely connected with cmd/go already, we piggybacked off of Go's GOCACHE. In particular, for each build cache entry per `go list`'s Export field, we would store a "garble" sibling file with that gob content. However, this was brittle for two reasons: 1) We were doing this without cmd/go's permission or knowledge. We were careful to use filename suffixes similar to Export files, meaning that `go clean` and other commands would treat them the same. However, this could confuse cmd/go at any point in the future. 2) cmd/go trims cache entries in GOCACHE regularly, to keep the size of the build and test caches under control. Right now, this means that every 24h, any file not accessed in the last five days is deleted. However, that trimming heuristic is done per-file. If the trimming removed Garble's sibling file but not the original Export file, this could cause errors such as "cannot load garble export file" which users already ran into. Instead, start using github.com/rogpeppe/go-internal/cache, an exported copy of cmd/go's own cache implementation for GOCACHE. Since we need an entirely separate directory, we introduce GARBLE_CACHE, defaulting to the "garble" directory inside the user's cache directory. For example, on Linux this would be ~/.cache/garble. Inside GARBLE_CACHE, our gob file cache will be under "build", which helps clarify that this cache is used when obfuscating Go builds, and allows placing other kinds of caches inside GARBLE_CACHE. For example, we already have a need for storing linker binaries, which for now still use their own caching mechanism. This commit does not make our cache properly resistant to removed files. The proof is that our seed.txtar testscript still fails the second case. However, we do rewrite all of our caching logic away from Export files, which in itself is a considerable refactor, and we add a few TODOs. One notable change is how we load gob files from dependencies when building the cache entry for the current package. We used to load the gob files from all packages in the Deps field. However, that is the list of all _transitive_ dependencies. Since these gob files are already flat, meaning they contain information about all of their transitive dependencies as well, we need only load the gob files from the direct dependencies, the Imports field. Performance is largely unchanged, since the behavior is similar. However, the change from Deps to Imports saves us some work, which can be seen in the reduced mallocs per obfuscated build. It's unclear why the binary size isn't stable. When reverting the Deps to Imports change, it then settles at 5.386Mi, which is almost exactly in between the two measurements below. I'm not sure why, but that metric appears to be slightly unstable. goos: linux goarch: amd64 pkg: mvdan.cc/garble cpu: AMD Ryzen 7 PRO 5850U with Radeon Graphics │ old │ new │ │ sec/op │ sec/op vs base │ Build-8 11.09 ± 1% 11.08 ± 1% ~ (p=0.796 n=10) │ old │ new │ │ bin-B │ bin-B vs base │ Build-8 5.390Mi ± 0% 5.382Mi ± 0% -0.14% (p=0.000 n=10) │ old │ new │ │ cached-sec/op │ cached-sec/op vs base │ Build-8 415.5m ± 4% 421.6m ± 1% ~ (p=0.190 n=10) │ old │ new │ │ mallocs/op │ mallocs/op vs base │ Build-8 35.43M ± 0% 34.05M ± 0% -3.89% (p=0.000 n=10) │ old │ new │ │ sys-sec/op │ sys-sec/op vs base │ Build-8 5.662 ± 1% 5.701 ± 2% ~ (p=0.280 n=10)
1 year ago
env GOCACHE=${WORK}/go-cache
env GARBLE_CACHE=${WORK}/garble-cache
# level1a has the regular Go build cached.
exec go build ./level1a
# level1b has the garble build cached, but our own cache files are gone.
exec garble build ./level1b
replace our caching inside GOCACHE with GARBLE_CACHE For each Go package we obfuscate, we need to store information about how we obfuscated it, which is needed when obfuscating its dependents. For example, if A depends on B to use the type B.Foo, A needs to know whether or not B.Foo was obfuscated; it depends on B's use of reflect. We record this information in a gob file, which is cached on disk. To avoid rolling our own custom cache, and since garble is so closely connected with cmd/go already, we piggybacked off of Go's GOCACHE. In particular, for each build cache entry per `go list`'s Export field, we would store a "garble" sibling file with that gob content. However, this was brittle for two reasons: 1) We were doing this without cmd/go's permission or knowledge. We were careful to use filename suffixes similar to Export files, meaning that `go clean` and other commands would treat them the same. However, this could confuse cmd/go at any point in the future. 2) cmd/go trims cache entries in GOCACHE regularly, to keep the size of the build and test caches under control. Right now, this means that every 24h, any file not accessed in the last five days is deleted. However, that trimming heuristic is done per-file. If the trimming removed Garble's sibling file but not the original Export file, this could cause errors such as "cannot load garble export file" which users already ran into. Instead, start using github.com/rogpeppe/go-internal/cache, an exported copy of cmd/go's own cache implementation for GOCACHE. Since we need an entirely separate directory, we introduce GARBLE_CACHE, defaulting to the "garble" directory inside the user's cache directory. For example, on Linux this would be ~/.cache/garble. Inside GARBLE_CACHE, our gob file cache will be under "build", which helps clarify that this cache is used when obfuscating Go builds, and allows placing other kinds of caches inside GARBLE_CACHE. For example, we already have a need for storing linker binaries, which for now still use their own caching mechanism. This commit does not make our cache properly resistant to removed files. The proof is that our seed.txtar testscript still fails the second case. However, we do rewrite all of our caching logic away from Export files, which in itself is a considerable refactor, and we add a few TODOs. One notable change is how we load gob files from dependencies when building the cache entry for the current package. We used to load the gob files from all packages in the Deps field. However, that is the list of all _transitive_ dependencies. Since these gob files are already flat, meaning they contain information about all of their transitive dependencies as well, we need only load the gob files from the direct dependencies, the Imports field. Performance is largely unchanged, since the behavior is similar. However, the change from Deps to Imports saves us some work, which can be seen in the reduced mallocs per obfuscated build. It's unclear why the binary size isn't stable. When reverting the Deps to Imports change, it then settles at 5.386Mi, which is almost exactly in between the two measurements below. I'm not sure why, but that metric appears to be slightly unstable. goos: linux goarch: amd64 pkg: mvdan.cc/garble cpu: AMD Ryzen 7 PRO 5850U with Radeon Graphics │ old │ new │ │ sec/op │ sec/op vs base │ Build-8 11.09 ± 1% 11.08 ± 1% ~ (p=0.796 n=10) │ old │ new │ │ bin-B │ bin-B vs base │ Build-8 5.390Mi ± 0% 5.382Mi ± 0% -0.14% (p=0.000 n=10) │ old │ new │ │ cached-sec/op │ cached-sec/op vs base │ Build-8 415.5m ± 4% 421.6m ± 1% ~ (p=0.190 n=10) │ old │ new │ │ mallocs/op │ mallocs/op vs base │ Build-8 35.43M ± 0% 34.05M ± 0% -3.89% (p=0.000 n=10) │ old │ new │ │ sys-sec/op │ sys-sec/op vs base │ Build-8 5.662 ± 1% 5.701 ± 2% ~ (p=0.280 n=10)
1 year ago
rm garble-cache
# level1c has the garble build cached with all files available.
exec garble build ./level1c
exec garble build
exec ./main
cmp stderr main.stderr
# verify with regular Go.
go build
exec ./main
cmp stderr main.stderr
-- go.mod --
module test/main
go 1.21
-- main.go --
package main
import (
"test/main/level1a"
"test/main/level1b"
"test/main/level1c"
)
func main() {
level1a.Print()
level1b.Print()
level1c.Print()
}
-- level1a/pkg.go --
package level1a
import (
"test/main/level1a/level2x"
"test/main/level1a/level2y"
)
func Print() { println(level2x.Value, level2y.Value) }
-- level1a/level2x/pkg.go --
package level2x
var Value = "1a/2x"
-- level1a/level2y/pkg.go --
package level2y
var Value = "1a/2y"
-- level1b/pkg.go --
package level1b
import (
"test/main/level1b/level2x"
"test/main/level1b/level2y"
)
func Print() { println(level2x.Value, level2y.Value) }
-- level1b/level2x/pkg.go --
package level2x
var Value = "1b/2x"
-- level1b/level2y/pkg.go --
package level2y
var Value = "1b/2y"
-- level1c/pkg.go --
package level1c
import (
"test/main/level1c/level2x"
"test/main/level1c/level2y"
)
func Print() { println(level2x.Value, level2y.Value) }
-- level1c/level2x/pkg.go --
package level2x
var Value = "1c/2x"
-- level1c/level2y/pkg.go --
package level2y
var Value = "1c/2y"
-- main.stderr --
1a/2x 1a/2y
1b/2x 1b/2y
1c/2x 1c/2y