You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
garble/testdata/script/reflect.txtar

606 lines
14 KiB
Plaintext

exec garble build
exec ./main
cmp stdout main.stdout
! binsubstr main$exe 'garble_main.go' 'test/main' 'importedpkg.' 'DownstreamObfuscated' 'SiblingObfuscated' 'IndirectObfuscated' 'IndirectNamedWithoutReflect' 'AliasIndirectNamedWithReflect' 'AliasIndirectNamedWithoutReflect' 'FmtTypeField' 'LocalObfuscated'
fix a number of issues involving types from indirect imports obfuscatedTypesPackage is used to figure out if a name in a dependency package was obfuscated or not. For example, if that package used reflection on a named type, it wasn't obfuscated, so we must have the same information to not obfuscate the same name downstream. obfuscatedTypesPackage could return nil if the package was indirectly imported, though. This can happen if a direct import has a function that returns an indirect type, or if a direct import exposes a name that's a type alias to an indirect type. We sort of dealt with this in two pieces of code by checking for obfPkg!=nil, but a third one did not have this check and caused a panic in the added test case: --- FAIL: TestScripts/reflect (0.81s) testscript.go:397: > env GOPRIVATE=test/main > garble build [stderr] # test/main panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8a5e39] More importantly though, the nil check only avoids panics. It doesn't fix the root cause of the problem: that importcfg does not contain indirectly imported packages. The added test case would still fail, as we would obfuscate a type in the main package, but not in the indirectly imported package where the type is defined. To fix this, resurrect a bit of code from earlier garble versions, which uses "go list -toolexec=garble" to fetch a package's export file. This lets us fill the indirect import gaps in importcfg, working around the problem entirely. This solution is still not particularly great, so we add a TODO about possibly rethinking this in the future. It does add some overhead and complexity, though thankfully indirect imports should be uncommon. This fixes a few panics while building the protobuf module.
3 years ago
binsubstr main$exe 'ReflectInDefined' 'ExportedField2' 'unexportedField2' 'IndirectUnobfuscated' 'IndirectNamedWithReflect'
[short] stop # no need to verify this with -short
# Check that the program works as expected without garble.
go build
exec ./main
cmp stdout main.stdout
-- go.mod --
module test/main
go 1.22
-- garble_main.go --
package main
import (
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/x509"
"crypto/x509/pkix"
"encoding/gob"
"encoding/json"
"fmt"
"math/big"
"os"
"reflect"
"unsafe"
"strings"
"sync"
"text/template"
"test/main/importedpkg"
"test/main/importedpkg2"
)
var Sink interface{}
func main() {
// Fields still work fine when they are not obfuscated.
fmt.Println(importedpkg.ReflectInDefinedVar.ExportedField2)
fmt.Println(importedpkg.ReflectInDefined{ExportedField2: 5})
// Type names are not obfuscated either, when reflection is used.
printfWithoutPackage("%T\n", importedpkg.ReflectTypeOf(2))
printfWithoutPackage("%T\n", importedpkg.ReflectTypeOfIndirect(4))
// More complex use of reflect.
v := importedpkg.ReflectValueOfVar
printfWithoutPackage("%#v\n", v)
method := reflect.ValueOf(&v).MethodByName("ExportedMethodName")
if method.IsValid() {
fmt.Println(method.Call(nil))
} else {
fmt.Println("method not found")
}
// Use of a common library using reflect, encoding/json.
enc, _ := json.Marshal(EncodingT{Foo: 3})
fmt.Println(string(enc))
// Another common library, text/template.
tmpl := template.Must(template.New("").Parse("Hello {{.Name}}."))
_ = tmpl.Execute(os.Stdout, struct{Name string}{Name: "Dave"})
fmt.Println() // Always print a newline.
// Another complex case, involving embedding and another package.
outer := &importedpkg.EmbeddingOuter{}
outer.InnerField = 3
enc, _ = json.Marshal(outer)
fmt.Println(string(enc))
// An edge case; the struct type is defined in a different package.
// Note that the struct type is unnamed, but it still has named fields.
// We only use reflection on it here, not the declaring package.
// As such, we should obfuscate the field name.
// Simply using the field name here used to cause build failures.
_ = reflect.TypeOf(importedpkg.UnnamedWithDownstreamReflect{})
fmt.Printf("%v\n", importedpkg.UnnamedWithDownstreamReflect{
fix a number of issues involving types from indirect imports obfuscatedTypesPackage is used to figure out if a name in a dependency package was obfuscated or not. For example, if that package used reflection on a named type, it wasn't obfuscated, so we must have the same information to not obfuscate the same name downstream. obfuscatedTypesPackage could return nil if the package was indirectly imported, though. This can happen if a direct import has a function that returns an indirect type, or if a direct import exposes a name that's a type alias to an indirect type. We sort of dealt with this in two pieces of code by checking for obfPkg!=nil, but a third one did not have this check and caused a panic in the added test case: --- FAIL: TestScripts/reflect (0.81s) testscript.go:397: > env GOPRIVATE=test/main > garble build [stderr] # test/main panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8a5e39] More importantly though, the nil check only avoids panics. It doesn't fix the root cause of the problem: that importcfg does not contain indirectly imported packages. The added test case would still fail, as we would obfuscate a type in the main package, but not in the indirectly imported package where the type is defined. To fix this, resurrect a bit of code from earlier garble versions, which uses "go list -toolexec=garble" to fetch a package's export file. This lets us fill the indirect import gaps in importcfg, working around the problem entirely. This solution is still not particularly great, so we add a TODO about possibly rethinking this in the future. It does add some overhead and complexity, though thankfully indirect imports should be uncommon. This fixes a few panics while building the protobuf module.
3 years ago
DownstreamObfuscated: "downstream",
})
// An edge case; the struct type is defined in package importedpkg2.
// importedpkg2 does not use reflection on it, so it's not obfuscated there.
// importedpkg uses reflection on a type containing ReflectInSiblingImport.
// If our logic is incorrect, we might inconsistently obfuscate the type.
// We should not obfuscate it when building any package.
fmt.Printf("%v\n", importedpkg2.ReflectInSiblingImport{
fix a number of issues involving types from indirect imports obfuscatedTypesPackage is used to figure out if a name in a dependency package was obfuscated or not. For example, if that package used reflection on a named type, it wasn't obfuscated, so we must have the same information to not obfuscate the same name downstream. obfuscatedTypesPackage could return nil if the package was indirectly imported, though. This can happen if a direct import has a function that returns an indirect type, or if a direct import exposes a name that's a type alias to an indirect type. We sort of dealt with this in two pieces of code by checking for obfPkg!=nil, but a third one did not have this check and caused a panic in the added test case: --- FAIL: TestScripts/reflect (0.81s) testscript.go:397: > env GOPRIVATE=test/main > garble build [stderr] # test/main panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8a5e39] More importantly though, the nil check only avoids panics. It doesn't fix the root cause of the problem: that importcfg does not contain indirectly imported packages. The added test case would still fail, as we would obfuscate a type in the main package, but not in the indirectly imported package where the type is defined. To fix this, resurrect a bit of code from earlier garble versions, which uses "go list -toolexec=garble" to fetch a package's export file. This lets us fill the indirect import gaps in importcfg, working around the problem entirely. This solution is still not particularly great, so we add a TODO about possibly rethinking this in the future. It does add some overhead and complexity, though thankfully indirect imports should be uncommon. This fixes a few panics while building the protobuf module.
3 years ago
SiblingObfuscated: "sibling",
})
fix a number of issues involving types from indirect imports obfuscatedTypesPackage is used to figure out if a name in a dependency package was obfuscated or not. For example, if that package used reflection on a named type, it wasn't obfuscated, so we must have the same information to not obfuscate the same name downstream. obfuscatedTypesPackage could return nil if the package was indirectly imported, though. This can happen if a direct import has a function that returns an indirect type, or if a direct import exposes a name that's a type alias to an indirect type. We sort of dealt with this in two pieces of code by checking for obfPkg!=nil, but a third one did not have this check and caused a panic in the added test case: --- FAIL: TestScripts/reflect (0.81s) testscript.go:397: > env GOPRIVATE=test/main > garble build [stderr] # test/main panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8a5e39] More importantly though, the nil check only avoids panics. It doesn't fix the root cause of the problem: that importcfg does not contain indirectly imported packages. The added test case would still fail, as we would obfuscate a type in the main package, but not in the indirectly imported package where the type is defined. To fix this, resurrect a bit of code from earlier garble versions, which uses "go list -toolexec=garble" to fetch a package's export file. This lets us fill the indirect import gaps in importcfg, working around the problem entirely. This solution is still not particularly great, so we add a TODO about possibly rethinking this in the future. It does add some overhead and complexity, though thankfully indirect imports should be uncommon. This fixes a few panics while building the protobuf module.
3 years ago
// Using type aliases as both regular fields, and embedded fields.
fix a number of issues involving types from indirect imports obfuscatedTypesPackage is used to figure out if a name in a dependency package was obfuscated or not. For example, if that package used reflection on a named type, it wasn't obfuscated, so we must have the same information to not obfuscate the same name downstream. obfuscatedTypesPackage could return nil if the package was indirectly imported, though. This can happen if a direct import has a function that returns an indirect type, or if a direct import exposes a name that's a type alias to an indirect type. We sort of dealt with this in two pieces of code by checking for obfPkg!=nil, but a third one did not have this check and caused a panic in the added test case: --- FAIL: TestScripts/reflect (0.81s) testscript.go:397: > env GOPRIVATE=test/main > garble build [stderr] # test/main panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8a5e39] More importantly though, the nil check only avoids panics. It doesn't fix the root cause of the problem: that importcfg does not contain indirectly imported packages. The added test case would still fail, as we would obfuscate a type in the main package, but not in the indirectly imported package where the type is defined. To fix this, resurrect a bit of code from earlier garble versions, which uses "go list -toolexec=garble" to fetch a package's export file. This lets us fill the indirect import gaps in importcfg, working around the problem entirely. This solution is still not particularly great, so we add a TODO about possibly rethinking this in the future. It does add some overhead and complexity, though thankfully indirect imports should be uncommon. This fixes a few panics while building the protobuf module.
3 years ago
var emb EmbeddingIndirect
emb.With.IndirectUnobfuscated = "indirect-with"
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
emb.With.DuplicateFieldName = 3
fix a number of issues involving types from indirect imports obfuscatedTypesPackage is used to figure out if a name in a dependency package was obfuscated or not. For example, if that package used reflection on a named type, it wasn't obfuscated, so we must have the same information to not obfuscate the same name downstream. obfuscatedTypesPackage could return nil if the package was indirectly imported, though. This can happen if a direct import has a function that returns an indirect type, or if a direct import exposes a name that's a type alias to an indirect type. We sort of dealt with this in two pieces of code by checking for obfPkg!=nil, but a third one did not have this check and caused a panic in the added test case: --- FAIL: TestScripts/reflect (0.81s) testscript.go:397: > env GOPRIVATE=test/main > garble build [stderr] # test/main panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8a5e39] More importantly though, the nil check only avoids panics. It doesn't fix the root cause of the problem: that importcfg does not contain indirectly imported packages. The added test case would still fail, as we would obfuscate a type in the main package, but not in the indirectly imported package where the type is defined. To fix this, resurrect a bit of code from earlier garble versions, which uses "go list -toolexec=garble" to fetch a package's export file. This lets us fill the indirect import gaps in importcfg, working around the problem entirely. This solution is still not particularly great, so we add a TODO about possibly rethinking this in the future. It does add some overhead and complexity, though thankfully indirect imports should be uncommon. This fixes a few panics while building the protobuf module.
3 years ago
emb.Without.IndirectObfuscated = "indirect-without"
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
emb.Without.DuplicateFieldName = 4
fix a number of issues involving types from indirect imports obfuscatedTypesPackage is used to figure out if a name in a dependency package was obfuscated or not. For example, if that package used reflection on a named type, it wasn't obfuscated, so we must have the same information to not obfuscate the same name downstream. obfuscatedTypesPackage could return nil if the package was indirectly imported, though. This can happen if a direct import has a function that returns an indirect type, or if a direct import exposes a name that's a type alias to an indirect type. We sort of dealt with this in two pieces of code by checking for obfPkg!=nil, but a third one did not have this check and caused a panic in the added test case: --- FAIL: TestScripts/reflect (0.81s) testscript.go:397: > env GOPRIVATE=test/main > garble build [stderr] # test/main panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8a5e39] More importantly though, the nil check only avoids panics. It doesn't fix the root cause of the problem: that importcfg does not contain indirectly imported packages. The added test case would still fail, as we would obfuscate a type in the main package, but not in the indirectly imported package where the type is defined. To fix this, resurrect a bit of code from earlier garble versions, which uses "go list -toolexec=garble" to fetch a package's export file. This lets us fill the indirect import gaps in importcfg, working around the problem entirely. This solution is still not particularly great, so we add a TODO about possibly rethinking this in the future. It does add some overhead and complexity, though thankfully indirect imports should be uncommon. This fixes a few panics while building the protobuf module.
3 years ago
fmt.Printf("%v\n", emb)
printfWithoutPackage("%#v\n", emb.With)
// TODO: don't obfuscate the embedded field name here
// printfWithoutPackage("%#v\n", importedpkg.ReflectEmbeddingAlias{})
indirectReflection(IndirectReflection{})
fmt.Println(FmtType{})
// Variadic functions are a bit tricky as the number of parameters is variable.
// We want to notice indirect uses of reflection via all variadic arguments.
_ = importedpkg.VariadicReflect(0, 1, 2, 3)
_ = importedpkg.VariadicReflect(0)
variadic := VariadicReflection{ReflectionField: "variadic"}
_ = importedpkg.VariadicReflect("foo", 1, variadic, false)
printfWithoutPackage("%#v\n", variadic)
testx509()
testGoSpew()
// Very complex reflection used by gorm
user := StatUser{}
find(&user)
// Similar to gorm with composite literals instead of direct assignments
userComp := StatCompUser{}
findComp(&userComp)
x := UnnamedStructInterface(importedpkg.ReflectUnnamedStruct(0))
x.UnnamedStructMethod(struct{ UnnamedStructField string }{UnnamedStructField: "field value"})
// Local names not used in reflection should not be in the final binary,
// even if they are embedded in a struct and become a field name.
type unexportedLocalObfuscated struct { LocalObfuscatedA int }
type ExportedLocalObfuscated struct { LocalObfuscatedB int }
type EmbeddingObfuscated struct {
unexportedLocalObfuscated
ExportedLocalObfuscated
}
// Ensure the types are kept in the binary. Use an anonymous type too.
_ = fmt.Sprintf("%#v", EmbeddingObfuscated{})
_ = fmt.Sprintf("%#v", struct{ExportedLocalObfuscated}{})
// reflection can see all type names, even local ones, so they cannot be obfuscated.
{
type TypeOfNamedField struct { NamedReflectionField int }
type TypeOfEmbeddedField struct { EmbeddedReflectionField int }
type TypeOfParent struct {
ReflectionField TypeOfNamedField
TypeOfEmbeddedField
}
t := reflect.TypeOf(TypeOfParent{})
fmt.Println("TypeOfParent's own name:", t.Name())
namedField, _ := t.FieldByName("ReflectionField")
namedFieldField, _ := namedField.Type.FieldByName("NamedReflectionField")
fmt.Println("TypeOfParent named:",
namedField.Type.Name(),
namedFieldField.Name,
)
embedField, _ := t.FieldByName("TypeOfEmbeddedField")
embedFieldField, _ := embedField.Type.FieldByName("EmbeddedReflectionField")
fmt.Println("TypeOfParent embedded:",
embedField.Type.Name(),
embedFieldField.Name,
)
}
y := UnnamedStructFields{}
y.unexportedGoGoProto = new(struct {
mu sync.Mutex
extensionMap map[int32]EncodingT
})
fix a number of issues involving types from indirect imports obfuscatedTypesPackage is used to figure out if a name in a dependency package was obfuscated or not. For example, if that package used reflection on a named type, it wasn't obfuscated, so we must have the same information to not obfuscate the same name downstream. obfuscatedTypesPackage could return nil if the package was indirectly imported, though. This can happen if a direct import has a function that returns an indirect type, or if a direct import exposes a name that's a type alias to an indirect type. We sort of dealt with this in two pieces of code by checking for obfPkg!=nil, but a third one did not have this check and caused a panic in the added test case: --- FAIL: TestScripts/reflect (0.81s) testscript.go:397: > env GOPRIVATE=test/main > garble build [stderr] # test/main panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8a5e39] More importantly though, the nil check only avoids panics. It doesn't fix the root cause of the problem: that importcfg does not contain indirectly imported packages. The added test case would still fail, as we would obfuscate a type in the main package, but not in the indirectly imported package where the type is defined. To fix this, resurrect a bit of code from earlier garble versions, which uses "go list -toolexec=garble" to fetch a package's export file. This lets us fill the indirect import gaps in importcfg, working around the problem entirely. This solution is still not particularly great, so we add a TODO about possibly rethinking this in the future. It does add some overhead and complexity, though thankfully indirect imports should be uncommon. This fixes a few panics while building the protobuf module.
3 years ago
}
type EmbeddingIndirect struct {
// With field names, to test selectors above.
With importedpkg.AliasIndirectNamedWithReflect
Without importedpkg.AliasIndirectNamedWithoutReflect
// Embedding used to crash garble, too.
importedpkg.AliasIndirectNamedWithReflect
}
func printfWithoutPackage(format string, v any) {
s := fmt.Sprintf(format, v)
if _, without, found := strings.Cut(s, "."); found {
s = without
}
fmt.Print(s)
}
type EncodingT struct {
Foo int
}
type RecursiveStruct struct {
*RecursiveStruct
list []RecursiveStruct
}
// This could crash or hang if we don't deal with loops.
var _ = reflect.TypeOf(RecursiveStruct{})
type IndirectReflection struct {
ReflectionField string
}
func indirectReflection(v any) {
fmt.Println(reflect.TypeOf(v).Field(0).Name)
}
type VariadicReflection struct {
ReflectionField string
}
type FmtType struct {
FmtTypeField int
}
// copied from github.com/davecgh/go-spew, which reaches into reflect's internals
func testGoSpew() {
flagValOffset := func() uintptr {
field, ok := reflect.TypeOf(reflect.Value{}).FieldByName("flag")
if !ok {
panic("reflect.Value has no flag field")
}
return field.Offset
}()
type flag uintptr
flagField := func(v *reflect.Value) *flag {
return (*flag)(unsafe.Pointer(uintptr(unsafe.Pointer(v)) + flagValOffset))
}
type t0 int
var t struct {
A t0
t0
a t0
}
vA := reflect.ValueOf(t).FieldByName("A")
va := reflect.ValueOf(t).FieldByName("a")
vt0 := reflect.ValueOf(t).FieldByName("t0")
flagvA := *flagField(&vA)
flagva := *flagField(&va)
flagvt0 := *flagField(&vt0)
if flagvA&flagva&flagvt0 == 0 {
panic("reflect.Value read-only flag has changed semantics")
}
type T0 int
var T struct {
A T0
T0
a T0
}
vA = reflect.ValueOf(T).FieldByName("A")
va = reflect.ValueOf(T).FieldByName("a")
vt0 = reflect.ValueOf(T).FieldByName("T0")
flagvA = *flagField(&vA)
flagva = *flagField(&va)
flagvt0 = *flagField(&vt0)
if flagvA&flagva&flagvt0 == 0 {
panic("reflect.Value read-only flag has changed semantics")
}
}
// encoding/x509 uses encoding/asn1, which uses reflect.
// In one place it depends on field names; that used to be broken by garble.
func testx509() {
priv, err := ecdsa.GenerateKey(elliptic.P521(), rand.Reader)
if err != nil {
panic(err)
}
template := x509.Certificate{
SerialNumber: big.NewInt(1),
Subject: pkix.Name{Organization: []string{"Acme Co"}},
}
derBytes, err := x509.CreateCertificate(rand.Reader, &template, &template, &priv.PublicKey, priv)
if err != nil {
panic(err)
}
_, err = x509.ParseCertificate(derBytes)
if err != nil {
panic(err)
}
}
type pingMsg struct {
Data string `sshtype:"192"`
}
type pongMsg struct {
Data string `sshtype:"193"`
}
// golang.org/x/crypto/ssh converts a reflected type to another type
func testSSH() {
var msg = pingMsg{
Data: "data",
}
json.Marshal(msg)
_ = pongMsg(msg)
}
// variations similar to ssh
type reflectedMsg struct {
Data string
}
type convertedMsg struct {
Data string
}
func reflectConvert() {
msg := reflectedMsg(convertedMsg{})
json.Marshal(msg)
}
type reflectedMsg2 struct {
Data string
}
type convertedMsg2 struct {
Data string
}
func unrelatedConvert() {
// only discoverable by rechecking the package
_ = convertedMsg2(reflectedMsg2{})
}
func reflectUnrelatedConv() {
var msg = reflectedMsg2{
Data: "data",
}
json.Marshal(msg)
}
type StatUser struct {
Id int64 `gorm:"primaryKey"`
User_Id int64
}
type StatCompUser struct {
Id int64 `gorm:"primaryKey"`
User_Id int64
}
type Transaction struct {
Statement Statement
}
type Statement struct {
Dest interface{}
Model string
}
func find(dest interface{}) {
tx := Transaction{}
tx.Statement.Dest = dest
execute(tx)
}
func findComp(dest interface{}) {
tx := Transaction{
Statement: Statement{
Dest: dest,
},
}
execute(tx)
}
func execute(db Transaction) {
stmt := db.Statement
v := reflect.TypeOf(stmt.Dest)
fmt.Println(v)
}
type UnnamedStructInterface interface {
UnnamedStructMethod(struct{ UnnamedStructField string })
}
// Some projects declare types with unnamed struct fields,
// and the entire type is used via reflection and cannot be obfuscated.
// However, when assigning to these fields, the use of inline anonymous struct types
// confused garble, and it did not obfuscate those inline structs as well.
// That resulted in "cannot use X as Y value in assignment" build errors.
var _ = reflect.TypeOf(UnnamedStructFields{})
type UnnamedStructFields struct {
// As seen in github.com/gogo/protobuf/proto.
unexportedGoGoProto *struct {
mu sync.Mutex
extensionMap map[int32]EncodingT
}
}
func gobStruct() {
type gobAlias struct {
Security []map[string]struct {
List []string
Pad bool
}
}
alias := gobAlias{}
gob.NewEncoder(os.Stdout).Encode(alias)
alias.Security = make([]map[string]struct {
List []string
Pad bool
}, 0, len([]string{}))
}
func gobMap() {
type gobAlias struct {
Security map[string]struct {
List []string
Pad bool
}
}
alias := gobAlias{}
gob.NewEncoder(os.Stdout).Encode(alias)
alias.Security = make(map[string]struct {
List []string
Pad bool
}, len([]string{}))
}
func gobChan() {
type gobAlias struct {
Security chan struct {
List []string
Pad bool
}
}
alias := gobAlias{}
gob.NewEncoder(os.Stdout).Encode(alias)
alias.Security = make(chan struct {
List []string
Pad bool
}, len([]string{}))
}
-- importedpkg/imported.go --
package importedpkg
import (
"fmt"
"reflect"
fix a number of issues involving types from indirect imports obfuscatedTypesPackage is used to figure out if a name in a dependency package was obfuscated or not. For example, if that package used reflection on a named type, it wasn't obfuscated, so we must have the same information to not obfuscate the same name downstream. obfuscatedTypesPackage could return nil if the package was indirectly imported, though. This can happen if a direct import has a function that returns an indirect type, or if a direct import exposes a name that's a type alias to an indirect type. We sort of dealt with this in two pieces of code by checking for obfPkg!=nil, but a third one did not have this check and caused a panic in the added test case: --- FAIL: TestScripts/reflect (0.81s) testscript.go:397: > env GOPRIVATE=test/main > garble build [stderr] # test/main panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8a5e39] More importantly though, the nil check only avoids panics. It doesn't fix the root cause of the problem: that importcfg does not contain indirectly imported packages. The added test case would still fail, as we would obfuscate a type in the main package, but not in the indirectly imported package where the type is defined. To fix this, resurrect a bit of code from earlier garble versions, which uses "go list -toolexec=garble" to fetch a package's export file. This lets us fill the indirect import gaps in importcfg, working around the problem entirely. This solution is still not particularly great, so we add a TODO about possibly rethinking this in the future. It does add some overhead and complexity, though thankfully indirect imports should be uncommon. This fixes a few panics while building the protobuf module.
3 years ago
"test/main/importedpkg/indirect"
"test/main/importedpkg2"
)
type ReflectTypeOf int
var _ = reflect.TypeOf(ReflectTypeOf(0))
type ReflectTypeOfIndirect int
var _ = reflect.TypeOf(new([]*ReflectTypeOfIndirect))
type ReflectValueOf struct {
ExportedField string
unexportedField string
}
func (r *ReflectValueOf) ExportedMethodName() string { return "method: " + r.ExportedField }
var ReflectValueOfVar = ReflectValueOf{ExportedField: "abc"}
var _ = reflect.TypeOf(ReflectValueOfVar)
type ReflectInDefined struct {
ExportedField2 int
unexportedField2 int
importedpkg2.ReflectInSiblingImport
}
var ReflectInDefinedVar = ReflectInDefined{ExportedField2: 9000}
var _ = reflect.TypeOf(ReflectInDefinedVar)
var _ = reflect.TypeOf([]*struct{EmbeddingOuter}{})
type EmbeddingOuter struct {
EmbeddingInner
Anon struct {
AnonField int
}
}
type EmbeddingInner struct {
InnerField int
}
type UnnamedWithDownstreamReflect = struct {
fix a number of issues involving types from indirect imports obfuscatedTypesPackage is used to figure out if a name in a dependency package was obfuscated or not. For example, if that package used reflection on a named type, it wasn't obfuscated, so we must have the same information to not obfuscate the same name downstream. obfuscatedTypesPackage could return nil if the package was indirectly imported, though. This can happen if a direct import has a function that returns an indirect type, or if a direct import exposes a name that's a type alias to an indirect type. We sort of dealt with this in two pieces of code by checking for obfPkg!=nil, but a third one did not have this check and caused a panic in the added test case: --- FAIL: TestScripts/reflect (0.81s) testscript.go:397: > env GOPRIVATE=test/main > garble build [stderr] # test/main panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8a5e39] More importantly though, the nil check only avoids panics. It doesn't fix the root cause of the problem: that importcfg does not contain indirectly imported packages. The added test case would still fail, as we would obfuscate a type in the main package, but not in the indirectly imported package where the type is defined. To fix this, resurrect a bit of code from earlier garble versions, which uses "go list -toolexec=garble" to fetch a package's export file. This lets us fill the indirect import gaps in importcfg, working around the problem entirely. This solution is still not particularly great, so we add a TODO about possibly rethinking this in the future. It does add some overhead and complexity, though thankfully indirect imports should be uncommon. This fixes a few panics while building the protobuf module.
3 years ago
DownstreamObfuscated string
}
fix a number of issues involving types from indirect imports obfuscatedTypesPackage is used to figure out if a name in a dependency package was obfuscated or not. For example, if that package used reflection on a named type, it wasn't obfuscated, so we must have the same information to not obfuscate the same name downstream. obfuscatedTypesPackage could return nil if the package was indirectly imported, though. This can happen if a direct import has a function that returns an indirect type, or if a direct import exposes a name that's a type alias to an indirect type. We sort of dealt with this in two pieces of code by checking for obfPkg!=nil, but a third one did not have this check and caused a panic in the added test case: --- FAIL: TestScripts/reflect (0.81s) testscript.go:397: > env GOPRIVATE=test/main > garble build [stderr] # test/main panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8a5e39] More importantly though, the nil check only avoids panics. It doesn't fix the root cause of the problem: that importcfg does not contain indirectly imported packages. The added test case would still fail, as we would obfuscate a type in the main package, but not in the indirectly imported package where the type is defined. To fix this, resurrect a bit of code from earlier garble versions, which uses "go list -toolexec=garble" to fetch a package's export file. This lets us fill the indirect import gaps in importcfg, working around the problem entirely. This solution is still not particularly great, so we add a TODO about possibly rethinking this in the future. It does add some overhead and complexity, though thankfully indirect imports should be uncommon. This fixes a few panics while building the protobuf module.
3 years ago
type (
AliasIndirectNamedWithReflect = indirect.IndirectNamedWithReflect
AliasIndirectNamedWithoutReflect = indirect.IndirectNamedWithoutReflect
)
var _ = reflect.TypeOf(ReflectEmbeddingAlias{})
type ReflectEmbeddingAlias struct {
ReflectEmbeddedAlias
}
type ReflectEmbeddedAlias = ReflectEmbeddingNamed
type ReflectEmbeddingNamed struct{}
func VariadicReflect(x any, ys ...any) int {
_ = reflect.TypeOf(x)
for _, y := range ys {
_ = reflect.TypeOf(y)
}
return len(ys)
}
type ReflectUnnamedStruct int
func (ReflectUnnamedStruct) UnnamedStructMethod(s struct{ UnnamedStructField string }) {
fmt.Println(reflect.TypeOf(s))
}
-- importedpkg2/imported2.go --
package importedpkg2
type ReflectInSiblingImport struct {
fix a number of issues involving types from indirect imports obfuscatedTypesPackage is used to figure out if a name in a dependency package was obfuscated or not. For example, if that package used reflection on a named type, it wasn't obfuscated, so we must have the same information to not obfuscate the same name downstream. obfuscatedTypesPackage could return nil if the package was indirectly imported, though. This can happen if a direct import has a function that returns an indirect type, or if a direct import exposes a name that's a type alias to an indirect type. We sort of dealt with this in two pieces of code by checking for obfPkg!=nil, but a third one did not have this check and caused a panic in the added test case: --- FAIL: TestScripts/reflect (0.81s) testscript.go:397: > env GOPRIVATE=test/main > garble build [stderr] # test/main panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8a5e39] More importantly though, the nil check only avoids panics. It doesn't fix the root cause of the problem: that importcfg does not contain indirectly imported packages. The added test case would still fail, as we would obfuscate a type in the main package, but not in the indirectly imported package where the type is defined. To fix this, resurrect a bit of code from earlier garble versions, which uses "go list -toolexec=garble" to fetch a package's export file. This lets us fill the indirect import gaps in importcfg, working around the problem entirely. This solution is still not particularly great, so we add a TODO about possibly rethinking this in the future. It does add some overhead and complexity, though thankfully indirect imports should be uncommon. This fixes a few panics while building the protobuf module.
3 years ago
SiblingObfuscated string
}
-- importedpkg/indirect/indirect.go --
package indirect
import "reflect"
var _ = reflect.TypeOf(IndirectNamedWithReflect{})
type IndirectNamedWithReflect struct {
IndirectUnobfuscated string
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
DuplicateFieldName int
fix a number of issues involving types from indirect imports obfuscatedTypesPackage is used to figure out if a name in a dependency package was obfuscated or not. For example, if that package used reflection on a named type, it wasn't obfuscated, so we must have the same information to not obfuscate the same name downstream. obfuscatedTypesPackage could return nil if the package was indirectly imported, though. This can happen if a direct import has a function that returns an indirect type, or if a direct import exposes a name that's a type alias to an indirect type. We sort of dealt with this in two pieces of code by checking for obfPkg!=nil, but a third one did not have this check and caused a panic in the added test case: --- FAIL: TestScripts/reflect (0.81s) testscript.go:397: > env GOPRIVATE=test/main > garble build [stderr] # test/main panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8a5e39] More importantly though, the nil check only avoids panics. It doesn't fix the root cause of the problem: that importcfg does not contain indirectly imported packages. The added test case would still fail, as we would obfuscate a type in the main package, but not in the indirectly imported package where the type is defined. To fix this, resurrect a bit of code from earlier garble versions, which uses "go list -toolexec=garble" to fetch a package's export file. This lets us fill the indirect import gaps in importcfg, working around the problem entirely. This solution is still not particularly great, so we add a TODO about possibly rethinking this in the future. It does add some overhead and complexity, though thankfully indirect imports should be uncommon. This fixes a few panics while building the protobuf module.
3 years ago
}
type IndirectNamedWithoutReflect struct {
IndirectObfuscated string
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
DuplicateFieldName int
}
-- main.stdout --
9000
{5 0 {}}
ReflectTypeOf
ReflectTypeOfIndirect
ReflectValueOf{ExportedField:"abc", unexportedField:""}
[method: abc]
{"Foo":3}
Hello Dave.
{"InnerField":3,"Anon":{"AnonField":0}}
{downstream}
{sibling}
stop loading obfuscated type information from deps If package P1 imports package P2, P1 needs to know which names from P2 weren't obfuscated. For instance, if P2 declares T2 and does "reflect.TypeOf(T2{...})", then P2 won't obfuscate the name T2, and neither should P1. This information should flow from P2 to P1, as P2 builds before P1. We do this via obfuscatedTypesPackage; P1 loads the type information of the obfuscated version of P2, and does a lookup for T2. If T2 exists, then it wasn't obfuscated. This mechanism has served us well, but it has downsides: 1) It wastes CPU; we load the type information for the entire package. 2) It's complex; for instance, we need KnownObjectFiles as an extra. 3) It makes our code harder to understand, as we load both the original and obfuscated type informaiton. Instead, we now have each package record what names were not obfuscated as part of its cachedOuput file. Much like KnownObjectFiles, the map records incrementally through the import graph, to avoid having to load cachedOutput files for indirect dependencies. We shouldn't need to worry about those maps getting large; we only skip obfuscating declared names in a few uncommon scenarios, such as the use of reflection or cgo's "//export". Since go/types is relatively allocation-heavy, and the export files contain a lot of data, we get a nice speed-up: name old time/op new time/op delta Build-16 11.5s ± 2% 11.1s ± 3% -3.77% (p=0.008 n=5+5) name old bin-B new bin-B delta Build-16 5.15M ± 0% 5.15M ± 0% ~ (all equal) name old cached-time/op new cached-time/op delta Build-16 375ms ± 3% 341ms ± 6% -8.96% (p=0.008 n=5+5) name old sys-time/op new sys-time/op delta Build-16 283ms ±17% 289ms ±13% ~ (p=0.841 n=5+5) name old user-time/op new user-time/op delta Build-16 687ms ± 6% 664ms ± 7% ~ (p=0.548 n=5+5) Fixes #456. Updates #475.
2 years ago
{{indirect-with 3} {indirect-without 4} { 0}}
IndirectNamedWithReflect{IndirectUnobfuscated:"indirect-with", DuplicateFieldName:3}
ReflectionField
{0}
VariadicReflection{ReflectionField:"variadic"}
*main.StatUser
*main.StatCompUser
struct { UnnamedStructField string }
TypeOfParent's own name: TypeOfParent
TypeOfParent named: TypeOfNamedField NamedReflectionField
TypeOfParent embedded: TypeOfEmbeddedField EmbeddedReflectionField