fix a data race with the global cachedBinary mechanism (#413)

Spotted by our friend "go test -race":

	WARNING: DATA RACE
	Write at 0x0000010522d8 by goroutine 69:
	  mvdan.cc/garble.readFile()
		  garble/main_test.go:124 +0x23a
	  mvdan.cc/garble.binsubstr()
		  garble/main_test.go:141 +0xc4
	  github.com/rogpeppe/go-internal/testscript.(*TestScript).run()
		  github.com/rogpeppe/go-internal@v1.8.1-0.20211023094830-115ce09fd6b4/testscript/testscript.go:496 +0x9e8
	  [...]

	Previous write at 0x0000010522d8 by goroutine 60:
	  mvdan.cc/garble.readFile()
		  garble/main_test.go:124 +0x23a
	  mvdan.cc/garble.binsubstr()
		  garble/main_test.go:141 +0xc4
	  github.com/rogpeppe/go-internal/testscript.(*TestScript).run()
		  github.com/rogpeppe/go-internal@v1.8.1-0.20211023094830-115ce09fd6b4/testscript/testscript.go:496 +0x9e8
	  [...]

This wasn't a data race that we spotted via failures in practice,
as it only affected test code since July.

The race is due to the fact that each test script runs as a parallel
sub-test within the same Go program, sharing all globals.
As such, a single "cached binary" global is read and written with races.

Moreover, note that the caching always missed.
I briefly rewrote the code to avoid the race via a sync.Map keyed by
absolute filenames, and while that removed the data race,
the caching never actually hit.

To have a cache hit, we need an absolute path to already be in the cache
and for it to not have been modified since it was last cached. That is:

	modify-bin-1 foo
	binsubstr foo 'abc' # miss
	binsubstr foo 'def' # hit; use the cached "/tmp/[...]/foo" entry

	modify-bin-2 foo
	binsubstr foo 'abc' # miss

However, the test scripts don't do contiguous binsubstr calls like
these. Instead, they join repeated binsubstr calls:

	modify-bin-1 foo
	binsubstr foo 'abc' 'def' # miss

	modify-bin-2 foo
	binsubstr foo 'abc' # miss

For that reason, remove the extra code entirely.
I didn't notice any change to the performance of "go test -short"
with a warm build cache, with:

	go test -c
	./garble.test -test.short #warm cache
	benchcmd -n 5 TestShort ./garble.test -test.short

	name       old time/op         new time/op         delta
	TestShort          4.62s ±12%          4.35s ±12%   ~     (p=0.310 n=5+5)

	name       old user-time/op    new user-time/op    delta
	TestShort          16.8s ± 3%          16.7s ± 3%   ~     (p=0.690 n=5+5)

	name       old sys-time/op     new sys-time/op     delta
	TestShort          7.28s ± 1%          7.26s ± 2%   ~     (p=0.841 n=5+5)

	name       old peak-RSS-bytes  new peak-RSS-bytes  delta
	TestShort          305MB ± 0%          306MB ± 0%   ~     (p=0.421 n=5+5)

Finally, start using "go test -race" on Linux on CI,
which should have made the PR back in July red before merging.
pull/415/head
Daniel Martí 3 years ago committed by GitHub
parent d2203c4cce
commit 5e1e4d710b
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

@ -11,8 +11,8 @@ jobs:
strategy:
matrix:
go-version: [1.17.x]
platform: [ubuntu-latest, macos-latest, windows-latest]
runs-on: ${{ matrix.platform }}
os: [ubuntu-latest, macos-latest, windows-latest]
runs-on: ${{ matrix.os }}
steps:
- name: Install Go
uses: actions/setup-go@v2
@ -24,6 +24,12 @@ jobs:
run: |
go version
go test ./...
- name: Test with -race
# macos and windows tend to be a bit slower,
# and it's rare that a race in garble would be OS-specific.
if: matrix.os == 'ubuntu-latest'
run: |
go test -race ./...
test-gotip:
if: ${{ false }} # TODO: support Go 1.18, see https://github.com/burrowers/garble/issues/385

@ -16,7 +16,6 @@ import (
"runtime"
"strings"
"testing"
"time"
"github.com/google/go-cmp/cmp"
"github.com/rogpeppe/go-internal/goproxytest"
@ -101,31 +100,6 @@ func TestScripts(t *testing.T) {
testscript.Run(t, p)
}
type binaryCache struct {
name string
modtime time.Time
content string
}
var cachedBinary binaryCache
func readFile(ts *testscript.TestScript, file string) string {
file = ts.MkAbs(file)
info, err := os.Stat(file)
if err != nil {
ts.Fatalf("%v", err)
}
if cachedBinary.modtime == info.ModTime() && cachedBinary.name == file {
return cachedBinary.content
}
cachedBinary.name = file
cachedBinary.modtime = info.ModTime()
cachedBinary.content = ts.ReadFile(file)
return cachedBinary.content
}
func createFile(ts *testscript.TestScript, path string) *os.File {
file, err := os.Create(ts.MkAbs(path))
if err != nil {
@ -138,7 +112,7 @@ func binsubstr(ts *testscript.TestScript, neg bool, args []string) {
if len(args) < 2 {
ts.Fatalf("usage: binsubstr file substr...")
}
data := readFile(ts, args[0])
data := ts.ReadFile(args[0])
var failed []string
for _, substr := range args[1:] {
match := strings.Contains(data, substr)

Loading…
Cancel
Save