use more bits for the obfuscated name hashes (#248)

We've been using four base64 characters for obfuscated names for a
while. And that has mostly worked, since most packages only have up to a
few hundred exported or unexported names at a time.

However, we have already encountered two collisions in the wild, which
can be reproduced with one seed but not another:

	[...] PsaN.hQyW is a field, not a method

	[...] byte is not a type

In both of those cases, we happened to run into a collision by chance.
And that's not terribly unlikely to begin with; even with just 100
names, the probability of a collision was about 0.03%. It dramatically
goes up if there are more names; with 500, we're already around 0.75%.

It's clear that four base64 chars is not enough to properly avoid
collisions in the vast majority of cases. But how many characters are
enough? The target should be that, even with a very large package and
lots of names, we should still practically never have a collision.

I did some basic estimation with "lots of names" being ten thousand,
with "practically never" being a one in a million chance. We need to go
all the way up to eight characters to reach that probability.

It's entirely possible that 7 or even 6 characters would be enough for
most users. However, collisions result in confusing errors which are
also hard to reproduce for us unless we can use exactly the same seed
and source code for a build.

So, play it safe, and use 8 characters. The constant now also has
documentation explaining how we arrived at that figure.
pull/247/head
Daniel Martí 4 years ago committed by GitHub
parent 5e3ba2fc09
commit a223147093
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

@ -173,7 +173,34 @@ func hashWith(salt []byte, name string) string {
if name == "" {
panic("hashWith: empty name")
}
const length = 4
// hashLength is the number of base64 characters to use for the final
// hashed name.
// This needs to be long enough to realistically avoid hash collisions,
// but short enough to not bloat binary sizes.
// The namespace for collisions is generally a single package, since
// that's where most hashed names are namespaced to.
// Using a "hash collision" formula, and taking a generous estimate of a
// package having 10k names, we get the following probabilities.
// Most packages will have far fewer names, but some packages are huge,
// especially generated ones.
// We also have slightly fewer bits in practice, since the base64
// charset has 'z' twice, and the first base64 char is coerced into a
// valid Go identifier. So we must be conservative.
// Remember that base64 stores 6 bits per encoded byte.
// The probability numbers are approximated.
//
// length (base64) | length (bits) | collision probability
// -------------------------------------------------------
// 4 24 ~95%
// 5 30 ~4%
// 6 36 ~0.07%
// 7 42 ~0.001%
// 8 48 ~0.00001%
//
// We want collisions to be practically impossible, so we choose 8 to
// end up with a chance of about 1 in a million even when a package has
// thousands of obfuscated names.
const hashLength = 8
d := sha256.New()
d.Write(salt)
@ -181,7 +208,7 @@ func hashWith(salt []byte, name string) string {
io.WriteString(d, name)
sum := make([]byte, nameBase64.EncodedLen(d.Size()))
nameBase64.Encode(sum, d.Sum(nil))
sum = sum[:length]
sum = sum[:hashLength]
// Even if we are hashing a package path, we still want the result to be
// a valid identifier, since we'll use it as the package name too.

Loading…
Cancel
Save