Dumb ways to die: Random Values in Pointers

Mmmm, pointy

I seem to be writing about one blog post a year at the moment. I always tell myself I want to write more, but it doesn’t happen. This, however, is the first post in a series. But I only have two ideas for posts, and one seems like a bad idea, so it could be a short series!

Every now and again I visit the Go slack, in particular the #performance channel and the #darkarts channel. I almost never say anything.

Recently someone brought up the question of whether they could put non-pointer values in unsafe.Pointer variables. The general response was “no, that’s a bad idea”. I agree, it seems like a bad idea.

But if we never explore bad ideas we’ll never … er, actually if we never explore bad ideas we’ll be absolutely fine.

So lets explore this bad idea.

So why is this a bad idea? The main thought is that you’ll perhaps crash Go’s garbage collector. The Go GC looks at every pointer visible to the program to see what memory is still in use and what memory can be freed. If it follows a pointer that’s not pointing to a valid memory address, it could crash.

So let’s try that. We’ll allocate a billion unsafe.Pointers and set them all to values that really aren’t likely to be valid pointers.

func TestRandomUnsafePointers(t *testing.T) {
	x := make([]unsafe.Pointer, 1e9)

	for i := range x {
		// Possible misuse of unsafe.Pointer? Definite misuse of unsafe.Pointer!
		x[i] = unsafe.Pointer(uintptr(i * 8))
	}

	runtime.GC()
    runtime.KeepAlive(x)
}

This code creates a slice of 1 billion unsafe.Pointers, then forces the GC to run. It doesn’t crash.

We can try again with truely(ish) random values. And do some silly things.

func TestRandomUnsafePointers2(t *testing.T) {
	x := make([]unsafe.Pointer, 1e9)

	for i := range x {
		// Possible misuse of unsafe.Pointer? Definite misuse of unsafe.Pointer!
		x[i] = unsafe.Pointer(uintptr(rand.Int64()))
	}

	runtime.GC()

	for range 10 {
		for i := range x {
			// Possible misuse of unsafe.Pointer? Definite misuse of unsafe.Pointer!
			x[i] = unsafe.Add(x[i], 3)
		}

		runtime.GC()
	}
    runtime.KeepAlive(x)
}

It still doesn’t crash.

What if we’re just not being smart enough?

Go may be looking at these values and thinking “well, if that’s a pointer it’s nothing I know about” and ignoring it. Go can interact with C, so it needs to be able to deal with memory that’s allocated outside of its control. I’ve also used memory allocated directly via syscalls with Go for many years without (fingers crossed) any issues.

Go may find it more difficult if the value of a pointer looks like memory it should care about. What if we store a value that was once a valid memory address allocated by Go itself, but we know is no longer valid?

Here we allocate a slice that’s big enough to always be allocated on the heap. We then take the address of the array that backs the slice, and place that in a uintptr. We know that Go does not treat uintptrs as pointers, so holding the value in a uintptr shouldn’t cause Go to hold onto the allocation.

If we then remove references to the slice, and force a GC, the memory should be freed.

	y := make([]int, 1e4)
	yptr := uintptr(unsafe.Pointer(unsafe.SliceData(y)))
	y = nil
	runtime.GC()

Now if we store this value in an unsafe.Pointer and run the GC again, we might run into trouble. Here’s the full test.

func TestUnsafePointerBadNumber(t *testing.T) {
	y := make([]int, 1e4)
	yptr := uintptr(unsafe.Pointer(unsafe.SliceData(y)))
	y = nil
	runtime.GC()
	runtime.GC()
	x := unsafe.Pointer(yptr)
	runtime.GC()
	runtime.GC()

	runtime.KeepAlive(x)
}

And indeed, this does cause a panic.

runtime: pointer 0xc000162000 to unallocated span span.base()=0xc000288000 span.limit=0xc000290000 span.state=0
runtime: found in object at *(0xc00005ff58+0x0)
object=0xc00005ff58 s.base()=0xc00005e000 s.limit=0xc000066000 s.spanclass=0 s.elemsize=2048 s.state=mSpanManual
:
:
 ...
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

If you’re wondering why there are multiple calls to runtime.GC(), so am I. It seems to make it crash more reliably.

What does that mean?

Go’s garbage collector is pretty robust and can handle a lot of abuse. You can store values in pointers that aren’t pointers the GC knows about, or aren’t even valid memory addresses.

But if you store a memory address that the GC thinks it controls, then it will panic. I think it’s panicing because if it finds a pointer to a piece of memory that it has freed, then it has perhaps not done its job properly. The panic is there primarily to detect and help diagnose GC bugs, and just happens to catch poor uses of unsafe.

If you store non-pointer values in unsafe.Pointer you’ll likely see no immediate problems. Your testing will likely not show any issues. But one day you’ll be flat on your face, and you’ll have no idea why.

My conclusions are as follows.

1. Don’t use unsafe.Pointer unless you really have to.
2. Don’t keep unsafe.Pointer values around unless you really really have to. Most safe operations with unsafe.Pointer use it only as a transient value while transforming something to something else.
3. You probably don’t have to.
4. Only store memory addresses in unsafe.Pointer.
5. Perhaps only memory addresses allocated outside of the Go runtime (e.g. by direct calls to the mmap syscall).
6. And even in that case Go’s not guaranteed not to change underneath you.