mem: Allocate at 4KiB boundaries in the fallback buffer pool. (#8705)

By rounding up to the nearest page, we avoid repeatedly allocating
similar sizes if requests happen to arrive in roughly increasing order.

The GCS client sends messages with 2MiB of data repeatedly when writing
a large object. Therefore it has to repeatedly allocate just over 2MiB.
This ultimately results in many, many allocations in the fallback buffer
pool. In practice rounding up yields at least a 10x reduction in RAM
when running 100 concurrent large writes. This is probably not unique to
GCS: anyone who sends large messages may be affected.

This change in simpleBufferPool seems worthwhile vs. adding a tier. We
use simpleBufferPool for any size greater than 1MiB, so this effectively
lets us discover a reasonably tight tier around any large message size
that comes in frequently. It increases infrequent allocation sizes by no
more than 0.4%.

RELEASE NOTES:
* mem: round up to nearest 4KiB for pool allocations larger than 1MiB

Author: cjc25

mem/buffer_pool.go

@@ -38,9 +38,11 @@ type BufferPool interface {
 	Put(*[]byte)
 }
 
+const goPageSize = 4 << 10 // 4KiB. N.B. this must be a power of 2.
+
 var defaultBufferPoolSizes = []int{
 	256,
-	4 << 10,  // 4KB (go page size)
+	goPageSize,
 	16 << 10, // 16KB (max HTTP/2 frame size used by gRPC)
 	32 << 10, // 32KB (default buffer size for io.Copy)
 	1 << 20,  // 1MB
@@ -172,7 +174,14 @@ func (p *simpleBufferPool) Get(size int) *[]byte {
 		p.pool.Put(bs)
 	}
 
-	b := make([]byte, size)
+	// If we're going to allocate, round up to the nearest page. This way if
+	// requests frequently arrive with small variation we don't allocate
+	// repeatedly if we get unlucky and they increase over time. By default we
+	// only allocate here if size > 1MiB. Because goPageSize is a power of 2, we
+	// can round up efficiently.
+	allocSize := (size + goPageSize - 1) & ^(goPageSize - 1)
+
+	b := make([]byte, size, allocSize)
 	return &b
 }