ReadPromise Interface

In one of my projects written in Go, I’m working with streams of data to craft an output.

I don’t want to buffer the output in memory as the amount of data is unbounded and could exceed the amount of available memory. Nor do I want to buffer to disk because it’s slow and error prone. I want to transform these streams into an output stream I can immediately return as a HTTP response body or send as a HTTP request body.

In Go, ‘incoming’ streams of data are represented with the standard library io.Reader interface:

type Reader interface {
	Read(p []byte) (n int, err error)
}

Anything that implements this interface takes an input buffer and is expected to copy data into this buffer, return the number of bytes copied, and possibly an error if EOF is reached.

An example of something that implements the Reader interface is os.File.

What the Reader interface cannot tell you is the total number of bytes available to read. You could keep calling Read until you reach EOF to count, but then you’d need to buffer all of the data in memory for it to be useful.

To be able to know how much data there is in advance, I’ve come up with the ReadPromise interface:

type ReadPromise interface {
	io.Reader
	Bytes() int64
}

This is a simple extension to Reader that adds a Bytes() method that returns the number of bytes initially available to read.

Here’s a distilled version of what I wrote to make os.File implement this interface:

func newFileReadPromise(f *os.File) ReadPromise {
	info, err := f.Stat()

	var size int64
	if err == nil {
		size = info.Size()
	}

	return &readPromise{
		io.LimitReader(f, size),
		size,
	}
}

type readPromise struct {
	io.Reader
	Nb int64
}

func (f *readPromise) Bytes() int64 {
	return f.Nb
}

There are some important things to note here.

First is that the os.File is wrapped with io.LimitReader. This is done because the file could grow after the initial Stat() and therefore the return value of Bytes() would be smaller than the number of bytes actually readable via Read().

Secondly, this assumes that no bytes have already been read from the File. In my real code, I have a helper function that opens files and returns a ReadPromise so this assumption is not made.

Lastly, the readPromise struct is distinct from the ReadPromise interface. Although there’s nothing stopping you from throwing away the interface and just passing around the readPromise struct, this is friendlier to usage across across packages.

Using this pattern, I know the size of all of my input streams and can calculate the exact size of my output stream by adding them together. There’s specific reasons I need this that I won’t get into here.

Here’s some quick tests I wrote to demonstrate it in action. Notice how the ReadPromise for /dev/zero returns EOF on first read because Stat() says it’s zero bytes in size. As we know, in reality, this file is infinite in size.

func TestFileReadPromiseFiveBytes(t *testing.T) {
	f, _ := os.Open("fivebytes.txt")
	frp := newFileReadPromise(f)

	t.Log(frp.Bytes())
	t.Log(frp.Read(make([]byte, 5)))
	t.Log(frp.Read(make([]byte, 5)))

	/*
		=== RUN   TestFileReadPromise
		main_test.go:12: 5
		main_test.go:13: 5 <nil>
		main_test.go:14: 0 EOF
		--- PASS: TestFileReadPromise (0.00s)
	*/
}

func TestFileReadPromiseZero(t *testing.T) {
	f, _ := os.Open("/dev/zero")
	frp := newFileReadPromise(f)

	t.Log(frp.Bytes())
	t.Log(frp.Read(make([]byte, 5)))
	t.Log(frp.Read(make([]byte, 5)))

	/*
		=== RUN   TestFileReadPromiseZero
		main_test.go:31: 0
		main_test.go:32: 0 EOF
		main_test.go:33: 0 EOF
		--- PASS: TestFileReadPromiseZero (0.00s)
	*/
}