In Rust, I want to read from a stream or file to fill a buffer, and detect EOF

by

In Rust, I want to read (using BufReader) into a buffer array to fill it. I want to read the exact number of bytes, except in case of EOF, where reading less is fine. I have implemented a helper method myself, as follows, and that works. However I wonder if there is no convenience method that already does exactly that.

pub fn read_fully<R: Read>(read: &mut R, buffer: &mut [u8]) -> Result<usize, Error> {
    let mut count: usize = 0;
    loop {
        let r = read.read(&mut buffer[count..])?;
        if r == 0 {
            return Ok(count);
        }
        count += r;
    }
}

With each existing method I found there seems to be a problem:

  • “read” may read less (e.g. for stdin).
  • “read_exact” doesn’t work well with EOF.
  • “read_to_end” expects a Vec and not an array, and will read everything, possibly expanding the vector.

  • 3

    read_to_end() is not what you want. It will read more bytes than wanted until the end of the file. I don’t think there is another method you can use, your custom code is fine.

    – 

  • 4

    One thing you may want to do is to ignore ErrorKind::Interrupted and just repeat the read in this case.

    – 

  • Yes I tried read_to_end() and found it can read more bytes… I changed the question. Thanks about the tip for ErrorKind::Interrupted: when reading about it, it seems the loop should ignore this and continue. This makes it more complicated, but it seems to make sense.

    – 

  • 1

    Quick note. You should take R: Read directly instead of &mut R per API guidelines

    – 

  • 1

    @ThomasMueller Mark it as mut (mut read: R).

    – 




Here’s an idea, which I’m not sure is better than yours, but it also works:

pub fn read_fully<R: Read>(read: &mut R, buffer: &mut [u8]) -> Result<usize, Error> {
    let mut data = Vec::with_capacity(buffer.len());
    read.take(buffer.len().try_into().unwrap())
        .read_to_end(&mut data)?;
    buffer[..data.len()].copy_from_slice(&data);
    Ok(data.len())
}

It’ll probably be much less efficient. Whether it’s clearer is up to you to decide.

It seems that ErrorKind::Interrupted should be ignored. So the solution would become:

pub fn read_fully<R: Read>(mut read: R, buffer: &mut [u8]) -> Result<usize, Error> {
    let mut count: usize = 0;
    loop {
        let r = read.read(&mut buffer[count..]);
        match r {
            Ok(0) => {
                return Ok(count)
            },
            Ok(x) => {
                count += x
            },
            Err(e) => {
                if e.kind() == ErrorKind::Interrupted {
                    // retry
                } else {
                    return Err(e);
                }
            }
        };
    }
}

Here’s how you can do it shorter and cleaner, but using an experimental API:

#![feature(read_buf, core_io_borrowed_buf)]

use std::io::{BorrowedBuf, Error, ErrorKind, Read};

pub fn read_fully<R: Read>(read: &mut R, buffer: &mut [u8]) -> Result<usize, Error> {
    let mut buffer = BorrowedBuf::from(buffer);
    match read.read_buf_exact(buffer.unfilled()) {
        Ok(()) => {}
        Err(err) if err.kind() == ErrorKind::UnexpectedEof => {}
        Err(err) => return Err(err),
    }
    Ok(buffer.len())
}

At this point, you may want to just have a BorrowedCursor as a parameter:

#![feature(read_buf, core_io_borrowed_buf)]

use std::io::{BorrowedCursor, Error, ErrorKind, Read};

pub fn read_fully<R: Read>(read: &mut R, buffer: BorrowedCursor<'_>) -> Result<(), Error> {
    match read.read_buf_exact(buffer) {
        Err(err) if err.kind() == ErrorKind::UnexpectedEof => Ok(()),
        v => v,
    }
}

This allows neat things like using an uninitialized buffer.

Leave a Comment