13. Read a large file

The task

Count the lines that contain ERROR in a log file. The file can be many gigabytes.

The versions

1. WorseReads the whole file into a String, copies every line into a Vec<String>, and panics if the file cannot be read.

   pub fn count_errors(path: &str) -> usize {
       let content = fs::read_to_string(path).unwrap(); // the whole file in memory
       let lines: Vec<String> = content.lines().map(|l| l.to_string()).collect(); // and again
       let mut count = 0;
       for line in lines {
           if line.contains("ERROR") {
               count += 1;
           }
       }
       count
   }

2. OKReads line by line with BufReader::lines(), and returns io::Result. One new String per line.

   pub fn count_errors(path: &Path) -> io::Result<usize> {
       let reader = BufReader::new(File::open(path)?);
       let mut count = 0;
       for line in reader.lines() {
           if line?.contains("ERROR") {
               count += 1;
           }
       }
       Ok(count)
   }

3. BetterReuses one String buffer with read_line, and takes any impl BufRead, so tests need no files.

   pub fn count_errors(path: &Path) -> io::Result<usize> {
       count_errors_in(BufReader::new(File::open(path)?))
   }
   
   /// Works on any buffered reader: a file, a network stream, or bytes in a test.
   pub fn count_errors_in(mut reader: impl BufRead) -> io::Result<usize> {
       let mut line = String::new(); // one buffer, reused for every line
       let mut count = 0;
       while reader.read_line(&mut line)? > 0 {
           if line.contains("ERROR") {
               count += 1;
           }
           line.clear();
       }
       Ok(count)
   }

The “Worse” version is Files.readAllLines with two extra problems:

• The whole file is in memory twice. read_to_string loads all of it, and the Vec<String> copies every line again. A 4 GB log needs about 8 GB of memory.
• It panics on a missing file. unwrap() turns “file not found” or “permission denied” into a crash. The test worse_panics_on_a_missing_file shows it. See S02: unwrap() / expect() in non-test code and S36: panic! used for normal errors.
• It also fails on invalid UTF-8. read_to_string returns an error for a file that is not valid UTF-8, and the unwrap() turns that into a panic too. Log files with one bad byte are common.

The “OK” version streams the file. BufReader reads in 8 KB blocks, and lines() gives one io::Result<String> per line. Memory use is one line at a time. File::open(path)? and line? pass I/O errors to the caller. This is the idiomatic answer, and it is fine for most programs.

The “Better” version removes the last cost: lines() allocates a new String for each line. read_line appends into a buffer that the loop reuses, so a file with a hundred million lines needs one allocation. The function also takes impl BufRead instead of a path: the same code works for a file, standard input, or a network stream, and the test better_reads_from_bytes passes a byte string with no file at all.

Java habit

There is no try-with-resources and no close(). The File inside the BufReader closes when the reader is dropped, at the end of the function, also when ? returns early with an error.

Verdict

Each version compared with the Worse version

Criterion	OK	Better
Correctness	● Same	● Same
All three count the same lines in a valid file.
Failure behavior	▲ Better	▲ Better
"Worse" panics on a missing file or invalid UTF-8. "OK" and "Better" return the io::Error.
Performance	▲ Better	▲ Better
"Worse" needs memory for the whole file twice. "OK" allocates one String per line. "Better" allocates one buffer in total.
Readability	● Same	● Same
"OK" is the shortest correct version. "Better" needs line.clear(), which is easy to forget.
API ergonomics	▲ Better	▲ Better
Both take &Path and return io::Result. "Better" also works on any reader, which makes it easy to test.

Trade-offs

• fs::read_to_string is correct for small files: a configuration file, a template. It is simpler, and the size is known to be small. It still needs ? instead of unwrap().
• line.clear() is required in the “Better” loop. Without it, read_line appends to the previous lines, and the count is wrong. A reviewer should check for it.
• For logs that may contain invalid UTF-8, read bytes with read_until(b'\n', &mut bytes) into a Vec<u8>, and search the bytes. Both lines() and read_line return an error for invalid UTF-8.
• The difference between “OK” and “Better” matters only for very large files. For most programs, “OK” is the right choice because it is the most common and readable.

Related

• Smells: S02: unwrap() / expect() in non-test code, S36: panic! used for normal errors, S11: collect() then iterate again.
• Concepts: Result, not exceptions, Strings, Iterators vs streams.