State machines and stateful parsing

Reach for split(',') and CSV looks solved in one line. Then a field contains a comma: the row "a,b",c is meant to hold two fields, a,b and c. Split on every comma and you get three pieces ("a, b", c), the quotes still attached and the first field torn in half.

The fix is to stop treating every comma as a separator. Walk the input one character at a time and track a single piece of state: am I currently inside a quoted field? A comma inside quotes is data; a comma outside quotes is a separator.

This "for each character, update some state, occasionally emit a result" pattern is called a state machine. It comes up in any non-trivial parsing task: JSON, command-line arguments, terminal escape sequences, markup languages.

A skeleton

fn parse(line: &str) -> Vec<String> {
    let mut fields = Vec::new();
    let mut current = String::new();
    let mut in_quotes = false;
    let mut chars = line.chars().peekable();

    while let Some(c) = chars.next() {
        match (c, in_quotes) {
            ('"', false) => in_quotes = true,
            ('"', true) if chars.peek() == Some(&'"') => {
                // Escaped quote inside a quoted field.
                current.push('"');
                chars.next();
            }
            ('"', true) => in_quotes = false,
            (',', false) => {
                fields.push(std::mem::take(&mut current));
            }
            (c, _) => current.push(c),
        }
    }
    fields.push(current);
    fields
}

Each less familiar tool removes one bit of bookkeeping from the loop:

• peekable() lets you look at the next character without consuming it. That lookahead matters when one character's meaning depends on the one after it, as in the "" -> " rule.
• match on a tuple (c, in_quotes) lets you express each transition as one arm. The alternatives stay flatter than they would with nested if/else blocks.
• std::mem::take gives you the current string and replaces it with an empty one in a single move. The old buffer moves into fields without a clone.

A note on while let

You used if let with Option earlier; while let repeats the same pattern until it stops matching. That extra control matters here because the loop sometimes calls chars.next() again to consume the second ".

You also used tuple patterns in let (a, b) = pair; here, match inspects the character and quote state together. A guard adds the lookahead check only to the escaped-quote arm.

These tests use the raw strings you met in the env-file parser, so the CSV examples can contain literal commas and quotes without an escape forest.

A useful tactic

When stateful parsing gets hairy, write the simple version first (split_once, split(',')) and let the easy tests pass. Then upgrade to the state-machine version for the harder cases. Failing tests give you concrete examples to think against, instead of trying to imagine every edge case up front.

For real CSV in production code, reach for the csv crate; it handles all the corners that this exercise glosses over. This exercise focuses on the state-machine loop rather than a production-ready CSV implementation.

A first pass: comma-splitting

Before tackling the messy realities of CSV (quotes, escapes, embedded commas), let's handle the trivial case: a line that's nothing but plain values separated by commas, possibly with surrounding whitespace. This is what most "I'll just split on commas" CSV parsers do, and it's also why so many of them break.

Use str::split and str::trim. Collect into a Vec<String>.

Useful from the standard library

• str::split with a ',' argument yields each comma-separated piece as a &str.
• str::trim drops leading/trailing whitespace from each piece.
• str::to_string in a map step turns the borrowed pieces into the owned Strings the return type wants.
• Iterator::collect finishes the chain. The body fits on one line: line.split(',').map(|s| s.trim().to_string()).collect().

Quotes, embedded commas, and escapes

Real CSV is a state machine in disguise. A field can be wrapped in double quotes, in which case any commas inside the quotes are part of the field, not separators. And a literal " inside a quoted field is encoded as "" (two quotes).

Implement the cases in the order shown by the tests:

1. Plain a,b,c and simply quoted "a","b","c" (the basic test).
2. Commas inside quoted fields: "a,b",c.
3. Escaped quotes: "a""b",c -> [a"b, c].

Walk the string character by character with a peekable iterator and keep a small in_quotes: bool flag. When you see " while already inside quotes, peek the next char: if it's another ", push a literal " and consume both; otherwise close the field.

Useful from the standard library

• str::chars is the entry point for character-level iteration.
• Iterator::peekable wraps the iterator so you can look ahead one character. Essential for the "" -> " rule.
• Peekable::peek returns Option<&Item> without advancing.
• std::mem::take swaps the current String with a fresh empty one in a single move. Cleaner than current.clone() followed by current.clear().
• A match (c, in_quotes) on the tuple lets you express each state transition as a single arm. Add a guard (if chars.peek() == Some(&'"')) for the escape rule.

Parsing a whole file

With a working line parser, the file-level parser is mostly plumbing: split on newlines, treat the first line as headers, and parse the rest as data rows.

Use str::lines to split: it handles trailing newlines gracefully, so "a,b\n" gives one line, not two.

You'll reuse parse_csv_line from the previous page. To keep this page independently runnable, its signature is re-declared here as a stub with todo!(). Paste your earlier solution into the stub or call into it.

Useful from the standard library

• str::lines yields each line as a &str, stripping \n and \r\n. A trailing newline does not create an empty trailing line.
• Iterator::next on the iterator pulls off the header line; an empty file should return empty headers and rows.
• Iterator::map + parse_csv_line over the remaining lines builds the rows.
• Iterator::collect to materialize both the headers and the rows into Vecs.

Wrapping up the CSV parser

The parser begins with split and trim, then adds state for quoted fields and escaped quotes before collecting headers and rows.

What we learned

• The same stateful parsing pattern appears in JSON, command lines, and terminal escape sequences. In each case, read one item, consult the current state, then update the state or emit a result.
• A peekable iterator lets you inspect what comes next without consuming it, as the "" -> " escape rule requires.
• match (token, state) { ... } over a tuple expresses each state transition in one line. Match guards (if cond) handle the cases where the transition depends on the lookahead.
• std::mem::take(&mut s) gives you the current value and replaces it with Default in one move. Cleaner than clone-then-clear when you're harvesting an accumulator.
• The simple split/trim version is worth writing first. It passes the easy tests and gives you a baseline; the state-machine upgrade then has concrete failing cases to react to.
• Hand production CSV files to the csv crate, which handles BOMs, custom delimiters, and escaped newlines inside fields. Keep the state-machine loop for parsers you do need to write yourself.