lib::tokenizer Class Reference

Zero-allocation streaming string tokenizer. More...

Public Member Functions
auto	empty () const noexcept -> bool
	True when all input has been consumed.
auto	next () noexcept -> std::optional< std::string_view >
	Extract the next token.
auto	next_pair () noexcept -> std::optional< std::pair< std::string_view, std::string_view > >
	Parse the next token as a key=value pair.
auto	peek () noexcept -> std::optional< std::string_view >
	Peek at the next token without consuming it.
auto	remaining () const noexcept -> std::string_view
	Return the unconsumed portion of the input.
auto	reset (std::string_view input) noexcept -> void
	Reset with a new input string.
auto	seek (usize pos) noexcept -> void
	Seek to a previously saved position.
auto	tell () const noexcept -> usize
	Return the current read position.
	tokenizer (std::string_view input, std::string_view delimiters=" \t\r\n") noexcept
	Construct a tokenizer over input.

Detailed Description

Zero-allocation streaming string tokenizer.

Constructor & Destructor Documentation

tokenizer()

lib::tokenizer::tokenizer ( std::string_view input, std::string_view delimiters = " \t\r\n" )

inlineexplicitexportnoexcept

Construct a tokenizer over input.

Parameters

input	The string to tokenize.
delimiters	Character set that separates tokens.

Member Function Documentation

next()

auto lib::tokenizer::next ( ) ->std::optional< std::string_view >

inlinenodiscardexportnoexcept

Extract the next token.

Returns

The next token, or nullopt if exhausted.

next_pair()

auto lib::tokenizer::next_pair ( ) ->std::optional< std::pair< std::string_view, std::string_view > >

inlinenodiscardexportnoexcept

Parse the next token as a key=value pair.

Returns

A (key, value) pair, or nullopt if exhausted.

---

The documentation for this class was generated from the following file:

• lib/string_parser.cxxm