dataframe-fastcsv-1.4.1.0: SIMD-accelerated CSV reader for the dataframe library.
Safe HaskellNone
LanguageHaskell2010

DataFrame.IO.CSV.Fast

Description

SIMD-accelerated CSV reader: a C scanner finds delimiter positions, then each column is parsed directly from the mmap'd byte slices into typed column builders. Chunk-parallel with -threaded + +RTS -N.

Synopsis

Documentation

fastReadCsv :: FilePath -> IO DataFrame Source #

Read a CSV file with the default options.

Example

Expand
ghci> D.fastReadCsv "./data/taxi.csv"

readCsvFast :: FilePath -> IO DataFrame Source #

Alias for fastReadCsv.

Example

Expand
ghci> D.readCsvFast "./data/taxi.csv"

fastReadTsv :: FilePath -> IO DataFrame Source #

Read a TSV file with the default options.

Example

Expand
ghci> D.fastReadTsv "./data/taxi.tsv"

readTsvFast :: FilePath -> IO DataFrame Source #

Alias for fastReadTsv.

Example

Expand
ghci> D.readTsvFast "./data/taxi.tsv"

fastReadCsvWithOpts :: CsvReader Source #

Like fastReadCsv but takes a ReadOptions record. Use this when you need to tune ragged-row handling, unclosed-quote handling, the whitespace-trimming knob, or the column selection; otherwise stick with fastReadCsv. The separator comes from columnSeparator, so this is a CsvReader the lazy scan can drive.

Example

Expand
ghci> D.fastReadCsvWithOpts D.defaultReadOptions{D.readColumns = Just ["id", "name"]} "./data/customers.csv"

fastReadTsvWithOpts :: ReadOptions -> FilePath -> IO DataFrame Source #

TSV counterpart to fastReadCsvWithOpts; pins columnSeparator to tab.

Example

Expand
ghci> D.fastReadTsvWithOpts D.defaultReadOptions{D.safeRead = D.MaybeRead} "./data/taxi.tsv"

fastReadCsvWithSchema :: Schema -> FilePath -> IO DataFrame Source #

Read a CSV and coerce each column to the type declared in the supplied Schema. Schema columns bypass inference: parsed straight from file bytes as the declared type, avoiding row-1 misclassification.

Example

Expand
ghci> schema = D.makeSchema [("id", D.schemaType @Int)]
ghci> D.fastReadCsvWithSchema schema "./data/customers.csv"

fastReadCsvProj :: [Text] -> FilePath -> IO DataFrame Source #

Read a CSV and return only the named columns, in the order given. Shorthand for readColumns. The SIMD scan and delimiter classification still run over the whole file, but unnamed columns are never extracted, parsed or allocated. Naming a column the file does not have raises a ColumnsNotFoundException.

Example

Expand
ghci> D.fastReadCsvProj ["id", "name"] "./data/customers.csv"

fastReadCsvFromBytes :: ByteString -> IO DataFrame Source #

In-memory version of fastReadCsv.

Example

Expand
ghci> D.fastReadCsvFromBytes "id,name\n1,Ada\n"

fastReadCsvFromBytesWithSchema :: Schema -> ByteString -> IO DataFrame Source #

In-memory version of fastReadCsvWithSchema.

Example

Expand
ghci> schema = D.makeSchema [("id", D.schemaType @Int)]
ghci> D.fastReadCsvFromBytesWithSchema schema "id,name\n1,Ada\n"

readSeparated :: Word8 -> ReadOptions -> FilePath -> IO DataFrame Source #

Reads via a private (WriteCopy) mmap so the buffer is never copied. The separator byte and columnSeparator in opts must agree; use fastReadCsvWithOpts unless you need to pass a delimiter byte directly.

Example

Expand
ghci> D.readSeparated 44 D.defaultReadOptions "./data/taxi.csv"  -- 44 = ','

readSeparatedFromBytes :: Word8 -> ReadOptions -> ByteString -> IO DataFrame Source #

Identical to readSeparated but reads from an already-loaded byte buffer (e.g. a slice of a memory-mapped file). Zero-copy: the bytes are only read, never modified or grown.

Example

Expand
ghci> D.readSeparatedFromBytes 44 D.defaultReadOptions "id,name\n1,Ada\n"  -- 44 = ','

readSeparatedFromBytesChunks :: Int -> Word8 -> ReadOptions -> ByteString -> IO DataFrame Source #

readSeparatedFromBytes with a forced chunk count, bypassing the capability/size gate. Intended for tests and benchmarks that need to pin the parallel split; 1 forces the sequential path.

Example

Expand
ghci> D.readSeparatedFromBytesChunks 1 44 D.defaultReadOptions "id,name\n1,Ada\n"  -- 44 = ','

getDelimiterIndices :: Word8 -> Int -> Vector Word8 -> IO (Vector CSize) Source #

Locate delimiter byte positions in the first originalLen bytes of csvFile. Treats an unclosed quoted field at EOF as a hard error; callers that want to suppress the exception can use getDelimiterIndicesPolicy with BestEffort.

data CsvParseError Source #

Exceptions raised by the fast CSV parser. Catchable with catch or try.

Constructors

CsvUnclosedQuote

The input ends with a quoted field that was never closed. The PCLMUL quote-parity chain in the SIMD scanner treats an unmatched " as if the rest of the file were inside quotes, so we refuse to return a silently corrupted DataFrame and raise instead.

CsvRaggedRow !Int !Int !Int

A row has a different number of fields from the header. Only raised when fastCsvOnRaggedRow is set to RaiseOnRagged. Carries the 0-based row index, the expected field count, and the actual field count.