| Safe Haskell | None |
|---|---|
| Language | Haskell2010 |
DataFrame.IO.CSV.Fast
Description
SIMD-accelerated CSV reader: a C scanner finds delimiter positions,
then each column is parsed directly from the mmap'd byte slices into
typed column builders. Chunk-parallel with -threaded + +RTS -N.
Synopsis
- fastReadCsv :: FilePath -> IO DataFrame
- readCsvFast :: FilePath -> IO DataFrame
- fastReadTsv :: FilePath -> IO DataFrame
- readTsvFast :: FilePath -> IO DataFrame
- fastReadCsvWithOpts :: CsvReader
- fastReadTsvWithOpts :: ReadOptions -> FilePath -> IO DataFrame
- fastReadCsvWithSchema :: Schema -> FilePath -> IO DataFrame
- fastReadCsvProj :: [Text] -> FilePath -> IO DataFrame
- fastReadCsvFromBytes :: ByteString -> IO DataFrame
- fastReadCsvFromBytesWithSchema :: Schema -> ByteString -> IO DataFrame
- readSeparated :: Word8 -> ReadOptions -> FilePath -> IO DataFrame
- readSeparatedFromBytes :: Word8 -> ReadOptions -> ByteString -> IO DataFrame
- readSeparatedFromBytesChunks :: Int -> Word8 -> ReadOptions -> ByteString -> IO DataFrame
- getDelimiterIndices :: Word8 -> Int -> Vector Word8 -> IO (Vector CSize)
- data CsvParseError
- = CsvUnclosedQuote
- | CsvRaggedRow !Int !Int !Int
Documentation
fastReadCsv :: FilePath -> IO DataFrame Source #
Read a CSV file with the default options.
Example
ghci> D.fastReadCsv "./data/taxi.csv"
fastReadTsv :: FilePath -> IO DataFrame Source #
Read a TSV file with the default options.
Example
ghci> D.fastReadTsv "./data/taxi.tsv"
fastReadCsvWithOpts :: CsvReader Source #
Like fastReadCsv but takes a ReadOptions record. Use this when
you need to tune ragged-row handling, unclosed-quote handling, the
whitespace-trimming knob, or the column selection; otherwise stick with
fastReadCsv. The separator comes from columnSeparator, so this is a
CsvReader the lazy scan can drive.
Example
ghci> D.fastReadCsvWithOpts D.defaultReadOptions{D.readColumns = Just ["id", "name"]} "./data/customers.csv"
fastReadTsvWithOpts :: ReadOptions -> FilePath -> IO DataFrame Source #
TSV counterpart to fastReadCsvWithOpts; pins columnSeparator to tab.
Example
ghci> D.fastReadTsvWithOpts D.defaultReadOptions{D.safeRead = D.MaybeRead} "./data/taxi.tsv"
fastReadCsvWithSchema :: Schema -> FilePath -> IO DataFrame Source #
Read a CSV and coerce each column to the type declared in the
supplied Schema. Schema columns bypass inference: parsed straight from
file bytes as the declared type, avoiding row-1 misclassification.
Example
ghci> schema = D.makeSchema [("id", D.schemaType @Int)]
ghci> D.fastReadCsvWithSchema schema "./data/customers.csv"
fastReadCsvProj :: [Text] -> FilePath -> IO DataFrame Source #
Read a CSV and return only the named columns, in the order given.
Shorthand for readColumns. The SIMD scan and delimiter
classification still run over the whole file, but unnamed columns are
never extracted, parsed or allocated. Naming a column the file does not
have raises a ColumnsNotFoundException.
Example
ghci> D.fastReadCsvProj ["id", "name"] "./data/customers.csv"
fastReadCsvFromBytesWithSchema :: Schema -> ByteString -> IO DataFrame Source #
In-memory version of fastReadCsvWithSchema.
Example
ghci> schema = D.makeSchema [("id", D.schemaType @Int)]
ghci> D.fastReadCsvFromBytesWithSchema schema "id,name\n1,Ada\n"
readSeparated :: Word8 -> ReadOptions -> FilePath -> IO DataFrame Source #
Reads via a private (WriteCopy) mmap so the buffer is never copied.
The separator byte and columnSeparator in opts must agree; use
fastReadCsvWithOpts unless you need to pass a delimiter byte directly.
Example
ghci> D.readSeparated 44 D.defaultReadOptions "./data/taxi.csv" -- 44 = ','
readSeparatedFromBytes :: Word8 -> ReadOptions -> ByteString -> IO DataFrame Source #
Identical to readSeparated but reads from an already-loaded byte
buffer (e.g. a slice of a memory-mapped file). Zero-copy: the bytes are
only read, never modified or grown.
Example
ghci> D.readSeparatedFromBytes 44 D.defaultReadOptions "id,name\n1,Ada\n" -- 44 = ','
readSeparatedFromBytesChunks :: Int -> Word8 -> ReadOptions -> ByteString -> IO DataFrame Source #
readSeparatedFromBytes with a forced chunk count, bypassing the
capability/size gate. Intended for tests and benchmarks that need to pin
the parallel split; 1 forces the sequential path.
Example
ghci> D.readSeparatedFromBytesChunks 1 44 D.defaultReadOptions "id,name\n1,Ada\n" -- 44 = ','
getDelimiterIndices :: Word8 -> Int -> Vector Word8 -> IO (Vector CSize) Source #
Locate delimiter byte positions in the first originalLen bytes of
csvFile. Treats an unclosed quoted field at EOF as a hard error; callers
that want to suppress the exception can use getDelimiterIndicesPolicy
with BestEffort.
data CsvParseError Source #
Constructors
| CsvUnclosedQuote | The input ends with a quoted field that was never closed. The
PCLMUL quote-parity chain in the SIMD scanner treats an unmatched
|
| CsvRaggedRow !Int !Int !Int | A row has a different number of fields from the header. Only
raised when |
Instances
| Exception CsvParseError Source # | |
Defined in DataFrame.IO.CSV.Fast.Index Methods toException :: CsvParseError -> SomeException # fromException :: SomeException -> Maybe CsvParseError # displayException :: CsvParseError -> String # | |
| Show CsvParseError Source # | |
Defined in DataFrame.IO.CSV.Fast.Index Methods showsPrec :: Int -> CsvParseError -> ShowS # show :: CsvParseError -> String # showList :: [CsvParseError] -> ShowS # | |
| Eq CsvParseError Source # | |
Defined in DataFrame.IO.CSV.Fast.Index Methods (==) :: CsvParseError -> CsvParseError -> Bool # (/=) :: CsvParseError -> CsvParseError -> Bool # | |