A runtime tag for a column’s element type.

Construct a SchemaType for the given a.

Examples

>>> :set -XTypeApplications
>>> schemaType @T.Text == schemaType @T.Text
True

>>> show (schemaType @Double)
"Double"

Construct a Schema from a list of (columnName, schemaType) pairs.

Derive an untyped Schema value plus per-column Expr accessors from a record type. The value-level companion to deriveSchemaFromType: $(deriveSchemaValues ''Order) generates orderSchema.

CSV read parameters.

How a reader decides each column's type.

Example

ghci> D.readCsvWithOpts D.defaultReadOptions{D.typeSpec = D.InferFromSample 500} "wide.csv"

Where a reader's column names come from.

Example

ghci> D.readCsvWithOpts D.defaultReadOptions{D.headerSpec = D.ProvideNames ["a", "b"]} "no_header.csv"

The default ReadOptions: infer types from a 100-row sample, treat the first row as a header, read every row and every column.

Example

ghci> D.readCsvWithOpts D.defaultReadOptions{D.columnSeparator = ';'} "data.csv"

Read CSV file from path and load it into a dataframe.

Example

ghci> D.readCsv "./data/taxi.csv"

Schema-driven CSV reader. Coerces each column to the type declared in Schema; columns absent from the schema fall back to inference and are still returned. To read only the schema's columns, pass schemaReadOptions to readCsvWithOpts.

import qualified DataFrame as D
df <- D.readCsvWithSchema schema "input.csv"

Read CSV file from path and load it into a dataframe.

Example

ghci> D.readCsvWithOpts "./data/taxi.csv" (D.defaultReadOptions { dateFormat = "%d%-m%-Y" })

Read TSV (tab separated) file from path and load it into a dataframe.

Example

ghci> D.readTsv "./data/taxi.tsv"

Read text file with specified delimiter into a dataframe.

Example

ghci> D.readSeparated (D.defaultReadOptions { columnSeparator = ';' }) "./data/taxi.txt"

Write a dataframe to a comma-separated file.

Example

ghci> D.writeCsv "./out.csv" df

Write a dataframe to a file using the given field separator.

Example

ghci> D.writeSeparated ';' "./out.txt" df

Parse a CSV string into a DataFrame using default options.

Example

ghci> D.fromCsv "id,name\n1,Ada\n"
Right (DataFrame ...)

Parse a lazy ByteString containing CSV data into a DataFrame using default options.

Example

ghci> D.fromCsvBytes "id,name\n1,Ada\n"

Options for reading Parquet data.

These options are applied in this order:

1. predicate filtering
2. column projection
3. row range
4. safe column promotion

Column selection for selectedColumns uses leaf column names only.

Default Parquet read options.

Equivalent to:

ParquetReadOptions
    { selectedColumns = Nothing
    , predicate = Nothing
    , rowRange = Nothing
    , safeColumns = False
    }

Read a parquet file from path and load it into a dataframe.

Example

ghci> D.readParquet "./data/mtcars.parquet"

Read a Parquet file using explicit read options.

Example

ghci> D.readParquetWithOpts
ghci|   (D.defaultParquetReadOptions{D.selectedColumns = Just ["id"], D.rowRange = Just (0, 10)})
ghci|   ".testsdata/alltypes_plain.parquet"

Read Parquet files from a directory or glob path.

This is equivalent to calling readParquetFilesWithOpts with defaultParquetReadOptions.

Read multiple Parquet files (directory or glob) using explicit options.

If path is a directory, all non-directory entries are read. If path is a glob, matching files are read.

For multi-file reads, rowRange is applied once after concatenation (global range semantics).

Example

ghci> D.readParquetFilesWithOpts
ghci|   (D.defaultParquetReadOptions{D.selectedColumns = Just ["id"], D.rowRange = Just (0, 5)})
ghci|   ".testsdata/alltypes_plain*.parquet"

A lazy query that has not been executed yet: a LogicalPlan tree whose execution is deferred until runDataFrame is called.

Execute the lazy query: optimise the logical plan, then stream-execute the resulting physical plan into a fully-materialised DataFrame.

Lift an already-loaded eager DataFrame into the lazy plan.

Scan a CSV file with the default comma separator and the in-tree strict reader. For the SIMD reader use scanCsvWith.

The Schema both types and selects: only the columns it names are read, matching scanParquet.

Example

ghci> schema = D.makeSchema [("id", D.schemaType @Int), ("name", D.schemaType @Text)]
ghci> L.runDataFrame (L.scanCsv schema "customers.csv")

Like scanCsv but with an explicit CSV reader (e.g. the SIMD reader fastReadCsvWithOpts from dataframe-fastcsv). The scan derives the reader's ReadOptions from the schema and separator, so any CsvReader projects.

Example

ghci> import qualified DataFrame.IO.CSV.Fast as Fast
ghci> L.runDataFrame (L.scanCsvWith Fast.fastReadCsvWithOpts schema "customers.csv")

Scan a character-separated file with the default strict reader.

Example

ghci> L.runDataFrame (L.scanSeparated ';' schema "customers.txt")

Like scanSeparated but with an explicit CSV reader.

Example

ghci> L.runDataFrame (L.scanSeparatedWith Fast.fastReadCsvWithOpts ';' schema "customers.txt")

Scan a Parquet file, directory of files, or glob pattern.

How parse failures are surfaced: NoSafeRead throws, MaybeRead yields Nothing (column wrapped Maybe a), EitherRead yields Left rawText (column wrapped Either Text a, preserving the original input).

Options controlling how text columns are parsed into typed values.

Sensible out-of-the-box parse options: infer from the first 100 rows, treat common nullish strings as missing, and expect ISO 8601 dates.

Resolve a column's effective SafeReadMode: the override if present, otherwise the default.

O(n * k) Filter rows by a given condition.

filter "x" even df

Sample a dataframe. The double parameter must be between 0 and 1 (inclusive).

Example

ghci> import System.Random
ghci> D.sample (mkStdGen 137) 0.1 df

O(k * n) Take a range of rows of a DataFrame.

O(k * n) Take the first n rows of a DataFrame.

O(k * n) Drop the first n rows of a DataFrame.

O(n) Selects a number of columns in a given dataframe.

select ["name", "age"] df

O(k * n) Take the last n rows of a DataFrame.

O(k * n) Drop the last n rows of a DataFrame.

O(k) a version of filter where the predicate comes first.

filterBy even "x" df

O(k) filters the dataframe with a boolean expression.

filterWhere (F.col @Int x + F.col y F.> 5) df

O(k) removes all rows with Nothing in a given column from the dataframe.

filterJust "col" df

O(k) returns all rows with Nothing in a give column.

filterNothing "col" df

O(n * k) removes all rows with Nothing from the dataframe.

filterAllJust df

O(n * k) keeps any row with a null value.

filterAllNothing df

O(k) cuts the dataframe in a cube of size (a, b) where a is the length and b is the width.

cube (10, 5) df

Criteria for selecting a column by name.

selectBy [byName "Age"] df

equivalent to:

select ["Age"] df

Criteria for selecting columns whose property satisfies given predicate.

selectBy [byProperty isNumeric] df

Criteria for selecting columns whose name satisfies given predicate.

selectBy [byNameProperty (T.isPrefixOf "weight")] df

Criteria for selecting columns whose names are in the given lexicographic range (inclusive).

selectBy [byNameRange ("a", "c")] df

Criteria for selecting columns whose indices are in the given (inclusive) range.

selectBy [byIndexRange (0, 5)] df

O(n) select columns by column predicate name.

O(k * n) select rows by index

selectRows [0, 2, 4] df

O(n) inverse of select

exclude ["Name"] df

Split a dataset into two. The first in the tuple gets a sample of p (0 <= p <= 1) and the second gets (1 - p). This is useful for creating test and train splits.

Example

ghci> import System.Random
ghci> D.randomSplit (mkStdGen 137) 0.9 df

Creates n folds of a dataframe.

Example

ghci> import System.Random
ghci> D.kFolds (mkStdGen 137) 5 df

Sample a dataframe, preserving per-stratum proportions.

Example

ghci> import System.Random
ghci> D.stratifiedSample (mkStdGen 42) 0.8 "label" df

Split a dataframe into two, preserving per-stratum proportions.

Example

ghci> import System.Random
ghci> D.stratifiedSplit (mkStdGen 42) 0.8 "label" df

O(k) Apply a function to a given column in a dataframe.

O(k) Apply a function to an expression in a dataframe and add the result into alias column.

O(k) Safe version of the apply function. Returns (instead of throwing) the error.

O(k) Apply a function to an expression in a dataframe and add the result into alias column but

Examples

>>> (z, df') = deriveWithExpr "z" (F.col @Int "x" + F.col "y") df
>>> filterWhere (z .>= 50)

O(k * n) Apply a function to given column names in a dataframe.

O(k) Convenience function that applies to an int column.

O(k) Convenience function that applies to an double column.

O(k * n) Apply a function to a column only if there is another column value that matches the given criterion.

applyWhere (<20) "Age" (const "Gen-Z") "Generation" df

O(k) Apply a function to the column at a given index.

Replace all instances of Nothing in a column with the given value. Throws when the column carries no nulls to replace.

O(k * n) group the dataframe by the given key columns, bucketing rows with an open-addressing hash table that re-verifies keys on each hash hit. Groups are numbered in first-appearance order; valueIndices/offsets follow by counting sort.

Aggregate a grouped dataframe using the expressions given. All ungrouped columns will be dropped.

Filter out all non-unique values in a dataframe.

O(k log n) Sorts the dataframe by a given row.

sortBy Ascending ["Age"] df

Sort order taken as a parameter by the sortBy function.

Join two dataframes using SQL join semantics.

Performs an inner join on two dataframes using the specified key columns. Returns only rows where the key values exist in both dataframes.

Example

ghci> df = D.fromNamedColumns [("key", D.fromList [K0, K1, K2, K3]), (A, D.fromList [A0, A1, A2, A3])]
ghci> other = D.fromNamedColumns [("key", D.fromList [K0, K1, K2]), (B, D.fromList [B0, B1, B2])]
ghci> D.innerJoin ["key"] df other

-----------------
 key  |  A  |  B
------|-----|----
 Text | Text| Text
------|-----|----
 K0   | A0  | B0
 K1   | A1  | B1
 K2   | A2  | B2

Performs a left join on two dataframes using the specified key columns. Returns all rows from the left dataframe, with matching rows from the right dataframe. Non-matching rows will have Nothing/null values for columns from the right dataframe.

Example

ghci> df = D.fromNamedColumns [("key", D.fromList [K0, K1, K2, K3]), (A, D.fromList [A0, A1, A2, A3])]
ghci> other = D.fromNamedColumns [("key", D.fromList [K0, K1, K2]), (B, D.fromList [B0, B1, B2])]
ghci> D.leftJoin ["key"] df other

------------------------
 key  |  A  |     B
------|-----|----------
 Text | Text| Maybe Text
------|-----|----------
 K0   | A0  | Just B0
 K1   | A1  | Just B1
 K2   | A2  | Just B2
 K3   | A3  | Nothing

Performs a right join on two dataframes using the specified key columns. Returns all rows from the right dataframe, with matching rows from the left dataframe. Non-matching rows will have Nothing/null values for columns from the left dataframe.

Example

ghci> df = D.fromNamedColumns [("key", D.fromList [K0, K1, K2, K3]), (A, D.fromList [A0, A1, A2, A3])]
ghci> other = D.fromNamedColumns [("key", D.fromList [K0, K1]), (B, D.fromList [B0, B1])]
ghci> D.rightJoin ["key"] df other

-----------------
 key  |  A  |  B
------|-----|----
 Text | Text| Text
------|-----|----
 K0   | A0  | B0
 K1   | A1  | B1

Equivalent to SQL join types.

Bridge a Haskell record type a to a typed-dataframe schema.

The schema is exposed as an associated type family Schema so that instances can pick it up from a Rep computation (see SchemaOf) or from an explicit list emitted by deriveSchemaFromType.

toColumns explodes a list of records into a list of named columns. fromColumns reconstructs the records from a DataFrame, returning Left err if a column is missing or has the wrong type.

Build an untyped DataFrame from a list of records.

data Order = Order { orderId :: Int64, region :: Text, amount :: Double }
$(deriveSchemaFromType ''Order)

xs :: [Order]
xs = [Order 1 "us" 10.0, Order 2 "eu" 20.0]

df :: DataFrame
df = fromRecords xs

Parse a list of records out of an untyped DataFrame.

Returns Left err on schema mismatch (missing column, wrong type).

The column names a schema declares, in schema order. Pass it to a reader's options to fetch only those columns:

D.readCsvWithOpts
    D.defaultReadOptions{D.readColumns = Just (schemaColumnNames @(Schema Customer))}
    "customers.csv"

Splice a binding for every column of the DataFrame read from a CSV file. Each binding has type Expr T where T is the inferred column type.

Like declareColumnsFromCsvFile but with custom ReadOptions.

Splice a binding for every column of a parquet file (or directory of parquet files). The schema is read from each file's metadata and merged.

Splice a binding for every column of df, named after the column. Column names that are not valid Haskell identifiers are sanitized (see sanitize).

Like declareColumns but prefixes every binding name with prefix_.

Like declareColumnsWithPrefix but takes an optional prefix.