data.checker is a package for helping with boilerplate data checks. It enables you to automate fundamental data checks which, while simple, can be time-consuming to implement.

data.checker

• Checks data against a user supplied schema that defines what columns and data types are expected

• Enables user to add additional custom data checks based on multiple columns

• Creates exports of the results for QA

Getting Started

Installation

Software requirements

To use this package, you’ll need the following software on your computer:

1. RStudio 2024.04.2 or later and R 4.5.0 or later
2. GIT 2.35.3 or later

To install this R package, you will first need to clone the repository to you local machine by running

git clone https://github.com/ONSdigital/data.checker.git

Open the project in RStudio and in the console run:

devtools::install()

The package will be installed in you R library.

Setup and Usage

data.checker requires an input dataframe and a data schema to validate against. A full list of checks performed by the data checker, alongside how to include custom checks can be found here. The schema can either be defined within the R script itself or saved to either a JSON or YAML file to be loaded by the data checker. We recommend that schemas be saved as either a JSON or YAML to simplify the process of adding additional checks and column information. Once defined, we can pass both the dataframe and schema, alongside an output filepath and format for the report and the option for hardchecks into the check_and_export function.

libary(data.checker)
df <- data.frame(
  age = c(10, 11, 13, 15, 22, 34, 80),
  sex = c("M", "F", "M", "F", "M", "F", "M")
)
my_schema <- list(
  check_duplicates = TRUE,
  check_completeness = FALSE,
  columns = list(
    age = list(type = "integer", optional = FALSE),
    sex = list(type = "character", optional = FALSE)
  )
)
check_and_export(data = df,
         schema = my_schema,
         file = "report.csv",
         format = "csv",
         hard_check =TRUE)

This will produce a report.csv containing the status of each of the validation checks. With hard_check set to TRUE, this will mean the code stops running if any validation checks fail. The report will still be produced before this stop so you can view and investigate the issue causing a fail.

Pre-Defined and Adding Custom Checks

Pre-Defined Checks

These checks can be included in the lists for individual columns in your schema, depending on the data type.