Overview
DataAudit is an R package for systematic data-quality auditing and
validation.
It provides tools for detecting common data-quality problems, defining reusable validation rules, generating structured audit reports, and calculating an overall data-quality score.
DataAudit supports checks for missing values, blank values, duplicates, infinite values, constant and near-zero variance variables, outliers, ranges, categories, data types, patterns, dates, identifiers, lengths, character case, whitespace, sequences, dependencies, uniqueness, and cross-variable consistency.
Installation
You can install DataAudit from GitHub with:
# install.packages("pak") pak::pak("vinodhpmd/DataAudit")
Then load the package:
library(DataAudit)Quick example
Create a dataset containing several data-quality problems:
dat <- data.frame( ID = c( "A001", "A002", "A002", "BAD" ), Species = c( "Dog", "Horse", "", "Cat" ), Age = c( 5, 40, -2, NA ), stringsAsFactors = FALSE ) dat
## ID Species Age
## 1 A001 Dog 5
## 2 A002 Horse 40
## 3 A002 -2
## 4 BAD Cat NA
General data audit
Use audit_data() for a general first-pass audit:
report <- audit_data( dat, include_nzv = FALSE ) report
## DataAudit Report
## ================
##
## Rows: 4
## Variables: 3
## Variables with issues: 2
## Duplicated rows: 0
## Total issues: 2
## Rule violations: 0
## Overall issues: 2
##
## Variables with issues
## ---------------------
## Variable Class Missing MissingPercent Blank Infinite Unique Constant
## Species character 0 0 1 0 4 FALSE
## Age numeric 1 25 0 0 3 FALSE
## NearZeroVariance Issue
## FALSE TRUE
## FALSE TRUE
##
## No duplicated rows detected.
Detailed results are available through:
report$overview
## Metric Value
## 1 Rows 4
## 2 Variables 3
## 3 Missing values 1
## 4 Blank values 1
## 5 Infinite values 0
## 6 Duplicated rows 0
## 7 Constant variables 0
## 8 Near-zero variance variables 0
## 9 Variables with issues 2
report$variables
## Variable Class Missing MissingPercent Blank Infinite Unique
## ID ID character 0 0 0 0 3
## Species Species character 0 0 1 0 4
## Age Age numeric 1 25 0 0 3
## Constant NearZeroVariance Issue
## ID FALSE FALSE FALSE
## Species FALSE FALSE TRUE
## Age FALSE FALSE TRUE
report$duplicates
## Row Duplicate
## 1 1 FALSE
## 2 2 FALSE
## 3 3 FALSE
## 4 4 FALSE
report$summary
## Rows Variables VariablesWithIssues DuplicateRows TotalIssues RuleViolations
## 1 4 3 2 0 2 0
## OverallIssues
## 1 2
Individual checks
Specific checks can also be performed independently.
missing_check(dat)## Variable Total Missing Complete MissingPercent CompletePercent
## 1 ID 4 0 4 0 100
## 2 Species 4 0 4 0 100
## 3 Age 4 1 3 25 75
blank_check(dat)## Variable TextVariable Missing Blank BlankPercent HasBlank
## ID ID TRUE 0 0 0 FALSE
## Species Species TRUE 0 1 25 TRUE
## Age Age FALSE 1 0 0 FALSE
duplicate_check(dat)## Row Duplicate
## 1 1 FALSE
## 2 2 FALSE
## 3 3 FALSE
## 4 4 FALSE
range_check( dat, variable = "Age", min = 0, max = 30 )
## Row Value BelowMinimum AboveMaximum OutOfRange
## 1 1 5 FALSE FALSE FALSE
## 2 2 40 FALSE TRUE TRUE
## 3 3 -2 TRUE FALSE TRUE
## 4 4 NA FALSE FALSE FALSE
category_check( dat, variable = "Species", allowed = c( "Dog", "Cat" ) )
## Row Value Missing ValidCategory InvalidCategory
## 1 1 Dog FALSE TRUE FALSE
## 2 2 Horse FALSE FALSE TRUE
## 3 3 FALSE FALSE TRUE
## 4 4 Cat FALSE TRUE FALSE
Validation rules
Reusable validation rules can be defined with audit_rules():
rules <- audit_rules( range = list( Age = c(0, 30) ), category = list( Species = c( "Dog", "Cat" ) ), unique = "ID", required = "Species", pattern = list( ID = "^A[0-9]{3}$" ) )
Apply the rules directly:
rule_results <- apply_audit_rules( dat, rules ) rule_results
## DataAudit Rule Validation
## =========================
##
## Status: Violations detected
## Total violations: 8
##
## Rule summary
## ------------
## Rule Violations
## Range 2
## Category 2
## Unique 2
## Required 1
## Pattern 1
##
## Variables with rule violations
## ------------------------------
## - Range: Age
## - Category: Species
## - Unique: ID
## - Required: Species
## - Pattern: ID
Integrated audit
Custom validation rules can be incorporated directly into the general audit:
report <- audit_data( dat, include_nzv = FALSE, rules = rules ) report
## DataAudit Report
## ================
##
## Rows: 4
## Variables: 3
## Variables with issues: 2
## Duplicated rows: 0
## Total issues: 2
## Rule violations: 8
## Overall issues: 10
##
## Variables with issues
## ---------------------
## Variable Class Missing MissingPercent Blank Infinite Unique Constant
## Species character 0 0 1 0 4 FALSE
## Age numeric 1 25 0 0 3 FALSE
## NearZeroVariance Issue
## FALSE TRUE
## FALSE TRUE
##
## No duplicated rows detected.
##
## Custom rule validation
## ----------------------
## Rule Violations
## Range 2
## Category 2
## Unique 2
## Required 1
## Pattern 1
##
## Total rule violations: 8
The report now combines automatic auditing and custom validation:
report$summary
## Rows Variables VariablesWithIssues DuplicateRows TotalIssues RuleViolations
## 1 4 3 2 0 2 8
## OverallIssues
## 1 10
report$rules
## DataAudit Rule Validation
## =========================
##
## Status: Violations detected
## Total violations: 8
##
## Rule summary
## ------------
## Rule Violations
## Range 2
## Category 2
## Unique 2
## Required 1
## Pattern 1
##
## Variables with rule violations
## ------------------------------
## - Range: Age
## - Category: Species
## - Unique: ID
## - Required: Species
## - Pattern: ID
A compact summary can be obtained with:
summary(report)## $dataset
## Rows Variables VariablesWithIssues DuplicateRows TotalIssues RuleViolations
## 1 4 3 2 0 2 8
## OverallIssues
## 1 10
##
## $variables_with_issues
## Variable Class Missing MissingPercent Blank Infinite Unique
## Species Species character 0 0 1 0 4
## Age Age numeric 1 25 0 0 3
## Constant NearZeroVariance Issue
## Species FALSE FALSE TRUE
## Age FALSE FALSE TRUE
##
## $duplicate_rows
## [1] Row Duplicate
## <0 rows> (or 0-length row.names)
##
## $rules
## Rule Violations
## 1 Range 2
## 2 Category 2
## 3 Unique 2
## 4 Required 1
## 5 Pattern 1
##
## attr(,"class")
## [1] "summary.DataAuditReport" "list"
Data-quality score
Calculate an overall data-quality score with:
score <- audit_score(report) score
## DataAudit Quality Score
## =======================
##
## Score: 54.55 / 100
## Quality: Poor
##
## Dataset
## -------
## Rows: 4
## Variables: 3
## Data cells: 12
##
## Issues
## ------
## Automatic issues: 2
## Rule violations: 8
## Overall issues: 10
The score ranges from 0 to 100, with higher scores indicating fewer detected data-quality issues relative to dataset size.
DataAudit classifies scores as:
- Excellent
- Good
- Moderate
- Poor
- Critical
The score is intended as a compact audit summary and should be interpreted alongside the underlying diagnostics and the intended use of the dataset.
Main functionality
DataAudit currently provides functions for:
- general data auditing;
- missing and blank values;
- whitespace problems;
- infinite values;
- duplicate and unique values;
- constant and near-zero variance variables;
- outliers;
- ranges and categories;
- data types;
- regular-expression patterns;
- dates and identifiers;
- character lengths and case;
- sequences and grouped sequences;
- dependencies;
- cross-variable consistency;
- reusable validation rules; and
- data-quality scoring.
Typical workflow
Raw data
|
v
audit_data()
|
+----> Automatic checks
|
+----> audit_rules()
|
v
Rule validation
|
v
DataAuditReport
|
+----> summary()
|
+----> audit_score()
Authors
Vinodh Kumar Obli Rajendran
ORCID: 0000-0002-7232-4122
Keerthi Aaradhana
License
DataAudit is released under the MIT License.