Streaming ‘JSON’ (‘ndjson’) has one ‘JSON’ record per-line and many modern ‘ndjson’ files contain large numbers of records. These constructs may not be columnar in nature, but it is often useful to read in these files and “flatten” the structure out to enable working with the data in an R ‘data.frame’-like context. Functions are provided that make it possible to read in plain ‘ndjson’ files or compressed (‘gz’) ‘ndjson’ files and either validate the format of the records or create “flat” ‘data.table’ structures from them.
The goal is to create a completely “flat” data.frame-like structure
from ndjson records in plain text ndjson files or gzip’d ndjson files.
CRAN has binaries for Windows and macOS. To build this on UNIX-like systems, you need at least g++4.9 or clang++. This is a forced requirement by the ndjson library.
The least painful way to do this is to install gcc >= 4.9 (and you
should install ccache while you’re at it) and mmodfiy ~/.R/Makevars
thusly:
An example of such files are the output from Rapid7 internet-wide scans, such as their HTTPS study. A gzip’d extract of 100,000 of one of those scans weighs in abt about 171MB. The records sometimes contain heavily nested JSON elements depending on how comprehensive the certificate data and other fields were. A typical record will look like this:
All of the certificate sub-field data elements have been expanded and we
have a highly performant data.table to work with. Just go see what you
have to do in jsonlite to get a similar output (and how long it will
take).
pryr::object_size(df) for that shows it’s consuming 394 MB, which
means we can read in many more extracts comfortably on a reasonably
configured system and most (if not all) of it on a well-configured AWS
box.
However, if you do end up trying to work with that scan data, it’s
highly recommended that you use jq to filter out the fields or records
you want into a more compact ndjson file.
There are no current plans for a stream_out() function since
jsonlite::stream_out() does a great job tossing data.frame-like
structures out to an ndjson file.
install.packages("ndjson") # NOTE: CRAN version is 0.8.0 # or remotes::install_github("hrbrmstr/ndjson")
NOTE: To use the ‘remotes’ install options you will need to have the {remotes} package installed.
library(ndjson) # current version packageVersion("ndjson") ## [1] '0.9.0'
flatten('{"top":{"next":{"final":1,"end":true},"another":"yes"},"more":"no"}') ## more top.another top.next.end top.next.final ## 1: no yes TRUE 1 f <- system.file("extdata", "test.json", package="ndjson") gzf <- system.file("extdata", "testgz.json.gz", package="ndjson") dplyr::glimpse(ndjson::stream_in(f)) ## Rows: 100 ## Columns: 8 ## $ args <int> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, … ## $ headers.Accept <chr> "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", … ## $ `headers.Accept-Encoding` <chr> "identity", "identity", "identity", "identity", "identity", "identity", "identity", … ## $ headers.Host <chr> "httpbin.org", "httpbin.org", "httpbin.org", "httpbin.org", "httpbin.org", "httpbin.… ## $ `headers.User-Agent` <chr> "Wget/1.18 (darwin15.5.0)", "Wget/1.18 (darwin15.5.0)", "Wget/1.18 (darwin15.5.0)", … ## $ id <dbl> 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23… ## $ origin <chr> "50.252.233.22", "50.252.233.22", "50.252.233.22", "50.252.233.22", "50.252.233.22",… ## $ url <chr> "http://httpbin.org/stream/100", "http://httpbin.org/stream/100", "http://httpbin.or… dplyr::glimpse(ndjson::stream_in(gzf)) ## Rows: 100 ## Columns: 8 ## $ args <int> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, … ## $ headers.Accept <chr> "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", … ## $ `headers.Accept-Encoding` <chr> "identity", "identity", "identity", "identity", "identity", "identity", "identity", … ## $ headers.Host <chr> "httpbin.org", "httpbin.org", "httpbin.org", "httpbin.org", "httpbin.org", "httpbin.… ## $ `headers.User-Agent` <chr> "Wget/1.18 (darwin15.5.0)", "Wget/1.18 (darwin15.5.0)", "Wget/1.18 (darwin15.5.0)", … ## $ id <dbl> 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23… ## $ origin <chr> "50.252.233.22", "50.252.233.22", "50.252.233.22", "50.252.233.22", "50.252.233.22",… ## $ url <chr> "http://httpbin.org/stream/100", "http://httpbin.org/stream/100", "http://httpbin.or… dplyr::glimpse(jsonlite::stream_in(file(f), flatten=TRUE, verbose=FALSE)) ## Rows: 100 ## Columns: 7 ## $ url <chr> "http://httpbin.org/stream/100", "http://httpbin.org/stream/100", "http://httpbin.or… ## $ id <int> 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23… ## $ origin <chr> "50.252.233.22", "50.252.233.22", "50.252.233.22", "50.252.233.22", "50.252.233.22",… ## $ headers.Host <chr> "httpbin.org", "httpbin.org", "httpbin.org", "httpbin.org", "httpbin.org", "httpbin.… ## $ `headers.Accept-Encoding` <chr> "identity", "identity", "identity", "identity", "identity", "identity", "identity", … ## $ headers.Accept <chr> "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", … ## $ `headers.User-Agent` <chr> "Wget/1.18 (darwin15.5.0)", "Wget/1.18 (darwin15.5.0)", "Wget/1.18 (darwin15.5.0)", … dplyr::glimpse(jsonlite::stream_in(gzfile(gzf), flatten=TRUE, verbose=FALSE)) ## Rows: 100 ## Columns: 7 ## $ url <chr> "http://httpbin.org/stream/100", "http://httpbin.org/stream/100", "http://httpbin.or… ## $ id <int> 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23… ## $ origin <chr> "50.252.233.22", "50.252.233.22", "50.252.233.22", "50.252.233.22", "50.252.233.22",… ## $ headers.Host <chr> "httpbin.org", "httpbin.org", "httpbin.org", "httpbin.org", "httpbin.org", "httpbin.… ## $ `headers.Accept-Encoding` <chr> "identity", "identity", "identity", "identity", "identity", "identity", "identity", … ## $ headers.Accept <chr> "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", "*/*", … ## $ `headers.User-Agent` <chr> "Wget/1.18 (darwin15.5.0)", "Wget/1.18 (darwin15.5.0)", "Wget/1.18 (darwin15.5.0)", …
Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.