The Tuva Project ships over 100 different healthcare datasets and loads them directly to your data warehouse. These datasets are essential for healthcare analytics, but are scattered all over the internet, maintained by different organizations, stored in different formats, and updated on different frequencies. Historically, data teams have spent a ton of time and energy maintaining these datasets. Tuva maintains them and makes it easy to load them to your data warehouse where your data team needs them.
We call these datasets Tuva Data Assets and they are organized into 6 main categories:
Terminology
Value Sets
Reference Data
Provider Data
Synthetic Data
Concept Library
Each category contains multiple datasets with size ranging from 50 MB to over 10 GB.
Over the years we’ve used different solutions for maintaining and versioning Data Assets. Originally we stored the datasets in GitHub inside the Tuva Project repository. This was great for versioning (GitHub inherently keeps track of any file changes over time) but pretty quickly the datasets grew too large for GitHub.
So a couple years ago we switched to using AWS S3 as the source of truth for all datasets. This solved the size problem (S3 is essentially unlimited storage) and we created a workaround for versioning using folders (each time we made a change to a file we would save it inside a new folder named with a new version number), but there was one serious drawback with S3: for obvious reasons we can’t make S3 open to the public for write access, so using S3 meant that collaboration with the open source community to maintain Data Assets was impossible (or at least significantly limited).
Enter DoltHub. DoltHub is essentially a version-controlled mysql database. And it’s open-source, so as long as your database is public you can use it for free. Which obviously we’re OK with :-)
So with the release of the Tuva Project 0.17.2 we’ve moved all Data Assets to DoltHub. There are 6 separate DoltHub databases — one for each category. You can see all the databases from the Tuva Health organization page on DoltHub here.
With DoltHub it’s easy to view and explore any dataset. Just click on the database and you can see every dataset within that database. Click on any table to see the top 50 rows in that table. You can use basic SQL to query the table and search for specific codes or terms.
If you want to help maintain any datasets, making a PR to the database is easy and is very similar to standard git workflow.
One additional note: we still use the same dbt macros and post-hooks to load Data Assets when you run Tuva, so nothing changes from a Tuva package usage standpoint.
Yesterday we released 0.17.2 of the Tuva Project dbt package which included the following improvements — full release notes here.
Tuva Data Assets on DoltHub: Tuva data assets are now organized as 6 versioned databases on DoltHub: terminology, value sets, reference data, provider data, concept library, and synthetic data. DoltHub is now the release source of truth, while S3, GCS, and Azure Blob remain the distribution layer used by the existing seed-loading macros.
Versioned Seed Releases: Package seeds and synthetic integration-test inputs now load from versioned release artifacts that have been copied to S3/Azure/GCS from DoltHub. Releases can be controlled with
tuva_seed_version, per-database overrides viatuva_seed_versions, and storage overrides viatuva_seed_buckets.CI Improvements: CI now uses dbt-style command routing instead of alias-style commands. For example,
/ci dbt build --full-refreshruns across all 6 warehouses, while/ci snowflake dbt runor/ci redshift dbt run --select core__medical_claimtarget specific warehouses and selections.Docs in the Tuva Repo: The docs site now lives in the
tuvarepo underdocs, which keeps docs closer to the code and metadata they describe.
Here are some exciting new healthcare data and analytics job postings from the past week. As always, you can find these postings and more in the Tuva Slack #jobs channel.
Cylinder Health is hiring a Senior Data Engineer
Zus Health is hiring an Analytics Engineering Manager
Thanks for reading and if you’d like to receive more content like this every week subscribe below. To learn more about the Tuva Project visit www.thetuvaproject.com.
No posts

Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.