Recent content on Beyond Measure

 

 
 
 
 
 
 
 
 

 
 
 
 
 People using this cell 1 
 
 1 10 100 1k 2k 
 
 
 4K UHD Buffering… 
 Your throughput 4K UHD 48 Mbps 
 
 the cell's capacity ≈ 48 Mbps total 
 
 Signal −95 dBm 4 / 4 · unchanged 
 Quality RSRQ −4.0 dB your slice: 100% 
 
 
 
 I was standing in a…
If you have a website these days you might be missing out on a lot of valuable data. You used to be able to just drop a simple widget on your site and track your visitors. While the accuracy of that has steadily declined with ad blockers and legislative requirements, it has been the go to for identifying users on your website or app. Now there is a new way of interacting with your online content…
So, you want real time and interactive, but handle large analytical data at the same time? Cloudflare Workers are small serverless functions that allow you to create fast, scalable apps on Cloudflare’s edge network. However, Cloudflare Workers are lightweight functions and therefore do not support native DuckDB bindings. They can, however, connect to MotherDuck through the Postgres endpoint…
It feels like there’s a new tool, framework, or SaaS product every hour. That feeling isn’t wrong.
I looked at a bunch of signals to validate this feeling: Show HN launches, GitHub repos, npm , Rust and PyPI package registries, even Product Hunt , the rate at which people ship new things is accelerating. Here’s the data. 
 Show HN: the shipping rate is accelerating 
…
TLDR: Use microbatching for large, time-based tables where robustness and backfilling is important. Always use multi-threading in dbt-duckdb, but more threads and more RAM is not always worth it. 

 Why We Built Microbatch Support for dbt-duckdb 
 I like a good benchmark as much as anyone, as long as it’s not benchmarketing. But benchmarks don’t tell the whole truth about…
We talk a lot about whether AI is smart enough to take our jobs. But I think we’re missing half the equation: it’s not just about being smart, it’s also about being cheap. Last month I backfilled roughly 60,000 historical job postings on selectfrom.work using an LLM. Each request extracted a summary, salary information, technologies mentioned, and job type. Total cost? $10. 
…
I’ll admit it, as a data engineer I still use Excel. It’s great for exploring unknown datasets or CSVs from stakeholders.
There is something just so convenient about seeing data in a tabular format in a tool that everyone has, knows and uses.
I’m here to tell you that you can 10x your Excel powers without even using AI. As much as I like the tabular format,
the…
If you work with API data long enough you eventually inherit a directory or bucket full of tiny JSON files—one per product, event batch, log minute, user snapshot—because “it was easy at the time.” That convenience tax shows up later as slow scans, schema anxiety, and rising warehouse bills. DuckDB lets you pay that tax once: consolidate early, then model against something stable and polite. 
…

 Nietzsche reading Hacker News. Not metaphorically, but literally: I can run faitch https://news.ycombinator.com/item?id=45168823 -t nietzsche and get the German philosopher’s take on whatever tech drama is unfolding today. Or have Wittgenstein analyze a job posting with his characteristic precision about language games and empty corporate speak. 
 This isn’t just a party…
Deploying Streamlit apps with dbt on Snowflake using a custom dbt materialization: a practical tutorial and exploration 
 If you want to jump straight into the code, you can find it on Github . 

 What if your dbt model could deploy a dashboard instead of creating a table? If you’ve worked with dbt for any amount of time, you’re familiar with the usual suspects: table ,…
If you’re using Snowflake, you know that things can get expensive very quickly if you don’t watch out. Whether it is turning down the default warehouse timeout settings from 10 to 1 minute, limiting the autoscaling, or making sure that your undersized warehouse times out when trying to process more data than it can handle instead of continuing throughout the weekend… Out of the…
If you’ve spent more than three days as an analytics engineer, you’ve probably had that moment: your dbt model runs, but the numbers look… off. Not broken, just weird enough to make you question your life choices (or at least your SQL). The code works, but the data? Not so much. And let’s be honest, most of these “surprises” start with the classic: “Let…
If you come from an analytics background you might still know what it was like to have folders full of notes and SQL files to keep track of your transformation queries for dashboards and analyses for business questions. Dbt solved a lot of the problems with that approach by keeping track of changing queries in a team setting through version control, splitting long queries into smaller building…
Have you ever wasted time or money because you made a decision based on incorrect data? Then you’ll appreciate the quality of your data. Fixing data quality issues, however, is a whole different beast. There are many different tools and technologies claiming to fix your data quality: data observability tools, data lineage, semantic layers, data catalogues, but to find the right solution to…
This post has been expanded to appear in my book The Fundamentals of Analytics Engineering , a perfect handbook if you are looking to learn more about
data modeling, data quality, data warehousing and data ingestion. 

 We all know that data, like wine and cheese, becomes more valuable when combined. And, just like wine and cheese, they can lead to serious headaches. Whether you are…
Have you ever seen certain small towns in your analytics reports with thousands of visitors? Or maybe a lot of traffic from a country where your products don’t even ship to? Google Analytics used to have a very nice feature that would allow you to see the Internet Service Provider (ISP) of the originating request. This made it easy to identify bot traffic and spam in your analytics account.…
With Google Analytics’ sub-par performance over the last few years, you may have been on the lookout for alternatives or just overwhelmed with the amount of LinkedIn posts claiming to have the ultimate list of alternatives. In any case, many of us have stopped a moment to think: “What do I actually want from an analytics tool?” I don’t know about you, but I want a cheap,…
Bots are a big part of the internet. Some are legitimate, like the Google Search crawler, but others have more nefarious purposes, like
scanning for vulnerabilities. If you want to understand which part of your traffic is related to bots or automation, you can use
a couple of attributes to identify this traffic. The user agent is probably the most important one, but also the referrer…
Imagine this: your website team has developed a great new feature. The developers are proud, the product owner is happy, and now they come to you to hear how it’s impacting sessions and conversions. You look at the data and realise they messed up the tracking implementation. 
 Snowplow fixes a lot of these tracking problems by giving you the ability to work with schemas, a codified…
With the recent update to Google Search Console (GSC) allowing us to export search console data to BigQuery it becomes
very easy to do some fun stuff. BigQuery has a ton of capabilities to process and slice data. To better understand the
trends of certain topics over time, we’ll extract those topics from the search queries directly so we can see how clicks
and impressions for…
How hard can it be to measure how many people visited your site? I recently spent a few days at a fantastic (web) analytics conference . After talking to so many smart people it suddenly struck me: we are not solving the problem of web analytics, we are solving how to work around the problems . Sounds too philosophical? Let me start with a picture first. 
 * 
 For a long time web analytics…
Incrementality is hard. Updating larges table with new information while also updating existing entries is not for the faint-hearted, but of course you are here because you’ve risen to the challenge. And so we shall conquer time and (storage) space together! The challenge ahead of us today is that of overwriting estimated data points with actual data points when they arrive (like the…
👋 Hey! I’m Dumky, Developer & Agent Experience Engineer at MotherDuck , the serverless analytics platform built on DuckDB. With a background as a Senior Analytics Engineer, I’m passionate about making data engineering accessible and practical. 
 I’m also a co-author of Fundamentals of Analytics Engineering and have built everything from €0.02/day Snowplow pipelines to…
How do you expose data in BigQuery with an API? I was recently looking for an easy way to do just that and make data from BigQuery publically available with little effort but still secure. The result of this effort is now available on my analytics page and if I want a JSON response with my BigQuery data, for example pageviews in the last 90 days all I have to do is call /analytics/json/pageviews .…
Update June 2023 : I have created a massive blog post on how to get started with Snowplow, Terraform, Docker and dbt that has some more insights on how to best run dbt in a container as well. This post is still valid, but know that there’s more if you are interested. 

 Dbt is a great tool to transform data in your data warehouse. It allows you to version control the rules and…
Over the last few years SQL has really started embracing its second adolescence —it’s nearly half a century old after all. It’s been riding the fast sportsbike —dbt— and now the Google BigQuery SQL dialect is trying out something more extreme: native integration with serverless functions, called BigQuery Remote Functions . Ok, maybe that doesn’t sound as exciting as a sportsbike,…
If you enjoy this post. I’ve developed an easy package called ‘consentcrawl’ to run a quick check on your site doing nothing more than running consentcrawl <<my-site.com>> in your terminal. 

There’s nothing like watching 20 browser windows pop-up on your screen to make you feel like a proper hacker. So I always grab on to any excuse for a browser automation…
Server-side tracking is all the rage these days, but let me tell you about the uber-coolest kid on the blockchain: edge analytics. I’m kidding, there’s no such thing as edge analytics (except maybe for IoT devices —story for another time—), but there is the possibility to intercept requests on the ’edge’ of the network. Using Cloudflare Workers, you can send data to Google…
If you’ve been around this blog before you might know that I know a thing or two about Google Tag Manager (GTM), the convenient tool that lets you run marketing and analytics scripts on your website. What you might not know is that I’m also a father of two young children, and I can tell you, kids are expensive! So, obviously, on sleepless nights, one starts to ponder all the ways to…
If you’re just looking for the code and a quick start, you can find everything on Github and you can find a live demo at docs.dumky.net . 
 Google Tag Manager is a great tool when you first get to know it: you can finally track that button in Google Analytics without having to wait 6 months on your dev team, hurrah! But after a few years reality kicks in. Now you’re three agencies…
One thing that has bothered me over the years is the weirdness of not having any possibility for setting up a pipeline in Google Tag Manager for when new container versions are published. Wouldn’t it be great not only for security reasons to keep an eye on container updates, but also to automatically update documentation, set up notifications in other services, or run some tests. But alas,…
“Google Page Speed says GTM is slowing down my site, can we remove it?” I’ve lost count of how many times I’ve heard that. I wrote about some best practices to improve performance before, but let me show you how to use GTM and still get that coveted 100/100 score on Google Page Speed Insights . We’ll do so by setting up a proxy with a cache and serve Google Tag…
The internet is a beautiful place. If you think chaos is beautiful, that is, because it is also a place where everyone and everything is hacked, abused, and manipulated for money, status or just the lolz. To prevent your precious Google Tag Manager implementation —and your entire site for that matter— from falling victim to malicious code taking over checkout funnels or secretly listening to form…
Google Sheets, the online Excel alternative, is basically the Swiss army knife of any modern knowledge worker. We’re adding to it’s multitude of uses and abuses today by creating a custom Tf-Idf model in Google Sheets. A Tf-Idf model can help us digest large quantities of text like reviews, comments, feedback, chat histories, and even 17th century philosophy books with ease. 
…
With terms like GDPR, Cookies, ePrivacy Directive, ePrivacy Regulation, ITP, ETP, most people’s heads will start spinning. Nonetheless, not complying with privacy regulation can be a costly business in three ways. 
 
 You can be fined for non-compliance. And, yes, regulators are actually handing out fines, despite anything your local ‘marketeer’ might say. And these fines…
When we think of analytics we think of tools like Google Analytics, Mixpanel or even Tableau or PowerBI. But analytics is not just a tool. When we understand the mechanisms behind these tools we are ready to see what each of them brings to the table and how they might solve the problems for the unique snowflakes that we are. 
 I’m known for being abstract and longwinding —blame the MA in…
Just a few days ago Google released a new version of their tag management system: Google Tag Manager Server-Side. Here’s why it’ll have a massive impact on analytics, advertising, A/B-testing, privacy and Google’s own cloud solutions. But first things first. What on earth is server-side tagging and why do I need it? 
 Tagging, the practice of adding small pieces of script and code to your…
Do you know if all your tags are used in Google Tag Manager or if you can remove some? Do you know if they work for all your users? Or if they fire on the right dataLayer events and how long they take to execute? Follow along for a complete solution to monitor your Google Tag Manager implementation including a dashboard . Tag monitoring will allow you to understand if important tags are actually…
Whether it’s because you don’t want your business to lose money or because you just like to watch videos on YouTube, automating your analytics monitoring can help you sit back and relax knowing that everything’s under control. This time we’ll look at how a simple Google Sheet can help you check your analytics on the fly with the Google Analytics Real Time Reporting API.…
When asking developers to add extra information to a site like a user ID, user login status, page category, product price, SKU’s or even product margins the conversation that ensues can be confusing. Do you want everything in a dataLayer push? 
 If you’re not sure what you want thing can get muddy. What’s the event name for the data layer push containing the page category?…
While the general idea of this post is still very much relevant. I’d recommend using Playwright instead of Puppeteer. It is much better maintained and documented and has support for all major browsers. Check out my post on using it to check cookie consent to get an idea of how Playwright works. 

 Testing your analytics setup regularly can help you prevent uncomfortable situations…
A note beforehand: though I really like Cypress, you might also want to think about using Puppeteer for your testing. 
 One thing that’s often overlooked in setting up your analytics is making sure it keeps working. When you add new tags or your developers deploy new versions it’s easy to forget about your site tracking until the revenue in your ecommerce reports drops to zero.…
Or why GA’s Web + App property with its event tracking framework is the future of web analytics. 
 Tell me, what do you do on the web? I promise I’ll be all GDPR and CCPA compliant about it and won’t tell anyone. But seriously, what do you do? Do you buy clothes online? Groceries? Do you read The New York Times? Or just watch videos on Reddit and Facebook? Do you look for jobs or look…
UPDATE: with the GTM server-side release I’ve added a section on server-side tagging 
 UPDATE 2: There’s a number 9 that shows you how to serve GTM through a caching proxy to increase performance 
 Google Tag Manager makes it incredibly easy to add marketing tags to your site. From registering ads conversions and transactions to sophisticated tags that segment users based on…
Just looking for the template? You can find it on Github . 
 Recently Google Tag Manager introduced custom templates. Custom templates are a way to make tag management even more manageable and scalable. A great and effective example is a custom template for the Facebook Pixel . In a way GTM’s custom templates can seem very limited, since they’re built on a very restricted, sandboxed version of…
This post was also published on the Dept blog on May 8th, 2019 
 I actually believe you will not make the targets for your online ads this year, and I’m not just saying it for the clicks. This is not your fault, or the developer’s fault, or the social or SEA consultant’s fault, or the creative’s fault. You can blame Apple, but don’t be too quick to judge. Last February, Apple introduced…
I have spent many hours of my life devoted to studying Thomas Hobbes’ book Leviathan. It was published in 1651 and is considered a starting point of modern political philosophy. It is a fascinating book, I even wrote a thesis on it, but one thing had always nagged me. You see, Hobbes writes very concise and structured arguments. He starts out by considering how people used to live before we…
What is the fastest way to transfer 100 petabytes to the cloud—that is to say, about 200 thousand times your laptop’s capacity? Who has that much information, you ask? Plenty of companies with years of digital documents, movie studios, government agencies, you name it. If you think your fiberoptic cable will help you out, think again. At this size and even with the fastest connection we are still…
Spoiler alert: it wasn’t Aristotle who said “Where the needs of the world and your talents cross, there lies your vocation.” Read on to get the full story. 
 We’ve all been there. We want to add a little spice to our presentation, give our paper a bit more eloquence, or just appear to have ‘deep thoughts’. So you type those reviled words ‘inspirational’ and ‘quote’ into your…

 It is stuck away all at the end of a letter. One of the many Isaac Newton wrote to his adversary Robert Hooke. Their disputes over their work, inventions, and science are still visible in the written words, but they also show a difference in attitude. They show a difference in view of how science works, how innovations come about. Where Hooke was known for his hubris and claimed ownership…