Recent content on Bryan Shalloway's Blog
TL;DR: I used Claude to build a pipeline that identifies likely owners of unclaimed property (lost or dormant assets held by the state) and just messaged the likely owners of ~$10M currently held by the state of CA. Initial notes on the project/approach are at brshallo/escheat-finder . There’s that classic economist joke: Two economists walk down a road and see a twenty dollar bill lying on…
Scenario 1: difference of binomial distributions Scenario 2: difference in negative binomial distributions Which game should you play if you’re better? Appendix Hypothesis tests on observational data TL;DR: A quick look at the binomial and negative binomial distributions through the lens of basketball shooting drills. A friend and I were doing basketball shooting drills. To make things…
What if you’re playing make-it-take-it? Summary Taking action Other considerations Appendix But you don’t play pickup basketball to infinity Table of imbalances Finite geometric series Scoring system in pickup basketball TLDR: For full court pickup basketball you should play by 2s and 3s. If you’re playing make-it-take-it, you can calculate the expected value of a possession with a geometric…
TLDR: Parts of Taleb’s critique of an, albeit overhyped study on Intermittent Fasting, seem specious. Veritasium recently put out a video on “The Problem With Science Communication” that details how news outlets and science journalism gravitate to big headlines that are often misleading, overstate the findings, or are outliers that go against the weight of evidence in the field. The video doesn’t…
There are many situations where you want to aggregate values, however if those values are on different scales or are related to measures of uncertainty, it’s typically more complicated than simply taking a simple mean or sum. You can’t take the aveage of p-values or standard deviations or statistical tests. You also can’t take the sum of confidence (or prediction) intervals to get to an interval…
When odds are helpful Ratio of odds (odds ratio) Communicating odds Appendix Ratios and fractions of events Many variables Other problems We grow up learning proportions, percentages, risks, probabilities. You encounter them when a teacher gives a grade on a test or a doctor describes the risk of an illness. On the other hand, we rarely interact with odds and when we do it’s often in contexts…
In this post I’ll walk through an example of how to convert between currencies. A challenge is that the conversion rate is constantly changing. If you have historical data you’ll want the conversion to be based on what the exchange rate was at the time. Hence the fields you need when doing currency conversion are: Date of transaction Start currency (what you’ll be converting from) End currency…
This is a follow-up to a short post I wrote on R Access to Twitter’s v2 API . In this post I’ll walk through a few more examples of pulling data from twitter using a mix of Twitter’s v2 API as well as the {rtweet} package 1 . I’ll pull all Twitter users that I ( brshallo ) have recently been engaged by (e.g. they like my tweet) or engaged with (e.g. I like their tweet). I’ll lean towards using…
The rtweet package is still the easiest way to GET and POST Twitter data from R. However its developers are currently working on adapting it to the new API. V2 comes with a variety of new features. The one I was interested in was being able to GET the users who liked a tweet. academictwitteR is probably the most established package that provides a quickstart entry point to the V2 API. However it…
Interactive network plots Julia Silge Blog David Robinson Tidy Tuesday R for Data Science Chapters My blog My gists In previous posts and threads I’ve alluded to the potential utility of visualizing the relationships between parsed functions/packages and files as a network plot. It can be helpful to review the relationship between your #rstats code files by looking at a network graph of them by…
Parsing code Parsing R files Parsing markdown files Binding files together Organizing snippets Appendix This post is part two in a series of posts introducing funspotr . See also: Identifying R Functions & Packages Used in GitHub Repos (funspotr part 2) Network plots of code collections (funspotr part 3) This post shows how funspotr can also be applied to parse gists: By functions or packages…
Documenting rstats posts Examples Julia Silge Blog David Robinson Tidy Tuesday R for Data Science Chapters Bryan Shalloway Blog TLDR: funspotr provides helpers for spotting the functions and packages in R and Rmarkdown files and associated github repositories. See Examples for catalogues of the functions/packages used in posts by Julia Silge, David Robinson, and others. See follow-up posts for…
NBA Playoffs and the Lakers Data Prep Scraping Betting Markets Steps Joining with FiveThirtyEight data Analysis How much does FiveThirtyEight differ from markets? Closing Thought Appendix Potential Reasons for the Difference Calculating percentiles of diff TLDR: FiveThirtyEight’s forecasts of NBA playoff berths seem to hold-up OK against betting markets. If you trust them, you should consider…
Macro in the Shell Example Setting-up Gaurd Rails Closing Appendix Related Alternative Other Resources There is many a data science meme degrading excel: (Google Sheets seems to have escaped most of the memes here.) While I no longer use it regularly for the purposes of analysis, I will always have a soft spot in my heart for excel 1 . Furthermore, using a “correct” set of data science tools often…
Quantile Regression Example Quantile Regression Forest Review Performance Coverage Interval Width Closing Notes Appendix Residual Plots Other Charts In this post I will build prediction intervals using quantile regression, more specifically, quantile regression forests. This is my third post on prediction intervals. Prior posts: Understanding Prediction Intervals (Part 1) Simulating Prediction…
Update Rough Idea Inspiration Procedure Example Simulate Prediction Interval Review Interval Width Coverage Closing Notes Appendix Conformal Inference Other Examples Using Simulation Confusion With Confidence Intervals Adjusting Procedure Alternative Procedure With CV Part 1 of my series of posts on building prediction intervals used data held-out from model training to evaluate the…
Providing More Than Point Estimates Considering Uncertainty Observation Specific Intervals A Few Things to Know About Prediction Intervals Prediction Intervals and Confidence Intervals Analytic Method of Calculating Prediction Intervals Visual Comparison of Prediction Intervals and Confidence Intervals Inference or Prediction? Cautions With Overfitting Generalizability Review Prediction Intervals…
Schooling & Early Career: Bryan graduated with honors in Cognitive Neuroscience from Washington University in St. Louis where he also studied American Culture and Political Science. He spent the majority of his high school and undergraduate summers in a Neuropathology research lab studying Alzheimer’s disease. After graduating, he started his career as a high school math teacher in Durham…
Annual Counts Raw data Reports HUD Supported Research Appendix HMIS Data Business Intelligence Other Resources Data Sources This write-up provides a broad overview of data sources and reports relevant for an independent researcher or analyst new to exploring data on people experiencing homelessness. The section on HMIS Data focuses specifically on those CoC’s in California supported by PATH…
Model Performance Metrics Lending Data Example Starter Code Weighting by Classification Outcomes Metrics Across Decision Thresholds Weighting by Observations Closing note Appendix Weights of Observations During and Prior to Modeling Notes on Cost Sensitive Classification Weighted Classification Metrics Questions on Cost Sensitive Classification Arriving at Weights Weighting in predictive modeling…
Create Data Association of ‘feature’ and ‘target’ Resample Build Models Rescale Predictions to Predicted Probabilities Appendix Density Plots Lift Plot Comparing Scaling Methods TLDR: In classification problems, under and over sampling 1 techniques shift the distribution of predicted probabilities towards the minority class. If your problem requires accurate probabilities you will need to adjust…
Simple Example Applying Incentives Takeaways of Resulting Distribution Think Carefully About Assumptions How to Set Assumptions Appendix Simple Assumptions Trade-offs In this post I will use incentives for sales representatives in pricing to provide examples of factors to consider when attempting to influence an existing distribution. For instance, if you have a lever that pushes prices from low…
Biases in Value Implications for Gambling Appendix Note on Prospect Theory Wallet Roulette In this post, I will describe an example of a game that produces many small wins for the player and occasional large wins for the house. Such a game could take advantage of psychological biases of individuals to prefer gains to be disaggregated and losses to be aggregated 1 as well as a general disposition…
A version of this question was asked on my alumni Slack channel. There were some excellent points brought up by those answering the question in the negative, including that… the practice is exploitative (or at least inconsiderate) of the interviewee’s time. it is not useful for the interviewing team (as the questions / scenarios are often so contrived as to be useless towards evaluating…
Load data Feature Engineering & Data Splits Lag Based Features (Before Split, use dplyr or similar) Data Splits Other Features (After Split, use recipes ) Model Specification and Training Model Evaluation Appendix Model Building with Hyperparameter Tuning Resources The new rsample::sliding_*() functions bring the windowing approaches used in slider to the sampling procedures used in the tidymodels…
TLDR: In this post I discuss using a national popular vote weighted by the electoral college to elect the president. This approach would empower voters by expanding political influence outside of ‘battleground states.’ It would also preserve the existing biases built into the American electoral college (thereby making such a system legislatively palatable across party affiliations and electoral…
What influences price? Simple linear regression model Inference and challenges Violation of model assumptions The tug-of-war between colinear inputs Improving model fit, considerations Closing notes and tips Appendix Pricing challenges Future pricing posts Dataset considerations Interpretability of machine learning methods Regularization and colinear variables Coefficients of a regularized model…
Things following you Codepen example How I made it Next steps Learning path and resources Closing thoughts Appendix Additional actions We spend the majority of our time in front of screens. It’s mostly one of computer/tablet/phone/tv 1 . These are largely platforms the user owns or controls. I’m surprised we don’t yet have more interactions with screens out in the world . Face detection and object…
Function expecting one column Functions allowing multiple columns Older approaches Appendix dplyr , the foundational tidyverse package, makes a trade-off between being easy to code in interactively at the expense of being more difficult to create functions with. The source of the trade-off is in how dplyr evaluates column names (specifically, allowing for unquoted column names as argument inputs).…
Learning R’s %>% Using the pipe operator ( %>% ) is one of my favorite things about coding in R and the tidyverse . However when it was first shown to me, I couldn’t understand what the #rstats nut describing it was so enthusiastic about. They tried to explain, “It means and then do the next operation.” When that didn’t click for me, they continued (while becoming ever more excited) “It passes the…
UPDATE Overview I. Nest and pivot II. Expand combinations III. Filter redundancies IV. Map function(s) V. Return to normal dataframe VI. Bind back to data Functionalize Example creating & evaluating features When is this approach inappropriate? Appendix Interactions example, tidymodels Expand via join Nested tibbles Pivot and then summarise Gif for social media Tweets UPDATE In May of 2021 I…
Riddler express Riddler classic Appendix Time to center Transform grid, rotate first Transform city, pretty This post contains solutions to FiveThirtyEight’s two riddles released 2020-02-14, Riddler Express and Riddler Classic . I created a toy package animatrixr to help with some of the visualizations and computations for my solutions 1 . Riddler express The riddle: Riddler City is a large…
This post is a continuation on my post from last week on Visualizing Matrix Transformations with gganimate . Both posts are largely inspired by Grant Sanderson’s beautiful video series The Essence of Linear Algebra and wanting to continue messing around with Thomas Lin Peterson’s fantastic gganimate package in R. As with the last post, I’ll describe trying to (very loosely) recreate a small part…
I highly recommend the fantastic video series Essence of Linear Algebra by Grant Sanderson . In this post I’ll walk through how you can use gganimate and the tidyverse to (very loosely) recreate some of the visualizations shown in that series. Specifically those on matrix transformations and changing the basis vectors 1 . This post is an offshoot of a post of my solutions to this week’s…
Riddler Express Riddler Classic Appendix On duplicates More than 9 numbers Define more rules Creating gif This post contains solutions to FiveThirtyEight’s two riddles released 2020-02-07, Riddler Express and Riddler Classic . Code for figures and solutions can be found on my github page . Riddler Express The riddle: From James Anderson comes a palindromic puzzle of calendars: This past Sunday was…
Riddler Express Riddler Classic Area of the base of the pyramid Height of the pyramid Encode functions and calculate volumes Appendix This post contains solutions to FiveThirtyEight’s two riddles released 2020-01-31, Riddler Express and Riddler Classic . Code for figures and solutions can be found on my github page . Riddler Express The riddle: At the recent World Indoor Bowls Championships in…
TLDR: Residential solar installations have gained popularity in the Seattle area over the last few years 1 . Prima facie, these seem to represent a suboptimal use of panels which could be more productive in regions of the country that are less overcast or have greater energy demands. The common inclination to “act locally” is sometimes misguided. Climate change and personal investments are global…
I did my best to convince Britney and my parents that we should start the morning with a ‘polar bear plunge’ in the ocean but was unsuccessful in convincing anyone (including myself) to participate. We slept in and had a relaxing morning before leaving the Airbnb at around 12PM. We headed to the Perlan, where we’d had dinner a few nights prior and which also served as the city’s science center 1 .…
We were out the door by 7:45AM and headed for the Blue Lagoon (where my parents had offered to treat us for the day). The regular Blue Lagoon pool had sold-out of tickets. Instead, we were ‘forced’ to get tickets to the Blue Lagoon Spa Retreat – setting the stage for the most luxurious four hours of my life. Changing room in the Blue Lagoon Spa Retreat In addition to access to the main lagoon, the…
Britney and I awoke several times in the night to the frigid cold. We would run across the street to a patch of trees where we could relieve ourselves and get away from the streetlights. I looked up to the stars and the clear night sky clinging faintly to a hope that whatever damage might be done to the car, it was all part of a plan for us to be at this exact point. For what? – maybe to see a…
We slept in a little later this morning (930AM), had Skyr parfaits with mom and dad for breakfast and more croissants from Brauð & Co. We drove back to the UNESCO World heritage site, Thingvellier, that we’d attempted to visit the morning before. We arrived close to sunrise (there was no creepy white pickup truck in the parking lot this time). The drive out was beautiful and easy. The browning…
I grabbed an assortment of croissants from Brauð & Co, half a block from our apartment. We were on the road headed East by 7:05AM. The morning was strikingly dark. Clouds obscured any starlight. A soft rain made everything reflective. Faint glimpses of small trees and twisted wooden branches looked like alien figures on the side of the road. I drove nervously, stooping over the wheel like my dad…
My parents, Britney and I landed in Reykjavik at 630AM. We’d taken an eight-and-a-half-hour overnight flight from Seattle. It was dark when we stepped off the plane and would remain dark until 11AM (in wintertime Iceland only gets 4-6 hours of sunlight a day). I packed the bags into haphazard piles that poured over the rear storage area of our rented Nissan X-trail. The air was a cold mix of rain…