Recent content on p < Whatever
Transitioning to using AI tools Despite working in the applied AI field for the last couple of years, I’ll admit that for a while I was resisting using it in my daily life, or my coding. This post is about transitioning to comfortably using AI, while leaving room for growth. At work, I found early on that LLMs could be extremely helpful for analyzing data in minutes, which would otherwise take a…
This post is about things I wish all eval tools provided (but many don’t). Over the last couple of years, I’ve used several different eval tools, from home-grown and pre-release products, to open source and startup-built tools with wide adoption. Most of them had inconsistent features at best, and glaring omissions for what I thought were obvious features, at worst. Now, to be fair, what I’m…
Recently, I’ve been surprised to note how many companies aren’t just returning to offices (ugh, open offices are the worst!), but are still actively unwilling to do remote work well. Here are some hints for things that I’ve learned can be helpful, after working remotely for a few years at a variety of companies - some that were remote-first or remote-only, and some that didn’t have mature…
Some folks love doing demo days, but I find it sometimes gets in the way of actual work, creates more work in the long run, and/or can be downright demoralizing (especially to backend teams). Here’s why. What is demo-driven development? • Building something specifically to be able to show it off • Rushing to get stuff in before an arbitrary deadline • A way to get people motivated by an…
Some observations on vibe coding I get a lot of questions about working with LLMs. Something I hear people discussing a lot is whether AI-generated code is better, or worse, or just more evil than human-generated code. I don’t think it’s an either/or question. To me, it’s just another tool: like most tools, there are ways to use them for good, or as weapons. I think vibe coding…
I’ve been doing a lot of work around grief lately. Both of my parents died in 2023. Two other friends also died in 2023 - both former coworkers. One was in his mid-thirties and had a rare brain tumor; the other died from COVID. My dad’s death last June came after several months in the ICU. He endured repeated rounds of dialysis, which he hated, and ventilator-associated pneumonia, which I hadn’t…
Life is growth. If we stop growing, technically and spiritually, we are as good as dead. —- Morihei Ueshiba I started studying aikido about 12 years ago, shortly after I moved to San Francisco. I had no idea how much I would enjoy it, or how much it would change my life. I didn’t know what it was when I first started, but the translation I like the best is “Path of Life…
This is one of those posts that I’m writing mostly because I’m still frequently hearing people say they’re struggling with the question of how to interview data people. I’ve blogged and spoken previously about the misery of being an interviewee, so suffice it to say I have a ton of empathy for how awful it is to go through a bad interview process, and how disappointing it…
I decided to embark on this comparison mostly out of curiosity. No tool is perfect for all use cases, that’s why we have forks, and spoons, and sometimes when we’re camping, sporks. Although Pachyderm claims to use a git-style approach for data and code versioning, there are aspects of the Pachyderm approach (like forking) that aren’t exactly like git. So one thing I wanted to…
Coming from a background in bench science, or what we affectionately referred to as “wetlab”, I like to test everything I do, and I like my tests to be fast and representative of what I expect to find when I run things “for real”. Most people I’ve met who are newer to data engineering find that it’s not immediately obvious how to write and run tests for data…
Recently, a coworker asked me what the difference is between data science and data engineering. She seemed to think that data engineering was “just analytics”, which is a definition I’ve often heard applied to data science, as well. After I got over my initial horror that she had been wondering this for months and hadn’t asked sooner, I started to explain: Data engineering is getting data,…
Question: How come we don’t have more diversity at our company? This is one of those posts, I’m writing it because I keep hearing people asking the same questions over and over. I hear these questions at meetups and in Slack groups every week, and I hate seeing people trying to reinvent the wheel when it’s really not necessary. Answer: You haven’t tried that hard. For the…
Recently, someone asked for help with setting up postgres in docker and connecting to it with python. While helping this person, I realized this is something that should be fairly straightforward with a simple set of instructions, but there aren’t a lot good beginner tutorials out there. So I decided to write this up because I’m sure it’s something other people would also find…
I was chatting with some friends recently, and this question came up: What does a good working relationship with your manager look like? This is the list I came up with. I’m not saying I’ve been perfect at all of these myself as a manager, but it’s what I strive for, and it’s what I look for in a good manager. They trust you They ask for your input when appropriate They…
I frequently hear Python referred to as a ‘scripting’ language, because it’s not compiled. Unfortunately, for this reason, a lot of people seem to assume you can’t write ‘real’ programs with it. This post is about moving beyond using Python as a scripting language. I’m assuming you’re already comfortable with basic python data types and methods.…
If you do a lot of data pipelining, you’ve probably heard a lot about Airflow by now. I gave a talk about it a while back at a meetup, and wrote a blog post about it. The gist of my pitch for Airflow was essentially “Look, it’s so much better than cron.” Fast-forward a year or two, and my team is using Pachyderm now. This post is about why I wanted to try Pachyderm, what I…
To make a long, mostly whiny story short, as part of my current role, I’ve been doing a lot of fighting with AWS to help support my team. Some of the things I’ve learned along the way are probably not obvious if you, like me, are relying mostly on AWS docs and other people’s advice, so I thought I’d collect some of them here. Best practices for storing big data on S3
The scene: I needed to process data from an s3 bucket using pyspark. The s3 bucket was owned by a different account. I had done this before. But this time, there was a twist: we needed to encrypt the data because of GDPR requirements. At the end of the processing, I needed to save the results to another s3 bucket for loading into Redshift. Thus began a weeks-long saga of learning about AWS the…
Why Spark? Lately I have been working on a project that requires cleaning and analyzing a large volume of event-level data. Originally, I did some exploratory data analysis on small samples of data (up to 15 million rows) using pandas, my usual data visualization tools, and multiprocessing. But then it was time to scale up. Why Spark is good for this Distributed processing means it’s very…
Airflow for hands-off ETL Almost exactly a year ago, I joined Yahoo , which more recently became Oath . The team I joined is called the Product Hackers, and we work with large amounts of data. By large amounts I meant, billions of rows of log data. Our team does both ad-hoc analyses and ongoing machine learning projects. In order to support those efforts, our team had initially written scripts to…
Over the years, I’ve done a few data science coding challenges for job interviews. My favorite ones included a data set and asked me to address both specific and open-ended questions about that data set. One of the first things I usually do is make a bunch of histograms. Histograms are great because it’s an easy way to look at the distribution of data without having to plot every…
Things I learned while following a tutorial on how to build reusable models with scikit-learn. When in doubt, go back to pandas. When in doubt, write tests. When in doubt, write helper methods to wrap existing objects, rather than creating new objects. Ingesting “clean” data is easy, right? Step 1 of this tutorial began with downloading data using requests , and saving that to a csv…
Here is a story about an interesting interview question and how I approached it. The company in question wasn’t interested in actually looking at my code, since I apparently tried to answer the wrong question. Given a deck of n unique cards, cut the deck c cards from the top and perform a perfect shuffle. A perfect shuffle is where you put down the bottom card from the top portion of the…
I don’t believe truth is a finite value. Truth is what we know right now. Every ten years or so, a major discovery gets overturned. Scientists are just people, and we’re wrong a lot. So one of the scariest things about doing research, or predictions, is trying to convince yourself, and other people, that what you think you’ve discovered is ‘real’. Or at least real…
When to test, and why: • Write a test for every method. • Write a test any time you find a bug! Then make sure the test passes after you fix the bug. • Think of tests as showing how your code should be used, and write them accordingly. The next person who’s going to edit your code, or even just use your code, should be able to refer to your tests to see what’s happening.
For better or worse, when you’re dealing with data pipelines of varying shapes and sizes, sometimes you need to combine objects that don’t match up evenly. For example, if you want to apply a condition via lookup, sometimes it makes sense to just do a merge. This creates a new column in your data table, and then you can use that for reference. This is an extremely simple example to…
Am I getting slower every day? If you’ve ever been a bike commuter, you’ve probably asked yourself this question. Thanks to these little devices we can now attach to ourselves or our bicycles, we can now use our own actual ride data to investigate these kinds of questions, as well as questions like these: If I’m going to work from home one day a week, which day would maximize my…
In continuing my series on investigating bike data , I ran into some interesting aspects of working with device data. I have some experience with devices, thanks to my many years of working in research labs. This post is about the fun of hunting down what’s working and what’s not. Things to consider when working with devices Are you using the device yourself? Are you interacting with…
So I have some bike data that I parsed out of XML and put into a pandas dataframe. Most of the questions I wanted to ask required that the timestamp of each ride segment, or lap, be used as the index along the x-axis of a plot. Non-obvious nuances of pandas datetime objects and indexes. You have to sort the dataframe by timestamps, before you can convert the timestamps to use as an index.
One thing I wanted to do with this data set was experiment with plotting methods. I had already done some exploratory plotting with regular matplotlib, so I had some vague ideas about what I wanted to do. First I had to select out subsets of data to compare. I knew that there were two types of rides: shorter trips in the city, and longer trips in the suburbs. I was feeling lazy, so I just did a…
One of the main reasons this project turned out to be interesting is that time series data has all kinds of gotchas. I never had to deal with a lot of this before, because the sorts of time series I did in my scientific life didn’t care about real-life things like time zones. We mostly just cared about calculating time elapsed. …tick…tick…tick Anyway one thing I wondered…
In an effort to advance my python skills, I spent some time slowly pecking away at the puzzles on pythonchallenge . I got stuck on most of the challenges, and either had to search for a hint, or ask for help from a friend, or both. This latest one was particularly instructive, and it had to do with zipfiles. I thought I knew what zip files were. I have used them since grad school, for transferring…
I know that from the outside, ‘science’ seems like The Place Where Scientists Live. But ‘science’ is not a monolithic, homogenous thing. Not all scientists are the same. Today someone called me a Biologist . But I was never really a Biologist . My undergraduate degree was in a chemistry department. My past life as a researcher was always very interdisciplinary. To better…
I have had a few pleasant job interviews. Here’s what’s different about those interviews, that made them really stand out from the others I’ve done. I’ll describe a specific example, and then give some specific suggestions. The hiring manager contacted me directly He had done his homework. He had looked at my GitHub repos. He told gave me pretty specific information about…
Once upon a time, a friend recruited me to do some technical writing for the company where he works now. Basically, they needed someone to quickly revise and update the documentation for their software. Most modern user-friendly software documentation isn’t just writing, though. It’s screenshots. A LOT of screenshots. So you don’t just write “click on the blue box”,…
I was looking for some fun time series data to hack on. Turns out bike computers provide a great way to interface between your friends’ unfounded modesty and their actual rides. One friend was kind enough to donate some data for my entertainment. This friend wears spandex, rides a titanium frame road bike, and sweats a lot. I knew he was a strong cyclist. He’s also relatively…
A lot of people have asked me why I’m not working in academia or biotech, how I learned to code, or how I got into data science . Here are some answers to some of those questions. Are you sure you wouldn’t rather be in academia? Yes and no. Yes, I wanted my own lab. No, we don’t always get what we want, even after 7 years of postdoc. Long story short: bad timing. It wasn’t…
This post is about a couple of painful problems I’ve had with reading in text files. They’re just text files, right? What could be hard about that? 1. Fun with encoding The problem: A friend sent me a dump of data from iTunes, as a table. Just a text file. She’s on a Mac. I’m on a Mac. But when I went to read in the file, Python choked in a way I hadn’t seen before.
So let’s say you’re thinking about attending a conference in tech, or some STEM field. Maybe you’ll be going alone. Maybe you’ve never been to a conference before, or this conference has a reputation for having, shall we say, “a higher risk of harassment”. Here’s a ‘game plan’ for things to keep in mind. I’m not saying anyone should ever…
Thanks to a friend who wants to help more women get into tech careers, last year I attended Developer Week , where I was impressed by a talk about Neo4j . Graph databases excited me right away, since this is a concept I’ve used for brainstorming since 3rd grade, when my teachers Mrs. Nysmith and Weaver taught us to draw webbings as a way to take notes and work through logic puzzles. In…
In a previous post , I wrote about using probability binning to create categories out of numerical data. This simplifying approach works well on various kinds of distributions. In this post, I wanted to show an example of how well it can work.
I spent about a year learning Python when I first got back into coding. A lot of people have asked me where to begin. Here’s a guide to what I found useful (or not) and why. Learn Python The Hard Way by Zed Shaw. I started here. I did the free online version. I like the advice he gives you to just type things in, even if you don’t know what’s happening yet. This required a bit of…
Once upon a time, I looked at a couple of data sets that involved geographical data. I wanted to actually plot the data on a map, so I had to do some shopping around for easy ways to do this quickly with python. What I wanted to do One of the data sets provided zipcodes, which I was able to convert to latitude and longitude (more on this below). At a minimum, I knew I wanted to be able to plot: a)…
More than once, and probably not for the last time, I have done a technical interview for which I was underprepared. I feel like no matter how much I try to prepare, I am always underprepared for technical interviews. I’m going to tell you about a time I was underprepared for a few reasons, including: a) It was the first interview where I was asked to write more than a couple lines of…
Inspired by a couple of great posts by Michael Gibson , I want to talk about what robustness means to me, as someone coming from a science background. I started working in a “wetlab” doing benchwork cancer research when I was in high school (I was 16). I learned early on that mistakes are: normal avoidable necessary. That may sound paradoxical, so I’ll explain. Some failures will…
Over the years, I’ve attended a lot of technical (read: software) talks. Once upon a time, I attended PyData , at Facebook. I’ve noticed at all of these events that the quality of presentations varies widely. If you’re like me, and you go to technical meetups and conferences around the Bay Area, for example, you’ll notice that some talks are well-rehearsed. But some seem…