Building good software is challenging but rewarding. On my blog, I share the things I'm learning as a practitioner in this exciting industry.
Alternate title: new codebase who dis? Delivered at: Southern DevFest 2020 . Slides available here . Find me on Twitter @SamuelDataT . Transcript 0:00 Alright, yeah, thank you all so much for being here. Great to be here. Let's get started. When I started studying computer science in college, it's kind of I realized that the software industry is kind of like this pizza. Stay with me, I know,…
We often fly blind in the world of machine learning. Our model outputs an estimate for revenue from clicking on a certain ad, or the amount of time until a new edition of a book comes out, or how long it will take to drive a certain route. Typically these estimates are in the form of a single number with implicitly high confidence. Trusting these models can be foolish! Perhaps our models are…
I have switched teams more often than I have had to implement an AVL tree, and you can guess which one of those two was taught in school. I wish someone had taught me how to join a new team! While learning a new codebase can be daunting, I've found a few things that work for me. You should do at least three things when joining a new team. The order of these three can be whatever you like, but all…
When I began working in analytics, one of the most miserable types of tasks I ended up doing was re-running an old Jupyter notebook. Often it failed part way through with some inscrutable error. Figuring out what was going on was challenging; how am I supposed to remember this particular notebook from five months ago? What's more, the underlying data sometimes stops getting updated, or a column…
Author's note: this is not what I usually write about. If you're an educator considering purchasing a product from apchemsolutions.com, please read on! Otherwise, don't worry about it. Executive summary An educator I know recently contacted me for some IT help with a product they had purchased from apchemsolutions.com (AKA AP Chem Solutions). In helping this person, I was appalled by the level of…
A key element of a trustworthy model is that it can give an estimate of its confidence in a given prediction. We've already talked about one way to do this for linear models , and today we'll talk about a technique for getting uncertainty estimates for any model. Let's continue using the fish dataset from last time: import os import pandas as pd fish = pd . read_csv ( os . path . expanduser (…
Last time, we began to talk about how to build models worthy of our users' trust. As a refresher, we said that trustworthy models require at least three things: Prediction -- An estimate for some unknown value Confidence -- A description of how uncertain the model is about the prediction Explanation -- The reasoning for which a model made its prediction Today, we'll take a pass at actually…
Machine learning has a trust problem. Discussions about the role that algorithms play in our lives have become national (if not global), with some raising important and legitimate questions about the biases inherent in these algorithms. In this environment, we wonder: what would it take for a model to be worthy of our trust? I recently read an illuminating piece by David Spiegelhalter called…
Everyone advises being a "lifelong learner," but not all learning is created equal. Many effective techniques are underutilized, and many common techniques are useless. We buy a book or attend a conference. If we're really dedicated, we may even jot a few notes down in the process. But rarely do we take a step back and ask what the most effective way is to develop the skills we care about.…
Most companies with personal development budgets are wasting their money. If the goal is to help employees master valuable skills, then we are misallocating funds to books and conferences. Instead, we should look at what the research on skill development says so that we can make decisions informed by data. Great people are hard to find, especially in software engineering and data science. The…
Alternate title: Help! My Classes are Imbalanced! Delivered at: ODSC West 2019 . Slides available here . DeveloperWeek Austin 2019 . Audio available here . Aspiring Data Scientist Community . Audio available here . Transcript below. AnacondaCON 2020 Find me on Twitter @SamuelDataT . Transcript The start of this story is that at one point I was in undergrad, and I was in this machine learning class…
Data scientists are uniquely positioned to provide leadership on their teams around risk and uncertainty. We are trusted by our coworkers to have an understanding of experimentation and data-driven decision making. This trust can be leveraged to improve processes, decisions, and (ultimately) the output of our teams. In Thinking in Bets, Annie Duke describes how to make good decisions, informed by…
Delievered at: All Things Open 2019 . Slides available here . PyData Austin 2019 . Recording available here . Find me on Twitter @SamuelDataT . Transcript Have you ever applied for a credit card? I know, I have. A few weeks ago, I was up late at night. And I suppose that my idea of a fun time is to try to get some credit card rewards. So all my friends are out partying, and I was like, "Man, I'm…
Think about the last time you gave a presentation. Were you nervous? Excited? Scared to look stupid in front of your boss? Confident in your ability to wow the others in the room? These emotions reveal that at our core, our biggest desire when presenting is to make ourselves look good. This desire is antithetical to a good presentation and harmful if not addressed directly. Our first priority must…
As developers, we deplore being interrupted. A stream of Twitter posts lament that a five minute meeting can harm productivity for hours. But it doesn't have to be like this: we can dramatically decrease the cost of being interrupted with a simple technique. What's more, implementing this technique makes us more effective and confident in our work. Problems with this narrative I can't talk about…
Feature importances are, well, important. We can use them to provide a rudimentary level of interpretability; if a feature has higher importance, it has greater impact on the target variable. Some machine learning models have an innate way of calculating feature importance (decision trees, for instance). Others don't have a way of doing this (for example, support vector machines using an RBF…
I like to keep track of how many miles I'm driving in my car. One conceivable way of doing this is to create a table in a Postgres database in which I can track this information. CREATE TABLE mileage ( observed_date DATE , observed_mileage INTEGER ); Unfortunately, I'm not always the most regular data collector. I often collect this data with gaps of days or months between each reading. INSERT…
Delivered at: AnacondaCON on 10 Apr 2018. Slides available here . Video embedded above. Windy City DevFest on 1 Feb 2019. Slides available here . Video available here . Transcript Have you ever applied for a credit card? A few weeks ago I was up late at night and apparently my idea of a good time is to try to churn some credit card rewards. So I was on a bank's website entering information about…
My first job at a real software company was after my sophomore year in college. As you might imagine, I learned a ton that summer. I learned about how companies organize themselves. I learned how software teams organize around different pieces of the product. I learned specific technical things. I learned how to read code. I learned about navigating large codebases. But the thing that that stuck…
Every once in a while, I'll have a need to do a one-to-many join, but keep only a certain row in the "many" table. For instance, say we have a system to track inventory in retail stores: CREATE TABLE store ( id SERIAL PRIMARY KEY , name TEXT ); CREATE TABLE inventory ( store_id INTEGER REFERENCES store , quantity INTEGER , item_name TEXT , PRIMARY KEY ( store_id , item_name ) ); INSERT INTO store…
It's a few months ago. I'm enjoying my job and trying to bring value to my team. A high-visibility project comes up. Despite being unfamiliar with the framework we plan to use, I'm excited to work on this project. I set out to "learn by doing", implementing a chunk of the project with this framework. After reading some code, writing some new code by trial and error, and asking a lot of questions,…
Photo by Álvaro Serrano I am not a poet. But when I was barely three years old, I was sitting at the kitchen table with my mother and older sister. They were working on rhyming words, because that is apparently the stage of childhood development my sister was in. My sister was struggling a little. My mom would ask her, "What's a word that rhymes with ball?" My sister replies, "bend." Mom says,…
We live in a world of imperfect information. When faced with a lack of data, we can make a guess. This guess could be far from the truth; it could be spot on. In Monte Carlo simulation, we repeatedly make guesses of some unknown value according to some distribution and are able to report on the results of that simulation to understand a little bit more about the unknown. While any one guess may be…
Delivered at: DevSpace on 14 Oct 2017. Slides available here . DataSciCon on 30 Nov 2017. Slides available here . Video available here . Transcript A wise man once said, "I got ninety-nine problems," and I can relate to that in some sense. Because on a day to day basis I run into problems; I run into things that aren't as easy as they should be or things that I want to be better. And I suspect…
Learning is a crucial part of being a software developer. Increases in knowledge and skill can make a significant impact in our work. But in an age where everyone seems to be in a time crunch, we want our learning to be as effective as possible. We want to get as much learning out of our books, courses, and videos as we can. Counterintuitively, I believe one of the best ways to maximize knowledge…
Are you familiar with the __call__ method in Python? By defining this method, an instance of your class can be called as though it were a function. Here's a contrived example solely to demonstrate how it works: class Foo : def __init__ ( self , name ): self . name = name self . call_ct = 0 print ( self . name , 'initialized' ) def __call__ ( self ): self . call_ct += 1 print ( self . name , self .…
Conferences can be both fun and valuable, but they can also be a huge waste of time. Here's some things I've learned that have helped me get the most out of conferences as an attendee. Consider skipping talks This advice sounds crazy, I know; the whole point of conferences is to hear the speakers, right? I'm not convinced. The most valuable experiences I've had at conferences are centered around…
Sometimes when I'm working on a project that involves web scraping, the actual scraping starts to slow me down. If you've ever re-run a script and then sat for a few minutes while your computer re-scraped the data, you know what I'm talking about. I've found two simple and practical ways to make this process significantly faster. For the sake of example, say we're crawling two links deep on the…
On his podcast Revisionist History , Malcolm Gladwell talks about the difference between "weak link" and "strong link" sports. A "weak link" sport is one in which the worst player's skill level has a large impact on the team's success. The example Gladwell gives is soccer, in which a long chain of events must go perfectly to score a point. To get the ball from one side of the field to a position…
Like many digital builders/hackers/makers, I have several projects that I want to put on the internet. I don't, however, want to have loads of servers to manage. Given that the highest-traffic project I maintain only around 100 unique visitors a day, it's feasible for me to host them all on the same node. I already deploy these things in Docker containers, so I imagined I could use nginx to solve…
The Braindead Megaphone, George Saunders This collection of nonfiction essays was enjoyable and thought-provoking. Saunders writes about the Iraq war, illegal immigration, and Dubai. He presents his experiences in a very relatable way. The Crying of Lot 49, Thomas Pynchon I didn't like this one. I put it down about halfway through; I just couldn't get into it. I didn't find anything about it…
Being present (paying attention to the here and now) and purposeful (taking actions to achieve some end) helps us to live more fulfilling lives. Jim Elliot once said, "wherever you are, be all there." While it is often painted on flowery backdrops and put on the walls of suburban homes, this quote holds useful advice for productivity and professional growth. If you're doing something, you should…
How many times in a row do I typically sneeze? Ever since I can remember, I've sneezed atypically. While most people sneeze once and are done, I often sneeze 3, 4, or even 8 times in a row. I was curious to see how often I sneeze a certain number of times in a row, so I kept track of every time I sneezed over the course of a week. Here are the results: All this time, I though it was fairly rare…
After we compile our code, why do we keep it? The machine is able to interpret the compiled code and perform the function we wanted it to, so why keep the original code around? Often, the answer to this question is that we may need to modify the program in the future. Modifying a bunch of 1's and 0's in a compiled file would be very costly in terms of developer time. We keep the original code…
On 18 Oct 2016, I gave a talk at Austin ACM SIGKDD on the k -nearest neighbors algorithm. Topics included some machine learning theory (approximation vs. generalization, VC dimension), the algorithm itself, proving the algorithm's performance, and some practical concerns around choosing k . Some other topics that I would probably include next time are similarity functions, high-dimensional spaces,…
Default arguments What is the output of this code? def foo(arg=[]): return arg my_list = foo() my_list.extend('abc') print(foo()) My first intution was that this would output the empty list ( [] ). However, the output is: ['a', 'b', 'c'] Why does this happen? Well, Python evaluates default arguments when the function is defined, rather than when it's run. As a consequence, if a default argument is…
Station Eleven, Emily St. John Mandel I picked up this book purely for entertainment, and it served that purpose well. The pacing is quick (at times excessively so), and the events are interesting. I found nothing about it revolutionary, but it was well-written. In particular, I liked the dialogue because it felt real (perhaps accentuated by the fact that I read the book aloud). As events unfold…
There are four hard things in life. If you can do these four things, the rest comes easy: Working hard Doing your best Telling the truth Taking responsibility for your actions My brothers and sisters, whenever you face trials of any kind, consider it nothing but joy, because you know that the testing of your faith produces endurance; and let endurance have its full effect, so that you may be…
This month, I read four books: Till We Have Faces, C.S. Lewis A Walk in the Woods, Bill Bryson Of Mice and Men, John Steinbeck Into the Wild, Jon Krakauer Till We Have Faces, C.S. Lewis I really enjoyed this one. It is a re-telling by Lewis of myth of Psyche and Cupid. I was unfamiliar with the original myth, but appreciated this book very much. I particularly enjoyed the clash in worldviews…
A View from the Bridge, Arthur Miller This is a script I read for my American Literature class. It was alright. Even though it has a distant setting, the characters still feel real. They each have their own needs/desires which conflict in certain ways, which causes the drama/action. My favorite scene is one in which they're all in the apartment, and Catherine and Rodolfo start dancing. The subtext…
Redirect, Timothy Wilson A book from a college professor about his (and others') research in "story editing" techniques that target the narratives we tell ourselves. I particularly appreciated the thoughts on happiness, as those are the most widely applicable. Because I'm not a parent, I couldn't quite get into the stuff about parenting as much. The discussions of various societal issues and how…
collections.namedtuple namedtuple can make your code a lot more readable. In an interview, that's helpful for a few reasons. First, it can help you demonstrate a good understanding of some of Python's standard libraries. Second, it helps you show off that you place importance on writing readable code. Third, it makes writing your code easier. If you're passing around tuples, it can be easy to…
Idea Two weeks ago, I went to HackTX and won a Leap Motion. While thinking about what things we could build with skeletal tracking and gesture recognition, I thought it would be cool to build a language learning tool (like Rosetta Stone) for American Sign Language. My friend Matt also thought it sounded cool, so we decided to build something like that at TAMUHack. Environment TAMUHack was fun. The…
BitTorrent is an inherently political technology which embodies decentralized political order. Additionally, it broadens the definition of art. Despite the negative side effects of the technology, BitTorrent is worth pursuing. What does it mean to say that a technology is political? Winner outlines two possibilities: I…offer…two ways in which artifacts can contain political properties. First are…