Web Scraping Boilerplate: Everything You Need to Start Your New Python Scraping Project (Batteries Included)
Hit the ground running on your next web scraping project, reusing my free github repository.
Hartley Brody - Articles
Hartley writes about full stack software development, marketing, and web scraping. Based in Boston, MA.
Latest posts
Web Scraping with Proxies: The Complete Guide to Scaling Your Web Scraper
Everything I tell my clients about how to purchase, configure and manage a strong proxy infrastructure for web scraping.
How to Avoid Bad Clients: 12 Red Flags I Look for to Dodge Nightmare Projects
A bad client can be an absolute nightmare, robbing you of your time, sanity and in some cases -- your paycheck. Here's what I've learned to look out for over the last 3 years.
Startup Security Guide: Minimum Viable Security Checklist for a Cloud-Based Web Application
Before you launch your new product to the world, make sure you're following the basic security guidelines in this checklist.
3 Years of Consulting: Settling into An Unexpected Career
I've now been an independent consultant for longer than I was ever a full time employee. I figured it was a good opportunity to peel back the curtain a bit and offer a 'behind the scenes' look at what life as a consultant has been like for me.
USB Rubber Ducky Tutorial: The Missing Quickstart Guide to Running Your First Keystroke Payload Hack
The USB Rubber Ducky is an awesome device for infosec testing and general mischief. Here are the missing step-by-step instructions for setting it up.
Building Facebook Messenger Bots That Don't Suck: 9 Anti-Patterns and Best Practices
A few days ago, the Facebook Messenger team launched version 1.4 of their platform . The most prominent changes for most bot developers are the enhancements to the persistent menu options. A lot of commentators noticed that the updates seem to steer bot developers away from building conversational, chat based interfaces. Indeed, there is now the ability for bot developers to completely disable the…
Second Edition of "The Ultimate Guide to Web Scraping" is Now Available
After four years, I figured that my book, The Ultimate Guide to Web Scraping was due for an upgrade. The internet has changed, websites have gotten more advanced, and I’ve learned a lot of new skills and tactics over the years that I wanted to incorporate into the book. I’ve also gotten a lot of feedback from some of the over 1,300 readers that helped to make the second edition of the book even…
Web Scraping Reference: A Simple Cheat Sheet for Web Scraping with Python
Once you’ve put together enough web scrapers, you start to feel like you can do it in your sleep. I’ve probably built hundreds of scrapers over the years for my own projects, as well as for clients and students in my web scraping course . Occasionally though, I find myself referencing documentation or re-reading old code looking for snippets I can reuse. One of the students in my course suggested…
10 Debugging Tips for Beginners: How to Troubleshoot and Fix Your Code Without Pulling Your Hair Out
This past summer, I launched a web scraping class targeted at total beginners . After talking to a lot of people who were having web scraping challenges over the past few years, I realized that many of them weren’t coders and didn’t have any technical backgrounds. Many people who were looking to learn web scraping had never coded before . So, when I was making the material and lessons for the…
How to Scrape Amazon.com: 19 Lessons I Learned While Crawling 1MM+ Product Listings
In its simplest form, web scraping is about making requests and extracting data from the response . For a small web scraping project, your code can be simple. You just need to find a few patterns in the URLs and in the HTML response and you’re in business. But everything changes when you’re trying to pull over 1,000,000 products from the largest ecommerce website on the planet. When crawling a…
Facebook Messenger Bot Tutorial: Step-by-Step Instructions for Building a Basic Facebook Chat Bot
First there were desktop software products, then everything moved to the web. Then there were email-based products and even SMS-based ones. The latest craze in software interfaces is messenger bots, and Facebook has the largest chat platform by a long shot. In this tutorial, I’ll show you how to build your own Facebook Messenger Chat Bot in python . We’ll use Flask for some basic web request…
7 Reasons I Won't Sign Your NDA Before a Coffee Meeting
In my work as a full stack web developer , I often meet clients who request that I sign a Non-Disclosure Agreement (NDA) at various stages of the project. By signing an NDA, the client is basically asking me to agree that I won’t take their idea and work on it myself, or share it with anyone else who will. For most entrepreneurs, that sounds like a smart idea, right? Here’s why I won’t sign them.
The 3 Mistakes Every Junior Developer Makes (And How to Stop Making Them)
As more and more people are learning to code , there are an increasing number of new developers in the work force. Code bootcamps are springing up everywhere that promise to land candidates a job with only a few months of experience. Having worked with a number of junior developers (and having been one myself, at one point ;) ) I’ve noticed a lot of the same mistakes crop up. In an effort to help…
Scaling Your Web App 101: Lessons in Architecture Under Load
It’s the classic champagne problem that most successful web apps will deal with – there are so many users on your site that things are starting to get bogged down. Pages load slowly, network connections start timing out and your servers are starting to creak under heavy load. Congratulations – your web app has hit scale! But now what? You need to keep everything online and want the user’s…
Focus on the Product, Not the Code: How I Build Software for Clients
While there are tons of resources designed to help people learn to code , there aren’t as many resources for helping people learn to build software products, at a higher level. “Writing code” is largely a vocational skill, just like swinging a hammer is – but presumably you’re using that skill to actually build something. In my experience, the reasons software projects fail – take too long, go…
How I Learned to Code in Only 6 Years: And You Can Too!
A few weeks ago, a friend texted me asking for advice. She was interested in learning how to code and wanted to know how I had done it. While I did take a handful of Computer Science classes in college, I consider most of my relevant, day-to-day software development skills to be things I picked up through self-guided learning. My initial advice for her was going to be pretty banal – sign up for…
Look Ma, No Servers! How Javascript is Changing the Modern Web Stack
Formerly titled “The Rise of the Server-less Web Stack” Javascript has lots of cool stuff built on top of it now. These days, there are tons of well-worn frameworks that bring all sorts of powerful programming paradigms into the browser. Want easy object-orientation? Use backbone. More of a functional programmer? There’s underscore, lodash and many others . And I can’t keep up with the latest…
Minimum Viable Git Best Practices for Small Teams
When I started as the first employee at Burstworks, the cofounders and I could easily hold the information about who was working on what at any given moment in our brains. But as we worked on new projects and the scope and size of the engineering team grew, all of our code mostly stayed organized in one central repository: Our high-performance ad server Data Pipeline One-off scripts Nightly jobs…
Lightning Fast Data Serialization in Python
A few months ago, I got a chance to dive into some Python code that was performing slower than expected. The code in question was taking tiny bits of data off of a queue, translating some values from strings to primary keys, and then saving the data back to another queue for another worker to process. The translation step should have been fast. We were loading the data into memory from a MySQL…
Preventing Web Scraping: Best Practices for Keeping Your Content Safe
Many content producers or site owners get understandably anxious about the thought of a web scraper culling all of their data, and wonder if there’s any technical means for stopping automated harvesting. Unfortunately, if your website presents information in a way that a browser can access and render for the average visitor, then that same content can be scraped by a script or application. Any…
The Full Time Employee's Guide to Generating Freelance Clients on the Side
There are many reasons to begin freelancing while still maintaining a full time job. Whether you want to work on new projects outside the scope of your current role or make some extra money each month – or maybe you’re hoping to eventually jump into freelancing full time – getting your first few clients can be really rewarding. But how should you get started? I’ll cover that in this article. We’ll…
Moving a Static Site to S3 Before My Girlfriend Got Out of the Shower
I’ve got an old Rackspace instance that I’ve been running a bunch of small sites on over the past 4 years. Lately it’s been causing me problems and sites will sporadically go down from time to time. I have been meaning to move several of the static sites onto a more appropriate static-file hosting service like Amazon’s Simple Storage Service, also known as simply “S3”. I’m on a trip in Denver with…
Becoming a Cold Weather Adventurer: Notes from MIT Outing Club's Winter School
Growing up, I was always an outdoorsy person, but cold New England winters kept me cooped up inside for a big chunk of the year. Last winter, I decided to take my first winter mountaineering and ice climbing lessons to start building the skills to become a year-round adventurer. Earlier this winter, a friend told me about MIT Outing Club’s annual Winter School in January. It was 16 hours of…
Peeling Back the ORM: Demystifying Relational Databases For New Web Developers
Most web developers building dynamic websites interact with databases every day. Relational databases like MySQL or Postgres are usually the first tool people reach for when their application needs to store data. But with the recent proliferation of web frameworks like Rails and Django, many web developers rely totally on Object-Relational Mappers (ORMs) for interacting with their database. In…
The "Ultimate Guide to Web Scraping" is Now Available
Feb 2017 Edit: The book has been updated. I wrote an article on web scraping last winter that has since been viewed almost 100,000 times . Clearly there are people who want to learn about this stuff, so I decided I’d write a book. A few months later, I’m happy to announce: The Ultimate Guide to Web Scraping . No prior knowledge of web scraping is necessary to follow along – the book is designed to…
How HTTPS Secures Connections: What Every Web Dev Should Know
How does HTTPS actually work? That was the question I set out to solve a few days ago for a project at work. As a web developer, I knew that using HTTPS to protect users’ sensitive data was A Very Good Idea, but I didn’t have much understanding about how it actually worked. How was data protected? How can a client and server create a secure connection if someone was already listening in on the…
Are You A Hacker, Developer or Engineer? (And Why it Matters)
In the software world, the terms “developer” and “engineer” are often used interchangeably to mean “someone who builds things with code.” Sometimes the word “hacker” gets thrown into the mix if the company is a startup or is trying to make an open job position sound more enticing. But what does it mean for someone to be a “developer” versus an “engineer”? Does it matter? If you’re trying to “level…
When Will We Stop Calling it "Mobile"?
“Mobile” is easily one of the biggest buzzwords of the decade, and everyone is starting to grok it – from marketers and web developers to your parents and grandparents. But what do we really mean when we talk about “mobile” in a development context? Usually, “mobile development” refers to developing native applications in Objective C or Java that take advantage of device-specific-APIs. In that…
Dependencies: Why Building Modern Applications is Like Using Magic
I launched another small app yesterday called Rooster, an SMS service that messages you the forecast every morning . It only took a few days to build and polish. But while the interface is remarkably simple, there’s a lot going on behind the scenes: User-provided location strings have to be converted to latitude & longitude pairs. Weather reports from dozens of agencies have to be aggregated and…
So You Want to Build a Chrome Extension?
I should start by saying that I just released my first chrome extension this afternoon called BuzzKill and I think you’ll like it: BuzzKill is a Chrome Extension that automatically removes all Buzz Feed content from your Facebook browsing experience. "This extension has helped me reclaim hours of my day!" -Happy User "I finally feel like I have my life back" -Happy User "With BuzzKill installed, I…
A Quick Primer on Writing Readable Python Code for New Developers
In general, code is written once in a few minutes (or hours) and then read and maintained for years to come. Ensuring that the code you write today is readable and makes sense to other engineers down the line is a Really Good Thing and you should always keep readability in mind as you’re writing code, even for small one-off scripts. This is an adaption of an article I wrote on the engineering wiki…
Become an Unsuspecting Hero
In the moments immediately after yesterday’s bombings , there were lots of volunteers and bystanders who didn’t expect to become heroes, but did. Their immediate actions saved lives. Every single citizen should know CPR and how to handle major blood loss events. If you don’t, please please read up and take a class. Spending a few hours learning these things now could give you the skills that end…
Convert New Users to Evangelists: Understanding the Post Sign Up Funnel
What do you want your users to do after they sign up? While getting people into your product is obviously A Very Good Thing, it isn’t time to pat yourself on the back just yet. To build an awesome product experience and an engaged user base, you really need to carefully focus on what happens after a user has signed up. Once they’re in the door, you want to show them value quickly. Help them avoid…
How to Set Up WordPress for Maximum Awesomeness
When most people set up their personal WordPress blog, they’ll pick a theme, throw the Google Analytics tracking code on it, and stop there. And while WordPress gives you a decent website out of the box, there’s a lot left to be desired. With an extra hour or two of setup, you can have a well-optimized, secure blog that pulls in new visitors and keeps out intruders. Most of these things are…
Startups: Get More Customers From Your Website by Understanding the Funnel
One of the rookie mistakes first-time entrepreneurs often make is to assume that visitors on their website are ready to buy their product or service right away. I’ve made this mistake myself. But in reality, the average visitor to your website isn’t ready to make a buying decision… yet. In order to grow a successful business, you need to understand the concept of “the funnel” and the stages that…
Overthinking Things
I’ve always been one to shoot from the hip. I go with my gut, and then reevaluate if necessary. After all, smart people change their minds a lot . But I’ve found it’s also really easy to get stuck overthinking some things. What tie should I wear? Which apartment should I live in? What do I want to do with my life? Sometimes there are so many options that we’re overwhelmed. Sometimes we get…
Two Startups in 9 Months: Lessons Learned as a Technical Co-Founder
The past few months, I’ve worked on a half-dozen side projects that were web-based applications. Two of them were even so big that I formed an LLC with my co-conspirators, did some marketing and tried my hand at a few sales, and even looked at raising a bit of money . The first product was a social network where students could share outfits and fashion ideas [ archive ]. The second was a music…
Growth Hacking: The Tale of a Marketer Who Writes Code
Last spring, I graduated from college and began working as a software engineer on the Product team at HubSpot . But a few weeks ago, I switched to the Marketing team, where I now work on SEO and driving leads from organic search and referral traffic. My path is an unusual one, and it puts me in quite a hybrid position. My day-to-day work is much different than it was on engineering – there are no…
I Don't Need No Stinking API: Web Scraping For Fun and Profit
If you’ve ever needed to pull data from a third party website, chances are you started by checking to see if they had an official API. But did you know that there’s a source of structured data that virtually every website on the internet supports automatically, by default? That’s right, we’re talking about pulling our data straight out of HTML – otherwise known as web scraping. Here’s why web…
Learning to be Choosy
One of the first emotional responses we must learn as children is how to handle disappointment and loss. Whether it’s learning to share a toy or coping with the loss of a family member, our formative years offer constant reminders that life doesn’t always work out how we want it. Coming to terms with that is a huge part of growing up. But I’m starting to see the other side of that coin. As a young…
Finding the Weather Balloon That Went to Space
In my last article , I talked about how I sent a weather balloon and video camera into space, but needed help finding the footage. I posted the article to Hacker News and promptly fell asleep, exhausted after a long day of adventure. When I woke up, my inbox was full of comments from people all over the world showing support and offering to help. Apparently, the post made it to the #3 spot on the…
Sending a Weather Balloon to Space
Like most of the nation, last weekend I watched the Red Bull Stratos live stream of Felix Baumgartner’s record breaking jump. For me, one of the best parts of the stream was the moment when the cabin finally depressurized and the door slid open. Suddenly, you could see both vastness of space and the bright blue curvature of the earth over his shoulder, and it took my breath away. It had always…
Practical Reasons to Choose Open Source
The Free and Open Source Software movement ( FOSS ) is almost as old as the market for software itself. Back in the 1980s, Richard Stallman started the Free Software Foundation as a reaction to the growing commercialization of software. He framed the issue not as a commercial one, but as a moral one: Thus, “free software” is a matter of liberty, not price. To understand the concept, you should…
The Real World: A Four Month Retrospective
Just a quick collection of things I’ve learned as I’ve transitioned from college into “the real world” these past 4 months. You’re a lot less active. On campus, I was always walking or biking around. Not only cause I enjoyed it, but because I had to. Every class, every meal, and every study group or party meant walking at least a few hundred yards. And I’d rarely be in one place for more than two…
Gangnam Style Parody
It’s been a while since HubSpot has made a fun video. The last one I was a part of was actually one of my first projects at HubSpot last spring. Huge shout out to HubSpot’s awesome video team and to all the background dancers (both the volunteers and the unwitting). I’m glad I get to work for a company that has this much fun. .
Asynchronize All The Things
One of the first steps of tackling a new web application is figuring out where bottlenecks might arise. Now, you don’t want to go too far down this route, or you’ll end up designing yourself into oblivion. As Donald Knuth famously said : Premature optimization is the root of all evil. But it’s still good to have a sense of where your system might get bogged down under heavy load. For example,…
Global Dominance: Olympics vs GDP
Along with about 35 million other Americans, I’ve been watching the Olympics in primetime every night after work. And while NBC has done a notoriously bad job at keeping the production fair and balanced , it’s still pretty easy to tell that some countries are pulling in way more medals than others. Currently, the United States has the most gold medals while China has the most overall medals and…
Saving Your First Million Dollars
A lot has been written on the topic of personal finance, but it all comes down to a fundamental fact: In our lives, we earn money, and we spend money – and in between, we accumulate money. Pretty obvious, right? So if we want to accumulate more money, we have to either earn more of it, or spend less of it. Still with me? The Power of Wealth In my college economics classes, we rarely dealt with the…
3 Ways To Raise Your First Capital
Yesterday, Sam and I met with a VC in a shiny new office building in downtown Cambridge. We knew we were punching above our weight class meeting with top-tier VCs at this stage, so we were simply hoping to pitch our concept , demo the site and get some advice on how to make it successful and raise money. Inevitably, the conversation turned to our competitors. “This is a very, very crowded space…