Build a Web Scraper in Python
Build an SEO-flavored crawler in Python that maps internal links and teaches you how scraping really works. Also available in:
Also available in:
Author
Boot.dev Team
Our expert in-house course authors are tenured software developers whose love for coding is only outstripped by a love for crafting addicting lessons.
6 Hours
of content
14 Addicting
lessons
Earn a certificate
of completion
Learn online
at your pace
What you'll learn in this Python course
Use Python to build a web crawler that scrapes pages. It's a tool that any SEO expert would be happy to have. You'll make HTTP requests and parse HTML to generate reports that can easily marshaled to standard output or a file. If you're interested in getting a job doing data analysis, this project will teach you how Python can be used to build a command-line application.
Chapter List
Join 989 students in the Build a Web Scraper in Python course
Read reviews of their learning experiences
The web crawler project is solid, but the async version is simultaneously too hand-holdy and hard to debug—the Data Analyst path doesn't cover the subtleties of async and recursive concurrency beforehand.
nokorikage
The Void
Very difficult course but very rewarding. However, I think it is best done alone as talking to boots only made things complicated
Jhomel Panogao
Philippines
Challenging, enjoyable course. Gives you working experience with async and http libraries. It's a great starting point to launch into more extensive analysis project opportunities.
Kaelin Curtsinger
Louisville, Kentucky
This was a fun and challenging (especially the concurrency stuff) guided project. I have a better handle on concurrency from taking this course.
Peter Schwenk
Newark, DE, USA
Hard and fun course
Diego Herrera
Costa Rica
Great project. The toughest part by far is concurrency (asyncio). In my view the level of difficulty is underrated in that respect.
Michael Martin
Andorra
Very hard, you will need to read some documentation.
Romulo Ferreira da Silva
São Carlos
Great project and test of skills, with real application. This project could be used as a piece of bigger projects
Jake Mackin
Pittsburgh, PA
good
WICKLIFF ORINA
Kenya
Showing 1-9 of 18
Mediocrity doesn't cut it anymore
The only way to become a great developer is to write a lot of code
Avoid tutorial hell
by writing a ton of code
Stay motivated with
a game-like curriculum
Build portfolio projects
to prove your skills
Delve deeper
into foundational concepts
Learn flexibly online
without interrupting your life
For 1% the price of college
to minimize your financial risk
Frequently asked Questions
Got questions? We've got answers
Yes! It's free to create an account and start learning. You'll get all the immersive and interactive features for free for a few chapters. After that, if you still haven't paid for a membership, you'll be in read-only (content only) mode.
Platform
Languages
Subjects
Support
Pricing
Community
