Pulling research into Claude has gotten easy. Tavily searches the web. Exa reads by meaning. Apify scrapes locked platforms. Firecrawl deep-reads a single page. Most people pick the one they’ve heard of, then wonder why it keeps hitting walls. Not because the tool is bad. Because they grabbed one tool for a job a different one owns. This guide names the four research MCPs worth keeping, and the job each one owns. Then how to run all four on free tiers.
In our Practical AI Builder inner circle, a member told me they love to use Firecrawl as the research MCP.
Then they asked: is there any reason to use a different MCP instead?
Yes. There is.
You’ve probably done the same thing. You find one scraping tool that works, and you point it at everything.
Then it hits a wall.
Amazon throws a login wall. A run reports “SUCCEEDED,” returns zero rows, and still bills you.
That was me.
There isn’t one best scraper. There are many. These four are my favorites, and each owns a different job.
Firecrawl deep-reads a URL you already have. It can’t find where the signal is. It can’t punch a login wall.
Search is one job. Meaning-match is another. Cracking a locked platform is a third.
I tested all four across 15 platforms: Reddit, Amazon, YouTube, Hacker News, Product Hunt...
Real runs. Real costs. Real walls.
One MCP billed me ten cents and returned nothing. Another pulled 25 clean posts for seven.
Match the tool to the job, and the walls mostly disappear.
This guide is that map.
Why one tool hits wall after wall: login walls, blocks, billable dead-ends
Tavily, Exa, Apify, Firecrawl: the four, named, one per wall
Mapping the territory: the best one for fast discovery
Find the complaint you can’t name: meaning-match, not exact words
Cracking the locked platforms: data where search can’t reach
Reading any URL: the deep-reader for a page you have
The Routing Card: which MCP for which job
By the end: you can reach for the right research MCP for any job, on free tiers, without forcing one tool everywhere
🎁 The Research MCP Routing Card: which MCP for which job, plus the free-tier setup for all four. Grab it at the end.
Point one MCP at every target and you don’t hit one wall. You hit a different wall each time.
I learned the four the hard way, testing across 15 platforms.
The walls one tool keeps slamming into:
The login wall. Amazon returns a sign-in page, not reviews. Firecrawl scrapes it and gets the login screen.
The engine-level block. X refuses every scraper before the request even runs. The response is “all scraping engines failed,” not a page.
The billable zero-row run. On Reddit, an Apify actor on a datacenter proxy returns nothing, reports “SUCCEEDED,” and still charges the start fee.
The snippets-only wall. Search LinkedIn and you get the post title and one line. The full post and its comments stay behind the login.
None of these is your fault.
Each is a different defense. No single tool is built to clear all four.
That is why “just use the one I like” breaks. You are not using a bad tool. You are using one tool for a wall it was never built to climb.
So I stopped hunting for the best scraper and started matching the tool to the wall.
Pulling research into Claude looks like one task. It’s four, and the walls from a minute ago were all the same mistake: one tool sent to do all of them.
Four jobs, and a different tool owns each:
Tavily finds where the signal is. Point it at the open web and it returns the threads, issues, and posts worth reading.
Exa matches by meaning. Describe the complaint you can’t keyword and it surfaces the post that never used your words.
Apify extracts from locked platforms. A marketplace of ready-made scrapers for the sources that demand a login: Reddit, YouTube, Amazon.
Firecrawl reads a page you already have. Hand it a URL and it comes back as clean markdown.
Find, match, extract, read. The searcher can’t read; the reader can’t search. That’s why the one tool you like keeps hitting a wall: you’re handing it three jobs it was never built for.
Knowing what each does is the easy part. The how, and which to reach for when, is the rest of this guide.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.