This page cannot be shown here. You can still read it on the original site — the toolbar below keeps your place in the directory.
crawdad is a simple, yet powerful alternative for scraping in a distributed, persistent manner (backed by Redis). It can do simple things, like generating site maps. It can also do complicated things, like extracting all the quotes from every page on a quotes website (tutorial follows). Install First get Docker which will be used for running Redis. Then you can simply download crawdad: $ wget…
crawdad is a simple, yet powerful alternative for scraping in a distributed, persistent manner (backed by Redis). It can do simple things, like generating site maps. It can also do complicated things, like extracting all the quotes from every page on a quotes website (tutorial follows).
Install First get Docker which will be used for running Redis.
Then you can simply download crawdad:
$ wget https://github.com/schollz/crawdad/releases/download/v3.0.0/crawdad_3.0.0_linux_amd64.zip $ unzip crawdad*zip $ sudo mv crawdad*amd64 /usr/local/bin/crawdad Unlike many other scraping frameworks, crawdad is a single binary that has no dependencies.Read on /software/crawdad/ ↗
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.