You are probably asking the wrong question about scraping

Question of the Week — a weekly read on what our community is figuring out in real time

On Monday 20 July a member put a very specific problem to the chat. He wanted to scrape fund offering documents filed with Hong Kong’s securities regulator, and he wanted to build a database of the risk factors inside them. He listed four facts. Anyone can download the documents, no login and no captcha. The page sits on the site’s robots.txt disallow list. The documents are written by asset managers, not the regulator. The terms of use say nothing about scraping.

Then he added the line that made it a good question rather than a request for free advice: “I just thought it’s a common issue that all LQs are facing here.”