I launched a new little project called botstop.io. You open the page and it instantly tells you whether your browser looks like a bot, then refines the verdict a moment later. No CAPTCHA, no puzzles, just passive signals.
An update regarding the state of this blog
In the Internet, each IP Address belongs to an autonomous system (AS). In this blog article, it is demonstrated how any IP address can be mapped to an AS number. The necessary information and sources to map each IP Address to an Autnomous System is provided as well, since they were not easy to find.
Instead of using one of the many battle proofed and reliable database solutions out there, I rather created my own solution. In this quick blog post, I am announcing the release of db.js - A in-memory database with persistant file storage.
It is not entirely trivial to find out if an IP address belongs to a datacenter / cloud provider. In this blog article, I try to find an algorithm that outputs with high confidence if an IPv4 / Ipv6 address belongs to a hosting provider or not.
In this blog post, I highlight the core differences between TLS 1.2 and TLS 1.3 and investigate how we can use several properties of the protocol to obtain fingerprinting entropy from TLS clients.
I in this blog post, I am investigating the current state of high precision JavaScript timers. High precision timing techniques were mostly used to launch CPU-level cache attacks such as Spectre and Meltdown from the browser. I am interested in other use cases though...
I (maybe) found another method to detect browsers that route their traffic through SOCKS/HTTP proxies. What do you think? Is this a valid method to detect Proxies? I need your help!
What it really takes to scrape without getting detected.
In this blog post, I demonstrate 7 different efficient ways how to detect a proxy server when the client is visiting a web server with a browser that has a proxy / VPN configured.
My personal opinion on economical and sociological topics in the near (2030) and mid-term (2050) future. This essay is divided into the following categories: Climate Crisis, Globalisation & Economy, Geopolitics & War, Advance of Medicine & Technology.
There are unique challenges when developing a passive bot detecting system. In this blog article, I explain some of the obstacles that need to be overcome in order to detect advanced bots without presenting a CAPTCHA. I also explain how bot programmers can benefit from the architectural challenges that bot detection systems inherently suffer from.
For security reasons, it's often helpful to check if an IP Address belongs to a datacenter or cloud computing provider such as Amazon AWS or Microsoft Azure. Therefore, I have developed a simple public API that helps you to check if an IP address belongs to a datacenter / cloud provider.
VPN's and Proxy Servers can be detected by comparing latencies measured with JavaScript in the browser with the corresponding latency of the TCP/IP handshake on the server.
In this blog article I demonstrate several bullet-proof ways how to detect Brightdata Data Collector as a bot without any doubt.
In this blog post I explain why it is best to avoid puppeteer and playwright for web scraping.
Detecting proxys can't be that hard? Can it?
Behavioral analysis is an interesting approach to detect bots. It surely is not the panacea for bot detection, but it certainly is an useful extension in your bot hunting tool belt.
TCP/IP fingerprinting is as old as the Internet itself. But this technique seems to have lost it's relevancy in our modern times. However, with the rise of Proxy and VPN Providers, TCP/IP fingerprinting becomes interesting again from a security perspective.
In this blog post I will demonstrate how it is possible to detect several scraping services: luminati.io , ScrapingBee , scraperapi.com , scrapingrobot.com , scrapfly.io .
In this blog post, I am talking about my several year long experience with web scraping and common mistakes I made along the road. The more I dive into web scraping, the more I realize how easy it is to take wrong decisions when scraping a site. For that reason, I compiled a list of seven common mistakes in regard to web scraping.
Android mobile devices give to any website device orientation and device motion data. This data is quite sensitive in nature and should not be granted to websites without obtaining explicit user consent.
The following write-up is an attempt to launch headful Google Chrome with Xvfb on AWS Lambda container.
Advanced bots use modern browsers and automation frameworks such as puppeteer and playwright. It becomes increasingly hard to distinguish bots from real human traffic, therefore, new methods are required.
In this article, various techniques to conduct port scanning from within the browser are developed. Modern JavaScript is used.
In this project, I make use of a method from early 2019 that demonstrates how to solve the Audio reCAPTCHA with Google's own Speech to Text API. This method still works, which is quite astonishing.
In this tutorial it is demonstrated how an Express App is deployed with Nginx as reverse proxy. Static files are served with Nginx and the real IP address is forwarded to the Express app.
There are many resources in the Internet that show how to detect uBlock Origin and Adblock Plus. However, after some research, it became clear that most detection methods are unreliable and cease to exist after a while. In this blog article, a reliable detection method for uBlock Origin and Adblock Plus is demonstrated. No external libraries. Just plain and simple JavaScript.
In this blog post, I will introduce a JavaScript library that allows to track various user interactions of website visitors. Several key problems that arise when creating a JavaScript analytics application will be discussed and solved in this blog post.
The chrome browser controlled via puppeteer doesn't support the dynamic change of proxies without restarting the browser. In this tutorial, I demonstrate how to implement this functionality with the help of a third party npm module named proxy-chain . This module acts as an intermediate proxy.
I am a heavy user of YouTube. I use it to listen to podcasts while cooking or in order to watch the latest documentaries before going to sleep. But lately, the extremely aggressive advertisement of YouTube sparked enough motivation within myself to remove YouTube ads for good. Google overdid it. I have enough.
Cross domain requests with <img> tags are not bound to the same origin policy. I will shed light on several possibilities how malicious web site owners can potentially abuse cross domain request done with <img> and script tags created with JavaScript.
In this article, I demonstrate how to reliably communicate JSON data to a cross domain server after the user is about to end or interrupt the browsing session by either: switching the focus to another page switching from the browser to another applicaton closing the tab closing the browser or any other means of terminating or interrupting the current browsing session. Mobile devices and desktop…
In this blog article I will introduce my most recent project: The distributed crawling infrastructure which allows to crawl any website with a low-level Http library or a fully fledged chrome browser configured to evade bot detection attempts. This introduction is divided into three distinct blog articles, because one blog article would be too large to cover this huge topic. (This article) The…
Quick script that demonstrates how to create s3 buckets in many regions.
This article makes an attempt to understand and predict the consequences of the rapid automation/computerization in the realm of human work. I make them based on my experiences as a software engineer, while I am fully aware that programming is not threatened to be eradicated by automation in the next decades. For that reason, I realize that I am holding a privileged position. In the second part of…
Find the updated blog post here. Chrome/Puppeteer has a couple of annoying issues when trying to use http/s proxies and socks proxies with the chrome browser controlled by puppeteer. The most pressing issues are the following: Dynamically changing proxy servers: Once the chrome browser is started, it is not possible to change the proxy configuration any longer. A restart is required to switch…
This blogs post demonstrates how puppeteer and the chrome browser can be used with http/s and socks4/5 proxies. For that reason, a proxy server is setup on Ubuntu 18.04 with squid3 and dante .
Thailand is the second most deadly country when it comes to traffic accidents. 80% of all deaths originate from people driving motorbikes. In this blog post, I try to share my experiences in the form of 5 survival tips in a honest way. I drove a scooter on 4 distinct tourist destinations in Thailand without a proper license and wasn't caught in a police checkpoint a single time.
In this quick blog post I outline why you probably don't want to go to Koh Phi Phi.
The results of my Master thesis named Model based fuzzing of the WPA3 Dragonfly handshake will be quickly discussed in this blog post. No severe vulnerabilities were discovered, mostly due to the limited deployment of WPA3 hardware since it's introduction. However, a DoS vulnerability in iwd was found.
Travel experiences during the days of the full moon party in Koh Phangan. The Jungle Experience Party is a huge scam, the Waterfall Party on Koh Phangan might be even worse.
In this blog post, it is explained how a lack of perfect information about the market allows the clever middleman to connect market supply with market demand by advertisement scrawping and lead crawling.
Scraping one million keywords is not a easy task. There are proxy problems, big data problems and reliability issues. In this blog post, the most valuable insights are shared.
In this blog post, we demonstrate how a web scraping function is deployed to the AWS cloud with puppeteer and headless chrome.
Implementing possible fuzzing strategies with boofuzz against the WPA3 SAE Dragonfly handshake. Dragonfly is the main ingredient of WPA3 certified routers and 802.11 devices.
I will shop an alternative approach to web scraping without using css selectors and XPath queries. We make use of the fact that most web pages visually render the information of interest in a coherent, structured way. This technique requires a remotely controllable web browser such as puppeteer, that is capable of rendering web pages visually.
A captcha is a mechanism to distinguish human users from automated programs (bot). There are many service providers in the Internet that have a major incentive to prevent bots from (ab)using their systems.
Tutorial that shows how to run an WPA3 access point with hostapd 2.7 and SAE Dragonfly Handshake.
Modern scraping now is mostly done with real browsers, configured to behave like real humans.