Recent content in Posts on Trickster Dev
The most common way to use mitmproxy for API traffic interception is to use it in a default forward proxy mode. One would run mitmproxy server on a laptop or desktop computer, then configure proxy settings and install X.509 certificate on a client device (e.g. smart phone). But there is more to mitmproxy than that. It can also be used as reverse proxy, transparent proxy (on Linux and macOS),…
When looking into some targets for web scraping, you may come across pages that contain a lot of data represented in JSON-like (but not quite JSON) format passed to self.__next_f.push() Javascript function calls. What’s going on here and how do we parse this stuff? To understand what this is about, we must go through a little journey across the technological landscape of the modern web. Note…
Previously I have covered how to set up mitmproxy with Android emulator or iOS device for the purposes of mobile app traffic interception. However, there is more to mobile app hacking, as not all apps allow their API calls to be hijacked by the simple setup I have described earlier. Some of the more secure apps implement a security mechanism called X.509 certificate pinning that entails checking…
JSON is the dominant data representation format for most of the modern RESTful APIs and certain automation/devops tools (Terraform, AWS CLI, kubectl, etc.) optionally output data in JSON format to make it machine readable. Some datasets (e.g. service price lists from health insurance companies) are available primarily as JSON files. For these reasons parsing, generation, modification and analysis…
Katana is CLI tool and Go library for automatically traversing (crawling) across all pages of given website(s) to map them out. It can work in two main modes - requests-based and through browser automation (headful or headless). To allow for discovery of API endpoints it can optionally do JavaScript parsing even when running in requests-based mode. Furthermore, Katana can do passive crawling by…
In this article we will explore some lesser known, but interesting and useful corners of Python standard library. Python dictionaries and lists are bread and butter for many applications, but might be too simple for more advanced data organisation. To provide more powerful containers for storing data in memory Python ships a collection module with things like: Deque - list-like data structure for…
Informational advantage is a form of power. On the flipside, exposure of sensitive information about a person or organisation can be a privacy/security problem. Sensitive Data Exposure is a type of vulnerability where software system (or user) makes sensitive data (API keys, user information, private documents, etc.) available to potential adversaries. For example, web app that lets users edit…
In computer networking a proxy server is intermediate party (software that may or may not be running on a separate server) to transfer application-level traffic between client and another server - a real destination of the connection. There are two major ways this can be done: message forwarding (largely limited to plaintext HTTP) and connection tunneling. SOCKS is a widely utilised protocol that…
Clutch.co is a web portal serving as B2B service company directory. One may want to scrape it to make a list of companies within a service niche to target them with some form of outreach. Company name, website URL, phone number, hourly rate, project size, social media URLs would be fields of the dataset we would create by scraping this site. But the thing is, Clutch is fighting scraping attempts…
SSH is well established protocol for securely accessing remote systems over the network for administration and devops purposes. A widely deployed OpenSSH software suite implements this protocol. But there is more to the SSH technology than reaching a remote shell over ssh(1) or copying files via scp(1)/sftp(1). We will go through some lesser known, somewhat advanced tricks and use cases that could…
Some internet research activities are based not only on the present data, but also on historical data that was, but no longer is posted online. As of late 2023, Internet Archive has over 842 billion web page snapshots stored and available for retrieval. We will go through a simple example of how scraping pre-crawled pages from Wayback Machine can be used to gather historical data for data science…
Apache httpd A little known fact is that macOS ships with Apache httpd. You can launch it by running sudo apachectl start . Configuration files are available in /etc/apache2 directory and default DocumentRoot is /Library/WebServer/Documents. Note however that macOS no longer ships PHP/Ruby/Perl/etc. so you will need to install a scripting language separately to do any kind of dynamic web…
Google Maps Platform contains large amount of POI (Point Of Interest) data that one might want to scrape for purposes such as lead generation, real estate OSINT, academic research and so on. However Google Maps web/mobile apps are highly complex pieces of software and may prove difficult to reverse engineer for data extraction. Once could be doing some browser automation for scraping, but the very…
Elasticsearch is open source server software that acts both as database and search engine, based on Apache Lucene library. It can be used to build distributed clusters for indexing and searching Enterprise-level amounts of data. Some sites that present large, searchable, mostly-textual datasets to the end user are based on elasticsearch as their backend data store and have frontend code talking…
mitmproxy is open source, interactive proxy server meant for interception, capture and analysis of network traffic for purposes such as penetration testing, debugging, troubleshooting, reverse engineering and privacy analysis. A common use case of this tool is to explore mobile application communications (typically API calls over HTTPS) for API scraping, automation and security testing. Earlier in…
Beautiful Soup (also known as bs4 ) is a Python module for parsing HTML (and also XML) documents. It is commonly used in web scraper development. Beautiful Soup takes an HTML string and parses it into a tree structure that reflects the structure of DOM tree, but has properties and methods related to accessing data and running queries. We can install it via PIP as beautifulsoup4 . Furthermore, some…
Many antibot solutions involve client-side JavaScript challenges - performing randomised computation, probing browser environment and so on. That may pose an obstacle for web scraper and automation bot development. In some cases this involves heavily obfuscated or even virtualised code that is hard to reverse engineer. Ideally we want to fully understand and reproduce the challenge in our code,…
GOAT is an online platform for retail sale and reselling of certain consumer products - primarily sneakers, apparel and electronics. It is estimated to have about 50 million active users that include some one million sellers. When there’s a major userbase, there’s a potential to extract monetary benefit by scraping the data and/or running some automations. But 1661 Inc., a company that…
Yelp is a major yellow pages portal in USA and some other countries. It provides large amount of data on various businesses - phone numbers, addresses, descriptions, reviews, working hours and so on. Yelp provides a public API , but it is rate-limited to 500 requests per 24 hours. Thus we will be doing some web scraping to extract data from Yelp web pages. Let us do some planning and strategy work…
With desktop GUI apps becoming generally becoming increasingly bloated and sluggish over time (largely thanks to growing popularity of Electron.js) some technical users are turning back to sofware using the simpler and leaner user interfaces - command line and pseudo-GUIs being rendered in ASCII/Unicode (text user interfaces). CLI tools have the additional benefit of being relatively easy to be…
Restringer is a modular Javascript deobfuscation tool that attempts to autodetect and undo some common JS obfuscation techniques. This tool was developed by PerimeterX for malware analysis purposes. Besides CLI tool and a NPM module (for custom deobfuscator development), Restringer is also available as web app . We will be taking a look into the inner workings of this tool for educational…
Scrapy framework provides a great deal of machinery for developing and operating web scrapers that is based on launching requests and parsing responses. However, sometimes it is desirable to introduce browser automation into a web scraping project. One may want to have code as general as possible across many target sites, address certain kinds of blocking (e.g. Javascript challenges) or simply…
Sometimes programmers really don’t want their code to tampered with during runtime. Code obfuscation makes it harder to read and understand, but by itself does not do anything against modification and dynamic analysis with a debugger. This is particularly applicable to client-side JavaScript code that is meant to be running in web browser environment. After all, the code is largely under…
World Wide Web is a network of HTML documents (pages) with hyperlinks between them. Consider a directed graph that consists of vertices representing pages and edges representing links between pages. For a given page, links from other pages to that page are known as backlinks. Backlinks are of significance to search engine ranking of the site, thus making them of importance to SEO people, digital…
Previously on Trickster Dev: Part 1 Part 2 By using AST transforms developed so far, JSFuck output for character @ can be simplified to: [][ "flat" ][ "constructor" ]( "return\"" + ([][ "flat" ][ "constructor" ]( "return/false/" )()[ "constructor" ]( "/" ) + [])[ 1 ] + [ 1 ] + [ 0 ] + [ 0 ] + "\"" )(); What about other characters that have null value in the MAPPING object in jsfuck.js? For H we…
Artificial intelligence, especially large language models such as ChatGPT are all the rage now. ChatGPT has the distinction of being the fastest growing product in the history of technology - it reached the first one million users in just a few days. As such, it is the talk of the global village now. There is no shortage of overly excited people posting their takes on how it will change the world…
In the world of desktop software, the concept of packer is not new. A packer is a tool that takes a binary executable file as input, applies transformations (e.g. compression, encryption, introducing anti-debugging tricks) and outputs a new, modified executable file that is different at binary level, but retains the functionality of original program. Some packers, such as UPX are only meant to…
Suppose you are looking to collect pricing data on male footwear from the official website of one of the industry leaders - Nike.com. There’s a product list page that seems like a good place to start, but the infinite scroll feature might seem puzzling to budding web scraper developers. In this post, we will go through developing a Scrapy project for scraping sneaker price data from this…
Not every website wants to let the data to be scraped and not every app wants to allow automation of user activity. If you work in scraping and automation at any capacity you certainly have dealt with sites that work just fine when accessed through normal browser throwing captchas or error pages at your bot. There are multiple security mechanisms that can cause this to happen. Today we will do a…
The front page of Scrapy project provides a basic example of Scrapy spider: class BlogSpider (scrapy . Spider): name = 'blogspider' start_urls = [ 'https://www.zyte.com/blog/' ] def parse (self, response): for title in response . css( '.oxy-post-title' ): yield { 'title' : title . css( '::text' ) . get()} for next_page in response . css( 'a.next' ): yield response . follow(next_page, self . parse)…
Bash is a Linux/UNIX program that reads users commands from the users, parses them and executes the appropriate programs through OS-specific APIs. Since it covers these APIs and provides some extra features on top of them this kind of program is called a shell. Bash is not merely an interface between keyboard and exec(2) et. al. It is also a scripting language and interpreter. Today we are going…
In the previous post we went through using Babel AST transforms to simplify JSFuck-generated unary and binary expressions. This was shown to undo a lot, but not all of the obfuscation. What we still have to do is to deal with certain API and runtime hacks that JSFuck leverages to obfuscate some of the characters that are not covered just by abusing type coercion and atomic components of JS (string…
Introduction JSFuck is a prominent JavaScript obfuscation tool that converts JS code into a rather weird looking form with only six characters: ()+[]! . For example, console.log(1) is turned into this: ([][( ! [] + [])[ + []] + ( ! [] + [])[ !+ [] +!+ []] + ( ! [] + [])[ +!+ []] + ( !! [] + [])[ + []]] + [])[ !+ [] +!+ [] +!+ []] + ( !! [] + [][( ! [] + [])[ + []] + ( ! [] + [])[ !+ [] +!+ []] + (…
Incoming HTTPS traffic can be fingerprinted by server-side systems to derive technical characteristics of client side systems. One way to do this is TLS fingerprinting that we have covered before on this blog and that is commonly done by antibot vendors as part of automation countermeasures suite. But that’s not all they do. Fingerprinting can be done at HTTP/2 level as well. Let us discuss…
In computer science, control flow graph is a graph that represents order of code block execution and transitions between blocks. For the purposes of todays topic, vertices in such graph are basic blocks of code that are to be executed consequentially and have no branching logic. Each basic block has an entry point and exit point. Edges in control graph represent transitions between basic blocks.…
String concealing is a code obfuscation technique that involves some sort of string constant recomputation (e.g. Base64 encoding or encryption with symmetric ciphers) being introduced into code. Furthermore, obfuscation solutions may introduce some variable and function indirection to further thwart reverse engineering. In this post we will be learning how to deal with both of these obstacles on…
Introducing constant recomputation is a commonly used code obfuscation technique. Let’s consider the following JavaScript snippet: const fourtyTwo = 42 ; const msg = "The answer is:" ; console . log ( msg , fourtyTwo ); We have one numeric constant ( fourtyTwo ) and one string constant ( msg ) that are passed into console.log() . Let us apply constant obfuscation by using obfuscator.io with…
In previous posts about using Babel for JavaScript deobfuscation, we have used NodePath.replaceWith() method to replace one node with another and NodePath.remove() to remove a single node. Since AST and it’s elements are mutable, we can also modify the AST without traversing it. But there is more to learn about AST modification than what we have seen before. We will go through some more AST…
In computer programming, a lexical scope of an identifier (name of function, class, variable, constant, etc.) is area within the code where that identifier can be used. Some identifiers have global scope meaning they can be used in the entire program. Others have narrowed-down scope that is limited to one single function or code block between curly braces. Identifier names can be reused between…
When doing Javascript deobfuscation work at AST level one will need to create new parts of AST for replacing the existing parts of AST being processed. We will go through 3 ways of doing that. For the sake of an example, let us try to build an AST for the following JS snippet: debugger ; console . log ( "!" ) debugger ; Babel parses this into two DebuggerStatement nodes and and one…
In previous posts we have went through several JavaScript obfuscation techniques and how they could be reversed by applying Abstract Syntax Tree transformations. AST manipulation is a powerful skill that is of particular importance in certain kinds of grayhat programming projects. Earlier, we have focused on how exactly AST could be changed to undo specific kinds of obfuscations. This time,…
On the front page of Scrapy framework there’s the following Python snippet: import scrapy class BlogSpider (scrapy . Spider): name = 'blogspider' start_urls = [ 'https://www.zyte.com/blog/' ] def parse (self, response): for title in response . css( '.oxy-post-title' ): yield { 'title' : title . css( '::text' ) . get()} for next_page in response . css( 'a.next' ): yield response .…
Yahoo! Finance is a prominent website featuring financial news, press releases, financial reports and quantitive data on various assets. We are going to go through some examples on how data could be scraped from this portal. We are going to scrape two kinds of data: fundamental infomation on how well various public companies are performing financially and stock price time series. But first we need…
Introduction I don’t suppose I need to do much explaining on the value of touch typing for developer productivity. Today developers gain additional productivity by using Integrated Development Environments such as PyCharm, Xcode, Visual Studio that provide features like auto-completion, enhanced code navigation, integration with compilers, debuggers and static analysers. All of that does not…
The default way of using Scrapy entails running scrapy startproject to generate a bunch of starter code across multiple files. For small scale scrapers that’s a bit of overkill. It’s far simpler to have a single Python script file that you can run when you want to scrape some data. The CrawlerProcess class in Scrapy framework enables us to develop such a script. For the sake of…
Many developers would agree that source code version control is a great thing that they would not imagine the modern software development without. What if it was applied for structured data as well? Yes, technically it is possible to save SQLite file or SQL dump into git repo, but that is rather clunky and outside the intended use case of git. For proper version control, we would want row-level…
On it’s own SMTP protocol does not do much validation on the authenticity of sender. One can spoof protocol headers at SMTP and MIME levels to send a message in another users name. That can be problematic as it makes spam and social engineering attacks easier. To address this problem, some email authentication technologies have been developed. We will discuss the three major ones: SPF, DKIM…
Introduction and big picture Email is a fairly old school technology meant to replicate postal service digitally. To understand how email works, we must know about network protocols that specify sets of data formats and data exchange rules for exchanging messages over the network. In the modern internet email sending part is conceptually (and sometimes technically) separate from receiving part.…
So, what is wget? Wget is a prominent command line tool to download stuff from the web. It has a fairly extensive feature set that includes recursive downloading, proxy support, site mirroring, cookie handling and so on. Let us go through some use cases of wget. To download a single file/page using wget, just pass the corresponding URL into argv[1] : $ wget http://www.textfiles.com/100/crossbow…
When doing recon part of penetration testing or bug bounty hunting engagements one may want to run various tools (such as port scanners, crawlers, vulnerability scanners, headless browsers and so on) in a VPS environment that will no longer be needed when the task at hand is complete. For large-scale scanning it is highly desirable to spread the workload across many servers. To address these need,…