Python, Machine learning, NLP, websec, etc.
Custom memory allocations and arenas are notoriously hard to deal with in Rust due to the ownership model. In a lot of projects, they can significantly improve performance. For example, let's take databases. Sending a query to a database produces a lot of temporary objects, and if the default allocator is used, you will need to allocate and deallocate thousands of small objects for each query,…
The single best signal for identifying a phishing domain is the domain registration date, and yet domain registrars still limit access to such data. For certificates, we have Certificate Transparency Logs where every issued TLS certificate is publicly visible within seconds of issuance. For domains, we used to use WHOIS protocol, which returned plaintext data: It was pretty hard to deal with,…
Recently, I've improved my Python library, hexora . I wrote it to detect malicious Python code using static analysis. In the new v.0.3.0 release, I've added new detections, and we now also use a simple machine learning model to analyze the whole file. The machine learning model uses code structure features, semantic features, and static code analysis to assess the entire Python file. Although the…
I did a lot of data engineering work in my career. When you work a lot with data, you often get quick requests to extract some cold data and process it. Since the data is cold, it usually resides on S3. For example, one of the typical requests in the past was to count unique values from an old MongoDB backup or CSV dump with 100GB of compressed data. I quickly learned that using throwaway Python…
When coming from relational databases, NULLs are the go-to for optional fields. Using them in ClickHouse can lead to unexpected and often unnoticeable performance degradation. This article explain why. PostgreSQL When using null values in PostgreSQL, you rarely notice any difference. In PG, columns are nullable by default and you can index them. Internally, each row in PostgreSQL has a bitmap that…
PyPI is the main repository for Python packages. One thing that I've noticed recently is the number of published packages per week. Let's look at published counts of new package versions per week: There are some dips in the data, but that's because of how the data was collected. We can see a clear increase in the number of published packages, especially in the last few months. Because of AI, the…
There is an ongoing surge of malicious repositories on GitHub, and the sad thing about it is that GitHub seems not to care much. About 10 days ago, I searched for a repo on DuckDuckGo and stumbled upon a fake GitHub repo. It mimics a legitimate repository, but instead of providing usual releases, it only provides malicious Windows binaries. Linux/MacOS binaries are not available, and the…
Fifteen years ago, I wanted to set up a note-taking system. At the time, Evernote was the tool everyone was talking about, so choosing it seemed like the right and easy decision. After storing around 500 notes for eight years, Evernote became a mess to use. It was bloated, heavily monetized, and slow to work with. So I wanted to switch. About that time came Notion. Everyone was talking about it. I…
At my work, we use ClickHouse to process billions of records and hundreds of terabytes of data. ClickHouse is fast, and its speed got me curious to learn some of its internals. Let's look at a few queries: SELECT count ( * ) FROM cluster . feed WHERE state = 'unknown' ┌────── count () ─┐ 1 . │ 129375618342 │ -- 129.38 billion [......]
Oh My Zsh is still getting recommended a lot. The main problem with Oh My Zsh is that it adds a lot of unnecessary bloat that affects shell startup time. Since OMZ is written in shell scripts, every time you open a new terminal tab, it has to interpret all those scripts. Most likely, you don't need OMZ at all. Here are the timings from the default setup with a few plugins (git,…