Recent content on reorchestrate
This post is the continuation of Your binary is no longer safe: Decompilation about the brute-force reverse engineering of binary (compiled) programs using Large Language Models (LLMs) to automate this two-part problem: decompilation and conversion to a modern programming language. This post covers the second part of the problem: conversion . Claude enters the game … Here is a 1:1…
This post is about the brute-force reverse engineering of binary (compiled) programs using Large Language Models (LLMs) to automate this two-part problem: decompilation and conversion to a modern programming language. This post covers the first part of the problem: decompilation . Update 24 February 2026 Based on feedback I have split this article into two posts: The first (this post) describes…
What is SQLite? In the past few years SQLite (not SQL-light) has had a surge of popularity as people have come to realise its power as an in-process, highly reliable SQL database engine as a backend for server processes rather than its traditional role of client or edge applications. This change in stance for SQLite has happened despite the authors almost actively discouraging its use for this…
JSON Web Tokens ( JWT ) are data structures that allow the holder of the token to assert claims that are able to be cryptographically verified. An example would be that a claim could hold a user claim role like administrator that if proven to be unaltered (using a shared public key) then could be used to provide access to certain administrator functions. Tools like PostgREST , Hasura and Supabase…
Plugins are a useful way to allow advanced users to add new functions your software without having to modify the main program itself. With interpreted languages like JavaScript or Python this can be relatively easy as the runtime itself is able to execute arbitrary instructions without them having to be compiled first. Rust is a compiled lanaguage so does not have a method of executing abritrary…
Debezium is a Database Change-Data-Capture (aka CDC) tool that is able to decode open source and proprietary database logs, normalize them to a standard payload format and push them into a series of Kafka topics. It implements the Confluent Kafka Connect interface so is built to be highly-available, has development supported by RedHat and commercial support from Confluent . If you are even…
Why Blob Storage Is Risky Many people use Amazon S3 or equivalent services to replace their relational database data warehouses as blob storage coupled with technologies like the compressed-columnar parquet file format offers a reasonably performant, massively scalable and cheap alternative. To understand why blob storage can be risky it is important to understand how Apache Spark executes writes…
History of the problem Since Hadoop was released in 2007 users have been struggling to use it to deploy reliable and scalable Extract-Transform-Load (ETL) data pipelines. This was exacerbated in the early days by the ecosystem still being in flux - it felt like every day there was another major Apache Foundation project being announced adding to the Hadoop ecosystem. Now with the convergence of…
One of the famous machine learning challenges is the performing handwritten character recognition (classification) over the MNIST database of handwritten digits . The MNIST dataset has a training set of 60,000 and a test set of 10,000 28x28 pixel images of handwritten digits and an integer value between 0 and 9 containing their true value. The current best scores are available on Wikipedia with…
Whilst playing with the MNIST dataset I found I needed a way of rotating images and so I decided to build an Affine Transform Transformer for Apache Spark ML. I have implemented the basic Affine Transformation operations: rotate , scaleX , scaleY , shearX , shearY , translateX , translateY . Any pixel which exceeds the image dimensions will be discarded.I am sure the code could be improved but…
I wrote this a while ago based on this excellent post and added a few more attributes. Given that Neo4j doesn’t have a datatype to deal with dates it might come in handy for you too. It will generate a calendar between the years specified at the top of the script ( 1970 to 2050 ) and create Day vertexes with attributes of year , month , day , dayName (day of week) and workDay (binary). It…
Update: This code has been approved and should appear in Apache Spark 2.0.0. The Binarizer transformer ( API ) is part of the core Apache Spark ML package. Its job is simple: compare a series of numbers against a threshold value and if the value is greater than the threshold then output 1.0 and if less than (or equal to) the threshold then output 0.0 . For example, if we apply the Binarizer…
As I have been playing with Apache Spark ML and needed a stemming algorithm I decided to have a go and write a custom transformer myself. As of Spark 1.5.2 Stemming has not been introduced ( should be in 1.7.0 ) but I have taken the Porter Stemmer Algorithm implemented in Scala by the ScalaNLP project and wrapped it as a Spark Transformer. Unfortunately, you are going to have to build Spark from…
Continues from Part 1 . 4 Execution 4.1 The Pipeline Now we have all the components of the pipeline ready all that is needed is to load them into the Spark ML Pipeline() . A pipeline helps with the sequencing of stages so that we can automate the pipeline in the image at the top of this post. When Pipeline() is fit() to the training set it will call the fit() method of the two estimator stages (…
The most exciting feature of Apache Spark is it’s ‘generality’ meaning the ability to rapidly take some text data, transform it to a graph structure and perform some network analysis with GraphX take that dataset and apply some machine learning algorithms with SparkML and store it in memory and query it using SparkSQL all within a single program of very little code. In this post…
In my previous post I wrote some code to demonstrate how to go from the raw database extracts provided monthly by WikiPedia through to loading into Apache Spark GraphX and running PageRank. In this post I will discuss my efforts to make that process more efficient which may be relevant to some of you trying to do proof-of-concept activities on less than ideal hardware. My test box (a standalone…
Recently I have spent a lot of time reading and learning about graphs and graph analytics which naturally drew me to Apache Spark GraphX having previously played with Neo4J . The benefits of GraphX are: fully open source scalable using the Apache Spark model written in Scala which I have been meaning to learn already has basic graph algorithms such as PageRank There is a great resource for…