Recent content on Data-Dive
I automated Duolingo by reverse engineering the Duolingo Android app. The result is the open-source Duobot. It is a complete and working command line automation for Duolingo written in Python.
In this how-to we look at using Caddy as a reverse proxy for serving a Python Dash Aap via gunicorn.
Project management and packaging in Python has always been messy and opaque. Several tools have tried to improve the situation but most have fallen short. This changes with Rye. A new, one-stop-shop solution to most challenges around Python development.
Reverse engineering the private API of an Android App used to be uncomplicated. One could simply intercept and decrypt the traffic between app and backend server using readily available tools. However, newer Android versions have introduced restrictions that make this more challenging. In addition, some apps are using certificate pinning to avoid having their API exposed. Hence, circumventing…
Marketing Mix Modeling is promoted as the answer to the most important and challenging question in marketing measurement: determining the effect of marketing and marketing channels on sales. However, this claim is misleading and should not be taken at face value. Let’s find out why!
Adding a dark mode to your Hugo blog is a good way to appeal to your readers. Many Hugo themes already implement this functionality but even adding it manually is simple enough. However, if you post code snippets with syntax highlighting your style will likely only match either light or dark mode. Let’s fix this by making the syntax highlighting style adaptive. One theme for light mode and…
We transcribe a live audio-stream in near real time using OpenAI-Whisper in Python. Our goal is to monitor it for keywords. Using fuzzy matching in the transcribed text, we trigger an alarm via Signal messenger on mention of our keywords.
We create a maintainable and reliable process for deploying dbt models to production on AWS. For that, we use a Docker runtime environment that will be run as a task on AWS ECS Fargate and triggered via Airflow. We use CI/CD for automating the deployment and making the life of our dbt users as easy as possible.
We build a model that can be prompted to generate human like positive and negative medical reviews in German. For that, we fine-tune GPT-2 on an unique data set using Tensorflow in Transformers. For fast results, we use TPUs on Google Colab.
Does the physicians rating platform Jameda discriminate against non-paying physicians? In a previous post, we analyzed this claim inconclusively. Using new data, we are able to gain new insights and shed light on the issue.
Using PyInstaller, you can package your Python scripts for common OS types. However, the need to build on each target OS separately can be cumbersome and limiting. Let’s solve this by building in the cloud! We learn to use GitHub Actions for a simple, convenient and free way to automatically build an app in the cloud and release it on GitHub.
Using a unique German data set containing ratings and comments on doctors, we build a Binary Text Classifier. In this third part, we introduce a state of the art model based on the transformer architecture, namely BERT. Using the transformers library for tensorflow, we push our ability to predict the sentiment of comments towards its limit.
Using a unique German data set containing ratings and comments on doctors, we build a Binary Text Classifier. In part 1 we’ve introduced a complete machine learning work flow that predicts ratings from comments. In this second part, we improve on our baseline by implementing a LSTM neural network model and using FastText embeddings. Using Keras for feature creation and prediction, we improve…
Using a unique German data set containing ratings and comments on doctors, we build a Binary Text Classifier. To do so, we implement a complete machine learning work flow that predicts ratings from comments. In this first part, we start with basic methods. We go through text pre processing, feature creation (TF-IDF), classification and model optimization. Finally, we evaluate our model’s…
In this post, I share results from a novel experiment on Tinder. We collect a unique dataset while swiping Tinder using different profile names and with heterosexual and homosexual search preferences. From this, we uncover vastly different patterns in matching and messaging behavior between sexes. More dramatically, we find strong evidence for name based discrimination on Tinder.
When plotting huge data sets using Python while keeping interactivity, Datashader is paramount. In this post, I demonstrate the abilities of this powerful and convenient library. We use an unique dataset containing a whole year of shared bike usage in Cologne to plot over a million locations on a map.
Do you want to build map visualizations in Python? Look no further than GeoViews. It is not only super simple to use but also offers several interactive features that make your visualization stand out. Using geo spatial data from our bike rental data set we explore some of the possibilities.
Tinder is hugely popular and offers fascinating data on online dating. Still, there are barely any analysis of tinder data. In this endeavor we change that, by analyzing an unique dataset of tinder profiles. The data contains thousands of female and male profiles collected by swiping as a hetero- and homosexual male. Using descriptive statistics, visualization and natural language processing we…
Including variables in a JupyterLab Notebook’s Markdown cells seems like a basic thing. Turns out it is not. It’s simply not yet supported out of the box. Here is how to do it anyways. Moreover, learn how to selectively hide code cells when exporting your Notebook. Improve the aesthetics and dynamic capabilities of your Notebook by using this simple approach.
Does the physicians rating platform Jameda discriminate against non-paying users? We investigate whether this claim made in an article by the newspaper ‘Die Zeit’ can be supported. For that, we conduct a thorough exploratory data analysis on an extensive data set of the platform. In the end, we come to a fact based conclusion.
Can’t find one of your android devices? It is on silent? My second Amazon Alexa skill saves you
In this multi part post, I will take a look at the bike rental system in Cologne. Following up on the previous work we start digging into the data using descriptive statistics. We look at some interesting findings and visualize them using matplotlib and seaborn.
Building on a basic choropleth map, we learn how to add additional interaction for the user. We visualize change in data patterns over time by allowing dynamic control over the plotted data. Moreover, we add animation to our visualization.
Using bike data from a previous post, we look at a different method to make a choropleth map. With the help of bokeh it is easy to create nice looking visualizations. They are interactive and can easily be integrated into web applications or websites.
Fahrrad Finder helps German users to quickly learn where the nearest sharing bikes are located.
Amazon has recently added an API for retrieving the location / address of an Alexa device. Learn how to quickly implement it using Python, and create new functionalities for your custom skills.
Alexa is Amazon’s voice guided assistant. It is affordable, has many helpful functions, and can be enhanced with custom skills. Are you missing a feature? Become a developer and implement it yourself! With basic Python knowledge you can quickly learn how to build your own skills.
In this multi part post, I will take a look at the bike rental system in Cologne. In the first part we applied web-scraping and XML parsing to retrieve data and build a dataset of bike locations. Now, we will deal with geospatial tools and map visualization in order to search for interesting patterns in the bike-rental data.
In this multi part post, I will take a look at the bike rental system in Cologne. We’ll be dealing with web-scraping, XML parsing, GeoData and map-visualization in order to search for interesting patterns in bike-rental data. Lets get started with gathering the data!
There are a handful solutions for dealing with unconfirmed bitcoin transactions. However, depending on the case only a few of those might be viable. In this post, I describe how to apply the child pays for parent method in electrum. It is a solution that works most times.
Including images in pelican is simple. Making them look pretty not so much. Here, I describe how to install a lightbox plugin that makes your blog look better.
Hi, I’m Michael. I’ve worked in academia as a research assistant in experimental economics. As a Data Analyst. And as a Data Scientist in marketing analytics. Currently, I’m working as a Data Engineer in a tech company. I like to explore new technologies. Moreover, I like data and coding. Whenever I find some time, I work on projects combining these things. By writing about them…