DuckDB for Geospatial
Using DuckDB for geospatial workflows for incredible performance boost! Leveling up my GeoPandas workflows for processing the 100M Foursquare dataset. Image Source: Foursquare
Geography & Coding
Recent content on Geography & Coding
Latest posts
Hetzner ARM Update Fix
Err:6 https://mirror.hetzner.com/ubuntu/packages jammy-updates/main arm64 Packages 404 Not Found [IP: 2a01:4ff:ff00::3:3 443] apt update & apt upgrade not working anymore on a Hetzer ARM VM? Here’s the solution.
English-only embedding models for multilingual docs
Text Embeddings: Speaking languages without learning them? tl;dr : some *-en models perform well on other languages too.
A performant embedding processing pipeline & tutorial for big XML/HTML data dumps
Let’s process 1.369.841 single English XML/HTML files and index them with BAAI/bge-base-en-v1.5 embeddings for semantic search! Sounds fun? Let’s go!
Semantic Search Tutorial
A full semantic search tutorial about: data mining with requests and beautifulsoup preprocessing in pandas chunking the document text in smaller paragraphs of the right size for the ML model creating embeddings for each chunk calculating the mean embedding for each document saving data as gzipped json (small file size & easy and fast to read in js with pako.js) creating a static web app based on…
Zulip Server Setup Tutorial: zulip.gis.chat
Let’s set up a public Zulip instance for our geospatial community! It’s live: zulip.gis.chat
SemanticFinder - frontend-only live semantic search with transformers.js
Semantic search right in your browser! Calculates the embeddings and cosine similarity client-side without server-side inferencing, using transformers.js and a quantized version of sentence-transformers/all-MiniLM-L6-v2 .
Semantic Search with Qdrant, Hugging Face, SentenceTransformers and transformers.js
Create a fully working semantic search stack with only Qdrant as vector database with built-in API and transformers.js using any huggingface model as your frontend-only embedding generator. No additional inference server needed! Image courtesy Qdrant & Hugging Face .
Geospatial Vector Search with Qdrant
Using qdrant for querying text data with vector search and geospatial filters without GPU (CPU only) Image courtesy Qdrant transformed with Stable Diffusion v2 by stability-ai .
Pyscript File Handling
[Work in progress] Useful PyScript file handling examples for all combinations: local/remote files and import/export.
Connect Pandas to a remote PostgreSQL database via SSH
Use Pandas to execute SQL statements in a remote database such as PostgreSQL via SSH
PyScript-fueled Jupyter Notebooks for your browser
A simple PyScript-fueled Jupyter Notebook running entirely in your browser - no server needed!
PyScript - A Game Changer For Your Browser
Execute Python right in your Browser! 1 2 3 4 5 6 7 8 9 10 11 12 13 14 <!--Load PyScript, Pyodide, Micropip and Python packages--> < script defer src = 'https://pyscript.net/alpha/pyscript.js' ></ script > < py-env > - pandas </ py-env > <!-- Write your Python code! --> < py-script > import pandas as pd from pyodide.http import open_url df = pd.read_csv(open_url('/downloads/iris.csv')) df </…
HyperLogLog Simply Explained
Floral foam 🌼, elephants 🐘 and evil mayors 😈: the useful HyperLogLog algorithm explained as if you were 10 years old!
Life Hacking 2.0: Adding Geographical Filters to Flat Search Websites with QGIS and Python
Create sophisticated walking isochrones for Milan’s metro stops and search for flats with immobiliare! Reproducible for all of Italy!
HDMI Remote Screen Control
Control all functions of your external monitor with a click on your (wireless) keyboard! Lower brightness, switch input, raise contrast or switch power modes with Python and Monitorcontrol. 1 2 3 4 from monitorcontrol import get_monitors for monitor in get_monitors (): with monitor : monitor . set_luminance ( 30 )
City Skylines
Create a free skyline for any city on earth!
Basic River Animations with Blender - Mosel Edition
To view this video please enable JavaScript, and consider upgrading to a web browser that supports HTML5 video Basic River Animations with QGIS, Aerialod and Blender - Mosel from Trier to Traben-Trarbach
How to create Paper (Mario) Maps
Pretty Paper (Mario) Maps quickly done with QGIS and Aerialod.
Guide to 3D OSM Building Models - Milan Edition
OSM data offers a vast amount of information about our cities. In this guide I focus on working and styling OSM building data for 3D-modeling in Aerialod for close-to-reality models. Tl;dr: Use OSM building levels, emulate roofs with shapeburst effect and use Sketchfab for interactive 3D models
Guide to 3D OSM City Models - Trier Edition
There are few cities providing open source LIDAR data for 3D-modeling. However, leveraging OSM building, road and landuse data gives you neat city models in no time!
Raster-Based 3D Models with QGIS, OSM and Aerialod
Creating 3D models from raster or vector data has never been that easy! This tutorial takes you through the whole process from vector/raster data to 3D in just 3 steps.
Tips for neat splash pages
SSH Port Forwarding Guide
Setting up proper port forwarding for SSH connections, servers or just test purposes is crucial but isn’t as straight forward as your connection should be. This post aims to be a quick guide for different use cases.
Convert Python Scripts to Windows Executables (.py to .exe) - Pyinstaller and Pandas | 2022 WORKING VERSION
Python and pandas are a great team for data science. But what if you need to deliver a script to a Windows client who does not have Python installed? This is a quick tutorial for converting Python scripts ( .py ) to standalone, one-file Windows executables ( .exe ). 1 pyinstaller -F my_script.py Get a 20Mb exe file that is fully functional on every Windows machine!
An Introduction to HyperLogLog
Have you ever wondered how HyperLogLog works? Or have you never heard of it at all? In this post I explain the wonderful algorithm of Flajolet et al. from scratch and in a very simple manner.
Life Hacking: Adding Geographical Filters to Flat Search Websites with Turf.js
Flat hunting can be a big pain - particularly for European metropoles. Most flat search platforms already offer good filter options but until now, I didn’t see any with custom geographic filters. Apart from drawing your own area of interest on a map, you seldomly find filters such as “How far is the next metro stop?” or “How close are important facilities?”. This…
Using HyperLogLog with Leaflet-Hexbins
Leaflet-Hexbins is a powerful tool to aggregate data and visualize it quickly on a map. It can size the hexbin radius based on a feature values. But what if we used HyperLogLog data to preserve user-privacy? This post will provide a convenient way to quickly perform on-the-fly HyperLogLog unions for leaflet-hexbins. No more backend-constraint for HyperLogLog There are many HyperLogLog…
HyperLogLog Javascript (js-hll) & Postgres (postgres-hll): making them talk to each other!
1 2 3 4 5 6 7 8 9 10 11 hex_array = […
Querying Postgres with Python Fastapi Backend and Leaflet-Geoman Frontend - Applying Geometry Filters
Querying Postgres with Python Fastapi Backend and Leaflet & Geoman Fronted - Applying Geometry Filters How to query a database with a user-defined geometry drawn on a Leaflet frontend? A very common use case for map applications are custom filters. Either thematic, temporal or value-based filters play a significant role but probably in a geographical context the most dominant one might be spatial…
Leaflet Data Visualization Strategies - Markers, Marker Clusters, Heatmap, Hexbins
Leaflet Data Visualization Strategies - Markers, Marker Clusters, Heatmap, Hexbins
Hugo blog routine - write, preview, edit, commit
1 2 3 4 5 6 7 8 ██████╗ ███████╗ ██████╗ ██████╗ ██████╗ ██████╗██╗ ██╗███████╗ ██╔════╝ ██╔════╝██╔═══██╗ ██╔══██╗██╔═══██╗██╔════╝██║ ██╔╝██╔════╝ ██║ ███╗█████╗ ██║ ██║ ██████╔╝██║ ██║██║ █████╔╝ ███████╗ ██║ ██║██╔══╝ ██║ ██║ ██╔══██╗██║ ██║██║ ██╔═██╗ ╚════██║ ╚██████╔╝███████╗╚██████╔╝██╗██║ ██║╚██████╔╝╚██████╗██║ ██╗███████║ ╚═════╝ ╚══════╝ ╚═════╝ ╚═╝╚═╝ ╚═╝ ╚═════╝ ╚═════╝╚═╝…
Run a Background Thread in Python - Torpy Image Download
1 2 3 import threading def start_thread ( func , name = None , args = []): threading . Thread ( target = func , name = name , args = args ) . start () How to start a background thread in Python to download images from an API.
VM Port Forwarding
Port Forwarding in Virtual Box - Postgres example
SSH in Virtual Machine Ubuntu Server 20.04
A quick guide to setup a SSH-conncection to Virtual Machine Ubuntu Server 20.04
Requests in Python with requests package and Torpy
1 2 3 import requests response = requests . get ( 'https://webscraper.io/test-sites/e-commerce/allinone' ) response
Using Jupyter Notebook as GIS
Clipping a geodataframe and displaying it on a basemap in Jupyter.
QGIS - How to use custom SVG-Markers
Use your own SVG as a QGIS marker!
A complete guide to preprocess Instagram post data
Scripts and tricks to preprocess Instagram data mined with Fast-Instagram-Scraper. Get it on a map!
Mining Instagram efficiently with Torpy - Fast Instagram Scraper
A fast and efficient Instagram Scraper based on Torpy. Scrapes posts for multiple hashtags and location ids .
A quick readthedocs Wiki with mkdocs
Setting up a readthedocs wiki like 1,2,3!
Debugging Python
1 2 3 import pdb from pdb import set_trace as bp bp () Debugging Python with pdb 🐞
Writing Your Own Scraper - Simple Instagram Scraper
Photo by NeONBRAND on Unsplash How to write your DYI Instagram Scraper
Deleting Browser Cache with Python Selenium and Chromedriver
Quick and dirty
Mining Instagram location IDs
1 2 3 4 5 6 7 8 9 10 11 { 'Jena' : '673710' , 'Jena-Zentrum' : '1006125655' , 'Plan B Boulderhalle Jena' : '613227777' , 'Kassablanca Jena' : '5752477' , 'Kulturarena Jena' : '574057843' , 'University of Jena' : '251848169' , 'Paradiespark Para - Jena Paradies' : '228419929' , 'Ernst-Abbe-Sportfeld' : '326665551' , ... } How to quickly mine location IDs from Instagram’s location explorer.
Processing data mined from instagram-scraper with Python and Pandas
Photo by Polina Razorilova on Unsplash Making use of Python and Pandas to quickly process mined social media data of instagram-scraper. Get most common hashtags!
Social Media Mining
A personal shortlist of well-working Python-based scrapers in 2020.
Creating a simple Ubuntu Bash shortcut
1 . x A daily routine of cd’ing in a directory and starting a server can easily be automated.
Creating a simple Windows Powershell shortcut
1 .\ g Automating a daily powershell git routine.
Most common Hugo pitfalls and their solutions
#1 Line endings CRLF vs. LF Theres a difference in how line endings are encoded. If you use VS Code and git, take care that line endings are handled the right way.