Note Summary PyCamp is the kind of community event that recharges your batteries and restores your hope that another way of relating to one another, beyond stressful competition and selfish goals, is not only possible but very real and achievable if we create the proper environment. Here are a few concrete examples and takeaways from PyCamp 2026 that will hopefully inspire you to look for similar…
Note TL;DR When defining a Python-based Kubeflow Pipeline component, you typically list dependencies using the packages_to_install argument in kfp.dsl.component or by baking them into a custom Docker image. I present here a custom_component wrapper that leverages uv to automate this process by inferring dependencies from dependency-group in the pyproject.toml file. See code repo here I presented…
NumFocus organized a hackathon, the NumHack 2024 , and together with Jurij Wollert and Francesco Bruzzesi I participated. We put in a few days on intense work and it paid off: We got 1st place in the “Build” category!🥳 The project we built is a prototype of an app that aims at empowering communities to build better cities. The app facilitates how citizens can report issues to the local…
JSON Lines is a common format encountered in modern data applications, as stated in this documentation : The JSON Lines text format, also called newline-delimited JSON, is a convenient format for storing structured data that may be processed one record at a time. It’s a great format for log files. It’s also a flexible format for passing messages between cooperating processes. For example Google…
Note Summary If parsing and validating the request significantly contributes to the processing time, there might be room to optimize your FastAPI REST API. The key is to use directly Starlette at one or two spots and to leverage some of Pydantic’s magic to accelerate validation. Part of FastAPI ’s merit is how well it leverages other tools to serve its purpose. FastAPI adds the layers it needs on…
Note TL;DR Numpy’s bytes format can be considerably faster than other formats to deserialize. When storing/retrieving vectors arrays just use the methods array.tobytes() and numpy.frombuffer() (instead of, for example, pickle.dumps/loads ). The Situation Let’s say you have a bunch of entities, e.g. product-ids of on online shop, for which you have a vector representation (think for example of a…
This post contains the code behind this video explanation: Code import torch from torch import tensor import numpy as np import torch.nn.functional as F from torch.utils.data import DataLoader, Dataset from torch.nn.utils.rnn import pad_sequence Imagine a supervised learning scenario of a classification task with sequential data as features and a binary target. Let’s simulate a toy dataset and…
seaborn makes our life easy when it comes to slicing and plotting data in Python. That awesome buffet of well balanced aesthetic and practical functionalities of its ergonomic API comes with a few caveats to consider though. Here’s one of them when trying to plot data including PyTorch tensors. Code import matplotlib.pyplot as plt import numpy as np import seaborn as sns from torch import tensor…
The built-in collections module is a handy bag of tools to be aware of. Here we explore collections.ChainMap, an elegant solution to efficiently carry out a look up over more than one dictionary. # Here's the gist of it, watch the video for more details. dict1 = { "a" : 1 , "b" : 2 , "c" : 3 , } dict2 = { "d" : 4 , "f" : 0 , } dict3 = { "g" : 6 , "h" : 7 , "f" : 10 , } dicts = (dict1, dict3,…
The situation This is a question on Stackoverflow: “Why do I get only one parameter from a statsmodels OLS fit?” As of today it has 52K views. So, if you ran into this problem, you’re not alone. OLS refers to Ordinary Least Squares , a method to estimate the parameters of a Linear Regression model. The fix: Instead of doing just import statsmodels.api as sm sm.OLS(y, X) do: X = sm.add_constant(X)…
Merging dictionaries Say we have these two dictionaries that we would like to merge: d1 = { "a" : 1 , "b" : 2 } d2 = { "a" : 2 , "c" : 3 , "d" : 4 } A kind of cannonical way to do it would be this: d3 = {} for d in [d1, d2]: for k, v in d.items(): d3[k] = v d3 {'a': 2, 'b': 2, 'c': 3, 'd': 4} Warning Notice that we are updating the items, so later appeared keys will overwrite the values under…
Let’s look at a common pattern in Python code: numbers = range ( 10 ) list (numbers) [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] def square(x): return x ** 2 results = [] for n in numbers: results.append(square(n)) results [0, 1, 4, 9, 16, 25, 36, 49, 64, 81] That’s fine. But Pythonistas often prefer list comprehensions like this: results = [square(n) for n in numbers] results [0, 1, 4, 9, 16, 25, 36, 49, 64,…
TL;DR: pandas.read_csv considers the word “null” as a NaN, which also means “zero” in German. The arguments na_values and keep_default_na offer a solution. It’s Friday an you set out to build a very sophisticated numbers translator in several languages: import pandas as pd numbers = pd.DataFrame({ "Spanish" : [ "cero" , "uno" , "dos" , "tres" ], "English" : [ "zero" , "one" , "two" , "three" ],…
TL; DR: Only use the form array *= something if you’re 100% sure you are doing the right thing, otherwise, just go for array = array * something . Let’s see why. We define two functions that to the eyes of many (including past me) do just the same. import numpy as np def multiply(array, scalar): array *= scalar # <-- handy short hand, right? ;) return array def multiply2(array, scalar): array =…
It’s not about the projections for the rest of 2020, I promise. Nor 2021. TL;DR: Imagine you are working with high-dimensional data, that is, the position of each data point in that multidimensional space can be represented by a large number of other features/coordinates. For example, you measure a bunch of properties of a product where each item has some values associated, say, size, cost of…
TL; DR: If you need to compute many vector pairwise metrics in batches, try sklearn.metrics.pairwise_distances_chunked The problem I had to compute pairwise cosine distances for a large list of high-dimensional vectors (e.g. word embedding ). After a couple of (very bad) possible solutions I found a reasonable one, of course, standing on the shoulders of giants: the sklearn function…