Imagine que você tenha uma tabela com as datas de início e fim de uma pessoa em um cargo. Considere que é possível que uma pessoa esteja em mais de um cargo simultaneamente. A tabela poderia ser definida como: user_id cargo_id data_inicio data_fim 1 1 2024-01-01 2024-03-15 1 2 2024-02-10 2024-04-30 2 3 2024-01-01 2024-01-31 2 4 2024-02-01 2024-03-31 3 5 2024-01-01 2024-01-31 3 6 2024-03-01…
Dbt is a great tool to define the data transformations using SQL. In a nutshell, it’s SQL plus Jinja2 templates. It enhances SQL by allowing you to link to dynamic tables, create macros to reuse common parts, add loops, etc. It also allows you to document your tables and add tests using a YAML file, generating a static documentation website describing all tables, their columns, and lineage (i.e.,…
Traduzido de Religion for the non-religious A mente… pode fazer um céu do inferno, ou um inferno do céu. - John Milton A mente é certamente seu próprio cosmos. - Alan Lightman Você vai à escola, estuda muito, obtém um diploma e fica satisfeito consigo mesmo. Mas você se tornou mais sábio? Você consegue um emprego, consegue coisas no trabalho, ganha responsabilidade, é pago mais, muda-se para uma…
To drop user-defined functions (UDF) in Snowflake, we need the function name and list of parameters. Depending on how many UDFs you have, this can become boring quickly. Instead of going one by one, run the SQL command below: SHOW USER FUNCTIONS IN ACCOUNT ; SELECT CONCAT ( 'DROP FUNCTION ' , "catalog_name" , '.' , "schema_name" , '.' , REGEXP_REPLACE ( "arguments" , ' RETURN .*$' , '' ), ';' ) AS…
I’ve been using Ubuntu since its virst version, Ubuntu 4.10. I tried other distributions (Fedora, Debian, Gentoo, etc.), but always ended up back in Ubuntu. It’s stable enough, easy to install, and things just work, so I don’t see a reason to change. All my servers are either Ubuntu or Debian as well, so the knowledge translates well. Anyway, I was still running Ubuntu 16.04, and over the years it…
You can use the \COPY command in Postgres to export the result of a SELECT to a file (CSV, TSV, etc.). However, what if the table is huge? I faced this problem where the resulting CSV had more than 50 GB. Thankfully, the \COPY command also allows passing the result to a regular program, so we’ll use the CLI program split and gzip to not only split the resulting file, but also compress it with…
I’m using django-sass-processor to automatically compile the SASS files in a Django 2.1 project. Everything runs fine locally, but on Heroku the pages appear unstyled. The problem is that Heroku only runs ./manage.py collectstatic when building a Django app. This simply copies the static files to the ./staticfiles folder. As the SCSS files weren’t compiled yet, only the *.scss files end up in that…
The Diário Oficial is a project by the Serenata de Amor Operation to extract the government purchases that had bidding exemption (because they are below a certain value) published in the official gazettes. Their intent is to get this data in a machine readable format, so we can look for suspicious purchases. In this post, we’ll walk through the code to understand how the gazette is parsed and its…
Transcribing a video is a tedious and time-consuming task. It used to be impossible for machines to do, but this isn’t the case anymore, just ask Siri, Alexa, Google Home, or any other of the many voice assitants. The problem becomes how a regular user this technology to convert their own videos? YouTube automatically generates subtitles for videos. What if we could use it to transcribe our own…
I needed to run a Docker container from inside another container. While it’s possible to run Docker inside Docker , the recommended way is running siblings containers. The challenge now is how to create a Docker container in the host machine from inside another Docker container. It’s easy in theory. You just have to map the host’s /var/run/docker.sock to the container’s, like: docker run -v…