Posts

What is Ibis and Why Would You Use It? Ibis is a backend agnostic query builder / dataframe API for Python. Think of it as an interface for dynamically generating & executing SQL queries via Python. Ibis provides a unified API for a variety of backends like Snowflake, BigQuery, DuckDB, polars or PySpark. But why would this be useful? I mainly use it for: Generating complex dynamic queries where (Jinja-)templating would becomes too messy Writing SQL generation as self-contained, easily testable Python functions Switching out a production OLAP database (like Snowflake) at test time with a local & version controlled DuckDB instance containing test data If you know SQL you already know Ibis....

The Need for Distributed Locks When you run into situations where you want to prevent two pieces of code ‒ possibly running on differen machines ‒ from running concurrently, you need a distributed lock. An easy solution to implement such a lock is to leverage a cloud storage service like Google Cloud Storage (GCS) or Amazon S3. Here you can find the full code example implementing this kind of lock....

Introduction dbt is a great tool for building & organising your ELT data pipelines. When deploying dbt yourself you can invoke dbt either through dbt core cli or through Python via dbtRunner. I will give you an example template on how to use the latter. You can find the full example in the Full Example section. Note: This example was build on dbt-core==1.8.3. dbtRunner may be subject to breaking changes so there’s no guarantee the provided code works as is with other dbt versions....

A Hackernews Client In an attempt to learn some frontend / mobile development I’ve decided to give Flutter a try. Flutter is cross-platform framework developed by Google which lets us build & deploy for web (iOS & Android), web and desktop from the same code base. I’m an avid Hacker News reader, they provide an API to query stories and comments, so I decided to build a Hackernews client app....

Introduction When you deploy applications you might want to protect them behind a login. If you’re deploying multiple applications it might not be feasible to add authentication for each deployment separately. Here I’ll show how to set up a load balancer with nginx and oauth2proxy using Keycloak to secure any app. Run the example You can find a working example of running a load balancer with authentication in this repository. You can use docker compose to run the example:...