Airflow + Kubernetes 101^•

Last modified on 01 Oct 2021.

Installation

sudo snap install microk8s --classic

# make an alias
alias k='microk8s.kubectl'

# add to sudo group
sudo usermod -a -G microk8s thi
sudo chown -f -R thi ~/.kube
# log out and log in again

We use k which stands for kubectl or microk8s.kubectl in this notebook!

k8s

Pods

Pod: (full): a group of one or more containers (such as Docker containers), with shared storage/network, and a specification for how to run the containers.

# list of pods
microk8s.kubectl get pods

# enter a pod
microk8s.kubectl exec -it <pod_id> bash

Namespaces

• Detail is here.
• Multiple virtual clusters (namespaces) backed by the same physical cluster.
• Motivation for using namespaces: here => one word: isolation!
• Avoid creating namespace with prefix kube-.

# list of current namespaces
k get namespace

# create
k create namespace <name_space>
# follows DBS label: https://bit.ly/2Cxge0K

# delete
k delete namespaces <name_space>
# This deletes everything under the namespace!

Airflow

Quickstart

Check here.

KubernetesPodOperator

• task_id="abc_xyz" (required): the name of task given in the airflow ui.
• image="localhost:32000/airflow-abc:debug" (required): docker image to use.
• namespace="default" (required, ref):
• in_cluster=True:
• trigger_rule="all_success": ref
• image_pull_policy="Always": if you changes something on the docker image.

If using a dictionary and feed to env_vars (error “TypeError: string indices must be integers”),

ex_var = {'a': 1, 'b': 2}
with DAG(
    # ...
) as dag:
    abc = KubernetesPodOperator(
        # ...
        env_vars = {
            "EX_VAR": str(ex_var)
        }
        # ...
    )

# used in another file
import os
import ast

ex_var = ast.literal_eval(os.environ["EX_VAR"])

References:

• API document: ref.
• KubernetesPodOperator Configuration – Google Cloud.

Branching

Airflow Errors

🔅 ERROR - Exception when attempting to create Namespaced Pod.

• Note that if you don’t use in_cluster=True, then you won’t run into this problem.^[ref]

• There may be something wrong with variables or their type of values. Check again carefully!

References

• Airflow official document.
• kubernetes blog – Airflow on Kubernetes

×Close