AVENTER UG (haftungsbeschränkt)

Update: Mesos Executer für Apache Airflow

08. January 2022

Andreas Peters

Unser Airflow Provider für Mesos ist nun in der Version 0.1.3 erschienen.

Changes v0.1.3

Support python >= 3.6
Add: Suppress mesos framework offers if there are no tasks in the queue
Add: support of custom docker environment (more details in the readme.md)
Change: Mesos task name would be generated by Airflow dag_id, task_id and execution_date
Python Modul

Die Installation erfolgt über:


pip install avmesos-airflow-provider

In der Airflow Konfiguration muss anschließend folgendes hinzugefügt werden:

[core]
executor = avmesos_airflow_provider.executors.mesos_executor.MesosExecutor

[mesos]
master = master.mesos:5050
framework_name = Airflow
checkpoint = True
failover_timeout = 604800
command_shell = True
task_cpu = 1
task_memory = 20000
authenticate = True
default_principal = <MESOS_MASTER_PRINCIPAL>
default_secret = <MESOS_MASTER_SECRET>
docker_image_slave = avhost/docker-airflow:v2.1.2
docker_volume_driver = local
docker_volume_dag_name = airflowdags
docker_volume_dag_container_path = /home/airflow/airflow/dags/
docker_sock = /var/run/docker.sock
docker_volume_logs_name = airflowlogs
docker_volume_logs_container_path = /home/airflow/airflow/logs/
docker_environment = '[{ "name":"<KEY>", "value":"<VALUE>" }, { ... }]'

Apache Aiflow Logo

Mesos Task
Airflow Task scheduled in Mesos

Update: Mesos Executer für Apache Airflow

Changes v0.1.3

Bereiche

Weitere Dokumente

Kontakt

AVENTER UG (haftungsbeschränkt)