Apache Airflow and OpenTelemetry

Описание

Presented by Dennis Ferruzzi & Howard Yoo at Airflow Summit 2023. Slides available at https://airflowsummit.org/slides/2023/ab1-1500-OpenTelemetry.pdf

OpenTelemetry is a vendor-neutral open-source (CNCF) observability framework that is supported by many vendors industry-wide. It is used for instrumenting, generation, collection, and exporting of data within systems which then are ingested by analytics tools that can provide tracing, metrics, and logs. It has long been the plan to adopt the OTel standard within Airflow, allowing builders and users to take advantage of valuable data that could help improve the efficiency, cost and performance of their systems.

Airflow Metrics Documentation - OpenTelemetry Setup: https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.htm #setup-opentelemetry
Better Apache Airflow Observability using OpenTelemetry blog post - https://medium.com/apache-airflow/better-apache-airflow-observability-using-opentemeletry-db9234bffeac