IT博客汇 | docker-compose安装airflow 2.0纪要

docker-compose安装airflow 2.0纪要

Kagura发表于 2021-01-06 06:11:30

本文所涉及完整配置文件:
https://github.com/KingFalse/veronica/blob/main/airflow2.0/docker-compose.yml

2020年12月17日airflow2.0发布,一直没有时间去更新,今天记录一下安装流程.

首先是docker-compose.yml文件,与1.x版本没多大变化,我遇到的主要是两个命令有变化:airflow upgradedb -> airflow db upgrade 跟 airflow create_user -> airflow users create

version: "3.7"
# 完整文件: https://github.com/KingFalse/veronica/blob/main/airflow2.0/docker-compose.yml
# 启动一个airflow实例并创建用户:admin 密码:admin123
x-airflow-env: 
  &airflow-env
  AIRFLOW__CORE__DEFAULT_TIMEZONE: system #解决UTC时区问题
  AIRFLOW__CORE__EXECUTOR: LocalExecutor #加载示例任务
  AIRFLOW__CORE__LOAD_EXAMPLES: 'True'
  AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS: 'False'
  AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgres+psycopg2://airflow:www.kagura.me@postgres:5432/airflow
  AIRFLOW__CORE__FERNET_KEY: 81HqDtbqAywKSOumSha3BhWNOdQ26slT6K0YaZeZyPs= 
  AIRFLOW_CONN_METADATA_DB: postgres+psycopg2://airflow:www.kagura.me@postgres:5432/airflow
  AIRFLOW_VAR__METADATA_DB_SCHEMA: airflow
  AIRFLOW__SCHEDULER__SCHEDULER_HEARTBEAT_SEC: 10
  AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 60 # dags目录自动刷新时间，60秒
  AIRFLOW__WEBSERVER__DEFAULT_UI_TIMEZONE: system #解决UTC时区问题
  AIRFLOW__WEBSERVER__RBAC: 'True'
  AIRFLOW__WEBSERVER__AUTHENTICATE: 'True'
  AIRFLOW__WEBSERVER__AUTH_BACKEND: airflow.contrib.auth.backends.password_auth #启用密码认证登陆
  AIRFLOW__WEBSERVER__EXPOSE_CONFIG: 'True'

x-airflow-vol: 
  &airflow-volumes
    - /etc/localtime:/etc/localtime
    - ./dags:/opt/airflow/dags
services:
  postgres:
    image: postgres:12
    container_name: airflow_postgres_1
    restart: always
    environment:
      - POSTGRES_USER=airflow
      - POSTGRES_PASSWORD=www.kagura.me
      - POSTGRES_DB=airflow
    volumes:
      - ./pgdata:/var/lib/postgresql/data/
    ports:
      - "5433:5432"

  scheduler:
    image: apache/airflow
    container_name: airflow_scheduler_1
    restart: always
    depends_on:
      - postgres
      - webserver
    environment:
      <<: *airflow-env
    ports:
      - "8793:8793"
    volumes: 
      *airflow-volumes
    command: scheduler
    healthcheck:
      test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
      interval: 30s
      timeout: 30s
      retries: 3

  webserver:
    image: apache/airflow
    container_name: airflow_webserver_1
    hostname: webserver
    restart: always
    depends_on:
      - postgres
    environment:
      <<: *airflow-env
    volumes: 
      *airflow-volumes
    ports:
      - "8080:8080"
    entrypoint: bash -c "
        airflow db upgrade
        && airflow users create -r Admin -u admin -e admin@example.com -f False -l King -p admin123
        && pip3 config set global.index-url https://pypi.mirrors.ustc.edu.cn/simple/
        && airflow webserver
      "
    healthcheck:
      test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
      interval: 30s
      timeout: 30s
      retries: 32

现在你已经安装好了一个开箱可用的airflow.打开浏览器:http://127.0.0.1:8080/ 应该能看见airflow登陆页面. 用户名:admin 密码:admin123 ,直接登陆即可使用

设置容器pip镜像

docker exec -u root airflow_webserver_1 pip3 config set global.index-url https://pypi.mirrors.ustc.edu.cn/simple/
docker exec -u root airflow_scheduler_1 pip3 config set global.index-url https://pypi.mirrors.ustc.edu.cn/simple/

安装其他的pip包

docker exec -u root airflow_webserver_1 pip3 install pymysql
docker exec -u root airflow_scheduler_1 pip3 install pymysql

设置容器apt镜像

docker exec -u root airflow_webserver_1 sed -i 's/deb.debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list

给容器安装apt包

docker exec -u root airflow_webserver_1 apt install -y git

安装airflow插件(在线代码编辑器)

docker exec -u root airflow_webserver_1 pip3 install airflow-code-editor