美文网首页
0. airflow的安装

0. airflow的安装

作者: 路小漫 | 来源:发表于2018-07-02 17:28 被阅读0次

1. 安装

airflow的安装非常简单,安装官方文档可以轻松完成,我是在macOS上安装的,Linux下应该是一样的,Windows就不清楚了。

# airflow needs a home, ~/airflow is the default,
# but you can lay foundation somewhere else if you prefer
# (optional)
export AIRFLOW_HOME=~/airflow

# install from pypi using pip
pip install apache-airflow

# initialize the database
airflow initdb

# start the web server, default port is 8080
airflow webserver -p 8080
  • airflow 需要设置一个环境变量,默认路径是~/airflow,当然你也可以选择其他路径
  • 使用pip安装
  • 初始化数据库
  • 启动web服务,默认端口号是8080

2.配置

运行完上面的安装命令后,airflow会默认创建环境变量$AIRFLOW_HOME对应的文件目录,并创建文件“airflow.cfg”。你可以打开$AIRFLOW_HOME/airflow.cfg或者在web端通过菜单Admin->Configuration查看文件的内容。PID文件在$AIRFLOW_HOME/airflow-webserver.pid目录下或者如果被系统共享的话在/run/airflow/webserver.pid目录下。(我的是在$AIRFLOW_HOME/airflow-webserver.pid这里)

airflow使用sqlite数据库,不需要你进行后端操作,可以快速的应用airflow。它与SequentialExecutor一起工作,它会按顺序运行任务实例。虽然功能有限,但是利用它可以快速的开始体验airflow的功能,并且可以访问UI和命令行工具。

下面是几个命令,会触发几个任务实例。运行下面的命令,能在DAG中看到job的运行状态。

# run your first task instance
airflow run example_bash_operator runme_0 2015-01-01
# run a backfill over 2 days
airflow backfill example_bash_operator -s 2015-01-01 -e 2015-01-02

3.扩展包

airflow有很多扩展包,如下图:

subpackage install command enables
all pip install apache-airflow[all] All Airflow features known to man
all_dbs pip install apache-airflow[all_dbs] All databases integrations
async pip install apache-airflow[async] Async worker classes for gunicorn
devel pip install apache-airflow[devel] Minimum dev tools requirements
devel_hadoop pip install apache-airflow[devel_hadoop] Airflow + dependencies on the Hadoop stack
celery pip install apache-airflow[celery] CeleryExecutor
crypto pip install apache-airflow[crypto] Encrypt connection passwords in metadata db
druid pip install apache-airflow[druid] Druid.io related operators & hooks
gcp_api pip install apache-airflow[gcp_api] Google Cloud Platform hooks and operators (using google-api-python-client)
jdbc pip install apache-airflow[jdbc] JDBC hooks and operators
hdfs pip install apache-airflow[hdfs] HDFS hooks and operators
hive pip install apache-airflow[hive] All Hive related operators
kerberos pip install apache-airflow[kerberos] kerberos integration for kerberized hadoop
ldap pip install apache-airflow[ldap] ldap authentication for users
mssql pip install apache-airflow[mssql] Microsoft SQL operators and hook, support as an Airflow backend
mysql pip install apache-airflow[mysql] MySQL operators and hook, support as an Airflow backend
password pip install apache-airflow[password] Password Authentication for users
postgres pip install apache-airflow[postgres] Postgres operators and hook, support as an Airflow backend
qds pip install apache-airflow[qds] Enable QDS (qubole data services) support
rabbitmq pip install apache-airflow[rabbitmq] Rabbitmq support as a Celery backend
s3 pip install apache-airflow[s3] S3KeySensor, S3PrefixSensor
samba pip install apache-airflow[samba] Hive2SambaOperator
slack pip install apache-airflow[slack] SlackAPIPostOperator
vertica pip install apache-airflow[vertica] Vertica hook support as an Airflow backend
cloudant pip install apache-airflow[cloudant] Cloudant hook
redis pip install apache-airflow[redis] Redis hooks and sensors

相关文章

  • 0. airflow的安装

    1. 安装 airflow的安装非常简单,安装官方文档可以轻松完成,我是在macOS上安装的,Linux下应该是一...

  • Ansible Role 大数据 之【airflow】

    Ansible Role: airflow 安装airflow 介绍 Airflow 被 Airbnb 内部用来创...

  • 005-Airflow安装

    环境准备-Airflow 1.1 Airflow安装 1.2 Airflow配置 在 /etc/profile 文...

  • airflow1.8+mysql5.6部署

    1.airflow安装 pip install airflowpip install airflow[celery...

  • airflow

    下载安装airflow 使用pip 安装 pip install apache-airflow 将默认的数据库sq...

  • airflow 升级1.10

    1.查看 airflow 配置 2.启动虚拟环境 3.找出 airflow 安装包的位置 4.指定 airflow...

  • Airflow 中文文档:安装

    获得气流 安装最新稳定版Airflow的最简单方法是使用pip : 您还可以安装Airflow,支持s3或post...

  • Airflow 安装

    前段时间部门要求研究airflow,大概记录一下安装的步骤。airflow是一个python实现的工作流管理平台,...

  • Airflow 安装

    平台:ubuntu apt-get install libmysqlclient-devapt-get insta...

  • 安装Airflow

    上一篇:etl bigdata[https://www.jianshu.com/p/4538e8c3b898]更多...

网友评论

      本文标题:0. airflow的安装

      本文链接:https://www.haomeiwen.com/subject/bnjluftx.html