airflow 安装配置celery+rabbitmq celery+redis

airflow 安装配置celery+rabbitmq celery+redis

AirFlow的安装可以参考​

这里介绍的是AirFlow 安装配置celery+rabbitmq   和   celery+redis

一、airflow 安装配置celery+rabbitmq

1、安装celery和rabbitmq组件

sudo pip install apache-airflow[celery]sudo pip install apache-airflow[rabbitmq]

2、系统安装rabbit

安装 erlang

sudo yum install erlang

安装 rabbitmq

sudo wget rpm --import yum install rabbitmq-server-3.6.15-1.el6.noarch.rpm

配置rabbitmq（设置用户名root，密码rabbitmq，创建虚拟主机airflow-rabbitmq）

[bigdata@bigdata airflow]$ suPassword: [root@bigdata airflow]# rabbitmqctl add_user root rabbitmqCreating user "root"[root@bigdata airflow]# rabbitmqctl add_vhost airflow-rabbitmqCreating vhost "airflow-rabbitmq"[root@bigdata airflow]# rabbitmqctl set_user_tags root airflow-rabbitmqSetting tags for user "root" to ['airflow-rabbitmq'][root@bigdata airflow]# rabbitmqctl set_permissions -p airflow-rabbitmq root ".*" ".*" ".*"Setting permissions for user "root" in vhost "airflow-rabbitmq"[root@bigdata airflow]# rabbitmq-plugins enable rabbitmq_managementThe following plugins have been enabled: amqp_client cowlib cowboy rabbitmq_web_dispatch rabbitmq_management_agent rabbitmq_managementApplying plugin configuration to rabbit@bigdata... started 6 plugins.[root@bigdata airflow]#

修改airflow配置

executor = CeleryExecutor

broker_url = amqp://root:rabbitmq@localhost:5672/airflow-rabbitmqresult_backend = amqp://root:rabbitmq@localhost:5672/airflow-rabbitmq

在启动worker 时，提示不能用root用户权限启动。修改方法：    echo  "export C_FORCE_ROOT="True"" >> /etc/profile    source /etc/profile

二、airflow 安装配置celery+redis

安装redis

sudo yum -y install redis

修改配置文件

sudo vim /etc/redis.conf#bind 127.0.0.1 // 注释掉，使redis允许远程访问requirepass 密码 // 修改这行，redis登录密码

开启redis服务

sudo systemctl start redis

python下安装redis库

sudo pip install redis

安装celery

sudo pip install celery

配置airflow.cfg

executor = CeleryExecutorbroker_url = redis://127.0.0.1:6379/0result_backend = redis://127.0.0.1:6379/0

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。