Dagobah 简介
Dagobash 是用 python 编写的一个简单的基于依赖关系的作业调度器。 Dagobah 允许你使用 Cron 语法安排调度工作，你可以非常简单的通过 web 界面点击拖动来定义每个工作的一系列任务的依赖关系。
Dagobash 可以让你重试失败的单个任务，给你发送有用的工作任务成功和失败的报告，并且可以在多种后台持久化你的数据，这样你就不用担心你的数据丢失了。
github 项目地址：Dagobah
安装
我的基础环境：
mac os
Python 2.7
mongo 3.2.3
安装过程
pip install pymongo
dagobahd 安装
pip install dagobah
启动 dagobah
dagobah dagobah 的默认端口是 9000
在启动的时候，其会生成一个配置文件 ~/.dagobahd.yml
# host and port for the Flask app. set host to 0.0.0.0 to listen on all IPs
host: 127.0.0.1
port: 9000
# the app's secret key, used for maintaining user sessions
# WARNING: change this to your own random value!
# an easy way to create a key with python is "import os; os.urandom(24)"
app_secret: 'g\xde\xf5\x06@K\xf5:\x1fmZ\xac\x1fO\xe8\xcd\xde\xcf\x90\xaeY7\x8c\x96'
# credentials for single-user auth
auth_disabled: False
password: dagobah
# choose one of the available backends
# None: Dagobah will not use a backend to permanently store data
# sqlite: use a SQLite database. see the SQLite section in this file
# mongo: store results in MongoDB. see the MongoBackend section in this file
backend: sqlite
# choose one of the available email templates
# None: Dagobah won't send you emails when a job finishes or fails
# text: Simple text format
# basic: Simple, tabular HTML email
email: basic
Logging:
# change to False to disable logging entirely
enabled: True
# specify a full path to the log file
# alternatively, specify "default" to log to a file in the dagobahd directory
logfile: default
# specify the log level to use
# choose one of [debug, info, warning, error, critical]
loglevel: info
Email:
# set host and port of the SMTP server to use to send mail
# e.g. host: smtp.gmail.com port: 587
host: smtp.gmail.com
port: 587
# email server authentication
user: None
password: None
# tls is required for some mail servers, specifically Gmail
use_tls: True
# from address in the emails from Dagobah.
# supports the following special variables within curly brackets {}
# {HOSTNAME}: the machine's hostname
from_address: dagobah@{HOSTNAME}
# list of email addresses to send reports to, e.g. ['myemail@gmail.com']
recipients: []
# sets whether Dagobah sends you emails on successful job completion
send_on_success: True
# sets whether Dagobah sends you emails on job and task failures
send_on_failure: True
SQLiteBackend:
# specify a full path to the sqlite database file
# alternatively, specify "default" to create a database file in the dagobahd directory
filepath: default
MongoBackend:
# connection details to a mongo database
host: localhost
port: 27017
db: dagobah
# names of collections within the db specified above
dagobah_collection: dagobah
job_collection: dagobah_job
log_collection: dagobah_log
# host and port for the Flask app. set host to 0.0.0.0 to listen on all IPs
host:127.0.0.1
port:9000
# the app's secret key, used for maintaining user sessions
# WARNING: change this to your own random value!
# an easy way to create a key with python is "import os; os.urandom(24)"
app_secret:'g\xde\xf5\x06@K\xf5:\x1fmZ\xac\x1fO\xe8\xcd\xde\xcf\x90\xaeY7\x8c\x96'
# credentials for single-user auth
auth_disabled:False
password:dagobah
# choose one of the available backends
# None: Dagobah will not use a backend to permanently store data
# sqlite: use a SQLite database. see the SQLite section in this file
# mongo: store results in MongoDB. see the MongoBackend section in this file
backend:sqlite
# choose one of the available email templates
# None: Dagobah won't send you emails when a job finishes or fails
# text: Simple text format
# basic: Simple, tabular HTML email
email:basic
Logging:
# change to False to disable logging entirely
enabled:True
# specify a full path to the log file
# alternatively, specify "default" to log to a file in the dagobahd directory
logfile:default
# specify the log level to use
# choose one of [debug, info, warning, error, critical]
loglevel:info
Email:
# set host and port of the SMTP server to use to send mail
# e.g. host: smtp.gmail.com port: 587
host:smtp.gmail.com
port:587
# email server authentication
user:None
password:None
# tls is required for some mail servers, specifically Gmail
use_tls:True
# from address in the emails from Dagobah.
# supports the following special variables within curly brackets {}
# {HOSTNAME}: the machine's hostname
from_address:dagobah@{HOSTNAME}
# list of email addresses to send reports to, e.g. ['myemail@gmail.com']
recipients:[]
# sets whether Dagobah sends you emails on successful job completion
send_on_success:True
# sets whether Dagobah sends you emails on job and task failures
send_on_failure:True
SQLiteBackend:
# specify a full path to the sqlite database file
# alternatively, specify "default" to create a database file in the dagobahd directory
filepath:default
MongoBackend:
# connection details to a mongo database
host:localhost
port:27017
db:dagobah
# names of collections within the db specified above
dagobah_collection:dagobah
job_collection:dagobah_job
log_collection:dagobah_log如果不想使用默认的端口和IP的话，可以修改配置文件
访问 Dagobah
以下截图借用官方的图片
登录
密码在配置文件中
添加 job
在 job 下添加任务
指定任务直接的依赖关系
这个点击任务的图片，然后拖动，就可以画出任务直接的关系了
给远程主机添加任务
日志会汇总之后发送邮件给你
源码安装 Dagobah
从 github 上获取源码
git clone https://github.com/thieman/dagobah.git
python setup.py install
1
2
3
gitclonehttps://github.com/thieman/dagobah.git
pythonsetup.pyinstall
为了能配置远程的服务器，你需要配置 SSH config
vim ~/.ssh/config
1
vim~/.ssh/config
加入以下信息
# Contents of ~/.ssh/config
Host test2
HostName 192.168.0.1
User root
IdentityFile ~/.ssh/id_rsa
Host test1
HostName 192.168.0.2
User root
IdentityFile ~/.ssh/id_rsa
1
2
3
4
5
6
7
8
9
# Contents of ~/.ssh/config
Hosttest2
HostName192.168.0.1
Userroot
IdentityFile~/.ssh/id_rsa
Hosttest1
HostName192.168.0.2
Userroot
IdentityFile~/.ssh/id_rsa
注： test2 是主机 192.168.0.1 的别名
配置 ssh 的免密，使用命令 ssh-keygen 和 ssh-copy-id，你懂的，详细过程就不介绍了
启动 dagobah
dagobahd
1
dagobahd
注：dagobah 安装后，其命令路径为 /usr/local/bin/dagobahd
1
2
注：dagobah安装后，其命令路径为/usr/local/bin/dagobahd
缺陷
不能定义 job 和 job 之间的依赖关系
不能定义 task 和 job 之间的依赖关系
task 和 task 依赖的话，如果中间的 task 失败，后续的 task 依然可以继续执行，其设计的原则是一个任务失败，不影响整体任务的执行，但是如果我是强依赖，这样就会有问题
优点
可以shell与python交互，这个组合起来你可以做非常多的功能
轻松定义前后关联，可视化操作，方便简单
基于cron表达式，你可以想到的调度时间都可以轻松实现
