DolphinScheduler task scheduling system v2.0.9

Apache DolphinScheduler (incubator, formerly EasyScheduler) is a big data distributed workflow task scheduling system. It mainly solves the problems of complex dependencies in ETL processing during the big data research and development process, but cannot intuitively monitor the health status of tasks. DolphinScheduler assembles Tasks in a DAG streaming manner and can monitor the running status of tasks in real time. It also supports operations such as retrying, recovering failed tasks from designated nodes, pausing, and killing tasks.

Design features: A distributed and easily scalable visual DAG workflow task scheduling system. Committed to solving the intricate dependencies in the data processing process, so that the scheduling system can be used out of the box in the data processing process. Its main objectives are as follows

1. Use a DAG diagram to associate tasks according to task dependencies, which can visually monitor the running status of tasks in real time.

2. Supports a variety of task types: Shell, MR, Spark, SQL (mysql, postgresql, hive, sparksql), Python, Sub_Process, Procedure, etc.

3. Supports workflow scheduled scheduling, dependency scheduling, manual scheduling, manual pause/stop/resume, and also supports operations such as failed retry/alarm, failed recovery from specified nodes, and Kill tasks.

4. Support workflow priority, task priority, task failover and task timeout alarm/failure

5. Support workflow global parameters and node custom parameter settings

6. Support online upload/download and management of resource files, and support online file creation and editing.

7. Support online viewing and scrolling of task logs, online downloading of logs, etc.

8. Implement cluster HA and achieve decentralization of Master cluster and Worker cluster through Zookeeper

9. Support online viewing of Master/Worker cpu load, memory, and cpu

10. Support workflow running history tree/Gantt chart display, task status statistics, and process status statistics.

11. Support complement number

12. Support multi-tenancy

13. Support internationalization

14. There are more waiting for partners to explore

Expand