light task scheduler
1.7.0 release
LTS (light-task-scheduler) is mainly used to solve distributed task scheduling problems and supports real-time tasks, scheduled tasks and Cron tasks. It has good scalability, scalability, robustness and stability and is used by many companies. We also hope that open source enthusiasts can contribute together.
Github address: https://github.com/ltsopensource/light-task-scheduler
oschina address: http://git.oschina.net/hugui/light-task-scheduler
Examples: https://github.com/ltsopensource/lts-examples
Document address (under update, this will prevail later): https://www.gitbook.com/book/qq254963746/lts/details
Both addresses will be updated simultaneously. If you are interested, please join the QQ group: 109500214 (group password: hello world) to discuss and improve together. The more people support it, the more motivation there will be to update it. If you like, remember to star in the upper right corner.
LTS mainly has the following four types of nodes:
The JobClient, JobTracker, and TaskTracker nodes are all无状态. Multiple ones can be deployed and dynamically deleted to achieve load balancing and larger load capacity, and the framework uses the FailStore strategy to make LTS have good fault tolerance.
The LTS registration center provides multiple implementations (Zookeeper, redis, etc.). The registration center exposes node information and elects masters. (Mongo or Mysql) stores task queues and task execution logs, netty or mina does underlying communication, and provides multiple serialization methods such as fastjson, hessian2, java, etc.
LTS supports task types:
Supports dynamic modification of task parameters, task execution time and other settings, supports dynamic addition of tasks in the background, supports Cron task suspension, supports manual stopping of executing tasks (conditional), supports task monitoring statistics, supports task execution monitoring of each node, JVM Monitoring, etc.
The figure below is a standard real-time task execution process.
Currently, the backend has a simple authentication function provided by ztajy. The username and password are in auth.cfg and can be modified by the user.
LTS can do without the Spring framework at all, but considering that many user projects use the Spring framework, LTS also provides support for Spring, including Xml and annotations, just introduce lts-spring.jar .
A business logger is provided on the TaskTracker side for use by applications. Through this business logger, business logs can be submitted to JobTracker. These business logs can be concatenated by task IDs, and the execution of tasks can be viewed in real time in LTS-Admin. schedule.
SPI expansion can achieve zero intrusion. It only needs to implement the corresponding interface and it can be used by LTS. The currently open expansion interfaces are
When the TaskTracker that is executing the task goes down, the JobTracker will immediately allocate all the tasks assigned to the downed TaskTracker to other normal TaskTracker nodes for execution.
Resource monitoring, task monitoring, etc. can be performed on JobTracker and TaskTracker nodes, and can be viewed in the LTS-Admin management background in real time to perform reasonable resource allocation.
The LTS framework provides support for four execution results, EXECUTE_SUCCESS , EXECUTE_FAILED , EXECUTE_LATER , and EXECUTE_EXCEPTION , and adopts corresponding processing mechanisms for each result, such as retrying.
The FailStore mechanism is used for node fault tolerance. Fail And Store will not affect the operation of the current application due to the instability of remote communication. For specific FailStore instructions, please refer to the FailStore instructions in the concept description.
The project is mainly built using maven, and currently provides packaging of shell scripts. Environment dependencies: Java(jdk1.6+) Maven
User usage is generally divided into two types:
You can upload the lts jar package to the local warehouse through the maven command. Add the corresponding repository in the parent pom.xml and upload it with the deploy command. For specific citation methods, please refer to the examples in lts.
Each module of lts needs to be packaged into a separate jar package, and all lts dependency packages need to be imported. For specific reference to which jar packages, you can refer to the examples in lts.
Two version scripts are provided (cmd)windows and (shell)linux for compilation and deployment:
Running the sh build.sh or build.cmd script in the root directory will generate the lts-{version}-bin folder in dist directory.
The following is its directory structure, in which the bin directory mainly contains the startup scripts of JobTracker and LTS-Admin. jobtracker contains the configuration file of JobTracker and the jar package that needs to be used. lts-admin is the war package and configuration file related to LTS-Admin. File structure of lts-{version}-bin
-- lts - $ { version }- bin
|-- bin
| |-- jobtracker . cmd
| |-- jobtracker . sh
| |-- lts - admin . cmd
| |-- lts - admin . sh
| |-- lts - monitor . cmd
| |-- lts - monitor . sh
| |-- tasktracker . sh
|-- conf
| |-- log4j . properties
| |-- lts - admin . cfg
| |-- lts - monitor . cfg
| |-- readme . txt
| |-- tasktracker . cfg
| |-- zoo
| |-- jobtracker . cfg
| |-- log4j . properties
| |-- lts - monitor . cfg
|-- lib
| |-- *. jar
|-- war
|-- jetty
| |-- lib
| |-- *. jar
|-- lts - admin . warconf/zoo , and then run sh jobtracker.sh zoo start . If you want to start two JobTracker nodes, then you need to copy a zoo, for example, name it zoo2 , modify the configuration file under zoo2 , and then run sh jobtracker.sh zoo2 start . The jobtracker-zoo.out log is generated in the logs folder.conf/lts-monitor.cfg and conf/lts-admin.cfg , and then run the sh lts-admin.sh or lts-admin.cmd script under bin . lts-admin.out log will be generated in the logs folder. If the startup is successful, the access address will be printed in the log. Users can access through this access address. The jar packages that need to be imported into lts include lts-jobclient-{version}.jar , lts-core-{version}.jar and other third-party dependent jars.
JobClient jobClient = new RetryJobClient ();
jobClient . setNodeGroup ( "test_jobClient" );
jobClient . setClusterName ( "test_cluster" );
jobClient . setRegistryAddress ( "zookeeper://127.0.0.1:2181" );
jobClient . start ();
// 提交任务
Job job = new Job ();
job . setTaskId ( "3213213123" );
job . setParam ( "shopId" , "11111" );
job . setTaskTrackerNodeGroup ( "test_trade_TaskTracker" );
// job.setCronExpression("0 0/1 * * * ?"); // 支持 cronExpression表达式
// job.setTriggerTime(new Date()); // 支持指定时间执行
Response response = jobClient . submitJob ( job );< bean id = "jobClient" class = "com.github.ltsopensource.spring.JobClientFactoryBean" >
< property name = "clusterName" value = "test_cluster" />
< property name = "registryAddress" value = "zookeeper://127.0.0.1:2181" />
< property name = "nodeGroup" value = "test_jobClient" />
< property name = "masterChangeListeners" >
< list >
< bean class = "com.github.ltsopensource.example.support.MasterChangeListenerImpl" />
</ list >
</ property >
< property name = "jobFinishedHandler" >
< bean class = "com.github.ltsopensource.example.support.JobFinishedHandlerImpl" />
</ property >
< property name = "configs" >
< props >
<!-- 参数 -->
< prop key = "job.fail.store" > leveldb </ prop >
</ props >
</ property >
</ bean > @ Configuration
public class LTSSpringConfig {
@ Bean ( name = "jobClient" )
public JobClient getJobClient () throws Exception {
JobClientFactoryBean factoryBean = new JobClientFactoryBean ();
factoryBean . setClusterName ( "test_cluster" );
factoryBean . setRegistryAddress ( "zookeeper://127.0.0.1:2181" );
factoryBean . setNodeGroup ( "test_jobClient" );
factoryBean . setMasterChangeListeners ( new MasterChangeListener []{
new MasterChangeListenerImpl ()
});
Properties configs = new Properties ();
configs . setProperty ( "job.fail.store" , "leveldb" );
factoryBean . setConfigs ( configs );
factoryBean . afterPropertiesSet ();
return factoryBean . getObject ();
}
} The jar packages that need to be imported into lts include lts-tasktracker-{version}.jar , lts-core-{version}.jar and other third-party dependent jars.
public class MyJobRunner implements JobRunner {
@ Override
public Result run ( JobContext jobContext ) throws Throwable {
try {
// TODO 业务逻辑
// 会发送到 LTS (JobTracker上)
jobContext . getBizLogger (). info ( "测试，业务日志啊啊啊啊啊" );
} catch ( Exception e ) {
return new Result ( Action . EXECUTE_FAILED , e . getMessage ());
}
return new Result ( Action . EXECUTE_SUCCESS , "执行成功了，哈哈" );
}
} TaskTracker taskTracker = new TaskTracker ();
taskTracker . setJobRunnerClass ( MyJobRunner . class );
taskTracker . setRegistryAddress ( "zookeeper://127.0.0.1:2181" );
taskTracker . setNodeGroup ( "test_trade_TaskTracker" );
taskTracker . setClusterName ( "test_cluster" );
taskTracker . setWorkThreads ( 20 );
taskTracker . start ();< bean id = "taskTracker" class = "com.github.ltsopensource.spring.TaskTrackerAnnotationFactoryBean" init - method = "start" >
< property name = "jobRunnerClass" value = "com.github.ltsopensource.example.support.MyJobRunner" />
< property name = "bizLoggerLevel" value = "INFO" />
< property name = "clusterName" value = "test_cluster" />
< property name = "registryAddress" value = "zookeeper://127.0.0.1:2181" />
< property name = "nodeGroup" value = "test_trade_TaskTracker" />
< property name = "workThreads" value = "20" />
< property name = "masterChangeListeners" >
< list >
< bean class = "com.github.ltsopensource.example.support.MasterChangeListenerImpl" />
</ list >
</ property >
< property name = "configs" >
< props >
< prop key = "job.fail.store" > leveldb </ prop >
</ props >
</ property >
</ bean > @ Configuration
public class LTSSpringConfig implements ApplicationContextAware {
private ApplicationContext applicationContext ;
@ Override
public void setApplicationContext ( ApplicationContext applicationContext ) throws BeansException {
this . applicationContext = applicationContext ;
}
@ Bean ( name = "taskTracker" )
public TaskTracker getTaskTracker () throws Exception {
TaskTrackerAnnotationFactoryBean factoryBean = new TaskTrackerAnnotationFactoryBean ();
factoryBean . setApplicationContext ( applicationContext );
factoryBean . setClusterName ( "test_cluster" );
factoryBean . setJobRunnerClass ( MyJobRunner . class );
factoryBean . setNodeGroup ( "test_trade_TaskTracker" );
factoryBean . setBizLoggerLevel ( "INFO" );
factoryBean . setRegistryAddress ( "zookeeper://127.0.0.1:2181" );
factoryBean . setMasterChangeListeners ( new MasterChangeListener []{
new MasterChangeListenerImpl ()
});
factoryBean . setWorkThreads ( 20 );
Properties configs = new Properties ();
configs . setProperty ( "job.fail.store" , "leveldb" );
factoryBean . setConfigs ( configs );
factoryBean . afterPropertiesSet ();
// factoryBean.start();
return factoryBean . getObject ();
}
}Parameter description
Generally, only one JobClient instance is needed in a JVM. Do not create a new JobClient instance for each task. This will greatly waste resources, because one JobClient can submit multiple tasks. The same JVM generally tries to maintain only one TaskTracker instance. If there are more instances, it may cause a waste of resources. When encountering a TaskTracker that needs to run multiple tasks, please refer to "One TaskTracker Performs Multiple Tasks" below.
Sometimes, business scenarios require the execution of multiple tasks. Some people may ask whether each task type requires a TaskTracker to execute. My answer is no. If you are in a JVM, it is best to use a TaskTracker to run multiple tasks, because using multiple TaskTracker instances in a JVM is a waste of resources (of course, when you have a large amount of certain tasks, you can Use a separate TaskTracker node to execute this task). So how can we implement a TaskTracker to perform multiple tasks? Below is the reference example I gave.
/**
* 总入口，在 taskTracker.setJobRunnerClass(JobRunnerDispatcher.class)
* JobClient 提交 任务时指定 Job 类型 job.setParam("type", "aType")
*/
public class JobRunnerDispatcher implements JobRunner {
private static final ConcurrentHashMap < String /*type*/ , JobRunner >
JOB_RUNNER_MAP = new ConcurrentHashMap < String , JobRunner >();
static {
JOB_RUNNER_MAP . put ( "aType" , new JobRunnerA ()); // 也可以从Spring中拿
JOB_RUNNER_MAP . put ( "bType" , new JobRunnerB ());
}
@ Override
public Result run ( JobContext jobContext ) throws Throwable {
Job job = jobContext . getJob ();
String type = job . getParam ( "type" );
return JOB_RUNNER_MAP . get ( type ). run ( job );
}
}
class JobRunnerA implements JobRunner {
@ Override
public Result run ( JobContext jobContext ) throws Throwable {
// TODO A类型Job的逻辑
return null ;
}
}
class JobRunnerB implements JobRunner {
@ Override
public Result run ( JobContext jobContext ) throws Throwable {
// TODO B类型Job的逻辑
return null ;
}
} Generally, when writing TaskTracker, you only need to test whether the implementation logic of JobRunner is correct, but you do not want to start LTS for remote testing. To facilitate testing, LTS provides JobRunner’s shortcut testing method. Just integrate com.github.ltsopensource.tasktracker.runner.JobRunnerTester into your own test class, and implement initContext and newJobRunner methods. Such as the example in lts-examples:
public class TestJobRunnerTester extends JobRunnerTester {
public static void main ( String [] args ) throws Throwable {
// Mock Job 数据
Job job = new Job ();
job . setTaskId ( "2313213" );
JobContext jobContext = new JobContext ();
jobContext . setJob ( job );
JobExtInfo jobExtInfo = new JobExtInfo ();
jobExtInfo . setRetry ( false );
jobContext . setJobExtInfo ( jobExtInfo );
// 运行测试
TestJobRunnerTester tester = new TestJobRunnerTester ();
Result result = tester . run ( jobContext );
System . out . println ( JSON . toJSONString ( result ));
}
@ Override
protected void initContext () {
// TODO 初始化Spring容器
}
@ Override
protected JobRunner newJobRunner () {
return new TestJobRunner ();
}
}For Quartz's Cron tasks, you only need to add some code to the Spring configuration to connect to the LTS platform.
< bean class = " com.github.ltsopensource.spring.quartz.QuartzLTSProxyBean " >
< property name = " clusterName " value = " test_cluster " />
< property name = " registryAddress " value = " zookeeper://127.0.0.1:2181 " />
< property name = " nodeGroup " value = " quartz_test_group " />
</ bean > @ SpringBootApplication
@ EnableJobTracker // 启动JobTracker
@ EnableJobClient // 启动JobClient
@ EnableTaskTracker // 启动TaskTracker
@ EnableMonitor // 启动Monitor
public class Application {
public static void main ( String [] args ) {
SpringApplication . run ( Application . class , args );
}
} All that's left is to add the corresponding configuration in application.properties. For details, see the example under com.github.ltsopensource.examples.springboot package in lts-example.
When the machine has two network cards on the internal network, sometimes if the user wants LTS traffic to go through the external network card, then he needs to change the mapping address of the host name to the external network card address in the host. The same goes for the internal network. .
If the node ID is set when the node is started, LTS will set a UUID as the node ID by default. The readability will be poor, but the uniqueness of each node can be guaranteed. If the user can ensure the uniqueness of the node ID by himself, he can setIdentity to set, for example, if each node is deployed on a machine (a virtual machine), then the identity can be set to the host name
Support SPI extensions for JobLogger, JobQueue, etc.
See issue#389
Working experience is more than three years
Education requirementsBachelor's degree
Expectation level P6 (senior Java engineer)/P7 (technical expert)
Job description
The membership platform is responsible for the user system of Alibaba Group, supports the user needs of various business lines within the group, and supports the user and business channels of the group's external cooperation. Including user login & authorization, session system, registration, account management, account security and other functions at each end, underlying user information services, session and credential management, etc., it is one of the core product lines of the group, carrying hundreds of billions of calls every day volume, peak QPS of tens of millions, and a globally distributed hybrid cloud architecture, etc.
As a Software Engineer, you will work on our core products that provide critical functionality for our business infrastructure. Depending on your interests and experience, you may work in one or more of the following areas: Globalization, User experience, data security, machine learning, system high availability and more.
