springbatch step数据共享 springbatch文档

    距离开始使用 Spring Batch 有一段时间了，一直没有时间整理，现在项目即将完结，整理下这段时间学习和使用经历。

一、定义与特点

       A lightweight, comprehensive batch framework designed to enable the development of robust batch applications vital for the daily operations of enterprise systems.

Spring Batch provides reusable functions that are essential in processing large volumes of records, including logging/tracing, transaction management, job processing statistics, job restart, skip, and resource management. It also provides more advanced technical services and features that will enable extremely high-volume and high performance batch jobs through optimization and partitioning techniques. Simple as well as complex, high-volume batch jobs can leverage the framework in a highly scalable manner to process significant volumes of information.

Features

• Transaction management
• Chunk based processing
• Declarative I/O
• Start/Stop/Restart
• Rety/Skip
• Web based administration interface (Spring Batch Admin)

二、简介

    Spring Batch 是一个依托 Spring，面向批处理的框架，可以应用于企业级数据处理系统。通过阅读官网文档，可以知道 Spring Batch 的核心组件包括 Job、Step 等。Spring Batch 不仅提供了统一的读写接口、丰富的任务处理方式、灵活的事务管理及并发处理，同时还支持日志、监控、任务重启与跳过等特性，大大简化了批处理应用开发，将开发人员从复杂的任务配置管理过程中解放出来，使他们可以更多地去关注核心的业务处理过程。

使用场景

• Commit batch process periodically
• Concurrent batch processing: parallel processing of a job
• Staged, enterprise message-driven processing
• Massively parallel batch processing
• Manual or scheduled restart after failure
• Sequential processing of dependent steps (with extensions to workflow-driven batches)
• Partial processing: skip records (e.g. on rollback)
• Whole-batch transaction: for cases with a small batch size or existing stored procedures/scripts

 

三、HelloWorld

程序简介：从指定路径的文本文件中逐行读取，获取用户的姓和名，并在处理器中拼接用户的姓名，最后输出用户的姓名。

操作系统：Win7 x64 旗舰版

开发环境：Eclipse 4.3 、JDK1.6

步骤：

1. 搭建开发工程

打开Eclipse， 新建 Java Project ，本例使用 SpringBatchTest 为项目名。

新建 lib 文件夹，导入 SpringBatch 的 Jar 包和其他依赖包。建立相关 package 和 class ，得到结构如下图：

其中 包 和 类

acc 存放访问控制类（本例准备存放作业测试类）

batch.listener

batch.processor

batch.reader

batch.writer

batch.mapper

batch.data

BatchServer.java

配置文件

spring-application-batch.xml  定义spring batch 核心组件和自定义作业

spring-application-resource.xml  定义spring 组件

spring-application-context.xml  根配置文件，引入使用的配置文件，并控制配置文件引入顺序

2. 编写配置文件和对应的程序代码

由于开发过程中，配置文件和程序是并行书写的，所以以下内容无特定顺序

(1) Spring Batch 配置文件及其中定义的组件实例

spring-application-batch.xml

1 <?xml version="1.0" encoding="UTF-8"?>
 2 <beans xmlns="http://www.springframework.org/schema/beans"
 3     xmlns:batch="http://www.springframework.org/schema/batch"
 4     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
 5     xsi:schemaLocation="http://www.springframework.org/schema/beans  
 6             http://www.springframework.org/schema/beans/spring-beans-3.0.xsd  
 7             http://www.springframework.org/schema/batch   
 8             http://www.springframework.org/schema/batch/spring-batch-2.2.xsd"
 9     default-autowire="byName">
10 
11     <!-- Spring Batch  内存模型 -->
12     <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" />
13     <bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
14         <property name="jobRepository" ref="jobRepository" />
15     </bean>
16     <bean id="taskExecutor" class="org.springframework.core.task.SyncTaskExecutor" />
17 
18 
19     <!-- 文本行与逻辑对象映射处理 -->
20     <bean id="customerLineMapper" class="cn.spads.batch.mapper.CustomLineMapper"/>
21     <bean id="lineTokenizer" class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer" >
22         <property name="delimiter" value=" "/>
23     </bean>
24 
25     <!-- Scope = step 变量后绑定固定写法，即可以在对象调用时绑定变量 -->
26     <bean id="customReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
27         <property name="lineMapper">
28             <bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
29                 <property name="lineTokenizer" ref="lineTokenizer"/>
30                 <property name="fieldSetMapper" ref="customerLineMapper"/>
31             </bean>       
32         </property>
33         <!-- 此处使用单一文件绝对路径 -->
34         <property name="resource" value="file:#{jobParameters['customFileAbPath']}"/>
35     </bean>
36 
37     <bean id="customProcessor" class="cn.spads.batch.processor.CustomProcessor"/>
38     <bean id="customWriter" class="cn.spads.batch.writer.CustomWriter"/>
39 
40     <bean id="customJobListener" class="cn.spads.batch.listener.CustomJobListener"/>
41     <bean id="customStepListener" class="cn.spads.batch.listener.CustomStepListener"/>
42 
43 
44     <batch:job id="customJob">
45         <batch:step id="customJob_first_step">
46             <batch:tasklet>
47                 <batch:chunk reader="customReader" processor="customProcessor" 
48                     writer="customWriter" commit-interval="100">
49                 </batch:chunk>
50                 <batch:listeners>
51                     <batch:listener ref="customStepListener" />
52                 </batch:listeners>
53             </batch:tasklet>
54         </batch:step>
55         <batch:listeners>
56             <batch:listener ref="customJobListener"/>
57         </batch:listeners>
58     </batch:job>
59 </beans>

由于本示例使用Spring Batch 提供 的固定长度文本加载实例(FlatFileItemReader),因此没有自定义 Reader。

LineVo.java

package cn.spads.batch.data;

/**
 * <b>文本行逻辑对象</b><br>
 * @author        Gaylen
 * @version        V1.1.0
 * history
 * 1.1.0, 2014年11月24日        Gaylen            FE
 * @since        Java 6.0
 */
public class LineVo {

    /** 行号 */
    private int id;
    /** 名 */
    private String givenName;
    /** 姓 */
    private String familyName;
    /** 全名 */
    private String fullName;

    public int getId() {
        return id;
    }

    public void setId(int id) {
        this.id = id;
    }

    public String getGivenName() {
        return givenName;
    }

    public void setGivenName(String givenName) {
        this.givenName = givenName;
    }

    public String getFamilyName() {
        return familyName;
    }

    public void setFamilyName(String familyName) {
        this.familyName = familyName;
    }

    public String getFullName() {
        return fullName;
    }

    public void setFullName(String fullName) {
        this.fullName = fullName;
    }
}

View Code

CustomLineMapper.java

package cn.spads.batch.mapper;

import org.springframework.batch.item.file.mapping.FieldSetMapper;
import org.springframework.batch.item.file.transform.FieldSet;
import org.springframework.validation.BindException;

import cn.spads.batch.data.LineVo;

/**
 * <b>文本行-逻辑对象映射</b><br>
 * @author        Gaylen
 * @version        V1.1.0
 * history
 * 1.1.0, 2014-11-24        Gaylen            FE
 * @since        Java 6.0
 */
public class CustomLineMapper implements FieldSetMapper<LineVo> {

    /**
     * <b>映射处理</b><br>
     * @param fieldSet
     * @return DelCommandBean
     */
    @Override
    public LineVo mapFieldSet(FieldSet fieldSet) throws BindException {
        LineVo lv = new LineVo();
        lv.setId(Integer.parseInt(fieldSet.readString(0)));
        lv.setGivenName(fieldSet.readString(1));
        lv.setFamilyName(fieldSet.readString(2));
        return lv;
    }
}

View Code

CustomProcessor.java

package cn.spads.batch.processor;

import org.springframework.batch.item.ItemProcessor;

import cn.spads.batch.data.LineVo;

/**
 * <b>处理器</b><br>
 * @author        Gaylen
 * @version        V1.1.0
 * history
 * 1.1.0, 2014年11月24日        Gaylen            FE
 * @since        Java 6.0
 */
public class CustomProcessor implements ItemProcessor<LineVo, LineVo> {

    @Override
    public LineVo process(LineVo item) throws Exception {
        if (item == null) {
            return null;
        }
        item.setFullName(new StringBuilder().append(item.getFamilyName() == null ? "*" : item.getFamilyName())
                .append(" - ")
                .append(item.getGivenName() == null ? "*" : item.getGivenName())
                .toString());
        return item;
    }
}

View Code

CustomWriter.java

package cn.spads.batch.writer;

import java.util.List;

import org.springframework.batch.item.ItemWriter;

import cn.spads.batch.data.LineVo;

/**
 * <b>输出</b><br>
 * @author        Gaylen
 * @version        V1.1.0
 * history
 * 1.1.0, 2014年11月24日        Gaylen            FE
 * @since        Java 6.0
 */
public class CustomWriter implements ItemWriter<LineVo> {

    @Override
    public void write(List<? extends LineVo> items) throws Exception {
        if (items == null || items.size() == 0) {
            System.out.println("error.");
        } else {
            for (LineVo lv : items) {
                System.out.println(lv.getFullName());
            }
        }

    }
}

View Code

CustomJobListener.java 和 CustomStepListener.java

BatchServer.java

package cn.spads.batch;

import java.util.Calendar;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;

import org.springframework.batch.core.Job;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.core.JobParameter;
import org.springframework.batch.core.JobParameters;
import org.springframework.batch.core.JobParametersInvalidException;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.batch.core.repository.JobExecutionAlreadyRunningException;
import org.springframework.batch.core.repository.JobInstanceAlreadyCompleteException;
import org.springframework.batch.core.repository.JobRestartException;

/**
 * <b>批处理服务接口</b><br>
 * @author        Gaylen
 * @version        V1.1.0
 * history
 * 1.1.0, 2014年11月24日        Gaylen            FE
 * @since        Java 6.0
 */
public class BatchServer {

    /** 类单例对象 */
    private static final BatchServer INSTANCE = new BatchServer();

    /**
     * 单例
     * @return
     */
    public static BatchServer getInstance() {
        return INSTANCE;
    }

    /**
     * 私有构造方法
     */
    private BatchServer() {

    }

    /**
     * <b>测试作业</b><br>
     * @param launcher
     * @param job
     * @param paraMap
     */
    public void execCustomJob(JobLauncher launcher, Job job, Map<String, Object> paraMap) {
        JobExecution result = this.executeBatchJob(launcher, job, this.getJobParameters(paraMap));
        System.out.println(result.toString());
    }

    /**
     * <b>得到作业选项</b><br>
     * 默认配置任务开始时间
     * @param paraMap
     * @return
     */
    private JobParameters getJobParameters(Map<String, Object> paraMap) {
        HashMap<String, JobParameter> parameters = new HashMap<String, JobParameter>();
        parameters.put("time", new JobParameter(Calendar.getInstance().getTimeInMillis()));
        String key = null;
        Object value = null;

        if (paraMap == null || paraMap.size() == 0) {
            return new JobParameters(parameters);
        }

        for (Entry<String, Object> entry : paraMap.entrySet()) {
            if (entry == null) {
                continue;
            }
            key = entry.getKey();
            value = entry.getValue();

            if (value instanceof Date) {
                parameters.put(key, new JobParameter((Date) value));
            } else if (value instanceof String || value instanceof Integer) {
                parameters.put(key, new JobParameter((String) value));
            } else if (value instanceof Double) {
                parameters.put(key, new JobParameter((Double) value));
            } else if (value instanceof Long) {
                parameters.put(key, new JobParameter((Long) value));
            }
        }

        return new JobParameters(parameters);
    }

    /**
     * <b>批处理执行器</b><br>
     * @param joblanuncher
     * @param job
     * @param parameters
     */
    public JobExecution executeBatchJob(JobLauncher launcher, Job job, JobParameters jobParameters) {
        JobExecution result = null;
        try {
            result = launcher.run(job, jobParameters);
        } catch (JobExecutionAlreadyRunningException e) {
            e.printStackTrace();
        } catch (JobRestartException e) {
            e.printStackTrace();
        } catch (JobInstanceAlreadyCompleteException e) {
            e.printStackTrace();
        } catch (JobParametersInvalidException e) {
            e.printStackTrace();
        }
        return result;
    }
}

View Code

(2) acc 包下 新建测试类 MainTest.java 并在 I盘 新建 SpringBatchTest.txt

package cn.spads.acc;

import java.util.HashMap;
import java.util.Map;

import org.springframework.batch.core.Job;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.FileSystemXmlApplicationContext;

import cn.spads.batch.BatchServer;

/**
 * <b>批处理测试入口</b><br>
 * @author        Gaylen
 * @version        V1.1.0
 * history
 * 1.1.0, 2014年11月24日        Gaylen            FE
 * @since        Java 6.0
 */
public class MainTest {

    static private String fileLocation = "I:/SpringBatchTest.txt";

    static private void testCustomJob(ApplicationContext context) {
        JobLauncher launcher = (JobLauncher) context.getBean("jobLauncher");
        Job job = (Job) context.getBean("customJob");
        Map<String, Object> paraMap = new HashMap<String, Object>();
        paraMap.put("customFileAbPath", fileLocation);
        BatchServer.getInstance().execCustomJob(launcher, job, paraMap);
    }

    public static void main(String[] args) {
        ApplicationContext context = new FileSystemXmlApplicationContext("config/spring-application-context.xml");
        testCustomJob(context);
    }
}

View Code

文本文件(SpringBatchTest.txt)内容如下：

1 三 张
2 四 李
3 五 王
4 六 马

View Code

通过以上步骤, project 目录结构应如下图:

3. 至此整个 HelloWorld 项目搭建完成， 可以运行程序，得到输出结果如下： 

1 2014-11-25 0:12:28 org.springframework.context.support.FileSystemXmlApplicationContext prepareRefresh
 2 信息: Refreshing org.springframework.context.support.FileSystemXmlApplicationContext@bf32c: startup date [Tue Nov 25 00:12:28 CST 2014]; root of context hierarchy
 3 2014-11-25 0:12:28 org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
 4 信息: Loading XML bean definitions from file [D:\LocalDEV\workspace43\SpringBatchTest\config\spring-application-context.xml]
 5 2014-11-25 0:12:28 org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
 6 信息: Loading XML bean definitions from file [D:\LocalDEV\workspace43\SpringBatchTest\config\spring-application-resource.xml]
 7 2014-11-25 0:12:28 org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
 8 信息: Loading XML bean definitions from file [D:\LocalDEV\workspace43\SpringBatchTest\config\spring-application-batch.xml]
 9 2014-11-25 0:12:28 org.springframework.beans.factory.support.DefaultListableBeanFactory registerBeanDefinition
10 信息: Overriding bean definition for bean 'customJob': replacing [Generic bean: class [org.springframework.batch.core.configuration.xml.SimpleFlowFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null] with [Generic bean: class [org.springframework.batch.core.configuration.xml.JobParserJobFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
11 2014-11-25 0:12:28 org.springframework.beans.factory.support.DefaultListableBeanFactory registerBeanDefinition
12 信息: Overriding bean definition for bean 'customReader': replacing [Generic bean: class [org.springframework.batch.item.file.FlatFileItemReader]; scope=step; abstract=false; lazyInit=false; autowireMode=1; dependencyCheck=0; autowireCandidate=false; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in file [D:\LocalDEV\workspace43\SpringBatchTest\config\spring-application-batch.xml]] with [Root bean: class [org.springframework.aop.scope.ScopedProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in BeanDefinition defined in file [D:\LocalDEV\workspace43\SpringBatchTest\config\spring-application-batch.xml]]
13 2014-11-25 0:12:28 org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
14 信息: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@157fb52: defining beans [transactionManager,jobRepository,jobLauncher,taskExecutor,customerLineMapper,lineTokenizer,customReader,customProcessor,customWriter,customJobListener,customStepListener,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,customJob_first_step,customJob,scopedTarget.customReader]; root of factory hierarchy
15 2014-11-25 0:12:28 org.springframework.batch.core.launch.support.SimpleJobLauncher run
16 信息: Job: [FlowJob: [name=customJob]] launched with the following parameters: [{time=1416845548796, customFileAbPath=I:/SpringBatchTest.txt}]
17 2014-11-25 0:12:28 org.springframework.batch.core.job.SimpleStepHandler handleStep
18 信息: Executing step: [customJob_first_step]
19 张 - 三
20 李 - 四
21 王 - 五
22 马 - 六
23 2014-11-25 0:12:28 org.springframework.batch.core.launch.support.SimpleJobLauncher run
24 信息: Job: [FlowJob: [name=customJob]] completed with the following parameters: [{time=1416845548796, customFileAbPath=I:/SpringBatchTest.txt}] and the following status: [COMPLETED]
25 JobExecution: id=0, version=2, startTime=Tue Nov 25 00:12:28 CST 2014, endTime=Tue Nov 25 00:12:28 CST 2014, lastUpdated=Tue Nov 25 00:12:28 CST 2014, status=COMPLETED, exitStatus=exitCode=COMPLETED;exitDescription=, job=[JobInstance: id=0, version=0, Job=[customJob]], jobParameters=[{time=1416845548796, customFileAbPath=I:/SpringBatchTest.txt}]

View Code

总结：本篇文章简单介绍了 Spring Batch，以及使用 Spring Batch 开发 HelloWorld 程序。