spring batch 把数据放到stepExecutionContext中 spring batch 多个step

转载

编程梦想实现家 2024-02-21 13:49:18

springbatch学习笔记

一、简介

二、项目搭建

1.创建项目

方式一：https://start.spring.io/ 网页创建项目导入开发工具
方式二：开发工具直接创建springboot项目配置batch相关依赖
注：方式一配置时，如出现找不到数据源错误，只需要在pom中配置数据源即可。如：

<dependency>
			<groupId>com.h2database</groupId>
			<artifactId>h2</artifactId>
			<scope>runtime</scope>
		</dependency>

2.入门例子编写

一个job可以创建多个step，每一个step都是一个执行步骤，需要用Bean注入进来。执行过程中只有step正常返回才会走下面的step。
step执行有两种方式：tasklet、chunk（ItemReader、ItemProcessor、ItemWriter）
flow：可以包含一个或多个step。

public Flow flowDemo(){
        return new FlowBuilder<Flow>("flowDemo").start(step1()).next(step2()).build();
    }

split：并发执行，每个step有不同的线程来调用。
job中可以用step也可以用flow，调用flow时，将flow中包含的step一起处理。

@Configuration
@EnableBatchProcessing
public class JobConfiguration {

    //需要创建任务对象
    @Autowired
    private JobBuilderFactory jobBuilderFactory;

    //任务执行由step决定
    //创建step对象的对象
    @Autowired
    private StepBuilderFactory stepBuilderFactory;

    //创建人物对象
    @Bean
    public Job helloWorldJob() {
        return jobBuilderFactory.get("helloWorldJob").start(step1()).build();
    }

    @Bean
    public Step step1() {
        return stepBuilderFactory.get("step1").tasklet(new Tasklet() {
            @Override
            public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
                System.out.println("helloworld");
                return RepeatStatus.FINISHED;
            }
        }).build();
    }
}

3.与持久化互联（采用mysql为例）

在application.properties中配置连接数据库

spring.datasource.username=root
spring.datasource.password=Caoxu0407@
spring.datasource.url=jdbc:mysql://localhost:3306/springbatch?useUnicode=true&characterEncoding=UTF-8&serverTimezone=UTC
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
spring.datasource.schema=classpath:/org/springframework/batch/core/schema-mysql.sql
spring.batch.initialize-schema=always

再次执行任务代码，数据库会自动创建与batch相关的表存储执行结果及相关信息。（schema-mysql.sql）

4.使用决策器

决策器：根据判断条件决定走哪个step，相当于if功能。
决策器校验代码：

public class MyDecider implements JobExecutionDecider {
    private int count;
    @Override
    public FlowExecutionStatus decide(JobExecution jobExecution, StepExecution stepExecution) {
        count++;
        if(count%2==0){
            return new FlowExecutionStatus("even");
        }else {
            return new FlowExecutionStatus("odd");
        }
    }
}

调用端代码：

@Bean
    public JobExecutionDecider myDecider(){
        return new MyDecider();
    }
    @Bean
    public Job deciderJob(){
        return jobBuilderFactory.get("deciderJob").start(step1())
                .next(myDecider())
                .from(myDecider()).on("even").to(step2())
                .from(myDecider()).on("odd").to(step3())
                .end()
                .build();
    }

5.监听器

用来监听批处理作业的执行情况，创建监听器可以通过实现接口或使用注解。
调用端：

@Configuration
public class ListenerDemo {
    @Autowired
    private JobBuilderFactory jobBuilderFactory;
    @Autowired
    private StepBuilderFactory stepBuilderFactory;
    @Bean
    public Job listenerJob() {
        return jobBuilderFactory.get("listenerJob")
                .start(step1())
                .listener(new MyJobListener())
                .build();
    }
    @Bean
    public Step step1() {
        return stepBuilderFactory.get("step1")
                .<String, String>chunk(2)//大小
                .faultTolerant()//容错机制
                .listener(new MyJobListener())//监听器
                .reader(read())
                .writer(writer())
                .build();
    }
    @Bean
    public ItemWriter<String> writer() {
        return new ItemWriter<String>() {
            @Override
            public void write(List<? extends String> list) throws Exception {
                for (String item : list) {
                    System.out.println(item);
                }
            }
        };
    }
    @Bean
    public ItemReader<String> read() {
        return new ListItemReader<>(Arrays.asList("java", "spring", "mybatis"));
    }
}

监听器端：
第一种方式：

public class MyChunkListener {

    @BeforeChunk
    public void beforeListener(ChunkContext context) {
        System.out.println(context.getStepContext().getStepName() + "before");
    }
    @AfterChunk
    public void afterListener(ChunkContext context) {
        System.out.println(context.getStepContext().getStepName() + "after");
    }
}

第二种方式：

public class MyJobListener implements JobExecutionListener {
    @Override
    public void beforeJob(JobExecution jobExecution) {
        System.out.println(jobExecution.getJobInstance().getJobName()+"before");
    }
    @Override
    public void afterJob(JobExecution jobExecution) {
        System.out.println(jobExecution.getJobInstance().getJobName()+"after");
    }
}

传递参数：需要使用监听器before方法给其赋值，在调用时调用即可。

6.Itemreader使用

业务层调用端：

@Configuration
public class ItemreaderDemo {
    @Autowired
    private JobBuilderFactory jobBuilderFactory;
    @Autowired
    private StepBuilderFactory stepBuilderFactory;
    @Bean
    public Job itemreaderDemoJob() {
        return jobBuilderFactory.get("itemreaderDemoJob")
                .start(itemreaderDemoJobStep())
                .build();
    }
    @Bean
    public Step itemreaderDemoJobStep() {
        return stepBuilderFactory.get("itemreaderDemoJobStep")
                .<String, String>chunk(2)
                .reader(itemreaderDemoRead())
                .writer(list -> {
                    for (String item : list) {
                        System.out.println(item + "...");
                    }
                })
                .build();
    }
    @Bean
    public MyReader itemreaderDemoRead() {
        List<String> data = Arrays.asList("aaa", "bbb", "ccc");
        return new MyReader(data);
    }
}

MyReader端：

public class MyReader implements ItemReader<String> {
    private Iterator<String> iterator;
    public MyReader(List<String> list) {
        this.iterator = list.iterator();
    }
    public String read() throws Exception, UnexpectedException {
        if (iterator.hasNext()) {
            return this.iterator.next();
        } else {
            return null;
        }
    }
}

7.从数据库中读取数据

业务层：

@Configuration
public class ItemreadeDbrDemo {
    @Autowired
    private JobBuilderFactory jobBuilderFactory;
    @Autowired
    private StepBuilderFactory stepBuilderFactory;
    @Autowired
    private DataSource dataSource;
    @Autowired
    @Qualifier("dbjdebcWriter")
      private ItemWriter<? super User> dbjdebcWriter;
    @Bean
    public Job itemreaderDbDemoJob() {
        return jobBuilderFactory.get("itemreaderDbDemoJob")
                .start(itemreaderDbDemoJobStep())
                .build();
    }
    @Bean
    public Step itemreaderDbDemoJobStep() {
        return stepBuilderFactory.get("itemreaderDbDemoJobStep")
                .<User, User>chunk(2)
                .reader(jdbcDemoRead())
                .writer(dbjdebcWriter)
                .build();
    }
    @Bean
    @StepScope
    public JdbcPagingItemReader<User> jdbcDemoRead() {
        JdbcPagingItemReader<User> reader = new JdbcPagingItemReader();
        reader.setDataSource(dataSource);
        reader.setFetchSize(2);
        reader.setRowMapper(new RowMapper<User>() {
            //读取到的数据转换成user对象
            @Override
            public User mapRow(ResultSet rs, int rowNum) throws SQLException {
                User user = new User();
                user.setId(rs.getInt(1));
                user.setUsername(rs.getString(2));
                user.setPassword(rs.getString(3));
                user.setAge(rs.getInt(4));
                return user;
            }
        });
        //指定sql
        MySqlPagingQueryProvider properties = new MySqlPagingQueryProvider();
        properties.setSelectClause("id,username,password,age");
        properties.setFromClause("from user");
        //指定根据那个字段进行排序
        Map<String, Order> sort = new HashMap<>(1);
        sort.put("id", Order.ASCENDING);
        properties.setSortKeys(sort);
        reader.setQueryProvider(properties);
        return reader;
    }
}

writer遍历层

@Component("dbjdebcWriter")
public class DbjdebcWriter implements ItemWriter<User> {
    @Override
    public void write(List<? extends User> list) throws Exception {
        for (User user : list) {
            System.out.println(user);
        }
    }
}

8.从文件中读取

业务层：

@Configuration
public class FileItemreaderDemo {
    @Autowired
    private JobBuilderFactory jobBuilderFactory;
    @Autowired
    private StepBuilderFactory stepBuilderFactory;
    @Autowired
    private DataSource dataSource;
    @Autowired
    @Qualifier("flatFileWriter")
    private ItemWriter<? super User> flatFileWriter;
    @Bean
    public Job fileItemreaderDemo() {
        return jobBuilderFactory.get("fileItemreaderDemo")
                .start(fileItemreaderDemoStep())
                .build();
    }
    @Bean
    public Step fileItemreaderDemoStep() {
        return stepBuilderFactory.get("fileItemreaderDemoStep")
                .<User, User>chunk(2)
                .reader(fileItemreaderDemoRead())
                .writer(flatFileWriter)
                .build();
    }
    @Bean
    @StepScope
    public FlatFileItemReader<User> fileItemreaderDemoRead() {
        FlatFileItemReader<User> reader = new FlatFileItemReader();
        reader.setResource(new ClassPathResource("user.txt"));
        //跳过第一行，根据情况选择
        reader.setLinesToSkip(1);
        //数据解析
        DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
        tokenizer.setNames(new String[]{"id","username","password","age"});
        //把解析出来的一行映射到user对象
        DefaultLineMapper<User> mapper = new DefaultLineMapper<> ();
        mapper.setLineTokenizer(tokenizer);
        mapper.setFieldSetMapper(new FieldSetMapper<User>() {
            @Override
            public User mapFieldSet(FieldSet fieldSet) throws BindException {
                User user = new User();
                user.setId(fieldSet.readInt("id"));
                user.setUsername(fieldSet.readString("username"));
                user.setPassword(fieldSet.readString("password"));
                user.setAge(fieldSet.readInt("age"));
                return user;
            }
        });
       mapper.afterPropertiesSet();
       reader.setLineMapper(mapper);
        return reader;
    }
}

遍历层

@Component("flatFileWriter")
public class FlatFileWriter implements ItemWriter<UserBean> {
    @Override
    public void write(List<? extends UserBean>list) throws Exception {
        for(UserBean user:list){
            System.out.println(user);
        }
    }
}

8.ItemWriter使用

业务层：

@Configuration
public class ItemWriterDemo {
    @Autowired
    private JobBuilderFactory jobBuilderFactory;
    @Autowired
    private StepBuilderFactory stepBuilderFactory;
    @Autowired
    @Qualifier("myWriter")
    private ItemWriter<String> myWriter;
    @Bean
    public Job itemWriterDemoJob() {
        return jobBuilderFactory.get("itemWriterDemoJob")
                .start(itemWriterDemoStep())
                .build();
    }
    @Bean
    public Step itemWriterDemoStep() {
        return stepBuilderFactory.get("itemWriterDemoStep")
                .<String, String>chunk(5)
                .reader(myRead())
                .writer(myWriter)
                .build();
    }
    @Bean
    public ItemReader<String> myRead() {
        List<String> items = new ArrayList<>();
        for (int i = 0; i <= 50; i++) {
            items.add("java" + i);
        }
        return new ListItemReader<String>(items);
    }
}

MyWriter遍历层

@Component("myWriter")
public class MyWriter implements ItemWriter<String> {
    @Override
    public void write(List<? extends String> list) throws Exception {
        System.out.println(list.size());
        for(String str:list){
            System.out.println(str);
        }
    }
}

9.读取数据并插入数据库

业务层：

@Configuration
public class ItemWriterDbDemo {
    @Autowired
    private JobBuilderFactory jobBuilderFactory;
    @Autowired
    private StepBuilderFactory stepBuilderFactory;

    @Autowired
    @Qualifier("faltFileReaderConfig")
    private ItemReader<User> faltFileReaderConfig;
    @Autowired
    @Qualifier("itemWriterDbConfig")
    private ItemWriter<User> itemWriterDbConfig;

    @Bean
    public Job itemWriterDbDemoJob() {
        return jobBuilderFactory.get("itemWriterDbDemoJob")
                .start(itemWriterDbDemoStep())
                .build();
    }
    @Bean
    public Step itemWriterDbDemoStep() {
        return stepBuilderFactory.get("itemWriterDbDemoStep")
                .<User, User>chunk(2)
                .reader(faltFileReaderConfig)
                .writer(itemWriterDbConfig)
                .build();
    }
    @Bean
    public ItemReader<String> myRead() {
        List<String> items = new ArrayList<>();
        for (int i = 0; i <= 50; i++) {
            items.add("java" + i);
        }
        return new ListItemReader<String>(items);
    }
}

读文件

@Configuration
public class FaltFileReaderConfig {
    @Bean
    public FlatFileItemReader<User> faltFileReaderConfig() {
        FlatFileItemReader<User> reader = new FlatFileItemReader();
        reader.setResource(new ClassPathResource("user.txt"));
        //跳过第一行，根据情况选择
        //reader.setLinesToSkip(1);
        //数据解析
        DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
        tokenizer.setNames(new String[]{"id","username","password","age"});
        //把解析出来的一行映射到user对象
        DefaultLineMapper<com.example.springbatch.itemreaderDb.User> mapper = new DefaultLineMapper<> ();
        mapper.setLineTokenizer(tokenizer);
        mapper.setFieldSetMapper(new FieldSetMapper<com.example.springbatch.itemreaderDb.User>() {
            @Override
            public com.example.springbatch.itemreaderDb.User mapFieldSet(FieldSet fieldSet) throws BindException {
                com.example.springbatch.itemreaderDb.User user = new User();
                user.setId(fieldSet.readInt("id"));
                user.setUsername(fieldSet.readString("username"));
                user.setPassword(fieldSet.readString("password"));
                user.setAge(fieldSet.readInt("age"));
                return user;
            }
        });
        mapper.afterPropertiesSet();
        reader.setLineMapper(mapper);
        return reader;
    }
}

写数据库

@Configuration
public class ItemWriterDbConfig   {
    @Autowired
    private DataSource dataSource;
    @Bean
    public JdbcBatchItemWriter<User> itemWriterDbConfig(){
        JdbcBatchItemWriter<User> writer = new JdbcBatchItemWriter<User>();
        writer.setDataSource(dataSource);
        writer.setSql("inster into user (id,username,password,age) values"+
                "(:id,:username:password:age)");
        writer.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<User>());
        return writer;
    }
}

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。