spring storm spring storm整合

转载

mob6454cc6e1f98 2023-06-14 16:42:13

文章标签 spring storm spring storm 集成 xml 文章分类 Storm 大数据

我们知道storm是一个独立的实时计算框架，而Spring是一个独立的轻量级容器框架，那么如何在storm框架中集成Spring框架，以便于在storm开发中利用Spring的控制反转(IoC)和面向切面(AOP)性能来做java对象的管理呢？首先我们应该知道Spring是如何来管理java对象的，storm是怎么运行拓扑的，知道了这个、些才能明白怎么将Spring集成到storm中。

我们经常接触到的Spring框架其实是在Spring的基础上封装的一个web框架，和懒人整合包（starter）Springboot，那么他们是怎么管理bean的呢？实际上，spring mvc和Springboot都是利用上下文ApplicationContext或者beanFactory作为容易来管理java对象的整个生命周期，其中Spring mvc是通过读取web.xml中的配置来初始化上下文环境并装配配置文件中的bean，而Springboot则是通过一个启动类来初始化上下文环境并装配启动类所在路径及其子路径下的bean。

那么我们来回顾一下storm的基本要素以及他是怎么启动运行一个storm拓扑的。storm的拓扑是由数据源spout和数据处理单元bolt组成的，spout是产生数据的地方，spout可以从文件读取数据，从Kafka获取数据（KafkaSpout），从网络获取数据，经过简单处理后发送出来，bolt接收到spout发出的数据，对数据经过一系列的处理，可以得到最终需要的数据，而拓扑将spout和bolt链接起来形成一个实时数据的处理链。

spout

public class TestWordSpout extends BaseRichSpout{
    private SpoutOutputCollector collector = null;
    public void open(Map map, TopologyContext topologyContext, SpoutOutputCollector spoutOutputCollector) {
        this.collector = spoutOutputCollector;
    }
    public void nextTuple() {
        Utils.sleep(1000);
        final String[] words = new String[]{"fdfs","fdfs","ffsdfs"};
        final Random random = new Random();
        final String word = words[random.nextInt(words.length)];
        collector.emit(new Values(word));
    }
    public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
        outputFieldsDeclarer.declare(new Fields("word"));
    }
}

bolt

public class ExclamationBolt extends BaseRichBolt{
    private OutputCollector collector = null;
    public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) {
        this.collector = outputCollector;
    }
    public void execute(Tuple tuple) {
        this.collector.emit(tuple,new Values(tuple.getString(0)+"!!!"));
        this.collector.ack(tuple);
    }
    public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
        outputFieldsDeclarer.declare(new Fields("word"));
    }
}

topology

public class WordsTopology {
    public static void main(String[] args) throws AlreadyAliveException, InvalidTopologyException {
        TopologyBuilder topologyBuilder = new TopologyBuilder();
        topologyBuilder.setSpout("word",new TestWordSpout(),1);
        topologyBuilder.setBolt("exclaim",new ExclamationBolt(),1).shuffleGrouping("word");
        Config config = new Config();
        config.setDebug(true);
        if(args != null && args.length > 0){
            config.setNumWorkers(3); //集群模式
            StormSubmitter.submitTopologyWithProgressBar(args[0],config,topologyBuilder.createTopology());
        }else{ //本地模式
            LocalCluster localCluster = new LocalCluster();
            localCluster.submitTopology("test",config,topologyBuilder.createTopology());
            Utils.sleep(30000);
            localCluster.killTopology("test");
            localCluster.shutdown();
        }
    }
}

可以看出storm是由main函数启动，并没有用到Spring框架，那么怎么才可以在storm中启动Spring的上下文环境呢？我们可以分别从Spring mvc和Spring boot中获得启发。

Spring mvc 框架形式

在Spring mvc框架中，实际上就是通过web.xml指定的配置文件来初始化上下文环境并实例化bean，那么参考我们在sprin mvc的测试类中的做法，我们可以手动的根据配置文件获取上下文环境并实例化bean（这部分一般在prepare方法中进行）：

ApplicationContext applicationContext = new ClassPathXmlApplicationContext("application-context.xml");
DimDataClient client = (DimDataClient) applicationContext.getBean("dimDataClient");

考虑到storm中的bolt是分布在集群中的各个节点上的，因此每一个bolt都需要启动各自的Spring上下文环境，因此需要在每个bolt中都使用上述代码获取上下文环境，并在ApplicationContext中获取相应的bean。

applicationcontext.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:util="http://www.springframework.org/schema/util"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd
       http://www.springframework.org/schema/context
       http://www.springframework.org/schema/context/spring-context-4.2.xsd
       http://www.springframework.org/schema/util
       http://www.springframework.org/schema/util/spring-util.xsd">

    <bean id="yourBean" class="com.sankuai.tdp.YourBean">
        <property name="order" value="2"/>
        <property name="ignoreUnresolvablePlaceholders" value="false"/>
    </bean>
</beans>

然后我们就可以在bolt中创建上下文环境并使用其中的对象了：

public class myBolt extends BaseBasicBolt {
    private YourBean yourBean;
    private static final ApplicationContext APPLICATION_CONTEXT = new ClassPathXmlApplicationContext("application-context.xml");
    static {
        YourBean = (YourBean) APPLICATION_CONTEXT.getBean("yourBean");
    }
    @Overeide
    public void prepare(Map map, TopologyContext topologyContext) {
        //prepare code
    }
    @Overeide
    public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
        outputFieldsDeclarer.declare(UpsRtSaveData.genFields());
    }
    @Overeide
    public void execute(Tuple tuple, BasicOutputCollector basicOutputCollector) {
        //excute code
    }
}

Spring boot框架形式

我们参考Springboot的启动类发现，启动类中实际上就是新建了一个SpringApplication并调用其run方法，因此我们可以自己写一个启动类，并手动调用这个启动类。

SpringStormApplication启动类

@SpringBootApplication
@ComponentScan(value = "com.xxx.storm")
public class SpringStormApplication {
    /**
     * 非工程启动入口，所以不用main方法
     * 加上synchronized的作用是由于storm在启动多个bolt线程实例时，如果Springboot用到Apollo分布式配置，会报ConcurrentModificationException错误
     */
    public synchronized static void run(String ...args) {
        SpringApplication app = new SpringApplication(SpringStormApplication.class);
        //我们并不需要web servlet功能，所以设置为WebApplicationType.NONE
        app.setWebApplicationType(WebApplicationType.NONE);
        //忽略掉banner输出
        app.setBannerMode(Banner.Mode.OFF);
        //忽略Spring启动信息日志
        app.setLogStartupInfo(false);
        app.run(args);
    }
}

有了这个启动类相当于建立了Spring的上下文环境，我们还需要拿到改上下文环境中的bean对象，这里需要实现ApplicationContextAware接口

@Component
public class BeanUtils implements ApplicationContextAware {
    private static ApplicationContext applicationContext = null;
    @Override
    public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
        if (BeanUtils.applicationContext == null) {
            BeanUtils.applicationContext = applicationContext;
        }
    }
    public static ApplicationContext getApplicationContext() {
        return applicationContext;
    }
    public static Object getBean(String name) {
        return getApplicationContext().getBean(name);
    }
    public static <T> T getBean(Class<T> clazz) {
        return getApplicationContext().getBean(clazz);
    }
    public static <T> T getBean(String name, Class<T> clazz) {
        return getApplicationContext().getBean(name, clazz);
    }
}

然后我们就可以在bolt中创建上下文环境并使用其中的对象了：

public class myBolt extends BaseBasicBolt {
    private YourBean yourBean;
    
    @Overeide
    public void prepare(Map map, TopologyContext topologyContext) {
        SpringStormApplication.run();
        YourBean = (YourBean) BeanUtils.getBean("yourBean");
        //prepare code
    }
    @Overeide
    public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
        outputFieldsDeclarer.declare(UpsRtSaveData.genFields());
    }
    @Overeide
    public void execute(Tuple tuple, BasicOutputCollector basicOutputCollector) {
        //excute code
    }
}

启动拓扑：在打包storm时需要引入插件

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.1.0</version>
    <configuration>
        <artifactSet>
            <excludes>
                <exclude>org.slf4j:slf4j-simple</exclude>
                <exclude>org.slf4j:slf4j-log4j12</exclude>
            </excludes>
        </artifactSet>
        <filters>
            <filter>
                <artifact>*:*</artifact>
                <excludes>
                    <exclude>META-INF/*.SF</exclude>
                    <exclude>META-INF/*.DSA</exclude>
                    <exclude>META-INF/*.RSA</exclude>
                </excludes>
            </filter>
        </filters>
        <!-- Additional configuration. -->
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <finalName>${project.artifactId}-${project.version}-with-dependencies</finalName>
            </configuration>
        </execution>
    </executions>
</plugin>

如果打包出错，要添加两个文件：spring.handlers和spring.schemas，也就是在插件中添加标签<transformers>

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.1.0</version>
    <configuration>
        <artifactSet>
            <excludes>
                <exclude>org.slf4j:slf4j-simple</exclude>
                <exclude>org.slf4j:slf4j-log4j12</exclude>
            </excludes>
        </artifactSet>
        <filters>
            <filter>
                <artifact>*:*</artifact>
                <excludes>
                    <exclude>META-INF/*.SF</exclude>
                    <exclude>META-INF/*.DSA</exclude>
                    <exclude>META-INF/*.RSA</exclude>
                </excludes>
            </filter>
        </filters>
        <!-- Additional configuration. -->
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <finalName>${project.artifactId}-${project.version}-with-dependencies</finalName>
                <transformers>
                    <transformer
                            implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                        <resource>META-INF/spring.handlers</resource>
                    </transformer>
                    <transformer
                            implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                        <resource>META-INF/spring.schemas</resource>
                    </transformer>
                </transformers>
            </configuration>
        </execution>
    </executions>
</plugin>

其中spring.handlers为：

http\://www.springframework.org/schema/p=org.springframework.beans.factory.xml.SimplePropertyNamespaceHandler
http\://www.springframework.org/schema/util=org.springframework.beans.factory.xml.UtilNamespaceHandler
http\://www.springframework.org/schema/aop=org.springframework.aop.config.AopNamespaceHandler
http\://www.springframework.org/schema/context=org.springframework.context.config.ContextNamespaceHandler
http\://www.springframework.org/schema/jee=org.springframework.ejb.config.JeeNamespaceHandler
http\://www.springframework.org/schema/lang=org.springframework.scripting.config.LangNamespaceHandler

spring.schemas为：

http\://www.springframework.org/schema/context/spring-context-2.5.xsd=org/springframework/context/config/spring-context-2.5.xsd
http\://www.springframework.org/schema/context/spring-context-3.0.xsd=org/springframework/context/config/spring-context-3.0.xsd
http\://www.springframework.org/schema/context/spring-context.xsd=org/springframework/context/config/spring-context-3.0.xsd
http\://www.springframework.org/schema/jee/spring-jee-2.0.xsd=org/springframework/ejb/config/spring-jee-2.0.xsd
http\://www.springframework.org/schema/jee/spring-jee-2.5.xsd=org/springframework/ejb/config/spring-jee-2.5.xsd
http\://www.springframework.org/schema/jee/spring-jee-3.0.xsd=org/springframework/ejb/config/spring-jee-3.0.xsd
http\://www.springframework.org/schema/jee/spring-jee.xsd=org/springframework/ejb/config/spring-jee-3.0.xsd
http\://www.springframework.org/schema/lang/spring-lang-2.0.xsd=org/springframework/scripting/config/spring-lang-2.0.xsd
http\://www.springframework.org/schema/lang/spring-lang-2.5.xsd=org/springframework/scripting/config/spring-lang-2.5.xsd
http\://www.springframework.org/schema/lang/spring-lang-3.0.xsd=org/springframework/scripting/config/spring-lang-3.0.xsd
http\://www.springframework.org/schema/lang/spring-lang.xsd=org/springframework/scripting/config/spring-lang-3.0.xsd
http\://www.springframework.org/schema/task/spring-task-3.0.xsd=org/springframework/scheduling/config/spring-task-3.0.xsd
http\://www.springframework.org/schema/task/spring-task.xsd=org/springframework/scheduling/config/spring-task-3.0.xsd

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。