Storm保证从spout发射出来的每个消息都被完全处理。该文章描述storm如何实现消息可靠处理,从storm的可靠性特性获得好处,用户需要做哪些工作。
- TopologyBuilder builder = new TopologyBuilder();
- builder.setSpout("sentences", new KestrelSpout("kestrel.backtype.com",
- 22133,
- "sentence_queue",
- new StringScheme()));
- builder.setBolt("split", new SplitSentence(), 10)
- .shuffleGrouping("sentences");
- builder.setBolt("count", new WordCount(), 20)
- .fieldsGrouping("split", new Fields("word"));
当消息树中的每个消息都已处理时,storm认为从一个spout出来的一个元组是“完全地处理”。当一个元组的消息树在指定时间内未完全地处理,Storm认为一个元组失败。可使用 Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS配置Timeout,默认值为30秒。
- public interface ISpout extends Serializable {
- void open(Map conf, TopologyContext context, SpoutOutputCollector collector);
- void close();
- void nextTuple();
- void ack(Object msgId);
- void fail(Object msgId);
- }
- public class SplitSentence implements IRichBolt {
- OutputCollector _collector;
- public void prepare(Map conf, TopologyContext context, OutputCollector collector) {
- _collector = collector;
- }
- public void execute(Tuple tuple) {
- String sentence = tuple.getString(0);
- for(String word: sentence.split(" ")) {
- _collector.emit(tuple, new Values(word));
- }
- _collector.ack(tuple);
- }
- public void cleanup() {
- }
- public void declareOutputFields(OutputFieldsDeclarer declarer) {
- declarer.declare(new Fields("word"));
- }
- }
- _collector.emit(new Values(word));
- List<Tuple> anchors = new ArrayList<Tuple>();
- anchors.add(tuple1);
- anchors.add(tuple2);
- _collector.emit(anchors, new Values(1, 2, 3));
- public class SplitSentence implements IBasicBolt {
- public void prepare(Map conf, TopologyContext context) {
- }
- public void execute(Tuple tuple, BasicOutputCollector collector) {
- String sentence = tuple.getString(0);
- for(String word: sentence.split(" ")) {
- collector.emit(new Values(word));
- }
- }
- public void cleanup() {
- }
- public void declareOutputFields(OutputFieldsDeclarer declarer) {
- declarer.declare(new Fields("word"));
- }
- }