java海量数据去重 java大数据去重

实测数据去重好方法

去重方法1

public class DataDealWithUtil {
 public static Predicate distinctByKey(Function<? super T, ?> keyExtractor) {
 Map<Object, Boolean> seen = new ConcurrentHashMap<>();
 return t -> seen.putIfAbsent(keyExtractor.apply(t), Boolean.TRUE) == null;
 }
 }resultList = resultList.stream().filter(DataDealWithUtil.distinctByKey(o -> o.getDistinct())).collect(Collectors.toList());

去重方法2

resultList = resultList .stream().collect(Collectors.collectingAndThen(Collectors.toCollection(() -> new TreeSet<>(Comparator.comparing(BusinessMsgBean :: getId))), ArrayList::new));

去重方法有很多，但这种方法我一直用，感觉很好用，分享一下。