电商项目实战-浏览量统计功能实现3

转载

mob604756f4c9c3 2021-07-22 10:15:00

文章标签 .net apache hadoop ide d3 文章分类 代码人生

1、统计页面的浏览量

将读入进来的文件
count
将一行记录做成一个固定的key，value赋值为1

2、PVStartApp.java

src/main/java/project/mrv1/PVStartApp.java

package project.mrv1;

/*
 * 第一版本浏览量的统计
 */

import mr.access.Access;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import java.io.IOException;

public class PVStartApp {

    public static void main(String[] args) throws Exception{

        Configuration configuration = new Configuration();

        Job job = Job .getInstance(configuration);
        job.setJarByClass(PVStartApp.class);

        job.setMapperClass(MyMapper.class);
        job.setReducerClass(MyReducer.class);

        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(LongWritable.class);

        job.setOutputKeyClass(NullWritable.class);
        job.setOutputValueClass(LongWritable.class);

        FileInputFormat.setInputPaths(job, new Path("access/raw/input/trackinfo_20130721.data"));
        FileOutputFormat.setOutputPath(job, new Path("output/v1/pvstart"));

        job.waitForCompletion(true);



    }

    static class MyMapper extends Mapper<LongWritable, Text, Text, LongWritable>{

        //将key固定
        private Text KEY = new Text("key");
        private LongWritable ONE = new LongWritable(1);

        //在这里要实现一个map方法
        //直接输入map+回车
        @Override
        protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
            context.write(KEY, ONE);
        }
    }

    static class MyReducer extends Reducer<Text, LongWritable, NullWritable, LongWritable>{

        //在这里要实现一个reduce方法
        //直接输入reduce+回车
        @Override
        protected void reduce(Text key, Iterable<LongWritable> values, Context context) throws IOException, InterruptedException {

            long count = 0;
            for (LongWritable value: values){
                count++;
            }

            context.write(NullWritable.get(), new LongWritable(count));
        }
    }

}

3、trackinfo_20130721.data

access/raw/input/trackinfo_20130721.data

20946835322http://www.yihaodian.com/1/?tracker_u=2225501&type=3http://www.baidu.com/s?wd=1%E5%8F%B7%E5%BA%97&rsv_bp=0&ch=&tn=baidu&bar=&rsv_spt=3&ie=utf-8&rsv_sug3=5&rsv_sug=0&rsv_sug1=4&rsv_sug4=313&inputT=42351号店1SKAPHD3JZYH9EE9ACB1NGA9VDQHNJMX1NY9TPPG4SWG71358HGRJGQHQQBXY9GF96CVU2225501\N124.79.172.232msessionid:YR9H5YU7RZ8Y94EBJNZ2P5W8DT37Q9JH,unionKey:22255012013-07-21 09:30:01\Nhttp://www.baidu.com/s?wd=1%E5%8F%B7%E5%BA%97&rsv_bp=0&ch=&tn=baidu&bar=&rsv_spt=3&ie=utf-8&rsv_sug3=5&rsv_sug=0&rsv_sug1=4&rsv_sug4=313&inputT=42351\Nnull-10Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; MATP; Media Center PC 6.0; .NET4.0C; InfoPath.2; .NET4.0E)Win32上海市12013-07-21 09:30:01上海市66\N\N\N\N2013-07-21
20947290187http://www.yihaodian.com/ctg/s2/c5287-%E5%A4%A7%E7%B1%B3/b/a28186,28184-s1-v0-p1-price-d0-f0-m1-rt0-pid-k/1/http://www.yihaodian.com/ctg/s2/c5287-%E5%A4%A7%E7%B1%B3/b1879/a281862SKAPHD3JZYH9EE9ACB1NGA9VDQHNJMX1NY9TPPG4SWG71358HGRJGQHQQBXY9GF96CVU22255011124.79.172.232msessionid:YR9H5YU7RZ8Y94EBJNZ2P5W8DT37Q9JH,uname:chen45311,unionKey:22255012013-07-21 09:32:30134885852http://www.baidu.com/s?wd=1%E5%8F%B7%E5%BA%97&rsv_bp=0&ch=&tn=baidu&bar=&rsv_spt=3&ie=utf-8&rsv_sug3=5&rsv_sug=0&rsv_sug1=4&rsv_sug4=313&inputT=4235110\N1-10Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; MATP; Media Center PC 6.0; .NET4.0C; InfoPath.2; .NET4.0E)Win32 93 1search_order_2上海市1null上海市66\N\N\N\N2013-07-21
20947323490http://www.yihaodian.com/cmsPage/show.do?pageId=49248&provinceId=1http://www.yihaodian.com/cmsPage/show.do?pageId=48963&provinceId=13SKAPHD3JZYH9EE9ACB1NGA9VDQHNJMX1NY9TPPG4SWG71358HGRJGQHQQBXY9GF96CVU2225501\N124.79.172.232msessionid:YR9H5YU7RZ8Y94EBJNZ2P5W8DT37Q9JH,uname:chen45311,unionKey:22255012013-07-21 09:37:04134885852http://www.baidu.com/s?wd=1%E5%8F%B7%E5%BA%97&rsv_bp=0&ch=&tn=baidu&bar=&rsv_spt=3&ie=utf-8&rsv_sug3=5&rsv_sug=0&rsv_sug1=4&rsv_sug4=313&inputT=423520\N1-10Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; MATP; Media Center PC 6.0; .NET4.0C; InfoPath.2; .NET4.0E)Win32cms_pic_48963_542760_0上海市1上海市66\N\N\N\N2013-07-21
20947273602https://passport.yihaodian.com/passport/login_input.dohttp://www.yihaodian.com/1/?tracker_u=2225501&type=33SKAPHD3JZYH9EE9ACB1NGA9VDQHNJMX1NY9TPPG4SWG71358HGRJGQHQQBXY9GF96CVU2225501\N124.79.172.232msessionid:YR9H5YU7RZ8Y94EBJNZ2P5W8DT37Q9JH,unionKey:22255012013-07-21 09:30:11\Nhttp://www.baidu.com/s?wd=1%E5%8F%B7%E5%BA%97&rsv_bp=0&ch=&tn=baidu&bar=&rsv_spt=3&ie=utf-8&rsv_sug3=5&rsv_sug=0&rsv_sug1=4&rsv_sug4=313&inputT=42353\N1-10Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; MATP; Media Center PC 6.0; .NET4.0C; InfoPath.2; .NET4.0E)Win32YHD_TOP_login上海市1上海市66\N\N\N\N2013-07-21
20947308318http://www.yihaodian.com/marketing/allproduct.htmlhttp://www.yihaodian.com/item/user/continueShopping.do?ref=detail_continue_shopping_B-1_1_52_search.ctg_13SKAPHD3JZYH9EE9ACB1NGA9VDQHNJMX1NY9TPPG4SWG71358HGRJGQHQQBXY9GF96CVU2225501\N124.79.172.232msessionid:YR9H5YU7RZ8Y94EBJNZ2P5W8DT37Q9JH,uname:chen45311,unionKey:22255012013-07-21 09:34:59134885852http://www.baidu.com/s?wd=1%E5%8F%B7%E5%BA%97&rsv_bp=0&ch=&tn=baidu&bar=&rsv_spt=3&ie=utf-8&rsv_sug3=5&rsv_sug=0&rsv_sug1=4&rsv_sug4=313&inputT=423516\N1-10Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; MATP; Media Center PC 6.0; .NET4.0C; InfoPath.2; .NET4.0E)Win32YHD_TOP_index_5上海市1上海市66\N\N\N\N2013-07-21
20948276566http://www.yihaodian.com/checkoutV3/index.dohttp://www.yihaodian.com/cart/cart.do?action=view3SKAPHD3JZYH9EE9ACB1NGA9VDQHNJMX1NY9TPPG4SWG71358HGRJGQHQQBXY9GF96CVU2225501\N124.79.172.232msessionid:YR9H5YU7RZ8Y94EBJNZ2P5W8DT37Q9JH,uname:chen45311,unionKey:22255012013-07-21 09:50:43134885852http://www.baidu.com/s?wd=1%E5%8F%B7%E5%BA%97&rsv_bp=0&ch=&tn=baidu&bar=&rsv_spt=3&ie=utf-8&rsv_sug3=5&rsv_sug=0&rsv_sug1=4&rsv_sug4=313&inputT=423567\N1-10Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; MATP; Media Center PC 6.0; .NET4.0C; InfoPath.2; .NET4.0E)Win32mobile上海市1上海市66\N\N\N\N2013-07-21

电商项目实战-浏览量统计功能实现3_ide