王炸科技头像

HadoopHelloWordExamples

来源:
       

  ?另外一个hadoop的入门demo,求平均数。是对WordCount这个demo的一个小小的修改。输入一堆成绩单(人名,成绩),然后求每个人成绩平均数,比如://?subject1.txt?a90?b80?c70?//subject2.txt?a100?b90?c80?求a,b,c这三个人的平均

  ?另外一个hadoop的入门demo,求平均数。是对WordCount这个demo的一个小小的修改。输入一堆成绩单(人名,成绩),然后求每个人成绩平均数,比如:

  //?subject1.txt

  ?a90

  ?b80

  ?c70

  ?//subject2.txt

  ?a100

  ?b90

  ?c80

  ?求a,b,c这三个人的平均分。解决思路很简单,在map阶段key是名字,value是成绩,直接output。reduce阶段得到了map输出的key名字,values是该名字对应的一系列的成绩,那么对其求平均数即可。

  ?这里我们实现了两个版本的代码,分别用TextInputFormat和KeyValueTextInputFormat来作为输入格式。

  ?TextInputFormat版本:

  ?

  importjava.util.;importjava.io.;importorg.apache.hadoop.conf.Configuration;importorg.apache.hadoop.fs.Path;importorg.apache.hadoop.io.Text;importorg.apache.hadoop.io.IntWritable;importorg.apache.hadoop.mapreduce.Mapper;importorg.apache.hadoop.mapreduce.Reducer;importorg.apache.hadoop.mapreduce.Job;importorg.apache.hadoop.mapreduce.lib.input.FileInputFormat;importorg.apache.hadoop.mapreduce.lib.output.FileOutputFormat;publicclassAveScore{publicstaticclassAveMapperextendsMapper{@Overridepublicvoidmap(Objectkey,Textvalue,Contextcontext)throwsIOException,InterruptedException{Stringline=value.toString();String[]strs=line.split("");Stringname=strs[0];intscore=Integer.parseInt(strs[1]);context.write(newText(name),newIntWritable(score));}}publicstaticclassAveReducerextendsReducer{@Overridepublicvoidreduce(Textkey,Iterablevalues,Contextcontext)throwsIOException,InterruptedException{intsum=0;intcount=0;for(IntWritableval:values){sum+=val.get();count++;}intaveScore=sum/count;context.write(key,newIntWritable(aveScore));}}publicstaticvoidmain(String[]args)throwsException{ConfigurationcOnf=newConfiguration();Jobjob=newJob(conf,"AverageScore");job.setJarByClass(AveScore.class);job.setMapperClass(AveMapper.class);job.setReducerClass(AveReducer.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class);FileInputFormat.addInputPath(job,newPath(args[0]));FileOutputFormat.setOutputPath(job,newPath(args[1]));System.exit(job.waitForCompletion(true)?0:1);}}

  KeyValueTextInputFormat版本;

  importjava.util.;importjava.io.;importorg.apache.hadoop.conf.Configuration;importorg.apache.hadoop.fs.Path;importorg.apache.hadoop.io.Text;importorg.apache.hadoop.io.IntWritable;importorg.apache.hadoop.mapreduce.Mapper;importorg.apache.hadoop.mapreduce.Reducer;importorg.apache.hadoop.mapreduce.Job;importorg.apache.hadoop.mapreduce.lib.input.FileInputFormat;importorg.apache.hadoop.mapreduce.lib.output.FileOutputFormat;importorg.apache.hadoop.mapreduce.lib.input.KeyValueTextInputFormat;importorg.apache.hadoop.mapreduce.lib.output.TextOutputFormat;publicclassAveScore_KeyValue{publicstaticclassAveMapperextendsMapper{@Overridepublicvoidmap(Textkey,Textvalue,Contextcontext)throwsIOException,InterruptedException{intscore=Integer.parseInt(value.toString());context.write(key,newIntWritable(score));}}publicstaticclassAveReducerextendsReducer{@Overridepublicvoidreduce(Textkey,Iterablevalues,Contextcontext)throwsIOException,InterruptedException{intsum=0;intcount=0;for(IntWritableval:values){sum+=val.get();count++;}intaveScore=sum/count;context.write(key,newIntWritable(aveScore));}}publicstaticvoidmain(String[]args)throwsException{ConfigurationcOnf=newConfiguration();conf.set("mapreduce.input.keyvaluelinerecordreader.key.value.separator","");Jobjob=newJob(conf,"AverageScore");job.setJarByClass(AveScore_KeyValue.class);job.setMapperClass(AveMapper.class);job.setReducerClass(AveReducer.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class);job.setInputFormatClass(KeyValueTextInputFormat.class);job.setOutputFormatClass(TextOutputFormat.class);FileInputFormat.addInputPath(job,newPath(args[0]));FileOutputFormat.setOutputPath(job,newPath(args[1]));System.exit(job.waitForCompletion(true)?0:1);}}

  输出结果为:

  ?a95

  ?b85

  ?c75

  ?

  作者:qiul12345发表于2013-8-2321:51:03原文链接

  阅读:113评论:0查看评论

HadoopHelloWordExamples-第1张图片

  原文地址:HadoopHelloWordExamples-求平均数,感谢原作者分享。