小編給大家分享一下NLineInputFormat的示例分析,希望大家閱讀完這篇文章之后都有所收獲,下面讓我們一起去探討吧!
成都創(chuàng)新互聯(lián)是專業(yè)的伊通網(wǎng)站建設(shè)公司,伊通接單;提供網(wǎng)站設(shè)計制作、成都網(wǎng)站制作,網(wǎng)頁設(shè)計,網(wǎng)站設(shè)計,建網(wǎng)站,PHP網(wǎng)站建設(shè)等專業(yè)做網(wǎng)站服務(wù);采用PHP框架,可快速的進行伊通網(wǎng)站開發(fā)網(wǎng)頁制作和功能擴展;專業(yè)做搜索引擎喜愛的網(wǎng)站,專業(yè)的做網(wǎng)站團隊,希望更多企業(yè)前來合作!
package com.test; import java.io.IOException; import java.util.Iterator; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.NLineInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; import org.apache.hadoop.util.Tool; import org.apache.hadoop.util.ToolRunner; /** * TextInputFormat處理的數(shù)據(jù)來自于一個InputSplit。InputSplit根據(jù)塊大小劃分。 * 由于每條記錄有長有短,所以,每個map任務(wù)處理的記錄數(shù)都不一樣 * NLineInputFormat決定每個map處理記錄數(shù)是相同的 */ public class WordCountNL extends Configured implements Tool { public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> { public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); StringTokenizer st = new StringTokenizer(line); while(st.hasMoreElements()) { context.write(new Text(st.nextElement().toString()), new IntWritable(1)); } } } public static class Combiner extends Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int count = 0; Iterator<IntWritable> it = values.iterator(); while(it.hasNext()) { count = count + it.next().get(); } context.write(key, new IntWritable(count)); } } public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int count = 0; Iterator<IntWritable> it = values.iterator(); while(it.hasNext()) { count = count + it.next().get(); } context.write(key, new IntWritable(count)); } } public int run(String[] args) throws IOException, InterruptedException, ClassNotFoundException { Configuration conf = this.getConf(); //設(shè)置每個map可以處理多少行數(shù)據(jù) //conf.set("mapreduce.input.lineinputformat.linespermap", "1"); conf.set(NLineInputFormat.LINES_PER_MAP, "1"); Job job = new Job(conf); job.setJobName(WordCountNL.class.getSimpleName()); job.setJarByClass(WordCountNL.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); job.setMapperClass(Map.class); job.setCombinerClass(Combiner.class); job.setReducerClass(Reduce.class); job.setInputFormatClass(NLineInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); job.waitForCompletion(true); return job.isSuccessful()?0:1; } public static void main(String[] args) throws Exception { int exit = ToolRunner.run(new WordCount(), args); System.exit(exit); } }
看完了這篇文章,相信你對“NLineInputFormat的示例分析”有了一定的了解,如果想了解更多相關(guān)知識,歡迎關(guān)注創(chuàng)新互聯(lián)行業(yè)資訊頻道,感謝各位的閱讀!
分享文章:NLineInputFormat的示例分析
分享URL:http://muchs.cn/article28/jpgsjp.html
成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián),為您提供建站公司、動態(tài)網(wǎng)站、、網(wǎng)站內(nèi)鏈、網(wǎng)站維護、移動網(wǎng)站建設(shè)
聲明:本網(wǎng)站發(fā)布的內(nèi)容(圖片、視頻和文字)以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主,如果涉及侵權(quán)請盡快告知,我們將會在第一時間刪除。文章觀點不代表本網(wǎng)站立場,如需處理請聯(lián)系客服。電話:028-86922220;郵箱:631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載,或轉(zhuǎn)載時需注明來源: 創(chuàng)新互聯(lián)