hadoop - Merging MapReduce output -

-

i have 2 mapreduce jobs produce files in 2 separate directories so:

 directory output1:  ------------------  /output/20140102-r-00000.txt  /output/20140102-r-00000.txt  /output/20140103-r-00000.txt  /output/20140104-r-00000.txt   directory output2:  ------------------  /output-update/20140102-r-00000.txt

i want merge these 2 directories in new directory /output-complete/ 20140102-r-00000.txt replaces original file in /output directory , of "-r-0000x" removed file name. 2 original directories empty , resulting directory should follows:

 directory output3:  -------------------  /output-complete/20140102.txt  /output-complete/20140102.txt  /output-complete/20140103.txt  /output-complete/20140104.txt

what best way this? can use hdfs shell commands? need create java program traverse both directories , logic?

you can use pig ...

get_data = load '/output*/20140102*.txt' using loader() store get_data "/output-complete/20140102.txt"

or hdfs command...

hadoop fs -cat '/output*/20140102*.txt' > output-complete/20140102.txt

single qoutes may not work, try double quotes


Comments

Post a Comment

Popular posts from this blog

java - WrongTypeOfReturnValue exception thrown when unit testing using mockito -

-
my test list<person> mylist; @test public void testisvalidperson() { mylist = new arraylist<person>(); mylist.add(new person("tom")); when(persondao.get(person)).thenreturn(mylist); when((persondao.get(person)).isempty()).thenreturn(false);//------exception thrown boolean result = service.isvalid("tom"); assertfalse(result); } method tested: public boolean isvalid(string person){ persondao = new persondao(); person personobj = new person(person); return (persondao.get(person).isempty())?false : true; } exception thrown: org.mockito.exceptions.misusing.wrongtypeofreturnvalue: boolean cannot returned get() get() should return list *** if you're unsure why you're getting above error read on. due nature of syntax above problem might occur because: 1. exception *might* occur in wrongly written multi-threaded tests. please refer mockito faq on limitations of concurrency testing. 2. sp
Read more

php - Magento - Deleted Base url key -

-
i working on cleaning url: http://mydomain.com/shop/shop/deals i going through settings in configuration tab in admin panel on magento , change secure , unsecure base urls remove first base url "shop" web page read: http://mydomain.com/shop/deals so changed unsecure base urls mydomain.com/shop mydomain.com , secure 12.345.67.89/shop 12.345.67.89. wrong thing do. i cannot reload page or admin console. how access these files through ftp , change correctly there?? when make changes in admin panel changes made in database not in file. have make changes in database. 1.go database , select core_config_data table under table find web/unsecure/base_url , web/secure/base_url . make changes here. don't forget clear cache in var/cache folder
Read more

android - How to disable Button if EditText is empty ? -

-
i have edittext , button in application. when button clicked,the text entered in edittext added listview. i want disable button if edittext empty.how ? this code button click imagebutton imb=(imagebutton)findviewbyid(r.id.btn_send); imb.setonclicklistener(new onclicklistener() { @override public void onclick(view arg0) { edittext et = (edittext)findviewbyid(r.id.edittext1); string str = et.gettext().tostring(); web1.add(str); toast.maketext(shoutsingleprogram.this, "you entered...."+str, toast.length_short).show(); adapter1.notifydatasetchanged(); et.settext(""); } }); } how can ? edittext1.addtextchangedlistener(new textwatcher() { @override public void ontextchanged(charsequence s, int start, int before, in
Read more