python - Pin down exact content location in html for web scraping urllib2 Beautiful Soup -

-

i'm new web scraping, have little exposure html file systems , wanted know if there better more efficient way search required content on html version of web page. currently, want scrape reviews product here: http://www.walmart.com/ip/29701960?wmlspartner=wlpa&adid=22222222227022069601&wl0=&wl1=g&wl2=c&wl3=34297254061&wl4=&wl5=pla&wl6=62272156621&veh=sem

for this, have following code:

url = http://www.walmart.com/ip/29701960? wmlspartner=wlpa&adid=22222222227022069601&wl0=&wl1=g&wl2=c&wl3=34297254061&wl4=&wl5=pla&wl6=6227215 6621&veh=sem review_url = url        #print review_url     #-------------------------------------------------------------------------     # scrape ratings     #-------------------------------------------------------------------------     page_no = 1     sum_total_reviews = 0     more = true      while (more):         #print "xxxx"         # open url review data         request = urllib2.request(review_url)         try:             #print "xxxx"             page = urllib2.urlopen(request)         except urllib2.urlerror, e:             #print "xxxxx"             if hasattr(e, 'reason'):                 print 'failed reach url'                 print 'reason: ', e.reason                 sys.exit()             elif hasattr(e, 'code'):                 if e.code == 404:                     print 'error: ', e.code                     sys.exit()          content = page.read()         #print content         soup = beautifulsoup(content)         results = soup.find_all('span', {'class': re.compile(r's_star_\d_0')})

with this, i'm not able read anything. i'm guessing have give accurate destination. suggestions ?

i understand question beautifulsoup, since haven't had success using in particular situation, suggest taking @ selenium.

selenium uses real browser - don't have deal parsing results of ajax calls. example, here's how can list of review titles , ratings first reviews page:

from selenium.webdriver.firefox import webdriver   driver = webdriver.webdriver() driver.get('http://www.walmart.com/ip/29701960?page=seeallreviews')  review in driver.find_elements_by_class_name('bvrrreviewdisplaystyle3main'):     title = review.find_element_by_class_name('bvrrreviewtitle').text     rating = review.find_element_by_xpath('.//div[@class="bvrrratingnormalimage"]//img').get_attribute('title')     print title, rating  driver.close()

prints:

renee culver loves clorox wipes 5 out of 5 men @ work 5 out of 5 clorox wipes 5 out of 5 ...

also, take account can use headless phantomjs browser (example).

another option make use of walmart api.

hope helps.


Comments

Post a Comment

Popular posts from this blog

java - WrongTypeOfReturnValue exception thrown when unit testing using mockito -

-
my test list<person> mylist; @test public void testisvalidperson() { mylist = new arraylist<person>(); mylist.add(new person("tom")); when(persondao.get(person)).thenreturn(mylist); when((persondao.get(person)).isempty()).thenreturn(false);//------exception thrown boolean result = service.isvalid("tom"); assertfalse(result); } method tested: public boolean isvalid(string person){ persondao = new persondao(); person personobj = new person(person); return (persondao.get(person).isempty())?false : true; } exception thrown: org.mockito.exceptions.misusing.wrongtypeofreturnvalue: boolean cannot returned get() get() should return list *** if you're unsure why you're getting above error read on. due nature of syntax above problem might occur because: 1. exception *might* occur in wrongly written multi-threaded tests. please refer mockito faq on limitations of concurrency testing. 2. sp
Read more

php - Magento - Deleted Base url key -

-
i working on cleaning url: http://mydomain.com/shop/shop/deals i going through settings in configuration tab in admin panel on magento , change secure , unsecure base urls remove first base url "shop" web page read: http://mydomain.com/shop/deals so changed unsecure base urls mydomain.com/shop mydomain.com , secure 12.345.67.89/shop 12.345.67.89. wrong thing do. i cannot reload page or admin console. how access these files through ftp , change correctly there?? when make changes in admin panel changes made in database not in file. have make changes in database. 1.go database , select core_config_data table under table find web/unsecure/base_url , web/secure/base_url . make changes here. don't forget clear cache in var/cache folder
Read more

android - How to disable Button if EditText is empty ? -

-
i have edittext , button in application. when button clicked,the text entered in edittext added listview. i want disable button if edittext empty.how ? this code button click imagebutton imb=(imagebutton)findviewbyid(r.id.btn_send); imb.setonclicklistener(new onclicklistener() { @override public void onclick(view arg0) { edittext et = (edittext)findviewbyid(r.id.edittext1); string str = et.gettext().tostring(); web1.add(str); toast.maketext(shoutsingleprogram.this, "you entered...."+str, toast.length_short).show(); adapter1.notifydatasetchanged(); et.settext(""); } }); } how can ? edittext1.addtextchangedlistener(new textwatcher() { @override public void ontextchanged(charsequence s, int start, int before, in
Read more