python - Pin down exact content location in html for web scraping urllib2 Beautiful Soup -

-

i'm new web scraping, have little exposure html file systems , wanted know if there better more efficient way search required content on html version of web page. currently, want scrape reviews product here: http://www.walmart.com/ip/29701960?wmlspartner=wlpa&adid=22222222227022069601&wl0=&wl1=g&wl2=c&wl3=34297254061&wl4=&wl5=pla&wl6=62272156621&veh=sem

for this, have following code:

url = http://www.walmart.com/ip/29701960? wmlspartner=wlpa&adid=22222222227022069601&wl0=&wl1=g&wl2=c&wl3=34297254061&wl4=&wl5=pla&wl6=6227215 6621&veh=sem review_url = url        #print review_url     #-------------------------------------------------------------------------     # scrape ratings     #-------------------------------------------------------------------------     page_no = 1     sum_total_reviews = 0     more = true      while (more):         #print "xxxx"         # open url review data         request = urllib2.request(review_url)         try:             #print "xxxx"             page = urllib2.urlopen(request)         except urllib2.urlerror, e:             #print "xxxxx"             if hasattr(e, 'reason'):                 print 'failed reach url'                 print 'reason: ', e.reason                 sys.exit()             elif hasattr(e, 'code'):                 if e.code == 404:                     print 'error: ', e.code                     sys.exit()          content = page.read()         #print content         soup = beautifulsoup(content)         results = soup.find_all('span', {'class': re.compile(r's_star_\d_0')})

with this, i'm not able read anything. i'm guessing have give accurate destination. suggestions ?

i understand question beautifulsoup, since haven't had success using in particular situation, suggest taking @ selenium.

selenium uses real browser - don't have deal parsing results of ajax calls. example, here's how can list of review titles , ratings first reviews page:

from selenium.webdriver.firefox import webdriver   driver = webdriver.webdriver() driver.get('http://www.walmart.com/ip/29701960?page=seeallreviews')  review in driver.find_elements_by_class_name('bvrrreviewdisplaystyle3main'):     title = review.find_element_by_class_name('bvrrreviewtitle').text     rating = review.find_element_by_xpath('.//div[@class="bvrrratingnormalimage"]//img').get_attribute('title')     print title, rating  driver.close()

prints:

renee culver loves clorox wipes 5 out of 5 men @ work 5 out of 5 clorox wipes 5 out of 5 ...

also, take account can use headless phantomjs browser (example).

another option make use of walmart api.

hope helps.


Comments

Post a Comment

Popular posts from this blog

php - Magento - Deleted Base url key -

-
i working on cleaning url: http://mydomain.com/shop/shop/deals i going through settings in configuration tab in admin panel on magento , change secure , unsecure base urls remove first base url "shop" web page read: http://mydomain.com/shop/deals so changed unsecure base urls mydomain.com/shop mydomain.com , secure 12.345.67.89/shop 12.345.67.89. wrong thing do. i cannot reload page or admin console. how access these files through ftp , change correctly there?? when make changes in admin panel changes made in database not in file. have make changes in database. 1.go database , select core_config_data table under table find web/unsecure/base_url , web/secure/base_url . make changes here. don't forget clear cache in var/cache folder
Read more

java - WrongTypeOfReturnValue exception thrown when unit testing using mockito -

-
my test list<person> mylist; @test public void testisvalidperson() { mylist = new arraylist<person>(); mylist.add(new person("tom")); when(persondao.get(person)).thenreturn(mylist); when((persondao.get(person)).isempty()).thenreturn(false);//------exception thrown boolean result = service.isvalid("tom"); assertfalse(result); } method tested: public boolean isvalid(string person){ persondao = new persondao(); person personobj = new person(person); return (persondao.get(person).isempty())?false : true; } exception thrown: org.mockito.exceptions.misusing.wrongtypeofreturnvalue: boolean cannot returned get() get() should return list *** if you're unsure why you're getting above error read on. due nature of syntax above problem might occur because: 1. exception *might* occur in wrongly written multi-threaded tests. please refer mockito faq on limitations of concurrency testing. 2. sp...
Read more

javascript - Tooltipster plugin not firing jquery function when button or any click even occur -

-
i want use tooltipster plugin click element in body show tooltip , want have close button inside tooltip close it. however use jquery click() function handle won't fired. tried solution in post. worked when tooltip triggered hover event. tooltipster plugin not firing jquery function when link in in tooltip clicked the original solution using hover in jsfiddle trigger: 'hover', instead using click show tooltip trigger: 'click', http://jsfiddle.net/bcqyl/7/ it won't fire click event in jquery block. can captured normal javascript function only. checked should code in tooltipster plugin locked , captured "click()" event inside tips when in click trigger mode. i tried other event onchange event of radio button fired when using code in original solution @evan right. tooltip doesn't exist in dom , can't bind/delegate action (even if go way $(document) after domready). if you're after close button inside of too...
Read more