Comments (14)
Hi NKoenig06, I was also facing this "selenium.common.exceptions.NoSuchElementException" error at times! In my experience with running this script you don't have to change sleep time(googling this error does show this type of solutions) or any part of the code instead just try after some time after closing everything out. For me it ran smoothly!!
from glassdoor-review-scraper.
Let me know if you want to scrap reviews for a specific company and will share with you..
from glassdoor-review-scraper.
The issue seems to be that glassdoor throws another log-in at you. As a quick work-around I just set the sleep time to 40 seconds before this step and logged in manually, data extraction worked perfectly afterwards.
Always happens after the line:
browser.get(args.url)
from glassdoor-review-scraper.
Interesting, it's not breaking there for me. I'm always getting to the following point:
- configuring browser
- Scraping up to 10 reviews
- Signing in to "username"
- Navigating to company reviews
- Extracting reviews from page 1
- Found 10 reviews on page 1
Then this error below. It seems to be saying it's not recognizing the method or selector. I made adjustments to the sleep time like you suggested and it didn't change me continuing to get stuck at the same spot despite trying multiple reviews pages.
Traceback (most recent call last):
File "main.py", line 461, in
main()
File "main.py", line 441, in main
reviews_df = extract_from_page()
File "main.py", line 295, in extract_from_page
data = extract_review(review)
File "main.py", line 281, in extract_review
res[field] = scrape(field, review, author)
File "main.py", line 264, in scrape
return fdictfield
File "main.py", line 156, in scrape_years
'reviewBodyCell').find_element_by_tag_name('p')
File "/home/nick/miniconda3/envs/scraping/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py", line 305, in find_element_by_tag_name
return self.find_element(by=By.TAG_NAME, value=name)
File "/home/nick/miniconda3/envs/scraping/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py", line 659, in find_element
{"using": by, "value": value})['value']
File "/home/nick/miniconda3/envs/scraping/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py", line 633, in _execute
return self._parent.execute(command, params)
File "/home/nick/miniconda3/envs/scraping/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/home/nick/miniconda3/envs/scraping/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"tag name","selector":"p"}
(Session info: headless chrome=71.0.3578.98)
(Driver info: chromedriver=2.45.615279 (12b89733300bd268cff3b78fc76cb8f3a7cc44e5),platform=Linux 4.15.0-46-generic x86_64)
from glassdoor-review-scraper.
It could be that glassdoor only requests your login after you want to switch to the next page. But in my experience the important thing is, that you log in a second time before you start scratching. The first time referring to the scripted log-in.
from glassdoor-review-scraper.
Hey all, sorry I don't have time to look into this right now. But it sounds like you're finding you're way pretty well!
Without having done any investigation, my guess is that Glassdoor changed either the HTML structure of the site or their login flow, which is causing these errors. If we can diagnose the precise cause and get a PR for it, I'll approve it!
from glassdoor-review-scraper.
Sounds like this has not been able to be reproduced? If so I'll close the issue.
from glassdoor-review-scraper.
I've experienced this same issue, but it's not always a problem. Doesn't seem to happen every time.
from glassdoor-review-scraper.
Replacing some line of codes helped me.
Original (3 places in the codes):
paging_control = browser.find_element_by_class_name('pagingControls')
Updated:
paging_control = browser.find_element_by_css_selector('.eiReviews__EIReviewsPageContainerStyles__pagination.noTabover.mt')
Original (2 places in the codes):
next_ = paging_control.find_element_by_class_name('next')
Updated:
next_ = paging_control.find_element_by_class_name('pagination__PaginationStyle__next')
from glassdoor-review-scraper.
It's looking like there may have been element changes either in Selenium or on Glassdoor.
I'm not completely familiar with Selenium, so I was wondering if someone had seen this issue;
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"tag name","selector":"p"}
same issue here.. tied above all, not working yet..
from glassdoor-review-scraper.
Let me know if you want to scrap reviews for a specific company and will share with you..
Hi, would you be able to scrap reviews for a specific company for me?
from glassdoor-review-scraper.
Let me know if you want to scrap reviews for a specific company and will share with you..
Hi, would you be able to scrap reviews for a specific company for me?
If you look at the html layout of Glassdoor a lot has changed. I think this specific repo would need to be updated to accommodate the html changes to continue working.
from glassdoor-review-scraper.
Let me know if you want to scrap reviews for a specific company and will share with you..
Hi, would you be able to scrap reviews for a specific company for me?
If you look at the html layout of Glassdoor a lot has changed. I think this specific repo would need to be updated to accommodate the html changes to continue working.
Hi Nick,
I have gotten pretty far with the code with some minor tweaks, but after I run the "main" function I get a "No Such Element Exception" that looks like this:
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".pagination__PaginationStyle__page.pagination__PaginationStyle__current"}
(Session info: chrome=86.0.4240.111)
I don't suppose you'd know a work around for this?
from glassdoor-review-scraper.
Hi @NKoenig06
I appreciate the tutorial you provided in your article posted online (https://nkoenig06.github.io/scrape-gd.html) on how to scrape online reviews. However, I am also encountering issues with the current repository that you linked. I think the specific issue is that Glassdoor keeps hitting with sign-up prompts, which the script is unable to address. This sounds similar to the issue you experienced previously, so I'm curious to know if you were able to resolve that?
If so, how?
Appreciate any help that you, or anyone else can provide!
from glassdoor-review-scraper.
Related Issues (20)
- --start_from_url NoSuchElementException HOT 2
- Error when running Example 1
- Error when running Example 1. Please help! HOT 2
- Getting a Syntax Error
- No Such Element Exception HOT 6
- Not pulling overall rating nor advice to management
- NoSuchElementException HOT 4
- Cloudflare anti-bot protection
- Issues with subratings
- Glassdoor new UI design HOT 2
- NoSuchElementException - {"method":"css selector","selector":"time"} HOT 4
- Credentials error despite passing them? HOT 1
- no such element: Unable to locate element: HOT 5
- The scraper seems to only pull the top 10 reviews regardless of how many pages it attempts to navigate to HOT 2
- NoSuchElementException - {"method":"css selector","selector":"time"}
- Getting error of element click intercepted
- Does not recognize URL as an argument
- Default decoder decode error
- Issue with ChromeDriver
- Scraper Utilization without Glassdoor Account?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from glassdoor-review-scraper.