Scraper throws error instead of pulling values from a webpage

[psf]

Scraper throws error instead of pulling values from a webpage

[kennethreitz]

Your script runs fine for me!

import requests_html

with requests_html.HTMLSession() as session:
    r = session.get('https://www.gdax.com/trade/LTC-EUR')
    r.html.render()
    print(r.html.find('.MarketInfo_market-num_1lAXs', first=True).text)

Result:

169.00 EUR

Slightly modified. Pyppeteer might not work on Windows. I'll have to test.

[kennethreitz]

Oh, make sure you're using python 3.6!

Here is the full traceback, sir. Link to the traceback "https://www.dropbox.com/s/40ohohrte9keplo/traceback.txt?dl=0". I know it's hard to give any solution hypothetically. However, if I get any clue why I'm facing such error then I might take any initiative to resolve that. I've already created a thread in StackOverflow to get any solution for but it seems you have released it very lately that is why few peoples are aware of it.

[kennethreitz]

Well pyppeteer requires python 3.6.

Thanks for the quick turnaround, sir. I'm gonna upgrade the version. It was an immense pleasure to have any solution from the author himself. I may get back if I face any issue after running the script when my version is update. Thanks sir.

[kennethreitz]

I'm testing on Windows as we speak.

[kennethreitz]

Windows works great! You just have to use Python 3.6.

[kennethreitz]

(You're using Python 3.5)

It's a revolutionary attempt. This library will make a new era undoubtedly.

[kennethreitz]

Awe, thank you :)

Do give Python 3.6 a try — it should "just work".

Hi again, sir!! I've already upgraded my Python version to 3.6.4. I'm on windows 7, 32 bit. When I execute the above script, I get a different traceback now. Here goes the full error. Thanks in advance, sir.

Exception in callback NavigatorWatcher.waitForNavigation.<locals>.watchdog_cb(<Task finishe...> result=None>) at C:\Users\ar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pyppeteer\navigator_watcher.py:49
handle: <Handle NavigatorWatcher.waitForNavigation.<locals>.watchdog_cb(<Task finishe...> result=None>) at C:\Users\ar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pyppeteer\navigator_watcher.py:49>
Traceback (most recent call last):
  File "C:\Users\ar\AppData\Local\Programs\Python\Python36-32\lib\asyncio\events.py", line 145, in _run
    self._callback(*self._args)
  File "C:\Users\ar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pyppeteer\navigator_watcher.py", line 52, in watchdog_cb
    self._timeout)
  File "C:\Users\ar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pyppeteer\navigator_watcher.py", line 40, in _raise_error
    raise error
concurrent.futures._base.TimeoutError: Navigation Timeout Exceeded: 3000 ms exceeded
Traceback (most recent call last):
  File "C:\Users\ar\AppData\Local\Programs\Python\Python36-32\demo.py", line 13, in <module>
    item = js.find('.MarketInfo_market-num_1lAXs',first=True).text
AttributeError: 'NoneType' object has no attribute 'find'
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\Users\ar\AppData\Local\Programs\Python\Python36-32\lib\shutil.py", line 387, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\ar\\.pyppeteer\\.dev_profile\\tmp2hnn897d\\CrashpadMetrics-active.pma'

[kennethreitz]

you might want to run it in an administrator mode cmd.exe

I've tried that as well, sir. Having the same error.

[kennethreitz]

try removing ~/.pyppeteer

[kennethreitz]

not sure what's wrong! no issues here.

I did a slight modification in my script to fetch the data. Now, I can get the desired content. However, along with the data I'm getting several errors. Take a look below. I have already marked the result with a marker. How can I kick out errors?

Modified script:

import requests_html

with requests_html.HTMLSession() as session:
    r = session.get('https://www.gdax.com/trade/LTC-EUR')
    r.html.render(sleep=10)
    item = r.html.find("[class^='MarketInfo_market-num_']",first=True).text
    print(item)

Error along with result (take a look at the long dot which indicates the value) :

Exception in callback NavigatorWatcher.waitForNavigation.<locals>.watchdog_cb(<Task finishe...> result=None>) at C:\Users\ar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pyppeteer\navigator_watcher.py:49
handle: <Handle NavigatorWatcher.waitForNavigation.<locals>.watchdog_cb(<Task finishe...> result=None>) at C:\Users\ar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pyppeteer\navigator_watcher.py:49>
Traceback (most recent call last):
  File "C:\Users\ar\AppData\Local\Programs\Python\Python36-32\lib\asyncio\events.py", line 145, in _run
    self._callback(*self._args)
  File "C:\Users\ar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pyppeteer\navigator_watcher.py", line 52, in watchdog_cb
    self._timeout)
  File "C:\Users\ar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pyppeteer\navigator_watcher.py", line 40, in _raise_error
    raise error
concurrent.futures._base.TimeoutError: Navigation Timeout Exceeded: 3000 ms exceeded
167.69 EUR         ====================================================
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\Users\ar\AppData\Local\Programs\Python\Python36-32\lib\shutil.py", line 387, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\ar\\.pyppeteer\\.dev_profile\\tmp4m18xik8\\CrashpadMetrics-active.pma'

Except for parsing javascript content, the library is awesome when it comes to play with css selector or xpath. It even supports pseudo selectors that BeautifulSoup could not. lxml library supports pseudo selectors, though.