tilman151/scrape-my-bike

Repository to scrape eBay Kleinanzeigen for bikes

scrape-my-bike's Issues

Consider using native requests

Hey,
Awesome project! Just an idea: using native requests instead of selenium saves lots of CPU and RAM.

Ebay Kleinanzeigen is rather unprotected - theres only an user agent filter, which can be bypassed easely (if in blacklist, shows message that IP range has been banned - dont get confused...). That way its easy to access the page - even without js capabilities.

Small concept below:

import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Opera/8.96 (Windows NT 5.01; sl-SI) Presto/2.9.199 Version/11.00"
}

r = requests.get("https://www.ebay-kleinanzeigen.de/s-berlin/fahrrad/k0l3331", headers=headers)


soup = BeautifulSoup(r.text, "html.parser")

for i in soup.find_all(attrs={"class": "lazyload-item"}):
    print(i)

In the html, you can also find things as the description, or, via the class, also detect wether its an ad or top.

I can open a pull request tomorrow (maybe).