This project aims to develop an automatic agent that scrapes Amazon orders, retrieving their details in a structured format. The initial focus is on Amazon Egypt due to the availability of existing orders for testing.
-
Initial Manual Process:
- Perform the entire data scraping process manually to understand each step thoroughly.
- This includes reading library documentation, understanding and debugging code errors, reading HTML contents of the pages, and interacting with page elements.
-
Incremental Automation:
- Abstract and automate each step identified in the manual process.
- Ensure the agent can handle various challenges, such as slow internet speeds and handling ads or popups.
-
Quality and Adaptability:
- Develop the agent with qualities such as robust abstraction of manual steps and adaptability to handle unexpected scenarios.
- Envision a "super agent" that can complete tasks with minimal input, such as "scrape Amazon orders using the following username and password."
- Agent 1: Generates Selenium code to fetch product details based on detailed instructions.
- Agent 2: Extracts data using Beautiful Soup code, also based on detailed instructions.
- Agent 1: Executes Selenium code multiple times, incorporating feedback from the website to improve performance.
- Agent 2: Explores HTML patterns to generate more efficient Beautiful Soup code.
- Agent 1 and Agent 2: Operate based on more general instructions, reducing the need for detailed guidance.
- Combine the functionalities of the two agents into a single, highly autonomous agent that can perform the entire scraping process with minimal input.
- Python 3.x
- Selenium
- WebDriver for your preferred browser (e.g., ChromeDriver)
-
Clone the repository:
git clone https://github.com/bely66/amazon-orders-scraper.git cd amazon-orders-scraper
-
Install the required packages:
pip install -r requirements.txt
-
Create .env file and add the following (I chose to use the .env file to store the API key and the Amazon credentials safely):
OPENAI_API_KEY='openai_api_key' # ask the user about their username and password amz_mail="amazon_email" amz_pass="amazon_password"
-
Run the manual scraping script to understand the process:
python login_amazon.py
-
Follow the instructions to navigate and extract order details manually.
The agent don't have access to the passwords, but has access to the variables containing them, this way the data isn't sent to openai's servers.
Run the automated agent script:
```bash
python main.py
```
- Add a way to scroll through all the products using the years filter.
- Extract the properties of all divs and send it to the Structurer agent to generate the suitable Beautiful Soup code.
- Add retrying mechanism while sending the trace of the mistakes to handle any code issues.
- Rewrite the prompt to be more generic, in the sense it responds to code outputs rather than only writing code.
- Find a way to have a brief overview of elements on the page in Selenium or BeautifulSoup so the agent can have the needed information to make an informed decision without exceeding the maximum length as HTMLs could be very long.
- It's advisable we get to the wanted performance from a very good model.
- Use the logs and documentation we have from that model to train a smaller model (a good Python model to be fine-tuned on Selenium and BeautifulSoup).
- We'll need to experiment with the model size but generally a 13b quantized model trained with LORA in 4bit will be perfect in terms of both accuracy, memory requirements, flexibility, and speed.
- Incorporate memory so that the agent can use past actions to learn