mark-thompson/whole-foods-deliverance

[Availability notifications, auto-checkout, slot preferences, cart tracking] for Whole Foods / Amazon Fresh

whole-foods-deliverance's Issues

Track cart state (item names/IDs/prices)

Ideally, this would also keep track of items in your cart and keep a record of which items are removed due to stock constraints (either silently, or as a result of an OOS alert)
Price changes could also be tracked (#21)

question rather than issue

first of all great concept and execution ! the code runs fine on python 3.7

In amazon final screen, it says select a day : Today or Tomorrow.

does the code check windows only for today ? most likely Tomorrow has more chances to have openings so I'd like the code to check both days. Is this the case ?

Refactor

This is getting kind of ugly. What was once a simple script now isn't so simple and probably deserves a refactor (looking at you, utils.py) and consolidation of many of the WebElement related methods

Add Prime Now compatibility

Given the differences in page layouts, this will require more than a config change

GUI

Might be nice to have a graphical option for setting config as well as control flow for those who don't spend all day in fancy text editors or on the command line.

Add Amazon Fresh compatibility

This may be as simple as adding additional site config / routes in config.py

Document code

"I'll get around to it eventually..."

Windows nav backslash

venv did not seem to work creating a virtual environment on windows. I was able to get virtualenv to work.

Windows navigation should be .\env\Scripts\activate
not . env/Scripts/activate

Handle login redirects

When using stored cookie data, navigating to the slot selection page fails occasionally due to a login redirect.

Configurable payment options?

During the Fresh checkout process, Amazon presents me with three cards to choose from; one is from a family member who's on my Prime family account, one is my Amazon store card, and one is my credit card. Unlike the regular cart checkout, the Fresh checkout never remembers the last card I used, and automatically selects the first one in the list instead. Unfortunately, that's my family member's card, not mine.

It would be handy if the config file had an option for the last 4 digits of the card to use, and if the script made sure that radio button was selected.

selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 81

on windows, I had some trouble with this, the solution was:

pip install --upgrade chromedriver-binary

Then add to imports:
from webdriver_manager.chrome import ChromeDriverManager

And update:

log.info('Invoking Selenium Chrome webdriver')
driver = webdriver.Chrome()

to:

    log.info('Invoking Selenium Chrome webdriver')
    driver = webdriver.Chrome(ChromeDriverManager().install())

Parse delivery window values

• Make the notification message more readable
• Remove extraneous info and unavailable time slots

Auto-checkout: Large orders take longer to complete checkout - selected slot may be lost.

The script is working well and picks the slot. It presses continue on the payment methods page. However it throws and exception on the next page which is the final confirmation place order page.

Running Windows + Chrome 81

`INFO:main:Attempting to select slot and checkout
INFO:main:Selecting slot: Tomorrow::8:00 AM - 10:00 AM
INFO:nav:Navigating <Route beginning at 'https://www.amazon.com/gp/buy/shipoptionselect/handlers/display.html' with 3 stops>
INFO:nav:Navigating <Waypoint ('xpath', "//*[contains(@Class, 'ufss-overview-continue-button')]") -> 'gp/buy/payselect/handlers/display.html'>
INFO:nav:Navigated to 'gp/buy/payselect/handlers/display.html'
INFO:nav:Navigating <Waypoint ('id', 'continue-top') -> 'gp/buy/spc/handlers/display.html'>
WARNING:nav:Current URL 'https://www.amazon.com/gp/buy/payselect/handlers/display.html' does not match target
Handling possible redirect (timeout in 10s)
Traceback (most recent call last):
File "C:\Users\Hello\Desktop\whole-foods-deliverance\nav.py", line 96, in navigate
WebDriverWait(driver, timeout).until(
File "C:\Users\Hello\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\selenium\webdriver\support\wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "run.py", line 170, in
main_loop(driver, args)
File "run.py", line 144, in main_loop
site_config.Routes.CHECKOUT.navigate(driver)
File "C:\Users\Hello\Desktop\whole-foods-deliverance\nav.py", line 100, in navigate
raise e
File "C:\Users\Hello\Desktop\whole-foods-deliverance\nav.py", line 78, in navigate
self.navigate_waypoint(driver, waypoint, timeout)
File "C:\Users\Hello\Desktop\whole-foods-deliverance\nav.py", line 59, in navigate_waypoint
raise NavigationException(
nav.NavigationException: Message: Navigation to 'gp/buy/spc/handlers/display.html' failed
`

Donations / Support

Dude, set up a tip jar for donations. I can't be the only one saving a ton of stress by using this. I need a helpful way to thank you!

ARM alternative to chromedriver-binary?

The chromedriver-binary package only includes an x86_64 binary, so this can't be run on a Raspberry Pi or other ARM system.

/usr/local/lib/python3.6/dist-packages/chromedriver_binary/chromedriver: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, for GNU/Linux 3.2.0, BuildID[sha1]=59cd88aeed037aaa10f05efbd725908f3f982ea3, stripped

Is there an alternative?

Handle out of stock items

Navigating to the slot selection page occasionally fails due to an out of stock alert for one of the items in cart. This may also affect slot page refreshes.
Could handle this with a user alert and await input, or add a flag to ignore and continue

Does refreshing the slots page over and over really work?

I would expect that it doesn't work, and that Amazon would intentionally disable it to discourage the use of scripts like this. I've modified run.py line 140 to replace driver.refresh() with build_route(site_config, 'SLOT_SELECT').navigate(driver) to replicate what a person might do when retrying. I'll run that for a while and see if it works any better. (Several hours with just the refresh didn't get me any slots.)

This might be a good command-line option to provide, to choose the refresh mode.

Encountered error AFTER successful checkout

I've been running commit 0a83c6 most of the morning, with no issues. I'd seen it occasionally cycling through the cart pages, but most of the time it was just refreshing. My conf.toml is an exact copy of conf_template.toml, and the command options I used were:

python run.py --service "Amazon Fresh" --checkout --ignore-oos --save-cart

A few minutes ago, it started seeing open slots, and after a couple of attempts to grab one it managed to finish the checkout process. After that, I got the error below. Chrome is sitting on the "Thank you, your Amazon Fresh order has been placed" confirmation page, https://www.amazon.com/gp/buy/thankyou/handlers/display.html?...

This is on Win10, btw. I was getting the audio alert about the slots being available, but I don't think I got one to confirm the order.

[2020-04-22 14:16:36] {get_slots} INFO: Found 2 slots:
Today::8:00 PM - 10:00 PM
Today::10:00 PM - 12:00 AM
[2020-04-22 14:16:36] {main_loop} INFO: Selecting slot: Today::8:00 PM - 10:00 PM
[2020-04-22 14:16:36] {navigate} INFO: Navigating <Route beginning at 'https://www.amazon.com/gp/buy/shipoptionselect/handlers/display.html' with 3 stops>
[2020-04-22 14:16:36] {navigate_waypoint} INFO: Navigating <Waypoint [('xpath', "//*[contains(@class, 'ufss-overview-continue-button')]"), ('xpath', "//input[@class='a-button-text a-declarative' and @type='submit']")] -> 'gp/buy/payselect/handlers/display.html'>
[2020-04-22 14:16:44] {navigate_waypoint} INFO: Navigated to 'gp/buy/payselect/handlers/display.html'
[2020-04-22 14:16:44] {navigate_waypoint} INFO: Navigating <Waypoint ('id', 'continue-top') -> 'gp/buy/spc/handlers/display.html'>
[2020-04-22 14:16:55] {navigate_waypoint} INFO: Navigated to 'gp/buy/spc/handlers/display.html'
[2020-04-22 14:16:55] {navigate_waypoint} INFO: Navigating <Waypoint ('xpath', "//input[contains(@class, 'place-your-order-button')]") -> 'gp/buy/thankyou/handlers/display.html'>
[2020-04-22 14:17:02] {handle_redirect} WARNING: Redirected to: 'https://www.amazon.com/gp/buy/spc/handlers/static-submit-decoupled.html/ref=ox_spc_place_order'
[2020-04-22 14:17:02] {alert} INFO: Alerting user with message: 'Encountered an error'
Traceback (most recent call last):
  File "C:\Users\Doug\source\repos\whole-foods-deliverance\deliverance\nav.py", line 109, in navigate
    self.navigate_waypoint(driver, waypoint, timeout,
  File "C:\Users\Doug\source\repos\whole-foods-deliverance\deliverance\nav.py", line 90, in navigate_waypoint
    raise NavigationException(
deliverance.exceptions.NavigationException: Message: Navigation to 'gp/buy/thankyou/handlers/display.html' failed


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run.py", line 217, in <module>
    main_loop(driver, args)
  File "run.py", line 168, in main_loop
    build_route(site_config, 'CHECKOUT', args).navigate(driver)
  File "C:\Users\Doug\source\repos\whole-foods-deliverance\deliverance\nav.py", line 112, in navigate
    handle_redirect(driver,
  File "C:\Users\Doug\source\repos\whole-foods-deliverance\deliverance\nav.py", line 46, in handle_redirect
    raise UnhandledRedirect()
deliverance.exceptions.UnhandledRedirect: Message: None

(unknown error: DevToolsActivePort file doesn't exist)

Assuming I have something misconfigured or a driver in a non default location?

Whole Foods page has UI change

This is the line:

whole-foods-deliverance/config.py

Line 95 in 1bbdf3b

(By.XPATH, "//*[contains(text(),'Checkout {}')]/..".format(

Currently the button is called “Check out Whole Foods” instead of “Checkout Whole Foods” (note the space). I didn’t check Amazon Fresh.

I think it’s safer to support both, in case Amazon engineers change their minds.

Add support for multiple day / multiple delivery option regions

Issuing the following command, it dumps out at the delivery times: python run.py -s 'Amazon Fresh' -c --debug

DEBUG:urllib3.connectionpool:http://127.0.0.1:51412 "POST /session/b2ebca87c5b21e30059084b779d79d97/element HTTP/1.1" 404 1867
DEBUG:selenium.webdriver.remote.remote_connection:Finished Request
DEBUG:selenium.webdriver.remote.remote_connection:POST http://127.0.0.1:51412/session/b2ebca87c5b21e30059084b779d79d97/element {"using": "css selector", "value": ".ufss-slotselect-container"}
DEBUG:urllib3.connectionpool:http://127.0.0.1:51412 "POST /session/b2ebca87c5b21e30059084b779d79d97/element HTTP/1.1" 404 1867
DEBUG:selenium.webdriver.remote.remote_connection:Finished Request
DEBUG:selenium.webdriver.remote.remote_connection:POST http://127.0.0.1:51412/session/b2ebca87c5b21e30059084b779d79d97/element {"using": "css selector", "value": ".ufss-slotselect-container"}
DEBUG:urllib3.connectionpool:http://127.0.0.1:51412 "POST /session/b2ebca87c5b21e30059084b779d79d97/element HTTP/1.1" 404 1867
DEBUG:selenium.webdriver.remote.remote_connection:Finished Request
ERROR:utils:Timed out waiting for target element: ('class name', 'ufss-slotselect-container')
INFO:utils:Dumping page source to: source_dump_1586829983526.html
DEBUG:selenium.webdriver.remote.remote_connection:GET http://127.0.0.1:51412/session/b2ebca87c5b21e30059084b779d79d97/source {}
DEBUG:urllib3.connectionpool:http://127.0.0.1:51412 "GET /session/b2ebca87c5b21e30059084b779d79d97/source HTTP/1.1" 200 547930
DEBUG:selenium.webdriver.remote.remote_connection:Finished Request
Traceback (most recent call last):
File "run.py", line 183, in
main_loop(driver, args)
File "run.py", line 133, in main_loop
slots = slots_available(driver, slot_prefs)
File "run.py", line 76, in slots_available
slots = get_slots(driver)
File "run.py", line 30, in get_slots
slot_container = get_element(driver, config.Locators.SLOT_CONTAINER)
File "/Users/user/whole-foods-deliverance/utils.py", line 56, in get_element
EC.presence_of_element_located(locator)
File "/Users/user/whole-foods-deliverance/env/lib/python3.7/site-packages/selenium/webdriver/support/wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

Chrome being controlled by automated software

My chrome is still showing that Chrome being controlled by automated software. Even though i am not using the software now.
Had to reinstall chrome to fix the issue.
Is there something I missed? How can I avoid this next time?

Handle throttle.html page

This page is pretty rare, but I've seen it now and then while trying to check out manually, and I just saw it twice while running the script:

[2020-04-14 11:33:48] {navigate} WARNING: Current URL 'https://www.amazon.com/gp/buy/error/handlers/throttle.html' does not match target
Handling possible redirect (timeout in 30s)
[2020-04-14 11:34:18] {navigate} ERROR: Timed out waiting for redirect to a valid dest
Current URL: 'https://www.amazon.com/gp/cart/view.html'

The challenge is that the page contains a button that's disabled initially, with a progress bar. It takes 20sec or so and then turns into an active button that takes you back to the cart page.

Once it's active, the source looks like this:

<a id="checkout-throttle-return-to-cart-announce" href="https://www.amazon.com/gp/cart/view.html" class="a-button-text" role="button">
      Continue
    </a>

timeout error?

ERROR: Timed out waiting for target element: ('class name', 'ufss-slotselect-container')
Traceback (most recent call last):
File "run.py", line 185, in
main_loop(driver, args)
File "run.py", line 141, in main_loop
slots = slots_available(driver, slot_prefs)
File "run.py", line 76, in slots_available
slots = get_slots(driver)
File "run.py", line 30, in get_slots
slot_container = get_element(driver, config.Locators.SLOT_CONTAINER)
File "/Users/BD/whole-foods-deliverance/utils.py", line 55, in get_element
WebDriverWait(driver, timeout).until(
File "/Users/BD/whole-foods-deliverance/env/lib/python3.8/site-packages/selenium/webdriver/support/wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

Add audio alerts for Windows

nav.py and wait.py through exceptions

@mark-thompson here is the traceback. Thanks!

INFO:main:No slots found :( waiting...
INFO:main:Checking for available slots
INFO:main:Found 1 slots:
Tomorrow::11:00 AM - 1:00 PM
INFO:notify:Alerting user with message: 'Delivery slots found'
ERROR:notify:send_sms() requires a config file at 'conf.toml' with key 'twilio'
ERROR:notify:send_telegram() requires a config file at 'conf.toml' with key 'telegram'
INFO:main:Attempting to select slot and checkout
INFO:main:Selecting slot: Tomorrow::11:00 AM - 1:00 PM
INFO:nav:Navigating <Route beginning at 'https://www.amazon.com/gp/buy/shipoptionselect/handlers/display.html' with 3 stops>
INFO:nav:Navigating <Waypoint ('xpath', "//*[contains(@Class, 'ufss-overview-continue-button')]") -> 'gp/buy/payselect/handlers/display.html'>
INFO:nav:Navigated to 'gp/buy/payselect/handlers/display.html'
INFO:nav:Navigating <Waypoint ('id', 'continue-top') -> 'gp/buy/spc/handlers/display.html'>
WARNING:nav:Current URL 'https://www.amazon.com/gp/buy/shipoptionselect/handlers/display.html' does not match target
Handling possible redirect (timeout in 10s)
Traceback (most recent call last):
File "/Users/aub/Repos/whole-foods-deliverance/nav.py", line 86, in navigate
EC.url_contains(waypoint.dest)
File "/Users/aub/Repos/whole-foods-deliverance/env/lib/python3.7/site-packages/selenium/webdriver/support/wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "run.py", line 147, in
site_config.Routes.CHECKOUT.navigate(driver)
File "/Users/aub/Repos/whole-foods-deliverance/nav.py", line 90, in navigate
raise e
File "/Users/aub/Repos/whole-foods-deliverance/nav.py", line 68, in navigate
self.navigate_waypoint(driver, waypoint, timeout)
File "/Users/aub/Repos/whole-foods-deliverance/nav.py", line 54, in navigate_waypoint
"Navigation to '{}' failed".format(waypoint.dest)
nav.NavigationException: Message: Navigation to 'gp/buy/spc/handlers/display.html' failed

Complete purchase upon finding open delivery slots

Ideally, this should not just notify you of open slots, but complete the purchase for you, or at least give you the option to complete the purchase automatically.
Ideas:

• Read delivery slot preferences via a config file and/or command line arg
• Format the open slots and accept and process SMS replies via Twilio webhooks

Retry slot select if checkout fails

Currently, if a slot becomes unavailable before the checkout process is complete, we are redirected to the slot select page and navigation fails.

Instead, we should iterate any remaining slots and try to checkout again until no slots remain

ERROR: Timed out waiting for target element: ('class name', 'ufss-slotselect-container')

Getting this error just after "Checking for available slots