[BUG] Suddenly blind

[aluminum-ice]

[BUG] Suddenly blind

[aluminum-ice]

@kennyHH I had a stupid error in my code. Try this version (just replace bettercap.py from v1.7.8 completely with this code):

import json
import logging
import requests
import websockets
import asyncio
import random

from requests.auth import HTTPBasicAuth
from time import sleep

requests.adapters.DEFAULT_RETRIES = 5 # increase retries number

ping_timeout  = 180
ping_interval = 15
max_queue = 10000

min_sleep = 0.5
max_sleep = 5.0

def decode(r, verbose_errors=True):
    try:
        return r.json()
    except Exception as e:
        if r.status_code == 200:
            logging.error("error while decoding json: error='%s' resp='%s'" % (e, r.text))
        else:
            err = "error %d: %s" % (r.status_code, r.text.strip())
            if verbose_errors:
                logging.info(err)
            raise Exception(err)
        return r.text


class Client(object):
    def __init__(self, hostname='localhost', scheme='http', port=8081, username='user', password='pass'):
        self.hostname = hostname
        self.scheme = scheme
        self.port = port
        self.username = username
        self.password = password
        self.url = "%s://%s:%d/api" % (scheme, hostname, port)
        self.websocket = "ws://%s:%s@%s:%d/api" % (username, password, hostname, port)
        self.auth = HTTPBasicAuth(username, password)

    def session(self):
        r = requests.get("%s/session" % self.url, auth=self.auth)
        return decode(r)

    async def start_websocket(self, consumer):
        s = "%s/events" % self.websocket

        # More modern version of the approach below
        # logging.info("Creating new websocket...")
        # async for ws in websockets.connect(s):
        #     try:
        #         async for msg in ws:
        #             try:
        #                 await consumer(msg)
        #             except Exception as ex:
        #                     logging.debug("Error while parsing event (%s)", ex)
        #     except websockets.exceptions.ConnectionClosedError:
        #         sleep_time = max_sleep*random.random()
        #         logging.warning('Retrying websocket connection in {} sec'.format(sleep_time))
        #         await asyncio.sleep(sleep_time)
        #         continue

        # restarted every time the connection fails
        while True:
            logging.info("creating new websocket...")
            try: 
                async with websockets.connect(s, ping_interval=ping_interval, ping_timeout=ping_timeout, max_queue=max_queue) as ws:
                    # listener loop
                    while True:
                        try:
                            async for msg in ws:
                                try:
                                    await consumer(msg)
                                except Exception as ex:
                                        logging.debug("error while parsing event (%s)", ex)
                        except websockets.exceptions.ConnectionClosedError:
                            try: 
                                pong = await ws.ping()
                                await asyncio.wait_for(pong, timeout=ping_timeout)
                                logging.warning('ping OK, keeping connection alive...')
                                continue
                            except:
                                sleep_time = min_sleep + max_sleep*random.random()
                                logging.warning('ping error - retrying connection in {} sec'.format(sleep_time))
                                await asyncio.sleep(sleep_time)
                                break
            except ConnectionRefusedError:
                sleep_time = min_sleep + max_sleep*random.random()
                logging.warning('nobody seems to be listening at the bettercap endpoint...')
                logging.warning('retrying connection in {} sec'.format(sleep_time))
                await asyncio.sleep(sleep_time)
                continue
            except OSError:
                sleep_time = min_sleep + max_sleep*random.random()
                logging.warning('connection to the bettercap endpoint failed...')
                logging.warning('retrying connection in {} sec'.format(sleep_time))
                await asyncio.sleep(sleep_time)
                continue


    def run(self, command, verbose_errors=True):
        while True:
            try:
                r = requests.post("%s/session" % self.url, auth=self.auth, json={'cmd': command})
            except requests.exceptions.ConnectionError as e:
                sleep_time = min_sleep + max_sleep*random.random()
                logging.warning("can't run my request... connection to the bettercap endpoint failed...")
                logging.warning('retrying run in {} sec'.format(sleep_time))
                sleep(sleep_time)
            else:
                break

        return decode(r, verbose_errors=verbose_errors)

[aluminum-ice]

Also if you want me to give you a Raspberry Pi 4B, I've got one I can send you, it's the least I could do given all the time you put in in making your fork. If so, just private message me your address.

I appreciate it but I have several RPi4B sitting around my desk. I don't need another one at this time. I don't use them for pwnagotchi. Maybe after I feel that the RPiZ2W is working the way I want it to, I will test the 4B and see how well it works on that one.

I have an alpha v1.7.8 version coming I want you to test to see if this problem is fixed. I will post a note here when its ready.

[aluminum-ice]

Yes I tested it extensively yesterday. The code no longer crashes out or lock up. Instead, when the web socket loses the connection it reconnects after a few seconds. However, I noticed that this inevitably seems to show at that point the device is blind. I suspect it's signaling that the nexmon driver has crashed or something. So my pwnagotchi reboots soon after.

I've also noticed that my pwnagotchi is now far better at capturing handshakes. At least qualitatively it's more successful at seeing APs and getting handshakes.

[aluminum-ice]

Yes I tested it extensively yesterday. The code no longer crashes out or lock up. Instead, when the web socket loses the connection it reconnects after a few seconds. However, I noticed that this inevitably seems to show at that point the device is blind. I suspect it's signaling that the nexmon driver has crashed or something. So my pwnagotchi reboots soon after.

I've also noticed that my pwnagotchi is now far better at capturing handshakes. At least qualitatively it's more successful at seeing APs and getting handshakes.

I’ve noticed that the watchdog seems to think it’s blind and does a reboot within 4-10 mins of boot on mine as I can see some brcm errors, however disabling the watchdog stopped this for me. Just got back from a trip to the shops and in got 90 handshakes in 45 mins, 0 blind epochs and it’s still going as we speak!

Yes, I turned off my watchdog for the same reason. Just set the max blind epochs in the config file to 5 and it will quickly reboot if it's truly blind. The watchdog is triggering falsely.

[rsun0525]

I think I was able to reproduce the issue again on my Raspberry Pi Zero 2 W using the built in WiFi, here's the /var/log/pwnagotchi.log (scroll from the bottom up to see).
pwnagotchi.log

[rsun0525]

Here's my 400 again, more times when it died, I just set it to if it's blind for 10 epochs to restart now, not sure if you want me to continue to upload these log files?
pwnagotchi.log

[aluminum-ice]

Let me look into these log files before adding more. I'll focus on the RPiZ2W since that is what I use and have.

[rsun0525]

No worries, I'm assuming the 4B problem is that you need to update some driver, as if I use the external USB Wi-Fi adaptor on it and disable the onboard Wi-Fi adapter it works, etc.

[aluminum-ice]

@rsun0525 Can you please edit "sudo nano /usr/bin/pwnagotchi-launcher" to be in debug mode when you start AUTO mode (add the --debug flag as you see below) and attach the log file for the RPiZ2W when it crashes? Please check the log file to ensure you see [DEBUG] entries.

if is_auto_mode; then
  /usr/local/bin/pwnagotchi --debug
else
  /usr/local/bin/pwnagotchi --manual
fi

[rsun0525]

Also if you want me to give you a Raspberry Pi 4B, I've got one I can send you, it's the least I could do given all the time you put in in making your fork. If so, just private message me your address.

[rsun0525]

@rsun0525 Can you please edit "sudo nano /usr/bin/pwnagotchi-launcher" to be in debug mode when you start AUTO mode (add the --debug flag as you see below) and attach the log file for the RPiZ2W when it crashes? Please check the log file to ensure you see [DEBUG] entries.

if is_auto_mode; then
  /usr/local/bin/pwnagotchi --debug
else
  /usr/local/bin/pwnagotchi --manual
fi

Done, I've got 1.7.7 running on a Raspberry Pi 400 (with external USB WiFi Adaptor) and a Raspberry Pi Zero 2W, both updated and rebooted just now.

[rsun0525]

Hopefully this caught it as I found the pwnagotchi rebooted itself.
pwnagotchi.gz
pwnagotchi.log

[rsun0525]

rebooted again
pwnagotchi.log

[rsun0525]

rebooted again
pwnagotchi.log

[rsun0525]

Sounds good, as soon as I can put 1.7.8, I'll put it on it. Thanks again for your time and effort!

[wpa-2]

Also if you want me to give you a Raspberry Pi 4B, I've got one I can send you, it's the least I could do given all the time you put in in making your fork. If so, just private message me your address.

I appreciate it but I have several RPi4B sitting around my desk. I don't need another one at this time. I don't use them for pwnagotchi. Maybe after I feel that the RPiZ2W is working the way I want it to, I will test the 4B and see how well it works on that one.

I have an alpha v1.7.8 version coming I want you to test to see if this problem is fixed. I will post a note here when its ready.

Thanks for all the awesome work .

[naturekid72]

When can we Update via Git pull?

[rsun0525]

Raspberry Pi Zero 2W running 1.7.8

pwnagotchi.log

[aluminum-ice]

Raspberry Pi Zero 2W running 1.7.8

pwnagotchi.log

What's the issue? There is no crash in that log file. The ephemeral port exhaustion issue is fixed. If you mean the 'blind bug' as in the brcm driver crashes, that's a different issue. It's a problem with nexmon not pwnagotchi.

[aluminum-ice]

I am going to branch my repo and try using DrSchottky's nexmon fork instead of the official nexmon. I think his fork is more up to date and actively developed. I want to see if that reduces or eliminates the brcm crashes that cause the blind bug. Have you seen any more crashes due to websockets running out of ephemeral TCP ports?

[rsun0525]

Yes, I've seen crashes which I think are related to websockets.

[insanelysick]

Took your latest built out for a trip through the city yesterday and started getting the below for tens of thousands of lines
"[2023-09-11 08:01:26,589] [ERROR] Lost websocket connection. Reconnecting..."

For giggles I took your bettercap.py and put it into Xbits build and didn’t get these errors for some reason.

[aluminum-ice]

Yes, I've seen crashes which I think are related to websockets.

If you've seen crashes, send me an example log. An entry saying [ERROR] is not a crash. A crash is what the earlier logs was saying where python was exiting completely.

[aluminum-ice]

Took your latest built out for a trip through the city yesterday and started getting the below for tens of thousands of lines

"[2023-09-11 08:01:26,589] [ERROR] Lost websocket connection. Reconnecting..."

For giggles I took your bettercap.py and put it into Xbits build and didn’t get these errors for some reason.

That's the way it's supposed to work. Before you never saw the error because bettercap was simply spinning up more and more websockets instead of reconnecting to an existing socket until it exhausted all the ephemeral TCP ports and then did a hard crash. What you're seeing is the correct behavior and not a crash.

[insanelysick]

Took your latest built out for a trip through the city yesterday and started getting the below for tens of thousands of lines
"[2023-09-11 08:01:26,589] [ERROR] Lost websocket connection. Reconnecting..."
For giggles I took your bettercap.py and put it into Xbits build and didn’t get these errors for some reason.

That's the way it's supposed to work. Before you never saw the error because bettercap was simply spinning up more and more websockets instead of reconnecting to an existing socket until it exhausted all the ephemeral TCP ports and then did a hard crash. What you're seeing is the correct behavior and not a crash.

Interesting, my unit became unresponsive and I had to pull the plug then found that error about 40,000 times in the logs. I’ll give it another run and try waiting out any unresponsiveness and see what happens

[aluminum-ice]

Took your latest built out for a trip through the city yesterday and started getting the below for tens of thousands of lines

"[2023-09-11 08:01:26,589] [ERROR] Lost websocket connection. Reconnecting..."

For giggles I took your bettercap.py and put it into Xbits build and didn’t get these errors for some reason.

That's the way it's supposed to work. Before you never saw the error because bettercap was simply spinning up more and more websockets instead of reconnecting to an existing socket until it exhausted all the ephemeral TCP ports and then did a hard crash. What you're seeing is the correct behavior and not a crash.

Interesting, my unit became unresponsive and I had to pull the plug then found that error about 40,000 times in the logs. I’ll give it another run and try waiting out any unresponsiveness and see what happens

I just saw the behavior you mentioned. A few reconnections is normal but not 40K. I had not seen this earlier when I tested it prior to release. There might still be a bug somewhere.

[aluminum-ice]

I think what is happening is that too many simultaneous reconnection attempts are happening in areas with many APs. This overwhelms the bettercap service and locks up. I'm going to test a change to randomize the reconnections.

[aluminum-ice]

@insanelysick @rsun0525 Please replace your bettercap.py file with the following code and let me know if you see the same problem "[2023-09-11 08:01:26,589] [ERROR] Lost websocket connection. Reconnecting..."

import logging
import requests
import websockets
import asyncio
import random

from requests.auth import HTTPBasicAuth
from time import sleep

requests.adapters.DEFAULT_RETRIES = 5 # increase retries number

ping_timeout  = 90
ping_interval = 60

max_sleep = 2.0

def decode(r, verbose_errors=True):
    try:
        return r.json()
    except Exception as e:
        if r.status_code == 200:
            logging.error("error while decoding json: error='%s' resp='%s'" % (e, r.text))
        else:
            err = "error %d: %s" % (r.status_code, r.text.strip())
            if verbose_errors:
                logging.info(err)
            raise Exception(err)
        return r.text


class Client(object):
    def __init__(self, hostname='localhost', scheme='http', port=8081, username='user', password='pass'):
        self.hostname = hostname
        self.scheme = scheme
        self.port = port
        self.username = username
        self.password = password
        self.url = "%s://%s:%d/api" % (scheme, hostname, port)
        self.websocket = "ws://%s:%s@%s:%d/api" % (username, password, hostname, port)
        self.auth = HTTPBasicAuth(username, password)

    def session(self):
        r = requests.get("%s/session" % self.url, auth=self.auth)
        return decode(r)

    async def start_websocket(self, consumer):
        s = "%s/events" % self.websocket

        # restarted every time the connection fails
        while True:
            logging.info("creating new websocket...")
            try: 
                async with websockets.connect(s, ping_interval=ping_interval, ping_timeout=ping_timeout) as ws:
                    # listener loop
                    while True:
                        try:
                            async for msg in ws:
                                try:
                                    await consumer(msg)
                                except Exception as ex:
                                        logging.debug("error while parsing event (%s)", ex)
                        except websockets.exceptions.ConnectionClosedError:
                            try: 
                                pong = await ws.ping()
                                await asyncio.wait_for(pong, timeout=ping_timeout)
                                logger.warning('ping OK, keeping connection alive...')
                                continue
                            except:
                                sleep_time = max_sleep*random.random()
                                logger.warning('ping error - retrying connection in {} sec'.format(sleep_time))
                                await asyncio.sleep(sleep_time)
                                break
            except ConnectionRefusedError:
                sleep_time = max_sleep*random.random()
                logger.warning('nobody seems to listen to the bettercap endpoint...')
                logger.warning('retrying connection in {} sec'.format(sleep_time))
                await asyncio.sleep(sleep_time)
                continue

    def run(self, command, verbose_errors=True):
        for _ in range(0,2):
            try:
                r = requests.post("%s/session" % self.url, auth=self.auth, json={'cmd': command})
            except requests.exceptions.ConnectionError as e:
                sleep_time = max_sleep*random.random()
                logging.exception("Request connection error (%s) while running command (%s)", e, command)
                logging.warning('Retrying run in {} sec'.format(sleep_time))
                sleep(sleep_time)
            else:
                break

        return decode(r, verbose_errors=verbose_errors)

[insanelysick]

Thans @aluminum-ice, I've burned a fresh copy of 1.7.8 and replaced bettercap.py with the above. I'll run it for a while and report back although I don't think I'll be heading into an area with a high amount of APs until early next week but I'll see what I can do.

[insanelysick]

Got these in the logs so far, which afterwards blindness sets in.

From Pwnagotchi.log
[2023-09-13 11:05:22,636] [ERROR] Request connection error (HTTPConnectionPool(host='localhost', port=8081): Max retries exceeded with url: /api/session (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x462a3bf0>: Failed to establish a new connection: [Errno 111] Connection refused'))) while running command (wifi.recon.channel 1,2,5,7,8,9,11,13)
raise err
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
raise MaxRetryError(_pool, url, error or ResponseError(cause))
[2023-09-13 11:05:23,660] [ERROR] Request connection error (HTTPConnectionPool(host='localhost', port=8081): Max retries exceeded with url: /api/session (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x44367630>: Failed to establish a new connection: [Errno 111] Connection refused'))) while running command (wifi.recon.channel 1,2,5,7,8,9,11,13)
raise err
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
raise MaxRetryError(_pool, url, error or ResponseError(cause))
return decode(r, verbose_errors=verbose_errors)
[2023-09-13 11:05:24,070] [ERROR] Request connection error (HTTPConnectionPool(host='localhost', port=8081): Max retries exceeded with url: /api/session (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x44355230>: Failed to establish a new connection: [Errno 111] Connection refused'))) while running command (set wifi.ap.ttl 368)
raise err
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
raise MaxRetryError(_pool, url, error or ResponseError(cause))
[2023-09-13 11:05:24,123] [ERROR] Request connection error (HTTPConnectionPool(host='localhost', port=8081): Max retries exceeded with url: /api/session (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x44355490>: Failed to establish a new connection: [Errno 111] Connection refused'))) while running command (set wifi.ap.ttl 368)
raise err
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
raise MaxRetryError(_pool, url, error or ResponseError(cause))
[2023-09-13 11:05:26,099] [ERROR] [ai] error while training (local variable 'r' referenced before assignment)
return decode(r, verbose_errors=verbose_errors)

and syslog is showing tons of these,
Sep 13 12:27:28 pwnagotchi bettercap-launcher[1267]: [12:27:28] [sys.log] [err] wifi could not inject WiFi packet: send: Resource temporarily unavailable

[kennyHH]

I had the same issue as other guys. My last log file has over 4 million lines of "[ERROR] Lost websocket connection. Reconnecting..." in approximately 40 minutes. I'll try your new solution tomorrow, and I've also included zipped log file since it's over 330mb in size when unpacked :)

pwnagotchi-6.gz

Edit : I've updated the bettercap.py file with the new version, and it seems like the issue with the excessive log file has been resolved. However, I'm still encountering errors, and the program becomes unresponsive. I took it for a walk once in the morning and once in the evening.

Errors start from lines :

Morning walk : line 274

Evening walk : line 1359

Log file : pwnagotchi.log

[rsun0525]

I "think" there's been an improvement since implementing the bettercap.py above, at least when it is blind, it's autorebooting itself. I'll try to put it on more pwnagotchi's and see if I can create more chances for it to fail and send up some logs. Thanks again for your time and effort, it's much appreicated!

[rsun0525]

Next thing to fix is support Raspberry Pi 4b internal wifi adaptor :-).

…

On Sat, Sep 16, 2023 at 10:56 aluminum-ice ***@***.***> wrote: Yes I tested it extensively yesterday. The code no longer crashes out or lock up. Instead, when the web socket loses the connection it reconnects after a few seconds. However, I noticed that this inevitably seems to show at that point the device is blind. I suspect it's signaling that the nexmon driver has crashed or something. So my pwnagotchi reboots soon after. I've also noticed that my pwnagotchi is now far better at capturing handshakes. At least qualitatively it's more successful at seeing APs and getting handshakes. — Reply to this email directly, view it on GitHub <#39 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHDXGCZDXO5LQRKKZU4I2YLX2W4ZVANCNFSM6AAAAAA4K2HMAU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[insanelysick]

Yes I tested it extensively yesterday. The code no longer crashes out or lock up. Instead, when the web socket loses the connection it reconnects after a few seconds. However, I noticed that this inevitably seems to show at that point the device is blind. I suspect it's signaling that the nexmon driver has crashed or something. So my pwnagotchi reboots soon after.

I've also noticed that my pwnagotchi is now far better at capturing handshakes. At least qualitatively it's more successful at seeing APs and getting handshakes.

I’ve noticed that the watchdog seems to think it’s blind and does a reboot within 4-10 mins of boot on mine as I can see some brcm errors, however disabling the watchdog stopped this for me. Just got back from a trip to the shops and in got 90 handshakes in 45 mins, 0 blind epochs and it’s still going as we speak!

[rsun0525]

Thanks for the tip. I will turn off watchdog and see.

…

On Sat, Sep 16, 2023 at 11:04 insanelysick ***@***.***> wrote: Yes I tested it extensively yesterday. The code no longer crashes out or lock up. Instead, when the web socket loses the connection it reconnects after a few seconds. However, I noticed that this inevitably seems to show at that point the device is blind. I suspect it's signaling that the nexmon driver has crashed or something. So my pwnagotchi reboots soon after. I've also noticed that my pwnagotchi is now far better at capturing handshakes. At least qualitatively it's more successful at seeing APs and getting handshakes. I’ve noticed that the watchdog seems to think it’s blind and does a reboot within 4-10 mins of boot on mine as I can see some brcm errors, however disabling the watchdog stopped this for me. Just got back from a trip to the shops and in got 90 handshakes in 45 mins, 0 blind epochs and it’s still going as we speak! — Reply to this email directly, view it on GitHub <#39 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHDXGC2KXSGATHWTWDQ2JJTX2W5ZPANCNFSM6AAAAAA4K2HMAU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[aluminum-ice]

Next thing to fix is support Raspberry Pi 4b internal wifi adaptor :-).

On Sat, Sep 16, 2023 at 10:56 aluminum-ice @.***> wrote:

Yes I tested it extensively yesterday. The code no longer crashes out or

lock up. Instead, when the web socket loses the connection it reconnects

after a few seconds. However, I noticed that this inevitably seems to show

at that point the device is blind. I suspect it's signaling that the nexmon

driver has crashed or something. So my pwnagotchi reboots soon after.

I've also noticed that my pwnagotchi is now far better at capturing

handshakes. At least qualitatively it's more successful at seeing APs and

getting handshakes.

—

Reply to this email directly, view it on GitHub

#39 (comment),

or unsubscribe

https://github.com/notifications/unsubscribe-auth/AHDXGCZDXO5LQRKKZU4I2YLX2W4ZVANCNFSM6AAAAAA4K2HMAU

.

You are receiving this because you were mentioned.Message ID:

@.***>

Haha. Not by me. I don't care about the 4B. I only care about the 2W because that is what I use. Someone else can submit PRs or fork my fork.

[insanelysick]

Checked in on the logs at the end of the day. 129 handshakes captured, no bettercap web socket issues even when in an area infested with APs and potentially 400-500 clients. Seems like it's sorted but will take it through the city on Monday to make sure. Thanks for the hard work dude, its been a long time coming but I think the end is in sight for getting the 02 stable.

[ryanbeard82]

Running smooth as eggs on this end too. Been running around all day with it in my pocket. No problemos.

[rsun0525]

For a crash on my Raspberry Pi 3B+

ifconfig still shows the device there:

pwnagotchi.log

[rsun0525]

Saw this, and I had to reboot it to start working again.

pwnagotchi.log

[rsun0525]

What I notice is that when my pwnagotchi hit this condition, I know to ssh in and just "reboot" to get it going again.

Is there some additional code you can add that say if you are in this state for several cycles, just do a "sudo reboot" on it?

[rsun0525]

In 1.7.9, Sometimes after a reboot (manual or automaticaly), I can't get it to start up, and the only solution I can find it to reboot again and it does start up.

pwnagotchi.log

[rsun0525]

Going to close this out open up a new ticket specific to bettercap.py failing