Failed to Login > Error setting up platform netgear (More Details Enclosed)

[matmaul]

Failed to Login > Error setting up platform netgear (More Details Enclosed)

[MatMaul]

merged, and I just added my logging changes. I'll merge another PR and release a new version this weekend. Please continue testing with the new commits, always helpful to validate a bit before release, and the more the better :)

[MatMaul]

You can change line 87/88 and LOGIN_BODY at line 380 to match your finding and see if it helps.

[gruijter]

Super! Together we'll bring that netgear soap mess down on its knees 👍

[caffeinatedMike]

Haha, all I can say is long day :P

[gruijter]

@caffeinatedMike BOOHOOHOO indeed :) At least we managed to get the new login method operational!

But are you sure the content-type should be 'application/xml'? In all the genie captures on my R7000 and R7800 the content-type is set to 'multipart/form-data'. ...

It would be great if you could test my node version, so that I get confirmation that your C3000 is supported. Testing from a (e.g. windows) desktop is real easy:

// INSTRUCTIONS FOR TESTING FROM DESKTOP:
install node (https://nodejs.org)
install this package: > npm i netgear
run the test (from the test folder): > node test password

[MatMaul]

@caffeinatedMike I'll probably release a new version this weekend. I am adding proper logging so we can get useful debug logs when debugging.

Can you test what fields are really necessary for you in the header ?
I am mainly interesting in removing Connection: Keep-Alive because I think we don't do keep-alive so we are kind of lying :)

If you can also test with multipart/form-data as a content type as mentioned by @gruijter .

[MatMaul]

It seems like it can't connect at all to the soap interface.
Is your router web interface running on port 80 ?
You should play with ssl parameter (boolean) too. Which I forgot to document :( bad me.

[caffeinatedMike]

This may have been an issue on my end (as you can see in this lengthy thread on the HA forums). Although, I cannot confirm yet, but will likely be able to test on my new rPi tomorrow. Long story short, I was trying to run Hassio on an old laptop to get a feel for it until my rPi came (saturday). Turns out there's a disconnect somewhere between the docker instance and my router. Blame's on me for trying to use an advanced install method to try out Hassio when I hate linux lol. I was just too anxious to get started and dove head-first into what seems to be an empty pool 🤕 😆

Once I get my raspberry pi all set up tomorrow I'll let you know if the issue persists.

[caffeinatedMike]

@MatMaul So, this isn't an issue with my setup (I don't believe). To further test, I'm running HA on my laptop (before bothering to config my Pi) and I'm still running into the issue. I have a feeling it has something to do with my router model. Something funny happened. The router/gateway is detected/configurable in HA, but once configured only shows speed statistics of the gateway, as is seen in the below screenshots

I've tried messing with the ssl parameter as you suggested and it made no difference. See below

device_tracker:
  - platform: netgear
    ssl: False
    host: 192.168.0.1
    port: 80
    username: REDACTED
    password: REDACTED
    interval_seconds: 30
    consider_home: 90
    new_device_defaults:
      track_new_devices: True
      hide_if_away: False

Here is a screenshot of the latest errors. For some reason they don't end up in the "full" log when pressing the refresh button, which HA suggests to do to view the whole log.

I've provided all error details, gathered by clicking on each error related to the netgear issues below

Unable to load C:\Users\Mike\AppData\Roaming\.homeassistant\known_devices.yaml: Config file not found: C:\Users\Mike\AppData\Roaming\.homeassistant\known_devices.yaml

Error talking to API
Traceback (most recent call last):
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 1321, in getresponse
    response.begin()
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 296, in begin
    version, status, reason = self._read_status()
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 257, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\socket.py", line 589, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\adapters.py", line 445, in send
    timeout=timeout
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\util\retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\packages\six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 1321, in getresponse
    response.begin()
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 296, in begin
    version, status, reason = self._read_status()
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 257, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\socket.py", line 589, in readinto
    return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Mike\AppData\Roaming\.homeassistant\deps\Python37\site-packages\pynetgear\__init__.py", line 272, in _make_request
    data=message, timeout=30, verify=False)
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 112, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\adapters.py", line 495, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

Failed to Login

Error setting up platform netgear

Really hoping you can provide some insight/fix ☹️
I'm up to help debug, make adjustments (whatever you can walk me through) to diagnose and resolve this issue.

[MatMaul]

I don't think I can help you with that, pretty sure it's on your side. It still completely fails to connect to the router, why I don't know.
Use your rPi first, you are on Windows, I am not even going further with that :)

[caffeinatedMike]

Just noting the two difference I was able to find when examining the PyNetgear's attempted communications with my router and what the Netgear Genie app shows.

The only two differences I found (and what I believe to be the issue, at least in my case) are as follows:

1. Different Login SOAPAction in Post Headers

• PyNetgear uses SOAPAction: urn:NETGEAR-ROUTER:service:ParentalControl:1#Authenticate
• My Router uses SOAPAction: urn:NETGEAR-ROUTER:service:DeviceConfig:1#SOAPLogin

2. Different Login XML in Post Body

• PyNetgear's Login XML, used to gain access to Router's SOAP API

<SOAP-ENV:Body>
<Authenticate>
  <NewUsername>{username}</NewUsername>
  <NewPassword>{password}</NewPassword>
</Authenticate>
</SOAP-ENV:Body>

• My router's Login XML, captured with Packet Capture android app.

<SOAP-ENV:Body>
<M1:SOAPLogin xmlns:M1="urn:NETGEAR-ROUTER:service:DeviceConfig:1">
  <Username>{username}</Username>
  <Password>{password}</Password>
</M1:SOAPLogin>
</SOAP-ENV:Body>

Since the initial login headers & XML Schema appears to be different for my router that would explain why the component fails to communicate with my router at all.

[caffeinatedMike]

Will do. I'll modify a local copy of the module once I'm home from work and have direct access to the router. If that ends up being the culprit how would you suggest a change be implemented. Would you want me to add a try/except statement in the login function to support both login scenerios? I suspect that my router uses a legacy login flow (seeing as it's not as new as the Nighthawks or N7000s that people report having success with). Or do you want me to just report back if the testing was successful and leave it to you to implement to keep your logic consistent?

[MatMaul]

I would say try the current one first and if it fails try SOAPLogin. You should be able to do it without changing anything outside of login method.

[MatMaul]

And I am lazy so I'll gladly take a tested pull request.
I insist on tested ;)

[caffeinatedMike]

Haha, will do! Just wanted to make sure it wouldn't be considered too hacky of an implementation. But, if all you care about is if it's tested that's fine by me :)

[gruijter]

Actually I found out that the R7000 and R7800 have the same login. Note that also the action.urn is different:

POST http://routerlogin.net/soap/server_sa/ HTTP/1.1
Accept: text/xml
SOAPAction: urn:NETGEAR-ROUTER:service:DeviceConfig:1#SOAPLogin
Cache-Control: no-cache
Pragma: no-cache
Accept-Encoding: gzip
Accept-Language: zh-CN,en,*
Content-type: multipart/form-data
User-Agent: SOAP Toolkit 3.0
Content-Length: 596
Host: routerlogin.net
Connection: Keep-Alive

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<SOAP-ENV:Envelope xmlns:SOAPSDK1="http://www.w3.org/2001/XMLSchema" xmlns:SOAPSDK2="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAPSDK3="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP-ENV:Header>
<SessionID>A7D88AE69687E58D9A00</SessionID>
</SOAP-ENV:Header>
<SOAP-ENV:Body>
<M1:SOAPLogin xmlns:M1="urn:NETGEAR-ROUTER:service:DeviceConfig:1">
  <Username>admin</Username>
  <Password>******</Password>
</M1:SOAPLogin>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

[MatMaul]

@gruijter I think it makes sense (which is kind of unusual with this API :-P ), the SOAPAction is the full qualifier of the action, while in the xml you specify the context (DeviceConfig:1) and the action is the actual XML tag.

[MatMaul]

Nevermind you were saying the same ^^

[MatMaul]

@caffeinatedMike you can use SERVICE_DEVICE_CONFIG as service parameter when calling _make_request since it is already defined.

[gruijter]

behaviour seems to be different though. I just changed it to ^^^^ In my node package. Now I can only do one action, and ater that it seems to be logged out again. Need to check what is going on...

[MatMaul]

You mean that your router can take both login method but they behave differently ? My god this API is such a garbage...

[gruijter]

yes, my R7800 takes both methods. My genie app uses what @caffeinatedMike found though.

[MatMaul]

Ok then I'll try this one on my Orbi to see.
We can perhaps put the other one first if it works for a majority of models, and call it each time before doing the action call.

[caffeinatedMike]

Oh man, just saw all these comments over since my last reply. It's so good to hear that it's not just me being crazy. The api is just a f***** mess. So, if I'm understanding you correctly, both my and the current methods will be implemented (after your tests, if successful) to broaden the model-coverage of the component. Are you saying I might actually finally be able to use my router as a device_tracker? 😃

[gruijter]

@MatMaul I found out that with this 'new' method for each request you must use the cookie that you get on first login. But as always (very nice) Netgear has made the cookie header different on different models. For R7800 it is 'Set-Cookie', and on R7000 it is 'set-cookie'. It seems to work now in my node version. But I need to test some more. If it works I will probably deprecate the old login method, since this new method seems to be more consistent.

**edit: never mind the capitals. That might be just a node thing.

[gruijter]

here an example of login where you receive the cookie:

HTTP/1.1 200 OK
CONTENT-LENGTH: 426
Set-Cookie: sess_id=8696e1659b88558de96329c07b59ba95ad80a218ca1d9ea7c1605d8ad33032e85c4095f6afc959120959ff55e97857cdf163d64f0ffcdd4e5a311994959da540; SameSite=Strict
CONTENT-TYPE: text/xml; charset="UTF-8"
SERVER: "OS/version" UPnP/1.0 "product/version"

<?xml version="1.0" encoding="UTF-8"?>
<soap-env:Envelope
        xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/"
        soap-env:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"
        >
<soap-env:Body>
    <m:SOAPLoginResponse
        xmlns:m="urn:NETGEAR-ROUTER:service:DeviceConfig:1">
    </m:SOAPLoginResponse>
    <ResponseCode>000</ResponseCode>
</soap-env:Body>
</soap-env:Envelope>

[gruijter]

@caffeinatedMike If you want to try the new login method on node, you can use my beta branch here:
https://github.com/gruijter/netgear.js/tree/beta

[caffeinatedMike]

I don't have node, I'm only setup with python atm :/

[gruijter]

I made my version so that it first does getCurrentSettings (can be done without login). It contains a loginMethod which is 2.0 for both my R7000 and R7800. I assume this method defines if you can use the new login method, or that you have to use the old one (all assumptions now...)

Anyway, I made it to use the new login method only if the getCurrentSettings.loginMethod >= 2

Enough work for today :)
over and out

[caffeinatedMike]

Wait, how do I test any of this for myself (in my home assistant setup)? I know nothing about node lol

And side note: My login method also states 2.0 >.<

[gruijter]

Lol. I know nothing on HA or python😂

[caffeinatedMike]

Lol whelp that leaves me at a loss 😆 I'll just try @MatMaul's suggestion

You can change line 87/88 and LOGIN_BODY at line 380 to match your finding and see if it helps.

And see how that goes. Then, I'll report back and we can go from there 😄

[MatMaul]

@gruijter can you show me an example of a request using the cookie ? I can't make it works while it looks good to me. Getting the cookie I managed to do but I can't manage to use it.

[gruijter]

This is what I do for the header on each request. Only put in the cookie if you previously got one, or it will fail.

				SOAPAction: action,
				'Cache-Control': 'no-cache',
				'User-Agent': 'node-netgearjs',
				'Content-Length': Buffer.byteLength(message),
				Connection: 'Keep-Alive',
			};
			if (this.cookie) {
				headers.Cookie = this.cookie;
			}

[MatMaul]

@gruijter Awesome thanks it's working now, I saw your code but I misinterpreted it.
I was setting Set-Cookie and not Cookie in the request headers.
I need to clean that up, we'll see tomorrow.

[caffeinatedMike]

@MatMaul I'll make a couple of adjustments to my fork for you to review. That way I can redeem myself a little bit for making your life a living hell these last two days 😛

I'll switch to using request.Session to make the cookie easier to manage & test out the previously discussed fixes.

[gruijter]

Here an example of logout command with use of the cookie

************ SOAPLogout **************
POST http://routerlogin.net/soap/server_sa/ HTTP/1.1
Accept: text/xml
SOAPAction: urn:NETGEAR-ROUTER:service:DeviceConfig:1#SOAPLogout
Cache-Control: no-cache
Pragma: no-cache
Accept-Encoding: gzip
Accept-Language: zh-CN,en,*
Content-type: multipart/form-data
User-Agent: SOAP Toolkit 3.0
Cookie: jwt_local=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOiIxMDY2OTkiLCJpc3MiOiJ3d3cubmV0Z2Vhci5jb20iLCJzdWIiOiJTT0FQIFRvb2xraXQgMy4wIn0=.9d780d44a9d2d7fa95758fd9b3b4f37f1b8a60984ebb058f331344821a4e5163
Content-Length: 531
Host: routerlogin.net
Connection: Keep-Alive

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<SOAP-ENV:Envelope xmlns:SOAPSDK1="http://www.w3.org/2001/XMLSchema" xmlns:SOAPSDK2="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAPSDK3="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP-ENV:Header>
<SessionID>A7D88AE69687E58D9A00</SessionID>
</SOAP-ENV:Header>
<SOAP-ENV:Body>
<M1:SOAPLogout xmlns:M1="urn:NETGEAR-ROUTER:service:DeviceConfig:1">
</M1:SOAPLogout>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

===========================
HTTP/1.0 200 OK
Content-Length: 385
Content-Type: text/xml; charset="UTF-8"
Server: Linux/2.6.15 uhttpd/1.0.0 soap/1.0

<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope
   xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
   SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<SOAP-ENV:Body>
<m:SOAPLogoutResponse xmlns:m="urn:NETGEAR-ROUTER:service:DeviceConfig:1"></m:SOAPLogoutResponse>
<ResponseCode>000</ResponseCode>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

[MatMaul]

I have the code working guys thanks. @caffeinatedMike if you want I push my branch and you can clean up my mess, and handle gracefully the 2 different ways to login.

[MatMaul]

@caffeinatedMike here it is:
https://github.com/MatMaul/pynetgear/tree/new-login

It's clean but the fallback to the old login method is missing.

[caffeinatedMike]

Yay! I'll have a look as soon as I'm home (about half hour from now).

[MatMaul]

Ok give me 10mn adding some missing retry after failing that I removed accidentally.
I'll tell you.

[caffeinatedMike]

Ok, I'm home now. So, I'll start fiddling whenever you get done.

[MatMaul]

Done. The code is clean, you have the login_v1 and login_v2 methods, you just need to put some try code in login.

[caffeinatedMike]

Awesome! I'm on it.

[MatMaul]

@caffeinatedMike if you have some free time after your setup works you can try implementing and test new methods to mess with the router settings, like SetBlockDeviceByMAC. It should be a matter of copy-pasting and adapting some stuff from the existing ones.

There are some pointers here and you can have more details in @gruijter js code.
#20

[caffeinatedMike]

I'll see what I can do!

[MatMaul]

Seems like you found your glasses ;-)

[MatMaul]

Hint: so your tests are meaningful you can do v1 then v2, to test the fall-back mechanism since your router fails with v1. Then once the retry logic is tested please put v2 as the first method to try. I can't easily test myself, both works here.

[caffeinatedMike]

I'm failing even with v2 atm :(

[caffeinatedMike]

Bahahahahahahahahahaha. @MatMaul you know what the issue's been all along? You never set the Content-Type header in the _get_soap_headers function 😆

Added it, then 💥 Device List showed in IDLE/Shell

[caffeinatedMike]

Pull request submitted 😃

To be honest, there's a lot that can be made more efficient in the module, but we can save that for another day 😉

About how long do you estimate these changes will take to make their way to the master?

[caffeinatedMike]

@gruijter At second look, my captured genie calls also use multipart/form-data. So, I suspect the Content-Type will work with both. I'm thinking the header just needs to be set really. But, I'll switch the type later, test on my python version and I'll also try your node library and let you know :)

@MatMaul Of course, I'll test all that when I get home from work today and resubmit the PR tonight after work :)

[caffeinatedMike]

@MatMaul Here's the updated commit to the PR in case you're able to test it within the next 8 hours. If you don't get around to it that's fine, I'll test when I get home in that case.

[caffeinatedMike]

@MatMaul I have tested the changes I made and I'm happy to say they are all functional. So, you can merge that PR whenever you have the time 😄

@gruijter I've been trying to run your test for you, but I keep failing. First, I'll point out one issue I was able to correct: you have an extra comma on line 22 process.argv[5],. Removing it got rid of the error. However, after that I receive a SyntaxError: Unexpected token function, which points to line 26 async function getRouterInfo() {

[caffeinatedMike]

I've been messing around with it in HA by copying the netgear component over into the custom_components directory, then adding my local pynetgear module inside there. Changed the import statements to point locally and been discovering/renaming my devices. So, I say so far so good :)

[gruijter]

@MatMaul I have tested the changes I made and I'm happy to say they are all functional. So, you can merge that PR whenever you have the time 😄

@gruijter I've been trying to run your test for you, but I keep failing. First, I'll point out one issue I was able to correct: you have an extra comma on line 22 process.argv[5],. Removing it got rid of the error. However, after that I receive a SyntaxError: Unexpected token function, which points to line 26 async function getRouterInfo() {

Hi @caffeinatedMike The comma is supposed to be there. If you remove it you get the other error :) What error were you getting before? Could you please try again (and install the new package that gives more info and captures the http response. > npm i netgear)

[caffeinatedMike]

@gruijter Unfortunately I just got into work again, so that means I won't be able to test until 8 hours from now at the earliest. Sorry >.<

[starkillerOG]

@MatMaul, I also have problems with pynetgear not beeing able to login to my router.
Everything is working in version 0.4.1, but after I upgraded homeassistant and got version 0.5.0 of pynetgear it failed to login to my R7000 router.

Has the fix for this issue already been merged in release 0.5.1?
I see it in the branch "new-login", but I don't see it in the master branch...