VPN Client Gateway Stuck on Requesting

[mr-canoehead]

VPN Client Gateway Stuck on Requesting

[mr-canoehead]

Just a quick update to let you know that I have fixed Issue #49; the project now works with the latest Raspbian Buster release, the 'Gateway status' section of the management web page is being populated properly.

Cheers,
Chris

[mr-canoehead]

Good question - I have only done full fresh installs when testing (following the install guide), never tried patching an existing system. I suppose you could download the latest master branch and then install the files that have changed since the last version.

Note: these instructions are off the top of my head, I have not tested them!

• copy the modified server app:

sudo cp vpn_client_gateway-master/application/vpncgw.py /opt/vpncgw/vpncgw.py

• install the modified systemd unit file:

sudo service vpncgw stop
sudo cp vpn_client_gateway-master/config/vpncgw.service /etc/systemd/system/vpncgw.service
sudo systemctl daemon-reload
sudo service vpncgw start

• copy the modified web page and javascript files:

sudo cp vpn_client_gateway-master/www/index.html /var/www/html/index.html
sudo cp vpn_client_gateway-master/www/scripts/main.js /var/www/html/scripts/main.js

• install the modified NGINX site configuration file:

for a setup with only the VPN Client Gateway installed:

sudo unlink /etc/nginx/sites-enabled/vpncgw.conf
sudo cp vpn_client_gateway-master/config/vpncgw.conf /etc/nginx/sites-available/vpncgw.conf
sudo ln -s /etc/nginx/sites-available/vpncgw.conf /etc/nginx/sites-enabled/vpncgw.conf
sudo service nginx restart

or for a combined Pi-hole + VPN Client Gateway setup:

sudo unlink /etc/nginx/sites-enabled/pihole-vpncgw.conf
sudo cp vpn_client_gateway-master/config/pihole-vpncgw.conf /etc/nginx/sites-available/pihole-vpncgw.conf
sudo ln -s /etc/nginx/sites-available/pihole-vpncgw.conf /etc/nginx/sites-enabled/pihole-vpncgw.conf
sudo service nginx restart

I think that should cover all the changes, all the prerequisites (Flask, gunicorn, eventlet, etc.) should already be installed and I don't think I added any new dependencies. Keep an eye on /var/log/syslog for any errors when starting the vpncgw service or when using the management web interface, if something is missing it will show up in the log.

[mr-canoehead]

Yes, that is easy to do using a rewrite rule in the site configuration file. For example, if I don't want the VPN Client Gateway html file to be the default index.html, I can rename it to vpncgw.html and then tell NGINX to forward all request for the URL http://10.0.1.2/vpngateway to http://10.0.1.2/vpncgw.html

Here's how I tested it on my setup:

• rename the html file:

sudo mv /var/www/html/index.html /var/www/html/vpncgw.html

• add a rewrite rule to the site configuration file, the rule looks like this:

location /vpngateway {
	rewrite ^/vpngateway$ /vpncgw.html;
}

• restart the NGINX service:

sudo service nginx restart

I have attached my site configuration file so that you can see where the rule should be placed inside the server block:

pihole-vpncgw.conf.txt

[mr-canoehead]

Whoops, figured out what is causing your problem: after I merged the dev branch back into the master branch my local copy was a few commits ahead of the repository. I have now pushed the changes, the github repo is up to date with all my changes.

Sorry about that!

On the plus side, I was able to test the patch instructions I posted previously in this thread, they work fine (except for one minor typo which I've since fixed). So, if you download the latest snapshot of the master branch and patch your install using the instructions above, it should solve the problem.

[mr-canoehead]

If you add the line 'management 0.0.0.0 7505' (without quotes) to your /etc/openvpn/server.conf file and then restart the openvpn service 'sudo service openvpn restart' it will enable the OpenVPN management interface, which will allow the vpncgw_monitor script to query the vpn service state.

Yes, you can host multiple web pages using your existing NGINX server configuration. It shouldn't cause any problems unless you clobber one of the existing VPN Client Gateway project files.

[mr-canoehead]

Does anything interesting show up in /var/log/syslog when you start the vpncgw service? Common issues are:

1. one or more Python modules are missing (can happen during install if pip times out while installing a module)

2. the application can't find the current VPN server ('remote' setting in /etc/openvpn/server.conf) in the server list (/opt/vpncgw/vpnservers.xml). Did you replace the standard server list with the Mullvad server list (copy /opt/vpncgw/vpn_providers/mullvad/vpnservers.xml to /opt/vpncgw/vpnservers.xml). Note that I created the Mullvad list some time ago, they may have added/removed/changed server names since then. Unfortunately I don't have a Mullvad account to test it with.

[markiljas]

Thank you for the suggestions! I've stopped and started the vpncgw service again, then checked the /var/log/syslog:

Aug 27 23:44:49 jeeves systemd[1]: Stopping VPN Client Gateway...
Aug 27 23:44:49 jeeves env[566]: [2019-08-27 23:44:49 +0000] [566] [INFO] Handling signal: term
Aug 27 23:44:49 jeeves env[566]: [2019-08-27 23:44:49 +0000] [807] [INFO] Worker exiting (pid: 807)
Aug 27 23:44:49 jeeves env[566]: [2019-08-27 23:44:49 +0000] [566] [INFO] Shutting down: Master
Aug 27 23:44:49 jeeves systemd[1]: vpncgw.service: Succeeded.
Aug 27 23:44:49 jeeves systemd[1]: Stopped VPN Client Gateway.
Aug 27 23:44:57 jeeves systemd[1]: Starting VPN Client Gateway...
Aug 27 23:44:59 jeeves env[14764]: [2019-08-27 23:44:59 +0000] [14764] [INFO] Starting gunicorn 19.9.0
Aug 27 23:44:59 jeeves env[14764]: [2019-08-27 23:44:59 +0000] [14764] [INFO] Listening at: http://0.0.0.0:8000 (14764)
Aug 27 23:44:59 jeeves env[14764]: [2019-08-27 23:44:59 +0000] [14764] [INFO] Using worker: eventlet
Aug 27 23:44:59 jeeves env[14764]: [2019-08-27 23:44:59 +0000] [14774] [INFO] Booting worker with pid: 14774
Aug 27 23:45:00 jeeves systemd[1]: Started VPN Client Gateway.

1. During the install there were no time outs, maybe on one or two occasion it reported that something was already installed (I believe on this one: sudo pip install Flask)

2. I've replaced the list during the install, but only this file:
   /var/www/html/vpnmgmt/vpnservers.xml
   not this: /opt/vpncgw/vpnservers.xml
   I've copied the server list to that folder as well now, but the problem remains. The server that Mullvad is currently connected to is the first one in the server list in the file (Sweeden): se-sto.mullvad.net

I also checked the server list and their ports, updated the 8 ones that were to be listed in the gateway, saved it like that in both files and rebooted the pi. However, the issue remained, it doesn't even show the flags or checklists, just everything empty (+requesting data) with the blue loading cog.

I think that's all, hopefully enough to shed some light on the matter. Thanks!

[markiljas]

I reverted to my old backup before installing everything in step 3, just to test will I reach the same error. I've been extra careful to do everything exactly as the guide suggests and I've even went through the Mullvad server list and updated all the ports to their most recent ones (I can share it here if you think it might help other people who use Mullvad), and I'm happy to report the gateway works! The only thing that is stuck loading is the gateway status (blue buffering thing keeps spinning), but it shows my current VPN server and I am able to switch to the others.

But what I'd like to ask is is it possible to set a static website and domain via NGINX without messing up the Gateway and PiHole? I'm using dynamic DNS over Cloudflare and I need my domain to point to my static website. Should I move Gateway to some other address (i.e. 10.0.1.2/gateway)? If so, how would be best to do it? Or is there a better way to achieve the desired result?

Thanks in advance!

P.S.
Also, is it possible to have 6 or 9 VPNs in the Gateway list? Or does it have to be exactly 8 for it to work properly? Didn't want to edit to test, since it works now, so didn't want to mess it up :)

[mr-canoehead]

1. Gateway status not loading (blue buffering thing keeps spinning): which OS are you running - what is the output of the following commands?
   cat /etc/os-release
uname -a

2. Static website / dynamic DNS / Cloudflare etc. Sorry, this is beyond the scope of what I'm currently able to test & support - you'll have to experiment on your own.

3. 6 or 9 VPNs in the Gateway list: Yes, you can have as many servers listed in the basicvpnservers section of vpnservers.xml as you like, the management web page is built dynamically based on the contents of the file. Note that each entry in the basicvpnservers section must have a corresponding entry in the vpnservers section.

[markiljas]

1. Currently running a backup, so can't access the terminal, but I know that I am running a latest Buster (fresh install from few days ago). I can post the outputs later if needed.

2. I understand, doing backup now so I'll experiment with that later.

3. Great!

Thank you very much!

[mr-canoehead]

'Gateway status section not loading' is a known issue when running on Buster (Issue #49), I haven't been able to figure out why WebSockets isn't working on Buster. Everything works just fine with the previous Raspbian release (Stretch).

Since there's already an open issue for that, I'm closing this one.

[markiljas]

Great, thanks for the update! How do I apply this fix, I tried to figure it out, but not exactly certain? Do I have to reinstall some part or? Thanks!

…

On 24 Sep 2019, at 04:25, Chris ***@***.***> wrote: Just a quick update to let you know that I have fixed Issue #49 <#49>; the project now works with the latest Raspbian Buster release, the 'Gateway status' section of the management web page is being populated properly. Cheers, Chris — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#51?email_source=notifications&email_token=AKNB6NG7OY2ZFKDDNUZLJG3QLF3CVA5CNFSM4IQFNA2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7M3CKY#issuecomment-534360363>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKNB6NAZ2XRZ6XH6XBGOZSTQLF3CVANCNFSM4IQFNA2A>.

[markiljas]

Wow, great! Thank you very much for these instructions, I’ll get on it right away. Worst comes to worst, I think I have a backup file before installing following the guide, so I can revert and install a clean copy. Just wanted to ask one more thing, I know it is not covered by the guide, but if by any chance it is not too much to ask – is there a way to map the gateway to a different address than the IP address of the Pi? I.e. 10.0.1.2/vpngateway instead of just 10.0.1.2? Thanks!

…

On 25 Sep 2019, at 00:49, Chris ***@***.***> wrote: Good question - I have only done full fresh installs when testing (following the install guide), never tried patching an existing system. I suppose you could download the latest master branch and then install the files that have changed since the last version. Note: these instructions are off the top of my head, I have not tested them! copy the modified server app: sudo cp vpn_client_gateway-master/application/vpncgw.py /opt/vpncgw/vpncgw.py install the modified systemd unit file: sudo service vpncgw stop sudo cp vpn_client_gateway-master/config/vpncgw.service /etc/systemd/system/vpncgw.service sudo systemctl daemon-reload sudo service vpncgw start copy the modified web page and javascript files: sudo cp vpn_client_gateway-master/www/index.html /var/www/html/index.html sudo cp vpn_client_gateway-master/www/scripts/main.js /var/www/html/scripts/main.js install the modified NGINX site configuration file: for a setup with only the VPN Client Gateway installed: sudo unlink /etc/nginx/sites-enabled/vpncgw.conf sudo cp vpncgw_client_gateway-master/config/vpncgw.conf /etc/nginx/sites-available/vpncgw.conf sudo ln -s /etc/nginx/sites-available/vpncgw.conf /etc/nginx/sites-enabled/vpncgw.conf sudo service nginx restart or for a combined Pi-hole + VPN Client Gateway setup: sudo unlink /etc/nginx/sites-enabled/pihole-vpncgw.conf sudo cp vpncgw_client_gateway-master/config/pihole-vpncgw.conf /etc/nginx/sites-available/pihole-vpncgw.conf sudo ln -s /etc/nginx/sites-available/pihole-vpncgw.conf /etc/nginx/sites-enabled/pihole-vpncgw.conf sudo service nginx restart I think that should cover all the changes, all the prerequisites (Flask, gunicorn, eventlet, etc.) should already be installed and I don't think I added any new dependencies. Keep an eye on /var/log/syslog for any errors when starting the vpncgw service or when using the management web interface, if something is missing it will show up in the log. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#51?email_source=notifications&email_token=AKNB6NFRZYIMPWWUYHX5TVLQLKKOTA5CNFSM4IQFNA2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7QBBEY#issuecomment-534778003>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKNB6NBFTF3LSLUVAWX5Q3TQLKKOTANCNFSM4IQFNA2A>.

[markiljas]

Great, that did it, remapped the gateway to 10.0.1.2/vpngateway successfully, thanks! However, even though I followed instructions for the update of the Gateway Status to the letter, it still behaves in the same way. I’ve checked the logs and the only thing I could find is that a file is missing (as seen in screenshot below), should I just copy it from the master? Could that be the reason? Sorry to nag, just realised that my backup point was a bit further down the road so it would take me several hours to set up everything again, so trying to find a way to patch this up if possible. Thanks!

…

On 25 Sep 2019, at 02:34, Chris ***@***.***> wrote: Yes, that is easy to do using a rewrite rule in the site configuration file. For example, if I don't want the VPN Client Gateway html file to be the default index.html, I can rename it to vpncgw.html and then tell NGINX to forward all request for the URL http://10.0.1.2/vpngateway <http://10.0.1.2/vpngateway> to http://10.0.1.2/vpncgw.html <http://10.0.1.2/vpncgw.html> Here's how I tested it on my setup: rename the html file: sudo mv /var/www/html/index.html /var/www/html/vpncgw.html add a rewrite rule to the site configuration file, the rule looks like this: location /vpngateway { rewrite ^/vpngateway$ /vpncgw.html; } I have attached my site configuration file so that you can see where the rule should be placed inside the server block: pihole-vpncgw.conf.txt <https://github.com/mr-canoehead/vpn_client_gateway/files/3650008/pihole-vpncgw.conf.txt> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#51?email_source=notifications&email_token=AKNB6NGBNJZFLOTSZDCWI3TQLKW3HA5CNFSM4IQFNA2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7QGSBY#issuecomment-534800647>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKNB6NF5HRYNVETOWAY3ML3QLKW3HANCNFSM4IQFNA2A>.

[markiljas]

Sorry for replying late, it was night over here :) That makes sense, I was wondering why all the repository was several months old, thought that maybe it didn’t refresh. However, I used the instructions from before with the new master branch, while also being careful to apply the change for gateway address before starting it up, but now the gateway is completely empty (screenshot below, added your address that you gave so you could see). I don’t see anything really off in syslog, but posting it as well in case you see something. Also, a little above in the log it sometimes mentions: systemd[1]: vpncgw.service: Can't open PID file /run/vpncgw/vpncgw.pid (yet?) after start-post: No such file or directory I did the patch three times, just in case I missed something (I’m using gateway+pihole setup), but it still shows up empty for some bizzare reason. However, I did notice that it still opens the (blank) gateway on both 10.0.1.2 and 10.0.1.2/vpngateway. If you have any suggestions on how to fix this, it would help tremendously!

…

On 25 Sep 2019, at 06:51, Chris ***@***.***> wrote: Whoops, figured out what is causing your problem: after I merged the dev branch back into the master branch my local copy was a few commits ahead of the repository. I have now pushed the changes, the github repo is up to date with all my changes. Sorry about that! On the plus side, I was able to test the patch instructions I posted previously in this thread, they work fine (except for one minor typo which I've since fixed). So, if you download the latest snapshot of the master branch and patch your install using the instructions above, it should solve the problem. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#51?email_source=notifications&email_token=AKNB6NGVC5NR3YYKHPMIEW3QLLU4FA5CNFSM4IQFNA2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7QSYEI#issuecomment-534850577>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKNB6NEOSQ46IZOFD47SDADQLLU4FANCNFSM4IQFNA2A>.

[markiljas]

Also, sending you na updated VPN server list for Mullvad if you want to include it. I went and checked each of them so I could update the list, thought it might be useful for someone using Mullvad as well. It mostly needed new port information :)

…

On 25 Sep 2019, at 06:51, Chris ***@***.*** ***@***.***>> wrote: Whoops, figured out what is causing your problem: after I merged the dev branch back into the master branch my local copy was a few commits ahead of the repository. I have now pushed the changes, the github repo is up to date with all my changes. Sorry about that! On the plus side, I was able to test the patch instructions I posted previously in this thread, they work fine (except for one minor typo which I've since fixed). So, if you download the latest snapshot of the master branch and patch your install using the instructions above, it should solve the problem. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#51?email_source=notifications&email_token=AKNB6NGVC5NR3YYKHPMIEW3QLLU4FA5CNFSM4IQFNA2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7QSYEI#issuecomment-534850577>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKNB6NEOSQ46IZOFD47SDADQLLU4FANCNFSM4IQFNA2A>.

[markiljas]

Update: I tried opening the client gateway on Firefox (usually I am on Safari), it's working fine. It even works on mobile Safari, but for some reason doesn't work on desktop version. Previously it has worked normally on Safari (except the requesting part), but now it is stuck on loading as in the screenshots I've sent. I've tried opening it in private window on Safari and it works fine, so I assume it is some cookie or something like that that needs to be deleted? I've tried removing it from history and removing my IP address from Website Data, but with no luck... Any idea how to fix this little detail?

Also, for some reason, when I view it it has green ticks next to all the elements except VPN state, which has a grey minus and it says n/a, even when I have selected a VPN from a specific state (i.e. USA, NY). Is that normal?

Thanks!

[mr-canoehead]

If the web page loads but is not being populated (i.e. the Current and New VPN server sections show "requesting data...") it could be a browser cache problem; on my desktop if I change the HTML, CSS, or Javascript for the page I need to force the browser to reload the page from the server instead of from the local cache. In Chrome & Firefox I do this with Ctrl+F5, not sure how it's done with Safari.
The "VPN state" refers to the state of the VPN connection (connected, connecting, authenticating, etc.) not a geographic territory. If that is not being populated, it could be that the vpn status file is not being generated (/run/vpncgw/vpncgw_status.json), or the script that queries the OpenVPN management interface isn't able to do so. The management interface is enabled via the 'management 0.0.0.0 7505' setting in /etc/openvpn/server.conf, if it is enabled you should be able to test it by connecting via the console on the Pi:
nc localhost 7505
You should a message like "INFO: OpenVPN Management Interface Version 1"
Note: use the 'exit' or 'quit' command to exit the management interface, otherwise it will block any other connections (including the script that tries to query it).

[markiljas]

It seems to be working properly now on all browsers, which is awesome! Thank you for helping me patch this. The only thing that I wasn't able to figure out is the "VPN state" part.

I've tried running the "nc localhost 7505" command in the terminal while connected to Pi but nothing happens, it just goes into the next input line. I've also went ahead and checked the two files you've mentioned, but didn't find any trace of 'management 0.0.0.0 7505'.

Not really sure what to do, I know this is relatively minor issue, since it is working, so if it's too complicated to debug, it doesn't matter, but if you think of something, cool!

Also, a short sidenote: now that the gateway is at 10.0.1.2/vpngateway, does that mean I can make another NIGNX site at 10.0.1.2? Or will it conflict with the gateway?

/run/vpncgw/vpncgw_status.json:

{
    "status_datetime": "1569538066",
    "currentserver": {
        "servername": "ch.mullvad.net",
        "enabled": true,
        "serverport": "1302"
    },
    "system": {
        "dns_ok": "True",
        "inet_ok": "True",
        "cpu_temp": "64.5",
        "cpu_load": "0.0",
        "mem_usage": "42.0"
    },
    "openvpn": {
        "service": "active"
    }
}

/etc/openvpn/server.conf:

client
dev tun0
proto udp
remote ch.mullvad.net 1302
cipher AES-256-CBC
resolv-retry infinite
nobind
persist-key
persist-tun
comp-lzo
verb 3
remote-cert-tls server
ping 10
ping-restart 60
auth-user-pass auth.txt
ca mullvad_ca.crt
tun-ipv6
script-security 2
tls-cipher TLS*****
route 170.178.190.196 255.255.255.255 net_gateway

The last line in the server.conf is for dynamic DNS, so disregard it :)

Thanks a bunch!

[markiljas]

Awesome, the line worked like a charm! Now everything is reading nicely.

Great, good to know, wanted to try hosting another web page on the default local address of Pi (10.0.1.2), but thought maybe best to check so I don't break anything.

Thank you very much for your help :)

[markiljas]

Tried a couple of times to make a static site, but each time I crash the gateway, so decided to ask, maybe I'm missing something stupidly obvious. I'm trying to make a simple static site that will be running along gateway site and link my domain to it. However, after I create a conf file (or edit the default in sites-available) and link it to sites-enabled, the gateway either stops working or the site is inaccessible (or both).

My guess is that I'm somehow "clobbering" one of the existing VPN Client Gateway project files as you said it yourself, so I was wondering if you could share some insight. Or what would be the correct steps to make a static site that will run along the VPN Client Gateway?

It would be of immense help if you could share some insight, I'm going crazy over this trying to figure it out without crashing everything.

Thanks!

[mr-canoehead]

If you create two site configuration files with servers listening on the same port then yes there will be a conflict; one of the server threads will fail to start because it can't bind to the same port (the port is already in use by the first server thread that starts). There are several options for configuring a static site, you could just put a static html file in the existing web root directory. For example, if I create the html file 'hello_world.html' and put it in /var/www/html I can then access the static web page via the url http://10.1.2.50/hello_world.html

Another option is to add another 'server' block to the existing site configuration file (or a separate site configuration file) with a different port number, e.g.:

server {
listen 8080 default_server;
etc...

This would tell NGINX to start another server thread listening on port 8080, which does not conflict with the existing server thread listening on port 80. With this configuration, you would access the static web page via the host name / IP + port number, e.g. http://10.1.2.50:8080/hello_world.html

You'll need to refer to the NGINX documentation for more information about configuring web servers as my experience is limited to a single internal (i.e. not internet-facing) site.

[markiljas]

I see, thank you for taking the time to write this up! The different port thing might be where I messed up, I'll try to go down that route and see the documentation. I was more worried if I unknowingly messed up something from the Gateway files that I needed to be wary of.

In any case, thanks!