barnacs/compy

HTTP/HTTPS compression proxy

compy's Issues

Compy fails to parse the compressed images

Description:
Some websites compress the images for transmission, yet Compy fails to recognize it can make an error when displaying the image.

Reason:
Compy can't recognize the compressed images.

Suggested fix:
Compy may use the ``content-encoding'' field in the response header to determine the compression methods and further decompress the images before display.

Minifier plugin triggers a Content encoding error if content is compressed with brotli by the target server

Hi,

For a reason unknown to me, compy triggers Content encoding errors on the client when both the browser and the target server support brotli, and when the minifier plugin is enabled.

Known failure cases which triggers Content encoding errors are:

• https://wp.gc.niqex.net
• https://mad.ninja
• https://google.com
• https://youtube.com

One solution to circumvent the issue is to mask br when passing the Accept-encoding headers, if the minify plugin is enabled.

The more elegant one would be to check why the content isn't encoded back in brotli when processed by the minify plugin.

Kind regards,

x509: requested SignatureAlgorithm does not match private key type

Getting the error message when trying to open some HTTPS web pages.
one example is aao25.com page does not load on client and message in compy log x509: requested SignatureAlgorithm does not match private key type.

Anyone got any ideas how to resolve?

TLS Oversized record

I tried to set up a Compy with HTTPS and MitM support, and I connected my iPhone to it but I keep getting these two errors whenever I try to load a page. I don't understand what I did wrong but i'm hoping you can help me.

http: TLS handshake error from 192.168.1.74:49989: tls: oversized record received with length 20037
http: TLS handshake error from 192.168.1.74:49994: tls: first record does not look like a TLS handshake

Group command-line flags

Presently compy sorts all its command-line flags such that related flags like cert and key are not adjacent. We may need to switch parsers for this.

Let's Encrypt instructions for .pem files?

I have generated Let's Encrypt files on a test server, where I have these .pem files to work with:

$ cat ./README

This directory contains your keys and certificates.

`privkey.pem`  : the private key for your certificate.
`fullchain.pem`: the certificate file used in most server software.
`chain.pem`    : used for OCSP stapling in Nginx >=1.3.7.
`cert.pem`     : will break many server configurations, and should not be used
                 without reading further documentation (see link below).

WARNING: DO NOT MOVE OR RENAME THESE FILES!
         Certbot expects these files to remain in this location in order
         to function properly!

We recommend not moving these files. For more information, see the Certbot
User Guide at https://certbot.eff.org/docs/using.html#where-are-my-certificates.

Which of these are compatible/should be used with the various cert/ca/key flags or the software itself of compy?

Diagnostics endpoint not working

I saw diagnostic endpoint in #22 but can't get it to work.

I keep getting following:

$ curl https://localhost:9999/ -kv
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 9999 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 592 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* 	 server certificate verification SKIPPED
* 	 server certificate status verification SKIPPED
* 	 common name: localhost (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: CN=localhost
* 	 start date: Wed, 18 Apr 2018 18:20:01 GMT
* 	 expire date: Sat, 15 Apr 2028 18:20:01 GMT
* 	 issuer: CN=localhost
* 	 compression: NULL
* ALPN, server accepted to use http/1.1
> GET / HTTP/1.1
> Host: localhost:9999
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Date: Wed, 18 Apr 2018 18:20:37 GMT
< Content-Length: 0
<
* Connection #0 to host localhost left intact

and in logs:

2018/04/18 18:20:37 serving request: /
2018/04/18 18:20:37 http: TLS handshake error from 127.0.0.1:47946: tls: first record does not look like a TLS handshake
2018/04/18 18:20:37 error forwarding request: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02" while serving request: http://localhost:9999/

I'm using docker version of compy without any special config.

One weird thing I've noticed is "while serving request: http://localhost:9999/" in logs even though I'm hitting proxy itself as https (because it works in HTTPS proxy mode).

Transcoding video streams?

It would be a nice feature to transcode video streams into lower resolutions or more efficient codecs.

Examples?

Can create any examples e.g. to intercept on MITM and do something with the payload, like refusing a forward.

Update install instructions / deps? Install currently fails.

Hi,
I tried installing using the latest version of Go on a Raspberry Pi. The instructions now require "go install [url]@Version", which fails at:

github.com/chai2010/webp

go/pkg/mod/github.com/chai2010/[email protected]/webp.go:22:9: undefined: webpGetInfo
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:26:20: undefined: webpDecodeGray
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:39:20: undefined: webpDecodeRGB
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:52:20: undefined: webpDecodeRGBA
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:68:14: undefined: webpDecodeGrayToSize
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:82:14: undefined: webpDecodeRGBToSize
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:96:14: undefined: webpDecodeRGBAToSize
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:109:7: undefined: toGrayImage
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:110:14: undefined: webpEncodeGray
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:119:14: undefined: webpEncodeRGB
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:119:14: too many errors
github.com/pixiv/go-libjpeg/jpeg: build constraints exclude all Go files in /root/go/pkg/mod/github.com/pixiv/[email protected]/jpeg
gopkg.in/kothar/brotli-go.v0/dec: build constraints exclude all Go files in /root/go/pkg/mod/gopkg.in/kothar/[email protected]/dec
gopkg.in/kothar/brotli-go.v0/enc: build constraints exclude all Go files in /root/go/pkg/mod/gopkg.in/kothar/[email protected]/enc

Note that I had to install libjpeg-9 instead of libjpeg-8 as the latter is now unavailable - would that cause the above issues? I'd love to use this project, but I can't tell if it's being maintained or not.

Add Chrome and Firefox client side instructions

Configuring the SSL certificates can confuse users. The README or wiki should have detailed instructions for how to establish both SSL trust to compy and the CA trust for MitM.

Some sites inaccessible, 502/403 errors

There are various sites I have found that cannot be accessed using this proxy with unhelpful error messages such as bad request or invalid reponse. I'm not a software developer so have had no luck tracking down what the issue is. If anyone gets 5 minuted can you look at
https://www.old-maps.co.uk/
or
https://www.realvnc.com/connect/download/vnc/windows/

Thanks, Jon.

Enhancement request: local http/https port change option

I'm running compy on my LAN machine, which also hosts a website with nginx via HTTP and HTTPS. If I try to access compy's local webpage where the total savings are displayed, nginx interferes with compy and nothing is displayed.

I guess an option to change the port where the proxy is listinening to HTTP/S connections would be the solution.

Just my two cents.

HTTP authentication

While running compy I found that others were using my instance and generated 1.6 GB of traffic. compy should include HTTP authentication to prevent this.

How could this be installed on OpenWRT/LEDE?

I am very interested in this project! It sounds awesome! I have no idea why it isn't more popular.

Anyways, I am unfamiliar with Go, so how I could I install this on a OpenWRT/LEDE?

How resource intensive is it? My router is has a 1.3 GHz dual-core with 512 MB of RAM.

Google App Engine

Deploying in GAE would allow users to run compy without allocating an entire VM for this purpose.

Need HTTPS authentication functionality

If I run compy with mitm support and supply username and password, on my remote computer browser and phone, whenever I try to access a site, a dialog pops up asking for username and password and then errors out. It say ERR_UNEXPECTED_PROXY_AUTH.
Note: authentication only works with http sites and not https

Transcode to AVIF

This format can reduce file size even more than WebP:

https://en.wikipedia.org/wiki/AV1#AV1_Image_File_Format_(AVIF)

Firefox 77 already has support behind a feature flag:

https://bugzilla.mozilla.org/show_bug.cgi?id=1625363

verbose logging

compy should support a --verbose flag so that it emits HTTP requests and responses like curl does. We should also attach some kind of request id since the current logging causes confusion with concurrent requests.

better Dockerfile

Are you interested in a better more up to date and slimer Dockerfile? I am currently building one and would contribute it if you like.

Some sites give certificate error when using mitm

Most of https sites work ok, but some of them fail with invalid certificate error. Here's an example list:
https://vk.com
https://youtube.com/
https://support.google.com/chrome/answer/6098869

TLS handshake error with http2

The newest commit seems to be not working with https connections.
Errors outputed from compy:

2016/03/28 15:06:03 http: TLS handshake error from *IP*:54255: EOF
2016/03/28 15:06:03 http: TLS handshake error from *IP*:54256: EOF
2016/03/28 15:06:03 http: TLS handshake error from *IP*:54257: EOF
2016/03/28 15:06:04 http: TLS handshake error from *IP*:54258: EOF
2016/03/28 15:06:04 http: TLS handshake error from *IP*:54259: EOF
2016/03/28 15:06:04 http: TLS handshake error from *IP*:54260: EOF

IP is hereby my locals computer IP.
With 6dae9bd the same setup was working.
Since a4a691b it no longer works.
Chrome shows ERR_SSL_PROTOCOL_ERROR

Command used for running compy:
./compy -ca ca.crt -cakey ca.key

EOF using docker image

Hello! Wishing to use the docker image that's available but getting EOF

#2

Same issue.

Compy can't deal with the `websocket`, which fails some interaction tasks like chatrooms and online services.

Description:
Compy can't deal with the websocket, which fails some interaction tasks like chatrooms and online services.

Suggested fix:
Compy can simply add a condition for websocket and directly forward the data in the websocket to the end users.

Enhancement request: add custom http headers

Add option via http headers to configure compression levels (gzip and brotli) and separate minification toggles for html, css, json, js

Some files become larger after compression/transcoding

I was testing Compy on some http websites and I am getting some strange numbers for the "amount transcoded" report. Example:
2017/10/03 07:53:27 transcoded: 53 -> 77 (145.3%)
2017/10/03 07:53:27 transcoded: 61 -> 74 (121.3%)
2017/10/03 07:52:56 transcoded: 61 -> 74 (121.3%)
2017/10/03 07:52:56 transcoded: 69 -> 93 (134.8%)
2017/10/03 07:52:28 transcoded: 0 -> 23 (+Inf%)

@gaul mention:

Very small HTML and images can increase in size due to overhead of compression metadata and possibly different formats have different overhead, e.g., GIF vs. WebP.

Perhaps a conditional statement could be included, such as "if x is this small or smaller then bypass transcoding/compression, else transcode/compress"?

Command line parsing issues

This is by no means critical, but I guess it is better to document it. If I do:

compy -cert cert.crt -key cert.key -ca ca.crt -cakey ca.key -jpeg 40 -gif false -gzip 9 -brotli 11 -minify true -host :1234

compy ignores the -host option:

2017/12/13 00:28:23 compy listening on :9999

But if I do
compy -cert cert.crt -key cert.key -ca ca.crt -cakey ca.key -host :1234 -jpeg 40 -gif false -gzip 9 -brotli 11 -minify true

The proxy now listens on port 1234

I think the command line is not correctly parsing the command line options on some situations.

Endpoint serving diagnostics

compy could serve diagnostics like its configuration, statistics, and more importantly the CA certificate needed to enable MitM support. The latter would make installing the root certificate more convenient on mobile:

https://mtersch.wordpress.com/2015/03/17/certificate-import-in-firefox-on-android/

Travis CI

Now that compy has unit tests via #12, we should set up Travis to prevent regressions.

Issues with certificates and Docker

Hi there,

I would be super excited to test the project but I'm having real trouble to set it up.

What I did:

• Installed the docker on a remote server with a domain name
• Tried to install all the certificates created by the docker in Keychain and Firefox (it still doesn't trust my DOMAIN:9999 though 😢 )
• Add a PAC File to Firefox via URL
• Every Request is dropped by Firefox (HSTS or just not Secure).

I get following log on the Docker side:

http2: server: error reading preface from client 37.167.90.155:49700: remote error: tls: bad certificate

Is there something obvious that I'm missing or what extra information can I provide you in order that you could help me?

Dockerfile error

when i try to deploy on heroku .
there error in line.
RUN /usr/local/go/bin/go get -d -v ./...
it returned a non-zero code: 1

How to use? Windows server install?

How to install it?

Add Dockerfile

This will make compy easier for non-programmers to use.

Compy may block the redirecting process of some websites

Description:
Compy may block the redirecting process of some websites

Reason:
Some websites may use user-agent to present different contents to different users, yet Compy doesn't attempt to forge the user's request.

Suggested fix:
Compy can record the UA when forwarding the user's data packets if necessary.

mitm: crypto/x509: system root pool is not available on Windows

Hi,
I'm trying to use compy with TLS and MitM support: (on Windows Server 2012 R2)
> compy -cert cert.crt -key cert.key -ca ca.crt -cakey ca.key
and:
not using mitm: crypto/x509: system root pool is not available on Windows
2018/03/04 08:01:11 compy listening on :9999

Compiler: go1.10.windows-amd64 & MinGW_x86_64-7.2.0-release-posix-seh-rt_v5-rev1

100% CPU load

Sometimes after heavy loads; one of child processes continues 100% loading and does not come back to normal load! (even if i'm not using it).

Client-side settings

Compy should allow clients to configure their compression settings via HTTP request headers, e.g., X-Compy-Quality. This is similar to the implicit configuration that the Accept and Accept-Encoding headers provide. Firefox and Chrome extensions allow modification of headers:

• https://addons.mozilla.org/en-us/firefox/addon/modify-header-value/
• https://chrome.google.com/webstore/detail/modheader/idgpnmonknjnojddfkpgkljpfnnfcklj?hl=en

Using Compy Locally

Hi

I tried setting up compy to serve compressed assets locally and I'm running into issues (probably with the way I'm using it).

Steps to reproduce

1. Create a simple HTML file containing anything you want.
2. Serve the file via a simple server (I use live-server which can be installed via npm) running on port 8080.
3. Run compy on the same port (8080).

Error

2020/04/20 13:45:03 compy listening on :8080
2020/04/20 13:45:03 listen tcp :8080: bind: address already in use
james@james-pc:~/Desktop/testing-compy$

Setup

OS: Ubuntu 18.04
Go version: 1.14.1

mitm issue

When i run with mitm enabled, i get this:

# compy -ca ca.crt -cakey ca.key
not using mitm: open : no such file or directory
2018/03/11 12:11:32 compy listening on :9999

OS is Debian 9 x64 and go 1.10

WebP recompression

compy should allow recompressing WebP images with a given quality setting.

Install failed "jpeglib.h"

I tried to install compy to my Raspberry Pi 3B with Rasbian Stretch and Go 1.10.1 but got this error:

pi@BM-Raspi-Server:~ $ go get github.com/barnacs/compy
# github.com/pixiv/go-libjpeg/jpeg
go/src/github.com/pixiv/go-libjpeg/jpeg/compress.go:6:21: fatal error: jpeglib.h: No such file or directory
 #include "jpeglib.h"
                     ^
compilation terminated.

How to fix it?

Also tried to install libjpeg9 and libjpeg-turbo but neither works.

Compy goes wrong when compressing some images, which makes the images unable to load.

Description:
Compy goes wrong when compressing some images, which makes the images unable to load.

Reason:
This is because Compy uses the content-type'' field in the HTTP response header to judge the type of the file. When some websites mistake thecontent-type'', Compy goes wrong!

Suggested fix:
A typical solution to this big is that Compy should judge the file type from the content (e.g., the header of the file) rather than merely using the content type.

Homebrew package

Homebrew could provide a package to make installation and upgrades easier for macOS users. We might need to start tagging releases for user-friendliness.

Enhancement request: GIF to WebM convert

Before compy I used RabbIT with ffmpeg, and convert GIF more than 30kb to WebM format. This allows you to leave GIF animated when the size is significantly reduced. Can you add similar functional to compy?

TLS error handshake

using latest commit, i get this error.
2017/06/09 07:50:01 http: TLS handshake error from 41.66.203.167:41837: read tcp 45.62.242.187:3130->41.66.203.167:41837: read: connection reset by peer 2017/06/09 07:50:01 http: TLS handshake error from 41.66.203.167:57673: read tcp 45.62.242.187:3130->41.66.203.167:57673: read: connection reset by peer 2017/06/09 07:50:03 http: TLS handshake error from 41.66.203.167:41127: read tcp 45.62.242.187:3130->41.66.203.167:41127: read: connection reset by peer 2017/06/09 07:50:04 http: TLS handshake error from 41.66.203.167:47938: read tcp 45.62.242.187:3130->41.66.203.167:47938: read: connection reset by peer 2017/06/09 07:50:05 http: TLS handshake error from 41.66.203.167:56411: read tcp 45.62.242.187:3130->41.66.203.167:56411: read: connection reset by peer

Content Type image/jpg is not recongized as JPEG

Some servers serve JPEG images with content type image/jpg instead of image/jpeg (the e).

The transcoder should catch these as well (compy.go):

        if *jpeg != 0 {
                tcjpeg :=  tc.NewJpeg(*jpeg)
                p.AddTranscoder("image/jpeg", tcjpeg)
                p.AddTranscoder("image/jpg", tcjpeg)
        }

@barnacs do you agree and if so do you want a pull-request for this change?

Need feature like timeout to prevent ram loss

I think compy has not any feature to empty unsued memory. For example if i do not use compy for hours or days it would take hundreds of megabytes even in idle state!

real data savings?

Hello, I've successfully setup compy on my computer. But, does it offer data savings as seen by the ISP when run on a local server? I used 127.0.0.1:9999 on my browser. Or should I run compy on a cloud server to provide data savings?

certificate error

I have installed the compy on centos 6.8. Everything I did followed the instructions, but I cannot use MitM method.

The compy was launched by command:
compy -cert cert.crt -key cert.key -ca ca.crt -cakey ca.key

and the compy continues showing the following errors:
remote error: tls: unknown certificate authority

I have added the generated ca.cert to Firefox.
Please help me to solve this problem.