kriechi/aws-s3-reverse-proxy

reverse-proxy all incoming S3 API calls to the public AWS S3 backend

aws-s3-reverse-proxy's Issues

noPrometheusMetrics should be PrometheusMetrics

It appears that your test for NoPrometheusMetrics needs to be "true" in order for the port to be created (i.e., the logic is reversed in main.go).

Thanks for writing a useful proxy for DirectConnect folk!

X-Amz-Date header missing or set multiple times

We have a Django Application ( Netbox) which uses the Django backend for S3.
This doesnt set headers but instead puts things into the URL which leads to this error:

X-Amz-Date header missing or set multiple times: &{GET /netbox-test/devicetype-images/2021-03-02_13_28_22-ASR-9910.jpg_700800.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=OMKK45MUP3A21NUFBAMO%2F20210901%2Ffx%2Fs3%2Faws4_request&X-Amz-Date=20210901T194621Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=8cd5167be8363d44b437e2ddb9b09dcb7e2622375e3122d18c370ca7b7ff1a4b HTTP/1.1 1 1 map[Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9] Accept-Encoding:[gzip, deflate, br] Accept-Language:[en-GB,en-US;q=0.9,en;q=0.8,de;q=0.7] Cookie:[CONSENTMGR=consent:true|ts:1622119709217; wt_ttv2_s_196380495960676=9999; SAPWP_active=1; wt3_sid=%3B943593591056212%3B700378032410688%3B896369902135262; utag_main=v_id:017b9e117cb60019cfa8f2fe36a202069008406101788$_sn:2$_ss:0$_st:1630497112174$ses_id:1630495283764%3Bexp-session$_pn:8%3Bexp-session; wt3_eid=%3B943593591056212%7C2162972062014115676%232163049134992127594%3B700378032410688%7C2162998334043534258%232163049531157297740%3B896369902135262%7C2162998334087666506%232163049531287865764; wt_rla=943593591056212%2C20%2C1630491121687%3B700378032410688%2C5%2C1630495284161%3B896369902135262%2C4%2C1630495284251; _oauth2_proxy=tNL3RjEJLxE0V-3hZk0_BQNt_kCNCCTQJHs7zeNpPEDRAiwa-QoWtQysdpspA_MLS5L9DAJQv7O8WDJ0I9lQzyQYoGfgoE142AyJsyVTVsjLuFIxyQW1JhVyuptBhN3lkUArNHlN15826fVZSbP2KXOGOb3UQj4LsOQmCVunaxeyeETs3kTqbigGz_SAi1Q7tgs8VZovKJUgeUovCXwla3Gjh8ZfD25GptALAyuhmlZJuxyKmDZ9qeSlr44IC0liz6rAsZ6c5SoB3Mg8dLcxR3wLjZgKrbhF9UAwX8E3p_HFXSZJqoYkLJPj7bd7DlTjLIcysfFq__laq7nfeHk805w2ZsrKVOYpF1UUL6sx6i-DJx-Kc0LMqXjb0KfOkxnjNhwPNwF_EBrEteyKZISVOEZTYiK60FJas6GIWgs9paZq8Vvn1wV8--NF9AFLouPrL78hAX0DKVgkqOhfcUYrazs_HHfXqKv4T_7vjUOqZmHO3YBdcmtCN0CfhRJejAUw1bpW-aH4k9FkI8x-imuq68cHUPu9xEQfOVBmZ4ta69JKqcW6uDl20X89VWMt0MDs5_jJnQQuYloEhwsYFQ3RuBTbnDgEjHhP1pLVLth8UW65KGNPlhTwypYzV-hnpvelvP_zELqEDjwd26dvZmBVJA4f5pVTCSefbjxOfDLopmsXKd5pj7hRqgC_pg7vBeWk-sesA5V1Ew-Io9IwzIBT9TGrtsssxSyz5U4QLsYI1YQw4cqU4rGf8NinGecBiSsUM91pTbKdsv_fIunzrsBEUQLmSnUnf2SESCIR3FU-nJcv0cMyTI1US2fBScpPB5h6KLd6PHg2fJf_CJvSdiPt-7PXdMjMeBfI37SOBkpQ-RYz_ShbzxBS54ZkbR_gHjqlIAPYH4L9DTPNcNwCCMUjTInYHfwaBgWbtocnbUh1W5l2XKLaUclcYHyObS2wchcf6PtYH4AGLC_K_Lov2eRuwgg6z3KTNg2Xr8phZ5fNry273qdf-W-ThMZAZIkl_GjsbwqOMpJzmKMzeXFiAxV2zWpLevVnA0HtsJdrjFgpvEcSIRjDf79f2noiY-EvpXHk-3Yy-dkip4ngA_4mzjk42grIsqQf0T4qpmzU9WOsjpcASKVDRq1Kg-FsZHxALOS6x06fXJkR_V6M7uEBZZuy_cL3r1XfDRcAMHaDpv5dCmYO5SIdTOEhAAuFPqCsIfussi77fqXmhG9RzjW8G2Ap_iX_LJbbqdaOuYFTIajPeFCjhmzBEvEM5ULl8qDKFYTaFwNoRinwjzrQblwK71Iv76e3CzEQQH5J9Xh3CjXxU6kkYynHEAfQgy9al6lFHEFGLKAqwvK6WZTmrx1yqvdhICcsu2s4K4Y-R0mOHIYYlz_zzNpGHlBGRZPFfiY0bx1UPLEbrrmj6xdXJNPYZim3n1H3z6ocJ_9J4v_1DohNlCdfIkvTm9i9x0n--1Ouy_QUy2TyUsD-6m0l_REuJePOILygxQ5T9RuRf3nQsl3P_GFQ2z2YrmCLpp7sAx61n5pldKPYzuLLgeoT7GizyQCmZW8d5_2R9mFGoZFVUVA9wA1pA3oEyj4rtKDvqp061pVamaknE9p427kJA5mlckRj5RZxMnKLyYUFA6HRO-rlEfT8xlDxMaNM8JAHGr4QyZO5GaXF5D9-6E_7RTVvM2v7Jz12lXYAmK6TxuV0XwRdCNYPjJfG-kXghXmPf1unNuMQpxTyBr6Nohd4CbJchuN95B1kz8_9KZTrDEe8zlSGIzAFVMF7jPPrVrM93jpaTbKphQ567zuQzVTUX0g4cRpwktJZzpefgfXbSSXIeUMzwAZ9lwBrUuOtsXqdGrTpd8iPma85FPwHvgQ8Fv0seco_AT-gZERjtWZJS5MdW01oMYbvHmDKTGv7gGg6ignjeYXw4vuWEooFQVT_sqQGDyRHf9X0-foqO0dIDFRiB0E2f1_Lu2VIBFNY8JS79WD0kWxo7cSIvyLniG1F7Ky0GnQ3V_XhlNxy8HdNoBOjhSTAt3vYMI_hP8IjNgd9K_DT9dIqTcd5kKMwsrLpEjKU9ygw99oeorZjohFblnQ1Yrs9Df-t3nR6kC8mCrIWDTMCEji7_Gr27VPHdYjXcFGbVugUe4gUkX3HBQUgpOQxKkJYBih2APBO2oTjuWFWVs3HKuTmfpOUe3RpMyC37khbRTx8GA6QPwbH1dtKPA0-mCXQ0k7P0XdizKM9OPoWJGsy3cKVy7scq9YkGRtPg0oCF89O21M6OFXK1LSzD36ZB5RJYJKIiauFOyQa3IupRYbPg85xm2enSk_lr9tG8dO1aBG30bHDq_X_y1SLzuG4Ax8hznFygFe10U4vopZ1X1MQC4tV1gVugmnIZQ2EXg_ScZr9kJA2ffWHwu2CsWBLPYK3HK4s4PukeOkQJVCUOoD3nZ3lgZLyu4f3w5a41zw7-6SQ8xc86FuPs_k6ZzNnCPgf0uNa6k95JelzPX-Y2LUBWkKTLl7jYVq4sJDrHmZ5Gy5JhO-EkGYxoUiYEy5BNQYKsJ9EBLa3g2k6gG2J4J7dAt-Tgi_cz6GwhHeTzxlbuuUeBgqt3ME_VrAfzJhqHwp87XW52X3ua03vhFLzckZnDAt2D1rdc2kET4gCV837K9AsR-pHzrgO-91LPw9xVq469C3UBPOMSRUWIZ-T-qG-ovkpN97z8SGwCuhZEfOnq_bN20JciT-THbC0szzLBE5kpNe0VPZ4HoOn_tjyMhCLI8WVn4xjoiqzEvc3OOOnkrinU9qUPRoNMXEvQBMdDAoswumhjwYqstFJ-qoAiduBFYpoVvB31iCTbyxfr2OaOWGA4CTaFdVXnU8XWgrDUkvPsF4xC4VgXbEs7sDAM1tOv0_E6Xc0yIqX4m9_lSJpfqcT-x0xTWfEUA6nUZUo49mIlc5HdAq_rEBvCWuGwuw_gaE_CNl4DJEIcnD1PXlp-kCyOktW4WZtGJ7GO3FZaLScVL_JRqdLlX1_jP-ExT9Fy15iKFrfjwzU80lrYAioOfzHw-rgWxxEWJu9DvqN0Qeu0ws59jnZjBCXhbnUle1SV5vTuVqv970vT4QZlkUgm9DtChw77q_voMwxg7rcx1FIKhzlh_jDqJgPvUis9Kb2K_O5k7OCm0iTPxYmBKRLflsOVgviBfudQMnu39EMi1J_O-2CTJBYSbQifVlvHPNdUmJh0CaqJFmhJ60UvLYyjf27A0mRwbe_zxUy81Aywr5pyN0JW6ZZEDU14nFUN1MyNZ3GvJEoWGokym1dbulsNkBPm5sgyAzyui7Icf-f5M9BkePbWoJMdMI7fmIjnL0suXxnR5KPeSTHclvzc-qIEhp-TlJ-da8GOouKzMC2s2bIRzTqkpSEkl6Gh7py369imYRl2AslaKVZCghZ3P_MYtD0l1ig1MnLcDwCqgH_jLG2fAOh7cZXA0slpF37pKNFIu3pZ1XS1a5nce3Gvzu4w--Q5SJbyD99hgjRJgY3Id7oZNGgvnQqOokPc2cnnRYvw9fke-8b_QCZWj_bfVIO376P|1630509542|9Kw5U7pvkklkmlVsk8pX2txQTPjl-H9TBpBeR-5hy8Y=; csrftoken=4jGnRUwzXAORCKRbEduMCRHBsStKM6l5Y3hQCFffudRVgKcqAnMP813Q8QQioYzP; sessionid=mqz6u2cu16xowpwk0kte09g60ksiad56] Referer:[https://netbox.hb.das-schiff.telekom.de/dcim/device-types/25/] Sec-Ch-Ua:[" Not A;Brand";v="99", "Chromium";v="92"] Sec-Ch-Ua-Mobile:[?0] Sec-Fetch-Dest:[document] Sec-Fetch-Mode:[navigate] Sec-Fetch-Site:[same-origin] Sec-Fetch-User:[?1] Upgrade-Insecure-Requests:[1] User-Agent:[Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36] X-Forwarded-For:[10.27.50.159] X-Forwarded-Host:[netbox.hb.das-schiff.telekom.de] X-Forwarded-Port:[443] X-Forwarded-Proto:[https] X-Real-Ip:[10.27.50.159] X-Request-Id:[0ac451fcb2b561c5bbfcc1aabafdbdfc] X-Scheme:[https]] {} <nil> 0 [] false netbox.hb.das-schiff.telekom.de map[] map[] <nil> map[] 192.168.90.70:37558 /netbox-test/devicetype-images/2021-03-02_13_28_22-ASR-9910.jpg_700800.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=OMKK45MUP3A21NUFBAMO%2F20210901%2Ffx%2Fs3%2Faws4_request&X-Amz-Date=20210901T194621Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=8cd5167be8363d44b437e2ddb9b09dcb7e2622375e3122d18c370ca7b7ff1a4b <nil> <nil> <nil> 0xc000396700}

Unable to use tool behind another Reverse Proxy

We wanted to use this proxy in Kubernetes, behind an Ingress Controller.
This seems to cause some issues,as im unable to make a connection:

time="2021-09-01T20:12:57Z" level=debug msg="Incoming request: GET /netbox-test?list-type=2&prefix=&delimiter=%2F&encoding-type=url HTTP/1.1\r\nHost: netbox.hb.das-schiff.telekom.de\r\nAccept-Encoding: identity\r\nAuthorization: AWS4-HMAC-SHA256 Credential=OMKK45MUP3A21NUFBAMO/20210901/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=df917a72082c74b48178fb1aaf8d89662b7857855f66527a68101ea2905c11ab\r\nUser-Agent: aws-cli/1.20.21 Python/3.9.6 Linux/5.14.0-1-MANJARO botocore/1.21.21\r\nX-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\nX-Amz-Date: 20210901T201256Z\r\nX-Forwarded-For: 10.27.50.159\r\nX-Forwarded-Host: netbox.hb.das-schiff.telekom.de\r\nX-Forwarded-Port: 443\r\nX-Forwarded-Proto: https\r\nX-Real-Ip: 10.27.50.159\r\nX-Request-Id: bc05d3323e143d19a0340e9bfb9c0e84\r\nX-Scheme: https\r\n\r\n"
time="2021-09-01T20:12:57Z" level=error msg="unable to proxy request" error="invalid signature in Authorization header"

URL Encoding causes signature to be incorrect

A URL that is coming in as http://myproxy.com/year=2020/month=04/day=28/hour=20/test.txt is being changed to http://myproxy.com/year%3D2020/month%3D04/day%3D28/hour%3D20/test.txt causing the signature not to be correct. I corrected this in my copy in handler.go adding the line proxyURL.RawPath = *&req.URL.Path in the assembleUpstreamReq function.

Does it work with vultr?

Vult now offers S3 buckets, does it work with it?