tootallnate/node-http-proxy-agent

An HTTP(s) proxy `http.Agent` implementation for HTTP endpoints

Note: The code that used to be here now lives in the proxy-agents monorepo.

node-http-proxy-agent's Issues

Property 'maxTotalSockets' is missing in type 'HttpProxyAgent' but required in type 'Agent'.

I get this error with @types/node version 12.19.2.

Stuck when pipe or request.write()

const fs = require('fs');
const url = require('url');
const https = require('https');
const HttpProxyAgent = require('http-proxy-agent');

let requestOptions = url.parse('https://127.0.0.1');

requestOptions.method = 'POST';
requestOptions.agent = new HttpProxyAgent('http://127.0.0.1:3000'); // <--- If comment this - all works fine.

let request = https.request(requestOption, (res) => {
res.pipe(process.stdout);
});

fs.createReadStream(__dirname + '/test.txt').pipe(request); // <--- Didn't work. request.write('Hello') didn't work too. Just stuck.

// Working only request.end('Hello'), but sometimes i need send files and use Readable streams.
// p.s. socks-proxy-agent working very fine with request.write('Hello') and .pipe(request).
// p.s.s. 127.0.0.1 - just for example.

404 Not Found tootallnate%2fonce from version 4.0.1

npm ERR! code E404
npm ERR! 404 Not Found - GET http://*****/@TooTallNate%2fonce - no such package available
npm ERR! 404
npm ERR! 404 '@tootallnate/once@1' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404 It was specified as a dependency of 'http-proxy-agent'
npm ERR! 404
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

npm ERR! A complete log of this run can be found in:
npm ERR! C:\build1\build_dca\rbuild.npm_logs\2020-02-13T08_53_45_450Z-debug.log

Protocol error when used with websocket

Hi,

When I use http proxy agent with websocket over https, protocol set is http.
I add this line in index.js in 68 :
if (parsed.protocol == null) parsed.protocol = parsed.port === 443 ? 'https:' : 'http:';
It's work fine.

Was it a good solution or did I do something wrong?

Jérôme

acccording to snyk old version of debug has ddos vulnerability. https as well

Testing /usr/lib/node_modules/npm...
✗ Low severity vulnerability found on [email protected]
- desc: Regular Expression Denial of Service (ReDoS)
- info: https://snyk.io/vuln/npm:debug:20170905
- from: [email protected] > [email protected] > [email protected] > [email protected] > [email protected]
Your dependencies are out of date, otherwise you would be using a newer debug than [email protected].
Try deleting node_modules, reinstalling and running `snyk test` again.
If the problem persists, one of your dependencies may be bundling outdated modules.

✗ Low severity vulnerability found on [email protected]
- desc: Regular Expression Denial of Service (ReDoS)
- info: https://snyk.io/vuln/npm:debug:20170905
- from: [email protected] > [email protected] > [email protected] > [email protected] > [email protected]
Your dependencies are out of date, otherwise you would be using a newer debug than [email protected].
Try deleting node_modules, reinstalling and running `snyk test` again.
If the problem persists, one of your dependencies may be bundling outdated modules.

Tested 358 dependencies for known vulnerabilities, found 1 vulnerability, 2 vulnerable paths.

setting auth: property on target

For some reason the contents did not save:

I needed to send a request where the target.auth was set to a value. I was wondering if it would to pass in an
options {
target: url,
auth: austhString
}

and have the target.auth property set to the options.auth passed in.

I need to POST data to number of different remote severs (which for legacy reasons) don't have CORS enabled and a single proxy.on('proxyReq', ..) handler doesn't really cut it.

Also options.auth would differ by API request

app.post('/csc/broker/storage/:tenant/:id', function(req, res){
app.post('/Hi5Cloud/ContentManagement/uploadFile/:token/:tenant/:id', function(req, res){
etc

function storageRequest(url, auth, req, res) {
    var opts =  {
        target: url,
        auth: auth
    };
    proxy.web(req, res, opts);

}

right now the options block that is used for proxy is something like teh block below with the auth:null (which gets passed along to the target and the connect fails)

{ target:
   { protocol: 'https:',
     slashes: true,
     auth: null,
     host: 'host',
     port: null,
     hostname: 'hostname',
     hash: null,
     search: null,
     query: null,
     pathname: '/ ... path ...'
     path: '/... path .../',
     href: ' https:// ... href ... ' 
},
prependPath: true }

I also not that even though I set https as the protocol to use, port:null

Please release new module to remove self-signed certs in pacakge from agent-base

https://github.com/TooTallNate/node-http-proxy-agent/tree/master/test

connect ETIMEDOUT

i copied and pasted the example code into my server file and I still cannot figure out why i'm getting a connect timeout issue.

Denial of Service high severity vulnerability for https-proxy-agent version 2.1.0

please help!

Exported API does not match README

In README this library is used as a class:

import ProxyAgent from 'http-proxy-agent';

new ProxyAgent(options);

In typescript definition the export is a function:

import createHttpProxyAgent from 'http-proxy-agent';

createHttpProxyAgent(options);

http.globalAgent

Hello!

Can we replace http.globalAgent with new instance of HttpProxyAgent ?

How to use this package to connect our crawler to Tor

Hi all,

I have a crawler to crawl pages and extract hyperlinks. It currently works well for common pages like google.com and I want to use it to crawl pages on Tor. I found this package and am interested in using it in my code to enable the crawler to work on Tor pages. My code is as follows:

var cheerio = require('cheerio');
var Crawler = require("simplecrawler");
var fs = require('fs');
var readline = require('readline');
var stream = require('stream');


var txtFile = "./test.txt";
var temp = [''];
var tree = {};
var readArr = [];
var i = 0;
var crawler = Crawler("http://www.torlinkbgs6aabns.onion/");

function configuringRobot(){
        
    crawler.timeout=3000000;
    crawler.listenerTTL=100000;
    /crawler.proxyPort=9050; //Tor proxy
    crawler.maxDepth=3;
    crawler.allowInitialDomainChange=false; //let the crawler to fetch links accessible from other domains.
    crawler.filterByDomain=false; //let the crawler to fetch links accessible from other domains.
    crawler.scanSubdomains=true;
    crawler.stripQuerystring=true; //strip queries to be fetched
    crawler.maxConcurrency=5;
    crawler.ignoreWWWDomain=false;
    crawler.acceptCookies = false;
}

function callingRobot(file, crawler){

    crawler.discoverResources = function(buffer, queueItem) { //to fetch hyperlinks
        var $ = cheerio.load(buffer.toString("utf8"));

        return $("a[href]").map(function () {
            return $(this).attr("href");
        }).get();

    };

    crawler.addDownloadCondition(function(queueItem, referrerQueueItem, callback) {
        i = i + 1;
        callback(null, i < 1000);
    });
    
    crawler.addFetchCondition(function(queueItem, referrerQueueItem, callback) {
        // We only ever want to move one step away from example.com, so if the
        // referrer queue item reports a different domain, don't proceed
        callback(null, referrerQueueItem.host === crawler.host);
    });
    
    crawler.on("fetch404", function(queueItem, responseObject)
    {
        console.log("URL: " + queueItem.url, "emits 404 Error");   
    });
    crawler.on("fetchcomplete", function(queueItem, responseBuffer, response) {
        console.log("I just received %s (%s status)", queueItem.url, queueItem.status);
        //console.log("It was a resource of type %s", response.headers['content-type']);
        var depth_key = queueItem.depth.toString();
        if (!(depth_key in tree)){
            tree[depth_key] = [];
        }
        tree[depth_key].push({parent: queueItem.referrer, child: queueItem.url});
        temp.push(["Child: "+ queueItem.url, "Parent: "+ queueItem.referrer]);
        //tempr.push(queueItem.referrer);
        fs.writeFileSync(file, temp);
        fs.writeFileSync(file, '\n');
        fs.writeFileSync(file, temp);
    });
    
    crawler.on("complete", function(){
        console.log("Final temp object: ");
        console.log(temp);
        console.log("Final tree:");
        console.log(tree);
    });
    
 
    crawler.start();
    
    
}// function f



configuringRobot();
callingRobot(txtFile, crawler);

Could you please let me know how to do this?

{ error: [Error: Hostname/IP doesn't match certificate's altnames] }

I have got this error when trying to connect to own https server with self-signed server.
How to disable certificate check?

Keepalive support?

How can I configure the http proxy agent to support keepalive? including sending the Proxy-Connection: Keep-Alive header?

note - I'm only interested in HTTP traffic and pooling sockets to the proxy server for http-forwarding requests. For https I'm already using https://github.com/mknj/node-keepalive-proxy-agent

License file?

Currently the license is embedded in the readme, can we have a separate license file?

Node 14: Not work in Termux (Mobile network 4G) ... with WiFi work ok.

Node 14: Not work in Termux (Mobile network 4G) ... with WiFi work ok.

without node-http-proxy-agent - work in both.

(node-fetch, axios tested)

for https - version.

Http(s)ProxyAgent is not compatible with node 14 Agent

Using the Http(s)ProxyAgent fails with the following error.

error TS2741: Property 'freeSockets' is missing in type 'HttpsProxyAgent' but required in type 'Agent'

This is due to the fact that the Http(s)ProxyAgent extends a custom Agent, that is not based on the node http.Agent.