Giter Site home page Giter Site logo

chill117 / proxy-verifier Goto Github PK

View Code? Open in Web Editor NEW
76.0 4.0 16.0 2.65 MB

NodeJS module to check proxies: if functional, anonymity level, tunneling, supported protocols.

License: MIT License

JavaScript 100.00%
proxies checking-proxies nodejs

proxy-verifier's Introduction

proxy-verifier

Build Status

Check that proxies are working, verify their anonymity level (transparent, anonymous, or elite), lookup their geographic location by IP address, check for other capabilities such as tunneling and available protocols (HTTP/S, SOCKS4/5).

Installation

Add to your application via npm:

npm install proxy-verifier --save

This will install proxy-verifier and add it to your application's package.json file.

API

testAll

testAll(proxy[, options], cb)

Runs all test methods for the given proxy. The options argument is passed through to every test method.

Usage:

var ProxyVerifier = require('proxy-verifier');

var proxy = {
	ipAddress: '127.0.0.1',
	port: 8888,
	protocol: 'http'
};

ProxyVerifier.testAll(proxy, function(error, result) {

	if (error) {
		// Some unusual error occurred.
	} else {
		// The result object will contain success/error information.
	}
});

Sample result:

{
	anonymityLevel: 'elite',
	protocols: {
		http: {
			ok: true
		}
	},
	tunnel: {
		ok: true
	}
}

testProtocol

testProtocol(proxy[, options], cb)

Check that the proxy works with the specified protocol. The options argument is passed through to the request() method which uses request.

Usage:

var ProxyVerifier = require('proxy-verifier');

var proxy = {
	ipAddress: '127.0.0.1',
	port: 8080,
	protocol: 'http'
};

ProxyVerifier.testProtocol(proxy, function(error, result) {

	if (error) {
		// Some unusual error occurred.
	} else {
		// The result object will contain success/error information.
	}
});

Sample result when the proxy is working:

{
	ok: true
}

Sample result when the proxy is not working:

{
	ok: false,
	error: {
		message: 'socket hang up',
		code: 'ECONNRESET'
	}
}

testProtocols

testProtocols(proxy[, options], cb)

Check that the proxy works with the specified protocols. The options argument is passed through to the request() method which uses request.

Usage:

var ProxyVerifier = require('proxy-verifier');

var proxy = {
	ipAddress: '127.0.0.1',
	port: 8080,
	protocols: ['http', 'https']
};

ProxyVerifier.testProtocols(proxy, function(error, results) {

	if (error) {
		// Some unusual error occurred.
	} else {
		// The results object contains a result object for each protocol.
	}
});

Sample results when the proxy is working for all its protocols:

{
	http: {
		ok: true
	},
	https: {
		ok: true
	}
}

Sample results when the proxy is not working for any of the protocols:

{
	http: {
		ok: false,
		error: {
			message: 'socket hang up',
			code: 'ECONNRESET'
		}
	},
	https: {
		ok: false,
		error: {
			message: 'socket hang up',
			code: 'ECONNRESET'
		}
	}
}

testAnonymityLevel

testAnonymityLevel(proxy[, options], cb)

Check the anonymity level of the proxy. The options argument is passed through to the request() method which uses request.

Usage:

var ProxyVerifier = require('proxy-verifier');

var proxy = {
	ipAddress: '127.0.0.1',
	port: 8080,
	protocol: 'http'
};

ProxyVerifier.testAnonymityLevel(proxy, function(error, anonymityLevel) {

	if (error) {
		// Some unusual error occurred.
	} else {
		// anonymityLevel will be a string equal to "transparent", "anonymous", or "elite".
	}
});

Anonymity levels explained:

  • transparent - The proxy does not hide the requester's IP address.
  • anonymous - The proxy hides the requester's IP address, but adds headers to the forwarded request that make it clear that the request was made using a proxy.
  • elite - The proxy hides the requester's IP address and does not add any proxy-related headers to the request.

testTunnel

testTunnel(proxy[, options], cb)

Check to see if the proxy supports HTTP tunneling. The options argument is passed through to the request() method which uses request.

Usage:

var ProxyVerifier = require('proxy-verifier');

var proxy = {
	ipAddress: '127.0.0.1',
	port: 8888,
	protocol: 'http'
};

ProxyVerifier.testTunnel(proxy, function(error, result) {

	if (error) {
		// Some unusual error occurred.
	} else {
		// The result object will contain success/error information.
	}
});

Sample result when the proxy supports tunneling:

{
	ok: true
}

Sample result when the proxy does not support tunneling:

{
	ok: false,
	error: {
		message: 'socket hang up',
		code: 'ECONNRESET'
	}
}

test

Use this method to create your custom tests. Example usage:

ProxyVerifier.test(proxy, {
    testUrl: 'https://www.google.com/?q=test',
    testFn: function(data, status, headers) {

        // Check the response data, status, and headers.

        // Throw an error if the test failed.
        throw new Error('Test failed!');

        // Do nothing if the test passed.
    }
}, function(error, results) {
    // Do something with error or results.
});

Contributing

There are a number of ways you can contribute:

  • Improve or correct the documentation - All the documentation is in this readme.md file. If you see a mistake, or think something should be clarified or expanded upon, please submit a pull request
  • Report a bug - Please review existing issues before submitting a new one; to avoid duplicates. If you can't find an issue that relates to the bug you've found, please create a new one.
  • Request a feature - Again, please review the existing issues before posting a feature request. If you can't find an existing one that covers your feature idea, please create a new one.
  • Fix a bug - Have a look at the existing issues for the project. If there's a bug in there that you'd like to tackle, please feel free to do so. I would ask that when fixing a bug, that you first create a failing test that proves the bug. Then to fix the bug, make the test pass. This should hopefully ensure that the bug never creeps into the project again. After you've done all that, you can submit a pull request with your changes.

Before you contribute code, please read through at least some of the source code for the project. I would appreciate it if any pull requests for source code changes follow the coding style of the rest of the project.

Now if you're still interested, you'll need to get your local environment configured.

Configure Local Environment

Step 1: Get the Code

First, you'll need to pull down the code from GitHub:

git clone https://github.com/chill117/proxy-verifier.git

Step 2: Install Dependencies

Second, you'll need to install the project dependencies as well as the dev dependencies. To do this, simply run the following from the directory you created in step 1:

npm install

Tests

This project includes an automated regression test suite. To run the tests:

npm test

Changelog

See changelog.md

License

This software is MIT licensed:

A short, permissive software license. Basically, you can do whatever you want as long as you include the original copyright and license notice in any copy of the software/source. There are many variations of this license in use.

Funding

This project is free and open-source. If you would like to show your appreciation by helping to fund the project's continued development and maintenance, you can find available options here.

proxy-verifier's People

Contributors

chill117 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

proxy-verifier's Issues

Cannot verify proxies

proxies-fetch.js (seems works ok)

var fs = require('fs');

var ProxyLists = require('proxy-lists');

var filename = 'proxies-unchecked.txt';

// Read current list of proxies
var proxiesAntidup = {};
if (fs.existsSync(filename)) {

    fs
        .readFileSync(filename, 'utf8')
        .split("\n")
        .forEach(function(proxy) {
            proxiesAntidup[proxy] = true;
        });
}

function parseProxies(protocol) {

    return new Promise(function(resolve, reject) {

        var options = {
            anonymityLevels: 'https' == protocol ? ['elite'] : null,
            protocols: [protocol],
            sourcesBlackList: ['bitproxies', 'kingproxies']
        };

        // `gettingProxies` is an event emitter object.
        var gettingProxies = ProxyLists.getProxies(options);

        gettingProxies.on('data', function(proxies) {

            // Received some proxies.
            proxies.forEach(function(proxy) {

                // Add proxy to the list
                proxy = proxy.ipAddress + "\t" + proxy.port + "\t" + protocol;
                if ( ! proxiesAntidup[proxy]) {

                    fs.appendFileSync(filename, proxy + "\n");
                    proxiesAntidup[proxy] = true;

                }

            });

        });

        gettingProxies.on('error', function(error) {
            // Some error has occurred.
            console.error(error);
        });

        gettingProxies.once('end', function() {
            // Done getting proxies.
            resolve();
        });

    });

}

Promise.all(['https', 'socks5'].map(function(protocol) {

    return parseProxies(protocol);

})).then(function() {

    process.exit(0);

});

proxies-check.js (has problems)

var fs = require('fs');

var ProxyVerifier = require('proxy-verifier');

var sourceFilename = 'proxies-unchecked.txt';
var outputFilename = 'proxies.txt';

if ( ! fs.existsSync(sourceFilename)) {

    process.exit(0);

}

// Read current list of proxies
var promises = fs
    .readFileSync(sourceFilename, 'utf8')
    .split("\n")
    .map(function(proxy) {

        proxy = proxy.split("\t");
        if ( ! proxy[0] || ! proxy[1] || ! proxy[2] ) {
            return;
        }

        var proxy = {
            ipAddress: proxy[0],
            port: proxy[1],
            protocol: proxy[2],
            protocols: ['https', 'socks5']
        };

        return new Promise(function(resolve, reject) {
            ProxyVerifier.testAll(proxy, function(error, result) {

                console.log(result);

                if (error) {
                    reject(error);
                    return;
                }

                if ('elite' == result.anonymityLevel && (result.protocols.https && result.protocols.https.ok || result.protocols.socks5 && result.protocols.socks5.ok)) {

                    proxy.protocol = result.protocols.socks5 && result.protocols.socks5.ok ? 'socks5' : 'https';
                    proxy.country = result.country;
                    resolve(proxy);
                    return;

                }

                reject();
            });
        });
    });

fs.unlinkSync(sourceFilename);

Promise
    .all(promises)
    .then(function(proxies) {

        proxies.forEach(function(proxy) {

            if (!proxy) {
                return;
            }

            fs.appendFileSync(outputFilename, JSON.stringify(proxy) + "\n");

        });

    }).then(function() {

        process.exit(0);

    }).catch(function(error) {

        if (error) {
            console.error(error);
        }

    });

I don't get any proxy. In console.log(result) all proxies are failed and process also hangs. Forgot how to use node-inspector or IDE debugger :(

gatherproxy crashed and server restart

Hi @chill117,
Today my app crash with below code adn server restarted:

W20160805-21:31:20.135(4.5)? (STDERR) /home/username/Projects/Proxy Assistant/Sources/node_modules/proxy-lists/sources/gatherproxy.js:313
W20160805-21:31:20.136(4.5)? (STDERR)       return cookie.substr(0, cookie.indexOf(';'));
W20160805-21:31:20.137(4.5)? (STDERR)                    ^
W20160805-21:31:20.137(4.5)? (STDERR) 
W20160805-21:31:20.137(4.5)? (STDERR) TypeError: Cannot read property 'substr' of undefined
W20160805-21:31:20.138(4.5)? (STDERR)     at Object.module.exports.getSessionCookie (/home/username/Projects/Proxy Assistant/Sources/node_modules/proxy-lists/sources/gatherproxy.js:313:16)
W20160805-21:31:20.138(4.5)? (STDERR)     at Request._callback (/home/username/Projects/Proxy Assistant/Sources/node_modules/proxy-lists/sources/gatherproxy.js:211:24)
W20160805-21:31:20.139(4.5)? (STDERR)     at Request.self.callback (/home/username/Projects/Proxy Assistant/Sources/node_modules/request/request.js:200:22)
W20160805-21:31:20.139(4.5)? (STDERR)     at emitTwo (events.js:87:13)
W20160805-21:31:20.139(4.5)? (STDERR)     at Request.emit (events.js:172:7)
W20160805-21:31:20.140(4.5)? (STDERR)     at Request.<anonymous> (/home/username/Projects/Proxy Assistant/Sources/node_modules/request/request.js:1067:10)
W20160805-21:31:20.140(4.5)? (STDERR)     at emitOne (events.js:82:20)
W20160805-21:31:20.140(4.5)? (STDERR)     at Request.emit (events.js:169:7)
W20160805-21:31:20.140(4.5)? (STDERR)     at IncomingMessage.<anonymous> (/home/username/Projects/Proxy Assistant/Sources/node_modules/request/request.js:988:12)
W20160805-21:31:20.141(4.5)? (STDERR)     at emitNone (events.js:72:20)
=> Exited with code: 1

I think manybe gatherproxy breaks. I think it's better to prevent server restart on error's like this or this.

Auth not working.

var ProxyVerifier = require('proxy-verifier');

const auth = btoa('user:pass');

var proxy = {
ipAddress: '208.202.250.44',
port: 8312,
protocol: 'http',
auth: Basic ${auth}
};

ProxyVerifier.testAll(proxy, function(error, result) {

if (error) {
console.log(error);
} else {
console.log(result);
}

});

How does this module works? Does it submit proxies to bitproxies.eu to SELL them?

Does this volume checks socks5 LOCALLY, or does it submit proxies to external service? As i see some references to external services in the code.

	_defaultTestUrl: 'http://bitproxies.eu/api/v2/check',
	_ipAddressCheckUrl: 'https://bitproxies.eu/api/v2/check',
	_tunnelTestUrl: 'https://bitproxies.eu/api/v2/check',

Bitproxies.eu:

Fast, easy API access to a large database of verified, active proxies (http, https, socks)

So could you clarify whether you submit proxies of library users to service where they're resold?

Add option to check a list of proxies in a .txt file

can you give me a index.js that does;

load proxies from pending.txt
check if site return 404, if it does log as success.txt

i am unsure wether the proxies are http/https also, so not sure what we could do bout that

Write tests

Write tests for the following proxy checks (verification methods):

  • ProxyVerifier.checks.protocols
  • ProxyVerifier.checks.protocol
  • ProxyVerifier.checks.anonymityLevel
  • ProxyVerifier.checks.country

Write tests for the following utility functions:

  • ProxyVerifier.request

References:

Add custom URLs check

Hi @chill117,
Is it possible to verify proxy based on specific URL?
For example check list for google-passed proxy or other specific URL.

Stop running check

proxy-verifier is ran concurrently using an array of promises against a larger list of proxy candidates. When enough working proxies have been found (e.g. just only 1), the other running proxy checks should be stopped while still running.

How to return result?

Help me please! I newbie in js. I need to return result of your check but i can`t.

Tried 1:

    let a = await ProxyVerifier.testProtocols(oneProxy,  async function(error, result) {
        return new Promise((resolve) => {
            return resolve(result);
        });
    });

Tried 2:

    let a = await ProxyVerifier.testProtocols(oneProxy,  async function(error, result) {
            return result;
    });

Tried 3:

let/var/const a = null;
    ProxyVerifier.testProtocols(oneProxy,  async function(error, result) {
            a  = result;
    });

Tried 4:

let a = {
    proxy: ProxyVerifier.testProtocols(oneProxy,  async function(error, result) {
            this.value = result;
    }),
   value: null
}

Maybe answer is simple, but i can`t understand how get result.

doesn't install

I've been trying to install the package but with npm later switched to yarn but doesn't really install. help.

Pass request option in Test function

Is it possible to set request options in Test function like testAll and other functions?
I can't see related note in docs. I want to increase request connection timeout value. Can you guide me about it?

Error: Callback was already called.

Hi @chill117,
I used this code and sometimes it's return below error and app server restarted at all:

W20160729-19:02:08.443(4.5)? (STDERR) /home/usename/Projects/Proxy Assistant/Sources/node_modules/proxy-verifier/node_modules/async/lib/async.js:43
W20160729-19:02:08.444(4.5)? (STDERR)             if (fn === null) throw new Error("Callback was already called.");
W20160729-19:02:08.444(4.5)? (STDERR)                              ^
W20160729-19:02:08.445(4.5)? (STDERR) 
W20160729-19:02:08.446(4.5)? (STDERR) Error: Callback was already called.
W20160729-19:02:08.446(4.5)? (STDERR)     at /home/usename/Projects/Proxy Assistant/Sources/node_modules/proxy-verifier/node_modules/async/lib/async.js:43:36
W20160729-19:02:08.446(4.5)? (STDERR)     at /home/usename/Projects/Proxy Assistant/Sources/node_modules/proxy-verifier/node_modules/async/lib/async.js:723:17
W20160729-19:02:08.446(4.5)? (STDERR)     at /home/usename/Projects/Proxy Assistant/Sources/node_modules/proxy-verifier/node_modules/async/lib/async.js:167:37
W20160729-19:02:08.447(4.5)? (STDERR)     at /home/usename/Projects/Proxy Assistant/Sources/node_modules/proxy-verifier/index.js:248:4
W20160729-19:02:08.447(4.5)? (STDERR)     at Request.<anonymous> (/home/usename/Projects/Proxy Assistant/Sources/node_modules/proxy-verifier/index.js:445:4)
W20160729-19:02:08.447(4.5)? (STDERR)     at emitOne (events.js:77:13)
W20160729-19:02:08.447(4.5)? (STDERR)     at Request.emit (events.js:169:7)
W20160729-19:02:08.448(4.5)? (STDERR)     at Request.onRequestError (/home/usename/Projects/Proxy Assistant/Sources/node_modules/proxy-verifier/node_modules/request/request.js:820:8)
W20160729-19:02:08.448(4.5)? (STDERR)     at emitOne (events.js:77:13)
W20160729-19:02:08.448(4.5)? (STDERR)     at ClientRequest.emit (events.js:169:7)
=> Exited with code: 1

This is my code:

            // proxy format: http://192.168.1.1:8080 (Protocol://IP:PORT)
            let proxyParts = proxy.proxy.split(":");
            let protocol = proxyParts[0].toLowerCase();
            let ipAddress = proxyParts[1].replace("//", "");
            let port = proxyParts[2];

            let proxyObj = {
                ipAddress: ipAddress,
                port: port,
                protocol: protocol
            };
            ProxyVerifier.testAll(proxyObj, Meteor.bindEnvironment((error, result) => {

                if (error) {
                    // Some unusual error occurred.
                } else {
                    // anonymityLevel will be a string equal to "transparent", "anonymous", or "elite".
                    if (result.anonymityLevel === "elite" && result.protocols[proxyObj.protocol].ok) {
                        Proxies.update({_id: proxy._id}, {$inc: {point: 1}});
                    } else {
                        Proxies.update({_id: proxy._id}, {$inc: {point: -1}});
                    }
                }
            }));

Do you have any suggestion?? My code is wrong or it's a bug?

use test as sync function

Hi @chill117 ,
I try using this library for check some proxies but after some check it's return Maximum number of open connections reached. message.
How we can use test function as sync function.
Can you help me?

Proxy Authentication Required

my code:

const auth=Buffer.from(`${proxyF[2]}:${proxyF[3]}`).toString('base64')
            proxyObj.auth = `Basic ${auth}`
 ProxyVerifier.testProtocols(proxyObj, function (error, results) {

    ....

});

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.