feross / simple-get Goto Github PK
View Code? Open in Web Editor NEWSimplest way to make http get requests. Supports HTTPS, redirects, gzip/deflate, streams in < 100 lines
License: MIT License
Simplest way to make http get requests. Supports HTTPS, redirects, gzip/deflate, streams in < 100 lines
License: MIT License
Lines 74 to 77 in e7a7411
This calls abort
on the request, and fires the callback. So you'd expect simpleGet
to have relinquished all execution back to the caller.
But in a super simple example of simply making a request to some URL, and exiting (nothing more), there is an observed 30s delay between when the callback is called with the error and before the process exits.
I looked into it, and it appears this is caused by the fact that the http
module from node defaults to dns.lookup
which is not cancellable.
I experimented with swapping this out with Tangerine
which maintains an internal AbortController
and adding a tangerine.cancel()
on any error and it worked flawlessly.
Slight hiccup though - tangerine's peer dependency is undici
so it might be unfortunate to pull that in if you're not using it. Ref: #64
But there's other DNS alternatives that don't require undici. Up to you, just figured I'd mention this here.
I know this is more-so a feature request - not an issue. Regardless, I imagine it would be a nice feature.
I'm working on a project that requires manual handling of redirects. Would it be possible to to add an option to disable automatic redirect following? I imagine I'm not the only person who requires such functionality.
Thanks.
Right now there doesn't seem to be a way to do
var req = get.post(...)
pump(coolStream, req)
Would be great to support that :)
Hi,
I was wondering if this module should begin to use https://npm.im/undici instead of the regular node http/s
client.
(Once we're ready to drop Node 8 support)
Hi,
I'm using this module from webtorrent, and i get an error from this module when using webseeding (webconn of bittorrent-swarm uses this module). The error is in this piece of code:
return simpleGet(opts, function (err, res) {
if (err) return cb(err)
var chunks = []
res.on('data', function (chunk) {
chunks.push(chunk)
})
res.on('end', function () {
cb(null, Buffer.concat(chunks), res)
})
})
Buffer.concat implementation uses item.copy() on every part of the list, except that if the length of chunks is 1, then it returns chunks[0]. In that case there's no error.
I think Buffer.concat expects array of Buffer or Buffer to be sent, but we're sending strings. I've got no error with the following code:
res.on('data', function (chunk) {
chunks.push(new Buffer(chunk))
})
But anyway the webseed fails. I'm opening an issue in webtorrent too, will ref to this.
Any advice? Thanks!
Is it possible to add option that will allows to skip body decompression?
I you are not against of this I can make PR.
I noticed if the request url is https timeout doesn't work, see example below
const get = require('simple-get')
const opts = {
url: 'http://example.com',
timeout: 10
}
get(opts, function (err, res) {
console.log('HTTP', err);
})
})
// result
// HTTP Error: Request timed out
// at ClientRequest.req.on (/home/runner/node_modules/simple-get/index.js:67:8)
// at ClientRequest.emit (events.js:180:13)
// at Socket.emitRequestTimeout (_http_client.js:641:42)
// at Object.onceWrapper (events.js:272:13)
// at Socket.emit (events.js:180:13)
// at Socket._onTimeout (net.js:396:8)
// at ontimeout (timers.js:466:11)
// at tryOnTimeout (timers.js:304:5)
// at Timer.listOnTimeout (timers.js:267:5)
const opts2 = {
url: 'https://example.com',
timeout: 10
}
get(opts2, function (err, res) {
console.log('HTTPS', err);
})
// result
//HTTPS null
Hello!
I'm opening this PR just to inform the new Node.js release v19 might break a few users that expect the connection will be closed after the first request.
Example:
sget({
method: 'POST',
url: 'localhost:3000',
}, (err, res, body) => {
...
})
sget({
method: 'POST',
url: 'localhost:3000',
agent: false, // this will obligate creating an agent for every request, so the connection won't be reutilized by default
}, (err, res, body) => {
...
})
Code:
class ApiController {
constructor() {
this.settings = apiSettings;
}
get() {
console.log(this);
request(apiSettings, (err, res, data) => {
if (err) {
throw err;
}
console.log('data', data);
console.log('data', res);
});
}
}
This library aims to be the smallest wrapper around HTTP, but it includes its test
directory in its installs. This directory is twice the size of the rest of the library.
Please exclude it with the package.json
files, or an .npmignore
.
The querystring API is considered Legacy. While it is still maintained, new code should use the URLSearchParams API instead.
Line 9 in 81eaf56
The npm package is deprecated also: https://www.npmjs.com/package/querystring
It'd be useful if this module had TS types.
Same as request
and got.stream
to allow things like `.pipe(fs.createWriteStream(fp))?
Maybe separate method, would be the way.
If the library allowed a promise in addition to a callback flow, it would be helpful for code utilizing async/await.
Happy to take a stab implementing if the feature is wanted.
Node 6.0 is still supported for another year, and it would be fantastic to see it being kept supported in simple-get.
I think the only problem is related to the use of the spread operator, so maybe we can remove that.
https://travis-ci.org/fastify/fastify-cli/jobs/370790022#L1283-L1303
/home/travis/build/fastify/fastify-cli/node_modules/simple-get/index.js:14
opts = typeof opts === 'string' ? { url: opts } : { ...opts }
^^^
SyntaxError: Unexpected token ...
at createScript (vm.js:56:10)
at Object.runInThisContext (vm.js:97:10)
at Module._compile (module.js:549:28)
at Object.Module._extensions..js (module.js:586:10)
at Module.load (module.js:494:32)
at tryModuleLoad (module.js:453:12)
at Function.Module._load (module.js:445:3)
at Module.require (module.js:504:17)
at require (internal/module.js:20:19)
at Object.<anonymous> (/home/travis/build/fastify/fastify-cli/test/start.js:8:35)
at Module._compile (module.js:577:32)
at Object.Module._extensions..js (module.js:586:10)
at Module.load (module.js:494:32)
at tryModuleLoad (module.js:453:12)
at Function.Module._load (module.js:445:3)
at Module.runMain (module.js:611:10)
If you do not have bandwidth, I'll be happy to help out with maintenance. This is my go-to http client.
(This means we cannot upgrade to [email protected] in fastify tests.)
Hello.
I am gonna use this package in IONIC, but I can't. (Originally, I am using webtorrent in IONIC)
When I run project, it shows error like this.
I tried to find solution, and fix it to add this in package.json
It works well when I run this.
ionic cordova emulate android -l
But when I build release apk, it doesn't work, because http package can't be used in IONIC or Angular.
It shows "protocol.request is not function"
Is there any solution for this?
time to drop commonjs support?
Since POST doesn't follow redirects (per design, see #35), I need to be able to detect when a redirect happens and figure out the new URL, so I can re-issue my POST request.
Here's what I have so far:
get('https://github.com/wmhilton/isomorphic-git', (err, res) => console.log(res.req.path))
That gets me the path, but I haven't figured out the best way to get the protocol/hostname/port.
Having a redirect history is probably the major benefit of using this library over node-fetch in my opinion.
Hello again, sorry for spam.
I have noticed the redirection system is somewhat faulty. When redirecting, it does not call url.resolve, so redirects will entirely malfunction if you got a relative path.
To solve this, navigate to the line within index.js, containing the comment // Follow 3xx redirects
and update it to
opts.url = url.resolve(url.format(opts), res.headers.location); // Follow 3xx redirects
Again, sorry for spamming the issues section.
Regards.
After upgrading to Node.js 14, I do not receive HTTP 500 responses with code of the form:
get.concat('http://example.com', function (err, res, data) {
} );
This can be fixed by checking explicitly for res.statusCode != 200
in simple-get's concat function, i.e.
simpleGet.concat = (opts, cb) => {
return simpleGet(opts, (err, res) => {
if (err) return cb(err)
if (res.statusCode != 200) return cb(null,res) // <---- added line
concat(res, (err, data) => {
It appears that a change in Node.js 14 means that the stream end event is no-longer called in this situation, so simple-concat never returns output.
URL, available in Node with whatwg-url (and require("url").URL
in v7), is not currently supported.
Hey @feross
Would you be in favor of a json: true
option similar to request
?
json - sets body to JSON representation of value and adds Content-type: application/json header. Additionally, parses the response body as JSON.
It feels like the majority of HTTP calls I'm making are JSON these days and it could be nice to avoid the Content-Type
/stringify
/parse
dance.
I understand this makes thing less simple, so no worries if you want to keep it out! ๐
SyntaxError: API error: Unexpected token < in JSON at position 0
at JSON.parse ()
at C:\xampp\htdocs\paper-hands-service-node\node_modules\simple-get\index.js:84:23
at PassThrough. (C:\xampp\htdocs\paper-hands-service-node\node_modules\simple-concat\index.js:8:13) at Object.onceWrapper (even at Object.onceWrapper (events.js:421:28)
at PassThrough.emit (events.js:315:20)
at endReadableNT (internal/streams/readable.js:1327:12)
at processTicksAndRejections (internal/process/task_queues.js:80:21)
I am fetching the Reddit API. After about 5 hours (18k requests later) of that, I am running into this error. I am not sure how this could happen given Reddit is always giving the correctly formatted JSON.
Hey @feross would you be open to a pull request to support basic auth? I know there's already OAuth but basic auth is still used here and there.
Hi @feross,
Would you be open to a PQ to add support for methods other than GET
or is that outside the scope of this library? It could be support with { method: 'POST', body: new Buffer(...) }
as opts
.
I'm asking since I need to a post and then follow the redirect I get back, I think that this is something quite common.
Thanks!
So I'm sending a JSON object via the POST method and using .concat
, but the problem is the endpoint is returning 204 No Content and simple-get
is trying to parse that. I think the correct behavior in this case would be to return an empty object or even null.
This doesn't look good in tests:
https://github.com/feross/simple-get/blob/master/test/basic.js#L11
I need to use simple-get
with ES5, which isn't do-able because of Object.assign({}, opts)
(and maybe some others..).
Currently simple-get will send a duplicated POST request if the response has a location header. I think this is not correct because POST requests are not idempotent. Moreover this is very common in REST applications, return a 204 status code with a Location header for a resource that should be accessed via GET (not POST).
I'm happy to file a PR if you agree that this should be fixed.
Found a minor issue: if a request redirects from hostname A to hostname B, then the host
header from the previous request must be cleared, else it will return a HTTP error 404 (or similar) from the destination server of hostname B.
Example httpbin test URL: https://httpbin.org/redirect-to?url=http%3A%2F%2Fexample.com%2F (redirects from httpbin.org to example.com).
Possible fix: add delete opts.headers['host']
to clear previous request's host
header in parseOptsUrl()
around line 103.
As a sidenote, you probably don't need delete opts.url
in parseOptsUrl()
at line 103. Should be safe to keep opts.url
.
Hi, this is a neat little library.
Playing around I noticed a possible bug with maxRedirects
. If its value is set to say 5
then up-to five redirects should be allowed after the main request, but it falls short by 1 (allowing only 4 redirects), due to opts.maxRedirects
being decremented before the actual check in the code at https://github.com/feross/simple-get/blob/master/index.js#L47. I suppose it should rather be:
if (opts.maxRedirects > 0) {
opts.maxRedirects -= 1
simpleGet(opts, cb)
}
It can be tested with this httpbin endpoint: https://httpbin.org/redirect/5
Also, I think the extra parseOptsUrl(opts)
within the redirection handler code at https://github.com/feross/simple-get/blob/master/index.js#L44 isn't necessary, as it will be called at https://github.com/feross/simple-get/blob/master/index.js#L15 anyways on the next request.
I am developing a project with Webtorrent that uses webseeds. I observed a weird behaviour with the range requests that were made to the webseed server. The downloads never completely finished, and my webseed was being choked (Although I am using WebTorrent 0.98.19), and the issue of choking webseeds has been already solved & released.
Observing the requests in the network, I could see that some of them were receiving 200 OK, instead of 206 Partial Content, and it turned out that the range header was not being sent:
I have tried in Firefox and Opera, and works properly. So I have been observing chrome://net-internals/#events and I see a strange behaviour. Somehow, when the cache is enabled this behaviour appears, but if in the network devTools the Disable cache
option is ON, then everything works fine, so it seems to be a problem with the Chrome caching.
These are logs of events of URL_REQUESTS, the first one is of a working request (206 partial content), while the second one is of a failing one. As you can see, after what seems to be a request to the cache, then the original request has the range header removed.
Working request:
3201472: URL_REQUEST
http://localhost:5000/ab3c6460-1682-4a2e-bac5-d59da9650ed6.enc
Start Time: 2017-10-13 16:31:39.561
t=69136 [st= 0] +REQUEST_ALIVE [dt=38]
--> priority = "MEDIUM"
--> url = "http://localhost:5000/ab3c6460-1682-4a2e-bac5-d59da9650ed6.enc"
t=69136 [st= 0] +URL_REQUEST_DELEGATE [dt=1]
t=69136 [st= 0] DELEGATE_INFO [dt=1]
--> delegate_blocked_by = "extension AdBlock"
t=69137 [st= 1] -URL_REQUEST_DELEGATE
t=69137 [st= 1] +URL_REQUEST_START_JOB [dt=36]
--> load_flags = 34624 (DO_NOT_SAVE_COOKIES | DO_NOT_SEND_AUTH_DATA | DO_NOT_SEND_COOKIES | MAYBE_USER_GESTURE | VERIFY_EV_CERT)
--> method = "GET"
--> url = "http://localhost:5000/ab3c6460-1682-4a2e-bac5-d59da9650ed6.enc"
t=69137 [st= 1] URL_REQUEST_DELEGATE [dt=0]
t=69137 [st= 1] HTTP_CACHE_CALLER_REQUEST_HEADERS
--> Origin: http://localhost:4200
X-DevTools-Emulate-Network-Conditions-Client-Id: 7405794b-d70d-47c3-bd12-0183cf81ed44
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
Range: bytes=14614528-14621551
Accept: */*
Referer: http://localhost:4200/files/list
Accept-Encoding: gzip, deflate, br
Accept-Language: es-ES,es;q=0.8
--> line = ""
t=69137 [st= 1] HTTP_CACHE_GET_BACKEND [dt=0]
t=69137 [st= 1] HTTP_CACHE_OPEN_ENTRY [dt=1]
--> net_error = -2 (ERR_FAILED)
t=69138 [st= 2] HTTP_CACHE_CREATE_ENTRY [dt=0]
t=69138 [st= 2] HTTP_CACHE_ADD_TO_ENTRY [dt=0]
t=69138 [st= 2] +HTTP_STREAM_REQUEST [dt=0]
t=69138 [st= 2] HTTP_STREAM_JOB_CONTROLLER_BOUND
--> source_dependency = 3201475 (HTTP_STREAM_JOB_CONTROLLER)
t=69138 [st= 2] HTTP_STREAM_REQUEST_BOUND_TO_JOB
--> source_dependency = 3201476 (HTTP_STREAM_JOB)
t=69138 [st= 2] -HTTP_STREAM_REQUEST
t=69139 [st= 3] +HTTP_TRANSACTION_SEND_REQUEST [dt=0]
t=69139 [st= 3] HTTP_TRANSACTION_SEND_REQUEST_HEADERS
--> GET /ab3c6460-1682-4a2e-bac5-d59da9650ed6.enc HTTP/1.1
Host: localhost:5000
Connection: keep-alive
Origin: http://localhost:4200
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
Accept: */*
Referer: http://localhost:4200/files/list
Accept-Encoding: gzip, deflate, br
Accept-Language: es-ES,es;q=0.8
Range: bytes=14614528-14621551
t=69139 [st= 3] -HTTP_TRANSACTION_SEND_REQUEST
t=69139 [st= 3] +HTTP_TRANSACTION_READ_HEADERS [dt=34]
t=69139 [st= 3] HTTP_STREAM_PARSER_READ_HEADERS [dt=34]
t=69173 [st=37] HTTP_TRANSACTION_READ_RESPONSE_HEADERS
--> HTTP/1.1 206 Partial Content
Access-Control-Allow-Origin: http://localhost:4200
Access-Control-Allow-Methods: GET,HEAD,PUT,POST,DELETE
Accept-Ranges: bytes
Content-Type: application/octet-stream
Content-Range: bytes 14614528-14621551/*
Date: Fri, 13 Oct 2017 14:31:39 GMT
Connection: keep-alive
Transfer-Encoding: chunked
t=69173 [st=37] -HTTP_TRANSACTION_READ_HEADERS
t=69173 [st=37] URL_REQUEST_DELEGATE [dt=0]
t=69173 [st=37] -URL_REQUEST_START_JOB
t=69173 [st=37] URL_REQUEST_DELEGATE [dt=1]
t=69174 [st=38] HTTP_TRANSACTION_READ_BODY [dt=0]
t=69174 [st=38] URL_REQUEST_JOB_FILTERED_BYTES_READ
--> byte_count = 3757
t=69174 [st=38] HTTP_TRANSACTION_READ_BODY [dt=0]
t=69174 [st=38] URL_REQUEST_JOB_FILTERED_BYTES_READ
--> byte_count = 3267
t=69174 [st=38] HTTP_TRANSACTION_READ_BODY [dt=0]
t=69174 [st=38] -REQUEST_ALIVE
Failing request:
3201529: URL_REQUEST
http://localhost:5000/ab3c6460-1682-4a2e-bac5-d59da9650ed6.enc
Start Time: 2017-10-13 16:31:40.439
t=70014 [st= 0] +REQUEST_ALIVE [dt=17]
--> priority = "MEDIUM"
--> url = "http://localhost:5000/ab3c6460-1682-4a2e-bac5-d59da9650ed6.enc"
t=70014 [st= 0] +URL_REQUEST_DELEGATE [dt=1]
t=70014 [st= 0] DELEGATE_INFO [dt=1]
--> delegate_blocked_by = "extension AdBlock"
t=70015 [st= 1] -URL_REQUEST_DELEGATE
t=70015 [st= 1] +URL_REQUEST_START_JOB [dt=14]
--> load_flags = 34624 (DO_NOT_SAVE_COOKIES | DO_NOT_SEND_AUTH_DATA | DO_NOT_SEND_COOKIES | MAYBE_USER_GESTURE | VERIFY_EV_CERT)
--> method = "GET"
--> url = "http://localhost:5000/ab3c6460-1682-4a2e-bac5-d59da9650ed6.enc"
t=70015 [st= 1] URL_REQUEST_DELEGATE [dt=0]
t=70015 [st= 1] HTTP_CACHE_CALLER_REQUEST_HEADERS
--> Origin: http://localhost:4200
X-DevTools-Emulate-Network-Conditions-Client-Id: 7405794b-d70d-47c3-bd12-0183cf81ed44
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
Range: bytes=14598144-14614527
Accept: */*
Referer: http://localhost:4200/player
Accept-Encoding: gzip, deflate, br
Accept-Language: es-ES,es;q=0.8
--> line = ""
t=70015 [st= 1] HTTP_CACHE_GET_BACKEND [dt=0]
t=70015 [st= 1] HTTP_CACHE_OPEN_ENTRY [dt=0]
t=70015 [st= 1] HTTP_CACHE_ADD_TO_ENTRY [dt=10]
t=70025 [st=11] HTTP_CACHE_READ_INFO [dt=0]
t=70025 [st=11] HTTP_CACHE_GET_BACKEND [dt=0]
t=70025 [st=11] HTTP_CACHE_OPEN_ENTRY [dt=0]
--> net_error = -2 (ERR_FAILED)
t=70025 [st=11] HTTP_CACHE_CREATE_ENTRY [dt=0]
t=70025 [st=11] HTTP_CACHE_ADD_TO_ENTRY [dt=0]
t=70025 [st=11] +HTTP_STREAM_REQUEST [dt=1]
t=70025 [st=11] HTTP_STREAM_JOB_CONTROLLER_BOUND
--> source_dependency = 3201532 (HTTP_STREAM_JOB_CONTROLLER)
t=70026 [st=12] HTTP_STREAM_REQUEST_BOUND_TO_JOB
--> source_dependency = 3201533 (HTTP_STREAM_JOB)
t=70026 [st=12] -HTTP_STREAM_REQUEST
t=70026 [st=12] +HTTP_TRANSACTION_SEND_REQUEST [dt=0]
t=70026 [st=12] HTTP_TRANSACTION_SEND_REQUEST_HEADERS
--> GET /ab3c6460-1682-4a2e-bac5-d59da9650ed6.enc HTTP/1.1
Host: localhost:5000
Connection: keep-alive
Origin: http://localhost:4200
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
Accept: */*
Referer: http://localhost:4200/player
Accept-Encoding: gzip, deflate, br
Accept-Language: es-ES,es;q=0.8
t=70026 [st=12] -HTTP_TRANSACTION_SEND_REQUEST
t=70026 [st=12] +HTTP_TRANSACTION_READ_HEADERS [dt=3]
t=70026 [st=12] HTTP_STREAM_PARSER_READ_HEADERS [dt=3]
t=70029 [st=15] HTTP_TRANSACTION_READ_RESPONSE_HEADERS
--> HTTP/1.1 416 Range Not Satisfiable
Access-Control-Allow-Origin: http://localhost:4200
Access-Control-Allow-Methods: GET,HEAD,PUT,POST,DELETE
Content-Type: text/plain; charset=utf-8
Content-Length: 21
Date: Fri, 13 Oct 2017 14:31:40 GMT
Connection: keep-alive
t=70029 [st=15] -HTTP_TRANSACTION_READ_HEADERS
t=70029 [st=15] URL_REQUEST_DELEGATE [dt=0]
t=70029 [st=15] -URL_REQUEST_START_JOB
t=70029 [st=15] URL_REQUEST_DELEGATE [dt=0]
t=70029 [st=15] HTTP_TRANSACTION_READ_BODY [dt=0]
t=70029 [st=15] HTTP_CACHE_WRITE_DATA [dt=1]
t=70030 [st=16] URL_REQUEST_JOB_FILTERED_BYTES_READ
--> byte_count = 21
t=70030 [st=16] HTTP_TRANSACTION_READ_BODY [dt=0]
t=70030 [st=16] HTTP_CACHE_WRITE_DATA [dt=0]
t=70031 [st=17] -REQUEST_ALIVE
My setup is Macbook Pro 13', macOS Sierra 10.12.6, Chrome 61.0.3163.100
I know the issue is probably not in the simple-get
project itself, but I wanted to post it here to see if anyone has seen similar behaviour, or knows a workaround.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.