Timeout Error Error: ESOCKETTIMEDOUT parsing https://www.nasdaq.com/feed/rssoutbound?category=Stocks

[MichaelWhite63]

Timeout Error Error: ESOCKETTIMEDOUT parsing https://www.nasdaq.com/feed/rssoutbound?category=Stocks

[alabeduarte]

Hi @MichaelWhite63

Thank you for raising an issue. I tried the example you provided and I also got the same issue although the XML feed is available if I request it in the browser. This library - feedparser-promised - is a wrapper for https://github.com/danmactough/node-feedparser and here are the things I just tried, without much success:

Bumping node-feedparser to latest version 🔴

I bumped https://github.com/danmactough/node-feedparser to the latest version (v2.2.10) and ran the smoke tests (yarn smoke) with the URL you provided and also got a timeout.

Trying out node-feedparser directly 🔴

I tried to use the https://github.com/danmactough/node-feedparser directly using the examples provided in their README and I also got a timeout.

Here's what I tried:

Go to https://npm.runkit.com/feedparser and put the following (same code as the README):

var FeedParser = require("feedparser")
var fetch = require('node-fetch'); // for fetching the feed

var req = fetch("https://www.nasdaq.com/feed/rssoutbound?category=Stocks")
var feedparser = new FeedParser();

req.then(function (res) {
  if (res.status !== 200) {
    throw new Error('Bad status code');
  }
  else {
    // The response `body` -- res.body -- is a stream
    res.body.pipe(feedparser);
  }
}, function (err) {
  // handle any request errors
});

feedparser.on('error', function (error) {
  // always handle errors
});

feedparser.on('readable', function () {
  // This is where the action is!
  var stream = this; // `this` is `feedparser`, which is a stream
  var meta = this.meta; // **NOTE** the "meta" is always available in the context of the feedparser instance
  var item;

  while (item = stream.read()) {
    console.log(">>>>>>>>>>>>", item);
  }
});

I'm sorry for not come up with the fix for that right away, I'd assume it will require more investigation in the weeds of https://github.com/danmactough/node-feedparser since this library (https://github.com/alabeduarte/feedparser-promised) is just a wrapper, so I would suggest creating an issue there (https://github.com/danmactough/node-feedparser/issues) and see how that goes. If it is something with the library feel free to bump with the new version or even use them directly!

Cheers

[MichaelWhite63]

Alabe, I very much appreciate your investigation. I tried it on feedparser as well realizing the same results that you did. I contacted the NASDAQ to ask if they have restricted access. They responded that they block 'curl' but have not yet responded if they block programmatic access such as this. Until I receive their response I am assuming that they can detect the difference between a browser and programmatic access such as https://github.com/danmactough/node-feedparser . If they are blocking access then they must be doing it for a reason and I doubt that I will be able to convince them to not restrict access. Assuming they purposefully block access I have two questions. First, do know the details of the initial connection between node-feedparser and a RSS server? NASDAQ's WAF must be able to detect that the connection is programmatic from 'a' library (any library) and not from a web browser. Allowing it to block access. Second, if so, is there any way to modify the initial connection to make it appear as if it is a browser and not programmatic. Thank you, Michael

…

On Sat, Sep 26, 2020 at 5:11 AM Alabe Duarte ***@***.***> wrote: Hi @MichaelWhite63 <https://github.com/MichaelWhite63> Thank you for raising an issue. I tried the example you provided and I also got the same issue although the XML feed is available if I request it in the browser. This library - feedparser-promised - is a wrapper for https://github.com/danmactough/node-feedparser and here are the things I just tried, without much success: Bumping node-feedparser to latest version I bumped https://github.com/danmactough/node-feedparser to the latest version (v2.2.10) and ran the smoke tests (yarn smoke) with the URL you provided and also got a timeout. Trying out node-feedparser directly I tried to use the https://github.com/danmactough/node-feedparser directly using the examples provided in their README and I also got a timeout. Here's what I tried: Go to https://npm.runkit.com/feedparser and put the following (same code as the README): var FeedParser = require("feedparser")var fetch = require('node-fetch'); // for fetching the feed var req = fetch("https://www.nasdaq.com/feed/rssoutbound?category=Stocks")var feedparser = new FeedParser(); req.then(function (res) { if (res.status !== 200) { throw new Error('Bad status code'); } else { // The response `body` -- res.body -- is a stream res.body.pipe(feedparser); }}, function (err) { // handle any request errors}); feedparser.on('error', function (error) { // always handle errors}); feedparser.on('readable', function () { // This is where the action is! var stream = this; // `this` is `feedparser`, which is a stream var meta = this.meta; // **NOTE** the "meta" is always available in the context of the feedparser instance var item; while (item = stream.read()) { console.log(">>>>>>>>>>>>", item); }}); I'm sorry for not come up with the fix for that right away, I'd assume it will require more investigation in the weeds of https://github.com/danmactough/node-feedparser since this library ( https://github.com/alabeduarte/feedparser-promised) is just a wrapper, so I would suggest creating an issue there ( https://github.com/danmactough/node-feedparser/issues) and see how that goes. If it is something with the library feel free to bump with the new version or even use them directly! Cheers — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#33 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5H2RJXUVFOKSAE4VTRK5LSHWV3XANCNFSM4RY7CBQQ> .

[alabeduarte]

Hi @MichaelWhite63

I'd assume that curl would fall into the same category as "programmatic". If they really block it I'm afraid it could be the case they are not open the RSS to use, hard to say. About different ways you can approach, I'd have a look at the "options" here:

Maybe there is an HTTP option you can pass in via request, more info here: https://github.com/alabeduarte/feedparser-promised#using-feedparser-options

Sorry not able to provide more info, let me know what you come up with!

Cheers

[MichaelWhite63]

Thank you for this information. One more question as I now have a reader and able to read your news. Do you have a RSS feed that contains the upgrades / downgrades you have on your web site?

…

On Sun, Sep 27, 2020 at 2:10 AM Alabe Duarte ***@***.***> wrote: Hi @MichaelWhite63 <https://github.com/MichaelWhite63> I'd assume that curl would fall into the same category as "programmatic". If they really block it I'm afraid it could be the case they are not open the RSS to use, hard to say. About different ways you can approach, I'd have a look at the "options" here: Maybe there is an HTTP option you can pass in via request, more info here: https://github.com/alabeduarte/feedparser-promised#using-feedparser-options Sorry not able to provide more info, let me know what you come up with! Cheers — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#33 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5H2RIMF3GHQSKN43RBQNLSH3JO3ANCNFSM4RY7CBQQ> .

[alabeduarte]

Hi @MichaelWhite63

I believe the quickest way to achieve that is to "Watch" this repo (throughout GitHub) so that you subscribe to any changes to the repo. Due to the nature of this library – wrapper – though, the tendency is to always get behind the original code so I'm considering deprecating it.

I'm closing the issue but let me know if there's anything outstanding that I can help you with.