sytelus/hackernewsdata

All stories and comments posted on Hacker News upto May 29, 2014

hackernewsdata's Issues

Torrent

Create a torrent for this.

641,071 IDs unaccounted for

Hi there,

Per my Reddit comment at http://uk.reddit.com/r/datasets/comments/26xqgs/downloading_all_of_hacker_news_posts_and_comments/ , there are 641k IDs that don't appear anywhere.

It looks like either your crawler or Algolia don't have a complete data set.

I manually checked some of the missing IDs, and some lead to deleted posts, the vast majority appear to lead to legitimate comments/links.

If it's a problem with your script, I guess that is easiest to fix. If it is a problem with Algolia, then I guess we're out of luck :(

Some HTML formatting broken

For the story https://news.ycombinator.com/item?id=7064470
I find this comment syntax on the site

What about the bnoordhuis story?<p>It seems not to be active again for his lib: <a href="https://github.com/joyent/libuv/graphs/contributors" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;joyent&#x2F;libuv&#x2F;graphs&#x2F;contributors</a><p>For people who don&#x27;t know, bnoordhuis exiled himself 2 months ago after the bashing the node.js community imposed him for not merging a pull about gender. It was a shame because he was in the top 3 contributors. (all details: <a href="https://github.com/joyent/libuv/pull/1015" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;joyent&#x2F;libuv&#x2F;pull&#x2F;1015</a>)

but this in the dump:

{"created_at":"2014-01-15T17:25:29Z","title":null,"url":null,"author":"hartator","points":38,"story_text":null,"comment_text":"What about the bnoordhuis story?<p>It seems not to be active again for his lib: <a href\"https://github.com/joyent/libuv/graphs/contributor\" rel\"nofollo\">https:&#x2F;&#x2F;github.com&#x2F;joyent&#x2F;libuv&#x2F;graphs&#x2F;contributors</a><p>For people who don&#x27;t know, bnoordhuis exiled himself 2 months ago after the bashing the node.js community imposed him for not merging a pull about gender. It was a shame because he was in the top 3 contributors. (all details: <a href\"https://github.com/joyent/libuv/pull/101\" rel\"nofollo\">https:&#x2F;&#x2F;github.com&#x2F;joyent&#x2F;libuv&#x2F;pull&#x2F;1015</a>)","num_comments":null,"story_id":7064470,"story_title":"The Next Phase of Node.js","story_url":"http://blog.nodejs.org/2014/01/15/the-next-phase-of-node-js/index.html","parent_id":7064470,"created_at_i":1389806729,"_tags":["comment","author_hartator","story_7064470"],"objectID":"7064601","_highlightResult":{"author":{"value":"hartator","matchLevel":"none","matchedWords":[]},"comment_text":{"value":"What about the bnoordhuis story?<p>It seems not to be active again for his lib: <a href\"https://github.com/joyent/libuv/graphs/contributor\" rel\"nofollo\">https:&#x2F;&#x2F;github.com&#x2F;joyent&#x2F;libuv&#x2F;graphs&#x2F;contributors</a><p>For people who don&#x27;t know, bnoordhuis exiled himself 2 months ago after the bashing the node.js community imposed him for not merging a pull about gender. It was a shame because he was in the top 3 contributors. (all details: <a href\"https://github.com/joyent/libuv/pull/101\" rel\"nofollo\">https:&#x2F;&#x2F;github.com&#x2F;joyent&#x2F;libuv&#x2F;pull&#x2F;1015</a>)","matchLevel":"none","matchedWords":[]},"story_title":{"value":"The Next Phase of Node.js","matchLevel":"none","matchedWords":[]},"story_url":{"value":"http://blog.nodejs.org/2014/01/15/the-next-phase-of-node-js/index.html","matchLevel":"none","matchedWords":[]}}}

notice the missing equals-signs in the a-href and a-nofollow.

Weirdly, this bug is only present in some of the data, ca. 12809 a's have no href, out of 982843 a tags.