georgjaehnig/webpages-to-ebook

Create an EPUB from a list of URLs. Standing on the shoulders of Wget, Readability and Pandoc.

Webpages to eBook

Create an EPUB from a list of URLs. Standing on the shoulders of

Wget → Readability → Pandoc

Create your own ebooks

Requirements

• Node.js
• NPM
• Wget: wget has to be in PATH.
• Pandoc: pandoc has to be in PATH.

Install

git clone https://github.com/georgjaehnig/webpages-to-ebook.git
cd webpages-to-ebook
npm install

Usage

1. Create a definition, like the examples in definitions/.
2. node index.js your-created-definition.yml [optional-definition-to-be-merged-with-the-first.yml]
3. Find your EPUB in output/epub/.

Documentation

• https://github.com/georgjaehnig/webpages-to-ebook/wiki/docs

Examples

• slatestarcodex.year.2020.epub, source
• slatestarcodex.year.2019.epub, source
• slatestarcodex.year.2018.epub, source
• slatestarcodex.year.2017.epub, source
• slatestarcodex.year.2016.epub, source
• slatestarcodex.year.2015.epub, source
• slatestarcodex.year.2014.epub, source
• slatestarcodex.year.2013.epub, source
• slatestarcodex.top.epub, source
• slatestarcodex.top.more.epub, source

Also some structuring with raw HTML is possible, check definitions/library_of_scott_alexandria.yml and in there raw: and tags:/title:.

webpages-to-ebook's Issues

June 2020 update?

I wonder if there's any chance you'd be able to update the compiled epubs to June 2020, using, say:
https://web.archive.org/web/*/www.slatestarcodex.com

question about the readability module dependency: mozila or luin ?

A small question about the readability dependency. On the README.md page you write:

Create an EPUB from a list of URLs. Standing on the shoulders of
Wget → Readability → Pandoc

where "Readability" links to, https://github.com/mozilla/readability

But your project actually uses, https://github.com/luin/readability , which actually installs a module called "node-readability"

I know that "luin" is probablily a fork or something pulling from "mozilla".
I just wanted to make sure there is a reason for this, and for not pulling directly from mozilla.

I ask this becaus I've been testing dozens of node based readability projects, and very frequently because they choose to name their binary "readability" you end up with a mess of different packages and/or their installed binary named "readability" .

Link title with URL

In readability.js, link the article title with its source URL.

Filter out hidden open threads?

The non-integer open threads do not actually contain any content, yet they appear in the example epubs.

Modify settings in Yaml

Allow to set Modify settings in definitions Yaml. Like

modify:
    content:
    - search: /(<img .*?)>/g
      replace: $1/>
    - search: /<br>/g
      replace: <br/>

TypeError: Cannot read property 'length' of undefined (book.content.length)

Hi,

I have an error thrown when I try to execute the module with the definitions/slatestarcodex.base.yml files:

$ node index.js definitions/slatestarcodex.base.yml
[hidden_path]/webpages-to-ebook/index.js:24
var count = book.content.length;
                         ^

TypeError: Cannot read property 'length' of undefined
    at Object.<anonymous> ([hidden_path]/webpages-to-ebook/index.js:24:26)
    at Module._compile (internal/modules/cjs/loader.js:702:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:713:10)
    at Module.load (internal/modules/cjs/loader.js:612:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:551:12)
    at Function.Module._load (internal/modules/cjs/loader.js:543:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:744:10)
    at startup (internal/bootstrap/node.js:238:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:572:3)

Thank you.

library_of_scott_alexandria.epub

Could you release the link to the "library_of_scott_alexandria.epub"? Some of the URLs no longer resolve.

Table Of Contents?

Great project. Is there a way to control the TOC that is generated. It is including every heading in each article and I would rather it just include the articles themselves.

Error on parsing. Skipping file.

Hi, I saw a bunch of errors when running make. I'm not sure if these are problems:

43a3ff1c870371cdded97974ee0adf27: Error on parsing. Skipping file.
eed3b866c8fac7bee2dbda868e35c96e: Error on parsing. Skipping file.
c6de499346d64f20a42ffd517bc5db61: Error on parsing. Skipping file.
fa3d324994c16cffcc27c28b7b91c753: Error on parsing. Skipping file.
10980f1ad7a933be56542d0febb5c2bd: Error on parsing. Skipping file.
eb3c22d6eded25cf6936b53f23ca821e: Error on parsing. Skipping file.
954ba7e05c3b0b84483bec8008738539: Error on parsing. Skipping file.
fcedbc82fc4e1fe39623036ccdc64350: Error on parsing. Skipping file.
96ced442ccbb85b965aa7e02524dd040: Error on parsing. Skipping file.
d4adf50b44fd609a244d5c175ea00341: Error on parsing. Skipping file.
1ad502cf24e012aa1039755c584634e3: Error on parsing. Skipping file.
825e51b6334e4e0d606e261b364df52c: Error on parsing. Skipping file.
6feb3d25cf316791ac1d9e338ca1e811: Error on parsing. Skipping file.
86c7a9c1a5e4373d7feb396923d64c32: Error on parsing. Skipping file.
e3905650a3d9055c7439dc3ac5d51450: Error on parsing. Skipping file.
55b6c3b2fc66c503aadcbdaa310f81f5: Error on parsing. Skipping file.
c432b2c7d0475bb734fc826298cb36a7: Error on parsing. Skipping file.
e7a3fc5a78f2f66e5d674b5b6d9599e9: Error on parsing. Skipping file.
8f6894eb6e9b0f30314b1ade6107149c: Error on parsing. Skipping file.
005436e0bcb90765bbdba48e041ff908: Error on parsing. Skipping file.
ff405d142101a00508f2755f47080496: Error on parsing. Skipping file.
04aaf413bbd8e080a7a1ada9e79bc971: Error on parsing. Skipping file.
f8b56c7f074d4f6a4a0dc3fa126280bf: Error on parsing. Skipping file.
a65f26048dc390e8a1c949523ec58f01: Error on parsing. Skipping file.
e338763a79d3a7ff596ba097df63e0ca: Error on parsing. Skipping file.
495a7d9f60df19cc587d156ac542188c: Error on parsing. Skipping file.
5a46e503ac3f9dea945471c0d7c29037: Error on parsing. Skipping file.
9d70944a6fbfc7014e79cebb4c8647af: Error on parsing. Skipping file.
4dbfb40d67c6f5548715817065e9ae35: Error on parsing. Skipping file.
565d2539c53f2c0bbefa8e87119923fa: Error on parsing. Skipping file.
0f8c8a4ae26805f0aacf696ffef7f277: Error on parsing. Skipping file.
7a7dc62a935b2a420f6e348fcfcfbb8a: Error on parsing. Skipping file.
13bdc234127eba7ba8e6302d488f4936: Error on parsing. Skipping file.
7dfbabcf679a377f68782975e13f1694: Error on parsing. Skipping file.
56fdd362bb4d699f976c2343e2579172: Error on parsing. Skipping file.
18e2515ce06a16f68fb283415e59e75f: Error on parsing. Skipping file.
085f9fcb923bb313c9e53c92928b9652: Error on parsing. Skipping file.
14f5eca6e62757d925e37292cde5c200: Error on parsing. Skipping file.
b3d64b7948dbd8c5af6466d7e9b796ef: Error on parsing. Skipping file.
adeab8dd983d05a7e9000d2e994d741c: Error on parsing. Skipping file.
7419e33a80dc7558edf25586b56e85df: Error on parsing. Skipping file.
86a1968f90a982e2dc5c78259bcba876: Error on parsing. Skipping file.
07d15ab9dac1a4f01a0051909c13aea5: Error on parsing. Skipping file.
e49c75f5b322b25a074ce2f170d6968c: Error on parsing. Skipping file.
d06c832e394911b2f886326ac938a426: Error on parsing. Skipping file.
1e868a5db9df7b2f71b46647f90dbdf2: Error on parsing. Skipping file.
b27db525d6ce48bf5031cccdf7441113: Error on parsing. Skipping file.
b4485689bbb648e393f0e1c3fa146e2f: Error on parsing. Skipping file.
92ba0c8eaac193b176c3cf5348600800: Error on parsing. Skipping file.
bd1f8f33c48c7add8c268a3cccad9d28: Error on parsing. Skipping file.
55c4e77b16dce128ff2ddc29c3330f26: Error on parsing. Skipping file.
d565ab64183cfec15239c719bde49013: Error on parsing. Skipping file.
d58b340da22f21b6332c9b21ea67db63: Error on parsing. Skipping file.
7959adacc334a945c55a848c04e446f0: Error on parsing. Skipping file.
5c07e69a2571bd0188501372d33e2957: Error on parsing. Skipping file.
d122d2b346e5e8627dd7cca4d01eec9b: Error on parsing. Skipping file.
73cf260d7fc9c324ec6c2aafe3bc455f: Error on parsing. Skipping file.
9122caac0a317cdd8feb4fe7d83788c9: Error on parsing. Skipping file.
69e183342290c3f0436fa2c22f1243e6: Error on parsing. Skipping file.
12dcddaf7b09571c09061d09602e806b: Error on parsing. Skipping file.
c8d0459c3876714bc7439948560ce755: Error on parsing. Skipping file.
bc9d876e8b14b3e9ae51dd3208e63a5c: Error on parsing. Skipping file.
cb17f6f1cbe4487cd3bc13f2adeb21e9: Error on parsing. Skipping file.
ffec3f276d3cc038055004d59132e235: Error on parsing. Skipping file.
26a3b4f82624feb3ad9d6db5cec13ea8: Error on parsing. Skipping file.
bb1315f36edef684321f025415c0a7a5: Error on parsing. Skipping file.
de8093e0f742358fb1d83272cf820cdb: Error on parsing. Skipping file.
04311aec4fa123038b6741cac4d48824: Error on parsing. Skipping file.
183e3a4570dab42323434709e4f88dcc: Error on parsing. Skipping file.
3d326b5ec716ca79b4b9baf5b4ff65fd: Error on parsing. Skipping file.
a13f64a78311d8a533db2fe3b72341c3: Error on parsing. Skipping file.
1673bd3702d219e30ea159f280dffaf3: Error on parsing. Skipping file.
38923f1f2301a6aa5d01ce9e91c47029: Error on parsing. Skipping file.
1dc5d0e1e94a7b5b377d2bae4f84e66c: Error on parsing. Skipping file.
81271614d29622095b63822be41c66d6: Error on parsing. Skipping file.
fbb3319fc69ad72e204374a9b753158a: Error on parsing. Skipping file.
545f12d12c702319f100a36dad8dfb79: Error on parsing. Skipping file.
b72c7ac87a3796fbb0aa2f6af24cce7e: Error on parsing. Skipping file.
a8371979e0cb8eca26c54d2996079cf7: Error on parsing. Skipping file.
b1d5120f0c6af83f1f9245291cb7b0f4: Error on parsing. Skipping file.
ee11c5d20e39e56e39df4abc960da816: Error on parsing. Skipping file.
5f6dfdea22093eb55d5da0a480888293: Error on parsing. Skipping file.
ed02aa33e5816fd91fdb881a3647647f: Error on parsing. Skipping file.
f55a1c317e6f26e22c6ddfb2460292bc: Error on parsing. Skipping file.
a70999fa39eb3f73e386f2a15c816c76: Error on parsing. Skipping file.
0428a292406fe45efaf5dad67f900fbd: Error on parsing. Skipping file.
97008b004414e215a63d74238de3a8a5: Error on parsing. Skipping file.
19df2a74f347888555ec9f1c2ce169bf: Error on parsing. Skipping file.
e943f338781dee485e24b48f7c15da28: Error on parsing. Skipping file.
348313892e248f3db72cea8fc8d1c83f: Error on parsing. Skipping file.
159b4bd50314804e17edd823931b684f: Error on parsing. Skipping file.
0852ad3ba4d43785fca67b8d6b7eb658: Error on parsing. Skipping file.
a8edfbe9502a5b043b5be63e3bee6eaa: Error on parsing. Skipping file.
fd4700c6124ed093d95f8d9f70d7d13a: Error on parsing. Skipping file.
1665cd3499e22945bc7f357829b84234: Error on parsing. Skipping file.
0b0e218b75c7e28e2a88be242e1a0b27: Error on parsing. Skipping file.
b0016c9bdbbe4d95c9a5ed713862ab2d: Error on parsing. Skipping file.
14256defd69060b0fdbf89352f0bc89b: Error on parsing. Skipping file.
a613618f5705b49115967d0e0ae1a0f2: Error on parsing. Skipping file.
8a86eb75c1a0f7557e05652fd11bb278: Error on parsing. Skipping file.
77204184bcbab20612e104334300764a: Error on parsing. Skipping file.
d9d3b9c611efc23e47849ac4136ccd54: Error on parsing. Skipping file.
1ca21254c424581cc14a1c3ef92db82c: Error on parsing. Skipping file.
730db9c4fc6624e7ff912c53b14998af: Error on parsing. Skipping file.
50fcd22ed3db3a6e92a22a4027b96d72: Error on parsing. Skipping file.
11f8f3ec7dc5850fcae5c40bc2bb827d: Error on parsing. Skipping file.
8c19d551829959a309e3145b6b3b4a92: Error on parsing. Skipping file.
f41810ffb3cfda34900395d734d6c881: Error on parsing. Skipping file.
bb59fe0b9c3e16956d4b27bf2b40378d: Error on parsing. Skipping file.
60c457b1a412798dcde71b2fb2b8ff59: Error on parsing. Skipping file.
b2d5e165a6aab796e8abadf8418b08d2: Error on parsing. Skipping file.
6fe297eed692dc35756aec86cc161183: Error on parsing. Skipping file.
58fe25b80f567a30721358974c4cdef2: Error on parsing. Skipping file.
466802da43b2fdf8a4a6345d8c98bf04: Error on parsing. Skipping file.
8058078794213a1bd7475a16d4940197: Error on parsing. Skipping file.
c2e4c02a8915f80aadc2691883ca9788: Error on parsing. Skipping file.
bb85e8e992eb23081236ef3e7739b010: Error on parsing. Skipping file.
912129b027722e54d950f137cfb88913: Error on parsing. Skipping file.
dcc79d66cffb6957dd0a797cda527145: Error on parsing. Skipping file.
341d3b6c48995db1eaade9663807cc7b: Error on parsing. Skipping file.
51ca953c16f5475242617e52895e5069: Error on parsing. Skipping file.
1ada5ce2472c8b2acc20a63538b20cc1: Error on parsing. Skipping file.
47c063ad58c7faa50d6696d871050fa6: Error on parsing. Skipping file.
e46f9fc6ee047c80d0ad8ef052090c4e: Error on parsing. Skipping file.
7b46bdedc7d79d0879f2af0c5b34940a: Error on parsing. Skipping file.
063220c9959aa888826c1cc42b174697: Error on parsing. Skipping file.
f250acd9c7a7b4c51356736386b6fc40: Error on parsing. Skipping file.
e28effc5cf1bcb33f8c5a6d47a6b1e5e: Error on parsing. Skipping file.
36eb6eb33e218b2265fa952c58c47c07: Error on parsing. Skipping file.
c0a63aad52302c9fade93f9dda053f58: Error on parsing. Skipping file.
2ea6844b98ed4ad8d675a9e309d49aa7: Error on parsing. Skipping file.
9b493d972db163e1487188fe51e56e47: Error on parsing. Skipping file.
de987e64fe248f6e84b5b6e15aed4576: Error on parsing. Skipping file.
3f12f9afb5631aa6edf5c24476d1aa43: Error on parsing. Skipping file.
399f1bbdab6db4ef8ae7561e8d70d81a: Error on parsing. Skipping file.
bb1ece9fda13588b6819f7cb6c01fba6: Error on parsing. Skipping file.
c5e2408ef967c420ff9522aab988134a: Error on parsing. Skipping file.
48e04a652017f1f71b68b71879c8bae2: Error on parsing. Skipping file.
cb41c1b3cad25aa9cf43665c97890759: Error on parsing. Skipping file.
a50753b548a3c42f67c221e210910c43: Error on parsing. Skipping file.
c7bc94ea62353c26457cd72eda32f975: Error on parsing. Skipping file.
760101c526aecf55c68de1379c3a047e: Error on parsing. Skipping file.
03bb074c9ffb16877107df1811472a1d: Error on parsing. Skipping file.
0ffc10de961000eec78725e21946fd65: Error on parsing. Skipping file.
f3ac174bef38db70513170c8039377fc: Error on parsing. Skipping file.
8434bb090082837a4c820f4652ee5dc9: Error on parsing. Skipping file.
4bf7be871653079ef8ec9798073946db: Error on parsing. Skipping file.
6cfcf1b6e897389015923de655cf0672: Error on parsing. Skipping file.
3f9314997300e44e1525a7ae55e69851: Error on parsing. Skipping file.
6a35111b4afa621feb58214cc25b8b11: Error on parsing. Skipping file.
0f31e56d72563d9685db7317c701051d: Error on parsing. Skipping file.
a97b2211f20acc3a3c21004f53d77aed: Error on parsing. Skipping file.
88ce026f07f42bb5493a597515b0ac6f: Error on parsing. Skipping file.
c3a14ea2aef7c262bb982b86503782da: Error on parsing. Skipping file.
ba84907be3faa9aaca0754fa98776955: Error on parsing. Skipping file.
0bb3ccea98f12632ca098e4e93631991: Error on parsing. Skipping file.
014552a754d15e293293a002c8e57ab5: Error on parsing. Skipping file.
6780d0a047f264a6a26103c48bf8fce1: Error on parsing. Skipping file.
882d55be471748dc132c9653d8b0e3d3: Error on parsing. Skipping file.
0fe8fd2ec82adf47b2190338296f6f0c: Error on parsing. Skipping file.
3632fbc80866a98ec2a3f521442ea20b: Error on parsing. Skipping file.
3693c62af6a81b04a775a5148e3fb3e7: Error on parsing. Skipping file.
afd2d458e998631b1dc036911dcc1de5: Error on parsing. Skipping file.
22b5c1f21e6f3ad0be105a23a2d1749b: Error on parsing. Skipping file.
56fbc3b0e6205fc57a9b7b896f999be8: Error on parsing. Skipping file.
f1ef710fb422e977749d6a06ca21fb9b: Error on parsing. Skipping file.
b44bd935b0126921b048e9b8e6cc8631: Error on parsing. Skipping file.
6dc29e0820323f78bf64ab338ea1ceab: Error on parsing. Skipping file.
696663f9118add0c30ca0e1524286075: Error on parsing. Skipping file.
af2db112596a425685b5340950a87faa: Error on parsing. Skipping file.
57cebce90d767eb2c718544537d9afe1: Error on parsing. Skipping file.
dbca715ffd34e0c5bc3ee781e7021a77: Error on parsing. Skipping file.
45d81517e242199a078487c38d2990e2: Error on parsing. Skipping file.
ca888b166b74630545c5a735fa6f0961: Error on parsing. Skipping file.
56b40c94fe01dc7d7d03e953ba10b637: Error on parsing. Skipping file.
93a860d640192b8b4203153bfa251555: Error on parsing. Skipping file.
ff920f457636e55b867da53e9d5c9150: Error on parsing. Skipping file.
a879296809dd7bc135a58f31da82a035: Error on parsing. Skipping file.
5b79f8fea5c7e4a8d1b4d31449ee82b9: Error on parsing. Skipping file.
7bfe702a12aeef8d9fb585f345d19216: Error on parsing. Skipping file.
d95cc2a7c7624781b4a57d35c73317ce: Error on parsing. Skipping file.
6dcbc566c6946d6dafecb5cddb77ebf8: Error on parsing. Skipping file.
bc8b2af45f8fb6e0a944963c78e9c026: Error on parsing. Skipping file.
29ec2c6cd8c43affcd37f09e164a6c84: Error on parsing. Skipping file.
ae7072ba9070a92ed61a26d65aba0480: Error on parsing. Skipping file.
c944c755492b8d3d0d6f6914734e2709: Error on parsing. Skipping file.
e8381d139e87620994eb5d9682928635: Error on parsing. Skipping file.
dce008be70b5e7b5385ad98892ed00cc: Error on parsing. Skipping file.
328a811212e65f7f92a71d48f9e1737b: Error on parsing. Skipping file.
3a8dc42f7d6cf087ca2860a0092e8872: Error on parsing. Skipping file.
5bf7ea8b8bed7be07ffcce9f48b80fcf: Error on parsing. Skipping file.
409af0931304d3fb0403552ef786187f: Error on parsing. Skipping file.
3e3b7878d10f7ae5f69af07c21f317b4: Error on parsing. Skipping file.
72b8ab280f1efc4c731297a02973f640: Error on parsing. Skipping file.
39e6569b124d54bfd9f43bb009dd8cd7: Error on parsing. Skipping file.
1d840304d139f38af7c46545e0e4b702: Error on parsing. Skipping file.
b9f3fe03af3ae818bccb50943ed8ee45: Error on parsing. Skipping file.
74f55419108b4003ddc1a4b18442ba01: Error on parsing. Skipping file.
0f71095e7ba7569dcce7f64e325d356f: Error on parsing. Skipping file.
3a7d6deb7b36cdce06d667cc6eb73f99: Error on parsing. Skipping file.
b2cd105d6dad6ba6f1abbe0bf0acf3c9: Error on parsing. Skipping file.
1d0de1fc643e689e72cbe8a7126ad02f: Error on parsing. Skipping file.
e19058373aba3ef0c1163512c475fb61: Error on parsing. Skipping file.
ff7c4a78664c0b2e6fb93d8c4b6207ba: Error on parsing. Skipping file.
a52b59768e9ff51149a231379cb3e929: Error on parsing. Skipping file.
6331a91dbb51dac2460bd32ce491898e: Error on parsing. Skipping file.
1694bb3ad7a60b5eac763615673f7440: Error on parsing. Skipping file.
ade86986f9aaa9143bb9cc9276b47625: Error on parsing. Skipping file.
65723bcda3af7e867742e776e66fc5ad: Error on parsing. Skipping file.
9091c8dfe29ab3c73766cae7004e7642: Error on parsing. Skipping file.
6b3cd92393d8ef38f2c9563f43ea6078: Error on parsing. Skipping file.
602b714907900d031f916e427b071fdb: Error on parsing. Skipping file.
dab6fca154cc6caa416d7264fe429283: Error on parsing. Skipping file.
776b3426b5a1c91f1cb056e39fe4aa6e: Error on parsing. Skipping file.
60be0483f2d59819baf678e741f80013: Error on parsing. Skipping file.
1d28cf52afdfb9ca228ff2399d6620c6: Error on parsing. Skipping file.
528ee901eb6f22a204cf0ce4aa012dfb: Error on parsing. Skipping file.
c6efb6efc1459183131eaf83c7af5532: Error on parsing. Skipping file.
c2811c3d0a444a53bf9713756612fa6f: Error on parsing. Skipping file.
3b14f3a2738d6b9ee3756b6d3f8023dd: Error on parsing. Skipping file.
e2defea35e0fbc329a97866bb71728b6: Error on parsing. Skipping file.
0d00e6c848a28aa887c16bf1cb851e54: Error on parsing. Skipping file.
2c1f878f4cdeca4d27e4fd3299bf4a09: Error on parsing. Skipping file.
f445564c1b23814f682fac9eb0417f1d: Error on parsing. Skipping file.
32d5ac95ece471839210788e1a6b4b15: Error on parsing. Skipping file.
b6f0a5e26670e949e7ff0e9e6b678854: Error on parsing. Skipping file.
8337350c04d6eb215c49f75fdc5d04de: Error on parsing. Skipping file.
984f83ab99cfa54a78590bee4633d5c0: Error on parsing. Skipping file.
87c39084d6c0fab282f5122cc11fd8a9: Error on parsing. Skipping file.
db763fcbfd6c54c6541e9339c2ca83b1: Error on parsing. Skipping file.
f738cf76ac75cb5c87ed4e49957c61a4: Error on parsing. Skipping file.
71db43fe366fba54f10ed120f43a929d: Error on parsing. Skipping file.
ce42713a76b14e09fca6bef9930d936d: Error on parsing. Skipping file.
5e4bffbf078a1b3b8a293adbfe2d7b03: Error on parsing. Skipping file.
b92c5e5818e769af190b62497de5d3e9: Error on parsing. Skipping file.
a6ee8802f4abe4b3a8ea08784b6ab3f3: Error on parsing. Skipping file.
706ca2d5291385890f70b13ec3f56f48: Error on parsing. Skipping file.
87e00532ec85974bcc7617a53f364d45: Error on parsing. Skipping file.
2176985f199b5145c58ddf265308d60d: Error on parsing. Skipping file.
ca4c4761491eb3f76150ae35a7f00e95: Error on parsing. Skipping file.
7be94ba4f201303a3180d98ee31da284: Error on parsing. Skipping file.

Check for prerequisites

Check if wget and pandoc are available, throw error otherwise.

Complete Works?

Hey! Thanks so much for doing this for SSC. If it's easy for you to do with your current setup, would you be willing to create a single epub with all of them, ie The Complete Works of Scott Alexander? Totally fine if you don't have time, but it'd be a great thing to have :)

Add publication date?

If Readablility provides a publication date, think about adding it.

Just a big thanks for this project

First of all, a big thanks for this project.
It's rare to see something cooked up with the simplicity, clareness and usefulness. that you came up with.

Making use of the "right tool for the job" like Pandoc or wget, instead of reinventing the wheell with even more new npm modules.

A very big upvote that you made your project some how to be "portable" and not force or make any "node/npm" package installation mess in the users computer.
This is especially important for users who value a clean and not bloated system, controlled by their package manager.

By choosing,
cd webpages-to-ebook
npm install.

And not recommending any "npm install -g XYZ" , or even worse "sudo npm ..."
This makes it extremelly attractive and usefull for users who are not iespecially interested in node/npm and choose not to have to cope or be forced with the mess of npm packages installation and updates.

It's a pitty your project isn't better known.

I'd like to suggest you to update your README.md page.
You should move tihis part to the very top of the page,

git clone ..
cd web..
node install ..

And stress the fact that your project is self-contained. It doesnt create any "global mess" of npm packages on your system. Maybe also put an example with a recipe for collecting things from different generic sites, like bbc.com or nyt.com. A very simple example of a YAML recipe, not a link to it.

The first time I saw it some months ago, I didnt pay much attention because
you have the (too) many "slatestarcodex" recipes listed very prominently at the top.
I thought, hmm, maybe some guy did a site specific scraper for one or two sites he likes. I'm not going to mess with npm packages just to ttest it.

TypeError: Cannot read property 'toString' of undefined (data.toString())

Hi,

I have an error when trying to execute on definitions/markmanson.yml:

node index.js definitions/markmanson.yml

[A lot of logs saying processing/downloading]

[hidden_path]/webpages-to-ebook/index.js:62
    let html = data.toString();
                    ^

TypeError: Cannot read property 'toString' of undefined
    at ReadFileContext.fs.readFile [as callback] ([hidden_path]/webpages-to-ebook/index.js:62:21)
    at FSReqWrap.readFileAfterOpen [as oncomplete] (fs.js:352:13)

Thank you.

JSON besides YAML

Can we have support for JSON config too? I read both formats map to the same data model, and it's a JS project after all, so maybe it's not that complex to add this feature?

TypeError on Windows

When I try to execute the script, I get
17897d873f2c14b18c7cf350d71e4637: downloading.
17897d873f2c14b18c7cf350d71e4637: downloaded.
4318533943cd7e3cc1c192b91ce26ed0: processing, URL: http://slatestarcodex.com/2016/02/24/two-attitudes-in-psychiatry/
[The same download messages for each other article in the yml.]
C:\Users\Dale\Desktop\webpages-to-ebook-master\index.js:62 let html = data.toString(); ^
TypeError: Cannot read property 'toString' of undefined at ReadFileContext.fs.readFile [as callback] (C:\Users\Dale\Desktop\webpages-to-ebook-master\index.js:62:21) at FSReqWrap.readFileAfterOpen [as oncomplete] (fs.js:422:13)

The downloads also seem to complete suspiciously fast. It could be that that's the part that's broken.