cloneableapp/cloneable

The best free, open source, all-in-one website downloader tool for offline browsing, archiving, and backups.

Cloneable

The all-in-one website downloader tool for offline browsing, archiving, backups, and more.

Screenshots

Downloads

Downloadable binaries are currently not available, but will be soon. For now, follow the steps in the Installation section.

Requirements

If you want to use your own version of wget, your system must have that installed and you must set that path in your settings. Binaries of wget is included with this software but they're not guaranteed to work properly on all systems.

Installation

To run Cloneable locally in development mode:

git clone [email protected]:CloneableApp/Cloneable.git
cd Cloneable
nvm install
nvm use
npm install
npm start

To get a working binary created in dist/:

nvm install
nvm use
npm install
npm postinstall # might be necessary
npm run app:dist

Information

Cloneable is a free and open source desktop application that can clone websites to your local computer automatically, with smart handling of links, images, files, stylesheets, and more, so sites load seamlessly for offline browsing.

It is built with Electron and React. Check the package.json for a list of all dependencies.

Behind the scenes, Cloneable relies heavily on wget.

TODO

• Add dark mode
• Add internationalization
• Add pagination to Clones page
• Add different format options to supply cookies (instead of just Netscape cookies.txt format)
• Add ability to stop and continue Clones, saving progress

Getting Help

Feel free to open an issue here, or email me at [email protected].

❤️ Donating

If you would like to support development of this project, please consider donating. We currently prefer Ko-fi to accept donations. Here's our Ko-fi page.

License

Cloneable is released under the GNU General Public License v3.0.

---

For further information check cloneable.app

cloneable's Issues

Clones not crawling

Cloneable seems to be doing a fantastic job of capturing single pages, but I've been unable to get it to crawl. Have got recursive maximum depth level set to -1. What am I missing?

Error: spawn /tmp/.mount_CloneaA0Kw0N/Resources/bin/wget ENOENT

Download/cloning fails with following error message:

Error: spawn /tmp/.mount_CloneaA0Kw0N/Resources/bin/wget ENOENT

How can I fix this?

Thank you ^^

Clonable seems abandoned -> use httrack instead

For everyone else who ends up here, because the website is great and SEO works well, but then runs into problems during installation (see #2)

Use httrack instead, it works:

brew install httrack
httrack

npm install fails on Mac

This is in a fresh git clone:

npm notice Beginning October 4, 2021, all connections to the npm registry - including for package installation - must use TLS 1.2 or higher. You are currently using plaintext http to connect. Please visit the GitHub blog for more information: https://github.blog/2021-08-23-npm-registry-deprecating-tls-1-0-tls-1-1/
npm notice Beginning October 4, 2021, all connections to the npm registry - including for package installation - must use TLS 1.2 or higher. You are currently using plaintext http to connect. Please visit the GitHub blog for more information: https://github.blog/2021-08-23-npm-registry-deprecating-tls-1-0-tls-1-1/
npm warn cleanup Failed to remove some directories [
npm warn cleanup   [
npm warn cleanup     '/Users/andig/htdocs/Cloneable/node_modules/electron-chromedriver',
npm warn cleanup     [Error: ENOTEMPTY: directory not empty, rmdir '/Users/andig/htdocs/Cloneable/node_modules/electron-chromedriver'] {
npm warn cleanup       errno: -66,
npm warn cleanup       code: 'ENOTEMPTY',
npm warn cleanup       syscall: 'rmdir',
npm warn cleanup       path: '/Users/andig/htdocs/Cloneable/node_modules/electron-chromedriver'
npm warn cleanup     }
npm warn cleanup   ],
npm warn cleanup   [
npm warn cleanup     '/Users/andig/htdocs/Cloneable/node_modules',
npm warn cleanup     [Error: ENOTEMPTY: directory not empty, rmdir '/Users/andig/htdocs/Cloneable/node_modules/electron-chromedriver'] {
npm warn cleanup       errno: -66,
npm warn cleanup       code: 'ENOTEMPTY',
npm warn cleanup       syscall: 'rmdir',
npm warn cleanup       path: '/Users/andig/htdocs/Cloneable/node_modules/electron-chromedriver'
npm warn cleanup     }
npm warn cleanup   ]
npm warn cleanup ]
npm error code 1
npm error path /Users/andig/htdocs/Cloneable/node_modules/better-sqlite3
npm error command failed
npm error command sh -c prebuild-install || npm run build-release
npm error > [email protected] build-release
npm error > node-gyp rebuild --release
npm error prebuild-install warn install No prebuilt binaries found (target=22.2.0 runtime=node arch=arm64 libc= platform=darwin)
npm error gyp info it worked if it ends with ok
npm error gyp info using [email protected]
npm error gyp info using [email protected] | darwin | arm64
npm error gyp info find Python using Python version 3.12.3 found at "/opt/homebrew/opt/[email protected]/bin/python3.12"
npm error gyp WARN install got an error, rolling back install
npm error gyp ERR! configure error
npm error gyp ERR! stack Error: ENOENT: no such file or directory, stat '/tmp/node-v16.14.0-headers.tar.gz'
npm error gyp ERR! System Darwin 23.5.0
npm error gyp ERR! command "/opt/homebrew/Cellar/node/22.2.0/bin/node" "/opt/homebrew/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild" "--release"
npm error gyp ERR! cwd /Users/andig/htdocs/Cloneable/node_modules/better-sqlite3
npm error gyp ERR! node -v v22.2.0
npm error gyp ERR! node-gyp -v v10.1.0
npm error gyp ERR! not ok

npm error A complete log of this run can be found in: /Users/andig/.npm/_logs/2024-06-02T18_45_09_731Z-debug-0.log

Does not build on Node 20

Node version: 20.15.0
OS version: Ubuntu 22.04 LTS
It occurs as of the latest revision of this codebase.

Here is the output of npm install:

npm error code 1
npm error path /home/zenulabidin/Downloads/Cloneable/node_modules/better-sqlite3
npm error command failed
npm error command sh -c prebuild-install || npm run build-release
npm error > [email protected] build-release
npm error > node-gyp rebuild --release
npm error
npm error
npm error make: Entering directory '/home/zenulabidin/Downloads/Cloneable/node_modules/better-sqlite3/build'
npm error   TOUCH ba23eeee118cd63e16015df367567cb043fed872.intermediate
npm error   ACTION deps_sqlite3_gyp_locate_sqlite3_target_copy_builtin_sqlite3 ba23eeee118cd63e16015df367567cb043fed872.intermediate
npm error   TOUCH Release/obj.target/deps/locate_sqlite3.stamp
npm error   CC(target) Release/obj.target/sqlite3/gen/sqlite3/sqlite3.o
npm error rm -f Release/obj.target/deps/sqlite3.a Release/obj.target/deps/sqlite3.a.ar-file-list; mkdir -p `dirname Release/obj.target/deps/sqlite3.a`
npm error ar crs Release/obj.target/deps/sqlite3.a @Release/obj.target/deps/sqlite3.a.ar-file-list
npm error   COPY Release/sqlite3.a
npm error   CXX(target) Release/obj.target/better_sqlite3/src/better_sqlite3.o
npm error rm ba23eeee118cd63e16015df367567cb043fed872.intermediate
npm error make: Leaving directory '/home/zenulabidin/Downloads/Cloneable/node_modules/better-sqlite3/build'
npm error prebuild-install warn install No prebuilt binaries found (target=20.15.0 runtime=node arch=x64 libc= platform=linux)
npm error gyp info it worked if it ends with ok
npm error gyp info using [email protected]
npm error gyp info using [email protected] | linux | x64
npm error gyp info find Python using Python version 3.10.12 found at "/usr/bin/python3"
npm error gyp http GET https://nodejs.org/download/release/v20.15.0/node-v20.15.0-headers.tar.gz
npm error gyp http 200 https://nodejs.org/download/release/v20.15.0/node-v20.15.0-headers.tar.gz
npm error gyp http GET https://nodejs.org/download/release/v20.15.0/SHASUMS256.txt
npm error gyp http 200 https://nodejs.org/download/release/v20.15.0/SHASUMS256.txt
npm error gyp info spawn /usr/bin/python3
npm error gyp info spawn args [
npm error gyp info spawn args '/home/zenulabidin/.config/nvm/versions/node/v20.15.0/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
npm error gyp info spawn args 'binding.gyp',
npm error gyp info spawn args '-f',
npm error gyp info spawn args 'make',
npm error gyp info spawn args '-I',
npm error gyp info spawn args '/home/zenulabidin/Downloads/Cloneable/node_modules/better-sqlite3/build/config.gypi',
npm error gyp info spawn args '-I',
npm error gyp info spawn args '/home/zenulabidin/.config/nvm/versions/node/v20.15.0/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
npm error gyp info spawn args '-I',
npm error gyp info spawn args '/home/zenulabidin/.cache/node-gyp/20.15.0/include/node/common.gypi',
npm error gyp info spawn args '-Dlibrary=shared_library',
npm error gyp info spawn args '-Dvisibility=default',
npm error gyp info spawn args '-Dnode_root_dir=/home/zenulabidin/.cache/node-gyp/20.15.0',
npm error gyp info spawn args '-Dnode_gyp_dir=/home/zenulabidin/.config/nvm/versions/node/v20.15.0/lib/node_modules/npm/node_modules/node-gyp',
npm error gyp info spawn args '-Dnode_lib_file=/home/zenulabidin/.cache/node-gyp/20.15.0/<(target_arch)/node.lib',
npm error gyp info spawn args '-Dmodule_root_dir=/home/zenulabidin/Downloads/Cloneable/node_modules/better-sqlite3',
npm error gyp info spawn args '-Dnode_engine=v8',
npm error gyp info spawn args '--depth=.',
npm error gyp info spawn args '--no-parallel',
npm error gyp info spawn args '--generator-output',
npm error gyp info spawn args 'build',
npm error gyp info spawn args '-Goutput_dir=.'
npm error gyp info spawn args ]
npm error gyp info spawn make
npm error gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
npm error In file included from ./src/better_sqlite3.lzz:11,
npm error                  from ../src/better_sqlite3.cpp:4:
npm error /home/zenulabidin/.cache/node-gyp/20.15.0/include/node/node.h:1248:7: warning: cast between incompatible function types from ‘void (*)(v8::Local<v8::Object>, v8::Local<v8::Value>, v8::Local<v8::Context>)’ to ‘node::addon_context_register_func’ {aka ‘void (*)(v8::Local<v8::Object>, v8::Local<v8::Value>, v8::Local<v8::Context>, void*)’} [-Wcast-function-type]
npm error  1248 |       (node::addon_context_register_func) (regfunc),                  \
npm error       |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
npm error /home/zenulabidin/.cache/node-gyp/20.15.0/include/node/node.h:1266:3: note: in expansion of macro ‘NODE_MODULE_CONTEXT_AWARE_X’
npm error  1266 |   NODE_MODULE_CONTEXT_AWARE_X(modname, regfunc, NULL, 0)
npm error       |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~
npm error /home/zenulabidin/.cache/node-gyp/20.15.0/include/node/node.h:1297:3: note: in expansion of macro ‘NODE_MODULE_CONTEXT_AWARE’
npm error  1297 |   NODE_MODULE_CONTEXT_AWARE(NODE_GYP_MODULE_NAME,                     \
npm error       |   ^~~~~~~~~~~~~~~~~~~~~~~~~
npm error ./src/better_sqlite3.lzz:67:1: note: in expansion of macro ‘NODE_MODULE_INIT’
npm error ./src/util/macros.lzz: In function ‘void SetPrototypeGetter(v8::Isolate*, v8::Local<v8::External>, v8::Local<v8::FunctionTemplate>, const char*, v8::AccessorGetterCallback)’:
npm error ./src/util/macros.lzz:157:21: error: ‘v8::AccessorSignature’ has not been declared
npm error ./src/util/binder.lzz: In static member function ‘static bool Binder::IsPlainObject(v8::Isolate*, v8::Local<v8::Object>)’:
npm error ./src/util/binder.lzz:37:51: error: ‘class v8::Object’ has no member named ‘CreationContext’; did you mean ‘GetCreationContext’?
npm error ./src/util/data.lzz: In function ‘v8::Local<v8::Value> Data::GetValueJS(v8::Isolate*, sqlite3_stmt*, int, bool)’:
npm error ./src/util/data.lzz:73:92: warning: this statement may fall through [-Wimplicit-fallthrough=]
npm error ./src/util/data.lzz:73:197: note: here
npm error ./src/util/data.lzz: In function ‘v8::Local<v8::Value> Data::GetValueJS(v8::Isolate*, sqlite3_value*, bool)’:
npm error ./src/util/data.lzz:77:81: warning: this statement may fall through [-Wimplicit-fallthrough=]
npm error ./src/util/data.lzz:77:175: note: here
npm error make: *** [better_sqlite3.target.mk:120: Release/obj.target/better_sqlite3/src/better_sqlite3.o] Error 1
npm error gyp ERR! build error 
npm error gyp ERR! stack Error: `make` failed with exit code: 2
npm error gyp ERR! stack at ChildProcess.<anonymous> (/home/zenulabidin/.config/nvm/versions/node/v20.15.0/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:209:23)
npm error gyp ERR! System Linux 5.15.0-107-generic
npm error gyp ERR! command "/home/zenulabidin/.config/nvm/versions/node/v20.15.0/bin/node" "/home/zenulabidin/.config/nvm/versions/node/v20.15.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild" "--release"
npm error gyp ERR! cwd /home/zenulabidin/Downloads/Cloneable/node_modules/better-sqlite3
npm error gyp ERR! node -v v20.15.0
npm error gyp ERR! node-gyp -v v10.1.0
npm error gyp ERR! not ok

The issue appears to be with using an old version of the better-sqlite3 pre-built binary which does not compile correctly with node 20 headers. I think it can be fixed by simply updating the dependency spec.

As a workaround, installing with Node 18 still works.

Update Node version as v16.13.1 is EOL

See: NixOS/nixpkgs#321583 (comment)

@ChrisMoutsos