philschmid/clipper.js

HTML to Markdown converter and crawler.

Clipper

Clipper is a Node.js command line tool that allows you to easily clip content from web pages and convert it to Markdown. It uses Mozilla's Readability library and Turndown under the hood to parse web page content and convert it to Markdown.

Clipper provides a quick and simple way to save bits of content from the web for personal archival or note taking purposes, similar to browser extensions like Evernote Web Clipper or Notion Web Clipper. However, Clipper runs entirely in the terminal so you don't need to install any extensions or sign up for accounts.

HTML	Markdown

Installation

npm install -g @philschmid/clipper

note: for crawling you need playwright and the browser dependencies.

Usage

Clip

Options:

• -i, --input <file> | <directory> - Input file (html) or directory to clip content from. If a directory is provided, all files in the directory will be clipped.
• -u, --url <url> - URL to clip content from
• -f, --format <format> - Output format (markdown, json) (default: markdown)
• -o, --output <file> - Output file for clipped content (default: output.md)

Examples:

1. Clip content from a URL:

clipper clip -u <url>

2. Clip content from a file:

clipper clip -i <file>

3. Clip content from a directory, convert a directory of HTML files to a jsonl file:

clipper clip -i <directory> -f json -o dataset.jsonl

Crawl

Warning

Only use this command if you know what you're doing. Crawling websites can be resource intensive and may cause issues for the website owner. Use at your own risk.

Options:

• -u, --url <url> - URL to crawl
• -g, --glob <glob> - Glob pattern to match URLs against
• -o, --output <file> - Output file for crawled content (default: dataset.jsonl)

Examples:

Crawl a site and clip all pages:

clipper crawl -u <url>

Results will be saved in dataset.jsonl file in the current directory.

Alternative use cases

Convert PDF to Markdown

If you want to convert a PDF to Markdown you can use poppler to convert the PDF to HTML and then use Clipper to convert the HTML to Markdown.

pdftohtml -c -s -noframes test.pdf test.html
clipper clip -i test.html

Local Development

• Clone the repo
• Run npm install
• Run npm run test -- clip -u https://huggingface.co/docs/transformers/index to test the CLI
• Run npm run test -- clip -i examples/ to test the CLI with directory input
• Run npm run test -- crawl -u https://awsdocs-neuron.readthedocs-hosted.com/en/v2.14.1/index.html -g https://awsdocs-neuron.readthedocs-hosted.com/en/v2.14.1/\*\*/\* to crawl the AWS Neuron docs
• Run npm run build to build for production
• Run npm install -g . to symlink the CLI for local testing
• Run clipper clip -u https://huggingface.co/docs/transformers/index to build for development
• Remove the symlink with npm r clipper -g

Credits

Clipper uses the following open source libraries:

• Mozilla Readability - For parsing article content
• Turndown - For converting HTML to Markdown
• Crawlee - For crawling websites

License

• Apache 2.0

Release to npm

1. Remove old build files with rm -rf dist
2. Update the version in package.json if minor or major version
3. Run npm run build
4. Run npm publish --access public
5. Create a new release on GitHub
6. Update the version in package.json to the next patch version

clipper.js's Issues

Add support for "directory" input instead of single file

Add support to provide a directory with multiple HTML files instead of a single file for "clipping". The idea would be to read the directory convert files to PDFs and then save them as a single dataset.jsonl

Issue with line wrapping long image names.

Hello,

Thanks for the very good tool! I used it to convert the Machine Learning Glossary page (which is a good test!) and everything worked seemingly alright but images because their names are wrapped and some software, e.g. Obsidian, do not cope up very well with that.

![A dataset broken into four equal groups of examples. In Round 1,
          the first three groups and used for training and the last group
          is used for testing. In Round 2, the first two groups and the last
          group are used for training, while the third group is used for
          testing. In Round 3, the first group and the last two groups are
          used for training, while the second group is used for testing.
          In Round 4, the first group is used is for testing, while the final
          three groups are used for training.](https://developers.google.com/static/machine-learning/glossary/images/k-folds.png)

Github does that said!

Cheers,

Thomas

parse error near `\n'

after install，i use clipper clip -u ，appear ：parse error near `\n'
help

punycode module is deprecated

Hi sir，Thanks for your great tool.

when I try to convert a html to markdown, I encounter an Error, google but not find any solutions. How to solve it ？

Thanks.

Add support for PDF input

Add support for PDFs as "clip" input.