clarketm/hugo-elasticsearch

Generate Elasticsearch indexes for Hugo static sites by parsing front matter

Hugo-Elasticsearch (HES)

Generate Elasticsearch indexes for Hugo static sites by parsing front matter.

Installation

Install with npm

$ npm install hugo-elasticsearch

Demo

Usage

CLI

NAME:
    hugo-elasticsearch
    hes (alias)

SYNOPSIS:
    hes [ opts... ]

DESCRIPTION:
    Generate Elasticsearch indexes from Hugo front matter.

OPTIONS:
    -i, --input path            Input path.                         (default: "content/**")
    -o, --output path           Output path.                        (default: "public/elasticsearch.json")
    -l, --language lang         Language [toml | yaml].             (default: "toml")
    -d, --delimiter delim       Delimiter [toml: +++ | yaml: ---].  (optional)
    -n, --name name             Index name.                         (optional)

Long form

$ hugo-elasticsearch \
 --input "content/**" \
 --output "public/elasticsearch.json" \
 --language "toml" \
 --delimiter "+++" \
 --index-name "posts"

Short form

$ hes \
 -i "content/**" \
 -o "public/elasticsearch.json" \
 -l "toml" \
 -d "+++" \
 -n "posts"

NPM Scripts

...
"scripts": {
  "index": "hes -i 'content/**' -o 'public/elasticsearch.json'"
  "index:toml": "hes -i 'content/toml/**' -o 'public/toml/elasticsearch.json' -l 'toml' -d '+++'"
  "index:yaml": "hes -i 'content/yaml/**' -o 'public/yaml/elasticsearch.json' -l 'yaml' -d '---'"
},
...

API

const hes = require('hugo-elasticsearch');

const Indexer = new hes({
  input: 'content/blog/**',
  output: 'public/static/elasticsearch.json',
  language: 'yaml',
  delimiter: '---',
  indexName: 'posts'
});

// Create index
Indexer.index()

// Setters
Indexer.setInput('content/blog/**');
Indexer.setOutput('public/static/elasticsearch.json');
Indexer.setLanguage('yaml');
Indexer.setDelimiter('---');
Indexer.setIndexName('posts');

Example

1. Create a directory named content.

$ mkdir 'content'

2. Create a markdown file with toml front matter in a file named content/test-toml.md.

$ cat > 'content/test-toml.md' <<EOF
+++
title = "Sample title"
description = "Sample description"
tags = [ "tag1" ]
+++

# Sample content header
Sample content body

EOF

3. Generate a newline delimited json file for indexing in Elasticsearch and output it to a file named public/elasticsearch.json.

$ hes -i 'content/**' -o 'public/elasticsearch.json'

4. Bulk upload your json file to a running Elasticsearch instance.

cURL

$ HOST="localhost"
$ PORT="9200"
$ INDEX="index"
$ TYPE="type"

$ curl \
  -H "Content-Type: application/x-ndjson" \
  -XPOST "$HOST:$PORT/$INDEX/$TYPE/_bulk" \
  --data-binary "@./public/elasticsearch.json"

{
  "took": 137,
  "errors": false,
  "items": [
    ...
  ]
}

JavaScript

const Elastic = require("elasticsearch");
const ndjson = require("ndjson");
const fs = require("fs");

const client = new Elastic.Client({host: 'localhost:9200'});

const fetchBulkJson = () => {
    return new Promise((resolve, reject) => {
      let lines = [];

      fs.createReadStream("./public/elasticsearch.json")
        .pipe(ndjson.parse())
        .on("data", line => lines.push(line))
        .on("end", () => resolve(lines))
        .on("error", err => reject(err));
    });
};

// Perform the bulk index operations in a single API call.
const bulkUpload = async () => {
    const json = await this.fetchBulkJson();
    return await client.bulk({ body: json });
};

Java, Python, Ruby, ...

Although the bulk upload examples above are only for cUrl and JavaScript, this format will work seamlessly with any one of the numerous Elasticsearch clients.

5. You content is now successfully indexed in Elasticsearch 👍. Happy elastic searching!

Refer to the content directory in the root of this project for examples of both yaml and toml content (i.e. .md files).

Refer to the public directory in the root of this project for examples of ndjson files (i.e. Elasticsearch index files) generated from both yaml and toml content.

Sites using hugo-elasticsearch

• https://blog.travismclarke.com/

License

Apache-2.0 © Travis Clarke

hugo-elasticsearch's Issues

is there any way to exclude a file from folder when generating json file for indexing

Hi
i am using this package in my hugo site.it is working fine.but now i need to exclude a file from indexing .is there any way to exclude certain files from indexing?
Thank you

error not showing when json file creation of md files

Hi @clarketm
when i am creating json file for indexing based on md files.it is getting struck in between.but didnot display any errors

/node_modules/hugo-elasticsearch/lib/hes.js:187 throw _iteratorError; ^ YAMLException: end of the stream or a document separator is expected at line 7, column 1: ^ at generateError (/node_modules/js-yaml/lib/js-yaml/loader.js:167:10) at throwError (/node_modules/js-yaml/lib/js-yaml/loader.js:173:9) at readDocument (/node_modules/js-yaml/lib/js-yaml/loader.js:1539:5) at loadDocuments (/node_modules/js-yaml/lib/js-yaml/loader.js:1575:5) at load (/node_modules/js-yaml/lib/js-yaml/loader.js:1596:19) at Object.safeLoad (/node_modules/js-yaml/lib/js-yaml/loader.js:1618:10) at module.exports (/node_modules/gray-matter/lib/parse.js:12:17) at parseMatter (/node_modules/gray-matter/index.js:109:17) at matter (/node_modules/gray-matter/index.js:50:10) at Function.matter.read (/node_modules/gray-matter/index.js:180:16)

is there any way to show the md file name or line number which causing this error..i have 100s of md files so it is very hard to find which md file is corrupted or causing the error
Thank you

Usage on Windows

I am trying to use hes on windows and I am unable to create any indexes. I followed the same steps of creating content directory and a sample .md file and ran the command

hes -i 'content**' -o 'public\elasticsearch.json'

I get the error no content found. Can you please provide an example for windows usage.

Please stop using remove-markdown

The library has a bunch of pretty bad things:

• It chokes on markdown tables
• It replaces things it shouldn't.

For the 2nd one, see e.g.: https://github.com/stiang/remove-markdown/blob/master/index.js#L33

An example:

'foo < bar baz >'.replace(/<[^>]*>/g, '')
"foo "

Ideally, the library should use a markdown parser, but remove-markdown is pretty far from it, so I'd suggest outright removing its use instead.

That said, I do not have a replacement to suggest :) not trying to be a smartass, really, just thought I'd give my opinion on a lib I had to monkey patch out of hugo-elasticsearch when adding it to my site.

Cheers,