elastic/esvm

Elasticsearch Version Manager

esvm

Elasticsearch Version Manager is a command line application used for development to manage different versions of Elasticsearch. It should not be used in production since it's probably a bad idea to wrap a process with Node.js. But nevertheless it's extremely useful if you need to develop against multiple versions of Elasticsearch (I'm looking at you Kibana team).

Installation

npm install esvm -g

Usage

Usage: esvm [options] <version>

  Options:

    -h, --help             output usage information
    -V, --version          output the version number
    -p, --purge            purge the data directory
    -f, --fresh            preform a fresh install
    -b, --branch           install from a branch release
    -n, --nodes <n>        the number of nodes to start
    -c, --config <file>    the config file to use
    -l, --list             list clusters in the config file
    --cluster-name <name>  the cluster name to use

Note: <version> can either be a semver expression of a named cluster (which is found in the configuration).

Configuration

esvm will look for an .esvmrc file in the current working directory. If it doesn't find one there it will continue to walk up the directory tree until it does. There are two main sections for the configuration file:

• defaults - The defaults to apply to every instance
• clusters - The named clusters

Both the defaults and named clusters will take the following options:

Option	Description
version	The semver statment for the released version of Elasticsearch to install. Will override branch and binary options, and install the latest version by default
branch	The nightly branch to install
binary	The path to the tarball to use. This can either be URL or file path.
directory	The directory where everything is installed. If the directory doesn't exist it will be created. (Default: ~/.esvm)
plugins	The plugins to install. This should be an array of plugin install directives.
purge	Purge the data directory when starting the server (Default: false).
fresh	Remove the current copy before installing a new copy (Default: false).
nodes	The number of nodes to start. This can either be a number or an array of config objects (Default: 1)
config	The config to start the server with.
shield	Configuration settings for Shield. Check the examples for help

For some example configurations, please consult the example files

esvm's Issues

Oops Error when running ~/.esvm/5.6.9/bin/elasticsearch -h

$ esvm 5
Apr 23, 2019 10:09 AM INFO cluster Downloading & Installing 5.6.9
Apr 23, 2019 10:09 AM INFO cluster Installing plugins
Apr 23, 2019 10:09 AM INFO cluster Starting 1 nodes
Oops Error: Command failed: /home/meowsus/.esvm/5.6.9/bin/elasticsearch -h

    at ChildProcess.exithandler (child_process.js:294:12)
    at ChildProcess.emit (events.js:189:13)
    at maybeClose (internal/child_process.js:970:16)
    at Socket.stream.socket.on (internal/child_process.js:389:11)
    at Socket.emit (events.js:189:13)
    at Pipe._handle.close (net.js:600:12)

I've tried removing the ~/.esvm/5.6.9 directory which triggered the new download, as seen above, but still the ChildProcess explodes.

I did just upgrade my system to Ubuntu 19.04.

Add cluster list functionality

I find myself checking my rc file pretty often to see how I've named my cluster configs and what's available.

Adding a -l, --list option that showed what configs I have available would be pretty handy.

Auto-update installs

There should be a way to tell esvm that it should check for a newer version for the version pattern specified. When it's found, install it, otherwise install the current version.

How to config when run es with a system property?

HI,
When I have run es with a system property like es.insecure.allow.root, something like this:

bin/elasticsearch -Des.insecure.allow.root=true

How to config esvm with the property? Thanks.

1.x builds not outputting elasticsearch log

starting up a 1.x node, stdout does not contain contain the elasticsearch output.

Example output (node is fully running).

Mar 3, 2016 4:04 PM INFO cluster Installing plugins
Mar 3, 2016 4:04 PM INFO cluster Starting 1 nodes

Error when starting the cluster with ingest-geoip plugin installed

When I try to start up a node with the ingest-geoip plugin installed via esvm, I get the following error:

https://gist.github.com/Bargs/eda140685f291438aab8

This error doesn't occur when manually starting the node with the elasticsearch binary, so it seems to be some compatibility issue between the plugin and esvm.

Add --prefix option for use in PATH

If I want to access one of the elasticsearch binaries (like plugin), I need to directly access them. It would be great to have a --prefix switch like brew so I could add something like the following line to my PATH variable for easy access:

PATH="$(esvm --prefix)/bin:$PATH"

Thanks for a great tool!

Oops Error: Unable to find a version for "*"

It used to work fine and I see a few versions installed in ~/.esvm, but recently I'm not able to either install or start any version:

Oops Error: Unable to find a version for "*"
    at /usr/local/opt/nvm/versions/node/v6.11.3/lib/node_modules/esvm/node_modules/libesvm/lib/resolveVersion.js:39:11
    at tryCatcher (/usr/local/opt/nvm/versions/node/v6.11.3/lib/node_modules/esvm/node_modules/bluebird/js/main/util.js:26:23)
    at Promise._settlePromiseFromHandler (/usr/local/opt/nvm/versions/node/v6.11.3/lib/node_modules/esvm/node_modules/bluebird/js/main/promise.js:507:31)
    at Promise._settlePromiseAt (/usr/local/opt/nvm/versions/node/v6.11.3/lib/node_modules/esvm/node_modules/bluebird/js/main/promise.js:581:18)
    at Async._drainQueue (/usr/local/opt/nvm/versions/node/v6.11.3/lib/node_modules/esvm/node_modules/bluebird/js/main/async.js:128:12)
    at Async._drainQueues (/usr/local/opt/nvm/versions/node/v6.11.3/lib/node_modules/esvm/node_modules/bluebird/js/main/async.js:133:10)
    at Immediate.Async.drainQueues (/usr/local/opt/nvm/versions/node/v6.11.3/lib/node_modules/esvm/node_modules/bluebird/js/main/async.js:15:14)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5)

esvm -l fails with:

TypeError: Cannot convert undefined or null to object
    at module.exports (/usr/local/opt/nvm/versions/node/v6.11.3/lib/node_modules/esvm/lib/formatClusters.js:15:29)
    at Object.<anonymous> (/usr/local/opt/nvm/versions/node/v6.11.3/lib/node_modules/esvm/bin/esvm.js:106:15)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.runMain (module.js:604:10)
    at run (bootstrap_node.js:389:7)
    at startup (bootstrap_node.js:149:9)
    at bootstrap_node.js:502:3

Same issue persists after deleting the cache folder and the ~/.esvm directory.

nodes array configuration not working

https://github.com/simianhacker/esvm/blob/master/sample/esvmrc#L7-L26 does not work. Throws the following error:

MALFORMED LOG MESSAGE: "EXCEPTION IN THREAD "MAIN" SETTINGSEXCEPTION[FAILED TO LOAD SETTINGS FROM [ELASTICSEARCH.JSON]]; NESTED: ELASTICSEARCHPARSEEXCEPTION[DUPLICATE SETTINGS KEY [CLUSTER.NAME] FOUND AT LINE NUMBER [1], COLUMN NUMBER [57], PREVIOUS VALUE [JAREDS-MBP.NETWORK.TDS.NET], CURRENT VALUE [JARED-TESTING]];" cluster undefined
Jan 26, 2016 10:16 PM MALFORMED LOG MESSAGE: "LIKELY ROOT CAUSE: ELASTICSEARCHPARSEEXCEPTION[DUPLICATE SETTINGS KEY [CLUSTER.NAME] FOUND AT LINE NUMBER [1], COLUMN NUMBER [57], PREVIOUS VALUE [JAREDS-MBP.NETWORK.TDS.NET], CURRENT VALUE [JARED-TESTING]]" cluster undefined

The mergeConfig looked simple enough, but I'm not spotting the issue. From the error above, it seems that the default config: { 'cluster.name': os.hostname() (previous value) is passing in addition to the configured value (current value).

If I take out the "cluster" object, it will launch a cluster; except that all the nodes in the cluster will have the same name, the last configured name in the nodes array.

List available versions

It would be nice if there was a flag to list the available versions of ES, perhaps using an npm-style matching pattern as well. Something like:

$ esvm -l
-> list of all available versions

$ esvm -l 1.x
-> list of all 1.x versions

It's possible it could just query github tags, as they should map to available tar packages.

Document how to use a local plugin

Highest version 2.4.0

I was curious why the highest version I can install with esvm is v2.4.0. I submitted a PR hoping that a bump to the libesvm library would solve the issue, but it didn't seem to.

I'd at least like to be able to install patch versions on the v2.4.x series.

elasticsearch 6.2.3 snapshot

How do I build a new snapshot or whatever to get the 6.2.3 version?
Currently I cannot develop anything for Kibana 6.2.3, since TEST_ES_BRANCH accepts a minor release only (which is also a branch) and that fetches a 6.2 branch, which is already way ahead of 6.2.3 and gives a 6.2.4-SNAPSHOT.

esvm -f 6.2.3   
Oops Error: Unable to find a version for "6.2.3" in "6.2.2", "6.2.1", "6.2.0", "6.1.3", "6.1.2", "6.1.1", "6.1.0", "6.0.1", "6.0.0", "5.6.8", "5.6.7", "5.6.6", "5.6.5", "5.6.4", "5.6.3", "5.6.2", "5.6.1", "5.6.0", "5.5.3", "5.5.2", "5.5.1", "5.5.0", "5.4.3", "5.4.2", "5.4.1", "5.4.0", "5.3.3", "5.3.2", "5.3.1", "5.3.0", "5.2.2", "5.2.1", "5.2.0", "5.1.2", "5.1.1", "5.0.2", "5.0.1", "5.0.0", "2.4.6", "2.4.5", "2.4.4", "2.4.3", "2.4.2", "2.4.1", "2.4.0", "2.3.5", "2.3.4", "2.3.3", "2.3.2", "2.3.1", "2.3.0", "2.2.2", "2.2.1", "2.2.0", "2.1.2", "2.1.1", "2.1.0", "2.0.2", "2.0.1", "2.0.0", "1.7.6", "1.7.5", "1.7.4", "1.7.3", "1.7.2", "1.7.1", "1.7.0", "1.6.2", "1.6.1", "1.6.0", "1.5.2", "1.5.1", "1.5.0", "1.4.5", "1.4.4", "1.4.3", "1.4.2", "1.4.1", "1.4.0", "1.3.9", "1.3.8", "1.3.7", "1.3.6", "1.3.5", "1.3.4", "1.3.3", "1.3.2", "1.3.1", "1.3.0", "1.2.4", "1.2.3", "1.2.2", "1.2.1", "1.2.0", "1.1.2", "1.1.1", "1.1.0", "1.0.3", "1.0.2", "1.0.1", "1.0.0", "0.90.9", "0.90.8", "0.90.7", "0.90.6", "0.90.5", "0.90.4", "0.90.3", "0.90.2", "0.90.1", "0.90.0"
    at /usr/local/lib/node_modules/esvm/node_modules/libesvm/lib/resolveVersion.js:39:11
    at tryCatcher (/usr/local/lib/node_modules/esvm/node_modules/bluebird/js/main/util.js:26:23)
    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/esvm/node_modules/bluebird/js/main/promise.js:507:31)
    at Promise._settlePromiseAt (/usr/local/lib/node_modules/esvm/node_modules/bluebird/js/main/promise.js:581:18)
    at Promise._settlePromises (/usr/local/lib/node_modules/esvm/node_modules/bluebird/js/main/promise.js:697:14)
    at Async._drainQueue (/usr/local/lib/node_modules/esvm/node_modules/bluebird/js/main/async.js:123:16)
    at Async._drainQueues (/usr/local/lib/node_modules/esvm/node_modules/bluebird/js/main/async.js:133:10)
    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/esvm/node_modules/bluebird/js/main/async.js:15:14)
    at runCallback (timers.js:794:20)
    at tryOnImmediate (timers.js:752:5)
    at processImmediate [as _immediateCallback] (timers.js:729:5)

Why does it build 6.2.4-SNAPSHOT if there's nothing above 6.2.2 in this list?

Change HOME to a function that is compatible with Windows and *nix

Override the version via command line arguments.

If you have a named cluster provide a way to override the version so you an do something like:

esvm --version 1.2 2m3d
esvm --version 1.3 2m3d
esvm --version 1.1 2m3d

Unhandled zlib error event when downloading 6.0.0

Here's the error:

~/Work $ esvm                
Nov 20, 2017 12:54 PM INFO cluster Downloading & Installing 6.0.0                                                     
events.js:182                
      throw er; // Unhandled 'error' event                 
      ^                      

Error: incorrect header check                              
    at Gunzip.zlibOnError (zlib.js:152:15)  

Esvm requires internet connection

Esvm makes a request to api.github.com to try and get the latest version of ES.

It should use the currently available version if there is one.

Consistent speed issues

So, hey.

This isn't going to be the most helpful of issues to open, but I'm going to open it anyway because it's an issue I'm definitely hitting.

My dev stack is OSX High Sierra -> Ruby 2.4.2 -> Rails 5.1.4 -> Mongo 3.4.10 -> Elasticsearch 5.6.4 (via esvm).

When I'm seeding my Rails app, after running esvm 5.6 to get it up and running, my seed takes about 50 minutes to complete, whereas if I start elasticsearch via ~/src/.esvm/5.6.4/bin/elasticsearch it takes the expected 10 or so to complete.

I've poured over the logs for a few days now and nothing is really pointing me to any obvious problems. esvm just seems like it's significantly causing indexing issues, from what I can tell. Just generally slow moving, and it's been happening, for reseeds, for the last two or so weeks.

Let me know if you have any ideas of how to debug this for you guys. Alternatively, if it's just too vague, feel free to close this issue.

React to elasticsearch dying

Currently, when the elasticsearch process is killed by some other tool esvm stays up, logs the "shutdown" message, and the prevents you from using ⌘-c.

Workaround: ⌘-d

Nested properties overwrite non-nested properties

We have one shield settings that is both part of the hierarchy of setting names, and also a leaf-node setting:
shield.transport.ssl is a boolean that turns ssl on and off
shield.transport.ssl.client.auth is a boolean that turns on/off client-certificate authentication

if shield.transport.ssl is set before shield.transport.ssl.client.auth it gives this error:

esvm (1.0) $ esvm

/usr/local/lib/node_modules/esvm/lib/explodeBy.js:9
        memo[_key] = value;
                   ^
TypeError: Cannot set property 'auth' of undefined
    at walk (/usr/local/lib/node_modules/esvm/lib/explodeBy.js:9:20)
    at walk (/usr/local/lib/node_modules/esvm/lib/explodeBy.js:12:9)
    at walk (/usr/local/lib/node_modules/esvm/lib/explodeBy.js:12:9)
    at walk (/usr/local/lib/node_modules/esvm/lib/explodeBy.js:12:9)
    at walk (/usr/local/lib/node_modules/esvm/lib/explodeBy.js:12:9)
    at walk (/usr/local/lib/node_modules/esvm/lib/explodeBy.js:12:9)
    at /usr/local/lib/node_modules/esvm/lib/explodeBy.js:14:7
    at forOwn (/usr/local/lib/node_modules/esvm/node_modules/lodash/dist/lodash.js:2105:15)
    at Function.forEach (/usr/local/lib/node_modules/esvm/node_modules/lodash/dist/lodash.js:3302:9)
    at module.exports (/usr/local/lib/node_modules/esvm/lib/explodeBy.js:4:5)

If shield.transport.ssl.client.auth comes first, it gets overridden by shield.transport.ssl

sudo requirement to run

$ esvm
Oops Error: EACCES, open '/usr/local/lib/node_modules/esvm/node_modules/libesvm/lib/../.cache.json'

When installed globally and when not running n, nvm, nave, etc., esvm doesn't launch correctly and required sudo to run. It probably shouldn't.

getcwd error when installing 5.6.8

This is what I get when I ran esvm 5 this morning:

$ esvm 5                                                                                                                                                                                                                     
Mar 15, 2018 11:08 AM INFO cluster Downloading & Installing 5.6.8                                                                                                                                                                             
Mar 15, 2018 11:08 AM INFO cluster Installing plugins                                                                                                                                                                                         
Mar 15, 2018 11:08 AM INFO cluster Starting 1 nodes                                                                                                                                                                                           
Oops Error: Command failed: /home/meowsus/.esvm/5.6.8/bin/elasticsearch -h                                                                                                                                                                    
sh: 0: getcwd() failed: No such file or directory                                                                                                                                                                                             
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory                                                                                                                           
chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory                                                                                                                                
                                                                                                                                                                                                                                              
    at ChildProcess.exithandler (child_process.js:275:12)                                                                                                                                                                                     
    at emitTwo (events.js:126:13)                                                                                                                                                                                                             
    at ChildProcess.emit (events.js:214:7)                 
    at maybeClose (internal/child_process.js:925:16)       
    at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)    
                                       
$ esvm 5                                  
/home/meowsus/.npm/lib/node_modules/esvm/node_modules/graceful-fs/polyfills.js:11                                      
    cwd = origCwd.call(process)                            
                  ^                                        

Error: ENOENT: no such file or directory, uv_cwd           
    at process.cwd (/home/meowsus/.npm/lib/node_modules/esvm/node_modules/graceful-fs/polyfills.js:11:19)              
    at Object.resolve (path.js:1167:25)                    
    at Function.Module._resolveLookupPaths (module.js:424:17)                                                          
    at Function.Module._resolveFilename (module.js:541:20) 
    at Function.Module._load (module.js:474:25)            
    at Module.require (module.js:596:17)                   
    at require (internal/module.js:11:18)                  
    at Object.<anonymous> (/home/meowsus/.npm/lib/node_modules/esvm/bin/esvm.js:11:21)                                 
    at Module._compile (module.js:652:30)                  
    at Object.Module._extensions..js (module.js:663:10)    
    at Module.load (module.js:565:32)                      
    at tryModuleLoad (module.js:505:12)                    
    at Function.Module._load (module.js:497:3)             
    at Function.Module.runMain (module.js:693:10)          
    at startup (bootstrap_node.js:188:16)                  
    at bootstrap_node.js:609:3    

I confirmed that running ~/.esvm/5.6.8/bin/elasticsearch directly works just fine.

ath.conf=/

what is this folder ath.conf=/ showing up in my project folders when running esvm?

Oops Error: Unable to find a version for "1.7.0"

Good evening guys, did anyone have a similar problem? I used to use a lot of esvm, but I had to migrate to evm (another version manager), the installation of Elasticsearch is correct and the application runs okay, but the esvm 1.7.0 command it returns:

Oops Error: Unable to find a version for "1.7.0"
    at /usr/lib/node_modules/esvm/node_modules/libesvm/lib/resolveVersion.js:39:11
    at tryCatcher (/usr/lib/node_modules/esvm/node_modules/bluebird/js/main/util.js:26:23)
    at Promise._settlePromiseFromHandler (/usr/lib/node_modules/esvm/node_modules/bluebird/js/main/promise.js:507:31)
    at Promise._settlePromiseAt (/usr/lib/node_modules/esvm/node_modules/bluebird/js/main/promise.js:581:18)
    at Async._drainQueue (/usr/lib/node_modules/esvm/node_modules/bluebird/js/main/async.js:128:12)
    at Async._drainQueues (/usr/lib/node_modules/esvm/node_modules/bluebird/js/main/async.js:133:10)
    at Immediate.Async.drainQueues (/usr/lib/node_modules/esvm/node_modules/bluebird/js/main/async.js:15:14)
    at runCallback (timers.js:570:20)
    at tryOnImmediate (timers.js:550:5)
    at processImmediate [as _immediateCallback] (timers.js:529:5)

Enhancement: esvm init

Similar to npm init or bower init, it would ask some questions and generate a baseline .esvm file from the answers.

Taking the idea even further, it might be useful to provide a yeoman style selection of items, like plugins, version, or even choosing existing cluster definitions from ~/.esvm

ES Plugin installation hangs on Windows

I suspect it's due to esvm trying to use the plugin shell script, instead of the plugin.bat file.

It does not error out though, it just sits there saying it's trying to install whatever plugin you've specified.

Unable to run esvm due to syntax error

I'm getting this when I try to run esvm:

» esvm

/usr/lib/node_modules/esvm/node_modules/rcloader/node_modules/rcfinder/index.js:138
      if (sync && err) throw err;
                             ^
SyntaxError: Unexpected string
    at Object.parse (native)
    at RcFinder.loader (/usr/lib/node_modules/esvm/node_modules/rcloader/node_modules/rcfinder/index.js:20:17)
    at get (/usr/lib/node_modules/esvm/node_modules/rcloader/node_modules/rcfinder/index.js:68:18)
    at respond (/usr/lib/node_modules/esvm/node_modules/rcloader/node_modules/rcfinder/index.js:112:37)
    at RcFinder.find (/usr/lib/node_modules/esvm/node_modules/rcloader/node_modules/rcfinder/index.js:158:14)
    at RcLoader.for (/usr/lib/node_modules/esvm/node_modules/rcloader/index.js:76:43)
    at Object.<anonymous> (/usr/lib/node_modules/esvm/bin/esvm.js:18:26)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)

Importing template files

it would be really handy to be able to import template files from a given path and have them copied into the cluster's config.

See http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.x/indices-templates.html#config

Something like this perhaps:

{
  "defaults": {
    "config": {
      "http.cors.enabled": true,
    },
    "templates": "./templates",
  },
  "clusters": {
    "my-rad-cluster": {
      "templates": "./rad-templates",
    }
  }
}

Where templates could be placed in the global config and/or the cluster configs. I suppose if both were used, it would just copy templates files from both locations.

--list should read the ~/.esvmrc file when no file arg is provided

I had expected the --list arg to show the configuration from the ~/.esvmrc, but if issued with no -c file.name it will startup elasticsearch.

https://github.com/simianhacker/esvm/blob/master/bin/esvm.js#L102-L106

Expected usage of branch flag

Looking at the existing ES branch list, I see a 1.4 branch:

However, when I try to install it, I get:

$ sudo esvm -f -b 1.4
Sep 17, 2014 5:31 PM INFO cluster Downloading & installing from "1.4" branch.

/usr/local/lib/node_modules/esvm/node_modules/progress/lib/node-progress.js:131
  complete = Array(completeLength + 1).join(this.chars.complete);

Is this a bug, or am I just doing it wrong?