vkhatri/chef-solrcloud

Chef cookbook to Manage Apache Solr

chef-solrcloud's Issues

Issues with setting default['solrcloud']['version'] and wrapper cookook.

Hi,
I have a wrapper cookbook which uses yours. Unfortunately setting default['solrcloud']['version'] in my cookbook works only partially. I uses it some places, like setting filenames and directories, but ie. when fetching .tgz from source it does not use it in creating URL string.

It works better when I use it within environment file to set attribute with version. Then it downloads and installs the desired version, but on the other hand fails to use the correct init script from cookbook (the if on > 5.2 fails)

Oh, I am trying to install solr 5.2.1, not sure if this is even fully supported by your cookbook, thou I have seen a lot of references in the codebase so I am giving it a try.

Installing dependencies for zk gem fail when apt-get update was not run first

Installing dependencies like GCC or patch will fail on Ubuntu when apt-get update was not run first. This happened for me on a fresh EC2 instance. Unfortunately this cannot be satisfied by another cookbook/recipe as installation of this gem and dependencies happen just after cookbook compilation step in Chef. In most cases include_recipe 'apt' somewhere before the package reference helps.

Not sure how this behaves on non-Debian/Ubuntu Linux flavours.

# Require for zk gem
%w(patch gcc make).each do |pkg|
  package pkg do
    action :nothing
    only_if { node['solrcloud']['install_zk_gem'] }
  end.run_action(:install)
end

add more specs

Could not find configName for collection samplecollection found:null

When I deploys SolrCloud using your book and example below, i ge tthe following error when i try to add a new core through the interface

• http://192.168.50.5:8080/solr/#/~cores
• screenshot: http://screencast.com/t/KBWPMsHPVG
• Error CREATEing SolrCore 'new_core': Unable to create core [new_core] Caused by: Could not find configName for collection samplecollection found:null

No matter what i try it seems that SolrCloud is not connecting to the zookeeper

Single Node SolrCloud Test Cluster Deployment

"default_attributes": {
  "solrcloud": {
    "zk_run": true,
    "port": "8080",
    "setup_user": true,
    "manager": true,
    "zkconfigsets": {
      "samplecollection": {}
    },
    "collections": {
      "samplecollection": {
        "collection_config_name": "samplecollection"
      }
    }
  }
}

add attribute for solr service start up log or add an option to choose either solr.log or startup log

Currently logs goes to a start up file and solr.log. Need an option to either choose one of them or make start up log configurable.

/usr/local/solr-4.10.0/etc/solr.keystore not generated

After switching to the latest release (0.3.8) i have a problem that the solr.keystore file is not generated properly.

Running "create-solr.keystore.sh"
Inside "/usr/local/solr-4.10.0/etc/" is working fine.

set default['solrcloud']['enable_ssl'] to false

IMHO, it is better to disable ssl by default

zk gem install is failing

Amazon AMI 2014-09

Gem::Installer::ExtensionBuildError

ERROR: Failed to build gem native extension.

    /opt/chef/embedded/bin/ruby extconf.rb

Building zkc.
tar xzf zkc-3.4.5.tar.gz 2>&1
patch -p0 < patches/zkc-3.4.5-yosemite-htonl-fix.patch 2>&1
patching file zkc-3.4.5/c/include/recordio.h
patching file zkc-3.4.5/c/src/recordio.c
patching file zkc-3.4.5/c/src/zookeeper.c
patching file zkc-3.4.5/c/tests/ZKMocks.cc
patch -p0 < patches/zkc-3.4.5-logging.patch 2>&1
patching file zkc-3.4.5/c/src/zookeeper.c
./configure --prefix=/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/zookeeper-1.4.9/ext --with-pic --without-cppunit --disable-dependency-tracking 2>&1
checking for doxygen... no
checking for perl... /usr/bin/perl
checking for dot... no
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for generated/zookeeper.jute.c... yes
checking for generated/zookeeper.jute.h... yes
checking for gcc... no
checking for cc... no
checking for cc... no
checking for cl... no
configure: error: no acceptable C compiler found in $PATH
See `config.log' for more details.
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers. Check the mkmf.log file for more
details. You may need configuration options.

fix foodcritic once for all

too many to paste here.

zkconfig lwrp does not work properly

LWRP zkconfigset depends alone on option node.solrcloud.manage_zkconfigsets. If this attribute was enabled post a chef run, no zk upload would happen.

better formatting for default sysconfig file parameters

Small typo in libraries/solrcloud.rb

  url << "&router.field=#{otps[:router_field]}" if opts[:router_field]

should be

  url << "&router.field=#{opts[:router_field]}" if opts[:router_field]

add spec kitchen vagrant etc. - branch: spec

Creating branch spec to add chefspec tests, kitchen tests and vagrant.

Feature request (backup and recover)

Hi,

we are using the solr cloud cookbook to provision a cluster on aws opsworks. And it is working
pritty good for now 👍

To improve day to day operations i would propose the following feature (and we could also implement it, but would appreciate your support and feedback):

As far as i see there is no possilility in solr cloud to create a full backup and snapshot of the cloud.
In the web and on the mailing list it is proposed to create index snapshots with the replication handler.
I think it would be good to support this in the coobook to create a snapshot for each not in a given location.

From reading the code i would propose the following steps:

1. Extend your solrcloud.rb library with a "Replication" class that triggers the replication handler
   to store a backup in a defined location on the cluster
2. Implement a lwrp to create a backup for every collection that has the "backup" flag configured
3. Add a create_backup recipe that triggers the backup creation on each node

Restoring the backup is as far as i see not so "straight forward".

What do you think? Do you have othere ideas or proposals?

update solr.xml template for v5.2.1

add attributes for jetty solr context

Helpful to configure context, webapp or war default value.

add test solrcloud_test cookbook

solrcloud should not unconditionally install tool chain (gcc, patch)

In our environment, we do not allow compilers to be installed on our VMs. So we package the zookeeper gems in to an RPM and install that before any dependencies (like zk).

Please add an attribute to prevent the following code from executing in tarball.rb:

# Require for zk gem
%w(patch gcc).each do |pkg|
  package pkg do
    action :nothing
  end.run_action(:install)
end

setup ci

testing in progress..

bump solr version to 5.2.1

add another attribute minor_version

Configset is not updated when "manage_zkconfigsets_source" is false and "manage_zkconfigsets" is true

I would expect that i can disabled the configset source management with "manage_zkconfigsets_source" = false and enable the "manage_zkconfigsets" = true and
with this setting can update the configsets folder by my own and let the cookbook upload the configset.

This works for the initial import but not for the update

make cookbook solr5 ready

bunch of references, but not necessary

https://cwiki.apache.org/confluence/display/solr/Upgrading+a+Solr+4.x+Cluster+to+Solr+5.0
https://cwiki.apache.org/confluence/display/solr/Taking+Solr+to+Production
https://cwiki.apache.org/confluence/display/solr/Major+Changes+from+Solr+4+to+Solr+5
http://lucene.apache.org/solr/5_0_0/changes/Changes.html
https://s.apache.org/Solr-Ref-Guide-PDF

add ssl support for v5.2

Initializing failure after Collection Create API call could create multiple replica set

create collection API call execution create multiple replica on node. If there is any failure or error in configuration after firing collection CREATE api, collection LWRP tries to create it again as the collection will not be present in collections LIST.

collection LWRP need more check and robustness.

One solution could help in minimizing the affect is to disable default attributes - default[:solrcloud][:manage_collections]. This attribute should be enabled for a node which runs chef-client precisely.

solr v5.2.1 service init always returns exit code 0

something to be fixed in solr package

add node attribute for solr major version

add attribute default['solrcloud']['major_version'] to lookup solr major version

Cusomt jetty context is not respected in collection resource provider

I propose to pass "solr/" as default and use the configured jetty context.

providers zkconfigset must refer to attribute :configset_name instead of :name

disable gc_log_opts gc_tune options in favor of java_options

for solr v5.2.x comp we have added two new attribute default['solrcloud']['solr_config']['gc_log_opts'] and default['solrcloud']['solr_config']['gc_tune'] to the cookbook.

IMHO, it would be better to just use node['solrcloud']['java_options'] and drop both of them completely.

this provides simplicity.

creating this issue in case of any objection.

restart solr service on sysconfig and init

add helper method for tarball checksums

this will provide support for older versions.

fix kitchen test

• kitchen test is broken for debian family, need to add apt-get update
• add few more configuration parameters to .kitchen.yml
• add port mapping 8888 -> 8983

solr user unable to start solr v5.2.1 service

there seems to be some issue with $SOLR_INCLUDE, $SOLR_INCLUDE file is not loaded and solr user is unable to start the service.

check whether to keep environment variable ZK_HOST or solr.xml zkHost for solr v5.2

lookup solr-core-version.jar instead of solr-version.war

solr 5 does not have war file, instead look up solr-core.version.jar file

example folder is renamed to server in solr5

perhaps it is better to add another attribute default['solrcloud']['server_base_dir_name'] to refer to base directory for install setup

rubocop

add attribute to support zookeeper chroot

gh-pages outdated

I almost looked away from the project since on http://vkhatri.github.io/chef-solrcloud/ it doesn't mention Solr 5 support.
Consider updating gh-pages or setting https://supermarket.chef.io/cookbooks/solrcloud as the project home page and getting rid of gh page.
Thanks

Error when running on opsworks

Hi,

when i include the cookbook on aws opsworks i have the problem. That solrcloud::tarball fails with the following error:

Cookbook Trace:

/var/lib/aws/opsworks/cache.stage2/cookbooks/solrcloud/recipes/tarball.rb:30:in require' /var/lib/aws/opsworks/cache.stage2/cookbooks/solrcloud/recipes/tarball.rb:30:infrom_file'
/var/lib/aws/opsworks/cache.stage2/cookbooks/opsworks_solrcloud/recipes/install.rb:7:in `from_file'

Relevant File Content:

/var/lib/aws/opsworks/cache.stage2/cookbooks/solrcloud/recipes/tarball.rb:

23:
24: chef_gem "zk" do
25: action :nothing
26: end.run_action(:install)
27:
28: Gem.clear_paths
29:
30>> require 'zk'
31: require 'net/http'
32: require 'json'
33:

Do you have any idea?

Attribute 'configset_name' is not used in 'zkconfigset' provider

In zkconfigset provider name attribute is used, insted of configset_name, as value of -configname paramerter passed to zkcli.

It can be useful to put configuration under different name into zookeeper. Especially in a case, when you have nested directory structure in cookbook files, for example /files/default/solr_config/[CONFIGNAME], and you want to avoid to create similar structure in zookeeper (configs/solr_config/[CONFIGNAME]).

node.solrcloud.manage_zkconfigsets_source triggers configSet re uplod to zookeeper for new nodes

Node attribute node.solrcloud.manage_zkconfigsets_source always re uplod configSets to zookeeper for new nodes.

This attribute needs to be disabled and must be enabled for limited set of nodes.

add new default attribute file for v5.2.1

ignore adminPath in solr.xml for v5

optional purge old versions archive

rotated solr.log is not compressed

solr.log rotated files are not compressed.

no core is loaded after upgrade to a new version

Upgrading solr to a new version does not load any core as it looses solr configsets.

Require a solution either to re-populate configsets or perform a backup from older configsets directory.

Solr not started after reboot

The solr server is not started after rebooting a cluster node, since /var/run/solr is not created and could not created during the init script execution.

setup zkconfigsets directory outside of solr_home

Directory default['solrcloud']['zkconfigsets_home'] is created under node['solrcloud']['install_dir'] which does not persists solr upgrade.

This directory by default to be created outside.