rubygems/gemstash

A RubyGems.org cache and private gem server

RubyGems

RubyGems is a package management framework for Ruby.

A package (also known as a library) contains a set of functionality that can be invoked by a Ruby program, such as reading and parsing an XML file. We call these packages "gems" and RubyGems is a tool to install, create, manage and load these packages in your Ruby environment.

RubyGems is also a client for RubyGems.org, a public repository of Gems that allows you to publish a Gem that can be shared and used by other developers. See our guide on publishing a Gem at guides.rubygems.org

Getting Started

Installing and managing a Gem is done through the gem command. To install a Gem such as Nokogiri which lets you read and parse XML in Ruby:

$ gem install nokogiri

RubyGems will download the Nokogiri Gem from RubyGems.org and install it into your Ruby environment.

Finally, inside your Ruby program, load the Nokogiri gem and start parsing your XML:

require 'nokogiri'

Nokogiri.XML('<h1>Hello World</h1>')

For more information about how to use RubyGems, see our RubyGems basics guide at guides.rubygems.org

Requirements

• RubyGems supports Ruby 3.0 or later.

Installation

RubyGems is already installed in your Ruby environment, you can check the version you have installed by running gem --version in your terminal emulator.

In some cases Ruby & RubyGems may be provided as OS packages. This is not a recommended way to use Ruby & RubyGems. It's better to use a Ruby Version Manager, such as rbenv or chruby. If you still want to use the version provided by your OS package manager, please also use your OS package manager to upgrade rubygems, and disregard any other installation instructions given below.

If you would like to manually install RubyGems:

• Download from https://rubygems.org/pages/download, unpack, and cd into RubyGems' src
• OR clone this repository and cd into the repository

Install RubyGems by running:

$ ruby setup.rb

For more details and other options, see:

$ ruby setup.rb --help

Upgrading RubyGems

To upgrade to the latest RubyGems, run:

$ gem update --system

See UPGRADING for more details and alternative instructions.

Release policy

RubyGems and Bundler are released in sync, although they do not share their major version number. It is planned that also their major version numbers will be sync'ed in the future.

The release policy is somewhat similar to the release policy of Ruby itself:

• Frequent patch releases (every 2-4 weeks) including bug fixes, minor enhancements, small features, or even medium sized features declared as experimental for battle testing.
• Yearly minor releases including bigger features, and minor breaking changes (affecting only edge cases and a very small set of users).
• Occasional major releases (replacing yearly minors) including major breaking changes.

Documentation

RubyGems uses rdoc for documentation. A compiled set of the docs can be viewed online at rubydoc.

RubyGems also provides a comprehensive set of guides which covers numerous topics such as creating a new gem, security practices and other resources at https://guides.rubygems.org

Getting Help

Filing Tickets

Got a bug and you're not sure? You're sure you have a bug, but don't know what to do next? In any case, let us know about it! The best place for letting the RubyGems team know about bugs or problems you're having is on the RubyGems issues page at GitHub.

Bundler Compatibility

See https://bundler.io/compatibility for known issues.

Supporting

RubyGems is managed by Ruby Central, a non-profit organization that supports the Ruby community through projects like this one, as well as RubyConf, RailsConf, and RubyGems.org. You can support Ruby Central by attending or sponsoring a conference, or by joining as a supporting member.

Contributing

If you'd like to contribute to RubyGems, that's awesome, and we <3 you. Check out our guide to contributing for more information.

Code of Conduct

Everyone interacting in the RubyGems project’s codebases, issue trackers, chat rooms, and mailing lists is expected to follow the contributor code of conduct.

gemstash's Issues

Logging

Add proper logging.

My thoughts for requirements:

• It should default the log to ~/.gemstash/server.log (or something like that)
• The log should have a limit by default, so we don't silently create a multi-hundred megabyte log... perhaps like 10MB or 50 or so by default?
• Utilize some existing logger so we can do debug/info/warn/error
• Look into what Puma logging does

Plugin for hosting documentation

Create a plugin which exposes endpoints for viewing Yard documentation, and possibly another plugin that integrates with Dash.

Run gemstash in a docker container

I put up a little repo that sets it up. Let me know if you want to take ownership or to make it a part of this project.

How does Gemstash compare to Geminabox?

Obviously Geminabox has been around for a while and Gemstash is relatively a new comer. It would be great to have a comparison on features, security, hosting setup etc. So the gem server shoppers can make a more informed decision on which one to pick.

NoMethodError - undefined method `<=' for nil:NilClass

I've been using Gemstash for several months, version 1.0.0. Today I redeployed Gemstash Docker container with version 1.0.2. After some time (1.5 h), I've got an error on bundle install:

Gem::RemoteFetcher::FetchError: bad response Internal Server Error 500 (http://my_rubygems_cache_public_host/gems/fog-local-0.3.0.gem)
An error occurred while installing fog-local (0.3.0), and Bundler cannot
continue.
Make sure that `gem install fog-local -v '0.3.0'` succeeds before bundling.

The logs say (and they are upside down):

[2016-09-24 19:45:24 +0000] - INFO - [7] 10.1.1.224 - - [24/Sep/2016:19:45:24 +0000] "GET /gems/fog-local-0.3.0.gem HTTP/1.1" 200 - 1.4742
/usr/local/bundle/gems/puma-2.16.0/lib/puma/thread_pool.rb:106:in `block in spawn_thread'
/usr/local/bundle/gems/puma-2.16.0/lib/puma/server.rb:270:in `block in run'
# ... shorthened
/usr/local/bundle/gems/gemstash-1.0.2/lib/gemstash/storage.rb:274:in `load_properties'
/usr/local/bundle/gems/gemstash-1.0.2/lib/gemstash/storage.rb:279:in `check_resource_version'
Did you mean? <=>:
[2016-09-24 19:45:24 +0000] - ERROR - 2016-09-24 19:45:24 - NoMethodError - undefined method `<=' for nil:NilClass
[2016-09-24 19:45:24 +0000] - INFO - [7] 172.17.0.1 - - [24/Sep/2016:19:45:24 +0000] "GET / HTTP/1.1" 302 - 0.0008
[2016-09-24 19:45:23 +0000] - INFO - Gem fog-local-0.3.0 is not cached, fetching gem
[2016-09-24 19:45:23 +0000] - INFO - [7] 10.1.1.224 - - [24/Sep/2016:19:45:23 +0000] "GET /gems/fog-google-0.1.0.gem HTTP/1.1" 200 - 1.7989
[2016-09-24 19:45:22 +0000] - INFO - [7] 10.1.1.224 - - [24/Sep/2016:19:45:22 +0000] "GET /gems/fog-google-0.1.0.gem HTTP/1.1" 200 - 1.6589
[2016-09-24 19:45:22 +0000] - INFO - [7] 10.1.1.224 - - [24/Sep/2016:19:45:22 +0000] "GET /gems/fog-google-0.1.0.gem HTTP/1.1" 200 - 1.6708
[2016-09-24 19:45:21 +0000] - INFO - Gem fog-google-0.1.0 is not cached, fetching gem

Don't you think there should be some check that version is not nil? https://github.com/bundler/gemstash/blob/master/lib/gemstash/storage.rb#L279

The build is broken, and we can finally see the logs

https://travis-ci.org/bundler/gemstash/jobs/129342999#L2067

Documentation and release sync

As mentioned on Slack..

Docs in master do not represent the currently available version.
At very least we can add a note to this effect and a link to releases?

As new features are added and documented a reference to their targeted version too?
eg "ERB support in YAML configuration file (0.2 and greater)" (although this means contributors need to have an idea of version numbers/releases or it's more work on release)

Faraday::ConnectionFailed

Howdy, and thanks for this great gem caching service!

Part of the reason I decided to try gemstash is because I'm on a slow T1 connection. So when I try to pull some larger gems I sometimes get:

ERROR - Connection failure - execution expired (Faraday::ConnectionFailed)

It would be great to be able to set an increased timeout value for Faraday. If I get a few mins. I'll drop in a PR here.

Thanks again!

Errors trying to bundle

I just installed gemstash and got the following errors trying to bundle:

$ bundle
Fetching gem metadata from http://localhost:9292/...........
Fetching version metadata from http://localhost:9292/..
Resolving dependencies...
Installing rake 10.5.0
Installing diff-lcs 1.2.5
Using thor 0.19.1
Installing rspec-support 3.4.1
Using bundler 1.11.2
Using nginxtra 1.8.0.11 from source at `.`

Gem::RemoteFetcher::FetchError: bad response Internal Server Error 500 (http://localhost:9292/gems/rspec-core-3.4.4.gem)

Gem::RemoteFetcher::FetchError: bad response Internal Server Error 500 (http://localhost:9292/gems/rspec-expectations-3.4.0.gem)

Gem::RemoteFetcher::FetchError: bad response Internal Server Error 500 (http://localhost:9292/gems/rspec-mocks-3.4.1.gem)
An error occurred while installing rspec-core (3.4.4), and Bundler cannot continue.
Make sure that `gem install rspec-core -v '3.4.4'` succeeds before bundling.

I looked into the log:

[2016-03-20 17:08:23 -0700] - INFO - Gem rspec-core-3.4.4 exists, returning cached gem
[2016-03-20 17:08:23 -0700] - INFO - 127.0.0.1 - - [20/Mar/2016:17:08:23 -0700] "GET /gems/rspec-core-3.4.4.gem HTTP/1.1" 200 145920 0.0060
[2016-03-20 17:08:23 -0700] - INFO - Gem rspec-expectations-3.4.0 exists, returning cached gem
[2016-03-20 17:08:23 -0700] - INFO - 127.0.0.1 - - [20/Mar/2016:17:08:23 -0700] "GET /gems/rspec-expectations-3.4.0.gem HTTP/1.1" 200 75264 0.0041
[2016-03-20 17:08:23 -0700] - INFO - Gem rspec-mocks-3.4.1 exists, returning cached gem
[2016-03-20 17:08:23 -0700] - INFO - 127.0.0.1 - - [20/Mar/2016:17:08:23 -0700] "GET /gems/rspec-mocks-3.4.1.gem HTTP/1.1" 200 74752 0.0030
[2016-03-20 17:08:24 -0700] - INFO - Gem rspec-3.4.0 is not cached, fetching gem
[2016-03-20 17:08:24 -0700] - ERROR - 2016-03-20 17:08:24 - Sequel::DatabaseError - SQLite3::SQLException: no such table: upstreams:
    ~/.rvm/gems/ruby-2.3.0/gems/sqlite3-1.3.11/lib/sqlite3/database.rb:91:in `initialize'
    ~/.rvm/gems/ruby-2.3.0/gems/sqlite3-1.3.11/lib/sqlite3/database.rb:91:in `new'
    ~/.rvm/gems/ruby-2.3.0/gems/sqlite3-1.3.11/lib/sqlite3/database.rb:91:in `prepare'
    ~/.rvm/gems/ruby-2.3.0/gems/sqlite3-1.3.11/lib/sqlite3/database.rb:266:in `query'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/adapters/sqlite.rb:187:in `block (2 levels) in _execute'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/database/logging.rb:35:in `log_yield'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/adapters/sqlite.rb:187:in `block in _execute'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/database/connecting.rb:251:in `block in synchronize'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/connection_pool/threaded.rb:101:in `hold'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/database/connecting.rb:251:in `synchronize'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/adapters/sqlite.rb:180:in `_execute'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/adapters/sqlite.rb:130:in `execute'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/dataset/actions.rb:952:in `execute'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/adapters/sqlite.rb:322:in `fetch_rows'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/dataset/actions.rb:833:in `with_sql_each'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/dataset/actions.rb:843:in `with_sql_first'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/dataset/placeholder_literalizer.rb:150:in `first'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/model/base.rb:878:in `block in def_finder_method'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/model/base.rb:112:in `[]'
    ~/.rvm/gems/ruby-2.3.0/gems/gemstash-1.0.1/lib/gemstash/db/upstream.rb:6:in `find_or_insert'
    ~/.rvm/gems/ruby-2.3.0/gems/gemstash-1.0.1/lib/gemstash/db/cached_rubygem.rb:9:in `block in store'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/database/transactions.rb:136:in `_transaction'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/database/transactions.rb:110:in `block in transaction'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/database/connecting.rb:251:in `block in synchronize'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/connection_pool/threaded.rb:105:in `hold'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/database/connecting.rb:251:in `synchronize'
    ~/.rvm/gems/ruby-2.3.0/gems/sequel-4.32.0/lib/sequel/database/transactions.rb:99:in `transaction'
    ~/.rvm/gems/ruby-2.3.0/gems/gemstash-1.0.1/lib/gemstash/db/cached_rubygem.rb:8:in `store'
    ~/.rvm/gems/ruby-2.3.0/gems/gemstash-1.0.1/lib/gemstash/gem_source/upstream_source.rb:178:in `block in fetch_remote_gem'
    ~/.rvm/gems/ruby-2.3.0/gems/gemstash-1.0.1/lib/gemstash/gem_fetcher.rb:16:in `block in fetch'
    ~/.rvm/gems/ruby-2.3.0/gems/gemstash-1.0.1/lib/gemstash/http_client.rb:56:in `get'
    ~/.rvm/gems/ruby-2.3.0/gems/gemstash-1.0.1/lib/gemstash/gem_fetcher.rb:13:in `fetch'
    ~/.rvm/gems/ruby-2.3.0/gems/gemstash-1.0.1/lib/gemstash/gem_source/upstream_source.rb:170:in `fetch_remote_gem'
    ~/.rvm/gems/ruby-2.3.0/gems/gemstash-1.0.1/lib/gemstash/gem_source/upstream_source.rb:159:in `fetch_gem'
    ~/.rvm/gems/ruby-2.3.0/gems/gemstash-1.0.1/lib/gemstash/gem_source/upstream_source.rb:122:in `serve_cached'
    ~/.rvm/gems/ruby-2.3.0/gems/gemstash-1.0.1/lib/gemstash/gem_source/upstream_source.rb:116:in `serve_gem'
    ~/.rvm/gems/ruby-2.3.0/gems/gemstash-1.0.1/lib/gemstash/web.rb:82:in `block in <class:Web>'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `call'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `block in compile!'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (3 levels) in route!'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:994:in `route_eval'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (2 levels) in route!'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1015:in `block in process_route'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in `catch'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in `process_route'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:973:in `block in route!'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in `each'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in `route!'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1085:in `block in dispatch!'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `catch'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1082:in `dispatch!'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `block in call!'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `catch'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `call!'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:895:in `call'
    ~/.rvm/gems/ruby-2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
    ~/.rvm/gems/ruby-2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
    ~/.rvm/gems/ruby-2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
    ~/.rvm/gems/ruby-2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
    ~/.rvm/gems/ruby-2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
    ~/.rvm/gems/ruby-2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
    ~/.rvm/gems/ruby-2.3.0/gems/rack-1.6.4/lib/rack/nulllogger.rb:9:in `call'
    ~/.rvm/gems/ruby-2.3.0/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/show_exceptions.rb:25:in `call'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:182:in `call'
    ~/.rvm/gems/ruby-2.3.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:2013:in `call'
    ~/.rvm/gems/ruby-2.3.0/gems/gemstash-1.0.1/lib/gemstash/gem_source/rack_middleware.rb:18:in `call'
    ~/.rvm/gems/ruby-2.3.0/gems/gemstash-1.0.1/lib/gemstash/env.rb:33:in `call'
    ~/.rvm/gems/ruby-2.3.0/gems/puma-2.16.0/lib/puma/commonlogger.rb:31:in `call'
    ~/.rvm/gems/ruby-2.3.0/gems/gemstash-1.0.1/lib/gemstash/logging.rb:56:in `call'
    ~/.rvm/gems/ruby-2.3.0/gems/rack-1.6.4/lib/rack/deflater.rb:35:in `call'
    ~/.rvm/gems/ruby-2.3.0/gems/puma-2.16.0/lib/puma/configuration.rb:81:in `call'
    ~/.rvm/gems/ruby-2.3.0/gems/puma-2.16.0/lib/puma/server.rb:557:in `handle_request'
    ~/.rvm/gems/ruby-2.3.0/gems/puma-2.16.0/lib/puma/server.rb:404:in `process_client'
    ~/.rvm/gems/ruby-2.3.0/gems/puma-2.16.0/lib/puma/server.rb:270:in `block in run'
    ~/.rvm/gems/ruby-2.3.0/gems/puma-2.16.0/lib/puma/thread_pool.rb:106:in `block in spawn_thread'

Here's my installed gems:

$ gem list

*** LOCAL GEMS ***

bigdecimal (1.2.8)
bundler (1.11.2)
bundler-unload (1.0.2)
dalli (2.7.6)
did_you_mean (1.0.0)
diff-lcs (1.2.5)
executable-hooks (1.3.2)
faraday (0.9.2)
faraday_middleware (0.10.0)
gem-wrappers (1.2.7)
gemstash (1.0.1)
io-console (0.4.5)
json (1.8.3)
lru_redux (1.1.0)
minitest (5.8.3)
multipart-post (2.0.0)
net-telnet (0.1.1)
power_assert (0.2.6)
psych (2.0.17)
puma (2.16.0)
rack (1.6.4)
rack-protection (1.5.3)
rake (10.5.0, 10.4.2)
rdoc (4.2.1)
rspec (3.4.0)
rspec-core (3.4.4)
rspec-expectations (3.4.0)
rspec-mocks (3.4.1)
rspec-support (3.4.1)
rubygems-bundler (1.4.4)
rvm (1.11.3.9)
sequel (4.32.0)
sinatra (1.4.7)
sqlite3 (1.3.11)
test-unit (3.1.5)
thor (0.19.1)
tilt (2.0.2)

Here is the gemfile.lock I was bundling with:

PATH
  remote: .
  specs:
    nginxtra (1.8.0.11)
      thor (~> 0.16)

GEM
  remote: https://rubygems.org/
  specs:
    diff-lcs (1.2.5)
    rake (10.5.0)
    rspec (3.4.0)
      rspec-core (~> 3.4.0)
      rspec-expectations (~> 3.4.0)
      rspec-mocks (~> 3.4.0)
    rspec-core (3.4.4)
      rspec-support (~> 3.4.0)
    rspec-expectations (3.4.0)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.4.0)
    rspec-mocks (3.4.1)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.4.0)
    rspec-support (3.4.1)
    thor (0.19.1)

PLATFORMS
  ruby

DEPENDENCIES
  nginxtra!
  rake (~> 10.0)
  rspec (~> 3.3)

BUNDLED WITH
   1.11.2

My first inclination is our loose dependencies might be biting us again, but I'm not really sure yet.

Add `gemstash status` command

It would be nice to be able to check if the server is running the same way you can start and stop it. :)

Command line config

This has been in the back of my head for a while, and has been referenced in #28, so I figured I would put my thoughts here.

Right now, all config is handled via a config file, but it would be neat to support all configuration directly as config options as well.

@indirect suggested -h/--host, -p/--port and -b/--bind, which we could expand to all the config options.

Another route I've thought about was along the lines of ssh config. For example, you can run ssh user@host -o Port=22 or ssh user@host -o "Port 22". You can also use all those config in your ssh config file, just like we would support in Gemstash.

Thus, it might look like this:

$ gemstash start --config bind=tcp://0.0.0.0:4242 --config rubygems_url=https://gemfury.io
$ gemstash start -c bind=tcp://0.0.0.0:4242 -c rubygems_url=https://gemfury.io

Would these command line options override any config found in the configuration file, even if a specific --config-file is provided? I would think yes, the command line should override.

Thoughts?

Are *.gem files cached for 30 minutes or indefinitely?

The gem dependencies from https://rubygems.org are cached for 30 minutes, so if you bundle again before that, you can successfully bundle without an Internet connection:

It's not clear to me what exactly happens.

1. Gem dependencies (what depends on what - the metadata) is cached for 30 minutes. Cached gems (*.gem files) are cached indefinitely.
2. Gem dependencies (that is, gems, as a dependency of a gem is another gem) is cached for 30 minutes. *.gem files disappear after 30 minutes.

Can you clarify here or, better yet, in the documentation itself?

---

Gemstash has saved my life. Fastly seems to have a problem recently at DigitalOcean/Gitlab.com CI runners and Gemstash helped the situation. Thanks for your great work.

Support running without memcached

Just for the sake of leanness. So you can pull and start for a quick test.

Enumerate version maintenance and version bumps

I think we should spell out in the documentation:

1. What we mean when we bump a major version
2. What we mean when we bump a minor version
3. How far back we will provide patch releases for bugs (maybe something like the last 2 minor versions of the last 2 major versions?)

I would like to follow semantic versioning, though to do so we should figure out what our API is, so we know when we are making a backwards incompatible change.

If we do so, then the biggest thing on my mind... is the new index format a backwards incompatible change or not? I think we could reasonably say that any HTTP requests happening behind the scenes are not really our API, and so I would consider it a minor update. Yet, it is still potentially a big change that introduces risk, and that might not be conveyed properly in a minor version update, so maybe that doesn't matter.

/cc @pcarranza @indirect

Bundle with full index against private gems

We should be able to bundle with --full-index against private gems hosted in Gemstash.

Private gems in the database?

Just an idea I've done in previous Rails projects that I've found useful.

If we store the private gem files in the database as blobs, we get a few free benefits:

• The Gemstash server could be served as a cluster by just pointing to the same database
• Backing up or moving the Gemstash server is as simple as backing up or moving the database
• If you don't care about backing up or moving cached upstream gems, backing up or moving becomes only dealing with the database
• Storing of the private gems and necessary info along with the cached dependency information can be in a single transaction that will either succeed or fail atomically

Some possible downfalls:

• Potentially a bit more complex or startling to understand if you are new to the code
• A user might be confused trying to find the gems and not seeing them in the storage (could be mitigated by also caching the files to disk)
• You might get less throughput (could be mitigated with caching in memory/disk)
• The blob size would limit the gem size (blobs could be specified to be pretty big though, so this is probably not be a real problem)

If you all feel this might be worthwhile to implement, I'd be happy to implement it in a branch so you can see the results.

Maybe clustering a Gemstash server is not a useful use case, or maybe backing up or moving the directory is just as easy as with the database (though I think there being 1 less step has value).

We can discuss here before I do anything. I think if we are going with this plan, though, we should probably do it before the initial release, to avoid needing to migrate the files into the database.

Do not have users require cgi

It seems strange to require users to require the cgi module in their gem file:

require "cgi"
source "http://localhost:9042/upstream/#{CGI.escape("http://127.0.0.1:9043")}"

Instead provide a better way with gemstash:

require "gemstash/bundler"
gemstash_source "http://localhost:9042/upstream", "http://127.0.0.1:9043"

Or perhaps:

require "gemstash/bundler"
gemstash_url "http://localhost:9042"
source gemstash_upstream("http://127.0.0.1:9043")

Implement the new index format

Implement the new index format using both the client and server gems used by Bundler and the Bundler API.

Yank a private gem

Yank private gems.

Since rubygems doesn't support yanking a host other than www.rubygems.org, we should add a temporary Thor task to make it easy.

The task should be clearly marked as temporary.

Here are my thoughts for behavior, please comment if it should differ:

• Yanked gem files will remain, in case of unyank
• Fetching gem version will result in 404
• Mark in database as yanked, and don't return in dependencies API

Explore not updating index unless explicitly calling `bundle update`

Explore the possibility of not updating the index unless bundle update is called (or perhaps if the index is over a certain age, like a month or 2 months or whatever). This means, if you create a bunch of rails apps (or otherwise reuse a common set of gems frequently), you won't refresh the index unless you explicitly want to bundle update, which saves you in bundling time.

I ran a test against rails 4.2.5 on a local gemstash server will the necessary gems cached versus bundling against rubygems.org directly. 54.471s vs 1m41.089s, then I tried pre-installing nokogiri (since the native building was taking more time than any other gem), and it resulted in 11.160s vs 32.879s.

This might be difficult or impossible to implement without causing potential failures when bundling... for example, if you add a requirement for a gem not yet fetched, then this might cause bundling to fail when it shouldn't, so this needs to be carefully thought out.

Bundle with full index against upstreams

We should be able to bundle with --full-index against upstreams (including the default server)

Add a page for design decisions and reasoning

There should be a page to describe various design decisions with a short background of what the decision was aiming for. Please add to this issue any decisions or questions that we can include.

Upstreams go to separate folders

Ensure different upstreams cache gems in segregated folders.

This is probably the final piece to complete #2

Do we have a plan to support S3 being the gem store?

Maybe having gems stored in S3 as the source of truth and cache locally for performance benefit?

Smarter fetcher and support resuming downloads when caching

So far it just pulls the remote gem and stores the headers.

We should remove headers that are S3 specific, and some other things that are there, and just return whatever actually provides value to the caller (etag, content-type, content-length, date, last-modified, etc)

Also we should support resuming if the size of the gem does not match with the content-length, to prevent half downloaded gems when downloading with unstable connections.

Send specific user agent to track usage

https://github.com/bundler/bundler/blob/master/lib/bundler/fetcher.rb#L140-L169

UI for interacting with private and cached gems

@chriseckhardt suggested we add a UI showing a list of the private gems like Stickler does.

It would be neat if we showed what is cached as well.

This might be best as a plugin.

Final 1.0 release tasks

• Remove the --pre added from #54 after 1.0.0 is released
• Changelog?
• Blog post with the changes in the DB model? (probably before releasing 1.0)

Please add to this list anything more we need.

private gems not found unless `gem search` has --source flag

My setup:
Late 2013 Macbook Pro 15-inch retina OS X 10.11.3
Ruby 2.3.0p0 (via RVM 1.26.11)
gem version 2.5.1

When I spin up a gemstash instance locally and push private gems, I am unable to find them in a gem search without appending the --source http://127.0.0.1/private to my command line invocation, regardless if I've set the gemstash as my gem source or not.

Steps to reproduce:

$ gem install gemstash
$ gemstash authorize # add key to ~/.gem/credentials
$ gemstash start
$ gem push --key gemstash --host http://127.0.0.1:9292/private my_private_gem-1.2.3.gem
$ gem source -r https://rubygems.org/
$ gem source -a http://127.0.0.1:9292/private
$ gem search -ar my_private_gem

*** REMOTE GEMS ***

$ gem search -ar my_private_gem --source http://127.0.0.1:9292/private

*** REMOTE GEMS ***
my_private_gem (1.2.3)

/versions and /info/* are redirected to rubygems.org

Gemstash is currently redirecting requests for /versions to https://rubygems.org/versions instead of the correct https://index.rubygems.org/versions. The endpoint for /info/* should also be updated to redirect to https://index.rubygems.org/info/*.

bundle update --verbose
HTTP GET https://internal.gemstash.server/versions
HTTP 302 Found
HTTP GET https://rubygems.org/versions
HTTP 200 OK
HTTP GET https://internal.gemstash.server/versions
HTTP 302 Found
HTTP GET https://rubygems.org/versions
HTTP 200 OK
The checksum of /versions does not match the checksum provided by the server! Something is wrong (local checksum is "\"f3e85d51952e0136d9a61006a82ab686\"", was expecting "W/\"f3e85d51952e0136d9a61006a82ab686\"").
HTTP GET https://internal.gemstash.server/api/v1/dependencies
HTTP 200 OK

This also has the side effect of causing a checksum mismatch due to rubygems.org serving weak ETags for the /versions endpoint. See rubygems/bundler#4764 for reference.

Make it easier to release java and ruby version of gem

Push a private gem

Allow pushing a private gem to a Gemstash server.

Feel free to ransack stickler

Many times I meant to add functionality to stickler to be a pass through cache for rubygems (copiousfreetime/stickler#7).

I'd just like to say, you are welcome to any and all of the code in stickler to help with this project. Stickler itself is a sinatra app that provides all the gem server functionality as a series of rack middlewares. You are welcome to take any and all code you want. If you you decide to use any of the code, I'm happy to re-license it as MIT (currently ISC).

If there is no interest, no worries. Just thought I would make the offer.

Deployment instructions

How do I get this up and running as a server? How do I set it up to act as a local cache for Rubygems.org? How do I get it running as a local private gem server? Where is the database stored? Where are gems stored? How can I upgrade a running server? How can I transfer a running server to another machine?

Import command

For users that have gem servers with a ton of private gems, it might be nice to import the gems as private gems in Gemstash.

Perhaps something like gemstash import <url>? Maybe an option to only import specific gems and/or versions?

It would also be possible to simply have the existing server be an upstream in Gemstash, but then again it might be nice to get rid of the other server and only maintain 1.

Task to actively pre-cache all of rubygems.org

If you're going to, say, railscamp, or mainland china, you might want to grab a copy of all the gems. Or at least the newest version of each gem. A task to make this a single command would be swell.

Add support for mysql as DB backend

Just that.

Plugins

I would like to expose a plugin mechanism once the basic features are done. This issue can capture ideas and considerations for implementing the plugins.

Some possible extension points for a Gemstash plugin:

• Hook in as a rack middleware
• Migrations that are run after Gemstash migrations are run
• Additional routes for sinatra handled by the plugin
• Deployed as a separate gem
• Maybe add the plugin via something like gemstash plugin <xyz>
• Alternatively, it could be a hidden feature supported by tweaking a config (like in Gemstash::Configuration, a plugins array listing the plugins to load)

Protected fetch

We have push behind an API key, as will be yank and unyank.

Should fetching private gems support the key as well?

Will bundler send API key from rubygems config?

Should we add that to bundler if not?

Maybe we send a 401 and bundler can look for API key from rubygems config? Maybe specify API key name in config and it will use it automatically?

Ugly trace vomited to bundler when upstream connection fails while resolving dependencies

Nothing to worry about, seems to be an edge case, but it outputted a lot of data that didn't make any sense.

Probably we need to guard ourselves from failing connections.

[2015-10-13 15:36:10 -0400] - INFO - Fetching dependencies: netrc, mime-types, http-cookie, rdoc, simplecov-html, docile, lockfile, power_assert, ast, slop, sexp_processor, little-plugger, loquacious, bones-git, bones-extras, abstract, metaclass, needle, jruby-pageant, echoe, arel, activemodel, tzinfo, activerecord-deprecated_finders, thread_safe, minitest
[2015-10-13 15:36:15 -0400] - ERROR - 2015-10-13 15:36:15 - Faraday::ConnectionFailed - execution expired:
    /home/pcarranza/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http.rb:879:in `initialize'
    /home/pcarranza/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http.rb:879:in `open'
    /home/pcarranza/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http.rb:879:in `block in connect'
    /home/pcarranza/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/timeout.rb:89:in `block in timeout'
    /home/pcarranza/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/timeout.rb:99:in `call'
    /home/pcarranza/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/timeout.rb:99:in `timeout'
    /home/pcarranza/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http.rb:878:in `connect'
    /home/pcarranza/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http.rb:863:in `do_start'
    /home/pcarranza/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http.rb:852:in `start'
    /home/pcarranza/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http.rb:1375:in `request'
    /home/pcarranza/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http.rb:1133:in `get'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:80:in `perform_request'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:40:in `block in call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:87:in `with_net_http_connection'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:32:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/faraday_middleware-0.10.0/lib/faraday_middleware/response/follow_redirects.rb:76:in `perform_with_redirection'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/faraday_middleware-0.10.0/lib/faraday_middleware/response/follow_redirects.rb:64:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/faraday-0.9.1/lib/faraday/rack_builder.rb:139:in `build_response'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/faraday-0.9.1/lib/faraday/connection.rb:377:in `run_request'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/faraday-0.9.1/lib/faraday/connection.rb:140:in `get'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/gemstash-0.1.0/lib/gemstash/http_client.rb:37:in `get'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/gemstash-0.1.0/lib/gemstash/dependencies.rb:77:in `fetch_from_web'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/gemstash-0.1.0/lib/gemstash/dependencies.rb:42:in `fetch'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/gemstash-0.1.0/lib/gemstash/dependencies.rb:23:in `fetch'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/gemstash-0.1.0/lib/gemstash/gem_source/dependency_caching.rb:15:in `serve_dependencies'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/gemstash-0.1.0/lib/gemstash/web.rb:34:in `block in <class:Web>'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:1610:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:1610:in `block in compile!'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:974:in `[]'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:974:in `block (3 levels) in route!'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:993:in `route_eval'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:974:in `block (2 levels) in route!'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:1014:in `block in process_route'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:1012:in `catch'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:1012:in `process_route'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:972:in `block in route!'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:971:in `each'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:971:in `route!'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:1084:in `block in dispatch!'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `block in invoke'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `catch'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `invoke'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:1081:in `dispatch!'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:906:in `block in call!'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `block in invoke'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `catch'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:1066:in `invoke'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:906:in `call!'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:894:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/rack-1.4.5/lib/rack/nulllogger.rb:9:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/rack-1.4.5/lib/rack/head.rb:9:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/show_exceptions.rb:21:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:181:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:2021:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/gemstash-0.1.0/lib/gemstash/gem_source/rack_middleware.rb:18:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/gemstash-0.1.0/lib/gemstash/env.rb:32:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/puma-2.14.0/lib/puma/commonlogger.rb:31:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/gemstash-0.1.0/lib/gemstash/logging.rb:42:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/rack-1.4.5/lib/rack/deflater.rb:13:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/puma-2.14.0/lib/puma/configuration.rb:78:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/puma-2.14.0/lib/puma/server.rb:541:in `handle_request'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/puma-2.14.0/lib/puma/server.rb:388:in `process_client'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/puma-2.14.0/lib/puma/server.rb:270:in `block in run'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/puma-2.14.0/lib/puma/thread_pool.rb:106:in `call'
    /home/pcarranza/.rvm/gems/ruby-2.2.1/gems/puma-2.14.0/lib/puma/thread_pool.rb:106:in `block in spawn_thread'

Authorization

Support a key based authorization.

Some ideas to work through:

• Should it use the same rubygems.org key?
• If not, should there be a command like gemstash push gem, which uses its own key?
• Should we direct user to specify a separate key manually?
• Maybe gemstash push gem is an alias for gem push, but uses a separate key file and passes it directly to gem command?
• Should key be protected on server?
• How to authorize on the server? gemstash authorize key?
• gem yank doesn't seem to work with a remote host... maybe support gemstash yank gem -v version?

Gemstash command outputs

Currently, gemstash start doesn't return any output, and gemstash stop returns Command stop sent success.

I think it's better to return outputs like "Gemstash started/stopped successfully!". If you're 👍 on this, I can work on it.

syntax error, unexpected tLABEL

Hi, I'm installing gemstash on a fresh ubuntu 14.04 vagrant box as per the README.md, and getting a syntax error. I'm not particularly familiar with ruby, so would be grateful for any pointers.

Thanks,

Joe

vagrant@gemstash-01:~$ sudo apt-get install ruby ruby-dev libsqlite3-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
ruby is already the newest version.
ruby-dev is already the newest version.
Suggested packages:
  sqlite3-doc
The following NEW packages will be installed:
  libsqlite3-dev
0 upgraded, 1 newly installed, 0 to remove and 10 not upgraded.
Need to get 439 kB of archives.
After this operation, 1,491 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://uk.archive.ubuntu.com/ubuntu/ trusty-updates/main libsqlite3-dev amd64 3.8.2-1ubuntu2.1 [439 kB]
Fetched 439 kB in 0s (2,137 kB/s)
Selecting previously unselected package libsqlite3-dev:amd64.
(Reading database ... 73881 files and directories currently installed.)
Preparing to unpack .../libsqlite3-dev_3.8.2-1ubuntu2.1_amd64.deb ...
Unpacking libsqlite3-dev:amd64 (3.8.2-1ubuntu2.1) ...
Setting up libsqlite3-dev:amd64 (3.8.2-1ubuntu2.1) ...
vagrant@gemstash-01:~$ sudo gem install gemstash
Fetching: dalli-2.7.6.gem (100%)
Fetching: lru_redux-1.1.0.gem (100%)
Fetching: puma-2.16.0.gem (100%)
Building native extensions.  This could take a while...
Fetching: sequel-4.38.0.gem (100%)
Fetching: rack-1.6.4.gem (100%)
Fetching: tilt-2.0.5.gem (100%)
Fetching: rack-protection-1.5.3.gem (100%)
Fetching: sinatra-1.4.7.gem (100%)
Fetching: thor-0.19.1.gem (100%)
Fetching: multipart-post-2.0.0.gem (100%)
Fetching: faraday-0.9.2.gem (100%)
Fetching: faraday_middleware-0.10.0.gem (100%)
Fetching: sqlite3-1.3.11.gem (100%)
Building native extensions.  This could take a while...
Fetching: gemstash-1.0.2.gem (100%)
Successfully installed dalli-2.7.6
Successfully installed lru_redux-1.1.0
Successfully installed puma-2.16.0
Successfully installed sequel-4.38.0
Successfully installed rack-1.6.4
Successfully installed tilt-2.0.5
Successfully installed rack-protection-1.5.3
Successfully installed sinatra-1.4.7
Successfully installed thor-0.19.1
Successfully installed multipart-post-2.0.0
Successfully installed faraday-0.9.2
Successfully installed faraday_middleware-0.10.0
Successfully installed sqlite3-1.3.11
Successfully installed gemstash-1.0.2
14 gems installed
Installing ri documentation for dalli-2.7.6...
Installing ri documentation for lru_redux-1.1.0...
Installing ri documentation for puma-2.16.0...
Installing ri documentation for sequel-4.38.0...
Installing ri documentation for rack-1.6.4...
Installing ri documentation for tilt-2.0.5...
Installing ri documentation for rack-protection-1.5.3...
Installing ri documentation for sinatra-1.4.7...
Installing ri documentation for thor-0.19.1...
Installing ri documentation for multipart-post-2.0.0...
Installing ri documentation for faraday-0.9.2...
Installing ri documentation for faraday_middleware-0.10.0...
Installing ri documentation for sqlite3-1.3.11...
Installing ri documentation for gemstash-1.0.2...
Installing RDoc documentation for dalli-2.7.6...
Installing RDoc documentation for lru_redux-1.1.0...
Installing RDoc documentation for puma-2.16.0...
Installing RDoc documentation for sequel-4.38.0...
Installing RDoc documentation for rack-1.6.4...
Installing RDoc documentation for tilt-2.0.5...
Installing RDoc documentation for rack-protection-1.5.3...
Installing RDoc documentation for sinatra-1.4.7...
Installing RDoc documentation for thor-0.19.1...
Installing RDoc documentation for multipart-post-2.0.0...
Installing RDoc documentation for faraday-0.9.2...
Installing RDoc documentation for faraday_middleware-0.10.0...
Installing RDoc documentation for sqlite3-1.3.11...
Installing RDoc documentation for gemstash-1.0.2...
vagrant@gemstash-01:~$ gemstash start
/var/lib/gems/1.9.1/gems/gemstash-1.0.2/lib/gemstash/cli/base.rb:7:in `<class:Base>': /var/lib/gems/1.9.1/gems/gemstash-1.0.2/lib/gemstash/env.rb:37: syntax error, unexpected tLABEL (SyntaxError)
    def initialize(config = nil, cache: nil, db: nil)
                                       ^
/var/lib/gems/1.9.1/gems/gemstash-1.0.2/lib/gemstash/env.rb:37: Can't assign to nil
    def initialize(config = nil, cache: nil, db: nil)
                                            ^
/var/lib/gems/1.9.1/gems/gemstash-1.0.2/lib/gemstash/env.rb:150: syntax error, unexpected keyword_end, expecting $end
        from /var/lib/gems/1.9.1/gems/gemstash-1.0.2/lib/gemstash/cli/base.rb:6:in `<class:CLI>'
        from /var/lib/gems/1.9.1/gems/gemstash-1.0.2/lib/gemstash/cli/base.rb:4:in `<module:Gemstash>'
        from /var/lib/gems/1.9.1/gems/gemstash-1.0.2/lib/gemstash/cli/base.rb:3:in `<top (required)>'
        from /var/lib/gems/1.9.1/gems/gemstash-1.0.2/lib/gemstash/cli/start.rb:8:in `<class:CLI>'
        from /var/lib/gems/1.9.1/gems/gemstash-1.0.2/lib/gemstash/cli/start.rb:5:in `<module:Gemstash>'
        from /var/lib/gems/1.9.1/gems/gemstash-1.0.2/lib/gemstash/cli/start.rb:4:in `<top (required)>'
        from /var/lib/gems/1.9.1/gems/gemstash-1.0.2/lib/gemstash/cli.rb:54:in `start'
        from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
        from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
        from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
        from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
        from /var/lib/gems/1.9.1/gems/gemstash-1.0.2/exe/gemstash:3:in `<top (required)>'
        from /usr/local/bin/gemstash:23:in `load'
        from /usr/local/bin/gemstash:23:in `<main>'

JRuby build is slow

The JRuby build is massively slower than MRI. I suspect this is due to JRuby warm up time happening in all the spawns in the integration specs. I've heard there are gems which can pre-warm the JVM... these may drastically speed up the build.

Errors are hidden in daemonized mode

If there is an error on startup, it is sometimes hidden when running in daemonized mode, while it will be visible running in non-daemonized mode.

Example: write a plugin against the plugins branch, have the initialization throw an error, then run in both daemonized mode and non-daemonized mode.

DB Migration policy?

I added a migration that altered an existing table originally created in another migration. The migration would be simplier if it was just folded into the original one, but it would require anyone with a Gemstash server to delete their database, migrate it manually, or otherwise deal with it. I think that is likely just a few test Gemstash instances right now amongst us, but I don't know for sure.

This brings up a question about DB migration policy... should we support someone cloning the repository against master and pulling the codebase and running from there? Or do we require they strictly go through the gem? I'm personally in favor of requiring they go through the gem, otherwise we would have to be super careful about any merges into master. This would allow me to merge the 2 migrations (maybe even all migrations into 1 migration?).

Thoughts?

Should I merge the 2 migrations? All 3 migrations? Leave as is and treat each migration going forward as if there is a production database in the wild?

Multiple gem servers

We use multiple gem sources in a single gemfile. It would be nice to point to gemstash and it will pull from all of them.

Raised exceptions should include an appropriate HTTP status code and respond better than a 500 error

Gemstash::GemPusher::ExistingVersionError raises and triggers a raw 500 Sinatra error. It should instead have a more appropriate HTTP status code with a nicer response. This goes for all Gemstash exceptions that get raised.

Make sure writes are thread safe

We are using Puma in a threaded environment. SQLite gest unhappy if you try to commit from 2 threads. Do we need a monitor around write uses, or does Sequel have a way of handling this natively? Do we need to reduce the DB connection pool to 1 when using SQLite?

Usage instructions

If someone has followed the deployment instructions (#19) and there is a server running, how do I use it? What if I want all of the app servers in my data center to use it? What if I want all of the pairing machines in the office to use it? What if I'm a developer with a laptop who is only sometimes at the office where gemstash is running? What if my Gemfile has multiple sources? What if I'm using private gems hosted on gemstash?

How do I push a private gem? How do I yank a private gem? How do I authenticate to the private gem server? How are my private gems protected from randos getting them if they see the source line in my Gemfile?