ember-cli-deploy / ember-cli-deploy-redis Goto Github PK

View Code? Open in Web Editor NEW

18.0 18.0 41.0 2.22 MB

An ember-cli-deploy-plugin to upload index.html to a Redis store

License: MIT License

JavaScript 97.86% HTML 2.14%

ember-cli-deploy-plugin

ember-cli-deploy-redis's Introduction

Ember CLI Deploy

Simple, flexible deployment for your Ember CLI app

Installation

ember install ember-cli-deploy

Quick start

After installation, choose plugins matching your deployment environment, configure your deployment script appropriately and you're ready to start deploying.

In-depth documentation

Visit the Docs site

Contributing

Clone the repo and run npm install. To run tests,

npm test

ember-cli-deploy-redis's People

Contributors

Stargazers

Watchers

ember-cli-deploy-redis's Issues

Sample nginx conf

How about sample nginx conf and some tutorial on how to set up it?

TypeError: this.readConfig is not a function

TypeError: this.readConfig is not a function is getting thrown when deploying.

TypeError: this.readConfig is not a function
TypeError: this.readConfig is not a function
    at Object.DeployPluginBase.extend.defaultConfig.redisDeployClient (/Users/andres/Developer/take2mfg/take2-shop/node_modules/ember-cli-deploy-lightning-pack/node_modules/ember-cli-deploy-redis/index.js:55:36)
    at CoreObject.extend.readConfig (/Users/andres/Developer/take2mfg/take2-shop/node_modules/ember-cli-deploy-lightning-pack/node_modules/ember-cli-deploy-redis/node_modules/ember-cli-deploy-plugin/index.js:59:30)
    at CoreObject.DeployPluginBase.extend.fetchInitialRevisions (/Users/andres/Developer/take2mfg/take2-shop/node_modules/ember-cli-deploy-lightning-pack/node_modules/ember-cli-deploy-redis/index.js:137:38)
    at Object._pipeline.register.fn (/Users/andres/Developer/take2mfg/take2-shop/node_modules/ember-cli-deploy/lib/tasks/pipeline.js:93:21)
    at Pipeline._notifyPipelinePluginHookExecution (/Users/andres/Developer/take2mfg/take2-shop/node_modules/ember-cli-deploy/lib/models/pipeline.js:175:19)
    at tryCatch (/Users/andres/Developer/take2mfg/take2-shop/node_modules/ember-cli/node_modules/rsvp/dist/rsvp.js:538:12)
    at invokeCallback (/Users/andres/Developer/take2mfg/take2-shop/node_modules/ember-cli/node_modules/rsvp/dist/rsvp.js:553:13)
    at /Users/andres/Developer/take2mfg/take2-shop/node_modules/ember-cli/node_modules/rsvp/dist/rsvp.js:628:16
    at flush (/Users/andres/Developer/take2mfg/take2-shop/node_modules/ember-cli/node_modules/rsvp/dist/rsvp.js:2373:5)

Appears to be related to this change ember-cli-deploy/ember-cli-deploy-plugin#15

Simple redis url does not work

Running a js app inside a container,

In deploy.js I have the following simple config

  ENV.redis = {
    url: process.env['REDIS_URL']
  }

The url in the env is REDIS_URL=redis://redis:6379/0

Apparently it will fail to parse as it will still try to connect to localhost (the default host).

To work around it I need to do the parsing myself:

  const { URL } = require('url');
  const redisURL = new URL(process.env['REDIS_URL'])

  ENV.redis = {
    host: redisURL.hostname,
    port: redisURL.port,
    username: redisURL.username,
    password: redisURL.password
  };

It will be ideal to have the parsing done by this library.

From the description of the plugin, This plugin uploads a file, presumably index.html, to a specified Redis store., it seems that uploading multiple files is not supported. In my use-case, besides an index.html, I would also like to upload a assets/print.css to Redis. Will such a use-case ever be supported by this plugin?

Thank you,

Yoran

Difficult Error When URL Missing "redis://"

I'm fairly new to the project, and didn't realize I need to prefix a URL option with "redis://". I know, it's a bonehead move... Any ways... The error you get is:

cleaning up...
Deploying [==>------] 33% [plugin: redis -> fetchInitialRevisions]events.js:183
      throw er; // Unhandled 'error' event
      ^

Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED 127.0.0.1:6379
    at Object._errnoException (util.js:1022:11)
    at _exceptionWithHostPort (util.js:1044:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1182:14)

I think we can do a bit better than this. Perhaps checking for this prefix is advisable, or least offering a suggestion to look for this when the failure happens? I'd be more than happy to help if such a solution is desirable.

How to use with sentinel?

I've been attempting to pass my own redisClient, so I could use sentinel-aware ioredis. It looks like this won't work, and the injection is only for testing, since redisClient needs decorations provided by this repo (upload, activate, etc) and it doesn't seem possible to use an alternate redis lib (since it relies on a particular method signature). Am I understanding this correctly? Anything I can do here?

Value already exists for key: app-name:index:default

- Value already exists for key: app-name:index:default
Value already exists for key: app-name:index:default
undefinedPipeline aborted

It's my first deploy of the app

"ember-cli-deploy": "0.5.1",
"ember-cli-deploy-build": "0.1.0",
"ember-cli-deploy-manifest": "0.1.0",
"ember-cli-deploy-redis": "0.1.0",
"ember-cli-deploy-s3": "0.1.0",

Key Prefix not Stored in Current Key

After upgrading from an older version of ember-deploy-redis I found that ember-cli-deploy-redis does not store the key prefix in the current key.

For example, assuming your project name is project-awesome, the legacy version would store the following in the project-awesome:current key:

project-awesome:abcdefghijklmnopqrstuvwxyz1234

The current version of ember-cli-deploy-redis stores the following in the project-awesome:current key:

abcdefghijklmnopqrstuvwxyz1234

Is this the intended behavior?

It's not a huge deal, but I did have to implement a workaround in my rails app for this, since some of my ember apps still use the legacy redis deploy and have yet to be updated.

Instead of this:

index_key = $redis.get('project-awesome:current')
content = $redis.get(index_key)

in order to maintain compatibility I need this:

index_key = fix_redis_key('project-awesome', $redis.get('project-awesome:current'))
content = $redis.get(index_key)

private
def fix_redis_key(project_name, key)
  return key if key.start_with?(project_name)
  "#{project_name}:#{key}"
end

Don't use prefix as key name for revisions list

It's confusing that the plugin uses the prefix as the key name for the revisions list. One expects a "prefix" to be before something. Let's change it to prefix + ":revisions"

Inconsistency with keyPrefix

My Redis config contains the following:

redis: {
    allowOverwrite: true,
    keyPrefix: 'zatresi',
    maxRecentUploads: 100
},

When I run ember deploy prod I get the following output:

- Deployed but did not activate revision eb65b18c84b6b6b373fa5315e6661a9b. To activate, run: ember deploy:activate prod --revision=eb65b18c84b6b6b373fa5315e6661a9b

In Redis, the key zatresi:eb65b18c84b6b6b373fa5315e6661a9b indeed contains the HTML content of the index.html file.

Next, I run:

ember deploy:activate prod --revision=eb65b18c84b6b6b373fa5315e6661a9b

This sets the value of they key zatresi:current to eb65b18c84b6b6b373fa5315e6661a9b instead of zatresi:eb65b18c84b6b6b373fa5315e6661a9b. I believe this is a regression from a much earlier version of the addon. At some point, the behavior changed.

The documentation says:

The unique revision number for the version of the file being uploaded to Redis. The Redis key will be a combination of the keyPrefix and the revisionKey. By default this option will use either the revisionKey passed in from the command line or the revisionData.revisionKey property from the deployment context.

Default: context.commandLineArgs.revisionKey || context.revisionData.revisionKey

I believe the default should instead be: keyPrefix + context.commandLineArgs.revisionKey || context.revisionData.revisionKey

Also, from this documentation it is not clear how I can override the revisionKey parameter. Can I pass a function which accepts the keyPrefix and/or context to it?

Push a JSON metatdata blob when deploying

As well as putting the latest revision on the set of revisions, it would be great to store the json blob that ember-cli-deploy-revision-data creates.

We would also need to remove the blob when the corresponding revision is removed from the list

didDeploy hook to publish changes

hi I would like to see something like pub/sub when deploying changes, so when a didDeploy event happens this add-on can exec a PUBLISH ${keyPrefix}:deploy ${revisionKey} and a fastboot-notifier can be listen on that channel events and trigger a notification to download the new app version. let me know if that sound good to you and if I can help implementing this. thanks

improve logging for fetchRevisions

Currently fetchRevisions is fired both on deploy and deploy:list

the log messages in the hook are NOT marked as verbose but they should be for the deploy command as they lead to confusing output for the users

Prember / Multiple File Support

Prember pre-renders routes into a static directory structure with index.html files at build time. I'd like to be able to use the lightning strategy for deployment and pre-render my static routes. This can be supported by grabbing the files from s3 as assets, but you lose the ability to effectively use revisions, uploading the html as assets using the s3 plugin means the index.html files will be overwritten as they are not fingerprinted. I would like to upload all the html into redis just as it stores the main ember index.html file to retain the ability to have revisions.

I've noticed in the past a PR was rejected, as a use case didn't really exist at the time and the suggestion of using the plugin multiple times as part of the pipeline was the correct advice. The past issues and pr were about uploading a single other file. An app may have tens or hundreds of static routes being prerended by prember so creating a plugin entry for each html file would be cumbersome, prone to error and not very scalable.

I'm creating this issue to spark at the least a discussion about if prembers rising popularity changes the conversation around supporting multiple file support and if it should be pursued as part of this plugin.

possible to upload multiple files?

I have run into an issue with compatibility of using the lightning deploy strategy and serving a service worker. the sw.js file needs to be served up from the root domain and not from a CDN url so I'm wondering if it's possible to deploy both the index.html and sw.js files to redis where I can then adapt my server to also dish out the service worker from the main domain?

so far I've tried updating the filePattern but it feels like this is set to locate 1 file to upload and doesn't support many files... but I'm likely missing something so thought I'd check here. TIA for the help!

Swap out then-redis for another redis lib

Backstory and investigation for this is in ember-cli-deploy/ember-cli-deploy/issues/327 . Specifically in this comment

Current Status

It looks like IORedis should cover all the cases that we need for this plugin

we need a champion willing to attempt a port to IORedis (see comment from @blimmer below)

Stash checksum and/or timestamp in Redis

Similar to #25, it would be fantastic if ember-deploy-redis would also stash a checksum (or the commit SHA) and/or a build date/time (or the commit timestamp) of the index to use in the the ETag and Last-Modified HTTP headers. Something like:

foo:index:current => '<!DOCTYPE html>...'
foo:index:current:meta => checksum '787b3dd284fd99202adfa4e18a28126d159552c5' 
                          timestamp 'Wed Sep 16 19:28:37 2015 +0200'

And then...

require 'sinatra'
require 'redis'

get '/' do
  content_type 'text/html'
  expires 3_600, :public

  redis = Redis.new
  index_key = redis.get("foo:current")
  index_key = "foo:#{params[:index_key]}" if params[:index_key]

  meta = redis.hgetall("#{index_key}:meta")
  etag meta[:checksum]
  last_modified meta[:timestamp]

  redis.get(index_key)
end

Stashing the content in the same hash as the metadata would be ideal (to limit the Redis hits) but would break backwards compatibility.

this.readConfig is not a function

Hello guys and merry christmas 🎅,

Without any version changes or any altered setup in my laptop I noticed $ ember deploy now throws an exception:

+- didFail
TypeError: this.readConfig is not a function
TypeError: this.readConfig is not a function
    at Object.redisDeployClient (/Users/mariogintili/dev/sunlight/web-client/node_modules/ember-cli-deploy-redis/index.js:55:36)
    at CoreObject.readConfig (/Users/mariogintili/dev/sunlight/web-client/node_modules/ember-cli-deploy-plugin/index.js:63:30)
    at CoreObject.fetchInitialRevisions (/Users/mariogintili/dev/sunlight/web-client/node_modules/ember-cli-deploy-redis/index.js:137:38)
    at Object.fn (/Users/mariogintili/dev/sunlight/web-client/node_modules/ember-cli-deploy/lib/tasks/pipeline.js:93:21)
    at Pipeline._notifyPipelinePluginHookExecution (/Users/mariogintili/dev/sunlight/web-client/node_modules/ember-cli-deploy/lib/models/pipeline.js:175:19)
    at tryCatch (/Users/mariogintili/dev/sunlight/web-client/node_modules/rsvp/dist/rsvp.js:538:12)
    at invokeCallback (/Users/mariogintili/dev/sunlight/web-client/node_modules/rsvp/dist/rsvp.js:553:13)
    at /Users/mariogintili/dev/sunlight/web-client/node_modules/rsvp/dist/rsvp.js:628:16
    at flush (/Users/mariogintili/dev/sunlight/web-client/node_modules/rsvp/dist/rsvp.js:2373:5)
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
|
Pipeline aborted

My package.json is as follows

    "ember-cli-deploy": "1.0.0-beta.1",
    "ember-cli-deploy-revision-data": "0.3.2",
    "ember-cli-deploy-display-revisions": "0.2.2",
    "ember-cli-deploy-lightning-pack": "0.6.7",
    "ember-cli-deploy-ssh-tunnel": "0.2.2",

I tried upgrading the ember-cli-deploy-lighting-pack to the latest release to date - 0.6.10

which now yields the following:

+- prepare
|  |
|  +- revision-data
- creating revision data using `file-hash`
- generated revision data for revision: `997780b824c374fc5ce269e95918b032`
|
+- didPrepare
|
+- fetchInitialRevisions
|  |
|  +- redis
- Listing initial revisions for key: `sunlight-web-client`
events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: Timed out while waiting for handshake
    at Timeout._onTimeout (/Users/mariogintili/dev/sunlight/web-client/node_modules/ssh2/lib/client.js:569:19)
    at ontimeout (timers.js:365:14)
    at tryOnTimeout (timers.js:237:5)
    at Timer.listOnTimeout (timers.js:207:5)

I can confirm no SSH keys in my setup have changed(both on my end and on our servers) and that I have a more than stable internet connection - any help is greatly appreciated. Some additional info:

ember --version
ember-cli: 2.7.0
node: 6.9.1
os: darwin x64

then-redis yarn warning

When using yarn to manage packages it throws the following warning:
warning "[email protected]" has unmet peer dependency "redis-commands@*".

and the then-redis maintainer has announced he is no longer maintaining then-redis and that users should switch to node_redis which now contains promise goodness.

So, is it possible to drop then-redis and replace it with node_redis?

Allow user to specify the allowed number of revisions to keep

url option broken

I've specified the 'url' key in my redis config object, but it does not seem to be taken in to account and deploy tries to use default host and port.

Using [email protected]

works find if I configure 'host', 'port' and 'password' instead

When deploying in dev app:index:current is nil

I might have misunderstood the way dev deploy worked, but when I 'ember deploy dev', the redis key 'app:index:current' returns nil and I only have 1 manifest_id that seems correct and it is 'default'.

wouldn't it be nice to have 'app:index:current' return 'default' ?

EDIT
using [email protected]

Should an exception be thrown for duplicate redis keys?

Hello peeps,

I'm working on a project that makes use of this library in our CI. We basically trigger a deployment every time a build passes successfully.

Given the current design of this project - and its defaults - retrying a successful build yields an error since the deploy script will output the same manifest, causing the following:

|    - Value already exists for key: zapnito-web:index:288d60899ac3cc175fcd2bbb758dc3de
|
+- didFail
Value already exists for key: zapnito-web:index:288d60899ac3cc175fcd2bbb758dc3de
undefined|
Pipeline aborted

I don't think retrying a successful build - where the code hasn't changed - should yield an error. Could we please change the behaviour on this to be a warning rather than an error?

How do we set the expire for each key?

Hi Folks,

We're running into a scenario where our Redis store reaches its memory capacity. When that happens, ember deploy fails because there is no room to deploy the index.html to our Redis instance.

We're using RedisToGo, so one approach may be to set our maxmemory-policy to allkeys-lru since that policy doesn't require keys to have expire set.

CONFIG SET maxmemory-policy allkeys-lru

Unfortunately, RedisToGo seems to have obfuscated the CONFIG command (for security reasons no doubt) and its default maxmemory-policy is volatile-lru which requires keys to have an expire set in order to make space for any new data added once max memory is met.

Without access to CONFIG our only option is to set an expire on our ember cli deployment keys. It doesn't appear that ember-cli-deploy-redis sets an expire by default. Is there a configuration option or some other approach we could take to set an expire during ember cli deploy?

Thanks Everyone!

then-redis is no longer maintained

Today, this package is using then-redis:

ember-cli-deploy-redis/package.json

Line 39 in b299ad5

"then-redis": "^2.0.1"

However, then-redis is no longer supported.

I noticed this because of this warning when I yarn install

warning "ember-cli-deploy-redis > [email protected]" has unmet peer dependency "redis-commands@*".

This package should be migrated to use async behavior in node_redis.

Connecting manually via redis-cli works but ember-cli-deploy-redis gets a "connect ETIMEDOUT" error

Redis is running on a server that is not publicly accessible (it's in a private subnet) so I'm using ssh tunnel to connect from a public facing server that is in the same VPC.

If I ssh into that public facing server, I can connect to the redis instance on the private server just fine using redis-cli. But I get I'm getting timeout errors when I try to deploy to production using the following config:

'redis': {
  allowOverwrite: true,
  keyPrefix: 'pigeon:index',
  url: 'redis://:' + env('REDIS_PASSWORD') + '@' + env('REDIS_HOST') + ':6379',
},

'ssh-tunnel': {
  username: env('SSH_USER'),
  host: env('SSH_HOST'),
  privateKeyPath: env('SSH_KEY'),
}

I'm guessing this is some sort of configuration issue on my end since I don't see any other issues about it. But I get this same error using the host, password, port configuration variation rather than specifying a connection url:

Error: Redis connection to internal-pigeon-db-production-1812613668.us-east-1.elb.amazonaws.com:6379 failed - connect ETIMEDOUT 192.168.1.148:6379
    at RedisClient.on_error (/Applications/MAMP/htdocs/pigeon-web-app/node_modules/ember-cli-deploy-lightning-pack/node_modules/ember-cli-deploy-redis/node_modules/redis/index.js:196:24)
    at Socket.<anonymous> (/Applications/MAMP/htdocs/pigeon-web-app/node_modules/ember-cli-deploy-lightning-pack/node_modules/ember-cli-deploy-redis/node_modules/redis/index.js:106:14)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)
    at emitErrorNT (net.js:1269:8)
    at nextTickCallbackWith2Args (node.js:442:9)
    at process._tickCallback (node.js:356:17)

Any ideas about what might be causing this?

Edit: Explained how I'm using ssh tunnel.

Handle heroku redis urls smartly

Heroku redis addon url's generally have a placeholder for the username, ie

redis://x:345932ywwe659weotwy@some-host:12345

That x in that url is pointless and breaks the redis package we use. The required url is:

redis://:345932ywwe659weotwy@some-host:12345

We should make this plugin handle the url if it includes the username placeholder.

Unhandled rejection Error: ERR invalid DB index

Hi,

with these plugin installed:

"ember-cli-deploy": "0.5.0-beta.2",
"ember-cli-deploy-lightning-pack": "0.2.0-beta.5",

whenever the pipeline contacts redis on Heroku I got ERR invalid DB index. example on activation:

|
+- activate
|  |
|  +- redis
|    - Activating revision `84549e8ecccb5d3ae31c1583f125d2dc`
Unhandled rejection Error: ERR invalid DB index
    at ReplyParser.<anonymous> (/Users/masciugo/dev/quenta/node_modules/ember-cli-deploy-lightning-pack/node_modules/ember-cli-deploy-redis/node_modules/redis/index.js:317:31)
    at ReplyParser.emit (events.js:107:17)
    at ReplyParser.send_error (/Users/masciugo/dev/quenta/node_modules/ember-cli-deploy-lightning-pack/node_modules/ember-cli-deploy-redis/node_modules/redis/lib/parser/javascript.js:296:10)
    at ReplyParser.execute (/Users/masciugo/dev/quenta/node_modules/ember-cli-deploy-lightning-pack/node_modules/ember-cli-deploy-redis/node_modules/redis/lib/parser/javascript.js:181:22)
    at RedisClient.on_data (/Users/masciugo/dev/quenta/node_modules/ember-cli-deploy-lightning-pack/node_modules/ember-cli-deploy-redis/node_modules/redis/index.js:547:27)
    at Socket.<anonymous> (/Users/masciugo/dev/quenta/node_modules/ember-cli-deploy-lightning-pack/node_modules/ember-cli-deploy-redis/node_modules/redis/index.js:102:14)
    at Socket.emit (events.js:107:17)
    at readableAddChunk (_stream_readable.js:163:16)
    at Socket.Readable.push (_stream_readable.js:126:10)
    at TCP.onread (net.js:538:20)
|    - ✔ Activated revision `84549e8ecccb5d3ae31c1583f125d2dc`
|
+- didActivate
|

anything to do with this issue? It doesn't seem to have any effects on the expected outcome, though.

Error in 1.0.1: this._listRevisions(...).then is not a function

TypeError: this._listRevisions(...).then is not a function
TypeError: this._listRevisions(...).then is not a function
    at CoreObject.fetchRevisions (/Users/lmelia/p/yapp/dashboard/node_modules/ember-cli-deploy-redis/lib/redis.js:75:43)
    at CoreObject.fetchInitialRevisions (/Users/lmelia/p/yapp/dashboard/node_modules/ember-cli-deploy-redis/index.js:149:47)