Document deploying index and assets to same bucket

[kerry350]

Document deploying index and assets to same bucket

[andrewbranch]

Yep, I’m having this issue too. Took me forever to diagnose. This plugin just randomly blows out other assets. Maybe I’ll open a PR...

What are the advantages of using two separate buckets? That seems to be common practice, but I can’t figure out why.

[andrewbranch]

Investigated: the offending code begins in _cleanupBucket:

if ((data.Contents.length - this.manifestSize) > 0) {
  var itemsToDelete = this._getItemsForDeletion(data);
  console.log(itemsToDelete);
  this._deleteItemsFromBucket(itemsToDelete)
  .then(resolve)
  .catch(reject);
}

The implementation of _getItemsForDeletion simply gets the n most recently modified files and selects everything else for deletion, where n is the number of revisions you want to keep around, defaulting to 5.

The quick and dirty solution, then, is to pass manifestSize: Infinity to the deploy.js options. The only downside is you’ll have to clean out revisions manually, should you want to keep the number down.

_getItemsForDeletion should probably restrict the files it marks for deletion to files in the root of the bucket (or specified root folder, if that’s an option) that match the pattern {indexPrefix}:{whatever hash pattern}.html. I may open a PR later this week to implement that, but setting manifestSize works for now.

[quiddle]

@andrewbranch Let me try to explain the thinking here.

The ember-deploy method deploys all assets to one location (probably an S3 bucket). These assets are all fingerprinted, so they don't overwrite previous assets. You can then test this deploy using the revision key. When you're ready, you activate the revision by swapping the index file with the correct revision. This also allows you to revert to a previous revision just as quickly. All of the previous assets continue to live in your bucket.

The asset deployment is a dumb process- it just uploads anything new. The index upload will depend on what tools you're using. This particular plugin (ember-deploy-s3-index) allows you to use a separate s3 bucket as your index manager, instead of a redis cache, which is the default (I think) index plugin.

I think your confusion comes from not having a complete understanding of the role of this plugin.

[andrewbranch]

I understand that. I didn’t at first, but by the time I dove into the source and filed this issue, I had a pretty complete picture of the process. But there’s no reason for this plugin to delete things that are not index revisions. Indiscriminately deleting everything but the most recent five uploads is also a dumb process, but there’s no reason we can’t make it smarter. This would allow people to use fewer buckets on S3, if they feel like that’s a thing they should do, and wouldn’t affect the paradigmatic use case of having separate buckets. This would only make this plugin more useful. What’s the downside?

[quiddle]

Go for it then! I don't really have a reason not to, except that it seems wrong (to me) that the assets and index revisions would live together. That's part of the reason they're designed as separate plugins in the first place. (Also, s3 buckets are free! What's the downside of having a separate bucket?) Your change wouldn't change my workflow, so no worries!

The plugin can definitely be smarter about handling the revision history. A lot of that depends on now the revisions are named, and I know there was talk about stabilizing that. There's a lot in a holding pattern as the API and workflow for ember-deploy 0.5.0 is worked out, which will introduce a new pipeline for deployment.

[treyhunner]

I may have improperly named this issue. I don't actually care about keeping all assets in the same bucket, but I'm assuming that means as essential to the end I'm looking for.

What I want:

• https://mydomain.com/robots.txt works
• https://mydomain.com/favicon.ico works
• https://mydomain.com/opensearch.xml works

If there might be a way to update or extend this plugin to allow serving more than just the homepage somehow (maybe through some sort of bucket-to-bucket redirect?), that would be great.

I may be willing to put some time into coding/documenting a solution for that issue, but I'm not sure what an appropriate solution would look like outside of putting all those files into the root of the index bucket.

[quiddle]

If you're trying to solve this with an S3 only solution, here's what I would do:

Upload those assets (robots.txt, favicon.ico, etc) to your asset bucket with the rest of your assets. You'll need to figure out how to set cache rules in the right manner, so you can update them as needed.

Then, on your index bucket, use Routing Rules to redirect requests for robots.txt, et al to your asset bucket.

That seems like it would work. Anyone else have any ideas?