Comments (11)
Do we really need/want something that will automatically delete files? I for one know that I would stop using that code entirely if that were in place. Web files are cheap, not having them there for consumption is expensive. Are there good use cases for it? Can we make sure it has an off switch when using sync?
from grunt-s3.
It's certainly a bit of a dangerous feature and it should be off by default. I am maintaining a large number of assets on s3. Changes are mostly additions but sometimes they are also deletes. Having a unified way to maintain the assets would be good.
from grunt-s3.
I'll take a stab at it. My interest is piqued. I'm wondering if it would be better to do a whitelist with rules, rather than a delete:on option.
For instance:
del: {
files: [ '/only/these/*/files/.js` ],// can be deleted
between : [ null, new Date(new Date().setDate(new Date().getDate() - 10 )) ]
}
So only delete from files, and if they are between date 0 and 10 days ago (don't delete anything in the last 10 days). The delete functionality would obviously be updated to use this, so that sync can take advantage?
The hardest part is really just knowing what files are on S3 to actually delete. I don't think it's actually possible with Knox and will require the AWS lib.
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/frames.html
from grunt-s3.
By the way thanks for implementing the initial sync functionality. For my use case the above filters would be complicating the solution, though they may be useful to someone else. I would be interested to hear the opinions of other people using this project. To me 'sync' implies the sync functionality will make whatever changes that are necessary to files stored on s3 to reflect the current state of the local files (PUT'ing and DELET'ing).
We could use https://github.com/segmentio/s3-lister. It uses a knox client to implement a streaming interface to listing a bucket. Though it probably makes sense to build this on top of AWS's node sdk.
from grunt-s3.
It is actually possible to get a list of files in a bucket with knox, so it should be very possible to add this in.
client.list({ prefix: 'my-prefix' }, function(err, data){
from grunt-s3.
The only problem with that, is that it can only list 1000 files at a time, so some paging functionality would have to be implemented.
from grunt-s3.
One thing at a time. Getting the first 1000 to work first would be a great step forward :)
from grunt-s3.
yes it would :)
from grunt-s3.
+1. I think this feature should be added to make able ability to sync the whole folder: upload files that not in bucket yet and delete objects that don't exist on file system. Also helper s3.list would be helpful for composing custom tasks.
from grunt-s3.
+1. It would be great to delete objects that no longer exist on the filesystem.
from grunt-s3.
+1 for this as well
from grunt-s3.
Related Issues (20)
- alpha.3 looses headers on upload HOT 1
- Fatal error: write EPIPE HOT 9
- Sync fails for files that don't exist in bucket HOT 4
- Clean up branches HOT 5
- Tests should run against an actual S3 bucket HOT 1
- Consider setting up readthedocs? HOT 1
- Delete Remote Folders
- Task runs without failing or uploading anything HOT 4
- Require templates from s3 HOT 1
- gzip option breaks s3 task with warning HOT 1
- Doesn't do anything HOT 6
- Whoops. HOT 1
- Anyone had the error Fatal error: write ECONNABORTED? HOT 2
- Files not being written into bucket folders HOT 4
- Random ETIMEOUT and hangups after uploading files HOT 3
- After upload invalidate file when it was changed
- Setting meta data HOT 2
- Support for AWS S3 CLI default secrets location
- No compatible version found: grunt-s3@~0.2.0
- Large file downloads fail with length > kMaxLength
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from grunt-s3.