ahaurw01/gulp-remember

A plugin for gulp that remembers and recalls files passed through it

gulp-remember's Issues

Why use gulp-remember when there is gulp-cached?

This probably isn't the right place for this question, but why use gulp-remember? Here's my build:

gulp.src SRC.coffee
    .pipe do sourcemaps.init
    .pipe cached 'coffee'
    .pipe (coffee COFFEE_OPTIONS).on 'error', (_.partialRight error, 'CoffeeScript')
    .pipe do sourcemaps.write
    .pipe ngAnnotate NGANNOTATE_OPTIONS
    .pipe concat 'bundle.js'
    .pipe gulp.dest DISTDIR

This produces the correct result as far as rebuilding the file that changed, and concating it with the rest of the files that haven't.

When I add in gulp-remember, the result seems to be the same, though each incremental rebuild takes 3x longer. What am I missing here?

Order issue, gulp-order not solving it

I am trying to use gulp-order as stated is needed if order is to be respected. I am trying but gulp-order is not doing anything. I created the exact repo to outline the problem available here.

https://github.com/newtonianb/gulp-order-problem
npm install && bower install && gulp js and see output in public/scripts.js. The coffeescript file (a.coffee) is being concatenated after the js file (b.js) eventhough the order states a should come first.

I also asked in gulp-order but I'm wondering if there is some incompatibility recently introduced or if I'm doing something wrong?

'use strict';
var gulp = require('gulp');
var $ = require('gulp-load-plugins')();

gulp.task('js', function (callback) {

    var coffeeFilter = $.filter(['**/*.coffee']);

    var src = [
        'resources/assets/bower/jquery/dist/jquery.min.js',
        'resources/assets/bower/jquery-migrate/jquery-migrate.min.js',
        'resources/assets/scripts/util/a.coffee',
        'resources/assets/scripts/b.js'
    ];

    var destName = 'scripts.js';
    var dest = 'public/assets';

    return gulp.src(src)

    .pipe(coffeeFilter)
    .pipe($.cached('coffee'))
    .pipe($.coffee())
    .pipe($.remember('coffee'))
    .pipe($.order(src, { base: process.cwd() }))
    .pipe(coffeeFilter.restore())

    .pipe($.concat(destName))
    .pipe(gulp.dest(dest));
});

Not piping what passed through, but piping destination files

Soo,

I just wrote a boilerplate kinda code,

like SO:

         var $ = require('gulp');
/*
    require gulp-load-plugins, and call it
    gulp-load-plugins will require individually,
    all the plugins installed in package.json at the root directory
    these required plugins, are lazy loaded
    that is, gulp-load-plugins will only require them when they are referenced in this file
    plugins are available for perusal, via camelcased names, minus gulp
    eg: gulp-clean-css is accessed by $$.cleanCss
*/
         var $$ = require('gulp-load-plugins')();

        function styles(src){
                  var stream= $.src('css/**/*.css')

                      .pipe($$.debug({title:'from - src',minimal:false}))

                       .pipe($$.cached('styles'))

                         .pipe($$.debug({title:'from - cached',minimal:false}))

                                 //do special stuff
                             .pipe($$.remember('styles'))

                                  .pipe($$.debug({title:'from - remember',minimal:false}))  

                                       .pipe($.dest('build/css'));
             return stream;
          }

I added those debug lines to know what is piped out at every step.

My css folder looks like so:
|-css
  |-test.css
  |-test4.css

Run 1 - Initial run, nothing in cache, everything is cached

So far so good - Nothing was in cache, so all files are piped by gulp-cached

Run2 - let us edit test4.css and gulp.watch will trigger styles task again

Expectation:

1. Both css files will be piped from gulp.src
2. Only test4.css has changed, so gulp-cached pipes only test4.css
3. Remember will have to pipe both test.css and test4.css under the css directory
   i.e, the path for both should be
   css/test.css
   css/test4.css

Result

Uh oh!!!! Where did that build come from?

I am curious

Strange Behavior on Gulp Remember

Hi there. I'm using your plugin and gulp-cached to optimize my image-resize task. Here in config.json, I have several presets for resize, like qvga, vga, hd720p, and so on...

To avoid nodejs Error: spawn ENOMEM, I break the resize task into multiple execution pipeline, sending finished event after each pipeline is finished.

Here is my gulp task:

var fse = require('fs-extra');
var gulp = require('gulp');
var _ = require('lodash');
var lP = require('gulp-load-plugins')();
var path = require('path');
var util = require('util');
var EventEmitter = require('events').EventEmitter;

gulp.task('image-resize', function( callback ){

  var options = fse.readJsonSync('config.json', { encoding : 'utf8' });
  var Pipeline = function(){
    this.queue = [];
  };

  util.inherits(Pipeline, EventEmitter);

  Pipeline.prototype.executeNextPipeline = function(){
    if (this.queue.length === 0) {
      // there's no more pipeline. Call callback to tell gulp that this task is done
      callback();
    } else {
      var pipeline = this.queue.shift();
      pipeline(); //execute it
    }
  };

  Pipeline.prototype.fillQueue = function(){
    var self = this;
    this.queue = _.map( options.media.image.resize, function(value, key){
      return function(){
        return gulp.src( path.join('content', '**', '*' + options.media.image.glob) )
          .pipe(lP.cached('imageResize', {optimizeMemory:true}))
          .pipe(lP.remember('imageResize'))
          .pipe(lP.rename(function (path) {path.basename += "_" + key; }))
          .pipe(lP.changed( 'dist' ))
          .pipe(lP.imageResize({
            width : value,
            quality : options.media.image.resizeQuality}))
          .pipe(lP.imagemin({
            optimizationLevel : 4,
            progressive : true,
            interlaced : true
          }))
          .pipe(lP.size({title: 'image-resize-' + key }))
          .pipe(gulp.dest('dist'))
          .on('end', function(){
            self.emit('finished');
          });
      };
    });
  };

  var pipeline = new Pipeline();

  // When one pipeline is finished, execute next
  pipeline.on('finished', pipeline.executeNextPipeline);

  // fill the pipeline first
  pipeline.fillQueue();

  // start pipeline Execution
  pipeline.executeNextPipeline();
});

What is so strange about it is, instead of outputting the correct image size and name, each files are just using the qvga preset, and the file name is recursively appended. Here are the screenshot:

Is this the correct behavior? I thought that gulp-remember works by remembering files that has been passed through it. And since I add .pipe(lP.remember('imageResize')) before .pipe(lP.rename(function (path) {path.basename += "_" + key; })), the file contents and file names shouldn't collide....

Adds file back to the stream with wrong path (i.e. path changes that happened after gulp-remember are already applied)

[Using gulp 3.9.0, gulp-cached 1.1.0, gulp-remember 0.3.0, gulp-rename 1.2.2]
[stackoverflow post: http://stackoverflow.com/questions/33072113/gulp-remember-seems-to-output-wrong-path]

We're using gulp as a build tool and gulp-cached together with gulp-remember to allow fast incremental rebuilds. Part of the files under build have to be moved to a different output directory and this has to happen in-stream (i.e. not in gulp.dest()), because we're zipping the results afterwards. We use gulp-rename for this.

However, when the build script is called multiple times (gulp.watch() for incremental rebuilds), then it seems to apply the gulp-rename transformation multiple times. It looks like gulp-remember is not actually outputting the files with the path it saw them last time, but with the path they got after the gulp-rename step.

I've narrowed the problem down to the following script:

var gulp = require('gulp');
var cached = require('gulp-cached');
var remember = require('gulp-remember');
var rename = require('gulp-rename');

function build() {
    gulp.src("src/**")
        .pipe(cached())
        .pipe(remember())
        .pipe(rename(function(path) {path.dirname = "resources/" + path.dirname;}))
        .pipe(gulp.dest('build/'));
}

gulp.task('default', [], function() {
    build();
    build();
});

Running this on a source directory src with just one file "myfile.js" produces the output file:

/build/resources/resources/myfile.js

If the second call to build() is removed, it produces correctly

/build/resources/myfile.js

And if we insert a third call to build(), it produces

/build/resources/resources/resources/myfile.js

I think gulp-remember should output the files with the path they had when they passed through it last time, not the path the files got after being processed further.

remember seems to cache

This is the code called by my watch task on any scss changes:

module.exports = function(gulp, plugins, config) {
    return function() {
        return gulp.src([config.srcPath + 'sass/**/*.scss'])
            .pipe(plugins.plumber())
            .pipe(plugins.cached('scss-lint'))
            .pipe(plugins.sourcemaps.init())
            .pipe(plugins.scsslint())
            .pipe(plugins.remember('scss-lint'))
            .pipe(plugins.sass().on('error', plugins.sass.logError))
            .pipe(plugins.autoprefixer({ browsers: ['last 2 versions'] }))
            .pipe(plugins.sourcemaps.write('.'))
            .pipe(gulp.dest(config.destPath + 'css'));
    };
};

scsslint receives all updated files and lints them just fine, but then sass() only compiles the old files from the first run. All subsequent runs triggered through my watch task will result in exactly the same css file - to cached() works fine, passing the updated file(s) to scsslint, but remember() seems to free only the files it remembers from the fist run. All sass() receives is always the same set of (cached) scss files.

When I comment out cached() and remember(), everything works fine. My workaround is to move the scsslint-command into a separate task that uses cached() which works fine, but isn't remember() just for not having to do that?

To me this seems to be an issue of remember(), or is my thinking wrong here?

Replace deprecated dependency gulp-util

gulp-util has been deprecated recently. Continuing to use this dependency may prevent the use of your library with the latest release of Gulp 4 so it is important to replace gulp-util.

The README.md lists alternatives for all the components so a simple replacement should be enough.

Your package is popular but still relying on gulp-util, it would be good to publish a fixed version to npm as soon as possible.

See:

• https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5
• gulpjs/gulp-util#143

Letter case

Paths SomeThing and something are equal. You should cache it in lower case and lowercase before check.

It does not handle files added and removed after run tasks

code is taken out as it is use.
Running the sample:

1. run the task
2. create tracked file
   2.1 task is done correctly
3. delete file
   3.1 remember it does not forget

It works correctly if the file was in the directory before starting the task. after removal of remember forgetting file.
but when the file is created in the directory and remove it remember not to forget about him.

what is the problem? maybe I made a mistake somewhere? correct me.
Thank you! :)

expose `caches` to module user

Hi there,

Is it possible to expose caches to module user, like the way gulp-cached did?

This is useful when someone want to delete remembered file on several gulp watchers programatically.

Like this:

gulp.task('watch', ['build'], function () {
    var watchList = [
        'less',
        'js',
        'html'
    ];
    var watchers = watchList.map(function (target) {
        var location = locations[target];
        var watcher =  gulp.watch(location, ['build:'+target]);
        watcher.on('change', function (msg) {
            if(msg.type === 'deleted') {
                // gulp-cached expose caches to users, so the following code works
                if(cache.caches[target]) {
                    delete cache.caches[target][msg.path];
                }
                // need a way to figure out whether the caches exists or not, cuz if the cache is not exists
                // the following code would throw an exception
                remember.forget(target, msg.path);
            }
        });
        return watcher;
    });
});

If expose the caches is not acceptable, do you think adding a check in gulpRemember.forget make more sense?

gulpRemember.forget = function (cacheName, path) {
  if (arguments.length === 1) {
    path = cacheName;
    cacheName = defaultName;
  }
  if (typeof cacheName !== 'number' && typeof cacheName !== 'string') {
    throw new PluginError(pluginName, 'Usage: require("gulp-remember").forget(cacheName, path); where cacheName is undefined, number or string and path is a string');
  }
  // If caches[cacheName] does not exist, the following line will cause an exception
  delete caches[cacheName][path];
};

Checking for changes accross teams

I want to use gulp-imagemin which is a very slow and processor intensive task and I am working in a team environment using Git. How can I make sure that an image is only ever processed once in my entire team? Where does gulp-changed-in-place save it's change log? Can it be checked-in to Git?

Weird behavior - Forgets source Jade files replacing them with compiled HTML

The problem is - on the first run, gulp-remember remembers all the files and initial compilation goes fine, but starting from the second run, a part of previously remembered files are replaced with compiled HTML files.

In the task below, all *.jade files are passed to the stream, but only root-to-base-folder files are being compiled.

'use strict';

const 
  gulp     = require('gulp'),
  combiner = require('stream-combiner2').obj,
  $        = require('gulp-load-plugins')();

module.exports = function(options) {
  return function() {
    return combiner(
      gulp.src(options.src, {
        since : gulp.lastRun(options.taskName)
      }),
      $.if(options.flags.debug, $.debug({title : 'DEBUG ' + options.taskName + ':src'})),

      $.if(function(file) {
        return /\.jade$/.test(file.relative);
      }, 
        combiner(
        $.remember('templates'),
        $.if(options.flags.debug, $.debug({title : 'DEBUG ' + options.taskName + ':remembered'}))    
      )),

      $.if(function(file) {
        return /^[a-zA-Z0-9-_]+\.jade$/.test(file.relative);
      },
        combiner(
          $.jade(),
          $.if(options.flags.debug, $.debug({title : 'DEBUG ' + options.taskName + ':compiled'})),
          gulp.dest(options.dest),
          $.if(options.flags.debug, $.debug({title : 'DEBUG ' + options.taskName + ':dest'}))
        ) 
      ),

    ).on('error', $.notify.onError(function(err) {
      console.log(err);
        return {
          title : 'Templates',
          message: err.message
        };
    }));  
  };
};

On the second run (triggered by gulp.watch), no files will be passed to $.jade() because all root jade files are respectfully replaced (replaced in gulp-remember cache) with compiled html files and there are no more files to match /^[a-zA-Z0-9-_]+\.jade$/.test(file.relative).

The question is how to avoid this behavior?
I want to remember only jade files and I want them to be deleted or replaced from the stream only with forget method.

Plugins which compile to different paths won't ever be removed from cache

Scenario:

You have a bunch of .coffee files.

During the build process you transpile those to .js and then you remember() them. Since you're remembering the transpiled versions, their path will be altered from path/to/file/foo.coffee to path/to/file/foo.js.

Then when you delete that file, the lookup for the path will not find the path/to/file/foo.js since that's not the original path name. So in other words, after you delete your source file, it will continue to be pulled back in and never actually removed.

This will happen for any file when the original extension changes during the build process.

After remembering() files are out of order from their original gulp.src

This took me a second to track down.

Imagine we have the original gulp.src of files in a very particular order.

["path/to/file1.js", "path/to/file2.js", "path/to/glob/**, "path/to/file3.js"]

When these first go through remember(), it will set key/vals of their path and file content.

Now let's imagine we modify file3.js.

Since we're also using gulp-cached, it will strip out all of the other non-modified files, so our stream will only consist of file3.js. When remember() runs it will replace the cache'd version, but immediately push this back into our stream as the first file.

Then flush runs which adds the remaining files. However, file3.js went into the stream first, which means it's now out of order from what it was originally sent in as. The fix here would be to replace the cached file contents with the new one, but to not push this downstream. When flush runs, it should just automatically add in all files again irrespective if we've seen them before.

EDIT:
This creates a secondary issue when deleting + adding back files, as these now would appear at the bottom of the build. I think this can be fixed by preserving the original key of the path, but null'ing out the file contents and avoiding processing that file. Working on a fix...

How to use this with gulp-sourcemaps?

Is it possible to have incremental build that still produce correct sourcemaps?