Time to find a non glob path increase with number of files in the same directory

[mrmlnc]

Time to find a non glob path increase with number of files in the same directory

[mrmlnc]

I prefer solution with is-glob with strict mode. In this case we needs have a separated tasks for static and dynamic patterns (just like the dynamic property in the ITask interface). Then, when processing a task, simply pass them to the correct adapters (fs and readdir-enhanced).

I've already started working on this issue.

[mrmlnc]

Hello, @pvdlg,

Good catch! Yeap, this is due to the fact that we first try to read the directory, and then try to apply patterns.

So we need fix this behaviour. Thanks for reporting!

[skarfacegc]

Would a check for file existence on each glob work?

const strippedGlobs = []
const results = [];
globs.forEach(glob =>{
  fs.access(glob, fs.constants.R_OK, err => {

    // file doesn't exist, add to the list of globs
    if(err){
      strippedGlobs.push(glob);
    } else {
      // file does exist, add to the result set
      results.push(glob);
    }
  })
})

[mrmlnc]

Must be fixed by [email protected]. Please, read release notes for more details.

Thanks for reporting ❤️!

[mrmlnc]

Just a few thoughts.

We can extract only constant patterns from all patterns, for example, using a is-glob package and run them as a separate task with different types (something like glob and regular). This requires changes in the task manager, creating adapters (glob, regular) and some magic in the base reader class. But this is first view.

[mrmlnc]

@skarfacegc,

Yeap, good idea!

I think it's possible. Now the task manager doesn't know anything about the File System and i think he shouldn't know anything about it. We can create a new manager, provider or something and check whether the pattern exists on the file system. We need to think about how to solve this problem correctly.

I'll invest time to this issue in the near future.

[skarfacegc]

If you can point me in the right direction I can take a look as well. I looked around but I couldn't wrap my head around how all the pieces fit together.

[pvdlg]

By checking for each string passed in the patterns wouldn't we loose performances for cases in which the pattern is an actual glob?

Passing a non glob path is still less common than passing a glob.
Maybe we should still use is-glob before testing on the FS.