Reading the filename implies reading the whole file

[flowboxio]

Reading the filename implies reading the whole file

[jberryman]

As we discussed the real issue isn't that you want to be able to reason about code precisely like the above, but that we don't want to be doing extra IO or reading files into memory unnecessarily when working with a large directory of big files, in readDirectoryWithL.

In the case of readDirectoryWithL readFile we don't have any choice but to have read the file, once we get to a File constructor, right? In that case the amount of actual IO depends on readFile and can't be reasoned about AFAICT.

Do you have another example that I could use as a direct test case? Let me know if I'm still misunderstanding, or have overlooked something; I haven't looked at this module for a while.

[wdanilo]

My example could be missunderstood - thank you for clarification.
So the problem is, that when we only want to list filenames, the whole files are read to memory (even if we don't need them). Of course this is connected to constructors, readFile (...) etc., but maybe there is some way to fix this behaviour?

(In the file data type the field 'file' could be another structure, that when accesed would load the file - or something like that - you simply need to separate the process of loading the file name from the process of loading it contents)

Sample code:

import System.Directory.Tree
import qualified Data.Foldable as F
import qualified Data.Traversable as T

main = do
    dir <- readDirectoryWithL readFile "test"
    let
        files = contents $ dirTree dir
        names = fmap name files
    print $ names

The above code reads all the files in directory "test" and prints their names. We do not need to read their content, but your library is loading each file to memory thought.
I think this is a serious bug concidering situation when these files are big (like 10Gb each).

[wdanilo]

Hello, is there any progress with this issue? :)

[jberryman]

Sorry, I meant to point out that you can use buildL or readDirectoryWithL return if you just care about file/dir names don't have any interest in the contents. I don't think there's anything else I can do about this.

Going to close, but let me know if you're not satisfied or I'm still not understanding.