Comments (4)
As we discussed the real issue isn't that you want to be able to reason about code precisely like the above, but that we don't want to be doing extra IO or reading files into memory unnecessarily when working with a large directory of big files, in readDirectoryWithL
.
In the case of readDirectoryWithL readFile
we don't have any choice but to have read the file, once we get to a File
constructor, right? In that case the amount of actual IO depends on readFile
and can't be reasoned about AFAICT.
Do you have another example that I could use as a direct test case? Let me know if I'm still misunderstanding, or have overlooked something; I haven't looked at this module for a while.
from directory-tree.
My example could be missunderstood - thank you for clarification.
So the problem is, that when we only want to list filenames, the whole files are read to memory (even if we don't need them). Of course this is connected to constructors, readFile (...) etc., but maybe there is some way to fix this behaviour?
(In the file data type the field 'file' could be another structure, that when accesed would load the file - or something like that - you simply need to separate the process of loading the file name from the process of loading it contents)
Sample code:
import System.Directory.Tree
import qualified Data.Foldable as F
import qualified Data.Traversable as T
main = do
dir <- readDirectoryWithL readFile "test"
let
files = contents $ dirTree dir
names = fmap name files
print $ names
The above code reads all the files in directory "test" and prints their names. We do not need to read their content, but your library is loading each file to memory thought.
I think this is a serious bug concidering situation when these files are big (like 10Gb each).
from directory-tree.
Hello, is there any progress with this issue? :)
from directory-tree.
Sorry, I meant to point out that you can use buildL
or readDirectoryWithL return
if you just care about file/dir names don't have any interest in the contents. I don't think there's anything else I can do about this.
Going to close, but let me know if you're not satisfied or I'm still not understanding.
from directory-tree.
Related Issues (11)
- Add free variable to Dir? HOT 2
- Test suite failure HOT 3
- Bifunctor support? HOT 3
- Weird behavior on /dev/fd with readDirectoryWithL HOT 5
- Zipper support? HOT 4
- Use `unsafeInterleaveIO` instead of `unsafePerformIO` HOT 1
- EXAMPLES/Examples.hs dependencies missing.
- Infinite loop when symlink refers to parent dir
- suggestion: add a function to simply collect recursivley the filenames in a directory HOT 1
- latest GHC support HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from directory-tree.