Comments (28)
Hi @drunkenbeaver,
your helicopter view of MS Word's file operations is certainly correct. At this level of detail I have been able to perform the steps until
Open Word.docx
Create ~$Word.docx
Save Word.docx
Create ~WRD0000.tmp
At this point, MS Word initiates a series of Read
operations on both the original and the temporary file (mostly just for the leading 512 bytes) and looks to be in some obscure way dissatisfied with the results. Next, MS Word pops up an error window citing "Word cannot complete the save due to a file permission error." and terminates the write operation.
This might be a general problem of Word not being able to update .docx files in place on a network shared volume - in which case there isn't much I can do about it.
Apart from that I have tracked down the error you reported back in December, so you'll at least be able to save a .docx file under a different name with the next DokanCloudFS release.
from dokancloudfs.
You're right - the latest changes that were introduced to support filesystems with "immutable" files (read: MEGA) seem to have broken something.
Please revert to v 1.0.4-alpha for the time being - I'll get to the root of this.
from dokancloudfs.
The crux of the problem seems to be that in Dokan Operations set allocation Size
var context = (StreamContext)info.Context;
context.Stream = scatterStream;
and the content is never copied from the Dokan context stream to the scatterStream so everything is blank :)
from dokancloudfs.
Could you please check if my latest checkin to the develop branch does any good for your problem?
I haven't quite gotten around to reproducing your observation with the call to SetAllocationSize()
, however, there definitely was a bug in determining file sizes for encrypted files - which may account for your hangs in Stream.CopyTo()
.
With the fix applied I was able to write a ~12Mb binary file to a FILE based volume, verify its CRC while on the mounted volume and copy it back to a normal HD intact.
from dokancloudfs.
I will check however I should have clarified before I am using the library with the encryption key set to "" which, I understood, would bypass the encryption all together, I am wondering if this could also account for the problems I am seeing
from dokancloudfs.
You are correct in assuming that encryptionKey = string.Empty
bypasses encryption.
I will doublecheck tonight that everything works as it should with encryption turned off.
BTW, do you access your mounted cloud volume through Windows Explorer or directly from .NET code?
My primary design goal for DokanCloudFS has been to enable cloud access for Windows Explorer, so there is a chance that not all code paths are covered for "raw" .NET - although this will definitely work for the CloudFS gateways.
from dokancloudfs.
I access the mounted cloud volume via windows Explorer,
I am still seeing issues with writes after the changes.
I wonder if the proxy file creation could be interfering as that seems to be a fairly recent change.
the reason I mentioned the SetAllocation is that I am seeing scenarios where Write is called before set allocation and as far as I can tell that written content would not end up in the scatterstream.
from dokancloudfs.
the reason I mentioned the SetAllocation is that I am seeing scenarios where Write is called before set allocation and as far as I can tell that written content would not end up in the scatterstream.
That's interesting. I've not seen this behavior so far. IIRC the only case where SetAllocationSize()
is not called first is for small files (< 4kb) where the entire content is written in a single Write()
.
Do you have a specific scenario that I could try to reproduce, like
- do you write a single file/multiple files/a single folder with its files/nested folders?
- what size are the affected files?
- what filesystem type is the originating volume?
- what OS platform are you testing on?
from dokancloudfs.
I was testing with small files.. but in those cases the file save hangs.
from dokancloudfs.
Ok, bad habit. I just saw that your reply came between my original comment and my edits.
If you'd be so kind as to give me a couple more specifics of your scenario I'd be happy to look into this tonight.
from dokancloudfs.
Sorry about that.
My setup is windows 8.1 and windows 10
I am testing with large txt files and small txt files from notepad as that uses simple file io and and office documents such as doc and docx files which has a tendency to use overlapped io
file system I am writing to is NTFS
all files are affected.
I am using BTSYNC that is selectively syncing files from a remote share but as far as access goes all my reads and writes are local and I am not seeing file access issues.
from dokancloudfs.
Please verify if #9c11912 improves anything.
from dokancloudfs.
These are tests with small files.
1: So new file saving no longer hangs but the files are written to the disk and then immediately are removed
2: If you attempt to open an existing txt file and then attempt to save it an error is thrown as the driver attempts to use the original stream that only has read permissions
when using the file sample cloud disk saving a file now returns a permissions error
Attached are the two log files from my notepad test with my virtual drive "CloudShare" and the "File"
virtual drive
for this test I literally started the drive,
opened up notepad
typed "this is a test"
and then hit the close button at which point it asked if I wanted to save and I selected each of the drives named the file something like test.txt and attempted to save.
CloudShare Simple Text Save.txt
File Simple Text Save.txt
from dokancloudfs.
Thanks very much for this detailed report - I now understand the reason behind your observed errors much better.
To be frank, I haven't viewed the online editing use case as very important so far and rather focused on a backup/sync scenario.
from dokancloudfs.
Ok, most of your simple test scenario should work now:
- I was able to create a new text file by saving a short text from Notepad exe.
- After restarting Notepad.exe and opening the text file I could save a modified text to the opened file.
All this was tested on a FILE gateway, but it should work just as well on any of the other gateways - with the exception of hubiC (currently not fully operational).
Thanks again for your cooperation, @adamrogas. Please feel free to report any additional shortcomings or to close this issue if there are none.
from dokancloudfs.
So I just tested this, the problem of initial writing seems to have been resolved.
but if I open a file in notepad, and type some changes and hit save or close it immediately comes back with "Invalid Parameter".....
at first glance it looks like what is really happening is the write cant access the file because the read already has the file open and has not released the file yet. this error then cascades down and ultimately the driver attempts to remove the file for some reason :) which luckily it can't for the same reason, "The File is locked"
I have included a screen shot of my system during this error, which gives a surprising amount of detail.
I have the call stack visible as well as the locals where you can see the error message,
from dokancloudfs.
Nice shot. But why do you think the amount of detail to be surprising?
from dokancloudfs.
The reason for the driver trying to remove a file when errors occur (except UnauthorizedAccessException
) is again related to my bias towards a backup/sync scenario.
In the presence of unrecoverable write errors in the cloud (use of a fully transactional file system is ruled out for the time being) I'd rather have a file missing entirely in my backup than keeping a corrupt copy. Especially when considering that there is no such thing as a partially corrupt file with AES encryption.
But I'd be interested in your view on this matter. Maybe there are other cases than authorization failure that can be compensated without purging the target file.
from dokancloudfs.
On the file deleting
I understand the concept on a new file creation, the problem, at least from my perspective is that the error that is occurring is nothing more than a conflict when attempting to gain access to write back to the file, so in that case nothing about the file has changed but yet the original file is potentially getting removed from the system.
Now from the perspective of a remote file system view for backups this makes tons of sense. In an interactive view of a file system this is incredibly dangerous.
On the open then save issue
The file is opened and kept open with read only locking so it then is failing when the attempt is made to open the file for writing by what is essentially another process,the assumption by the file system and the application is that the same "process" will be the ones actually making the writes to the file system.
In the case of this driver being a proxy in the middle that is no longer the case and that is where a majority of the file issues are coming from in the dokan projects that are actually proxing file system access.
from dokancloudfs.
On the file deleting
I understand the concept on a new file creation, the problem, at least from my perspective is that the error that is occurring is nothing more than a conflict when attempting to gain access to write back to the file, so in that case nothing about the file has changed but yet the original file is potentially getting removed from the system.
I totally agree with your view. What I'd like to do is identify better those situations when "nothing about the file has changed" so there is no danger of the file having gotten corrupted by a partial write. I started checking for UnauthorizedAccessException
because in this case I can be fairly certain that the target file remains unchanged. The problem arises with the unspecific IOException
that is thrown in the file locking case. The same Exception could also signal an aborted network file transfer. At least this is what I think will happen - please correct my if I'm wrong.
On the open then save issue
...
You sound like you have run into some painful issues with Dokan in the past :-(.
I'm basically aware of the implications behind read/write file access contention. However, I have so far failed in tracking down the actual reason for this contention to occur in DokanCloudFS. Whenever I break into the debugger in CloudOperations.WriteFile()
the read-Stream
has already been disposed and the write succeeds. It is only when letting the driver run free that the Exception is thrown.
Annoying case of Heisenbug - maybe related to a threading issue.
from dokancloudfs.
Also, I have so far only observed the contention for the local FILE
gateway in the unusual case of turned-off encryption.
This is hardly a primary use case for DokanCloudFS.
from dokancloudfs.
I thought about writing a file manager that was responsible for all of the actual io that gave out handles to to the processes as they were accessing the files, the theory is that the same process doesn't have to abide by the locking even if it's in conflict so it never really comes up for these apps when they are working with the disk directly.
So no file close until there are zero handles and everything else would be virtual.
-Adam
Sent from my iPhone
On May 13, 2016, at 4:04 PM, viciousviper [email protected] wrote:
On the file deleting
I understand the concept on a new file creation, the problem, at least from my perspective is that the error that is occurring is nothing more than a conflict when attempting to gain access to write back to the file, so in that case nothing about the file has changed but yet the original file is potentially getting removed from the system.I totally agree with your view. What I'd like to do is identify better those situations when "nothing about the file has changed" so there is no danger of the file having gotten corrupted by a partial write. I started checking for UnauthorizedAccessException because in this case I can be fairly certain that the target file remains unchanged. The problem arises with the unspecific IOException that is thrown in the file locking case. The same Exception could also signal an aborted network file transfer. At least this is what I think will happen - please correct my if I'm wrong.
On the open then save issue
...You sound like you have run into some painful issues with Dokan in the past :-(.
I'm basically aware of the implications behind read/write file access contention. However, I have so far failed in tracking down the actual reason for this contention to occur in DokanCloudFS. Whenever I break into the debugger in CloudOperations.WriteFile() the read-Stream has already been disposed and the write succeeds. It is only when letting the driver run free that the Exception is thrown.
Annoying case of Heisenbug - maybe related to a threading issue.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
from dokancloudfs.
Understood, it's a fairly unique use case.
Sent from my iPhone
On May 13, 2016, at 4:12 PM, viciousviper [email protected] wrote:
Also, I have so far only observed the contention for the local FILE gateway in the unusual case of turned-off encryption.
This is hardly a primary use case for DokanCloudFS.—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
from dokancloudfs.
Nevertheless I may have found an ugly workaround - at least it worked for me in about 10 consecutive manual tests.
Please try the above two and below one commits (FileGateway must be updated, too).
from dokancloudfs.
Hi @adamrogas,
have you had a chance to check if the latest changes to CloudFS - or version 1.0.6-alpha for that matter - fixed your situation?
from dokancloudfs.
It seems that the problem is still present while saving the MS Office files (.docx file opened - trying to save changes, two temporary files created by MS Word) - tested with Box.
There is no problem with classic Text files.
Exception occurs in Read method of ScatterStream class (during the SetAllocationSize dokan operation).
from dokancloudfs.
Hi @drunkenbeaver,
made some headway with the MS Office file handling issue.
I now have an internal version working that is able to open, edit and save (under a different name) a .docx file in MS Word 2016.
What I couldn't work out is in-place saving of a modified .docx file from within MS Word. The DokanCloudFS console trace looks to be without errors but still Word complains about some missing file system rights or other on the original file. I don't see any unsupported calls (like setting a readonly file attribute) nor are any Exceptions raised or any non-success DokanResult
s returned.
I'm inclined to leave it at that for the time being and rather bundle up the current state of affairs into a new beta release.
Maybe someone with deeper knowledge of the inner workings of MS Office file operations can help out: What is it that MS Word expects the file system to do after it saves (and apparently verifies by re-reading) a temporary ~WRD0000.tmp
file?
from dokancloudfs.
Hi there! First of all, I'm really happy that you still actively maintain this code base and I look forward to switch to develop branch to check out the latest progress on this and relevant issues.
This is what I've found MS is doing while Opening, Saving or Closing it's documents (.docx eg.) where Word.docx is a sample file I'm working with:
Open Word.docx
Create ~$Word.docx
Save Word.docx
Create ~WRD0000.tmp
Rename Word.docx > ~WRL00001.tmp
Rename ~WRD00000.tmp > Word.docx
Delete ~WRL00001.tmp
Close Word.docx
Delete ~$Word.docx
You're probably familiar with this or similar scenario.
It's quite difficult to track what's happening here, especially if at the endpoint you're trying to version these documents (ofc, original Word.docx is actually the only file that's really of interest to the users).
from dokancloudfs.
Related Issues (20)
- Add Baidu Cloud Support HOT 10
- Openstack swift HOT 2
- nuget packages fail HOT 6
- Mounter runtime problem HOT 16
- Protect sensitive credentials and keys using Windows crypto API HOT 2
- Wasted an entire 16 hours trying to use this library HOT 2
- No gateway is registered for schema HOT 5
- build error referencing wrong version HOT 1
- Mounting Google storage bucket or folder HOT 5
- Paste or drop file action [question] HOT 5
- Add Mail.ru Cloud Support HOT 4
- Support for Java. HOT 5
- google drive not working HOT 6
- Yandex causing error HOT 1
- Webdav mount problem HOT 4
- Huge memory consumption during opening big files in Google Drive HOT 1
- Can this be used with directory mounts? HOT 8
- Google Drive ArgumentException: Trying to set a IgorSoft.CloudFS.Interface.RootName
- new content not syncing down from box
- Why true random access is not allowed
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dokancloudfs.