Windows: error shown with symlinked files

[fcorbelli]

Windows: error shown with symlinked files

[fcorbelli]

Thank you for your recommendation.
Could you please do an actual example, even just for a single symlinked file?

"38385: WARN expected 0 getted for ".

should be a little different

38385: WARN expected 0 getted 34 for thepippo.txt

It is indeed important to understand in which portion of the program (I think I understand it, but not for sure) the difference is generated
In this example I made a symlink to a folder...

Z:\>dir
 Il volume nell'unità Z è RamDisk
 Numero di serie del volume: 8A05-AD69

 Directory di Z:\

27/04/2022  14:35    <DIR>          nza_copy
               0 File              0 byte
               1 Directory  84.879.110.144 byte disponibili

Z:\>mklink collegamento2 z:\nza_copy /d
collegamento simbolico creato per collegamento2 <<===>> z:\nza_copy

Z:\>dir
 Il volume nell'unità Z è RamDisk
 Numero di serie del volume: 8A05-AD69

 Directory di Z:\

27/04/2022  14:45    <SYMLINKD>     collegamento2 [z:\nza_copy]
27/04/2022  14:35    <DIR>          nza_copy
               0 File              0 byte
               2 Directory  84.879.110.144 byte disponibili

and it works OK

Z:\>zpaqfranz a 1 z:\collegamento2\*
zpaqfranz v54.12-experimental (HW BLAKE3), SFX64 v52.15, compiled Mar 20 2022
Integrity check type: XXHASH64+CRC-32 + CRC-32 by fragments
Creating 1.zpaq at offset 0 + 0
Adding 909.162.975 (867.04 MB) in 1.558 files  at 2022-04-27 14:46:55
Non-latin (UTF-8)             2
 82.09% 00:00:00 ( 711.73 MB) -> ( 342.98 MB) of ( 867.04 MB)  177.93 MB/sec
1.786 +added, 0 -removed.

0 + (909.162.975 -> 826.472.213 -> 437.368.167) = 437.368.167
Forced XLS/PPT has included 17.408 bytes in 1 files

6.078 seconds (000:00:06)  (all OK)

Maybe you use FILE symlinks instead of folder?
Can you please post a run with the -debug switch?
Maybe you get this ARCHIVE;REPARSE_POINT;

Can I prepare, in case, a specific debug version, so that you can test it, by putting it here (Windows 64 bit)?

[houbit]

Sorry my text was incomplete, used angle brackets without thinkin:

NTFS with symbolic link on a FILE:

"38385: WARN expected 0 getted for N".

Further on this results in
"38721: HOUSTON something seems wrong: expected M, done M+N"
"..."
"... almost certainly incomplete"
(that is definetly wrong, the archive is fine)`

Seems like the symlinked file was read (N bytes) but counted as 0 bytes.

Full example, zpaqfranz unpacked, symlink notes.txt.symlnk -> notes.txt:
'dir' =>
Verzeichnis von A:_zbackup\zpaq_test 27.04.2022 14:51

.
27.04.2022 14:51 ..
20.03.2022 18:41 293.202 blake3_windows_gnu.S
20.03.2022 18:41 5.677 CHANGELOG.md
20.03.2022 18:41 15.391 differences715.txt
20.03.2022 18:41 470 Makefile
20.03.2022 18:41 43.609 notes.txt
27.04.2022 14:51 notes.txt.symlnk [A:_zbackup\zpaq_test\notes.txt]
20.03.2022 18:41 27.754 README.md
27.04.2022 14:51 1.065.373 x.zpaq
20.03.2022 18:41 209.486 zpaq206.pdf
20.03.2022 18:41 1.841.725 zpaqfranz.cpp
10 Datei(en), 3.502.687 Bytes
2 Verzeichnis(se), xxxxxxxx Bytes frei

"zpaq a test *" (sorry, renamed it to zpaq for my own convinience only) =>
A:_zbackup\zpaq_test>zpaq a test *
zpaqfranz v54.12-experimental (HW BLAKE3), SFX64 v52.15, compiled Mar 20 2022
Integrity check type: XXHASH64+CRC-32 + CRC-32 by fragments
Creating test.zpaq at offset 0 + 0
Adding 2.437.314 (2.32 MB) in 9 files at 2022-04-27 15:07:40
38385: WARN expected 0 getted 43.609 for notes.txt.symlnk 9 +added, 0 -removed. 0 + (2.437.314 -> 2.437.314 -> 1.065.373) = 1.065.373 38271: HOUSTON something seems wrong: expected 2.437.314, done 2.480.923
38271: Corrupted source files? Lost connection? Cannot access? Media full?
38271: =>The updated .zpaq archive is almost certainly incompleted 0.234 seconds (00:00:00) (with errors)

[fcorbelli]

Sorry my text was incomplete, used angle brackets without thinkin:

NTFS with symbolic link on a FILE:

OK, this make sense (I never use symlink to file, only to folders :-)

"38385: WARN expected 0 getted for N".

Further on this results in "38721: HOUSTON something seems wrong: expected M, done M+N" "..." "... almost certainly incomplete" (that is definetly wrong, the archive is fine)`

Seems like the symlinked file was read (N bytes) but counted as 0 bytes.

Exactly. The file size is taken the first time from the filesystem (and in this function, scandir (), will need to be corrected) and a second time after the block reading (to calculate the hash). In a normal situation the two data coincide. The foo.txt file was 1000 bytes BEFORE and will be 1000 AFTER

However, this does not always happen, as "someone" may change a file while the compression is in progress (can takes time, ever hours to complete)
So the file pippo.txt FIRST could be 1000 bytes large (... someone changes it ...) and THEN become let's say 1001. In this case the warning is shown

As mentioned, I would need, please, a run with -debug to understand if my interpretation is correct.
Thanks

[houbit]

Same constallations,
"zpaq a test * -debug | tee test.log"

test.log:
`
zpaqfranz v54.12-experimental (HW BLAKE3), SFX64 v52.15, compiled Mar 20 2022
franz:DEBUG very verbose (-debug)
21026: check_if_password of test.zpaq
Integrity check type: XXHASH64+CRC-32 + CRC-32 by fragments
Creating test.zpaq at offset 0 + 0
00000010 MY new t .
DIRECTORY;

00000010 MY new t ..
DIRECTORY;

00000020 MY new t blake3_windows_gnu.S
ARCHIVE;

00000020 MY new t CHANGELOG.md
ARCHIVE;

00000020 MY new t differences715.txt
ARCHIVE;

00000020 MY new t Makefile
ARCHIVE;

00000020 MY new t notes.txt
ARCHIVE;

00000420 MY new t notes.txt.symlnk
ARCHIVE;REPARSE_POINT;

00000020 MY new t README.md
ARCHIVE;

00000020 MY new t zpaq206.pdf
ARCHIVE;

00000020 MY new t zpaqfranz.cpp
ARCHIVE;

Default sort
Adding 2.437.314 (2.32 MB) in 9 files 20006: converting to localtime 2022-04-27 13:31:56
20028: localtime is 2022-04-27 15:31:56
at 2022-04-27 15:31:56
Model2: XXHASH64 2CB19869E0642796 CHANGELOG.md
Mode2: XXHASH64: <<2CB19869E0642796>> CRC32 <> CHANGELOG.md
Model2: XXHASH64 5046DFD533DC8DA5 Makefile
Mode2: XXHASH64: <<5046DFD533DC8DA5>> CRC32 <> Makefile
Model2: XXHASH64 51D9BB16C4688B8C README.md
Mode2: XXHASH64: <<51D9BB16C4688B8C>> CRC32 <> README.md
Model2: XXHASH64 AB2D326F6ADC598D blake3_windows_gnu.S
Mode2: XXHASH64: <> CRC32 <<15956CE2>> blake3_windows_gnu.S
Model2: XXHASH64 D27913528DDE6FA4 differences715.txt
Mode2: XXHASH64: <> CRC32 <<1030B35B>> differences715.txt
Model2: XXHASH64 C498B2481EA2C766 notes.txt
Mode2: XXHASH64: <> CRC32 <<010AA2D6>> notes.txt
Model2: XXHASH64 C498B2481EA2C766 notes.txt.symlnk
38385: WARN expected 0 getted 43.609 for notes.txt.symlnk
Mode2: XXHASH64: <> CRC32 <<010AA2D6>> notes.txt.symlnk
Model2: XXHASH64 EA4527C120B47032 zpaq206.pdf
Mode2: XXHASH64: <> CRC32 <<26121D8D>> zpaq206.pdf
Model2: XXHASH64 8DCD3E429C38D817 zpaqfranz.cpp
Mode2: XXHASH64: <<8DCD3E429C38D817>> CRC32 <<4ACAE94D>> zpaqfranz.cpp

9 +added, 0 -removed.

38271: HOUSTON something seems wrong: expected 2.437.314, done 2.480.923
38271: Corrupted source files? Lost connection? Cannot access? Media full?
38271: =>The updated .zpaq archive is almost certainly incompleted
33026: call xcommand on errorcode==2
Entering xcommand()
i_command ||
i_parameter |38271: HOUSTON something seems wrong expected vs done|
Exit because empty i_command
`
If you need something from a 'special/prepared' version of zpaqfranz let me know.

Thank you

[fcorbelli]

54_13j.zip
Can you please check the attached pre-release, with and without -debug?

Thanks

[houbit]

I am VERY sorry about the delay, was in really bad health troubles.

..\54_13j a test * | tee ..\54_13j.log
..\54_13j a test * -debug | tee ..\54_13j-debug.log

54_13j.log:
`zpaqfranz v54.13j-experimental (HW BLAKE3), SFX64 v52.15, compiled Apr 28 2022
Integrity check type: XXHASH64+CRC-32 + CRC-32 by fragments
Creating test.zpaq at offset 0 + 0
Adding 2.437.314 (2.32 MB) in 9 files at 2022-06-13 11:47:12
38385: WARN expected 0 getted 43.609 for notes.txt.symlnk

9 +added, 0 -removed.

38271: HOUSTON something seems wrong: expected 2.437.314, done 2.480.923
38271: Corrupted source files? Lost connection? Cannot access? Media full?
38271: =>The updated .zpaq archive is almost certainly incompleted
54_13j-debug.logzpaqfranz v54.13j-experimental (HW BLAKE3), SFX64 v52.15, compiled Apr 28 2022
franz:DEBUG very verbose (-debug)
21026: check_if_password of test.zpaq
Integrity check type: XXHASH64+CRC-32 + CRC-32 by fragments
Creating test.zpaq at offset 0 + 0
00000010 MY new t .
DIRECTORY;

00000010 MY new t ..
DIRECTORY;

00000020 MY new t blake3_windows_gnu.S
ARCHIVE;

00000020 MY new t CHANGELOG.md
ARCHIVE;

00000020 MY new t differences715.txt
ARCHIVE;

00000020 MY new t Makefile
ARCHIVE;

00000020 MY new t notes.txt
ARCHIVE;

00000420 MY new t notes.txt.symlnk
ARCHIVE;REPARSE_POINT;

29300: maybe Windows-symlinked file, fake size 0 real size 0 *
00000020 MY new t README.md
ARCHIVE;

00000020 MY new t zpaq206.pdf
ARCHIVE;

00000020 MY new t zpaqfranz.cpp
ARCHIVE;

Default sort
Adding 2.437.314 (2.32 MB) in 9 files 20006: converting to localtime 2022-06-13 09:50:50
20028: localtime is 2022-06-13 11:50:50
at 2022-06-13 11:50:50
Model2: XXHASH64 2CB19869E0642796 CHANGELOG.md
Mode2: XXHASH64: <<2CB19869E0642796>> CRC32 <> CHANGELOG.md
Model2: XXHASH64 5046DFD533DC8DA5 Makefile
Mode2: XXHASH64: <<5046DFD533DC8DA5>> CRC32 <> Makefile
Model2: XXHASH64 51D9BB16C4688B8C README.md
Mode2: XXHASH64: <<51D9BB16C4688B8C>> CRC32 <> README.md
Model2: XXHASH64 AB2D326F6ADC598D blake3_windows_gnu.S
Mode2: XXHASH64: <> CRC32 <<15956CE2>> blake3_windows_gnu.S
Model2: XXHASH64 D27913528DDE6FA4 differences715.txt
Mode2: XXHASH64: <> CRC32 <<1030B35B>> differences715.txt
Model2: XXHASH64 C498B2481EA2C766 notes.txt
Mode2: XXHASH64: <> CRC32 <<010AA2D6>> notes.txt
Model2: XXHASH64 C498B2481EA2C766 notes.txt.symlnk
38385: WARN expected 0 getted 43.609 for notes.txt.symlnk
Mode2: XXHASH64: <> CRC32 <<010AA2D6>> notes.txt.symlnk
Model2: XXHASH64 EA4527C120B47032 zpaq206.pdf
Mode2: XXHASH64: <> CRC32 <<26121D8D>> zpaq206.pdf
Model2: XXHASH64 8DCD3E429C38D817 zpaqfranz.cpp
Mode2: XXHASH64: <<8DCD3E429C38D817>> CRC32 <<4ACAE94D>> zpaqfranz.cpp

9 +added, 0 -removed.

38271: HOUSTON something seems wrong: expected 2.437.314, done 2.480.923
38271: Corrupted source files? Lost connection? Cannot access? Media full?
38271: =>The updated .zpaq archive is almost certainly incompleted
33026: call xcommand on errorcode==2
Entering xcommand()
i_command ||
i_parameter |38271: HOUSTON something seems wrong expected vs done|
Exit because empty i_command
`

[fcorbelli]

I hope you health is OK

Model2: XXHASH64 C498B2481EA2C766 notes.txt.symlnk
38385: WARN expected 0 getted 43.609 for notes.txt.symlnk
(...)
38271: HOUSTON something seems wrong: expected 2.437.314, done 2.480.923

How do you have created
notes.txt.symlnk?

Thank you
PS the latest pre-release is this one

54_16e.zip

[houbit]

Thank You, I am fine now.

Just made a batch to create and run a test case:

• create directories ./node ./node/sub ./bro, each with a sample.txt and symlinks from ./node to the three sample.txt files (just in case, there may be differences when adding from other targets then the actual dir).
• run zpaqfranz with / without '-debug' on ./node
• zip the batch, the used executable and all the log and results to zpaqfranz_test.zip

! The symlinks are created by Windows own mklink.com.
! Sadly you need administrator privileges to use this and also for the batch.

here the results:
zpaqfranz_test.zip

Anycase I hope this helps.
Kind regards

[fcorbelli]

Please test the current 55.4 release
Thank you