Comments (18)
We’ll need to check whether this happens in 4.3.1.
from irods.
issue still exists for 4.3.1 (on a vanilla iRODS server, just created)
irods@ubuntu2004sudo:~/ton$ ls -l
total 8
-rw-rw-r-- 1 irods irods 13 Apr 29 15:33 testfile
-rw-rw-r-- 1 irods irods 5 Apr 29 15:34 testfile.shorter
irods@ubuntu2004sudo:~/ton$ iput testfile
irods@ubuntu2004sudo:~/ton$ ichksum testfile
testfile sha2:68chYPpvzc51Hl1ymMjfWD5hMe7BYfr1IJQVngXG01A=
irods@ubuntu2004sudo:~/ton$ ils -L testfile
rods 0 demoResc 13 2024-04-29.15:34 & testfile
sha2:68chYPpvzc51Hl1ymMjfWD5hMe7BYfr1IJQVngXG01A= generic /var/lib/irods/Vault/home/rods/testfile
irods@ubuntu2004sudo:~/ton$ cat testfile.shorter >/var/lib/irods/Vault/home/rods/testfile
irods@ubuntu2004sudo:~/ton$ ichksum testfile
testfile sha2:68chYPpvzc51Hl1ymMjfWD5hMe7BYfr1IJQVngXG01A=
irods@ubuntu2004sudo:~/ton$ ichksum -f testfile
remote addresses: 127.0.0.1 ERROR: chksumDataObjUtil: rcDataObjChksum error for /tempZone/home/rods/testfile status = -512000 UNIX_FILE_READ_ERR
remote addresses: 127.0.0.1 ERROR: chksumUtil: chksum error for /tempZone/home/rods/testfile, status = -512000 status = -512000 UNIX_FILE_READ_ERR
irods@ubuntu2004sudo:~/ton$ ienv
irods_version - 4.3.1
from irods.
Well, there we go. Bug!
from irods.
Further analysis reveals that the recalculation of the checksum is performed only for the initial bytes of the data file upto the filesize as registered with the replica. Hence for (independently) enlarged data files, iRODS will calculate and register an incorrect checksum. See example below where the recalculated checksum should but does not differ from the previous checksum:
iput testfile file
irods@ubuntu2004sudo:~/ton$ ils -L
/tempZone/home/rods:
rods 0 demoResc 13 2024-04-29.17:07 & file
generic /var/lib/irods/Vault/home/rods/file
irods@ubuntu2004sudo:~/ton$ ls -l /var/lib/irods/Vault/home/rods/file
-rw------- 1 irods irods 13 Apr 29 17:07 /var/lib/irods/Vault/home/rods/file
irods@ubuntu2004sudo:~/ton$ ichksum file
file sha2:68chYPpvzc51Hl1ymMjfWD5hMe7BYfr1IJQVngXG01A=
irods@ubuntu2004sudo:~/ton$ ils -L
/tempZone/home/rods:
rods 0 demoResc 13 2024-04-29.17:07 & file
sha2:68chYPpvzc51Hl1ymMjfWD5hMe7BYfr1IJQVngXG01A= generic /var/lib/irods/Vault/home/rods/file
irods@ubuntu2004sudo:~/ton$ cat testfile testfile >/var/lib/irods/Vault/home/rods/file
irods@ubuntu2004sudo:~/ton$ ls -l /var/lib/irods/Vault/home/rods/file
-rw------- 1 irods irods 26 Apr 29 17:09 /var/lib/irods/Vault/home/rods/file
irods@ubuntu2004sudo:~/ton$ ichksum -f file
file sha2:68chYPpvzc51Hl1ymMjfWD5hMe7BYfr1IJQVngXG01A=
irods@ubuntu2004sudo:~/ton$ ils -L
/tempZone/home/rods:
rods 0 demoResc 13 2024-04-29.17:07 & file
sha2:68chYPpvzc51Hl1ymMjfWD5hMe7BYfr1IJQVngXG01A= generic /var/lib/irods/Vault/home/rods/file
irods@ubuntu2004sudo:~/ton$
Now the good news is that a likely quick fix could be to first update the replica's filesize attribute using data file stat info, then proceed to calculate the checksum.
from irods.
Agreed - this makes the error (and fix) more consistent. Thanks.
from irods.
If we update the size of the size of the replica in the catalog, I'm wondering what it means about any sibling replicas which may still reflect the information in the catalog. Should the untouched sibling replicas be considered stale? Or should the replica found to be different from what is recorded be considered stale?
from irods.
I think with an ichksum -f
, perhaps we just stat()
the replica first, then read that number of bytes? Ignore the value in the catalog?
Oh... wait... the catalog would still have the wrong size information - so it's not for ichksum
to update that field in the catalog? Or is it?
What is the right way to tell the catalog to update its filesize information for a registered replica?
from irods.
Oh, I see... We are already updating the checksum (via -f
?) and so updating the size while we're at it wouldn't be much different. We can just do whatever it normally does if there's a difference.
from irods.
Right, that's the question. Would updating the size break any assumptions by other moving parts?
from irods.
Now updating attributes might cause an update of the replica's modify_ts right? Would that have any impact on decisions regarding good/stale status of the replica and its siblings?
from irods.
I don't think updating with ichksum -f
should update the status of the replica... if it was stale, it should remain stale... and if it was good, it should remain good...
(is that true? ichksum -f
of a stale replica of the correct size... only adds the checksum to the catalog... investigating...)
(answer: yes, see below)
Some testing with 4.3.1...
The file on disk for replica 1 is only 3 bytes:
$ ls -l /tmp/thechildvault/home/rods/another.txt
-rw------- 1 irods irods 3 Apr 29 14:14 /tmp/thechildvault/home/rods/another.txt
But the catalog shows 8 bytes:
irods$ ils -L another.txt
rods 0 demoResc 5 2024-04-29.14:08 & another.txt
sha2:I5YJnGwIT6S5vqyfDVLPO+nPjUcEDvEniD1TK1eQzXQ= generic /var/lib/irods/Vault/home/rods/another.txt
rods 1 thechild 8 2024-04-29.14:07 X another.txt
sha2:M6eyFQZfLuhjXvtyYgvCaaHvuIm6MCZWAzTac2Z0I3Q= generic /tmp/thechildvault/home/rods/another.txt
Checksum fails with UNIX_FILE_READ_ERR
:
irods$ ichksum -f -n1 another.txt
remote addresses: 127.0.0.1 ERROR: chksumDataObjUtil: rcDataObjChksum error for /tempZone/home/rods/another.txt status = -512000 UNIX_FILE_READ_ERR
remote addresses: 127.0.0.1 ERROR: chksumUtil: chksum error for /tempZone/home/rods/another.txt, status = -512000 status = -512000 UNIX_FILE_READ_ERR
And the logs in the server...
{"log_category":"legacy","log_level":"info","log_message":"[+]\t/irods_source/server/api/src/rsFileChksum.cpp:350:int file_checksum(RsComm *, const char *, const char *, const char *, const char *, rodsLong_t, char *) : status [Unknown iRODS error] errno [] -- message [file_checksum - fileRead failed for [/tmp/thechildvault/home/rods/another.txt].]\n\n","request_api_name":"DATA_OBJ_CHKSUM_AN","request_api_number":629,"request_api_version":"d","request_client_user":"rods","request_host":"127.0.0.1","request_proxy_user":"rods","request_release_version":"rods4.3.1","server_host":"xxx.xxx.xxx","server_pid":3831197,"server_timestamp":"2024-04-29T18:14:40.305Z","server_type":"agent","server_zone":"tempZone"}
{"log_category":"legacy","log_level":"error","log_message":"file_checksum - The size of the replica recorded in the catalog is greater than the size in storage.","request_api_name":"DATA_OBJ_CHKSUM_AN","request_api_number":629,"request_api_version":"d","request_client_user":"rods","request_host":"127.0.0.1","request_proxy_user":"rods","request_release_version":"rods4.3.1","server_host":"xxx.xxx.xxx","server_pid":3831197,"server_timestamp":"2024-04-29T18:14:40.305Z","server_type":"agent","server_zone":"tempZone"}
Most interesting is this line...
"log_message":"file_checksum - The size of the replica recorded in the catalog is greater than the size in storage."
Then, updating the catalog to be 'smaller' than the file in storage...
irods$ iadmin modrepl logical_path /tempZone/home/rods/another.txt replica_number 1 DATA_SIZE 1
The catalog now shows a 1-byte file for replica 1:
irods$ ils -L another.txt
rods 0 demoResc 5 2024-04-29.14:08 & another.txt
sha2:I5YJnGwIT6S5vqyfDVLPO+nPjUcEDvEniD1TK1eQzXQ= generic /var/lib/irods/Vault/home/rods/another.txt
rods 1 thechild 1 2024-04-29.14:07 X another.txt
sha2:M6eyFQZfLuhjXvtyYgvCaaHvuIm6MCZWAzTac2Z0I3Q= generic /tmp/thechildvault/home/rods/another.txt
Running ichksum -f
again, this time, succeeds...
irods$ ichksum -f -n1 another.txt
another.txt sha2:qqlAJmTxpB9A67xSyZk+tmrrNmYClY/fqig7ceZNsSM=
irods$ ils -L another.txt
rods 0 demoResc 5 2024-04-29.14:08 & another.txt
sha2:I5YJnGwIT6S5vqyfDVLPO+nPjUcEDvEniD1TK1eQzXQ= generic /var/lib/irods/Vault/home/rods/another.txt
rods 1 thechild 1 2024-04-29.14:07 X another.txt
sha2:qqlAJmTxpB9A67xSyZk+tmrrNmYClY/fqig7ceZNsSM= generic /tmp/thechildvault/home/rods/another.txt
So... maybe nothing needs to happen?
The server reports when it cannot read the entirety of the file (because the file on disk is larger than the size in the catalog).
from irods.
What is the right way to tell the catalog to update its filesize information for a registered replica?
If something is touching the vault, and that's your design goal... then you have to manage the expectations that the filesizes need to be updated as well.
You can do that with iadmin modrepl
OR with a trim and re-registration of that replica.
I'm now thinking ichksum
should not be in the business of updating filesizes in the catalog.
from irods.
According to the iRODS design [Rajasekar et al. 2006], the dirty bit (hence stale/good status) facilitates synchronization of replicas after changing one of the copies. This implies that status changes applied to the replicas of a data object should be linked to requests that write new data to a particular replica. This would suggest that changes to replica state stale/good should not be linked to checksum verify/update actions.
In the case where there is a discrepancy between ICAT held replica attributes and data file attributes, we typically expect a media failure or equivalent. Checksum verification allows us to find such discrepancies, where we regard the ICAT the trusted source.
As an exception, a deliberate user action to update the ICAT with a checksum calculated from a data file is an attempt to ensure that the replica attributes truthfully reflect the attributes of the data file.
In this light, one could argue the the modify_ts of a replica should reflect the last-modified timestamp of the datafile, and not change due to metadazta operations on checksum attributes (or file size attributes).
from irods.
On the experiment changing file size in the ICAT: this would indeed to a modified checksum as less bytes are involved in the calculation. The resulting checksum however does not properly reflect the content of the entire data file, even though 'optically' it seems okay.
from irods.
I agree that touching a checksum should not affect the modify_ts of the replica. And I think the current behavior matches that expectation.
The checksum matches the content contained within the filesize-in-the-catalog. I think this is also correct and good.
If the ichksum -f
is not allowed to 'fix' the filesize (which I currently believe it should not), then everything seems to be behaving as expected.
Your original posting above has...
Expected behavior
Replica checksum attribute in ICAT is updated with calculated checksum of related data file.
a) Do you agree we cannot / should not do update the checksum without also updating the filesize in the catalog?
b) Do you agree that we should not touch the filesize in the catalog with this operation?
And if yes to both... then there is nothing to do here?
from irods.
Indeed the current behavior seems to be consistent after all. iRODS manages only the data contained in a part of the data file bounded by the size attribute of the replica. It makes sense when one regards the data file as an arbitrary sized container. I tested other client tools such as iget and istream and they respect this boundary as well, only the 'managed' part of a data file is downloaded.
Hence the conclusion must be that this is not a bug, it is intended behavior. Agree with points a and b. It might be worthwhile to add a warning in ichksum documentation and the related microservice to clarify that the checksum value only represents the part of the data file indicated by the replica size attribute, not the entire datafile.
from irods.
Excellent.
It might be worthwhile to add a warning in ichksum documentation and the related microservice to clarify that the checksum value only represents the part of the data file indicated by the replica size attribute, not the entire datafile.
We can do that. Marking this as a documentation issue.
from irods.
all done. closing.
from irods.
Related Issues (20)
- `KW_CFG_PLUGIN_TYPE_AUTHENTICATION` is "auth" but configuration JSON schema requires "authentication"
- increased agent allocated memory after client-server roundtrips
- Add `-t` option to `istream`
- irods server refuses to start due to schema issue in the hosts host_resolution.json for iRODS 4.3.2 HOT 4
- `irods_assert.hpp` declares `irods::assert()` but it is never defined
- add -j option to itree
- GenQuery2: Investigate replacement for `no distinct`
- Add plugin interface for new authentication plugin framework
- Investigate delay server migration and servers using identical hostnames HOT 1
- Expose configuration option for changing the size of the buffer used to calculate checksums HOT 8
- `iinit` shows "iRODS password" prompt for non-native auth schemes
- Deprecate parallel transfer over high ports
- Remove parallel transfer over high ports HOT 7
- irepl in backup mode reports wrong error HOT 4
- error running ichksum HOT 7
- Make `rsDataObjUnlink` warning less noisy HOT 3
- Do not allow strict ACLs to be disabled
- Grant public group read permissions on root and zone collections HOT 1
- add SHA512 as available checksum algorithm to ichksum and configuration files
- Absorb functionality of the Metadata Guard rule engine plugin into the server
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from irods.