NO_NEXT_RESC_FOUND when uploading data object using PRC if resource has minimum free space set

[stsnel]

NO_NEXT_RESC_FOUND when uploading data object using PRC if resource has minimum free space set

[alanking]

This sounds very related to #7154 and irods/python-irodsclient#462.

Have you tried removing the storage resource(s) from the resource hierarchy and attempting to write data directly to them? I'm guessing that you will get a different error (namely, USER_FILE_TOO_LARGE).

[trel]

Very good - let us know how it goes.

Marking this as duplicate for now.

[trel]

also... related to #3841 ?

[stsnel]

I considered #3841 before submitting this issue, but I thought this was different, because I couldn't reproduce the issue we're seeing on our environments using iput. This issue is also not intermittent.

Creating a data object directly on a unixfilesystem resource that has free_space and minimum_free_space_for_create_in_bytes configured results in a irods.exception.USER_FILE_TOO_LARGE when using PRC. It looks like this is indeed a symptom of #7154 and irods/python-irodsclient#462 .

Thank you for the information 👍 Knowing the root cause, I think we will be able to work around it.

[trel]

Since #7154 is closed - closing this issue as well.

The workaround for using Python is to set the filesize to 0 when talking to a pre-4.3.1 iRODS server.

[stsnel]

Information for other people who encounter the same issue:

We ended up using the following solution for the time being: https://utrechtuniversity.github.io/yoda/design/processes/automatic-resource-balancing.html

It basically involves keeping track of which resources are (almost) full in a Redis key-value store. Our pep_resource_resolve_hierarchy_pre checks the resource information in Redis when a new data object is created, and changes votes so that new data objects get created on resources that have enough free space.

[kript]

Apologies for commenting on a closed Issue, but I just came across this and would it be possible to get some further clarification please?

I think that this affects all versions of iRODS before 4.3.1 but was introduced at least after 4.2.7 (since I've been able to use the PRC on a 4.2.7 system but not tried the most recent PRC version). It would be really good to know;

1. what version this got introduced in.
2. If the version of the PRC is related (i.e. can you use an earlier one? I think not but...)
3. If other API's (e.g. C) are affected.

@trel does mention that

The workaround for using Python is to set the filesize to 0 when talking to a pre-4.3.1 iRODS server.

However, the documentation for the PRC doesn't mention this as of a few minutes ago, and its not clear to me what affect setting the file size will have on other operations that might depend on it (e.g. it might try and create a file on a resource without sufficient space, or fail to recognise a partial upload?). Could we get an update to the PRC docs to mention this workaround please (especially if its only affecting the PRC, but even if it affects the other API's too).

Thanks!

[alanking]

1. what version this got introduced in.

I haven't taken the time to track this down, but I can almost guarantee that 4.2.9 is the first version in which this is an issue.

2. If the version of the PRC is related (i.e. can you use an earlier one? I think not but...)

I think the PRC has been leaning on the -1 dataSize behavior for a long time.

The inappropriate type choice for the problematic member variable has been in iRODS for a long time and has only "recently" become an issue because the usage of the member variable is leaning on the signed-ness which had been assumed but never actually real.

3. If other API's (e.g. C) are affected.

All APIs - regardless of language - are affected by this because the problem is in a type used by the voting machinery deep in the server. We added a test using the C client API call and demonstrated the same problem. The reason this doesn't manifest in other clients is because the dataSize is not being set to -1 by the iput, istream, etc. machinery.

However, the documentation for the PRC doesn't mention this as of a few minutes ago, and its not clear to me what affect setting the file size will have on other operations that might depend on it (e.g. it might try and create a file on a resource without sufficient space, or fail to recognise a partial upload?).

Setting the file size to 0 is equivalent to setting it to -1 in a streaming situation. The server only cares that the dataSize is a non-positive value, so 0 or -1 should result in the same behavior.

As for the insufficient space question, streaming data to the resource would result in the same issue regardless of whether -1 or 0 is used. Streaming inherently doesn't know how much data will be uploaded. If the size of the data is known, a put should be used instead of just opening the object. ...However, this also would not solve the issue as specifying the dataSize via put is not supported yet :( Please see irods/python-irodsclient#462

Could we get an update to the PRC docs to mention this workaround please (especially if its only affecting the PRC, but even if it affects the other API's too).

We could add a Troubleshooting section to the PRC documentation for this, sure. I've created an issue for this here: irods/python-irodsclient#481

Hope that helps...