Comments (8)
The reason we treat that case as a warning is because it doesn't signify that the implementation doesn't support the profile; it interferes with our testing since we can't get a resource, but we can move on with other testing. The scope of the interop validator is to verify requirements spelled out in the profile, and failures are constituted based on how the service does not meet the profile. Based on the profile specification, you either meet the profile or you don't; the profile syntax doesn't really have a "warning" end result.
from redfish-interop-validator.
Right, I understand a bit better. But then, in the output, there is not much difference between a "Supported" failing test (showing PASS
) and a "Mandatory" PASS
, rendering the "Supported" useless unless manual inspection of all the results (and for clarity inside the profile). Maybe adding an "Unsupported" status, then?
Same for "IfImplemented" but here you absolutely accept that the property might not be implemented and if it is, you can check values so a specific type of output might not be necessary.
from redfish-interop-validator.
Both of those cases are still requirements in terms of PASS/FAIL criteria and don't give leeway for warnings.
"Supported" indicates the property is required, and as long as it's on at least one resource instance, The usage of this came about because a system might not be fully populated, and so resources can show up with a state of "Absent", and thus have a very short list of properties. You don't want to fail service for not surfacing a property that isn't relevant to the system configuration, but still need to test that the property is present for cases where the configuration is expected to have it, such as when only half of your DIMMs are populated.
"IfImplemented" recognizes the fact that not all systems are designed the same and have underlying hardware differences. You don't want to use a Redfish profile to define hardware requirements; the scope is to test the manageability exposed by the BMC. For example, a user might have requirements to collect fan information in a consistent way, but not every hardware vendor would necessarily have fans as part of the system design. It would be in bad practice to require a BMC to report fan resources if none exist, so using "IfImplemented" in this case would make sense. However, we do have a gap in our testing since we don't have a way to inspect these types of requirements to know if the system under test is expected to have that functionality. At this time, we just treat "IfImplemented" as "Recommended", meaning if we don't find the property or resource, that's considered a PASS. We do have an open issue about this gap (#105).
from redfish-interop-validator.
Fair enough, I apparently didn't grasp the concept of those before. But then there is no workaround to indicate something "optional".
And for "Recomended", what is the use case? Simply allow the tool to check without failing or logging anything? Wouldn't it make sense to output differently?
from redfish-interop-validator.
"Recommended" is optional, but the author of the profile leans towards "I'd rather see this implemented than not".
Everything that's not mentioned in the profile is truly optional; the profile really is meant to document requirements that a user cannot live without.
Profiles are also not meant to restrict the capabilities of a service. So, a profile cannot forbid a service from implementing something, meaning that they have the flexibility to implement the entire data model, even if the profile's user has no use case for it.
from redfish-interop-validator.
In this case, I think it would still make sense to improve the output to differentiate a "Recommended" that is not present in the response. Because then, it's a green and a "PASS" whether it is present or not. It should not generate an error, but the operator would still want to know at a glance whether it was present without necessarily having to check manually the value or "DNE" entry.
from redfish-interop-validator.
@Arno500 you might want to take a look at the Profile output mode of the Redfish doc-generator https://github.com/DMTF/Redfish-Tools/tree/main/doc-generator
Given a profile document, that tool will produce a "schema guide" with each of the requirements pulled from the profile. That may give you more context for comparing an implementation to the profile.
But I do see your point that the tool doesn't provide a "wishlist" of support for all the non-required items shown in the profile. I think we should consider some support for that - but it would be a different mode of operation, as that output would be geared towards an implementer. The goal of the tool is to VALIDATE an implementation against a profile. But using it to create a "to do" list for an implementation is certainly a useful function.
from redfish-interop-validator.
Thank you @jautor for your input. This is indeed a very interesting tool that you have here, but I am on the "user" side, so it won't be of much help in my case :)
What I had more in mind was to give a shopping list to OEMs, so they can implement the features we need. We could have two different profiles (a minimal profile where it's only required features and a wishlist) as well. But I really liked the idea of something differentiating between "WE NEED THAT" and "well, you apparently don't have this one, but you know, that would be really awesome and we are gearing towards that in the future, know that it could impact our capacity to buy your servers".
And all of this in one clear output with fails and wished (at least this is my vision). So maybe it doesn't need another mode of operation but it would be perfectly fitting for this use case.
from redfish-interop-validator.
Related Issues (20)
- About Redfish Interoperability Profile Specification v1.6.0 HOT 2
- Chassis resource fail errors HOT 2
- Use `@Redfish.WriteableProperties` to valiate `WriteRequirement` HOT 2
- JSON Pointer syntax works for 'CompareProperty' but not 'UseCaseKeyProperty' HOT 3
- Value Comparison is not checked in ConditionalRequirements HOT 3
- Need ability to parse redfish URI trees in the profiles HOT 2
- Need ability to parse wildcards HOT 2
- Physicalcontext comparison error HOT 11
- --schema and --required_profiles_dir argument usage confirm HOT 2
- False positive results when "Recommended" or "IfImplemented" object does not exist. HOT 1
- Validation of Read/Write properties - false positive results.
- "Comparison" tests can show invalid failure if a property is not present
- Run scripts will pop "not enough values to unpack"
- Response time values are either 0 or missing
- Resource-level IfPopulated requirements get treated as Manadatory
- Interop validator is returning error for the write requirement property HOT 1
- Interop Validator is throwing error for write requirement for Readonly URI HOT 2
- Implement the RequiredResourceProfile schema property
- config.pu prints password in clear text.....
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from redfish-interop-validator.