Comments (6)
Does the actual schema contain any Open API links?
from schemathesis.
@Stranger6667 I don't think I understand the reason behind your question. The actual schema is quite complex and to be honest really irrelevant for the bug. I ran the code above in the description with the dummy schema and it produced the same result.
So for the reproducer it does not contain any links (or any real data) as the whole schema needed to hit the bug is just this:
{
"openapi": "3.0.2",
"info": {},
"paths": {}
}
from schemathesis.
The reason for the question is that stateful testing requires Open API links to be defined to run stateful tests:
To specify how different operations depend on each other, we use a special syntax from the Open API specification - Open API links
Perhaps it does not clearly state that running stateful tests requires specifying at least one connection between two API operations (though it is briefly mentioned in the How it works behind the scenes section).
Similarly, the error message is not tuned to reflect the API testing aspect. It comes from the underlying framework and merely reflects the implementation detail.
Other than that, the intention behind the error message is to show there is no point in running stateful tests if there are no stateful transitions in the schema. I.e. such config would be equivalent to the regular tests generated by schema.parametrize
, but in a randomized order.
Hopefully, if the error message would reflect this aspect and provide some actionable guidance on how to setup stateful tests, then it would be easier to use.
Do these details clarify the reason behind the actual behavior? If you have any suggestions or more questions, I'd be happy to discuss them.
from schemathesis.
@Stranger6667 Thanks for the explanation. My understanding from the docs was that if there are no links for cases, they are generated same as stateless. So if there are no links at all, the test set will be same as stateless except in random order.
The error Type APIWorkflow defines no rules
made me think I did not specify settings, it did not point me to "you have no links in you Open API schema". It would be great if you can make the error clearer with guidelines. 👍
I think I owe you an explanation why I'm trying to run stateful tests without links.
I hit an issue with current implementation of stateless tests, where originally we use lazy loading to postpone the schema load. URL to API contains version in it and it is not know at the time we load module. We've had a workaround to manually specify it using env var, but I rewrote it to remove this and actually check the version on the server.
Which means I need the server, which is available after running a fixture.
But to be able to use it for schemathesis I need to move the schema loading to fixture too. And I did not find a way to do this with stateless tests, but stateful support this. Is there a reason behind this?
So I cannot easily use pytest.mark.skipif
based on version since fixtures run after that. I created a custom mark defined by fixture that provides the version. For now I guess I'll have to move the server fixture to collection hook (ughhh) to have the server available before schemathesis lazy loading is called (when decorators are called). This has a benefit of not having to work around pytest.mark.skipif
, but I'll have to move cleanup to some hook called after test execution. Not as clean as fixture...
from schemathesis.
Thank you for providing the context!
My understanding from the docs was that if there are no links for cases, they are generated same as stateless.
I changed this behavior aspect some time ago, but forgot to update the docs accordingly. Will work on the clarifications, so they will be shipped in the next release
But to be able to use it for schemathesis I need to move the schema loading to fixture too. And I did not find a way to do this with stateless tests, but stateful support this. Is there a reason behind this?
I think that lazy loading should work:
from contextlib import asynccontextmanager
from fastapi import FastAPI
import pytest
import schemathesis
@pytest.fixture
def web_app(db):
# some dynamically built application
# that depends on other fixtures
app = FastAPI()
@asynccontextmanager
async def lifespan(_: FastAPI):
await db.connect()
yield
await db.disconnect()
return schemathesis.from_dict(app.openapi())
schema = schemathesis.from_pytest_fixture("web_app")
@schema.parametrize()
def test_api(case):
...
In this example, the schema is loaded inside the web_app
fixture - is it something you are looking for?
So I cannot easily use pytest.mark.skipif based on version since fixtures run after that. I created a custom mark defined by fixture that provides the version. For now I guess I'll have to move the server fixture to collection hook (ughhh) to have the server available before schemathesis lazy loading is called (when decorators are called). This has a benefit of not having to work around pytest.mark.skipif, but I'll have to move cleanup to some hook called after test execution. Not as clean as fixture...
Interesting! Maybe it could be that the mark is propagated to the fixture, which loads the server (it should be available via the request.node.get_closest_marker("pytest.mark.skipif")
) and as the version is loaded, then check it and call pytest.skip
if needed?
from schemathesis.
I changed this behavior aspect some time ago, but forgot to update the docs accordingly. Will work on the clarifications, so they will be shipped in the next release
Thanks!
I think that lazy loading should work:
Sadly we already have this and it does not work for our case. It seems all decorators are calculated before they are applied. So decorator order does not matter and I cannot first skip and than load schema. To use lazy loading with version check I need to add 2nd version check to the web_app
fixture.
Interesting! Maybe it could be that the mark is propagated to the fixture, which loads the server (it should be available via the
request.node.get_closest_marker("pytest.mark.skipif"))
and as the version is loaded, then check it and callpytest.skip
if needed?
Interesting approach, I did not think of using the skipif
marker itself as trigger to load the version variable. This way I can make the code that triggers version load to be called only when the skipif
is encountered. It narrows it a little, but assumes we don't have other skipif
than those based on version. Plus the code would probably have to be at the same place as now - the collection hook. Having it in fixture as decorator still evaluates the lazy loaded schema.
from schemathesis.
Related Issues (20)
- OAS validation for binary stream payloads of Contract Type application/pdf, image/jpeg, image/png, application/msword HOT 8
- More useful reports on internal errors
- strawberry scalar of type JSON HOT 3
- [BUG] hypothesis.errors.InvalidArgument: test has already been decorated with a settings object. HOT 10
- schemathesis and fastapi with elasticsearch HOT 4
- [FEATURE] Corpus for input values
- [BUG] Parametrization incompatibility with falcon ASGI apps HOT 5
- [FEATURE] Expose the way to configure header generation
- [FEATURE] Add a __main__.py to schemathesis.cli HOT 5
- [FEATURE] Add doco on hooks.py HOT 3
- [BUG] Schemathesis fails with pytest integration but works with CLI HOT 3
- [BUG] curl code samples omit non-printable characters HOT 3
- [BUG] astroid crashes with `RecursionError` in a pytest session if schemathesis is installed HOT 1
- [BUG] pytest plugin broken and triggering warnings/missreported errors on modern pluggy HOT 1
- [BUG] Setting body after make_case() causes request to be sent without a body HOT 1
- [BUG] TypeError HOT 4
- `allow_x00=False` not affecting Headers and Cookies [BUG] HOT 4
- [BUG] HTTPConnection/HTTPSConnection is not iterable HOT 3
- [FEATURE]: Improve output on `MaxRetryError`
- Schemathesis is generating invalid payload when we use examples and value objects in openapi spec HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from schemathesis.