pschanely / hypothesis-crosshair Goto Github PK
View Code? Open in Web Editor NEWLevel-up your Hypothesis tests with CrossHair
License: MIT License
Level-up your Hypothesis tests with CrossHair
License: MIT License
Hi,
I was curious to try hypothesis-crosshair, this is just reporting some issues which popped up in my first attempt.
1. Import of deprecated package
File "/home/jobh/mambaforge/envs/hypothesis/lib/python3.11/site-packages/crosshair/libimpl/relib.py", line 2, in <module>
from sre_parse import ANY # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jobh/mambaforge/envs/hypothesis/lib/python3.11/sre_parse.py", line 2, in <module>
warnings.warn(f"module {__name__!r} is deprecated",
DeprecationWarning: module 'sre_parse' is deprecated
================================================================================= short test summary info ==================================================================================
FAILED crosshair/test_basic.py::test_needs_solver - DeprecationWarning: module 'sre_parse' is deprecated
==================================================================================== 1 failed in 0.07s =====================================================================================
I think this package should be imported as re._parser
in python 3.11+. Next, with -Wignore
:
2. Error on first run
File "/home/jobh/src/hypothesis/crosshair/test_basic.py", line 4, in test_needs_solver
@given(st.integers())
^^^
File "/home/jobh/src/hypothesis/hypothesis-python/src/hypothesis/core.py", line 1658, in wrapped_test
raise the_error_hypothesis_found
File "/home/jobh/mambaforge/envs/hypothesis/lib/python3.11/site-packages/crosshair/libimpl/builtinslib.py", line 1101, in __hash__
return self.__index__().__hash__()
^^^^^^^^^^^^^^^^
File "/home/jobh/mambaforge/envs/hypothesis/lib/python3.11/site-packages/crosshair/libimpl/builtinslib.py", line 1112, in __index__
space = context_statespace()
^^^^^^^^^^^^^^^^^^^^
File "/home/jobh/mambaforge/envs/hypothesis/lib/python3.11/site-packages/crosshair/statespace.py", line 247, in context_statespace
raise CrosshairInternal
crosshair.util.CrosshairInternal
================================================================================= short test summary info ==================================================================================
FAILED crosshair/test_basic.py::test_needs_solver - crosshair.util.CrosshairInternal
==================================================================================== 1 failed in 0.13s =====================================================================================
Interestingly, this failure goes away if I run again (until the .hypothesis
db directory is removed). So it seems that the example has been found, and has been recorded by hypothesis, before the error happens.
================================================================================= short test summary info ==================================================================================
FAILED crosshair/test_basic.py::test_needs_solver - assert 123456789 != 123456789
==================================================================================== 1 failed in 0.37s =====================================================================================
(yay!)
I'm not really sure what's causing this, but it accounts for 351 out of 386 failures on the latest nocover
run of HypothesisWorks/hypothesis#4034. I suspect most of the affected tests would then pass, but either way triage would be a lot easier.
Build logs here; representative traceback:
File "python3.10/site-packages/hypothesis/core.py", line 1699, in wrapped_test
raise the_error_hypothesis_found
File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "python3.10/site-packages/hypothesis_crosshair_provider/crosshair_provider.py", line 8, in <module>
import crosshair.core_and_libs # Needed for patch registrations
File "python3.10/site-packages/crosshair/core_and_libs.py", line 129, in <module>
_make_registrations()
File "python3.10/site-packages/crosshair/core_and_libs.py", line 74, in _make_registrations
arraylib.make_registrations()
File "python3.10/site-packages/crosshair/libimpl/arraylib.py", line 157, in make_registrations
register_type(array, make_array)
File "python3.10/site-packages/crosshair/core.py", line 553, in register_type
raise CrosshairInternal(f'Duplicate type "{typ}" registered')
crosshair.util.CrosshairInternal: Duplicate type "<class 'array.array'>" registered
https://crosshair-web.org/?crosshair=0.1&python=3.8&gist=53cb15615819ffd4e29fe0c29b1cfe38
this may not be the smallest counterexample, but it was as small as I could easily make it. min(n, 1)
is to avoid crosshair finding domain errors.
This is the root cause of hypothesis-crosshair erroring on this property:
@given(st.decimals(min_value=1.0, max_value=1.5))
def f(n):
pass
f()
Talking to a friend about using hypothesis+crosshair as an easy-to-use Z3 interface, my main caveat was that we can find counterexamples but not use Z3's optimization power. But if we hacked on this a bit, maybe we could...
Obviously it'd only work for cases where hypothesis.target()
is called with a symbolic object, but so long as we let you add a shim to report those cases I think this would work!
It's possible this is a symptom of a separate issue. I haven't looked into it and am just logging it to ensure we don't forget.
This seems superficially similar to #11, but I don't know if it's related. I don't know if using timedelta
is critical to reproducing or not (when I tried to manually shrink via st.timedeltas
-> st.integers
, I got #11).
@given(st.timedeltas())
def outer(val):
@given(st.timedeltas(min_value=val, max_value=val))
def inner(v):
assert v == val
inner()
outer()
Traceback (most recent call last):
File "/Users/tybug/Desktop/Liam/coding/sandbox/hypothesis_/sandbox4.py", line 31, in <module>
outer()
File "/Users/tybug/Desktop/Liam/coding/sandbox/hypothesis_/sandbox4.py", line 26, in outer
def outer(val):
^
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/core.py", line 1657, in wrapped_test
raise the_error_hypothesis_found
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/core.py", line 1624, in wrapped_test
state.run_engine()
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/core.py", line 1152, in run_engine
runner.run()
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py", line 757, in run
self._run()
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py", line 1219, in _run
self.generate_new_examples()
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py", line 975, in generate_new_examples
self.test_function(data)
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py", line 531, in test_function
raise Flaky(
hypothesis.errors.Flaky: Inconsistent results from replaying a failing test case!
last: INTERESTING from RecursionError at /opt/homebrew/lib/python3.12/site-packages/crosshair/tracers.py:163
this: OVERRUN
#20 did indeed fix the ~dozens of returning a symbolic type
errors that I was getting before, but in a few rare cases it looks like that's uncovered a new failure mode: this CI job has three tests failing with
File ".../hypothesis_crosshair_provider/crosshair_provider.py", line 293, in realize
return self.export_value(value)
File ".../hypothesis_crosshair_provider/crosshair_provider.py", line 280, in export_value
return deep_realize(value)
File ".../crosshair/core.py", line 258, in deep_realize
return deepcopyext(value, CopyMode.REALIZE, {})
File ".../crosshair/copyext.py", line 47, in deepcopyext
obj = obj.__ch_realize__() # type: ignore
File ".../crosshair/libimpl/builtinslib.py", line 3819, in __ch_realize__
return bytes(tracing_iter(self.inner))
File ".../crosshair/tracers.py", line 463, in tracing_iter
value = next(itr)
File ".../crosshair/libimpl/builtinslib.py", line 2537, in __iter__
if not space.smt_fork(idx < my_smt_len):
File ".../crosshair/statespace.py", line 1043, in smt_fork
return self.choose_possible(expr, probability_true)
File ".../crosshair/statespace.py", line 854, in choose_possible
self.check_timeout()
File ".../crosshair/statespace.py", line 841, in check_timeout
raise PathTimeout
crosshair.util.PathTimeout
I'm not really sure what we should do here. Maybe we should have a known exception to indicate that the current test case can't be realized so that Hypothesis can skip it as if for assume(False)
? (raised/caught here) Increasing the timeout will only make the problem less common rather than fixing it, but at the cost of rather slow tests. Maybe that - or removing it entirely - is worth it though?
I think this only happens under certain conditions since pschanely/CrossHair#263 uses st.text()
and works fine. Maybe when no meaningful decisions are made on the symbolic?
This issue also occurs with st.characters()
. Found while trying to run hypothesis' test suite under crosshair (HypothesisWorks/hypothesis#4022 (review)). I'm tentatively calling this a crosshair-side issue, but if you discover otherwise let me know π
@settings(backend="crosshair", deadline=None)
@given(st.text())
def f(s):
pass
f()
My debug output:
1272686.164| |pre_path_hook() No coverage biasing in effect. ( 0 code locations)
1272686.165| |condition_parser() Using parsers: []
1272686.166| |per_test_case_context_manager() starting iteration 1
1272686.168| |_next_name() Drawing str_01
1272686.172| |find_model_value() *** Begin Not Deterministic Debug ***
1272686.173| |find_model_value() Model value node expected; found <class 'crosshair.statespace.DeatchedPathNode'> instead.
1272686.173| |find_model_value() Traceback: (<module> sandbox4.py:36) (f sandbox4.py:33) (wrapped_test core.py:1623) (run_engine core.py:1151) (run engine.py:736) (_run engine.py:1198) (generate_new_examples engine.py:881) (cached_test_function engine.py:1432) (test_function engine.py:434) (__stoppable_test_function engine.py:307) (_execute_once_for_engine core.py:1047) (execute_once core.py:961) (__exit__ contextlib.py:144) (per_test_case_context_manager crosshair_provider.py:110) (deep_realize core.py:251) (deepcopyext copyext.py:37) (__ch_realize__ builtinslib.py:3106) (__index__ builtinslib.py:1113) (find_model_value statespace.py:937) (test_stack util.py:221)
1272686.173| |find_model_value() *** End Not Deterministic Debug ***
1272686.174| |per_test_case_context_manager() ended iteration (exception: NotDeterministic: ) (per_test_case_context_manager crosshair_provider.py:110) (deep_realize core.py:251) (deepcopyext copyext.py:37) (__ch_realize__ builtinslib.py:3106) (__index__ builtinslib.py:1113) (find_model_value statespace.py:939)
1272686.174| |per_test_case_context_manager() no decisions made; ignoring this iteration
1272686.197| |export_value() WARNING: export_value() requested before test case complered (<module> sandbox4.py:36) (f sandbox4.py:33) (wrapped_test core.py:1623) (run_engine core.py:1151) (run engine.py:736) (_run engine.py:1198) (generate_new_examples engine.py:881) (cached_test_function engine.py:1432) (test_function engine.py:458) (post_test_case_hook crosshair_provider.py:286) (export_value crosshair_provider.py:277) (test_stack util.py:221)
1272686.203| |__init__() CrosshairInternal
1272686.204| |__init__() Stack trace:
File "/Users/tybug/Desktop/Liam/coding/sandbox/hypothesis_/sandbox4.py", line 36, in <module>
f()
File "/Users/tybug/Desktop/Liam/coding/sandbox/hypothesis_/sandbox4.py", line 33, in f
@given(st.text())
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/core.py", line 1623, in wrapped_test
state.run_engine()
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/core.py", line 1151, in run_engine
runner.run()
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py", line 736, in run
self._run()
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py", line 1198, in _run
self.generate_new_examples()
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py", line 881, in generate_new_examples
zero_data = self.cached_test_function(bytes(BUFFER_SIZE))
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py", line 1432, in cached_test_function
self.test_function(data)
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py", line 510, in test_function
self.__stoppable_test_function(data)
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py", line 307, in __stoppable_test_function
self._test_function(data)
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/core.py", line 1041, in _execute_once_for_engine
result = self.execute_once(data)
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/core.py", line 964, in execute_once
result = self.test_runner(data, run)
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/core.py", line 737, in default_executor
return function(data)
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/core.py", line 875, in run
kw, argslices = context.prep_args_kwargs_from_strategies(
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/control.py", line 157, in prep_args_kwargs_from_strategies
obj = check(self.data.draw(s, observe_as=f"generate:{k}"))
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/data.py", line 2423, in draw
return strategy.do_draw(self)
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/strategies/_internal/lazy.py", line 167, in do_draw
return data.draw(self.wrapped_strategy)
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/data.py", line 2417, in draw
return strategy.do_draw(self)
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/strategies/_internal/strings.py", line 117, in do_draw
return data.draw_string(
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/data.py", line 2182, in draw_string
node = self._pop_ir_tree_node("string", kwargs, forced=forced)
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/data.py", line 2330, in _pop_ir_tree_node
if not ir_value_permitted(node.value, node.ir_type, kwargs):
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/data.py", line 1089, in ir_value_permitted
if len(value) < kwargs["min_size"]:
File "/opt/homebrew/lib/python3.12/site-packages/crosshair/libimpl/builtinslib.py", line 1112, in __index__
space = context_statespace()
File "/opt/homebrew/lib/python3.12/site-packages/crosshair/statespace.py", line 247, in context_statespace
raise CrosshairInternal
File "/opt/homebrew/lib/python3.12/site-packages/crosshair/util.py", line 594, in __init__
debug(" Stack trace:\n" + "".join(traceback.format_stack()))
Traceback (most recent call last):
File "/Users/tybug/Desktop/Liam/coding/sandbox/hypothesis_/sandbox4.py", line 36, in <module>
f()
File "/Users/tybug/Desktop/Liam/coding/sandbox/hypothesis_/sandbox4.py", line 33, in f
@given(st.text())
^^^
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/core.py", line 1656, in wrapped_test
raise the_error_hypothesis_found
File "/opt/homebrew/lib/python3.12/site-packages/crosshair/libimpl/builtinslib.py", line 1112, in __index__
space = context_statespace()
^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.12/site-packages/crosshair/statespace.py", line 247, in context_statespace
raise CrosshairInternal
crosshair.util.CrosshairInternal
These assertions in Hypothesis check that if forced is True
, then p > 0
, and vice-versa at the other end - because it would be invalid to generate the forced value otherwise. However, the following example fails with Crosshair:
from hypothesis import settings
from hypothesis.stateful import RuleBasedStateMachine, rule, run_state_machine_as_test
@run_state_machine_as_test
@settings(backend="crosshair", deadline=None)
class IntListRules(RuleBasedStateMachine):
@rule()
def a(self): pass
@rule()
def b(self): pass
I'm reporting it here because of the draw_boolean()
symptom, but I actually find it pretty plausible that it's an issue in Crosshair's reasoning about the internal FeatureStrategy
(see here). As usual, found in HypothesisWorks/hypothesis#4034 π
Hypothesis' observability features were originally designed for the benefit of users (HypothesisWorks/hypothesis#3797), but have also turned out to be useful for developers - e.g. this discussion led to bugfixes and performance improvements.
Recent work on HypothesisWorks/hypothesis#4034 has me thinking that observability information from Crosshair would actually be pretty valuable for us as maintainers, to answer questions like:
Practically speaking, what do we actually need to do here?
decide what to measure (schema here)
status_reason
, if something on the crosshair side was responsible for a Status.INVALID
result, as if for assume(False)
. I considered whether we need custom statuses for e.g. #21 and concluded that the status_reason
metadata was a better fit.features
; they're intended to be about the runtime behavior of the code under test.
event()
is currently disabled under Crosshair to reduce premature realization, but we could support it nicely by e.g. deferring it to the end of the test case where we realize anyway.timing
observations are meant to be disjoint, so that the sum is total runtime. We therefore probably want to put internal timings in the metadata
instead.metadata
: the catch-all put-anything-here section.connect it up
PrimitiveProvider
protocol and call it hereuse the new information to understand what's going on, improve hypothesis-crosshair, iterate on observability, etc.
Would you find this useful? If so, setting up metadata passthrough would be pretty easy π
This doesn't find an error:
@settings(backend="crosshair", deadline=None, database=None)
@given(st.floats(1.0, 2.0)) # or other bounds
def f(x):
assert False
f()
I see lots of IgnoreAttempt
in debug logs. Probably crosshair is giving up generating inputs here when it shouldn't be?
Here's a quick one to either fix or dismiss as intended; I'm fairly sure the condition bounds for smallest_nonzero_magnitude
should be inclusive <=
as smallest_nonzero_magnitude
is allowed. https://github.com/pschanely/hypothesis-crosshair/blob/main/hypothesis_crosshair_provider/crosshair_provider.py#L222-L227.
from hypothesis import given, settings
from hypothesis import strategies as st
settings.register_profile("crosshair", backend="crosshair", deadline=None)
settings.load_profile("crosshair")
@given(st.integers())
def outer(n1):
@given(st.integers(min_value=n1, max_value=n1))
def inner(n2):
assert n1 == n2
inner()
outer()
stacktrace:
... (truncated)
File "/Users/tybug/Desktop/Liam/coding/sandbox/hypothesis_/sandbox4.py", line 36, in outer
def outer(n1):
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/core.py", line 830, in test
return self.test(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tybug/Desktop/Liam/coding/sandbox/hypothesis_/sandbox4.py", line 37, in outer
@given(st.integers(min_value=n1, max_value=n1))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/strategies/_internal/numbers.py", line 112, in integers
@defines_strategy(force_reusable_values=True)
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/strategies/_internal/utils.py", line 79, in cached_strategy
if cache_key in cache:
^^^^^^^^^^^^^^^^^^
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/cache.py", line 97, in __contains__
return key in self.keys_to_indices
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.12/site-packages/crosshair/libimpl/builtinslib.py", line 908, in __eq__
return numeric_binop(ops.eq, self, other)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.12/site-packages/crosshair/libimpl/builtinslib.py", line 462, in numeric_binop
raise CrosshairInternal("Numeric operation on symbolic while not tracing")
crosshair.util.CrosshairInternal: Numeric operation on symbolic while not tracing
I think we're using the kwargs to st.integers
as a cache key to reuse strategies, which includes n1
, and somehow n1
is realized outside of a crosshair scope (or maybe just outside of its own outer
crosshair scope since we may have jumped to a new one with inner
?).
Idle musing: something we should consider more broadly is nested backend scopes. A user may request backend A on outer
and a different backend B on inner
. Ideally this would just workβ’οΈ, but of course it may not be that simple to support.
Counterexample extracted from https://github.com/HypothesisWorks/hypothesis/blob/d8c17832141f587fec9ef89895ed01a4e9c1650d/hypothesis-python/tests/cover/test_datetimes.py#L52-L54.
Regular CrossHair checks TypeError messages for certain CrossHair type names and will ignores paths when it finds them. We probably need this logic in the hypothesis plugin as well.
When life gets too hard for the solver, it'll raise an UnknownSatisfiability error, and then there's a good chance that our argument realization code produce the same exception, failing to find concrete values for the draws. We probably need to extend the solver timeout during realization and/or wipe the solver state and pretend it was a useless/trivial iteration.
This is the root cause of hypothesis test failures like returning a symbolic type from provider.realize(obj)
(comment, CI)
I'm seeing many related internal IgnoreAttempt
raises here.
For instance:
@given(st.datetimes())
@settings(backend="crosshair", deadline=None)
def f(n):
print("call")
f()
takes ~75 IgnoreAttempts
over 5 seconds until it prints its first call
.
Other offenders are st.dates()
(not that bad) and st.emails()
.
This may not be surprising given crosshair is working with a low level IR, but I wonder if this could be alleviated in some way?
Here's a nice cursed one for you, probably low priority:
from hypothesis import strategies as st
from hypothesis import given, settings
settings.register_profile("crosshair", backend="crosshair", deadline=None)
settings.load_profile("crosshair")
@given(st.builds(dict).map(dict.items))
def f(x):
print(x)
f()
...
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/data.py", line 2447, in draw
return strategy.do_draw(self)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/strategies/_internal/strategies.py", line 844, in do_draw
result = self.pack(x) # type: ignore
^^^^^^^^^^^^
TypeError: descriptor 'items' for 'dict' objects doesn't apply to a 'ShellMutableMap' object
while generating 'x' from builds(dict).map(dict.items)
Hypothesis uses this to register a strategy for typing.ItemsView
.
Somehow, kwargs["max_value"]
(and only max_value
) on an integer ir node becomes symbolic here and errors when we go to cache based off it. This only happens on dates/datetimes so I'm suspecting crosshair interception there causes some of our code to be colored symbolically, but hypothesis' date strategy code may also be at fault.
@given(st.dates())
def f(d):
pass
f()
... (truncated)
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py", line 475, in test_function
self._cache(data)
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py", line 381, in _cache
self.__data_cache_ir[key] = result
~~~~~~~~~~~~~~~~~~~~^^^^^
File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/cache.py", line 111, in __setitem__
i = self.keys_to_indices[key]
~~~~~~~~~~~~~~~~~~~~^^^^^
File "/opt/homebrew/lib/python3.12/site-packages/crosshair/libimpl/builtinslib.py", line 1143, in __hash__
return self.__index__().__hash__()
^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.12/site-packages/crosshair/libimpl/builtinslib.py", line 1154, in __index__
space = context_statespace()
^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.12/site-packages/crosshair/statespace.py", line 247, in context_statespace
raise CrosshairInternal
crosshair.util.CrosshairInternal
Here's a logging patch that narrows the problem, if it's helpful:
diff --git a/hypothesis-python/src/hypothesis/internal/conjecture/engine.py b/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
index eb326f59a..708e48904 100644
--- a/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
+++ b/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
@@ -456,6 +456,7 @@ class ConjectureRunner:
if self.settings.backend != "hypothesis":
for node in data.examples.ir_tree_nodes:
value = data.provider.post_test_case_hook(node.value)
+ print("kwargs should not be symbolic", type(node.kwargs["max_value"]))
expected_type = {
"string": str,
"float": float,
With this, I get
kwargs should not be symbolic <class 'int'>
kwargs should not be symbolic <class 'int'>
kwargs should not be symbolic <class 'int'>
kwargs should not be symbolic <class 'int'>
kwargs should not be symbolic <class 'int'>
kwargs should not be symbolic <class 'int'>
kwargs should not be symbolic <class 'int'>
kwargs should not be symbolic <class 'int'>
kwargs should not be symbolic <class 'int'>
kwargs should not be symbolic <class 'int'>
kwargs should not be symbolic <class 'int'>
kwargs should not be symbolic <class 'int'>
kwargs should not be symbolic <class 'int'>
kwargs should not be symbolic <class 'int'>
kwargs should not be symbolic <class 'int'>
kwargs should not be symbolic <class 'int'>
kwargs should not be symbolic <class 'int'>
kwargs should not be symbolic <class 'int'>
kwargs should not be symbolic <class 'crosshair.libimpl.builtinslib.SymbolicInt'>
followed by the above stacktrace error.
Let's see what works and what doesn't!
import time as tm, pytest
from functools import wraps
from hypothesis import given, settings, strategies as st
@settings(backend="crosshair")
@given(st.text("abcdefg"))
def test_hits_internal_assert(x):
assert set(x).issubset(set("abcdefg"))
@pytest.fixture(scope="function")
def _patch_time(monkeypatch):
def time():
nonlocal current_time
current_time += 0.1
return current_time
current_time = tm.time()
monkeypatch.setattr(tm, "time", wraps(tm.time)(time))
monkeypatch.setattr(tm, "monotonic", wraps(tm.monotonic)(time))
# monkeypatch.setattr(tm, "perf_counter", wraps(tm.perf_counter)(time))
@settings(backend="crosshair")
@given(n=st.integers())
def test_crosshair_does_not_like_monkeypatching(_patch_time, n): ...
# crosshair.register_contract.ContractRegistrationError:
# Pre- and postconditons and skip_body should not differ when registering multiple contracts for the same function
Originally posted by @Zac-HD in HypothesisWorks/hypothesis#3914 (comment)
right now these parameters are ignored, which leads to crosshair generating unsound ir nodes:
@given(st.characters())
def f(c):
assert len(c) == 1
f()
the above gives a flaky error due to this; the root cause of invalid lengths is maybe more clear with this example:
@given(st.characters())
def f(c):
print(len(c)) # 0, 1, 2, ...
f()
values = []
@settings(backend="crosshair", deadline=None, database=None)
@given(st.integers())
def f(x):
values.append(x)
f()
# crosshair.util.CrosshairInternal: Numeric operation on symbolic while not tracing
print("values", values)
We do indeed call post_test_case_hook
-> deep_realize
here, but presumably that returns the realized value rather than realizing the value and any references in place. Is deep-realizing all references even possible to achieve?
Arguably this is an abuse of hypothesis, but we do use this technique in hypothesis tests, eg. cc @Zac-HD since this might involve a wider viability discussion.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.