ligurio/semgrep-rules

semgrep rules for flakiness, missed error handling, Lua antipatterns and pitfalls.

semgrep rules

This repo holds patterns for finding non-determinism and missed error handling in C, Python and Go source code, and rules for Lua source code.

The rules currently supports semgrep and coccinelle.

To run a single semgrep rule:

$ semgrep -f rules/<lang>/<rule>.yml .

To run all semgrep rules:

$ semgrep --config rules/<lang>/

To run a single coccinelle rule:

$ spatch --sp-file coccinelle/mmap_map_failed.cocci --very-quiet --dir ~/sources/tarantool/src/

Lua rules

General rules

• basic/cmp_by_reference
• basic/fd_leak
• basic/func_inside_func
• basic/immutable_looping_variables
• basic/init_rng_without_seed
• basic/loadstring
• basic/magic_number
• basic/pcall_err_handling
• basic/pcall_with_method
• basic/print
• basic/require_inside_func
• basic/trace_enabled
• basic/unsafe_function
• basic/use_fd_after_close
• basic/writing_to_file_in_read_mode

LuaJIT-specific rules

• luajit/jit_off
• luajit/jit/nyi
• luajit/jit/2.1
• luajit/jit/partial

Tarantool-specific rules

• tarantool/box/box_cfg_raw_access
• tarantool/box/grant_guest_full_access
• tarantool/box/missed_if_not_exist
• tarantool/box/set_trigger_once
• tarantool/crypto/insecure-hash-algorithm
• tarantool/digest/insecure-hash-algorithm
• tarantool/vshard/bad_hash_func

Flakiness

Non-determinism is a source of test flakiness.

References

• An empirical analysis of flaky tests - Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, Darko Marinov
• Empirical Analysis of Factors and their Effect on Test Flakiness - Practitioners’ Perceptions - Azeem Ahmad, Ola Leifler, Kristian Sandahl
• Root Causing Flaky Tests in a Large-Scale Industrial Setting - Wing Lam, Patrice Godefroid, Suman Nath, Anirudh Santhiar, Suresh Thummalapenta
• What is the Vocabulary of Flaky Tests? - Gustavo Pinto, Breno Miranda, Supun Dissanayake, Marcelo d'Amorim, Christoph Treude, Antonia Bertolino
• Eradicating Non-Determinism in Tests - Martin Fowler

Error handling

Error handling is importance for application reliability. Nice description was given by Dan Luu:

Proper error handling code is hard. Bugs in error handling code are a major cause of bad problems. This means that the probability of having sequential bugs, where an error causes buggy error handling code to run, isn't just the independent probabilities of the individual errors multiplied. It's common to have cascading failures cause a serious outage. There's a sense in which this is obvious -- error handling is generally regarded as being hard. If I mention this to people they'll tell me how obvious it is that a disproportionate number of serious postmortems come out of bad error handling and cascading failures where errors are repeatedly not handled correctly. But despite this being “obvious”, it's not so obvious that sufficient test and static analysis effort are devoted to making sure that error handling works.

See also Aspirator, a simple tool that finds serious bugs in Java exception handler.

References

• The Do's and Don'ts of Error Handling - Joe Armstrong (GOTO 2018)
• Finding Error-Handling Bugs in Systems Code Using Static Analysis - Cindy Rubio-González, Ben Liblit
• Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems - Ding Yuan, Yu Luo, Xin Zhuang, Guilherme Renna Rodrigues, Xu Zhao, Yongle Zhang, Pranay U. Jain, and Michael Stumm, University of Toronto (USENIX)
• Improving the Quality of Error-Handling Code in Systems Software using Function-Local Information - Suman Saha

semgrep-rules's Issues

Add test smells detector

• "tsDetect: an open source test smells detection tool" https://dl.acm.org/doi/10.1145/3368089.3417921
• https://testsmells.org/assets/publications/FSE2020_TechnicalPaper.pdf

Add rules with code smells

• SonarCube rules https://rules.sonarsource.com/python/type/Code%20Smell

Thread sleep

https://github.com/kzaikin/test-smells/tree/master/src/test/java/reliability/thread_sleep

Add Tarantool rules

semgrep has a Lua support with develop status, see https://semgrep.dev/docs/language-support/

This is a list of possibly interesting things that should be discovered in Lua source code:

• (LuaJIT specific) performance tips
  • https://stackoverflow.com/questions/7167566/luajit-2-optimization-guide
  • http://wiki.luajit.org/Numerical-Computing-Performance-Guide
• (LuaJIT specific) http://wiki.luajit.org/NYI
• (Tarantool-specific) grep :select не должен ничего найти
• (Tarantool-specific) аккуратное вычсиление bucket_id только в одном месте программы
• (Tarantool-specific) все файберы имеют имена
• (Tarantool-specific) все потенциально тяжелые циклы имеют внутри илд
• (Tarantool-specific) метрики экспортируются
• Можно маркировки вводить типа «raises FiberIsCancelled», «returns nil, err» и проверять корректность вызывающих функций.

Close all resources

I/O: Tests can be flaky because of I/O operations (besides network). One example we encountered was a test that would fail whenever the file it was trying to read had not been properly closed. Make sure all your tests properly close all resources and use appropriate synchronization for shared resources.

https://mir.cs.illinois.edu/lamyaa/publications/fse14.pdf

Floating point operations

Floating point operations: We came across tests that were flaky because of floating point operations. Floating point operations are by nature imprecise and require careful attention to avoid underflows, overflows, or issues with non-associative addition.

https://mir.cs.illinois.edu/lamyaa/publications/fse14.pdf

Setup testing in CI

semgrep --test --test-ignore-todo

https://github.com/returntocorp/semgrep-rules/blob/develop/.github/workflows/semgrep-rules-test.yml

Missed SO_REUSEADDR and SO_REUSEPORT are reasons of flakiness

See for example https://stevenjohnstone.net/posts/so_reuaseaddr-golang/ and https://man7.org/linux/man-pages/man7/socket.7.html

• Python: https://docs.python.org/3/library/socket.html
• Lua: https://w3.impa.br/~diego/software/luasocket/tcp.html
• C: https://man7.org/linux/man-pages/man7/socket.7.html
• Go: SetsockoptInt https://gist.github.com/joliver/4ccd58605e07e8edf71904b172d95513

Compare unordered structs

• C https://en.cppreference.com/w/cpp/container/unordered_map
• Go: reflect.DeepEqual(a, b)

Resource leak

Resource leak: Tests can be flaky if the application or tests do not properly acquire and release resources (such as memory or database connections). To avoid such issues, consider using resource pools and make sure that resources are returned to the pool when they are no longer needed.

https://mir.cs.illinois.edu/lamyaa/publications/fse14.pdf

Dependence on time

Time: Tests can be flaky if they depend on system time. System time is highly platform-dependent. If you expect your test to run on other machines, do not use system time.

• Source: https://mir.cs.illinois.edu/lamyaa/publications/fse14.pdf
• Examples: https://github.com/kzaikin/test-smells/tree/master/src/test/java/reliability/thread_sleep

Unordered collections

Unordered collections: Tests can be flaky when they make assumptions about the order of elements in an unordered collection. If your test is working with a set or a JSON object, don’t make any assumptions about the order, simply check for existence of elements in the collection.

Source: https://mir.cs.illinois.edu/lamyaa/publications/fse14.pdf
See LuaJIT/LuaJIT#719

• C https://en.cppreference.com/w/cpp/container/unordered_map
• Go: reflect.DeepEqual(a, b)
• semgrep/semgrep-rules#211

Possible sources of non-determinism

possible sources

Python: catch all exceptions

Avoid generic, catch-all exception handling.

Antipattern:

try:
    do_something()
except:
    print("Caught it!")

except example below:

try:
    f = open('myfile.txt')
    s = f.readline()
    i = int(s.strip())
except IOError as (errno, strerror):
    print("I/O error({0}): {1}".format(errno, strerror))
except ValueError:
    print("Could not convert data to an integer.")
except:
    print("Unexpected error:", sys.exc_info()[0])
    raise

• https://docs.python.org/3/tutorial/errors.html
• https://docs.python.org/3.10/library/exceptions.html#exception-hierarchy
• https://github.com/justinamiller/Exception-guidelines#dos-and-donts-for-exceptions

Add rules to highlight infrastructure issues

Infrastructure issues
Random incidents, like Continuous Integration (CI) node failures, network
issues, database outage etc.

Resource Leaks: https://martinfowler.com/articles/nonDeterminism.html

Remote Services: https://martinfowler.com/articles/nonDeterminism.html

Lack of Isolation: https://martinfowler.com/articles/nonDeterminism.html

https://github.com/kdeldycke/awesome-falsehood

http://xunitpatterns.com/Fragile%20Test.html

https://github.com/zulip/zulip/blob/master/tools/semgrep.yml

Selenium: https://sqa.stackexchange.com/questions/5240/are-selenium-functional-tests-reliable-enough-to-be-worthwhile

Java https://wiki.saucelabs.com/display/DOCS/How+to+Deal+with+Flaky+Java+Tests

Java, Selenium: https://habr.com/ru/company/jugru/blog/416757/

Tensorflow tensorflow/tensorflow#3103

Tensorflow https://www.twosigma.com/insights/article/a-workaround-for-non-determinism-in-tensorflow/

https://www.slideshare.net/jezhumble/creating-maintainable-automated-acceptance-tests

https://jkschin.com/2017/06/30/non-determinism.html

https://testing.googleblog.com/2009/06/my-selenium-tests-arent-stable.html

https://engineering.salesforce.com/flaky-tests-and-how-to-avoid-them-25b84b756f60

https://hackernoon.com/flaky-tests-a-war-that-never-ends-9aa32fdef359

https://www.eviltester.com/page/flaky/

https://sqa.stackexchange.com/questions/8508/what-are-anti-patterns-in-test-automation

https://sqa.stackexchange.com/questions/5240/are-selenium-functional-tests-reliable-enough-to-be-worthwhile

https://mestachs.wordpress.com/2012/08/13/selenium-best-practices/

примеры flaky кода https://www.programcreek.com/python/example/94767/flaky.flaky

тикеты на flaky тесты https://openedx.atlassian.net/browse/EDUCATOR-4134?filter=10600

Understanding Flaky Tests: The Developer’s Perspective -> "3 RQ1 – FLAKY TESTS: NATURE, ORIGIN, AND FIXING"

An Empirical Analysis of Flaky Tests

Empirical Analysis of Factors and their Effect on Test Flakiness - Practitioners’ Perceptions

Making System User Interactive Tests Repeatable: When and What Should we Control?

https://martinfowler.com/articles/nonDeterminism.html

https://schneide.blog/2019/07/22/non-determinism-in-c/

Kubernetes: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/flaky-tests.md#avoiding-flakes (We have a goal of 99.9% flake free tests. This means that there is only one flake in one thousand runs of a test.)

https://github.com/diy1/aspirator/tree/master/chord-src-2.1 (Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-intensive Systems)

Add useful information to a README

describe a common reasons of flakiness

unsafe_function: add io.popen

https://github.com/ligurio/semgrep-rules/blob/master/rules/lua/basic/unsafe_function.lua

Time bomb data

https://codingcraftsman.wordpress.com/2022/01/24/time-bomb-data/

Randomness

Randomness: Tests can be flaky if they use random number generators without accounting for the full range of possible results. If your test uses a random number generator, consider controlling for the seed and carefully consider all the possible values that can be returned.

• Source: https://mir.cs.illinois.edu/lamyaa/publications/fse14.pdf
• Examples: https://github.com/kzaikin/test-smells/tree/master/src/test/java/reliability/random