tikv/agatedb

A persistent key-value storage in rust.

Website | Documentation | Community Chat

TiKV is an open-source, distributed, and transactional key-value database. Unlike other traditional NoSQL systems, TiKV not only provides classical key-value APIs, but also transactional APIs with ACID compliance. Built in Rust and powered by Raft, TiKV was originally created by PingCAP to complement TiDB, a distributed HTAP database compatible with the MySQL protocol.

The design of TiKV ('Ti' stands for titanium) is inspired by some great distributed systems from Google, such as BigTable, Spanner, and Percolator, and some of the latest achievements in academia in recent years, such as the Raft consensus algorithm.

If you're interested in contributing to TiKV, or want to build it from source, see CONTRIBUTING.md.

TiKV is a graduated project of the Cloud Native Computing Foundation (CNCF). If you are an organization that wants to help shape the evolution of technologies that are container-packaged, dynamically-scheduled and microservices-oriented, consider joining the CNCF. For details about who's involved and how TiKV plays a role, read the CNCF announcement.

---

With the implementation of the Raft consensus algorithm in Rust and consensus state stored in RocksDB, TiKV guarantees data consistency. Placement Driver (PD), which is introduced to implement auto-sharding, enables automatic data migration. The transaction model is similar to Google's Percolator with some performance improvements. TiKV also provides snapshot isolation (SI), snapshot isolation with lock (SQL: SELECT ... FOR UPDATE), and externally consistent reads and writes in distributed transactions.

TiKV has the following key features:

• Geo-Replication

  TiKV uses Raft and the Placement Driver to support Geo-Replication.

• Horizontal scalability

  With PD and carefully designed Raft groups, TiKV excels in horizontal scalability and can easily scale to 100+ TBs of data.

• Consistent distributed transactions

  Similar to Google's Spanner, TiKV supports externally-consistent distributed transactions.

• Coprocessor support

  Similar to HBase, TiKV implements a coprocessor framework to support distributed computing.

• Cooperates with TiDB

  Thanks to the internal optimization, TiKV and TiDB can work together to be a compelling database solution with high horizontal scalability, externally-consistent transactions, support for RDBMS, and NoSQL design patterns.

Governance

See Governance.

Documentation

For instructions on deployment, configuration, and maintenance of TiKV,see TiKV documentation on our website. For more details on concepts and designs behind TiKV, see Deep Dive TiKV.

Note:

We have migrated our documentation from the TiKV's wiki page to the official website. The original Wiki page is discontinued. If you have any suggestions or issues regarding documentation, offer your feedback here.

TiKV adopters

You can view the list of TiKV Adopters.

TiKV software stack

• Placement Driver: PD is the cluster manager of TiKV, which periodically checks replication constraints to balance load and data automatically.
• Store: There is a RocksDB within each Store and it stores data into the local disk.
• Region: Region is the basic unit of Key-Value data movement. Each Region is replicated to multiple Nodes. These multiple replicas form a Raft group.
• Node: A physical node in the cluster. Within each node, there are one or more Stores. Within each Store, there are many Regions.

When a node starts, the metadata of the Node, Store and Region are recorded into PD. The status of each Region and Store is reported to PD regularly.

Quick start

Deploy a playground with TiUP

The most quickest to try out TiKV with TiDB is using TiUP, a component manager for TiDB.

You can see this page for a step by step tutorial.

Deploy a playground with binary

TiKV is able to run separately with PD, which is the minimal deployment required.

1. Download and extract binaries.

$ export TIKV_VERSION=v7.5.0
$ export GOOS=darwin  # only {darwin, linux} are supported
$ export GOARCH=amd64 # only {amd64, arm64} are supported
$ curl -O  https://tiup-mirrors.pingcap.com/tikv-$TIKV_VERSION-$GOOS-$GOARCH.tar.gz
$ curl -O  https://tiup-mirrors.pingcap.com/pd-$TIKV_VERSION-$GOOS-$GOARCH.tar.gz
$ tar -xzf tikv-$TIKV_VERSION-$GOOS-$GOARCH.tar.gz
$ tar -xzf pd-$TIKV_VERSION-$GOOS-$GOARCH.tar.gz

2. Start PD instance.

$ ./pd-server --name=pd --data-dir=/tmp/pd/data --client-urls="http://127.0.0.1:2379" --peer-urls="http://127.0.0.1:2380" --initial-cluster="pd=http://127.0.0.1:2380" --log-file=/tmp/pd/log/pd.log

3. Start TiKV instance.

$ ./tikv-server --pd-endpoints="127.0.0.1:2379" --addr="127.0.0.1:20160" --data-dir=/tmp/tikv/data --log-file=/tmp/tikv/log/tikv.log

4. Install TiKV Client(Python) and verify the deployment, required Python 3.5+.

$ pip3 install -i https://test.pypi.org/simple/ tikv-client

from tikv_client import RawClient

client = RawClient.connect("127.0.0.1:2379")

client.put(b'foo', b'bar')
print(client.get(b'foo')) # b'bar'

client.put(b'foo', b'baz')
print(client.get(b'foo')) # b'baz'

Deploy a cluster with TiUP

You can see this manual of production-like cluster deployment presented by @c4pt0r.

Build from source

See CONTRIBUTING.md.

Client drivers

• Go (The most stable and widely used)
• Java
• Rust
• C

If you want to try the Go client, see Go Client.

Security

Security audit

A third-party security auditing was performed by Cure53. See the full report here.

Reporting Security Vulnerabilities

To report a security vulnerability, please send an email to TiKV-security group.

See Security for the process and policy followed by the TiKV project.

Communication

Communication within the TiKV community abides by TiKV Code of Conduct. Here is an excerpt:

In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.

Social Media

• Twitter
• Blog
• Reddit
• Post questions or help answer them on Stack Overflow

Slack

Join the TiKV community on Slack - Sign up and join channels on TiKV topics that interest you.

License

TiKV is under the Apache 2.0 license. See the LICENSE file for details.

Acknowledgments

• Thanks etcd for providing some great open source tools.
• Thanks RocksDB for their powerful storage engines.
• Thanks rust-clippy. We do love the great project.

agatedb's Issues

[dev] Tracking issue for database persistence

Currently, we simply delete table files after we drop the last reference Arc to TableInner. This causes several issues.

• We had to forget (leak) table manually, thus causing leak sanitizer complain. https://github.com/tikv/agatedb/pull/60/checks?check_run_id=1452618735
• By using mem::forget, we also have to manually close underlying file objects.

[dev] panic on drop causes process to abort

As seen in failed CI in #88 , agatedb will abort when we panic on drop. I implemented a partial solution the we ignore all failures (e.g. PoisonedLock, failed to remove file) and simply print them. We could later find a better way to solve this issue.

agatedb 中 的 yatp 是否可以拿出来放到AgateOptions中

看起来，现在agatedb并没有类似rocksdb的Column family的东西，我一开始以为table是类似Column family，仔细看了一下代码，感觉table只是一个sst文件（没有compact）。

我的想法是，搞多个agatedb，每个相当于一个Column family

但是，现在每个agatedb都持有一份独立的yatp，开多个agatedb，就会有一堆线程。

我的想法是，把yatp放到参数中去，这样多个agatedb就可以共用一个yatp，不知道这个想法是否合理。

Merge branch develop into master

Development Task

Merge develop into master part by part.

Development Progress

The pull requests merged at develop branch are as follows, I would try to apply them into master.

• vlog

  • #84 @zhangjinpeng1987 #143
  • #88 @zhangjinpeng1987 #127

• memtable

  • #45 @GanZiheng #126
  • #46 @GanZiheng #126

• levels

  • #47 @zhangjinpeng1987 #138
  • #50 @zhangjinpeng1987 #148
  • #52 @GanZiheng #129
  • #54 @GanZiheng #129
  • #75 @GanZiheng #129
  • #73 @GanZiheng #129
  • #77 @GanZiheng #129
  • #78 @GanZiheng #129
  • #101 @GanZiheng #129

• txn

  • #57 @GanZiheng #132
  • #67 @GanZiheng
    • #136
    • #143
  • #72 @GanZiheng #143

• agate

  • #58 @GanZiheng #127
  • #62 @GanZiheng #127
  • #60 @GanZiheng #123
  • #87 @GanZiheng #170
  • #91 @GanZiheng #170

• bench

  • #86 @GanZiheng #169
  • #90 @GanZiheng #169
  • #93 @GanZiheng #167
  • #94 @GanZiheng #167
  • #96 @GanZiheng #167
  • #99 @GanZiheng #167
  • #102 @GanZiheng #167

Proposal: Ingest External SST

This issue describe a possible way to ingest external SST file to a running agatedb instance just as rocksdb.

Background

TiKV use ingest external SST feature for some use case. To integrate agatedb into TiKV, we need this feature.

From Creating and Ingesting SST files · facebook/rocksdb Wiki (github.com), ingesting a SST needs follow steps:

1. copy or link the file(s) into the DB directory
2. block (not skip) writes to the DB and assign correct sequence number to ingested file
3. flush memtable if range overlap (SST file's key range overlap with memtable)
4. assign the file to LSM-tree

Things go different in agatedb.

Analysis

Step 1 and step 4 are the same.

For step 3, it is unnecessary to flush memtable when overlap for agatedb. Agatedb reads all levels when get and picks the latest one (less or equal to read_ts), so it's okay to have new keys below old keys in LSM-tree. This really makes the whole process efficient. (Please point out if I'm wrong)

For step 2, it is the most difficult: How to protect ACID during and after ingestion?

Agatedb has ACID transaction (SSI). The architecture is totally different from rocksdb. We have a read_ts and a commit_ts and we have conflict check when commit.

Possible Implementation

Main idea: Regard ingesting SST files as writing large range of keys and make the whole process a transaction.

Ingesting files shall be an expensive job.

Time Point: before commit

1. Add files, fetch new file id from LevelsController then move or copy files to DB dir with allocated file id(name).
2. Verify file checksum (optional) and get file infomation (just open Table?).
3. Make a new transaction, mark update as true (add a ingest flag for transaction?).
4. (TBD) other checks…

Check files before starting a txn.

Time Point: commit

1. Same as current, but add ingest range field to CommittedTxn in CommitInfo which contains smallest and largest keys in each ingest files.
   This is for fast but inaccurate conflict check, otherwise we need to calc every key hash in SST which really takes time.
2. Set commit_ts as Table's global_version
3. Send to write channel as usual (see below)
4. Process Ingest, find suitable level for each files. (Which will take write lock of LevelHandler , and will block read process of other txn)
5. Write manifest file. The file's global_version is stored in manifest.

A question here. I see we protect the order of Requests send to write channel the same as commit_ts and I wonder why? For WAL in order?

If ingest job can be out-of-order, I think it's possible and better to make ingest job running in current thread and not sending to write channel.

You can see that the whole commit process has only one small I/O — append to manifest. But when picking level, it takes time if there are many tables in this level and this happens under a write lock of LevelHandler .

Conflict Check

There are two ways for conflict check as I described in TP: commit (step 1) .

1. Calculate every key's hash in ingested files and then conflict check works as same right now. For performance problem, we can calculate key hash before transaction.
2. Add ingest range field to CommittedTxn in CommitInfo .
   And for any update txn, when adding read keys, mark smallest and largest read key for this txn. When checking conflict, beside checking hash, also check if read range overlap with ingest range.
   This is really inaccurate and will cause many txn conflict but I think it makes sense.

The second way needs to refactor transaction much more.

(A more simple way is to break ACID…hhh)

Global Version

This concept is the same as rocksdb. For ingested files, all keys has wrong version and it's unreasonable to modify each. When file has a global version, every key in this file has this version.

In latest rocksdb, this value is stored in manifest to avoid random write to ingested files.

We need to add a field in meta.proto and update Table iterator (also block iterator) to fit this feature.

global_version may have ambiguity. Maybe another name when impl.

Discuss

1. A new transaction struct only for ingest or refactor current?
2. Which way to check conflict? (or any better idea?)
3. Send to write channel or done in current thread?
4. Any good idea to impliment this feature?
5. Strategy to pick level. I think from the bottom to check if there is overlap is nice.
6. What if external files (in one job) has overlap range? Rocksdb will put all files at level0.

Remove temporary directories in test

This can leave dirty file in temporary directory on Windows.

Originally posted by @BusyJay in #26 (comment)

[dev] compaction deadlock

compact #1 on level 0
5976
error when fill L0 to Lbase CustomError("KeyRange { left: b\"00000e2e\\0\\0\\0\\0\\0\\0\\0\\0\", right: b\"00000eb3\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\", inf: false } overlap with this level [KeyRange { left: b\"\", right: b\"\", inf: true }]")
5977
error while compaction: CustomError("only compactor zero can compact L0 to L0")
5978
compact #1 on level 0
5979
error when fill L0 to Lbase CustomError("KeyRange { left: b\"00000e2e\\0\\0\\0\\0\\0\\0\\0\\0\", right: b\"00000eb3\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\", inf: false } overlap with this level [KeyRange { left: b\"\", right: b\"\", inf: true }]")

https://github.com/tikv/agatedb/pull/73/checks?check_run_id=1484484527

Proposal: a better structure for LevelHandler

Problem

LevelHandler will re-sort the whole sst file in one level. But if there are 100K files in one level, once sort operation may cost 20~100ms (I bench it in my macbook 2020). And if there are many compaction jobs running together, they may block the read thread too long.

Solution

I propose a two-level b+ tree for LevelHandler and it will split the sst into much page. Every time it changed it only copy the origin page and update one row of it, and then replace it in a short time with holding mutex. I think two-level is enough because sort one thousand string can be finished in 1ms.

Remove unused functions from binary.rs

#29

Transaction 的 set(key,value) 不会增加版本号，当前进程读取是新数据，关闭进程后重新读取就会读取旧数据

Transaction 的 set(key,value) 不会增加版本号，当前进程读取是新数据，关闭进程后重新读取就会读取旧数据

读取都是用的

pub fn get(&self, key: impl Into<BytesMut>) -> Result<Value> {
    let key = key_with_ts(key.into(), std::u64::MAX);
    self.core.get(&key)
}

(clippy::uninit_assumed_init): this call for this type may be undefined behavior

agatedb/src/value.rs

Line 129 in 4aa6420

let mut arr: [u8; 12] = unsafe { MaybeUninit::uninit().assume_init() };

should change to

let mut arr: [MaybeUninit<u8>; 12] = unsafe { MaybeUninit::uninit().assume_init() };

or

let mut arr = [0u8; 12];

---

error: this call for this type may be undefined behavior
   --> src/value.rs:129:42
    |
129 |         let mut arr: [u8; 12] = unsafe { MaybeUninit::uninit().assume_init() };
    |                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: `#[deny(clippy::uninit_assumed_init)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#uninit_assumed_init

feature request: LSM-tree support split/merge

LSM-tree support split/merge.

Integrate with TiKV

Development Task

Integrate with TiKV (3-4 weeks)

Development Progress

TBD

[dev] ts cannot be 0

Current logic makes key with timestamp 0 not accessible. Will investigate into timestamp-related logic.

为什么db.update中set的key-value不生效?

运行程序 https://github.com/rmw-lib/agatedb/tree/cli/examples

    db.update(|tx| {
        use agatedb::{key_with_ts, util::unix_time};
        use bytes::BytesMut;
        let key = key_with_ts(BytesMut::from("3"), unix_time());
        tx.set(key, "2".into())
    })?;

然后读取，读取不到

但是这样写入就可以读取到

    pub fn set(&self, key: impl Into<BytesMut>, value: impl Into<Bytes>) -> Result<()> {
        use crate::util::unix_time;
        let key = key_with_ts(key.into(), unix_time());
        let req = Request {
            entries: vec![Entry::new(key, value.into())],
            ptrs: vec![],
            done: None,
        };
        self.write_to_lsm(req)
    }

get函数定义如下

    pub fn get(&self, key: impl Into<BytesMut>) -> Result<Value> {
        let key = key_with_ts(key.into(), std::u64::MAX);
        self.core.get(&key)
    }

运行演示 截图如下

write_to_lsm我感觉是一个写入磁盘操作，是不是Transaction执行之后也需要手工运行个什么来写入磁盘？难道数据库被drop的时候不会自动写入磁盘吗？

我记得redis和rocksdb都一个设置，可以配置落盘的周期，比如让程序每3秒钟落盘一次，有这样的配置吗？

Feature Reqeust: shard LRU list

AgateDB keep all files open, both SST files and VLog files. If the dataset is large and there are many files, most files may not will touch in a long time. In this scenario we may need a LRU list to maintain those most active files to keep the number of opened files under a specified limit.

Also, after we introduced block cache which used to keep the max memory usage of AgateDB under a limitation to prevent OOM issue, we need the LRU list to maintain those active data blocks.

[dev] dropPrefixes will cause dead loop

Should iter.next before continue.

https://github.com/tikv/agatedb/blob/develop/src/db.rs#L537

(thanks @soundOfDestiny !)

Running tests with code coverage is endless after switch compaction on

#148 start the background compaction threads, but seems this will cause the "Running Tests With Code Coverage" endless.

Feature Request: async IO (io_uring on cloud block storage)

Async IO is an effective mechanism to improve the IO utilization by using few threads. When comes to cloud disk, aka cloud block storage, the IO latency is higher than local disk, if using synchronous read/write in that situation, threads will be blocked longer. And this will lower the CPU utilization. So async IO is required more necessary when come to cloud disk.

https://kernel.dk/io_uring.pdf io_uring is a comprehensive async IO interface of Linux, AgateDB can introduce the async IO mechanism powered by lower-level io_uring.

Developing roadmap

Stage 1: Finish basic implementation

• Merge the develop branch to master, and reach the same test coverage ratio as bager
  • #125
  • introduce code-cov bot #131
• Construct a comparable benchmark between RocksDB and agatedb
  • port https://github.com/dgraph-io/badger-bench #169
  • continuing benchmark #173

Stage 2: Integrate to TiKV

• Refine TiKV engine trait
• Support features that agatedb doesn't have
  • column family
  • manual compact range
  • ingest
  • user property

Stage 3: Compatibile to old TiKV

• RocksDB SST format compatibility
• Support upgrade from RocksDB directly(reading RocksDB manifest at initial)

Stage 4: Optimization

• User timestamp support
• Async IO support

[dev] lock order inversion in LevelsController

==================
230
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16915)
231
  Cycle in lock order graph: M354 (0x7b1400005000) => M352 (0x7b14000050a0) => M354
232

233
  Mutex M352 acquired here while holding mutex M354 in thread T17:
234
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
235
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
236
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
237
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
238
    #4 agatedb::levels::Core::pick_compact_levels /home/runner/work/agatedb/agatedb/src/levels.rs:659 (agatedb-2d8b3cfdced45fcd+0x2505bb)
239
    #5 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:717 (agatedb-2d8b3cfdced45fcd+0x24174f)
240
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
241
    #7 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
242
    #8 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
243
    #9 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
244
    #10 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
245
    #11 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
246
    #12 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
247
    #13 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
248
    #14 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
249
    #15 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
250
    #16 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
251
    #17 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
252
    #18 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
253
    #19 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
254
    #20 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
255
    #21 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
256
    #22 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
257
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
258
    #24 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
259

260
    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
261

262
  Mutex M354 acquired here while holding mutex M352 in thread T17:
263
    #0 pthread_rwlock_wrlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1365 (agatedb-2d8b3cfdced45fcd+0x400f6)
264
    #1 std::sys::unix::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 (agatedb-2d8b3cfdced45fcd+0x1f016e)
265
    #2 std::sys_common::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:48 (agatedb-2d8b3cfdced45fcd+0x1d80d7)
266
    #3 std::sync::rwlock::RwLock<T>::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:264 (agatedb-2d8b3cfdced45fcd+0x198471)
267
    #4 agatedb::levels::Core::fill_tables_l0_to_lbase /home/runner/work/agatedb/agatedb/src/levels.rs:163 (agatedb-2d8b3cfdced45fcd+0x248cad)
268
    #5 agatedb::levels::Core::fill_tables_l0 /home/runner/work/agatedb/agatedb/src/levels.rs:224 (agatedb-2d8b3cfdced45fcd+0x249fc7)
269
    #6 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:558 (agatedb-2d8b3cfdced45fcd+0x24e97c)
270
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
271
    #8 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
272
    #9 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
273
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
274
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
275
    #12 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
276
    #13 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
277
    #14 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
278
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
279
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
280
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
281
    #18 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
282
    #19 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
283
    #20 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
284
    #21 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
285
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
286
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
287
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
288
    #25 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
289
    #26 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
290

291
  Thread T17 'agatedb-1' (tid=16933, running) created by thread T15 at:
292
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-2d8b3cfdced45fcd+0x3da7b)
293
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-2d8b3cfdced45fcd+0x67582b)
294
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-2d8b3cfdced45fcd+0x440041)
295
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-2d8b3cfdced45fcd+0x440a6a)
296
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-2d8b3cfdced45fcd+0x4671b6)
297
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-2d8b3cfdced45fcd+0x467d91)
298
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-2d8b3cfdced45fcd+0x467c8c)
299
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:72 (agatedb-2d8b3cfdced45fcd+0x17e172)
300
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:233 (agatedb-2d8b3cfdced45fcd+0x137bd4)
301
    #9 agatedb::db::tests::with_agate_test::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:618 (agatedb-2d8b3cfdced45fcd+0x1f3ace)
302
    #10 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x252f49)
303
    #11 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x259779)
304
    #12 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x122919)
305
    #13 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0xc695a)
306
    #14 __rust_try 1dzu5xa3oeqaq90p:? (agatedb-2d8b3cfdced45fcd+0xc744b)
307
    #15 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0xc66c8)
308
    #16 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x122b79)
309
    #17 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x259019)
310
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b797)
311
    #19 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
312
    #20 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
313
    #21 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
314

315
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 in std::sys::unix::rwlock::RWLock::read
316
==================
317
==================
318
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16915)
319
  Cycle in lock order graph: M353 (0x7b1400005230) => M354 (0x7b1400005000) => M353
320

321
  Mutex M354 acquired here while holding mutex M353 in thread T17:
322
    #0 pthread_rwlock_wrlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1365 (agatedb-2d8b3cfdced45fcd+0x400f6)
323
    #1 std::sys::unix::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 (agatedb-2d8b3cfdced45fcd+0x1f016e)
324
    #2 std::sys_common::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:48 (agatedb-2d8b3cfdced45fcd+0x1d80d7)
325
    #3 std::sync::rwlock::RwLock<T>::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:264 (agatedb-2d8b3cfdced45fcd+0x198471)
326
    #4 agatedb::levels::Core::fill_tables_l0_to_lbase /home/runner/work/agatedb/agatedb/src/levels.rs:163 (agatedb-2d8b3cfdced45fcd+0x248cad)
327
    #5 agatedb::levels::Core::fill_tables_l0 /home/runner/work/agatedb/agatedb/src/levels.rs:224 (agatedb-2d8b3cfdced45fcd+0x249fc7)
328
    #6 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:558 (agatedb-2d8b3cfdced45fcd+0x24e97c)
329
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
330
    #8 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
331
    #9 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
332
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
333
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
334
    #12 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
335
    #13 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
336
    #14 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
337
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
338
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
339
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
340
    #18 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
341
    #19 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
342
    #20 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
343
    #21 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
344
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
345
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
346
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
347
    #25 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
348
    #26 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
349

350
    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
351

352
  Mutex M353 acquired here while holding mutex M354 in thread T17:
353
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
354
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
355
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
356
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
357
    #4 agatedb::levels::compaction::CompactStatus::delete /home/runner/work/agatedb/agatedb/src/levels/compaction.rs:178 (agatedb-2d8b3cfdced45fcd+0xe21db)
358
    #5 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:581 (agatedb-2d8b3cfdced45fcd+0x24f1ad)
359
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
360
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
361
    #8 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
362
    #9 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
363
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
364
    #11 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
365
    #12 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
366
    #13 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
367
    #14 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
368
    #15 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
369
    #16 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
370
    #17 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
371
    #18 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
372
    #19 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
373
    #20 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
374
    #21 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
375
    #22 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
376
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
377
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
378
    #25 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
379

380
  Thread T17 'agatedb-1' (tid=16933, running) created by thread T15 at:
381
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-2d8b3cfdced45fcd+0x3da7b)
382
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-2d8b3cfdced45fcd+0x67582b)
383
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-2d8b3cfdced45fcd+0x440041)
384
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-2d8b3cfdced45fcd+0x440a6a)
385
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-2d8b3cfdced45fcd+0x4671b6)
386
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-2d8b3cfdced45fcd+0x467d91)
387
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-2d8b3cfdced45fcd+0x467c8c)
388
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:72 (agatedb-2d8b3cfdced45fcd+0x17e172)
389
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:233 (agatedb-2d8b3cfdced45fcd+0x137bd4)
390
    #9 agatedb::db::tests::with_agate_test::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:618 (agatedb-2d8b3cfdced45fcd+0x1f3ace)
391
    #10 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x252f49)
392
    #11 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x259779)
393
    #12 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x122919)
394
    #13 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0xc695a)
395
    #14 __rust_try 1dzu5xa3oeqaq90p:? (agatedb-2d8b3cfdced45fcd+0xc744b)
396
    #15 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0xc66c8)
397
    #16 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x122b79)
398
    #17 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x259019)
399
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b797)
400
    #19 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
401
    #20 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
402
    #21 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
403

404
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 in std::sys::unix::rwlock::RWLock::write
405
==================
406
==================
407
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16915)
408
  Cycle in lock order graph: M354 (0x7b1400005000) => M353 (0x7b1400005230) => M354
409

410
  Mutex M353 acquired here while holding mutex M354 in thread T17:
411
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
412
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
413
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
414
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
415
    #4 agatedb::levels::compaction::CompactStatus::delete /home/runner/work/agatedb/agatedb/src/levels/compaction.rs:178 (agatedb-2d8b3cfdced45fcd+0xe21db)
416
    #5 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:581 (agatedb-2d8b3cfdced45fcd+0x24f1ad)
417
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
418
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
419
    #8 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
420
    #9 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
421
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
422
    #11 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
423
    #12 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
424
    #13 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
425
    #14 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
426
    #15 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
427
    #16 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
428
    #17 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
429
    #18 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
430
    #19 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
431
    #20 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
432
    #21 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
433
    #22 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
434
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
435
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
436
    #25 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
437

438
    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
439

440
  Mutex M354 acquired here while holding mutex M353 in thread T17:
441
    #0 pthread_rwlock_wrlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1365 (agatedb-2d8b3cfdced45fcd+0x400f6)
442
    #1 std::sys::unix::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 (agatedb-2d8b3cfdced45fcd+0x1f016e)
443
    #2 std::sys_common::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:48 (agatedb-2d8b3cfdced45fcd+0x1d80d7)
444
    #3 std::sync::rwlock::RwLock<T>::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:264 (agatedb-2d8b3cfdced45fcd+0x198471)
445
    #4 agatedb::levels::Core::fill_tables_l0_to_lbase /home/runner/work/agatedb/agatedb/src/levels.rs:163 (agatedb-2d8b3cfdced45fcd+0x248cad)
446
    #5 agatedb::levels::Core::fill_tables_l0 /home/runner/work/agatedb/agatedb/src/levels.rs:224 (agatedb-2d8b3cfdced45fcd+0x249fc7)
447
    #6 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:558 (agatedb-2d8b3cfdced45fcd+0x24e97c)
448
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
449
    #8 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
450
    #9 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
451
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
452
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
453
    #12 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
454
    #13 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
455
    #14 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
456
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
457
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
458
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
459
    #18 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
460
    #19 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
461
    #20 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
462
    #21 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
463
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
464
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
465
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
466
    #25 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
467
    #26 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
468

469
  Thread T17 'agatedb-1' (tid=16933, running) created by thread T15 at:
470
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-2d8b3cfdced45fcd+0x3da7b)
471
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-2d8b3cfdced45fcd+0x67582b)
472
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-2d8b3cfdced45fcd+0x440041)
473
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-2d8b3cfdced45fcd+0x440a6a)
474
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-2d8b3cfdced45fcd+0x4671b6)
475
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-2d8b3cfdced45fcd+0x467d91)
476
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-2d8b3cfdced45fcd+0x467c8c)
477
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:72 (agatedb-2d8b3cfdced45fcd+0x17e172)
478
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:233 (agatedb-2d8b3cfdced45fcd+0x137bd4)
479
    #9 agatedb::db::tests::with_agate_test::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:618 (agatedb-2d8b3cfdced45fcd+0x1f3ace)
480
    #10 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x252f49)
481
    #11 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x259779)
482
    #12 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x122919)
483
    #13 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0xc695a)
484
    #14 __rust_try 1dzu5xa3oeqaq90p:? (agatedb-2d8b3cfdced45fcd+0xc744b)
485
    #15 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0xc66c8)
486
    #16 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x122b79)
487
    #17 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x259019)
488
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b797)
489
    #19 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
490
    #20 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
491
    #21 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
492

493
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 in std::sys::unix::rwlock::RWLock::read
494
==================
495
==================
496
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16915)
497
  Cycle in lock order graph: M352 (0x7b14000050a0) => M354 (0x7b1400005000) => M359 (0x7b1400005370) => M352
498

499
  Mutex M354 acquired here while holding mutex M352 in thread T17:
500
    #0 pthread_rwlock_wrlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1365 (agatedb-2d8b3cfdced45fcd+0x400f6)
501
    #1 std::sys::unix::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 (agatedb-2d8b3cfdced45fcd+0x1f016e)
502
    #2 std::sys_common::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:48 (agatedb-2d8b3cfdced45fcd+0x1d80d7)
503
    #3 std::sync::rwlock::RwLock<T>::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:264 (agatedb-2d8b3cfdced45fcd+0x198471)
504
    #4 agatedb::levels::Core::fill_tables_l0_to_lbase /home/runner/work/agatedb/agatedb/src/levels.rs:163 (agatedb-2d8b3cfdced45fcd+0x248cad)
505
    #5 agatedb::levels::Core::fill_tables_l0 /home/runner/work/agatedb/agatedb/src/levels.rs:224 (agatedb-2d8b3cfdced45fcd+0x249fc7)
506
    #6 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:558 (agatedb-2d8b3cfdced45fcd+0x24e97c)
507
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
508
    #8 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
509
    #9 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
510
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
511
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
512
    #12 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
513
    #13 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
514
    #14 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
515
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
516
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
517
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
518
    #18 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
519
    #19 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
520
    #20 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
521
    #21 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
522
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
523
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
524
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
525
    #25 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
526
    #26 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
527

528
    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
529

530
  Mutex M359 acquired here while holding mutex M354 in thread T17:
531
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
532
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
533
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
534
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
535
    #4 agatedb::levels::Core::pick_compact_levels /home/runner/work/agatedb/agatedb/src/levels.rs:659 (agatedb-2d8b3cfdced45fcd+0x2505bb)
536
    #5 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:717 (agatedb-2d8b3cfdced45fcd+0x24174f)
537
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
538
    #7 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
539
    #8 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
540
    #9 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
541
    #10 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
542
    #11 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
543
    #12 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
544
    #13 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
545
    #14 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
546
    #15 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
547
    #16 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
548
    #17 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
549
    #18 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
550
    #19 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
551
    #20 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
552
    #21 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
553
    #22 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
554
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
555
    #24 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
556

557
  Mutex M352 acquired here while holding mutex M359 in thread T17:
558
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
559
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
560
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
561
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
562
    #4 agatedb::levels::Core::fill_tables /home/runner/work/agatedb/agatedb/src/levels.rs:233 (agatedb-2d8b3cfdced45fcd+0x24a272)
563
    #5 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:571 (agatedb-2d8b3cfdced45fcd+0x24ed9d)
564
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
565
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
566
    #8 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
567
    #9 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
568
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
569
    #11 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
570
    #12 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
571
    #13 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
572
    #14 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
573
    #15 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
574
    #16 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
575
    #17 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
576
    #18 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
577
    #19 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
578
    #20 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
579
    #21 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
580
    #22 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
581
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
582
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
583
    #25 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
584

585
  Thread T17 'agatedb-1' (tid=16933, running) created by thread T15 at:
586
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-2d8b3cfdced45fcd+0x3da7b)
587
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-2d8b3cfdced45fcd+0x67582b)
588
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-2d8b3cfdced45fcd+0x440041)
589
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-2d8b3cfdced45fcd+0x440a6a)
590
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-2d8b3cfdced45fcd+0x4671b6)
591
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-2d8b3cfdced45fcd+0x467d91)
592
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-2d8b3cfdced45fcd+0x467c8c)
593
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:72 (agatedb-2d8b3cfdced45fcd+0x17e172)
594
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:233 (agatedb-2d8b3cfdced45fcd+0x137bd4)
595
    #9 agatedb::db::tests::with_agate_test::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:618 (agatedb-2d8b3cfdced45fcd+0x1f3ace)
596
    #10 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x252f49)
597
    #11 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x259779)
598
    #12 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x122919)
599
    #13 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0xc695a)
600
    #14 __rust_try 1dzu5xa3oeqaq90p:? (agatedb-2d8b3cfdced45fcd+0xc744b)
601
    #15 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0xc66c8)
602
    #16 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x122b79)
603
    #17 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x259019)
604
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b797)
605
    #19 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
606
    #20 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
607
    #21 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
608

609
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 in std::sys::unix::rwlock::RWLock::write
610
==================
611
==================
612
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16915)
613
  Cycle in lock order graph: M344 (0x7b1400002b20) => M342 (0x7b1400002ad0) => M344
614

615
  Mutex M342 acquired here while holding mutex M344 in thread T14:
616
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
617
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
618
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
619
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
620
    #4 agatedb::levels::Core::pick_compact_levels /home/runner/work/agatedb/agatedb/src/levels.rs:659 (agatedb-2d8b3cfdced45fcd+0x2505bb)
621
    #5 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:717 (agatedb-2d8b3cfdced45fcd+0x24174f)
622
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
623
    #7 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
624
    #8 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
625
    #9 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
626
    #10 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
627
    #11 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
628
    #12 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
629
    #13 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
630
    #14 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
631
    #15 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
632
    #16 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
633
    #17 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
634
    #18 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
635
    #19 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
636
    #20 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
637
    #21 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
638
    #22 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
639
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
640
    #24 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
641

642
    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
643

644
  Mutex M344 acquired here while holding mutex M342 in thread T14:
645
    #0 pthread_rwlock_wrlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1365 (agatedb-2d8b3cfdced45fcd+0x400f6)
646
    #1 std::sys::unix::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 (agatedb-2d8b3cfdced45fcd+0x1f016e)
647
    #2 std::sys_common::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:48 (agatedb-2d8b3cfdced45fcd+0x1d80d7)
648
    #3 std::sync::rwlock::RwLock<T>::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:264 (agatedb-2d8b3cfdced45fcd+0x198471)
649
    #4 agatedb::levels::Core::fill_tables_l0_to_lbase /home/runner/work/agatedb/agatedb/src/levels.rs:163 (agatedb-2d8b3cfdced45fcd+0x248cad)
650
    #5 agatedb::levels::Core::fill_tables_l0 /home/runner/work/agatedb/agatedb/src/levels.rs:224 (agatedb-2d8b3cfdced45fcd+0x249fc7)
651
    #6 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:558 (agatedb-2d8b3cfdced45fcd+0x24e97c)
652
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
653
    #8 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
654
    #9 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
655
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
656
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
657
    #12 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
658
    #13 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
659
    #14 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
660
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
661
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
662
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
663
    #18 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
664
    #19 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
665
    #20 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
666
    #21 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
667
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
668
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
669
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
670
    #25 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
671
    #26 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
672

673
  Thread T14 'agatedb-1' (tid=16930, running) created by thread T11 at:
674
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-2d8b3cfdced45fcd+0x3da7b)
675
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-2d8b3cfdced45fcd+0x67582b)
676
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-2d8b3cfdced45fcd+0x440041)
677
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-2d8b3cfdced45fcd+0x440a6a)
678
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-2d8b3cfdced45fcd+0x4671b6)
679
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-2d8b3cfdced45fcd+0x467d91)
680
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-2d8b3cfdced45fcd+0x467c8c)
681
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:72 (agatedb-2d8b3cfdced45fcd+0x17e172)
682
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:233 (agatedb-2d8b3cfdced45fcd+0x137bd4)
683
    #9 agatedb::db::tests::with_agate_test::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:618 (agatedb-2d8b3cfdced45fcd+0x1f404e)
684
    #10 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x252ea9)
685
    #11 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x2597e9)
686
    #12 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x122989)
687
    #13 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0xc6b3a)
688
    #14 __rust_try 1dzu5xa3oeqaq90p:? (agatedb-2d8b3cfdced45fcd+0xc744b)
689
    #15 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0xc6278)
690
    #16 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x122bf9)
691
    #17 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x258bf9)
692
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b6f7)
693
    #19 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
694
    #20 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
695
    #21 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
696

697
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 in std::sys::unix::rwlock::RWLock::read
698
==================
699
==================
700
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16915)
701
  Cycle in lock order graph: M343 (0x7b14000028f0) => M344 (0x7b1400002b20) => M343
702

703
  Mutex M344 acquired here while holding mutex M343 in thread T14:
704
    #0 pthread_rwlock_wrlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1365 (agatedb-2d8b3cfdced45fcd+0x400f6)
705
    #1 std::sys::unix::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 (agatedb-2d8b3cfdced45fcd+0x1f016e)
706
    #2 std::sys_common::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:48 (agatedb-2d8b3cfdced45fcd+0x1d80d7)
707
    #3 std::sync::rwlock::RwLock<T>::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:264 (agatedb-2d8b3cfdced45fcd+0x198471)
708
    #4 agatedb::levels::Core::fill_tables_l0_to_lbase /home/runner/work/agatedb/agatedb/src/levels.rs:163 (agatedb-2d8b3cfdced45fcd+0x248cad)
709
    #5 agatedb::levels::Core::fill_tables_l0 /home/runner/work/agatedb/agatedb/src/levels.rs:224 (agatedb-2d8b3cfdced45fcd+0x249fc7)
710
    #6 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:558 (agatedb-2d8b3cfdced45fcd+0x24e97c)
711
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
712
    #8 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
713
    #9 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
714
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
715
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
716
    #12 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
717
    #13 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
718
    #14 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
719
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
720
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
721
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
722
    #18 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
723
    #19 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
724
    #20 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
725
    #21 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
726
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
727
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
728
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
729
    #25 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
730
    #26 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
731

732
    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
733

734
  Mutex M343 acquired here while holding mutex M344 in thread T14:
735
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
736
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
737
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
738
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
739
    #4 agatedb::levels::compaction::CompactStatus::delete /home/runner/work/agatedb/agatedb/src/levels/compaction.rs:178 (agatedb-2d8b3cfdced45fcd+0xe21db)
740
    #5 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:581 (agatedb-2d8b3cfdced45fcd+0x24f1ad)
741
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
742
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
743
    #8 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
744
    #9 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
745
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
746
    #11 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
747
    #12 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
748
    #13 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
749
    #14 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
750
    #15 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
751
    #16 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
752
    #17 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
753
    #18 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
754
    #19 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
755
    #20 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
756
    #21 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
757
    #22 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
758
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
759
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
760
    #25 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
761

762
  Thread T14 'agatedb-1' (tid=16930, running) created by thread T11 at:
763
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-2d8b3cfdced45fcd+0x3da7b)
764
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-2d8b3cfdced45fcd+0x67582b)
765
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-2d8b3cfdced45fcd+0x440041)
766
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-2d8b3cfdced45fcd+0x440a6a)
767
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-2d8b3cfdced45fcd+0x4671b6)
768
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-2d8b3cfdced45fcd+0x467d91)
769
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-2d8b3cfdced45fcd+0x467c8c)
770
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:72 (agatedb-2d8b3cfdced45fcd+0x17e172)
771
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:233 (agatedb-2d8b3cfdced45fcd+0x137bd4)
772
    #9 agatedb::db::tests::with_agate_test::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:618 (agatedb-2d8b3cfdced45fcd+0x1f404e)
773
    #10 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x252ea9)
774
    #11 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x2597e9)
775
    #12 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x122989)
776
    #13 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0xc6b3a)
777
    #14 __rust_try 1dzu5xa3oeqaq90p:? (agatedb-2d8b3cfdced45fcd+0xc744b)
778
    #15 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0xc6278)
779
    #16 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x122bf9)
780
    #17 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x258bf9)
781
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b6f7)
782
    #19 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
783
    #20 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
784
    #21 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
785

786
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 in std::sys::unix::rwlock::RWLock::write
787
==================
788
==================
789
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16915)
790
  Cycle in lock order graph: M344 (0x7b1400002b20) => M343 (0x7b14000028f0) => M344
791

792
  Mutex M343 acquired here while holding mutex M344 in thread T14:
793
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
794
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
795
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
796
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
797
    #4 agatedb::levels::compaction::CompactStatus::delete /home/runner/work/agatedb/agatedb/src/levels/compaction.rs:178 (agatedb-2d8b3cfdced45fcd+0xe21db)
798
    #5 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:581 (agatedb-2d8b3cfdced45fcd+0x24f1ad)
799
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
800
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
801
    #8 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
802
    #9 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
803
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
804
    #11 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
805
    #12 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
806
    #13 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
807
    #14 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
808
    #15 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
809
    #16 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
810
    #17 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
811
    #18 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
812
    #19 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
813
    #20 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
814
    #21 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
815
    #22 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
816
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
817
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
818
    #25 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
819

820
    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
821

822
  Mutex M344 acquired here while holding mutex M343 in thread T14:
823
    #0 pthread_rwlock_wrlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1365 (agatedb-2d8b3cfdced45fcd+0x400f6)
824
    #1 std::sys::unix::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 (agatedb-2d8b3cfdced45fcd+0x1f016e)
825
    #2 std::sys_common::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:48 (agatedb-2d8b3cfdced45fcd+0x1d80d7)
826
    #3 std::sync::rwlock::RwLock<T>::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:264 (agatedb-2d8b3cfdced45fcd+0x198471)
827
    #4 agatedb::levels::Core::fill_tables_l0_to_lbase /home/runner/work/agatedb/agatedb/src/levels.rs:163 (agatedb-2d8b3cfdced45fcd+0x248cad)
828
    #5 agatedb::levels::Core::fill_tables_l0 /home/runner/work/agatedb/agatedb/src/levels.rs:224 (agatedb-2d8b3cfdced45fcd+0x249fc7)
829
    #6 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:558 (agatedb-2d8b3cfdced45fcd+0x24e97c)
830
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
831
    #8 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
832
    #9 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
833
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
834
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
835
    #12 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
836
    #13 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
837
    #14 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
838
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
839
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
840
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
841
    #18 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
842
    #19 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
843
    #20 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
844
    #21 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
845
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
846
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
847
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
848
    #25 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
849
    #26 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
850

851
  Thread T14 'agatedb-1' (tid=16930, running) created by thread T11 at:
852
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-2d8b3cfdced45fcd+0x3da7b)
853
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-2d8b3cfdced45fcd+0x67582b)
854
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-2d8b3cfdced45fcd+0x440041)
855
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-2d8b3cfdced45fcd+0x440a6a)
856
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-2d8b3cfdced45fcd+0x4671b6)
857
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-2d8b3cfdced45fcd+0x467d91)
858
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-2d8b3cfdced45fcd+0x467c8c)
859
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:72 (agatedb-2d8b3cfdced45fcd+0x17e172)
860
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:233 (agatedb-2d8b3cfdced45fcd+0x137bd4)
861
    #9 agatedb::db::tests::with_agate_test::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:618 (agatedb-2d8b3cfdced45fcd+0x1f404e)
862
    #10 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x252ea9)
863
    #11 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x2597e9)
864
    #12 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x122989)
865
    #13 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0xc6b3a)
866
    #14 __rust_try 1dzu5xa3oeqaq90p:? (agatedb-2d8b3cfdced45fcd+0xc744b)
867
    #15 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0xc6278)
868
    #16 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x122bf9)
869
    #17 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x258bf9)
870
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b6f7)
871
    #19 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
872
    #20 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
873
    #21 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
874

875
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 in std::sys::unix::rwlock::RWLock::read
876
==================
877
==================
878
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16915)
879
  Cycle in lock order graph: M342 (0x7b1400002ad0) => M344 (0x7b1400002b20) => M349 (0x7b1400002a80) => M342
880

881
  Mutex M344 acquired here while holding mutex M342 in thread T14:
882
    #0 pthread_rwlock_wrlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1365 (agatedb-2d8b3cfdced45fcd+0x400f6)
883
    #1 std::sys::unix::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 (agatedb-2d8b3cfdced45fcd+0x1f016e)
884
    #2 std::sys_common::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:48 (agatedb-2d8b3cfdced45fcd+0x1d80d7)
885
    #3 std::sync::rwlock::RwLock<T>::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:264 (agatedb-2d8b3cfdced45fcd+0x198471)
886
    #4 agatedb::levels::Core::fill_tables_l0_to_lbase /home/runner/work/agatedb/agatedb/src/levels.rs:163 (agatedb-2d8b3cfdced45fcd+0x248cad)
887
    #5 agatedb::levels::Core::fill_tables_l0 /home/runner/work/agatedb/agatedb/src/levels.rs:224 (agatedb-2d8b3cfdced45fcd+0x249fc7)
888
    #6 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:558 (agatedb-2d8b3cfdced45fcd+0x24e97c)
889
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
890
    #8 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
891
    #9 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
892
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
893
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
894
    #12 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
895
    #13 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
896
    #14 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
897
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
898
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
899
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
900
    #18 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
901
    #19 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
902
    #20 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
903
    #21 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
904
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
905
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
906
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
907
    #25 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
908
    #26 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
909

910
    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
911

912
  Mutex M349 acquired here while holding mutex M344 in thread T14:
913
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
914
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
915
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
916
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
917
    #4 agatedb::levels::Core::pick_compact_levels /home/runner/work/agatedb/agatedb/src/levels.rs:659 (agatedb-2d8b3cfdced45fcd+0x2505bb)
918
    #5 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:717 (agatedb-2d8b3cfdced45fcd+0x24174f)
919
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
920
    #7 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
921
    #8 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
922
    #9 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
923
    #10 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
924
    #11 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
925
    #12 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
926
    #13 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
927
    #14 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
928
    #15 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
929
    #16 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
930
    #17 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
931
    #18 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
932
    #19 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
933
    #20 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
934
    #21 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
935
    #22 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
936
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
937
    #24 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
938

939
  Mutex M342 acquired here while holding mutex M349 in thread T14:
940
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
941
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
942
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
943
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
944
    #4 agatedb::levels::Core::fill_tables /home/runner/work/agatedb/agatedb/src/levels.rs:233 (agatedb-2d8b3cfdced45fcd+0x24a272)
945
    #5 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:571 (agatedb-2d8b3cfdced45fcd+0x24ed9d)
946
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
947
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
948
    #8 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
949
    #9 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
950
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
951
    #11 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
952
    #12 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
953
    #13 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
954
    #14 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
955
    #15 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
956
    #16 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
957
    #17 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
958
    #18 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
959
    #19 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
960
    #20 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
961
    #21 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
962
    #22 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
963
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
964
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
965
    #25 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
966

967
  Thread T14 'agatedb-1' (tid=16930, running) created by thread T11 at:
968
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-2d8b3cfdced45fcd+0x3da7b)
969
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-2d8b3cfdced45fcd+0x67582b)
970
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-2d8b3cfdced45fcd+0x440041)
971
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-2d8b3cfdced45fcd+0x440a6a)
972
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-2d8b3cfdced45fcd+0x4671b6)
973
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-2d8b3cfdced45fcd+0x467d91)
974
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-2d8b3cfdced45fcd+0x467c8c)
975
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:72 (agatedb-2d8b3cfdced45fcd+0x17e172)
976
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:233 (agatedb-2d8b3cfdced45fcd+0x137bd4)
977
    #9 agatedb::db::tests::with_agate_test::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:618 (agatedb-2d8b3cfdced45fcd+0x1f404e)
978
    #10 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x252ea9)
979
    #11 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x2597e9)
980
    #12 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x122989)
981
    #13 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0xc6b3a)
982
    #14 __rust_try 1dzu5xa3oeqaq90p:? (agatedb-2d8b3cfdced45fcd+0xc744b)
983
    #15 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0xc6278)
984
    #16 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x122bf9)
985
    #17 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x258bf9)
986
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b6f7)
987
    #19 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
988
    #20 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
989
    #21 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
990

991
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 in std::sys::unix::rwlock::RWLock::write
992
==================
993
test db::tests::test_flush_memtable ... ok
994
==================
995
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16915)
996
  Cycle in lock order graph: M344 (0x7b1400002b20) => M343 (0x7b14000028f0) => M344
997

998
  Mutex M343 acquired here while holding mutex M344 in thread T14:
999
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
1000
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
1001
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
1002
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
1003
    #4 agatedb::levels::compaction::CompactStatus::delete /home/runner/work/agatedb/agatedb/src/levels/compaction.rs:178 (agatedb-2d8b3cfdced45fcd+0xe21db)
1004
    #5 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:581 (agatedb-2d8b3cfdced45fcd+0x24f1ad)
1005
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
1006
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
1007
    #8 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
1008
    #9 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
1009
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
1010
    #11 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
1011
    #12 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
1012
    #13 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
1013
    #14 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
1014
    #15 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
1015
    #16 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
1016
    #17 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
1017
    #18 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
1018
    #19 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
1019
    #20 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
1020
    #21 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
1021
    #22 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
1022
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
1023
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
1024
    #25 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
1025

1026
    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
1027

1028
  Mutex M344 acquired here while holding mutex M343 in thread T14:
1029
    #0 pthread_rwlock_wrlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1365 (agatedb-2d8b3cfdced45fcd+0x400f6)
1030
    #1 std::sys::unix::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 (agatedb-2d8b3cfdced45fcd+0x1f016e)
1031
    #2 std::sys_common::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:48 (agatedb-2d8b3cfdced45fcd+0x1d80d7)
1032
    #3 std::sync::rwlock::RwLock<T>::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:264 (agatedb-2d8b3cfdced45fcd+0x198471)
1033
    #4 agatedb::levels::Core::fill_tables_l0_to_lbase /home/runner/work/agatedb/agatedb/src/levels.rs:163 (agatedb-2d8b3cfdced45fcd+0x248cad)
1034
    #5 agatedb::levels::Core::fill_tables_l0 /home/runner/work/agatedb/agatedb/src/levels.rs:224 (agatedb-2d8b3cfdced45fcd+0x249fc7)
1035
    #6 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:558 (agatedb-2d8b3cfdced45fcd+0x24e97c)
1036
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
1037
    #8 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
1038
    #9 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
1039
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
1040
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
1041
    #12 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
1042
    #13 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
1043
    #14 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
1044
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
1045
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
1046
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
1047
    #18 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
1048
    #19 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
1049
    #20 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
1050
    #21 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
1051
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
1052
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
1053
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
1054
    #25 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
1055
    #26 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
1056

1057
  Thread T14 'agatedb-1' (tid=16930, running) created by thread T11 at:
1058
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-2d8b3cfdced45fcd+0x3da7b)
1059
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-2d8b3cfdced45fcd+0x67582b)
1060
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-2d8b3cfdced45fcd+0x440041)
1061
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-2d8b3cfdced45fcd+0x440a6a)
1062
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-2d8b3cfdced45fcd+0x4671b6)
1063
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-2d8b3cfdced45fcd+0x467d91)
1064
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-2d8b3cfdced45fcd+0x467c8c)
1065
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:72 (agatedb-2d8b3cfdced45fcd+0x17e172)
1066
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:233 (agatedb-2d8b3cfdced45fcd+0x137bd4)
1067
    #9 agatedb::db::tests::with_agate_test::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:618 (agatedb-2d8b3cfdced45fcd+0x1f404e)
1068
    #10 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x252ea9)
1069
    #11 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x2597e9)
1070
    #12 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x122989)
1071
    #13 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0xc6b3a)
1072
    #14 __rust_try 1dzu5xa3oeqaq90p:? (agatedb-2d8b3cfdced45fcd+0xc744b)
1073
    #15 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0xc6278)
1074
    #16 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x122bf9)
1075
    #17 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x258bf9)
1076
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b6f7)
1077
    #19 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
1078
    #20 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
1079
    #21 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
1080

1081
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 in std::sys::unix::rwlock::RWLock::read
1082
==================
1083
==================
1084
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16915)
1085
  Cycle in lock order graph: M349 (0x7b1400002a80) => M344 (0x7b1400002b20) => M348 (0x7b1400002a30) => M349
1086

1087
  Mutex M344 acquired here while holding mutex M349 in thread T14:
1088
    #0 pthread_rwlock_wrlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1365 (agatedb-2d8b3cfdced45fcd+0x400f6)
1089
    #1 std::sys::unix::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 (agatedb-2d8b3cfdced45fcd+0x1f016e)
1090
    #2 std::sys_common::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:48 (agatedb-2d8b3cfdced45fcd+0x1d80d7)
1091
    #3 std::sync::rwlock::RwLock<T>::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:264 (agatedb-2d8b3cfdced45fcd+0x198471)
1092
    #4 agatedb::levels::Core::fill_tables_l0_to_lbase /home/runner/work/agatedb/agatedb/src/levels.rs:163 (agatedb-2d8b3cfdced45fcd+0x248cad)
1093
    #5 agatedb::levels::Core::fill_tables_l0 /home/runner/work/agatedb/agatedb/src/levels.rs:224 (agatedb-2d8b3cfdced45fcd+0x249fc7)
1094
    #6 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:558 (agatedb-2d8b3cfdced45fcd+0x24e97c)
1095
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
1096
    #8 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
1097
    #9 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
1098
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
1099
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
1100
    #12 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
1101
    #13 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
1102
    #14 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
1103
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
1104
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
1105
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
1106
    #18 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
1107
    #19 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
1108
    #20 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
1109
    #21 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
1110
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
1111
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
1112
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
1113
    #25 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
1114
    #26 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
1115

1116
    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
1117

1118
  Mutex M348 acquired here while holding mutex M344 in thread T14:
1119
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
1120
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
1121
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
1122
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
1123
    #4 agatedb::levels::Core::pick_compact_levels /home/runner/work/agatedb/agatedb/src/levels.rs:659 (agatedb-2d8b3cfdced45fcd+0x2505bb)
1124
    #5 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:717 (agatedb-2d8b3cfdced45fcd+0x24174f)
1125
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
1126
    #7 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
1127
    #8 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
1128
    #9 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
1129
    #10 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
1130
    #11 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
1131
    #12 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
1132
    #13 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
1133
    #14 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
1134
    #15 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
1135
    #16 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
1136
    #17 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
1137
    #18 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
1138
    #19 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
1139
    #20 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
1140
    #21 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
1141
    #22 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
1142
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
1143
    #24 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
1144

1145
  Mutex M349 acquired here while holding mutex M348 in thread T14:
1146
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
1147
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
1148
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
1149
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
1150
    #4 agatedb::levels::Core::fill_tables /home/runner/work/agatedb/agatedb/src/levels.rs:233 (agatedb-2d8b3cfdced45fcd+0x24a272)
1151
    #5 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:571 (agatedb-2d8b3cfdced45fcd+0x24ed9d)
1152
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
1153
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
1154
    #8 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
1155
    #9 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
1156
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
1157
    #11 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
1158
    #12 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
1159
    #13 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
1160
    #14 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
1161
    #15 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
1162
    #16 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
1163
    #17 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
1164
    #18 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
1165
    #19 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
1166
    #20 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
1167
    #21 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
1168
    #22 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
1169
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
1170
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
1171
    #25 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
1172

1173
  Thread T14 'agatedb-1' (tid=16930, running) created by thread T11 at:
1174
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-2d8b3cfdced45fcd+0x3da7b)
1175
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-2d8b3cfdced45fcd+0x67582b)
1176
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-2d8b3cfdced45fcd+0x440041)
1177
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-2d8b3cfdced45fcd+0x440a6a)
1178
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-2d8b3cfdced45fcd+0x4671b6)
1179
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-2d8b3cfdced45fcd+0x467d91)
1180
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-2d8b3cfdced45fcd+0x467c8c)
1181
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:72 (agatedb-2d8b3cfdced45fcd+0x17e172)
1182
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:233 (agatedb-2d8b3cfdced45fcd+0x137bd4)
1183
    #9 agatedb::db::tests::with_agate_test::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:618 (agatedb-2d8b3cfdced45fcd+0x1f404e)
1184
    #10 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x252ea9)
1185
    #11 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x2597e9)
1186
    #12 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x122989)
1187
    #13 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0xc6b3a)
1188
    #14 __rust_try 1dzu5xa3oeqaq90p:? (agatedb-2d8b3cfdced45fcd+0xc744b)
1189
    #15 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0xc6278)
1190
    #16 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x122bf9)
1191
    #17 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x258bf9)
1192
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b6f7)
1193
    #19 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
1194
    #20 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
1195
    #21 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
1196

1197
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 in std::sys::unix::rwlock::RWLock::write

[Feature Request]: Add some architectural outlook and roadmap

Could we add some roadmap or architectural outlook like sled-architectural-outlook?

Feature Request: bi-direction skiplist

SkipList is a single direction skiplist, it is ok when we retrieve values in ascend order, but it is slow when we call prev to scan the skiplist in descend order.

Descend reading usually happens when we want to read latest records run query like SELECT * FROM t ORDER BY time DESC [LIMIT n] in SQL database. This is a very common scenario. So it is necessary for AgateDB to introduce the bi-direction skiplist to improve the descend reading performance.

Issue template is confusing

As organization template is used by default, the create issue page is really confusing. We could later copy TiKV issue template here.

Segmentation violation on leak sanitizer test

As shown in #143 (comment), we may not pass the leak sanitizer test due to the SEGV error:

Stack trace

test ops::transaction_test::normal_db::test_conflict ... LeakSanitizer:DEADLYSIGNAL
==9609==ERROR: LeakSanitizer: SEGV on unknown address 0x7f7858d75ff8 (pc 0x55cbb28f0094 bp 0x7f780e4c68e0 sp 0x7f780e4c6060 T1023)
==9609==The signal is caused by a READ memory access.
    #0 0x55cbb28f0094  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x86094) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #1 0x55cbb28ea3e2  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x803e2) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #2 0x55cbb28eea2f  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x84a2f) (BuildId: 0[219](https://github.com/tikv/agatedb/runs/6933910360?check_suite_focus=true#step:5:220)e8618c4e0fa0937fde79c12d993f79cbee5f)
    #3 0x55cbb2d502ae  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x4e62ae) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #4 0x55cbb2dd92b1  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x56f2b1) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #5 0x55cbb2ea2709  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x638709) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #6 0x55cbb2ea2b44  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x638b44) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #7 0x55cbb2ea2f1f  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x638f1f) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #8 0x55cbb2e9bcfb  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x631cfb) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #9 0x55cbb2e9b396  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x631396) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #10 0x55cbb2e9aef8  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x630ef8) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #11 0x55cbb2e9aedd  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x630edd) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #12 0x55cbb2e9fc50  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x635c50) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #13 0x55cbb2d22e12  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x4b8e12) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #14 0x55cbb29[228](https://github.com/tikv/agatedb/runs/6933910360?check_suite_focus=true#step:5:229)00  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0xb8800) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #15 0x55cbb2936721  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0xcc721) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #16 0x55cbb2a2772e  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x1bd72e) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #17 0x55cbb2a9b721  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x[231](https://github.com/tikv/agatedb/runs/6933910360?check_suite_focus=true#step:5:232)721) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #18 0x55cbb2ae4714  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x27a714) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #19 0x55cbb299b857  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x131857) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #20 0x55cbb2ae56f7  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x27b6f7) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #21 0x55cbb2a750ef  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x20b0ef) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #22 0x55cbb295821b  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0xee21b) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #23 0x55cbb29fd75f  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x19375f) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #24 0x55cbb2924926  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0xba926) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #25 0x55cbb2925a2a  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0xbba2a) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #26 0x55cbb2923ed0  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0xb9ed0) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #27 0x55cbb296850f  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0xfe50f) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #28 0x55cbb2953fa2  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0xe9fa2) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #29 0x55cbb299a5dd  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x1305dd) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #30 0x55cbb2e4b32c  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x5e132c) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #31 0x55cbb2e4b39f  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x5e139f) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #32 0x55cbb2d52501  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x4e8501) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #33 0x7f7859879608  (/lib/x86_64-linux-gnu/libpthread.so.0+0x8608) (BuildId: 7b4536f41cdaa5888408e82d0836e33dcf436466)
    #34 0x7f7859649132  (/lib/x86_64-linux-gnu/libc.so.6+0x11f132) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee)
LeakSanitizer can not provide additional info.
SUMMARY: LeakSanitizer: SEGV (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x86094) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f) 
==9609==ABORTING
error: test failed, to rerun pass '--lib'
Error: The process '/home/runner/.cargo/bin/cargo' failed with exit code 23

If we test it locally, we could additionally get error information as below.

Additional error information

error: address range table at offset 0x58280 has a premature terminator entry at offset 0x582b0
error: address range table at offset 0x58280 has a premature terminator entry at offset 0x582d0
error: address range table at offset 0x58280 has a premature terminator entry at offset 0x582f0
error: address range table at offset 0x687a0 has a premature terminator entry at offset 0x68860
error: address range table at offset 0x687a0 has a premature terminator entry at offset 0x68880
error: address range table at offset 0x72700 has a premature terminator entry at offset 0x72780
error: address range table at offset 0x72700 has a premature terminator entry at offset 0x727a0
error: address range table at offset 0x73510 has a premature terminator entry at offset 0x73540
error: address range table at offset 0x82280 has a premature terminator entry at offset 0x82290

Current workaround is running tests serially and reduce the concurrency in a certain test.

Not sure the cause of this issue.

Feature Request: cache system with precious memory usage limitation

Currently every sst is opened in mmap mode, we can't control the memory usage of agatedb with this mode, it will use as much as it can. This may cause OOM when agatedb embedded in other system (like in mongo's early days', its mmap engine caused many issues).

Implement a cache system that can control its total memory usage is very important for a storage engine, especially on the cloud environment where the resource is strictly limited. RocksDB's block-cache is good reference, it can limit the total memory usage and provide impressive access performance.

Read lock on LevelHandler while get

When getting value from LSM, agatedb iters on levelhandlers and calls LevelHandler::get() under a RwLockReadGuard which will cause the whole read I/O path on this level under a read lock. It is always not a good idea to do I/O under lock even it is a read lock.

See:

agatedb/src/levels.rs

Line 931 in 475e17b

let v = handler.read()?.get(key);

I think such read lock is only necessary for fetching associated Tables from LevelHandler and the exact read work can be done outside the lock.

This logic is same as what it is in badger.
https://github.com/dgraph-io/badger/blob/7d159dd923ac9e470ffa76365f9b29ccbd038b0d/levels.go#L1602
The comment says it will call h.RLock() and h.RUnlock(), but exactly they are called in getTableForKey() which only locked when getting tables.
https://github.com/dgraph-io/badger/blob/master/level_handler.go#L241-L243

BTW, as Table might be removed (I/O) when the struct drop and this might happened when modifying LevelHandler under write lock. So this might influence read? I' mot sure whether it's right?

Feature Request: get key only

Some times we just want to check if a key is existed in the DB, for example when we call transaction put to insert a row to TiKV we need to check if the key exist(in TiKV we call it write conflict check).

Currently we use get to check if the key exist, but the get will also return the value which will cost extra memcpy. If we can add a key only read option for get, we can eliminate the memcpy of value part.

一个我搞不定的错误 No such file or directory (os error 2)

代码见我的fork的cli分支 https://github.com/rmw-lib/agatedb/tree/cli/examples

我的想法是给 agatedb 写一个命令行工具，也可以当做学习agatedb的demo代码

代码是在 #178 (simple agatedb get put) 基础上继续开发的

但是目前卡在了这个错误上 No such file or directory (os error 2)，因为这个错误没代码行提示，我也搞不清楚为啥出错

截图如下

另外，每次运行 /tmp/adb 下面都会多一个 .mem文件，感觉这也不太正常

Implement Storage Engine

Development Task

Implement Storage Engine, R/W API and Transaction API (3-4 weeks)

Development progress

This section tracks what has been done (on develop branch)

• Implement LSM tree and compaction
  • memtable
  • L0 table
  • multi-levels with simple compaction policy
  • compaction to base
  • persistent database (currently it's just write-only)
    • delete unused files
    • manifest
    • persist files and recover
  • compaction
• Implement value log
• Implement database iterator
• Implement MVCC and Transaction API
• Patches and Fixes
  • implement merge iterator
  • implement concat iterator
  • fix varint

Pull Requests

This section tracks progress on master branch

• memtable #28
  • first, refactor current structures #37
  • WAL #55
  • vlog #82
• varint #29
• merge iterator #30
• concat iterator #65
• extra tests #38

Add sanitizer check

There are some unsafe code in the project, enabling sanitizer check can detect memory issues in the early stage and provide more confidence in memory safety.

This is a guide about how to enable the check: https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/sanitizer.html.

Feature Request: separate filter&index blocks and data blocks into two dedicated files, and SST supports range filter

When scan a range[start, end), LSM-tree based engine will create a iterator for each level and seek to the proper position for the corresponding SST file(whose range[smallest, biggest) cover [start, end) ). This will open the SST file and read some content of this SST even there is no key covered by range[start, end), for example the SST contains keys [a, e, f], and the scan range is [b, c).

This will open and read is not necessary if we can know this SST doesn't contain keys in the scan range. https://www.cs.cmu.edu/~pavlo/papers/mod601-zhangA-hm.pdf described a type named succinct range filter, which can achieve such goal. This can benefit even more when these SST files are stored in remote storage like cloud disk or distributed file system on the cloud.

Let's using the AWS EFS as an example, EFS has active and inactive tiers, it will move these inactive files to the inactive tier automatically to reduce the total storage cost. When the files in inactive tier are requested, EFS will move them to the active tier with some throughput fee. If we want to leverage such fundamental cloud storage, we need to consider how to make good use of its inactive tier to reduce the total storage cost, at the same time, we need to consider how to reduce the data transfer between its inactive tier and active tier.

If we separate the fitler&index block and the data blocks into 2 dedicated files, The index&filter files maybe accessed frequently, and they probably stay in the active tier of EFS, these long-time un-touched data files will stay in the inactive tier of EFS. This works well when the workload is pure point get, because bloom-filter will reduce these unnecessary touching of data files, but when the workload contains range queries, range filter make sense rather than bloom-filter.

create_if_not_exists 参数感觉可以删除掉，目前代码中并没有相关的判断，目录总是创建的。

[dev] pending_write in txn cannot be read

https://github.com/tikv/agatedb/blob/develop/src/ops/transaction.rs#L228

should return Ok when pending_write has been found.

(thanks @soundOfDestiny)!

Proposal: use ConcurrentSkiplist for memtable

Problem

Now AgateDB will write data in only one thread because the structure skiplist only support one-write-multiple-read. But if there are a large number of keys, it will cost much more time in writing memtable.

Solution

RocksDB has implemented one ConcurrentSkiplist for multiple thread, see more details in https://github.com/facebook/rocksdb/blob/main/memtable/inlineskiplist.h#L907 . I think we can port it into agatedb and then create a threadpool for writing memtable.

Proposal for new AgateIterator interface

The iterator trait can panic due to incorrect usage. I propose a safer design:

trait KeyValuePair {
    fn key(&self) -> &[u8];
    fn value(&self) -> Value;
}

trait Iterator {
    type KeyValuePair;

    fn next(&mut self) -> Result<KeyValuePair>;
    fn seek(&mut self) -> Result<KeyValuePair>;
    fn rewind(&mut self) -> Result<KeyValuePair>;
    fn current(&self) -> Option<KeyValuePair>;
}

KeyValuePair trait allows customization of key and value, so it's still lazily loaded. And key and value methods will not have to check for valid any more. Iterator doesn't have to store the error either.

This interface is for future refactor, doesn't have to be implemented in this PR.

Originally posted by @BusyJay in #30 (comment)

Compile error: no method named `steal_batch_with_limit_and_pop` found for struct `Injector` in the current scope

➜  agatedb git:(master) cargo check
    Checking yatp v0.0.1 (https://github.com/tikv/yatp.git#7ed25299)
error[E0599]: no method named `steal_batch_with_limit_and_pop` found for struct `Injector` in the current scope
   --> /root/.cargo/git/checkouts/yatp-e704b73c3ee279b6/7ed2529/src/queue/multilevel.rs:183:37
    |
183 |         self.level_injectors[level].steal_batch_with_limit_and_pop(&self.local_queue, steal_limit)
    |                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: there is a method with a similar name: `steal_batch_and_pop`

For more information about this error, try `rustc --explain E0599`.
error: could not compile `yatp` due to previous error
➜  agatedb git:(master) cd /root/.cargo/git/checkouts/yatp-e704b73c3ee279b6/7ed2529/
➜  7ed2529 git:(master) ✗ cargo check
    Finished dev [unoptimized + debuginfo] target(s) in 0.12s

#192

yatp could not preempt threads

After introducing MVCC and txn, we now have 3 long-running background threads.

• memtable flush
• batching write request
• compaction * 3

If we put all threads in one thread pool, one of them won't even begin to execute in GitHub Action. Current workaround is to use std::thread for flush and write, and pool for compaction. We may later investigate how to integrate all threads into a thread pool.

Benchmark

Development Task

Do benchmarks on agatedb (1-2 weeks)

Development Progress

• micro benchmark with badger-bench
  • plan
  • bench
• macro benchmark with TiKV

data race in thread sanitizer test

#49

WARNING: ThreadSanitizer: data race (pid=17810)
191
  Write of size 8 at 0x7b740000fa00 by thread T11:
192
    #0 free /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:708 (agatedb-6848b01562dc6aa7+0x39d38)
193
    #1 std::sys::unix::alloc::<impl core::alloc::global::GlobalAlloc for std::alloc::System>::dealloc /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/alloc.rs:42 (agatedb-6848b01562dc6aa7+0x5814a5)
194
    #2 __rdl_dealloc /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/alloc.rs:364 (agatedb-6848b01562dc6aa7+0x4e6040)
195
    #3 alloc::alloc::dealloc /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:102 (agatedb-6848b01562dc6aa7+0x40c368)
196
    #4 <alloc::alloc::Global as core::alloc::AllocRef>::dealloc /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:237 (agatedb-6848b01562dc6aa7+0x40cd57)
197
    #5 alloc::alloc::box_free /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:332 (agatedb-6848b01562dc6aa7+0x40c629)
198
    #6 core::ptr::drop_in_place /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:175 (agatedb-6848b01562dc6aa7+0x3fe5da)
199
    #7 core::mem::drop /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/mem/mod.rs:882 (agatedb-6848b01562dc6aa7+0x3f999f)
200
    #8 <crossbeam_epoch::atomic::Owned<T> as core::ops::drop::Drop>::drop /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/atomic.rs:760 (agatedb-6848b01562dc6aa7+0x405c1c)
201
    #9 core::ptr::drop_in_place /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:175 (agatedb-6848b01562dc6aa7+0x3fe2bb)
202
    #10 core::mem::drop /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/mem/mod.rs:882 (agatedb-6848b01562dc6aa7+0x3f97ff)
203
    #11 crossbeam_epoch::guard::Guard::defer_unchecked::{{closure}} /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/guard.rs:195 (agatedb-6848b01562dc6aa7+0x40b72e)
204
    #12 crossbeam_epoch::deferred::Deferred::new::call /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/deferred.rs:46 (agatedb-6848b01562dc6aa7+0x4089a6)
205
    #13 crossbeam_epoch::deferred::Deferred::call /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/deferred.rs:77 (agatedb-6848b01562dc6aa7+0x408c95)
206
    #14 <crossbeam_epoch::internal::Bag as core::ops::drop::Drop>::drop /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:201 (agatedb-6848b01562dc6aa7+0x3f6cef)
207
    #15 core::ptr::drop_in_place /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:175 (agatedb-6848b01562dc6aa7+0x3fdf5b)
208
    #16 core::ptr::drop_in_place /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:175 (agatedb-6848b01562dc6aa7+0x3fe00f)
209
    #17 core::mem::drop /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/mem/mod.rs:882 (agatedb-6848b01562dc6aa7+0x3f9853)
210
    #18 crossbeam_epoch::internal::Global::collect /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:295 (agatedb-6848b01562dc6aa7+0x3f7410)
211
    #19 crossbeam_epoch::internal::Local::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:497 (agatedb-6848b01562dc6aa7+0x382b77)
212
    #20 crossbeam_epoch::collector::LocalHandle::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/collector.rs:75 (agatedb-6848b01562dc6aa7+0x3b3474)
213
    #21 crossbeam_epoch::default::pin::{{closure}} /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:23 (agatedb-6848b01562dc6aa7+0x381eb6)
214
    #22 crossbeam_epoch::default::with_handle::{{closure}} /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:43 (agatedb-6848b01562dc6aa7+0x381b54)
215
    #23 std::thread::local::LocalKey<T>::try_with /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:272 (agatedb-6848b01562dc6aa7+0x398688)
216
    #24 crossbeam_epoch::default::with_handle /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:42 (agatedb-6848b01562dc6aa7+0x381a9c)
217
    #25 crossbeam_epoch::default::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:23 (agatedb-6848b01562dc6aa7+0x381e46)
218
    #26 crossbeam_deque::Stealer<T>::steal_batch_and_pop /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-deque-0.7.3/src/lib.rs:956 (agatedb-6848b01562dc6aa7+0x3a9b19)
219
    #27 yatp::queue::single_level::LocalQueue<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/queue/single_level.rs:87 (agatedb-6848b01562dc6aa7+0x3cefd4)
220
    #28 yatp::queue::LocalQueue<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/queue.rs:98 (agatedb-6848b01562dc6aa7+0x39bfee)
221
    #29 yatp::pool::spawn::Local<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/spawn.rs:255 (agatedb-6848b01562dc6aa7+0x3ce4e1)
222
    #30 yatp::pool::worker::WorkerThread<T,R>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:28 (agatedb-6848b01562dc6aa7+0x39b926)
223
    #31 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:44 (agatedb-6848b01562dc6aa7+0x39bbff)
224
    #32 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-6848b01562dc6aa7+0x3d44da)
225
    #33 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-6848b01562dc6aa7+0x398126)
226
    #34 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-6848b01562dc6aa7+0x3b49da)
227
    #35 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x3a049a)
228
    #36 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x3d1609)
229
    #37 __rust_try yatp.exejrn79-cgu.8:? (agatedb-6848b01562dc6aa7+0x3d584b)
230
    #38 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x3d14b7)
231
    #39 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x3a0b6a)
232
    #40 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-6848b01562dc6aa7+0x3b4723)
233
    #41 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x38342b)
234
    #42 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557b0b)
235
    #43 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557c06)
236
    #44 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-6848b01562dc6aa7+0x519e03)
237

238
  Previous atomic read of size 8 at 0x7b740000fa00 by thread T7:
239
    #0 __tsan_atomic64_load /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp:539 (agatedb-6848b01562dc6aa7+0x7cf1e)
240
    #1 core::sync::atomic::atomic_load /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/sync/atomic.rs:2354 (agatedb-6848b01562dc6aa7+0x710a8e)
241
    #2 core::sync::atomic::AtomicUsize::load /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/sync/atomic.rs:1486 (agatedb-6848b01562dc6aa7+0x3fca06)
242
    #3 crossbeam_epoch::atomic::Atomic<T>::load /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/atomic.rs:209 (agatedb-6848b01562dc6aa7+0x404886)
243
    #4 <crossbeam_epoch::sync::list::Iter<T,C> as core::iter::traits::iterator::Iterator>::next /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/sync/list.rs:240 (agatedb-6848b01562dc6aa7+0x40716e)
244
    #5 crossbeam_epoch::internal::Global::try_advance /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:316 (agatedb-6848b01562dc6aa7+0x3f760c)
245
    #6 crossbeam_epoch::internal::Global::collect /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:281 (agatedb-6848b01562dc6aa7+0x3f7265)
246
    #7 crossbeam_epoch::internal::Local::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:497 (agatedb-6848b01562dc6aa7+0x382b77)
247
    #8 crossbeam_epoch::collector::LocalHandle::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/collector.rs:75 (agatedb-6848b01562dc6aa7+0x3b3474)
248
    #9 crossbeam_epoch::default::pin::{{closure}} /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:23 (agatedb-6848b01562dc6aa7+0x381eb6)
249
    #10 crossbeam_epoch::default::with_handle::{{closure}} /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:43 (agatedb-6848b01562dc6aa7+0x381b54)
250
    #11 std::thread::local::LocalKey<T>::try_with /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:272 (agatedb-6848b01562dc6aa7+0x398688)
251
    #12 crossbeam_epoch::default::with_handle /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:42 (agatedb-6848b01562dc6aa7+0x381a9c)
252
    #13 crossbeam_epoch::default::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:23 (agatedb-6848b01562dc6aa7+0x381e46)
253
    #14 crossbeam_deque::Stealer<T>::steal_batch_and_pop /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-deque-0.7.3/src/lib.rs:956 (agatedb-6848b01562dc6aa7+0x3a9b19)
254
    #15 yatp::queue::single_level::LocalQueue<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/queue/single_level.rs:87 (agatedb-6848b01562dc6aa7+0x3cefd4)
255
    #16 yatp::queue::LocalQueue<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/queue.rs:98 (agatedb-6848b01562dc6aa7+0x39bfee)
256
    #17 yatp::pool::spawn::Local<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/spawn.rs:255 (agatedb-6848b01562dc6aa7+0x3ce4e1)
257
    #18 yatp::pool::worker::WorkerThread<T,R>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:28 (agatedb-6848b01562dc6aa7+0x39b926)
258
    #19 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:44 (agatedb-6848b01562dc6aa7+0x39bbff)
259
    #20 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-6848b01562dc6aa7+0x3d44da)
260
    #21 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-6848b01562dc6aa7+0x398126)
261
    #22 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-6848b01562dc6aa7+0x3b49da)
262
    #23 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x3a049a)
263
    #24 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x3d1609)
264
    #25 __rust_try yatp.exejrn79-cgu.8:? (agatedb-6848b01562dc6aa7+0x3d584b)
265
    #26 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x3d14b7)
266
    #27 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x3a0b6a)
267
    #28 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-6848b01562dc6aa7+0x3b4723)
268
    #29 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x38342b)
269
    #30 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557b0b)
270
    #31 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557c06)
271
    #32 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-6848b01562dc6aa7+0x519e03)
272

273
  Thread T11 'agatedb-1' (tid=17822, running) created by thread T9 at:
274
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-6848b01562dc6aa7+0x3ac9b)
275
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-6848b01562dc6aa7+0x51997b)
276
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-6848b01562dc6aa7+0x3b3eef)
277
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-6848b01562dc6aa7+0x3b4a9a)
278
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-6848b01562dc6aa7+0x3d4056)
279
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-6848b01562dc6aa7+0x3d4c31)
280
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-6848b01562dc6aa7+0x3d4b2c)
281
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:69 (agatedb-6848b01562dc6aa7+0xae5d3)
282
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:216 (agatedb-6848b01562dc6aa7+0x13ef74)
283
    #9 agatedb::db::tests::with_agate_test /home/runner/work/agatedb/agatedb/src/db.rs:577 (agatedb-6848b01562dc6aa7+0xb8c10)
284
    #10 agatedb::db::tests::test_flush_memtable_bigvalue /home/runner/work/agatedb/agatedb/src/db.rs:660 (agatedb-6848b01562dc6aa7+0xf37f2)
285
    #11 agatedb::db::tests::test_flush_memtable_bigvalue::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:659 (agatedb-6848b01562dc6aa7+0xba864)
286
    #12 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x18eb87)
287
    #13 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a43a1)
288
    #14 test::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:517 (agatedb-6848b01562dc6aa7+0x259afb)
289
    #15 test::run_test::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:508 (agatedb-6848b01562dc6aa7+0x2590d7)
290
    #16 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a4114)
291
    #17 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x2335bb)
292
    #18 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x2a0ddd)
293
    #19 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x203ba2)
294
    #20 __rust_try test.cvxxcx5e-cgu.1:? (agatedb-6848b01562dc6aa7+0x20deab)
295
    #21 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x2039b8)
296
    #22 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x2a31bd)
297
    #23 test::run_test_in_process /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:544 (agatedb-6848b01562dc6aa7+0x25a02d)
298
    #24 test::run_test::run_test_inner::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:450 (agatedb-6848b01562dc6aa7+0x25989f)
299
    #25 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-6848b01562dc6aa7+0x2a27b6)
300
    #26 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-6848b01562dc6aa7+0x22cc8a)
301
    #27 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x2a0d6a)
302
    #28 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x203af9)
303
    #29 __rust_try test.cvxxcx5e-cgu.1:? (agatedb-6848b01562dc6aa7+0x20deab)
304
    #30 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x203357)
305
    #31 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x2a324a)
306
    #32 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-6848b01562dc6aa7+0x22c9d3)
307
    #33 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a41eb)
308
    #34 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557b0b)
309
    #35 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557c06)
310
    #36 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-6848b01562dc6aa7+0x519e03)
311

312
  Thread T7 'agatedb-1' (tid=17818, running) created by thread T4 at:
313
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-6848b01562dc6aa7+0x3ac9b)
314
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-6848b01562dc6aa7+0x51997b)
315
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-6848b01562dc6aa7+0x3b3eef)
316
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-6848b01562dc6aa7+0x3b4a9a)
317
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-6848b01562dc6aa7+0x3d4056)
318
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-6848b01562dc6aa7+0x3d4c31)
319
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-6848b01562dc6aa7+0x3d4b2c)
320
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:69 (agatedb-6848b01562dc6aa7+0xae5d3)
321
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:216 (agatedb-6848b01562dc6aa7+0x13ef74)
322
    #9 agatedb::db::tests::with_agate_test /home/runner/work/agatedb/agatedb/src/db.rs:577 (agatedb-6848b01562dc6aa7+0xb9330)
323
    #10 agatedb::db::tests::test_flush_memtable /home/runner/work/agatedb/agatedb/src/db.rs:652 (agatedb-6848b01562dc6aa7+0xf37b2)
324
    #11 agatedb::db::tests::test_flush_memtable::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:651 (agatedb-6848b01562dc6aa7+0xba724)
325
    #12 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x18e917)
326
    #13 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a43a1)
327
    #14 test::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:517 (agatedb-6848b01562dc6aa7+0x259afb)
328
    #15 test::run_test::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:508 (agatedb-6848b01562dc6aa7+0x2590d7)
329
    #16 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a4114)
330
    #17 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x2335bb)
331
    #18 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x2a0ddd)
332
    #19 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x203ba2)
333
    #20 __rust_try test.cvxxcx5e-cgu.1:? (agatedb-6848b01562dc6aa7+0x20deab)
334
    #21 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x2039b8)
335
    #22 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x2a31bd)
336
    #23 test::run_test_in_process /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:544 (agatedb-6848b01562dc6aa7+0x25a02d)
337
    #24 test::run_test::run_test_inner::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:450 (agatedb-6848b01562dc6aa7+0x25989f)
338
    #25 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-6848b01562dc6aa7+0x2a27b6)
339
    #26 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-6848b01562dc6aa7+0x22cc8a)
340
    #27 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x2a0d6a)
341
    #28 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x203af9)
342
    #29 __rust_try test.cvxxcx5e-cgu.1:? (agatedb-6848b01562dc6aa7+0x20deab)
343
    #30 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x203357)
344
    #31 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x2a324a)
345
    #32 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-6848b01562dc6aa7+0x22c9d3)
346
    #33 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a41eb)
347
    #34 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557b0b)
348
    #35 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557c06)
349
    #36 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-6848b01562dc6aa7+0x519e03)
350

351
SUMMARY: ThreadSanitizer: data race /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/alloc.rs:42 in std::sys::unix::alloc::<impl core::alloc::global::GlobalAlloc for std::alloc::System>::dealloc
352
==================
353
==================
354
WARNING: ThreadSanitizer: data race (pid=17810)
355
  Write of size 8 at 0x7b740000fa08 by thread T11:
356
    #0 free /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:708 (agatedb-6848b01562dc6aa7+0x39d38)
357
    #1 std::sys::unix::alloc::<impl core::alloc::global::GlobalAlloc for std::alloc::System>::dealloc /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/alloc.rs:42 (agatedb-6848b01562dc6aa7+0x5814a5)
358
    #2 __rdl_dealloc /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/alloc.rs:364 (agatedb-6848b01562dc6aa7+0x4e6040)
359
    #3 alloc::alloc::dealloc /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:102 (agatedb-6848b01562dc6aa7+0x40c368)
360
    #4 <alloc::alloc::Global as core::alloc::AllocRef>::dealloc /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:237 (agatedb-6848b01562dc6aa7+0x40cd57)
361
    #5 alloc::alloc::box_free /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:332 (agatedb-6848b01562dc6aa7+0x40c629)
362
    #6 core::ptr::drop_in_place /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:175 (agatedb-6848b01562dc6aa7+0x3fe5da)
363
    #7 core::mem::drop /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/mem/mod.rs:882 (agatedb-6848b01562dc6aa7+0x3f999f)
364
    #8 <crossbeam_epoch::atomic::Owned<T> as core::ops::drop::Drop>::drop /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/atomic.rs:760 (agatedb-6848b01562dc6aa7+0x405c1c)
365
    #9 core::ptr::drop_in_place /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:175 (agatedb-6848b01562dc6aa7+0x3fe2bb)
366
    #10 core::mem::drop /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/mem/mod.rs:882 (agatedb-6848b01562dc6aa7+0x3f97ff)
367
    #11 crossbeam_epoch::guard::Guard::defer_unchecked::{{closure}} /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/guard.rs:195 (agatedb-6848b01562dc6aa7+0x40b72e)
368
    #12 crossbeam_epoch::deferred::Deferred::new::call /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/deferred.rs:46 (agatedb-6848b01562dc6aa7+0x4089a6)
369
    #13 crossbeam_epoch::deferred::Deferred::call /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/deferred.rs:77 (agatedb-6848b01562dc6aa7+0x408c95)
370
    #14 <crossbeam_epoch::internal::Bag as core::ops::drop::Drop>::drop /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:201 (agatedb-6848b01562dc6aa7+0x3f6cef)
371
    #15 core::ptr::drop_in_place /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:175 (agatedb-6848b01562dc6aa7+0x3fdf5b)
372
    #16 core::ptr::drop_in_place /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:175 (agatedb-6848b01562dc6aa7+0x3fe00f)
373
    #17 core::mem::drop /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/mem/mod.rs:882 (agatedb-6848b01562dc6aa7+0x3f9853)
374
    #18 crossbeam_epoch::internal::Global::collect /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:295 (agatedb-6848b01562dc6aa7+0x3f7410)
375
    #19 crossbeam_epoch::internal::Local::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:497 (agatedb-6848b01562dc6aa7+0x382b77)
376
    #20 crossbeam_epoch::collector::LocalHandle::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/collector.rs:75 (agatedb-6848b01562dc6aa7+0x3b3474)
377
    #21 crossbeam_epoch::default::pin::{{closure}} /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:23 (agatedb-6848b01562dc6aa7+0x381eb6)
378
    #22 crossbeam_epoch::default::with_handle::{{closure}} /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:43 (agatedb-6848b01562dc6aa7+0x381b54)
379
    #23 std::thread::local::LocalKey<T>::try_with /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:272 (agatedb-6848b01562dc6aa7+0x398688)
380
    #24 crossbeam_epoch::default::with_handle /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:42 (agatedb-6848b01562dc6aa7+0x381a9c)
381
    #25 crossbeam_epoch::default::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:23 (agatedb-6848b01562dc6aa7+0x381e46)
382
    #26 crossbeam_deque::Stealer<T>::steal_batch_and_pop /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-deque-0.7.3/src/lib.rs:956 (agatedb-6848b01562dc6aa7+0x3a9b19)
383
    #27 yatp::queue::single_level::LocalQueue<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/queue/single_level.rs:87 (agatedb-6848b01562dc6aa7+0x3cefd4)
384
    #28 yatp::queue::LocalQueue<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/queue.rs:98 (agatedb-6848b01562dc6aa7+0x39bfee)
385
    #29 yatp::pool::spawn::Local<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/spawn.rs:255 (agatedb-6848b01562dc6aa7+0x3ce4e1)
386
    #30 yatp::pool::worker::WorkerThread<T,R>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:28 (agatedb-6848b01562dc6aa7+0x39b926)
387
    #31 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:44 (agatedb-6848b01562dc6aa7+0x39bbff)
388
    #32 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-6848b01562dc6aa7+0x3d44da)
389
    #33 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-6848b01562dc6aa7+0x398126)
390
    #34 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-6848b01562dc6aa7+0x3b49da)
391
    #35 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x3a049a)
392
    #36 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x3d1609)
393
    #37 __rust_try yatp.exejrn79-cgu.8:? (agatedb-6848b01562dc6aa7+0x3d584b)
394
    #38 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x3d14b7)
395
    #39 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x3a0b6a)
396
    #40 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-6848b01562dc6aa7+0x3b4723)
397
    #41 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x38342b)
398
    #42 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557b0b)
399
    #43 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557c06)
400
    #44 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-6848b01562dc6aa7+0x519e03)
401

402
  Previous atomic read of size 8 at 0x7b740000fa08 by thread T7:
403
    #0 __tsan_atomic64_load /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp:539 (agatedb-6848b01562dc6aa7+0x7cf1e)
404
    #1 core::sync::atomic::atomic_load /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/sync/atomic.rs:2355 (agatedb-6848b01562dc6aa7+0x710aa9)
405
    #2 core::sync::atomic::AtomicUsize::load /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/sync/atomic.rs:1486 (agatedb-6848b01562dc6aa7+0x3fca06)
406
    #3 crossbeam_epoch::epoch::AtomicEpoch::load /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/epoch.rs:93 (agatedb-6848b01562dc6aa7+0x40b2d7)
407
    #4 crossbeam_epoch::internal::Global::try_advance /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:325 (agatedb-6848b01562dc6aa7+0x3f76c6)
408
    #5 crossbeam_epoch::internal::Global::collect /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:281 (agatedb-6848b01562dc6aa7+0x3f7265)
409
    #6 crossbeam_epoch::internal::Local::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:497 (agatedb-6848b01562dc6aa7+0x382b77)
410
    #7 crossbeam_epoch::collector::LocalHandle::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/collector.rs:75 (agatedb-6848b01562dc6aa7+0x3b3474)
411
    #8 crossbeam_epoch::default::pin::{{closure}} /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:23 (agatedb-6848b01562dc6aa7+0x381eb6)
412
    #9 crossbeam_epoch::default::with_handle::{{closure}} /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:43 (agatedb-6848b01562dc6aa7+0x381b54)
413
    #10 std::thread::local::LocalKey<T>::try_with /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:272 (agatedb-6848b01562dc6aa7+0x398688)
414
    #11 crossbeam_epoch::default::with_handle /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:42 (agatedb-6848b01562dc6aa7+0x381a9c)
415
    #12 crossbeam_epoch::default::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:23 (agatedb-6848b01562dc6aa7+0x381e46)
416
    #13 crossbeam_deque::Stealer<T>::steal_batch_and_pop /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-deque-0.7.3/src/lib.rs:956 (agatedb-6848b01562dc6aa7+0x3a9b19)
417
    #14 yatp::queue::single_level::LocalQueue<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/queue/single_level.rs:87 (agatedb-6848b01562dc6aa7+0x3cefd4)
418
    #15 yatp::queue::LocalQueue<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/queue.rs:98 (agatedb-6848b01562dc6aa7+0x39bfee)
419
    #16 yatp::pool::spawn::Local<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/spawn.rs:255 (agatedb-6848b01562dc6aa7+0x3ce4e1)
420
    #17 yatp::pool::worker::WorkerThread<T,R>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:28 (agatedb-6848b01562dc6aa7+0x39b926)
421
    #18 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:44 (agatedb-6848b01562dc6aa7+0x39bbff)
422
    #19 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-6848b01562dc6aa7+0x3d44da)
423
    #20 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-6848b01562dc6aa7+0x398126)
424
    #21 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-6848b01562dc6aa7+0x3b49da)
425
    #22 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x3a049a)
426
    #23 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x3d1609)
427
    #24 __rust_try yatp.exejrn79-cgu.8:? (agatedb-6848b01562dc6aa7+0x3d584b)
428
    #25 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x3d14b7)
429
    #26 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x3a0b6a)
430
    #27 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-6848b01562dc6aa7+0x3b4723)
431
    #28 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x38342b)
432
    #29 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557b0b)
433
    #30 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557c06)
434
    #31 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-6848b01562dc6aa7+0x519e03)
435

436
  Thread T11 'agatedb-1' (tid=17822, running) created by thread T9 at:
437
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-6848b01562dc6aa7+0x3ac9b)
438
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-6848b01562dc6aa7+0x51997b)
439
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-6848b01562dc6aa7+0x3b3eef)
440
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-6848b01562dc6aa7+0x3b4a9a)
441
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-6848b01562dc6aa7+0x3d4056)
442
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-6848b01562dc6aa7+0x3d4c31)
443
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-6848b01562dc6aa7+0x3d4b2c)
444
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:69 (agatedb-6848b01562dc6aa7+0xae5d3)
445
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:216 (agatedb-6848b01562dc6aa7+0x13ef74)
446
    #9 agatedb::db::tests::with_agate_test /home/runner/work/agatedb/agatedb/src/db.rs:577 (agatedb-6848b01562dc6aa7+0xb8c10)
447
    #10 agatedb::db::tests::test_flush_memtable_bigvalue /home/runner/work/agatedb/agatedb/src/db.rs:660 (agatedb-6848b01562dc6aa7+0xf37f2)
448
    #11 agatedb::db::tests::test_flush_memtable_bigvalue::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:659 (agatedb-6848b01562dc6aa7+0xba864)
449
    #12 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x18eb87)
450
    #13 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a43a1)
451
    #14 test::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:517 (agatedb-6848b01562dc6aa7+0x259afb)
452
    #15 test::run_test::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:508 (agatedb-6848b01562dc6aa7+0x2590d7)
453
    #16 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a4114)
454
    #17 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x2335bb)
455
    #18 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x2a0ddd)
456
    #19 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x203ba2)
457
    #20 __rust_try test.cvxxcx5e-cgu.1:? (agatedb-6848b01562dc6aa7+0x20deab)
458
    #21 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x2039b8)
459
    #22 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x2a31bd)
460
    #23 test::run_test_in_process /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:544 (agatedb-6848b01562dc6aa7+0x25a02d)
461
    #24 test::run_test::run_test_inner::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:450 (agatedb-6848b01562dc6aa7+0x25989f)
462
    #25 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-6848b01562dc6aa7+0x2a27b6)
463
    #26 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-6848b01562dc6aa7+0x22cc8a)
464
    #27 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x2a0d6a)
465
    #28 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x203af9)
466
    #29 __rust_try test.cvxxcx5e-cgu.1:? (agatedb-6848b01562dc6aa7+0x20deab)
467
    #30 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x203357)
468
    #31 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x2a324a)
469
    #32 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-6848b01562dc6aa7+0x22c9d3)
470
    #33 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a41eb)
471
    #34 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557b0b)
472
    #35 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557c06)
473
    #36 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-6848b01562dc6aa7+0x519e03)
474

475
  Thread T7 'agatedb-1' (tid=17818, running) created by thread T4 at:
476
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-6848b01562dc6aa7+0x3ac9b)
477
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-6848b01562dc6aa7+0x51997b)
478
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-6848b01562dc6aa7+0x3b3eef)
479
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-6848b01562dc6aa7+0x3b4a9a)
480
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-6848b01562dc6aa7+0x3d4056)
481
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-6848b01562dc6aa7+0x3d4c31)
482
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-6848b01562dc6aa7+0x3d4b2c)
483
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:69 (agatedb-6848b01562dc6aa7+0xae5d3)
484
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:216 (agatedb-6848b01562dc6aa7+0x13ef74)
485
    #9 agatedb::db::tests::with_agate_test /home/runner/work/agatedb/agatedb/src/db.rs:577 (agatedb-6848b01562dc6aa7+0xb9330)
486
    #10 agatedb::db::tests::test_flush_memtable /home/runner/work/agatedb/agatedb/src/db.rs:652 (agatedb-6848b01562dc6aa7+0xf37b2)
487
    #11 agatedb::db::tests::test_flush_memtable::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:651 (agatedb-6848b01562dc6aa7+0xba724)
488
    #12 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x18e917)
489
    #13 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a43a1)
490
    #14 test::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:517 (agatedb-6848b01562dc6aa7+0x259afb)
491
    #15 test::run_test::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:508 (agatedb-6848b01562dc6aa7+0x2590d7)
492
    #16 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a4114)
493
    #17 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x2335bb)
494
    #18 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x2a0ddd)
495
    #19 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x203ba2)
496
    #20 __rust_try test.cvxxcx5e-cgu.1:? (agatedb-6848b01562dc6aa7+0x20deab)
497
    #21 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x2039b8)
498
    #22 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x2a31bd)
499
    #23 test::run_test_in_process /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:544 (agatedb-6848b01562dc6aa7+0x25a02d)
500
    #24 test::run_test::run_test_inner::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:450 (agatedb-6848b01562dc6aa7+0x25989f)
501
    #25 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-6848b01562dc6aa7+0x2a27b6)
502
    #26 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-6848b01562dc6aa7+0x22cc8a)
503
    #27 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x2a0d6a)
504
    #28 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x203af9)
505
    #29 __rust_try test.cvxxcx5e-cgu.1:? (agatedb-6848b01562dc6aa7+0x20deab)
506
    #30 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x203357)
507
    #31 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x2a324a)
508
    #32 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-6848b01562dc6aa7+0x22c9d3)
509
    #33 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a41eb)
510
    #34 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557b0b)
511
    #35 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557c06)
512
    #36 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-6848b01562dc6aa7+0x519e03)
513

514
SUMMARY: ThreadSanitizer: data race /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/alloc.rs:42 in std::sys::unix::alloc::<impl core::alloc::global::GlobalAlloc for std::alloc::System>::dealloc
515
==================

Implement Basic Structures

Development Task

Implement basic data structures (2-3 weeks)

Development Progress

• main tasks
  • add CI integration
  • switch to prost
  • table builder
  • SST and iterator
  • SST checksum
  • SST and iterator benchmark
  • bloom filter
  • cache (seems not necessary)
  • vLog (requires integration with GC, only implement basic struct in this stage)
• bug fixes and patches
  • enable clippy and enable warn_unused
  • fix skiplist benchmark failures
  • fix test timeout failures
  • add sanitizer #10
  • use jemalloc in benchmark to avoid fluctuations
• improvements
  • use mmapfile instead of file+vec to improve read throughput
  • auto calculate length, and eliminate the need to store len along with Bytes or Vec
  • implement some memory copy optimization
  • use Result<()> instead of iter.valid()

Pull Requests

• add CI integration #4
• switch to prost #5
• fix benchmark and test failures #11
• prepare for SST and iterator implementation #13 #14
• use jemalloc in benchmark to avoid fluctuations #16
• implement SST and iterator #15 (need split)
  • Refactor in table builder #19
  • Block Iterator #21
  • Iterator #22
  • Table #25
  • Tests and Benchmarks #26
• add sanitizer #17
• SST checksum #18
• bloom filter #20
• mmap #23

Feature Request: Adaptive Radix Tree memtable

As https://db.in.tum.de/~leis/papers/ART.pdf showed, the adaptive radix tree is mush more faster than other comparable search tree/list.

The skiplist is not concurrent but could be used concurrently

The arena in skiplist does not support concurrent operations(#116), but it could be used concurrently when allow_concurrent_write is set to true.

Concurrent tests such as bench_read_write_skiplist will crash with [SIGSEGV](signal: 11, SIGSEGV: invalid memory reference) when reducing the arena capacity, for example changing the https://github.com/tikv/agatedb/blob/master/skiplist/benches/bench.rs#L47 size from 512 << 20 to 512.

Supporting the concurrent arena allocator type may be necessary.

能给一些入门使用的例子吗

compact_prios exceeds max_levels

This panic is not guaranteed to be triggered. Will look into details later.

thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Any', /home/skyzh/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool.rs:46:26
stack backtrace:
   0: rust_begin_unwind
             at /rustc/f5230fbf76bafd86ee4376a0e26e551df8d17fec/library/std/src/panicking.rs:495:5
   1: core::panicking::panic_fmt
             at /rustc/f5230fbf76bafd86ee4376a0e26e551df8d17fec/library/core/src/panicking.rs:92:14
   2: core::option::expect_none_failed
             at /rustc/f5230fbf76bafd86ee4376a0e26e551df8d17fec/library/core/src/option.rs:1268:5
   3: core::result::Result<T,E>::unwrap
             at /home/skyzh/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:973:23
   4: yatp::pool::ThreadPool<T>::shutdown
             at /home/skyzh/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool.rs:46:17
   5: <agatedb::db::Agate as core::ops::drop::Drop>::drop
             at ./src/db.rs:98:9
   6: core::ptr::drop_in_place
             at /home/skyzh/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:175:1
   7: agatedb::db::tests::with_agate_test::{{closure}}
             at ./src/db.rs:617:9
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[Question] Is this still being developed?

I am looking for an embedded KV store in Rust and found this crate.
Just want to know the status of this work.