oceanbase/miniob

MiniOB is a compact database that assists developers in understanding the fundamental workings of a database.

MiniOB 介绍

MiniOB 是 OceanBase 团队基于华中科技大学数据库课程原型，联合多所高校重新开发的、专为零基础的同学设计的数据库入门学习项目。MiniOB 的目标是为在校学生、数据库从业者、爱好者或对基础技术感兴趣的人提供一个友好的数据库学习项目，更好地将理论、实践进行结合，提升同学们的工程实战能力。

MiniOB 整体代码简洁，容易上手，设计了一系列由浅入深的题目，帮助同学们从零基础入门，迅速了解数据库并深入学习数据库内核。MiniOB 简化了许多模块，例如不考虑并发操作、安全特性和复杂的事务管理等功能，以便更好地学习数据库实现原理。我们期望通过 MiniOB 的训练，同学们能够熟练掌握数据库内核模块的功能和协同关系，并具备一定的工程编码能力，例如内存管理、网络通信和磁盘 I/O 处理等, 这将有助于同学在未来的面试和工作中脱颖而出。

文档

代码配套设计文档和相关代码注释已经生成文档，并通过 GitHub Pages 发布。您可以直接访问：MiniOB GitHub Pages.

快速上手

为了帮助开发者更好地上手并学习 MiniOB，建议阅读以下内容：

1. MiniOB 框架介绍
2. 如何编译 MiniOB 源码
3. 如何运行 MiniOB
4. 使用 GitPod 开发 MiniOB
5. doxygen 代码文档

为了帮助大家更好地学习数据库基础知识，OceanBase社区提供了一系列教程。更多文档请参考 MiniOB GitHub Pages。建议学习：

1. 《从0到1数据库内核实战教程》 视频教程
2. 《从0到1数据库内核实战教程》 基础讲义
3. 《数据库管理系统实现》 华中科技大学实现教材

系统架构

MiniOB 整体架构如下图所示:

其中:

• 网络模块(NET Service)：负责与客户端交互，收发客户端请求与应答；
• SQL解析(Parser)：将用户输入的SQL语句解析成语法树；
• 语义解析模块(Resolver)：将生成的语法树，转换成数据库内部数据结构；
• 查询优化(Optimizer)：根据一定规则和统计数据，调整/重写语法树。(部分实现)；
• 计划执行(Executor)：根据语法树描述，执行并生成结果；
• 存储引擎(Storage Engine)：负责数据的存储和检索；
• 事务管理(MVCC)：管理事务的提交、回滚、隔离级别等。当前事务管理仅实现了MVCC模式，因此直接以MVCC展示；
• 日志管理(Redo Log)：负责记录数据库操作日志；
• 记录管理(Record Manager)：负责管理某个表数据文件中的记录存放；
• B+ Tree：表索引存储结构；
• 会话管理：管理用户连接、调整某个连接的参数；
• 元数据管理(Meta Data)：记录当前的数据库、表、字段和索引元数据信息；
• 客户端(Client)：作为测试工具，接收用户请求，向服务端发起请求。

OceanBase 大赛

全国大学生计算机系统能力大赛（以下简称“大赛”）是由系统能力培养研究专家组发起，全国高等学校计算机教育研究会、系统能力培养研究项目示范高校共同主办、OceanBase 承办，面向高校大学生的全国性数据库大赛。 大赛面向全国爱好数据库的高校学生，以“竞技、交流、成长”为宗旨，搭建基于赛事的技术交流平台，促进高校创新人才培养机制，不仅帮助学生从0开始系统化学习 OceanBase 数据库理论知识，提升学生数据库实践能力，更能帮助学生走向企业积累经验，促进国内数据库人才的发展，碰撞出创新的火花。

OceanBase 初赛基于一套适合初学者实践的数据库实训平台 MiniOB，代码量少，易于上手学习，包含了数据库的各个关键模块，是一个系统性的数据库学习平台。基于该平台设置了一系列由浅入深的题目，以帮助同学们更好"零"基础入门。

为了帮助大家能在大赛中取得好成绩，我们提供了一系列的教程和指导，帮助大家更好地学习数据库基础知识，更好地完成大赛题目。 欢迎大家查看《从0到1数据库内核实战教程》 视频教程，视频中包含了代码框架的介绍和一些入门题目的讲解。

由于MiniOB是一个持续演进的产品，视频教程中有些内容会与最新代码有冲突，建议大家参考讲解中的思路。

大赛的初赛是在MiniOB上进行的，同学们可以在前几届的题目上进行提前训练，可以让自己比别人提前一步。大家在日常训练时可以在MiniOB 训练营 上提交代码进行测试。

在提交前, 请参考并学习 训练营使用说明。

如果大家在大赛中或使用训练营时遇到一些问题，请先查看大赛 FAQ。

在线开发平台

搭建开发环境是一个比较耗时而且繁琐的事情，特别是对于初学者。为了让大家更快地上手 MiniOB，本仓库基于 Gitpod 建立了快速在线开发平台。点击下面的按钮即可一键体验（建议使用 Chrome 浏览器）。

注意：这个链接打开的是MiniOB主仓库的开发环境，同学们需要通过gitpod进入自己的仓库开发环境。

首次进入 Gitpod 时需要安装一些依赖。安装完成后，终端界面会显示 "Dependency installed successfully"。运行 bash build.sh --make -j4 命令即可编译 MiniOB。

Contributing

OceanBase 社区热情欢迎每一位对数据库技术热爱的开发者，期待与您携手开启思维碰撞之旅。无论是文档格式调整或文字修正、问题修复还是增加新功能，都是参与和贡献 OceanBase 社区的方式之一。MiniOB 现在开放了一些新功能的开发，欢迎有兴趣的同学一起共建，希望我们共同成长。如果你对MiniOB不熟悉也没关系，可以直接联系我们，我们将会有人指导上手。现在就开始您的首次贡献吧！更多详情，请参考 社区贡献。

Contributors

感谢所有为 MiniOB 项目做出贡献的同学们！

License

MiniOB 采用 木兰宽松许可证，第2版, 可以自由拷贝和使用源码, 当做修改或分发时, 请遵守 木兰宽松许可证，第2版.

社区组织

• OceanBase 社区交流群 33254054
• OceanBase 大赛官方交流群 35326455
• OceanBase 官方论坛
• MiniOB 开发者微信群(添加 hnwyllmm_126 为好友，备注 MiniOB，邀请入群)

miniob's Issues

undefined reference to `typeinfo for testing::Test'

编译完成所有deps后，编译miniob时，报错：
95%] Built target path_test

[ 95%] Building CXX object unitest/CMakeFiles/md5_test.dir/md5_test.cpp.o

[ 96%] Linking CXX executable ../bin/md5_test

[ 96%] Built target md5_test

[ 96%] Building CXX object unitest/CMakeFiles/log_test.dir/log_test.cpp.o

[ 97%] Linking CXX executable ../bin/log_test

/usr/bin/ld: CMakeFiles/log_test.dir/log_test.cpp.o:(.data.rel.ro._ZTI35checkRotateTest_CheckRoateTest_Test[_ZTI35checkRotateTest_CheckRoateTest_Test]+0x10): undefined reference to `typeinfo for testing::Test'

/usr/bin/ld: CMakeFiles/log_test.dir/log_test.cpp.o:(.data.rel.ro._ZTI35testEnableTest_CheckEnableTest_Test[_ZTI35testEnableTest_CheckEnableTest_Test]+0x10): undefined reference to `typeinfo for testing::Test'

collect2: error: ld returned 1 exit status

make[2]: *** [unitest/CMakeFiles/log_test.dir/build.make:105: bin/log_test] Error 1

make[1]: *** [CMakeFiles/Makefile2:497: unitest/CMakeFiles/log_test.dir/all] Error 2

make: *** [Makefile:149: all] Error 2

实现resolver模块

在resolver模块中讲“字符串”语法树转换为可执行语法树，并对元数据做校验

使用docker报错，无法启动server

• 使用docker拉取镜像后，执行build.sh
• 启动server后报错，找不到jsoncpp依赖等

stage线程分配问题

由于代码逻辑错误，将所有stage处理都分配给DefaultThreads线程池。

penggan666:

test penggan666 complex-sub-query

no prompt in client while without readline

handler.h is unused

the file handler.h is not referenced by any file

miniob-test

大家找一块自己的地方，在一个comment里面，可以重复提交测试，每次comment变更，都可以检测到。
在一个时间段内，一个团队的任务提交，只会有一个任务执行。
使用方法：

# 执行测试
test <your-github-name> [case1] [case2] ...

# 查看失败原因
view <your-github-name> <case1> [case2] ...

FAQ:

• 查看失败原因长时间没有返回结果
  如果返回的数据中包含二进制字符，即不是纯文本文件，那么执行结果无法反馈到github中，目前无法看到结果。

• 每个case的名称可以看这个链接
  https://oceanbase-partner.github.io/lectures-on-dbms-implementation/miniob-topics
  basic测试的名称就叫 basic

how_to_build文档不准确

how_to_build文档的描述步骤，不能完全按照描述来操作。
比如GCC，在MAC上不需要安装；libevent的git add submodule操作，需要在git clone miniob之后才能执行

构建镜像的时候下载 cmake 一直会超时

我这边是直接 wget 拉对应版本的 cmake 的，看样子 ob 这边是把指定版本的 cmake 做成了一个 rpm 包放在 oss 上面了? 不知道这个服务器在哪，我这边本地访问还有使用 github action 访问都是不通的，这个链接是不是要换一下？

[question]能编译成功但observer无法启动

问题：miniob在build文件夹下编译成功后未能成功启动observer

init.cpp文件中的第82行，

std::map<std::string, std::string> log_section = properties.get(log_section_name);
其中properties.get()函数似乎没有获取，log_section的size仍为0，想问下是什么原因呢

环境

Windows11，WSL2，Ubuntu22.04
gcc 11.2.0，cmake 3.24.1

运行

yulinux@yu:/mnt/e/github/miniob/build$ make
-- The C compiler identification is GNU 11.2.0
-- The CXX compiler identification is GNU 11.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- This is SOURCE dir 
-- This is BINARY dir 
-- This is Project source dir /mnt/e/github/miniob
-- This is PROJECT_BINARY_DIR dir /mnt/e/github/miniob/build
-- HOME dir: /home/yulinux
-- This is UNIX
Disable debug
CMAKE_CXX_FLAGS is  -Wall -DCMAKE_EXPORT_COMPILE_COMMANDS=1  -O2 -g
CMAKE_INSTALL_PREFIX has been set as /usr/local
Install target dir is /usr/local
Begin to build common
-- This is PROJECT_BINARY_DIR dir /mnt/e/github/miniob/build/deps/common
-- This is PROJECT_SOURCE_DIR dir /mnt/e/github/miniob/deps/common
Use /mnt/e/github/miniob/deps/common/conf/ini.cpp
Use /mnt/e/github/miniob/deps/common/io/io.cpp
Use /mnt/e/github/miniob/deps/common/io/roll_select_dir.cpp
Use /mnt/e/github/miniob/deps/common/lang/bitmap.cpp
Use /mnt/e/github/miniob/deps/common/lang/mutex.cpp
Use /mnt/e/github/miniob/deps/common/lang/string.cpp
Use /mnt/e/github/miniob/deps/common/log/log.cpp
Use /mnt/e/github/miniob/deps/common/math/md5.cpp
Use /mnt/e/github/miniob/deps/common/math/random_generator.cpp
Use /mnt/e/github/miniob/deps/common/math/regex.cpp
Use /mnt/e/github/miniob/deps/common/metrics/console_reporter.cpp
Use /mnt/e/github/miniob/deps/common/metrics/histogram_snapshot.cpp
Use /mnt/e/github/miniob/deps/common/metrics/log_reporter.cpp
Use /mnt/e/github/miniob/deps/common/metrics/metrics.cpp
Use /mnt/e/github/miniob/deps/common/metrics/metrics_registry.cpp
Use /mnt/e/github/miniob/deps/common/metrics/reporter.cpp
Use /mnt/e/github/miniob/deps/common/metrics/reservoir.cpp
Use /mnt/e/github/miniob/deps/common/metrics/sampler.cpp
Use /mnt/e/github/miniob/deps/common/metrics/timer_snapshot.cpp
Use /mnt/e/github/miniob/deps/common/metrics/uniform_reservoir.cpp
Use /mnt/e/github/miniob/deps/common/mm/mem_pool.cpp
Use /mnt/e/github/miniob/deps/common/os/os.cpp
Use /mnt/e/github/miniob/deps/common/os/path.cpp
Use /mnt/e/github/miniob/deps/common/os/pidfile.cpp
Use /mnt/e/github/miniob/deps/common/os/process.cpp
Use /mnt/e/github/miniob/deps/common/os/process_param.cpp
Use /mnt/e/github/miniob/deps/common/os/signal.cpp
Use /mnt/e/github/miniob/deps/common/seda/callback.cpp
Use /mnt/e/github/miniob/deps/common/seda/event_dispatcher.cpp
Use /mnt/e/github/miniob/deps/common/seda/example_stage.cpp
Use /mnt/e/github/miniob/deps/common/seda/init.cpp
Use /mnt/e/github/miniob/deps/common/seda/kill_thread.cpp
Use /mnt/e/github/miniob/deps/common/seda/metrics_report_event.cpp
Use /mnt/e/github/miniob/deps/common/seda/metrics_stage.cpp
Use /mnt/e/github/miniob/deps/common/seda/seda_config.cpp
Use /mnt/e/github/miniob/deps/common/seda/stage.cpp
Use /mnt/e/github/miniob/deps/common/seda/stage_event.cpp
Use /mnt/e/github/miniob/deps/common/seda/thread_pool.cpp
Use /mnt/e/github/miniob/deps/common/seda/timer_stage.cpp
Use /mnt/e/github/miniob/deps/common/time/datetime.cpp
Use /mnt/e/github/miniob/deps/common/time/timeout_info.cpp
-- This is the func_static OUTPUT_NAME:common
Install conf/ini.h to /usr/local/common/include/conf
Install defs.h to /usr/local/common/include/
Install io/io.h to /usr/local/common/include/io
Install io/roll_select_dir.h to /usr/local/common/include/io
Install io/select_dir.h to /usr/local/common/include/io
Install lang/bitmap.h to /usr/local/common/include/lang
Install lang/defer.h to /usr/local/common/include/lang
Install lang/mutex.h to /usr/local/common/include/lang
Install lang/serializable.h to /usr/local/common/include/lang
Install lang/string.h to /usr/local/common/include/lang
Install log/log.h to /usr/local/common/include/log
Install math/md5.h to /usr/local/common/include/math
Install math/random_generator.h to /usr/local/common/include/math
Install math/regex.h to /usr/local/common/include/math
Install metrics/console_reporter.h to /usr/local/common/include/metrics
Install metrics/histogram_snapshot.h to /usr/local/common/include/metrics
Install metrics/log_reporter.h to /usr/local/common/include/metrics
Install metrics/metric.h to /usr/local/common/include/metrics
Install metrics/metrics.h to /usr/local/common/include/metrics
Install metrics/metrics_registry.h to /usr/local/common/include/metrics
Install metrics/reporter.h to /usr/local/common/include/metrics
Install metrics/reservoir.h to /usr/local/common/include/metrics
Install metrics/sampler.h to /usr/local/common/include/metrics
Install metrics/snapshot.h to /usr/local/common/include/metrics
Install metrics/timer_snapshot.h to /usr/local/common/include/metrics
Install metrics/uniform_reservoir.h to /usr/local/common/include/metrics
Install mm/debug_new.h to /usr/local/common/include/mm
Install mm/mem.h to /usr/local/common/include/mm
Install mm/mem_pool.h to /usr/local/common/include/mm
Install os/os.h to /usr/local/common/include/os
Install os/path.h to /usr/local/common/include/os
Install os/pidfile.h to /usr/local/common/include/os
Install os/process.h to /usr/local/common/include/os
Install os/process_param.h to /usr/local/common/include/os
Install os/signal.h to /usr/local/common/include/os
Install seda/callback.h to /usr/local/common/include/seda
Install seda/class_factory.h to /usr/local/common/include/seda
Install seda/event_dispatcher.h to /usr/local/common/include/seda
Install seda/example_stage.h to /usr/local/common/include/seda
Install seda/init.h to /usr/local/common/include/seda
Install seda/kill_thread.h to /usr/local/common/include/seda
Install seda/metrics_report_event.h to /usr/local/common/include/seda
Install seda/metrics_stage.h to /usr/local/common/include/seda
Install seda/seda_config.h to /usr/local/common/include/seda
Install seda/seda_defs.h to /usr/local/common/include/seda
Install seda/stage.h to /usr/local/common/include/seda
Install seda/stage_event.h to /usr/local/common/include/seda
Install seda/stage_factory.h to /usr/local/common/include/seda
Install seda/thread_pool.h to /usr/local/common/include/seda
Install seda/timer_stage.h to /usr/local/common/include/seda
Install time/datetime.h to /usr/local/common/include/time
Install time/timeout_info.h to /usr/local/common/include/time
Install version.h to /usr/local/common/include/
Begin to build miniob
-- This is PROJECT_BINARY_DIR dir /mnt/e/github/miniob/build/src
-- This is PROJECT_SOURCE_DIR dir /mnt/e/github/miniob/src
Begin to build obclient
-- Looking for include files stdio.h, readline/readline.h
-- Looking for include files stdio.h, readline/readline.h - not found
readline is not found
Use /mnt/e/github/miniob/src/obclient/client.cpp
Binary directory:/mnt/e/github/miniob/build/src/../bin
Begin to build observer
-- This is PROJECT_BINARY_DIR dir /mnt/e/github/miniob/build/src/observer
-- This is PROJECT_SOURCE_DIR dir /mnt/e/github/miniob/src/observer
MAIN SRC: /mnt/e/github/miniob/src/observer/main.cpp
Use /mnt/e/github/miniob/src/observer/event/session_event.cpp
Use /mnt/e/github/miniob/src/observer/event/sql_event.cpp
Use /mnt/e/github/miniob/src/observer/init.cpp
Use /mnt/e/github/miniob/src/observer/main.cpp
Use /mnt/e/github/miniob/src/observer/net/server.cpp
Use /mnt/e/github/miniob/src/observer/rc.cpp
Use /mnt/e/github/miniob/src/observer/session/session.cpp
Use /mnt/e/github/miniob/src/observer/session/session_stage.cpp
Use /mnt/e/github/miniob/src/observer/sql/executor/execute_stage.cpp
Use /mnt/e/github/miniob/src/observer/sql/expr/expression.cpp
Use /mnt/e/github/miniob/src/observer/sql/expr/tuple_cell.cpp
Use /mnt/e/github/miniob/src/observer/sql/operator/delete_operator.cpp
Use /mnt/e/github/miniob/src/observer/sql/operator/index_scan_operator.cpp
Use /mnt/e/github/miniob/src/observer/sql/operator/insert_operator.cpp
Use /mnt/e/github/miniob/src/observer/sql/operator/predicate_operator.cpp
Use /mnt/e/github/miniob/src/observer/sql/operator/project_operator.cpp
Use /mnt/e/github/miniob/src/observer/sql/operator/table_scan_operator.cpp
Use /mnt/e/github/miniob/src/observer/sql/optimizer/optimize_stage.cpp
Use /mnt/e/github/miniob/src/observer/sql/parser/lex.yy.c
Use /mnt/e/github/miniob/src/observer/sql/parser/parse.cpp
Use /mnt/e/github/miniob/src/observer/sql/parser/parse_stage.cpp
Use /mnt/e/github/miniob/src/observer/sql/parser/resolve_stage.cpp
Use /mnt/e/github/miniob/src/observer/sql/parser/yacc_sql.tab.c
Use /mnt/e/github/miniob/src/observer/sql/plan_cache/plan_cache_stage.cpp
Use /mnt/e/github/miniob/src/observer/sql/query_cache/query_cache_stage.cpp
Use /mnt/e/github/miniob/src/observer/sql/stmt/delete_stmt.cpp
Use /mnt/e/github/miniob/src/observer/sql/stmt/filter_stmt.cpp
Use /mnt/e/github/miniob/src/observer/sql/stmt/insert_stmt.cpp
Use /mnt/e/github/miniob/src/observer/sql/stmt/select_stmt.cpp
Use /mnt/e/github/miniob/src/observer/sql/stmt/stmt.cpp
Use /mnt/e/github/miniob/src/observer/sql/stmt/update_stmt.cpp
Use /mnt/e/github/miniob/src/observer/storage/common/condition_filter.cpp
Use /mnt/e/github/miniob/src/observer/storage/common/db.cpp
Use /mnt/e/github/miniob/src/observer/storage/common/field_meta.cpp
Use /mnt/e/github/miniob/src/observer/storage/common/index_meta.cpp
Use /mnt/e/github/miniob/src/observer/storage/common/meta_util.cpp
Use /mnt/e/github/miniob/src/observer/storage/common/record_manager.cpp
Use /mnt/e/github/miniob/src/observer/storage/common/table.cpp
Use /mnt/e/github/miniob/src/observer/storage/common/table_meta.cpp
Use /mnt/e/github/miniob/src/observer/storage/default/default_handler.cpp
Use /mnt/e/github/miniob/src/observer/storage/default/default_storage_stage.cpp
Use /mnt/e/github/miniob/src/observer/storage/default/disk_buffer_pool.cpp
Use /mnt/e/github/miniob/src/observer/storage/index/bplus_tree.cpp
Use /mnt/e/github/miniob/src/observer/storage/index/bplus_tree_index.cpp
Use /mnt/e/github/miniob/src/observer/storage/index/index.cpp
Use /mnt/e/github/miniob/src/observer/storage/mem/mem_storage_stage.cpp
Use /mnt/e/github/miniob/src/observer/storage/trx/trx.cpp
Use /mnt/e/github/miniob/src/observer/util/comparator.cpp
Binary directory:/mnt/e/github/miniob/build/src/observer/../../bin
Archive directory:/mnt/e/github/miniob/build/src/observer/../../lib
Begin to build test
-- This is PROJECT_BINARY_DIR dir /mnt/e/github/miniob/build/test/perf
-- This is PROJECT_SOURCE_DIR dir /mnt/e/github/miniob/test/perf
 -Wall -DCMAKE_EXPORT_COMPILE_COMMANDS=1  -O2 -g
Build client_performance_test according to /mnt/e/github/miniob/test/perf/client_performance_test.cpp
Begin to build unitest
-- This is PROJECT_BINARY_DIR dir /mnt/e/github/miniob/build/unitest
-- This is PROJECT_SOURCE_DIR dir /mnt/e/github/miniob/unitest
 -Wall -DCMAKE_EXPORT_COMPILE_COMMANDS=1  -O2 -g
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
Build bitmap_test according to /mnt/e/github/miniob/unitest/bitmap_test.cpp
Build bp_manager_test according to /mnt/e/github/miniob/unitest/bp_manager_test.cpp
Build bplus_tree_test according to /mnt/e/github/miniob/unitest/bplus_tree_test.cpp
Build log_test according to /mnt/e/github/miniob/unitest/log_test.cpp
Build md5_test according to /mnt/e/github/miniob/unitest/md5_test.cpp
Build mem_pool_test according to /mnt/e/github/miniob/unitest/mem_pool_test.cpp
Build path_test according to /mnt/e/github/miniob/unitest/path_test.cpp
Build pidfile_test according to /mnt/e/github/miniob/unitest/pidfile_test.cpp
Build rc_test according to /mnt/e/github/miniob/unitest/rc_test.cpp
Build record_manager_test according to /mnt/e/github/miniob/unitest/record_manager_test.cpp
-- Configuring done
-- Generating done
-- Build files have been written to: /mnt/e/github/miniob/build
Consolidate compiler generated dependencies of target common
[ 21%] Built target common
Consolidate compiler generated dependencies of target common_static
[ 41%] Built target common_static
Consolidate compiler generated dependencies of target obclient
[ 42%] Built target obclient
Consolidate compiler generated dependencies of target observer
[ 66%] Built target observer
Consolidate compiler generated dependencies of target observer_static
[ 90%] Built target observer_static
Consolidate compiler generated dependencies of target client_performance_test
[ 91%] Built target client_performance_test
Consolidate compiler generated dependencies of target bitmap_test
[ 91%] Built target bitmap_test
Consolidate compiler generated dependencies of target bp_manager_test
[ 92%] Built target bp_manager_test
Consolidate compiler generated dependencies of target bplus_tree_test
[ 93%] Built target bplus_tree_test
Consolidate compiler generated dependencies of target log_test
[ 94%] Built target log_test
Consolidate compiler generated dependencies of target md5_test
[ 95%] Built target md5_test
Consolidate compiler generated dependencies of target mem_pool_test
[ 96%] Built target mem_pool_test
Consolidate compiler generated dependencies of target path_test
[ 97%] Built target path_test
Consolidate compiler generated dependencies of target pidfile_test
[ 98%] Built target pidfile_test
Consolidate compiler generated dependencies of target rc_test
[ 99%] Built target rc_test
Consolidate compiler generated dependencies of target record_manager_test
[100%] Built target record_manager_test
yulinux@yu:/mnt/e/github/miniob/build$ ./bin/observer -s miniob.sock -f observer.ini
Successfully load observer.ini
Not set log file name, use default observer.log
Output configuration
Begin dump configuration
[]

Finish dump configuration

Extend one pool, this->size:128, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:256, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:384, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:512, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:640, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:768, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:896, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:1024, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:1152, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:1280, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:1408, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:1536, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:1664, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:1792, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:1920, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:2048, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:2176, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:2304, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:2432, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:2560, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:2688, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:2816, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:2944, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:3072, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:3200, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:3328, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:3456, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:3584, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:3712, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:3840, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:3968, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:4096, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:4224, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:4352, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:4480, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:4608, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:4736, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:4864, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:4992, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:5120, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:5248, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:5376, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:5504, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:5632, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:5760, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:5888, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:6016, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:6144, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:6272, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:6400, item_num_per_pool:128, this->name:BufPool.
Extend one pool, this->size:6400, item_num_per_pool:128, this->name:BufPool.
(Re)Starting State: Pid: 271 Time: Tue Sep 20 10:24:52 2022

The process Name is observer
Successfully init_event_history, EventHistory:0, MaxEventHops:100
Configuration hasn't set ThreadPools
Failed to init thread pool

SedaConfig: unable to initialize seda stages
Failed to init seda configuration!
Shutdown due to failed to init!
Successfully do cleanup, this->name:BufPool.
Shutdown Cleanly!
yulinux@yu:/mnt/e/github/miniob/build$ ps -A
  PID TTY          TIME CMD
    1 ?        00:00:00 init
    9 tty1     00:00:00 init
   10 tty1     00:00:00 bash
  272 tty1     00:00:00 ps

event_set is a deprecated function

event_set is used in server.cpp, but it is deprecated

cannot find gdb in docker

I cannot find the gdb in docker image, but it is important

'client_performance_test' failed

What happened:
In the current version, client_performance_test.cpp uses DEFAULT_PORT as socket communication port, which is different from '*.ini' file provided by the community in ./etc directory. It leads to failed to connect error.
(And guidance commands are not suitable for this, cuz it uses *.sock file)

Should we just change the PORT used in client_performance_test.cpp to the port set by .ini file?

miniob-test2

大家找一块自己的地方，在一个comment里面，可以重复提交测试，每次comment变更，都可以检测到。
在一个时间段内，一个团队的任务提交，只会有一个任务执行。
使用方法：

# 执行测试
test <your-github-name> [case1] [case2] ...

# 查看失败原因
view <your-github-name> <case1> [case2] ...

FAQ:

• 查看失败原因长时间没有返回结果
  如果返回的数据中包含二进制字符，即不是纯文本文件，那么执行结果无法反馈到github中，目前无法看到结果。

• 每个case的名称可以看这个链接
  https://oceanbase-partner.github.io/lectures-on-dbms-implementation/miniob-topics
  basic测试的名称就叫 basic

BanBuDu0

test BanBuDu0 date

简化编译，将依赖模块以external project的方式引入miniob

lex_sql.l 格式混乱

lex_sql.l 格式比较乱

cmake TARGET_LINK_DIRECTORIES in version 3.13

Begin to build miniob
-- This is PROJECT_BINARY_DIR dir /root/miniob/build/src
-- This is PROJECT_SOURCE_DIR dir /root/miniob/src
Begin to build obclient
CMake Error at src/obclient/CMakeLists.txt:11 (TARGET_LINK_DIRECTORIES):
Unknown CMake command "TARGET_LINK_DIRECTORIES".

-- Configuring incomplete, errors occurred!
See also "/root/miniob/build/CMakeFiles/CMakeOutput.log".
[root@host-10-208-65-5 build]# cmake -version
cmake version 3.12.1

TARGET_LINK_DIRECTORIES can be use in version >=3.13

基本的比如insert,delete操作有问题

比如我此图，上面的查询是我更改了delete的逻辑，在dispose_page的时候如果把页面删除掉了就让pageCount--，才不会出现delete后select出错的情况。
还有，原生的insert和delete在执行后均不能在磁盘上持久化，miniob的事务代码中并没有此逻辑，并且也没有让我们实现事务的功能。

'basic' test failed

branch: main

CMakeLists.txt: “-DCMAKE_EXPORT_COMPILE_COMMANDS=1”不应该放在此处.

miniob的小伙伴们你们好，我使用vim搭配coc.nvim以及clangd实现c/c++的自动补全，因此对compile_commands.json有需求。

当前的CMakeLists.txt已经涉及compile_commands.json的生成，但是这个FLAG的位置应该摆错了。

miniob/CMakeLists.txt

Line 40 in c611d5f

SET(CMAKE_COMMON_FLAGS "${CMAKE_COMMON_FLAGS} -Wall -DCMAKE_EXPORT_COMPILE_COMMANDS=1")

虽然CMAKE_COMMON_FLAGS这个变量感觉上是设置了cmake的FLAGS，但是最后其实是赋值给了CMAKE_CXX_FLAGS，而-DCMAKE_EXPORT_COMPILE_COMMANDS=1这个参数是用在cmake上的，而不是gcc。

解决方法是：

1、在CMakeLists.txt开头位置附近加上set(CMAKE_EXPORT_COMPILE_COMMANDS ON)

2、或在运行cmake时加上参数，即cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=1 ..

这样就能生成compile_commands.json：

test <a1445605672> [basic测试]

asan doc is need

Table::make_record 直接使用memcpy，可能导致内存越界访问

memcpy(record + field->offset(), value.data, field->len());

考虑这行代码，如果value.type 是CHARS，并且value.data 的实际长度比较小，比如2个字符，那么执行memcpy会出现越界访问内存的情况

pthread_setname_np 函数的引入导致在Mac上编译失败

请问这里是有bug吗？我是Mac环境，拉取main分支最新的编译出错了。

-- from @Joenhle

此BUG由 #79 引入

load data 去掉了一些空白字符

按照load data 的原意，如果csv文件中包含了一些空白字符，那么也应该导入进来，比如:
1| a |1
那么中间的字符串应该是' a '，而不是'a'

observer 不能运行

按照文档编译成功了，但是运行 observer 报错，如下：

➜  bin git:(main) ✗ ./observer 
Successfully load ../etc/observer.ini
Not set log file name, use default observer.log
Output configuration 
Begin dump configuration
[]

Finish dump configuration 

(Re)Starting State: Pid: 11171 Time: Sat Oct 16 20:55:43 2021

The process Name is observer
Successfully init_event_history, EventHistory:0, MaxEventHops:100
Configuration hasn't set ThreadPools
Failed to init thread pool

SedaConfig: unable to initialize seda stages
Failed to init seda configuration!
Shutdown due to failed to init!

Code format all code by clang-format

test pistacio520

test pistacio520

空跑CPU太高

空跑时，占用一个CPU。
环境：MacOS arm平台
占用CPU的栈信息：

(lldb) bt
* thread #11
  * frame #0: 0x0000000104fdc50c libcommon.1.0.0.dylib`std::__1::__tree<std::__1::__value_type<common::TimerToken, common::StageEvent*>, std::__1::__map_value_compare<common::TimerToken, std::__1::__value_type<common::TimerToken, common::StageEvent*>, bool (*)(common::TimerToken const&, common::TimerToken const&), false>, std::__1::allocator<std::__1::__value_type<common::TimerToken, common::StageEvent*> > >::erase(std::__1::__tree_const_iterator<std::__1::__value_type<common::TimerToken, common::StageEvent*>, std::__1::__tree_node<std::__1::__value_type<common::TimerToken, common::StageEvent*>, void*>*, long>, std::__1::__tree_const_iterator<std::__1::__value_type<common::TimerToken, common::StageEvent*>, std::__1::__tree_node<std::__1::__value_type<common::TimerToken, common::StageEvent*>, void*>*, long>) + 4
    frame #1: 0x0000000104fd3168 libcommon.1.0.0.dylib`std::__1::map<common::TimerToken, common::StageEvent*, bool (*)(common::TimerToken const&, common::TimerToken const&), std::__1::allocator<std::__1::pair<common::TimerToken const, common::StageEvent*> > >::erase(this=0x0000000112804ce0 size=1, __f=std::__1::map<common::TimerToken, common::StageEvent *, bool (*)(const common::TimerToken &, const common::TimerToken &), std::__1::allocator<std::__1::pair<const common::TimerToken, common::StageEvent *> > >::const_iterator @ 0x000000016b734d00, __l=std::__1::map<common::TimerToken, common::StageEvent *, bool (*)(const common::TimerToken &, const common::TimerToken &), std::__1::allocator<std::__1::pair<const common::TimerToken, common::StageEvent *> > >::const_iterator @ 0x000000016b734cf8) at map:1309:25
    frame #2: 0x0000000104fd246c libcommon.1.0.0.dylib`common::TimerStage::check_timer(this=0x0000000112804c00) at timer_stage.cpp:450:18
    frame #3: 0x0000000104fd0470 libcommon.1.0.0.dylib`common::TimerStage::start_timer_thread(arg=0x0000000112804c00) at timer_stage.cpp:423:11
    frame #4: 0x0000000183e9b878 libsystem_pthread.dylib`_pthread_start + 320

[compile issue] Could NOT find OpenSSL

Issue Report

Mac OS, running cmake as it describes in this doc: how to build
At the 'libevent' step, got errors like this:

build git:(5df3037d) cmake .. -DEVENT_DISABLE_OPENSSL=ON                                
CMake Warning at CMakeLists.txt:663 (message):
  Cannot check if kqueue works with pipes when crosscompiling, use
  EVENT__FORCE_KQUEUE_CHECK to be sure (this requires manually running a test
  program on the cross compilation target)


CMake Error at /usr/local/Cellar/cmake/3.18.4/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:165 (message):
  Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
  system variable OPENSSL_ROOT_DIR (missing: OPENSSL_CRYPTO_LIBRARY
  OPENSSL_INCLUDE_DIR)
Call Stack (most recent call first):
  /usr/local/Cellar/cmake/3.18.4/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:458 (_FPHSA_FAILURE_MESSAGE)
  /usr/local/Cellar/cmake/3.18.4/share/cmake/Modules/FindOpenSSL.cmake:486 (find_package_handle_standard_args)
  CMakeLists.txt:844 (find_package)

Running ENV

Software:

    System Software Overview:

      System Version: macOS 12.4 (21F79)
      Kernel Version: Darwin 21.5.0
      Boot Volume: MacintoshHD
      Boot Mode: Normal
      Secure Virtual Memory: Enabled
      System Integrity Protection: Enabled
      Time since boot: 7 days 13 minutes

Hardware:

    Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: MacBookPro16,2
      Processor Name: Quad-Core Intel Core i5
      Processor Speed: 2 GHz
      Number of Processors: 1
      Total Number of Cores: 4
      L2 Cache (per Core): 512 KB
      L3 Cache: 6 MB
      Hyper-Threading Technology: Enabled
      Memory: 16 GB
      System Firmware Version: 1731.120.10.0.0 (iBridge: 19.16.15071.0.0,0)
      OS Loader Version: 540.120.3~6
      Activation Lock Status: Enabled

duplicate codes in bplus_tree.h and tuple_cell.cpp

the codes for attribute compare is duplicate

int operator()(const char *v1, const char *v2) const {
    switch (attr_type_) {
    case INTS: {
      return *(int *)v1 - *(int *)v2;
    }
      break;
    case FLOATS: {
      float result = *(float *)v1 - *(float *)v2;
      if (-1e-6 < result && result < 1e-6) {
        return 0;
      }
      return result > 0 ? 1 : -1;
    }
    case CHARS: {
      return strncmp(v1, v2, attr_length_);
    }
    default:{
      LOG_ERROR("unknown attr type. %d", attr_type_);
      abort();
    }
    }
  }

the other

int TupleCell::compare(const TupleCell &other) const
{
  if (this->attr_type_ == other.attr_type_) {
    switch (this->attr_type_) {
    case INTS: return compare_int(this->data_, other.data_);
    case FLOATS: return compare_float(this->data_, other.data_);
    case CHARS: return compare_string(this->data_, this->length_, other.data_, other.length_);
    default: {
      LOG_WARN("unsupported type: %d", this->attr_type_);
    }
    }
  } else if (this->attr_type_ == INTS && other.attr_type_ == FLOATS) {
    float this_data = *(int *)data_;
    return compare_float(&this_data, other.data_);
  } else if (this->attr_type_ == FLOATS && other.attr_type_ == INTS) {
    float other_data = *(int *)other.data_;
    return compare_float(data_, &other_data);
  }
  LOG_WARN("not supported");
  return -1; // TODO return rc?
}

miniob 在 m1 芯片的 mac 上调试的问题

想问一下 miniob 应该是还没有适配 m1 的 air 对吧，目前我拉取官方的镜像显示的是只支持 linux/amd64 平台。

无法执行git submodule add

尝试根据 doc 添加 deps/libevent 和 deps/googletest 时均失败，报错"already exists in the index"

使用 "git ls-files deps/libevent" 可以看到相关信息，怀疑是cache 导致

执行如下清理之后可以正常执行git add submodule add

git rm --cached deps/libevent
git rm --cached deps/googletest

rm -rf deps/libevent
rm -rf deps/googletest

test OBserver-BUPT basic date update select-meta select-tables aggregation-func drop-table insert join-tables null unique simple-sub-query multi-index text expression complex-sub-query order-by group-by

需要增加.gitignore

增加.gitignore防止误提交一些文件或目录

client 增加更便捷的操作模式

当前的client实现比较简单，并且不支持超长字符串输入，可以考虑使用readline模块实现

编译选项增加-Werror可以提前发现一些BUG

增加-Werror，可以帮助提前发现一些问题，比如函数没有返回值。
现在的CMake中增加了-Wall，但是还存在一些编译告警，消灭编译告警后，才可以加上-Werror

一键安装开发环境的脚本

如果不使用docker，希望可以提供一个更小白的，能够可以一键搭建起来开发环境的脚本。

mac 上使用gcc 12.2编译失败

将单元测试模块设置为可选择依赖

build googletest failed

when execute make command according to build.md step by step, it failed like this:

make[2]: *** [googletest/CMakeFiles/gtest.dir/src/gtest-all.cc.o] Error 1
make[1]: *** [googletest/CMakeFiles/gtest.dir/all] Error 2

tried to update gcc from 4.8 to upper version, but it didn't work.

Solution: checkout to googletest release version instead of using master branch to build

build google test step

git submodule add https://github.com/google/googletest deps/googletest 
cd deps
cd googletest
git checkout release-1.8.1  #add this step to checkout to release-1.8.1: the lateast release version
mkdir build
cd build
cmake ..
make
sudo make install

server.cpp:send中关闭连接后，应该终止会话

在server.cpp: send中，发送消息出现异常，会close_connection，但是没有终止会话，导致还可能再次通过连接发送消息，引起coredump

ASAN

m1 系统 server 关闭后没有执行刷盘导致重启数据失效

m1 系统 kill server 之后会因为文件没有刷盘而导致重启获取不到数据
用 ls -l 查看可以发现在 kill 后文件没有修改

test_bingandbing

test bingandbing drop-table select-meta

Add lecture

Add database lecture

rewrite B+ tree

The B+ tree's implementaion isn't good, it had better be reimplemented. The speed is low, at the same time, there is no unit test for it.

penggan666:

test penggan666 complex-sub-query