man-group/arcticdb

ArcticDB is a high performance, serverless DataFrame database built for the Python Data Science ecosystem.

arcticdb's Issues

Wipe code history prior to OSS

• Remove all branches
• Start history of master

Apply BSL License

• Add the BSL header to all source files
• Adding the BSL license text to the license/ directory

Make the static key map static

It's a bit sad at the moment that we use a semimap but it's in a runtime context so it just reverts to std::unordered_map behaviour. Fix it so that it is genuinely compile-time.

Build Instructions on Readme

Remove thread local OpContext and pass arguments directly as parameters

The static thread_locals are potentially unpredictable in their order of initialization and destruction, and make the possible influences on the operation of a function indirect and hard to reason about. Replace this with normal parameter objects.

Pandas interface for writing sparse data

• Providing an API to write sparse data using the standard ArcticDB Pandas API. Null/None values will not be stored.

Defragmentation/Compaction

Tasks

• #33

Misc small docs improvements

• - Link to arcticdb.com from first "ArcticDB" string on getting started page
• - Version maps not compacted - say that accessing early versions when there are many will be slower than more recent versions, with compaction API coming soon
• - Ability to change some lib config settings after lib creation time coming soon
• - Get rid of LibraryConfiguration docs page and just link to LibraryOptions API docs
• - Explanation of NaN handling
• - Explanation of Categoricals handling
• - Coming soon - compaction API (solves data is fragmented problem)

Add refresh_symbol_list_cache method to Library

We need a way for people to dig themselves out of a corrupted symbol list, and the alternative is documenting something like lib._nvs.version_store._clear_symbol_list_cache()

Increase Python Pickle protocol to 4

We currently have the pickle protocol set as 2 for Python 2 compat.

We should be able to increase to 4 as a performance improvement.

(See internal ticket RAP-6273).

Usage Instructions on Readme

Advanced usage notebook

Test

Support for additional data types

FAQ - Arrow integration timeline

Manually test version 1.0.0

License information on README

Fix incorrect handling of negative numpy strides

Remove PyMongo dep from public API

FAQ - IO/CPU Thread configuration

This should include information on how this is managed (by default) when running on K8S.

Expose dedup and snapshot dedup as library options

Modify MongoClient to produce a runtime error when failing to read a segment

Publish documentation on Cloudflare

Simplify symbol list caching logic

Debugging/toolbox usage notebook

GitHub Actions-based Build Tooling (Releases)

• Wheel as artifact
• Runs auditwheel
• Ensure build exports debug symbols but builds in release mode
• Test that a core dump can be retrieved from the release build, and debugged using the debug build
• Ensure we can release off of GitHub

For the releases:

• Only take place on release branches
• Tag the release

Update image on "History of ArcticDB" page with 2023, BBG deal, and first release

Documentation Link on Readme

Conda build

GitHub Actions-based Build Tooling (Pull Requests)

• Runs tests

Scan for unwanted references to Man

Upload 1.0.0 to PyPi

Getting started notebook

Add documentation to repository

Apply BSL header to files

FAQ - Comparisons with Parquet/DeltaLake

Expose column/row slicing sizes as configurable library options

FAQ - Library configuration options

Reject incompatible append/updates

CoC & Contributing information on Readme

Docs accepted data/index types

• What index types are supported
• What data types are supported
• What combination of types are supported by what combination of operation

Blog review

Introduce error handling framework

FAQ - Common error messages

This should include details on how to configure ArcticDB logging and errors, such as:

• How to configure the different logging sink levels
• How to configure the AWS SDK logging level
• Information on how the Python logs relate to the C++ logs

Document how the symbol list (inc compaction) works

Apply LibraryOptions to lmdb libs (currently just S3)

GitHub Actions-based Build Tooling (documentation)

Flesh out ArcticDB Data Segments in the on-disk storage section

Reset version number to 1.0.0

Introduce initial defragmentation API