tomarrell/lbadd

LBADD: An experimental, distributed SQL database

G'day 👋

🛠 I build things from time to time. Contributions are welcome.

🖋 I also maintain a blog where I write about some of my projects. For example, how I built the keyboard that I use on a daily basis.

lbadd's Issues

parser: statement ending in whitespace causes panic

Statements ending in whitespaces, such as SELECT cause a panic.

panic: runtime error: index out of range [7] with length 7 [recovered]
        panic: runtime error: index out of range [7] with length 7

goroutine 19 [running]:
testing.tRunner.func1(0xc0000f2e00)
        /usr/local/go/src/testing/testing.go:874 +0x3a3
panic(0x13282c0, 0xc0000167c0)
        /usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/tomarrell/lbadd/internal/parser/scanner.(*ruleBasedScanner).ConsumeRune(0xc0000ee880)
        /Users/tsatke/Development/private/github.com/tomarrell/lbadd/internal/parser/scanner/rule_based_scanner.go:154 +0x77
github.com/tomarrell/lbadd/internal/parser/scanner.(*ruleBasedScanner).applyRule(0xc0000ee880, 0x13b251a, 0x12e7ae0)
        /Users/tsatke/Development/private/github.com/tomarrell/lbadd/internal/parser/scanner/rule_based_scanner.go:99 +0x13c
github.com/tomarrell/lbadd/internal/parser/scanner.(*ruleBasedScanner).computeNext(0xc0000ee880, 0x0, 0xc000113b01)
        /Users/tsatke/Development/private/github.com/tomarrell/lbadd/internal/parser/scanner/rule_based_scanner.go:84 +0x6a
github.com/tomarrell/lbadd/internal/parser/scanner.(*ruleBasedScanner).Peek(...)
        /Users/tsatke/Development/private/github.com/tomarrell/lbadd/internal/parser/scanner/rule_based_scanner.go:61
github.com/tomarrell/lbadd/internal/parser/scanner.(*ruleBasedScanner).Next(0xc0000ee880, 0x12e7ae0, 0x13b251a)
        /Users/tsatke/Development/private/github.com/tomarrell/lbadd/internal/parser/scanner/rule_based_scanner.go:54 +0x78
github.com/tomarrell/lbadd/internal/parser/scanner.TestRuleBasedSannerStatementEndingInWhitespace(0xc0000f2e00)
        /Users/tsatke/Development/private/github.com/tomarrell/lbadd/internal/parser/scanner/rule_based_scanner_test.go:191 +0x260
testing.tRunner(0xc0000f2e00, 0x1368268)
        /usr/local/go/src/testing/testing.go:909 +0xc9
created by testing.(*T).Run
        /usr/local/go/src/testing/testing.go:960 +0x350
FAIL    github.com/tomarrell/lbadd/internal/parser/scanner      0.018s

btree: finish btree implementation

• search
• insert
• delete

Purpose of this Package? Does it provide any performance improvement against sqlite, or similar?

Purpose of this Package? Does it provide any performance improvement against sqlite, or similar?

how to handle tool dependencies

How should we handle tool dependencies?

• Should our Makefile go get them?
• Should we introduce a tools.go file with blank imports to the tools?

executor: build executor and define its commands

The executor needs to take an instruction, and perform the necessary operations to the storage layer.

The executor is able to execute a command, which is the intermediary representation of one parsed sql statement.

• define command model (#43)
• create executor (#42)

build IR codegen from parsed query

internal/parser: implement 7 query expressions

• 7.1 row value constructor
• 7.2 row value expression
• 7.3 table value constructor
• 7.4 table expression
• 7.5 from clause
• 7.6 table reference
• 7.7 row pattern recognition clause
• 7.8 row pattern measures
• 7.9 row pattern common syntax
• 7.10 joined table
• 7.11 JSON table
• 7.12 where clause
• 7.13 group by clause
• 7.14 having clause
• 7.15 window clause
• 7.16 query specification
• 7.17 query expression

internal/parser: typo in error message

Error message unsupported construct has a typo in word length.

error while parsing command: unsupported construct: EOF() at (1:5) offset 4 lenght 0

Steps to reproduce

Try to parse asdf

create security policy

• update SECURITY.md
• provide public key for encrypting sensitive messages
• provide email for receiving encrypted messages

internal/parser: implement parser for SQL-2016 standard

Currently the parser supports only extremely basic SELECT statements.

We need to add support for SELECT, INSERT, CREATE, UPDATE, DELETE

Upgrade memguard to 0.22.2

There has been a version increment, and the change log states a fix of a deadlock. We do not use the functionality that the bug affected, however, upgrading would be the right thing to do.

Literal "table1" erroneously recognised as keyword Table.

Describe the bug
A clear and concise description of what the bug is.

The keyword table is being recognised by the scanner in the case of the word table1 instead of a literal table1

To Reproduce
Steps to reproduce the behavior:

Test for the keyword table1

Expected behavior
A clear and concise description of what you expected to happen.
Literal table1 instead of KeywordTable

Literal needs to be replaced with NumericLiteral

Describe the bug
Post addition of NumericLiteral to keyword set,
the implementation in https://github.com/tomarrell/lbadd/blob/master/internal/parser/simple_parser_rules.go#L353 needs to check for only NumericLiteral instead of Literal.

Additional context

NumericLiterals consist of only number, whereas Literals can be all alphanumeric permutations. SIgned numbers must accept only +/-Number and currently, accepts +/-xxxxx too.

continuous builds

At some point it would make sense to introduce continuous builds, preferably GitHub Actions. This would be especially useful to ensure high code quality and consistency across contributors, given that the CI performs not only building and testing, but also linting.

tool/cmp: remove dependency to spew

The ast comparison tool currently uses spew to generate a comparable string output. This dependency should be removed when possible.

errcheck Failed

⚠ errcheck Failed

repl.go:46:21:	r.executor.execute(instr)

Originally posted by @github-actions in #10 (comment)

remove golint from CI

As per golang/lint#176 (comment):
golint is not designed to be a part of a build/CI pipeline. From the README:

In short, this tool is not, and will never be, trustworthy enough for its suggestions to be enforced automatically, for example as part of a build process.

We have staticcheck, errcheck, gosec as well as a ton of other analyzers, which should be reliable enough to remove golint.

parser: single table selects should not be considered join statements

Describe the bug
When parsing a select statement that selects over a single table, that table name is wrapped in the join clause in the AST. It should however be contained in the TableOrSubquery part of the SelectCore.

To Reproduce
Parse SELECT col FROM users WHERE otherColumn IS someValue.
This will yield a SelectCore like follows.

(*ast.SelectCore)(0xc00015aa00)({
 Select: (token.tok) KeywordSelect(SELECT),
 Distinct: (token.Token) <nil>,
 All: (token.Token) <nil>,
 ResultColumn: ([]*ast.ResultColumn) (len=1 cap=1) {
  (*ast.ResultColumn)(0xc0001266c0)({
   Expr: (*ast.Expr)(0xc00017e840)({
    LiteralValue: (token.tok) Literal(col),
    ...
   }),
   ...
  })
 },
 From: (token.tok) KeywordFrom(FROM),
 JoinClause: (*ast.JoinClause)(0xc00011b0e0)({
  TableOrSubquery: (*ast.TableOrSubquery)(0xc0001a0100)({
   ...
   TableName: (token.tok) Literal(users),
   ...
  }),
  ...
 }),
 TableOrSubquery: ([]*ast.TableOrSubquery) <nil>,
 Where: (token.tok) KeywordWhere(WHERE),

Expected behavior
The TableOrSubquery must not be wrapped into the JoinClause.

Additional context
The fix must be implemented inside the parser package, internal/parser/simple_parser.go to be exact.

• If there is no JOIN specified, the tables must be in the TableOrSubquery of the SelectCore
• If there is a JOIN specified, all joined tables must be inside the JoinClause

testissue

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here, like the $go env output, if you were developing on the project.

internal/cli: implement rich cli

The current CLI implementation is called simple_cli, and for a reason.
It doesn't even support going through old commands via up-arrow, such as the common console.

The aim of this issue is, to implement a rich CLI with the help of https://github.com/c-bata/go-prompt.

build a conversion tool from AST to IR

Depends on #2

internal/parser: implement 6 scalar expressions

• 6.1 data type
• 6.2 field definition
• 6.3 value expression primary
• 6.4 value specification and target specification
• 6.5 contextually typed value specification
• 6.6 identifier chain
• 6.7 column reference
• 6.8 SQL parameter reference
• 6.9 set function specification
• 6.10 window function
• 6.11 nested window function
• 6.12 case expression
• 6.13 cast specification
• 6.14 next value expression
• 6.15 field reference
• 6.16 subtype treatment
• 6.17 method invocation
• 6.18 static method invocation
• 6.19 new specification
• 6.20 attribute or method reference
• 6.21 dereference operation
• 6.22 method reference
• 6.23 reference resolution
• 6.24 array element reference
• 6.25 multiset element reference
• 6.26 row pattern navigation operation
• 6.27 JSON value function
• 6.28 value expression
• 6.29 numeric value expression
• 6.30 numeric value function
• 6.31 string value expression
• 6.32 string value function
• 6.33 JSON value constructor
• 6.34 JSON query
• 6.35 datetime value expression
• 6.36 datetime value function
• 6.37 interval value expression
• 6.38 interval value function
• 6.39 boolean value expression
• 6.40 array value expression
• 6.41 array value function
• 6.42 array value constructor
• 6.43 multiset value expression
• 6.44 multiset value function
• 6.45 multiset value constructor

internal/parser: implement statement parsing

• parse SQLStmt
• parse AlterTableStmt
• parse AnalyzeStmt
• parse AttachStmt
• parse BeginStmt
• parse CommitStmt
• parse CreateIndexStmt
• parse CreateTableStmt
• parse CreateTriggerStmt
• parse CreateViewStmt
• parse CreateVirtualTableStmt
• parse DeleteStmt
• parse DeleteStmtLimited
• parse DetachStmt
• parse DropIndexStmt
• parse DropTableStmt
• parse DropTriggerStmt
• parse DropViewStmt
• parse QualifiedTableName
• parse InsertStmt
• parse ParenthesizedExpressions
• parse ReindexStmt
• parse SavepointStmt
• parse ReleaseStmt
• parse RollbackStmt
• parse SelectStmt
• parse SelectCore
• parse UpdateStmt
• parse UpdateSetter
• parse UpdateStmtLimited
• parse UpsertClause
• parse VacuumStmt
• parse WithClause
• parse RecursiveCte
• parse CteTableName
• parse Expr
• parse FilterClause
• parse OverClause
• parse RaiseFunction
• parse OrderingTerm
• parse ResultColumn
• parse NamedWindow
• parse WindowDefn
• parse FrameSpec
• parse TableConstraint
• parse ForeignKeyClause
• parse CommonTableExpression
• parse CompoundOperator
• parse TableOrSubquery
• parse JoinClause
• parse JoinClausePart
• parse JoinConstraint
• parse JoinOperator
• parse ColumnDef
• parse ColumnConstraint
• parse ColumnNameList
• parse ConflictClause
• parse TypeName
• parse IndexedColumn

parser: finish rule implementation

Rule implementation must be completed. Implementors must only work inside exactly one function at a time. When implementing another function, this must be coordinated across all implementors.

• parse SQLStmt #102
• parse AlterTableStmt 5e2d484
• parse AnalyzeStmt #75
• parse AttachStmt #71
• parse BeginStmt #77
• parse CommitStmt #77
• parse CreateIndexStmt #79
• parse CreateTableStmt #96
• parse CreateTriggerStmt #102
• parse CreateViewStmt #102
• parse CreateVirtualTableStmt #102
• parse DeleteStmt #83
• parse DeleteStmtLimited #102
• parse DetachStmt #73
• parse DropIndexStmt #102
• parse DropTableStmt #102
• parse DropTriggerStmt #102
• parse DropViewStmt #102
• parse QualifiedTableName #83
• parse InsertStmt #102
• parse ParenthesizedExpressions #83
• parse ReindexStmt #102
• parse SavepointStmt #102
• parse ReleaseStmt #102
• parse RollbackStmt #77
• parse SelectStmt #83
• parse SelectCore #83
• parse UpdateStmt #102
• parse UpdateSetter #102
• parse UpdateStmtLimited #102
• parse UpsertClause #102
• parse VacuumStmt #74
• parse WithClause #83
• parse RecursiveCte #83
• parse CteTableName #83
• parse Expr
• parse FilterClause #102
• parse OverClause #102
• parse RaiseFunction #102
• parse OrderingTerm #83
• parse ResultColumn #83
• parse NamedWindow #83
• parse WindowDefn #83
• parse FrameSpec #83
• parse TableConstraint #96
• parse ForeignKeyClause #96
• parse CommonTableExpression #83
• parse CompoundOperator #83
• parse TableOrSubquery #83
• parse JoinClause #83
• parse JoinClausePart #83
• parse JoinConstraint #83
• parse JoinOperator #83
• parse ColumnDef
• parse ColumnConstraint #96
• parse ColumnNameList #102
• parse ConflictClause
• parse TypeName
• parse IndexedColumn #79

driver: create first integration test

This is a test issue. Create the first issue-based integration test in driver/test/issue<number>_test.go and call the test function TestIssue<number>(...).

parser: literal must be allowed to be a single question mark

Prepared statements require placeholders, which shall be literals with the content ?.
They must be emitted as common literals. The executor can then expect the arguments, one for every placeholder.

docs: create document with component explanation

It's been pointed out that it's not entirely clear what the purpose of each component in the database is.

Therefore we should have a document which takes a high level approach to describing the function of each component.

raft: implement protocol

This will be a progress tracker for implementation of raft.
This doesn't include tests, it's a separate issue (#125 ) though this is a part of #121.

• Leader Election
• Append Entries
• Heartbeats
• Request Votes

parser: add parser support for any SQL dialect

Currently, the root node of an AST has to be *ast.SQLStmt. This could be changed to

type SQLStmt interface {
    Dialect() Dialect
}

type Dialect uint16
const (
    Lbadd Dialect = iota
    SQLite
    Oracle
    PostgreSQL
    MySQL
)

When parsing, the dialect has to be known. Maybe we can implement a dialect detection algorithm, which suggests dialects for a specific input.
When compiling, the compiler can check the dialect, and check, if it is supported/enabled/etc..
A Dialect has to be associated with a set of rules, to let the scanner know, what ruleset to use. It also should be associated with (probably) a reflect.Type, which is the actual type of the root node (like *ast.SQLStmt, which will then probably be something like *ast.Lbadd or *ast.SQLite).

Maybe the dialect implementation should rather look like

type Dialect struct {
    Name     string
    Ruleset  *ruleset.Ruleset
    StmtType reflect.Type
}

Scanner doesnt support `NULLS` keyword.

Describe the bug

Scanner doesnt have support for NULLS keyword.

Additional context
Due to non-existance of keyword in

lbadd/internal/parser/scanner/ruleset/ruleset_default.go

Line 60 in fb91b74

defaultKeywords = map[string]token.Type{"ABORT": token.KeywordAbort, "ACTION": token.KeywordAction, "ADD": token.KeywordAdd, "AFTER": token.KeywordAfter, "ALL": token.KeywordAll, "ALTER": token.KeywordAlter, "ANALYZE": token.KeywordAnalyze, "AND": token.KeywordAnd, "AS": token.KeywordAs, "ASC": token.KeywordAsc, "ATTACH": token.KeywordAttach, "AUTOINCREMENT": token.KeywordAutoincrement, "BEFORE": token.KeywordBefore, "BEGIN": token.KeywordBegin, "BETWEEN": token.KeywordBetween, "BY": token.KeywordBy, "CASCADE": token.KeywordCascade, "CASE": token.KeywordCase, "CAST": token.KeywordCast, "CHECK": token.KeywordCheck, "COLLATE": token.KeywordCollate, "COLUMN": token.KeywordColumn, "COMMIT": token.KeywordCommit, "CONFLICT": token.KeywordConflict, "CONSTRAINT": token.KeywordConstraint, "CREATE": token.KeywordCreate, "CROSS": token.KeywordCross, "CURRENT": token.KeywordCurrent, "CURRENT_DATE": token.KeywordCurrentDate, "CURRENT_TIME": token.KeywordCurrentTime, "CURRENT_TIMESTAMP": token.KeywordCurrentTimestamp, "DATABASE": token.KeywordDatabase, "DEFAULT": token.KeywordDefault, "DEFERRABLE": token.KeywordDeferrable, "DEFERRED": token.KeywordDeferred, "DELETE": token.KeywordDelete, "DESC": token.KeywordDesc, "DETACH": token.KeywordDetach, "DISTINCT": token.KeywordDistinct, "DO": token.KeywordDo, "DROP": token.KeywordDrop, "EACH": token.KeywordEach, "ELSE": token.KeywordElse, "END": token.KeywordEnd, "ESCAPE": token.KeywordEscape, "EXCEPT": token.KeywordExcept, "EXCLUDE": token.KeywordExclude, "EXCLUSIVE": token.KeywordExclusive, "EXISTS": token.KeywordExists, "EXPLAIN": token.KeywordExplain, "FAIL": token.KeywordFail, "FILTER": token.KeywordFilter, "FIRST": token.KeywordFirst, "FOLLOWING": token.KeywordFollowing, "FOR": token.KeywordFor, "FOREIGN": token.KeywordForeign, "FROM": token.KeywordFrom, "FULL": token.KeywordFull, "GLOB": token.KeywordGlob, "GROUP": token.KeywordGroup, "GROUPS": token.KeywordGroups, "HAVING": token.KeywordHaving, "IF": token.KeywordIf, "IGNORE": token.KeywordIgnore, "IMMEDIATE": token.KeywordImmediate, "IN": token.KeywordIn, "INDEX": token.KeywordIndex, "INDEXED": token.KeywordIndexed, "INITIALLY": token.KeywordInitially, "INNER": token.KeywordInner, "INSERT": token.KeywordInsert, "INSTEAD": token.KeywordInstead, "INTERSECT": token.KeywordIntersect, "INTO": token.KeywordInto, "IS": token.KeywordIs, "ISNULL": token.KeywordIsnull, "JOIN": token.KeywordJoin, "KEY": token.KeywordKey, "LAST": token.KeywordLast, "LEFT": token.KeywordLeft, "LIKE": token.KeywordLike, "LIMIT": token.KeywordLimit, "MATCH": token.KeywordMatch, "NATURAL": token.KeywordNatural, "NO": token.KeywordNo, "NOT": token.KeywordNot, "NOTHING": token.KeywordNothing, "NOTNULL": token.KeywordNotnull, "NULL": token.KeywordNull, "OF": token.KeywordOf, "OFFSET": token.KeywordOffset, "ON": token.KeywordOn, "OR": token.KeywordOr, "ORDER": token.KeywordOrder, "OTHERS": token.KeywordOthers, "OUTER": token.KeywordOuter, "OVER": token.KeywordOver, "PARTITION": token.KeywordPartition, "PLAN": token.KeywordPlan, "PRAGMA": token.KeywordPragma, "PRECEDING": token.KeywordPreceding, "PRIMARY": token.KeywordPrimary, "QUERY": token.KeywordQuery, "RAISE": token.KeywordRaise, "RANGE": token.KeywordRange, "RECURSIVE": token.KeywordRecursive, "REFERENCES": token.KeywordReferences, "REGEXP": token.KeywordRegexp, "REINDEX": token.KeywordReindex, "RELEASE": token.KeywordRelease, "RENAME": token.KeywordRename, "REPLACE": token.KeywordReplace, "RESTRICT": token.KeywordRestrict, "RIGHT": token.KeywordRight, "ROLLBACK": token.KeywordRollback, "ROW": token.KeywordRow, "ROWS": token.KeywordRows, "SAVEPOINT": token.KeywordSavepoint, "SELECT": token.KeywordSelect, "SET": token.KeywordSet, "TABLE": token.KeywordTable, "TEMP": token.KeywordTemp, "TEMPORARY": token.KeywordTemporary, "THEN": token.KeywordThen, "TIES": token.KeywordTies, "TO": token.KeywordTo, "TRANSACTION": token.KeywordTransaction, "TRIGGER": token.KeywordTrigger, "UNBOUNDED": token.KeywordUnbounded, "UNION": token.KeywordUnion, "UNIQUE": token.KeywordUnique, "UPDATE": token.KeywordUpdate, "USING": token.KeywordUsing, "VACUUM": token.KeywordVacuum, "VALUES": token.KeywordValues, "VIEW": token.KeywordView, "VIRTUAL": token.KeywordVirtual, "WHEN": token.KeywordWhen, "WHERE": token.KeywordWhere, "WINDOW": token.KeywordWindow, "WITH": token.KeywordWith, "WITHOUT": token.KeywordWithout}

internal/parser: implement AST

• 6.27 JSON value function
• 6.28 value expression
• 6.29 numeric value expression
• 6.30 numeric value function
• 6.31 string value expression
• 6.32 string value function
• 6.33 JSON value constructor
• 6.34 JSON query
• 6.35 datetime value expression
• 6.36 datetime value function
• 6.37 interval value expression
• 6.38 interval value function
• 6.39 boolean value expression
• 6.40 array value expression
• 6.41 array value function
• 6.42 array value constructor
• 6.43 multiset value expression
• 6.44 multiset value function
• 6.45 multiset value constructor

compiler: implement conversion from ast to command

A command is created by converting an AST to a command.
The function doing this is called command.From(*ast.SQLStmt).
This function must be implemented.

golint Failed

⚠ golint Failed

Found 3 lint suggestions; failing.

Show Detail

btree.go:169:10: if block ends with a return statement, so drop this else and outdent its block
repl.go:14:1: exported function NewRepl should have comment or be unexported
repl.go:14:16: exported func NewRepl returns unexported type *lbadd.repl, which can be annoying to use

Originally posted by @github-actions in #10 (comment)

parser: with statement hangs while parsing

Describe the bug
When parsing WITHe AS(VALUES(1,myExpr2)) DELETE FROM myTable, the parser hangs.

Expected behavior
This should be parsed or at least display an error.

testissue2

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

internal/parser: define error handling for error cases

The error handling of the parser has to be defined. Depending on the error, the parser could either skip a token, skip as many tokens as neccessary or just reset its state. The strategy has to be defined and must be dependent on the current state of the parser.

slack channel

Is there a slack channel or similar that facilitates communication?

command: define commands

Commands are a more functional representation of an AST that was parsed by the parser.
The structure must be defined, in order to represent an AST, so that everything in the AST can be executed.
Some information can be discarded, such as location information of tokens or constant token values.

nopanic: nopanic should check whether error is recovered

Currently, the nopanic analyzer only checks if there is a panic in the codebase. If so, an error is reported, no further checks done. This analyzer shall be extended, so that it only reports the panic as disallowed, if one or more of the following rules apply:

• the panic occurs inside the main package
• the panic is not recovered from within the same package
• the panic panics anything that does not implement the error interface
• the panic panics nil

I. e. panics are allowed, if and only if all of the following rules apply:

• they propagate an error up the callstack in the same package
• they are recovered from, so that it can't leave the package and can't crash the application
• if they are recovered, it must be ensured that every path actually catches the error

Examples

func main() {
    defer func(){
        if err := recover(); err != nil {
            fmt.Println(err)
        }
    }()
    panic("") // allowed
}

func main() {
    defer func(){
        if err := recover(); err != nil {
            if myErr, ok := err.(*MyErr); ok {
                fmt.Println(myErr)
            }
            panic(err) // not allowed
        }
    }()
    panic("") // allowed
}

parser: implement fuzzy testing

implement fuzzy testing using go-fuzz for the sql parser.

• implement a Fuzz(data []byte) function
• add fuzzing target to Makefile
• use https://github.com/dvyukov/go-fuzz-corpus/tree/master/sqlparser/corpus as initial corpus if possible

Additional resources

• https://github.com/fuzzitdev/example-go
• https://medium.com/@dgryski/go-fuzz-github-com-arolek-ase-3c74d5a3150c

define intermediary representation

staticcheck Failed

⚠ staticcheck Failed

btree.go:187:3: empty branch (SA9003)
btree_test.go:291:3: field order is unused (U1000)
executor.go:16:6: type record is unused (U1000)
executor.go:19:6: type row is unused (U1000)
executor.go:23:2: field columns is unused (U1000)
executor.go:24:2: field rows is unused (U1000)
executor.go:25:2: field rowsAffected is unused (U1000)
query.go:6:2: field conditions is unused (U1000)
query.go:7:2: field updates is unused (U1000)
query.go:8:2: field inserts is unused (U1000)
query.go:72:6: type condition is unused (U1000)

Checks Document

Originally posted by @github-actions in #10 (comment)

define consensus protocol

The database needs to handle multiple nodes over a network.

Prior art in this area is exists in https://github.com/rqlite/rqlite and can possibly be used as inspiration

We are looking for something which can exist as a barebones implementation to begin with, which is well isolated from the rest of the system to allow for further extension.

docs: add contributing document

We need some info describing how to contribute to the project.

executor: implement executor

An executor is the connection between the database input service and the database layer. Somewhere, an SQL statement is parsed and converted to the IR, which are commands that the executor can execute.
The executor can then execute the appropriate methods and functions on the database in order to achieve, what the command describes.

internal/parser: rule based scanner should not skip to whitespace

At the moment, the rule based scanner skips to the next whitespace, if no rules can be applied. What it should do instead is, consume one rune and mark it as unexpected token, then try all rules again at the next rune.

unhandled errors in repl.go

⚠ gosec Failed

Summary:
   Files: 12
   Lines: 896
   Nosec: 0
  Issues: 1

Show Detail

Results:


[/github/workspace/repl.go:46] - G104 (CWE-703): Errors unhandled. (Confidence: HIGH, Severity: LOW)
  > r.executor.execute(instr)


Summary:
   Files: 12
   Lines: 896
   Nosec: 0
  Issues: 1

Code Reference

Originally posted by @github-actions in #10 (comment)

raft: integration tests

We need extensive integration tests on the raft cluster covering all possibilities.
The test must use the underlying cluster package and actually create clusters instead of mocking them.

Following are the scenarios we need to consider.

• Normal raft operation.
• Leader failure.
• Leader recovery.
• Multiple leader problem.
• Network partitions and their recovery.
• Follower join.
• Follower graceful shutdown.
• Mock tests where the non-mocked node is a leader.
• Mock tests where the non-mocked node is a follower.

move tomarrell/lbadd to own organization

The point in time to do this is NOT NOW, however, we should maybe think about this at some point.

Besides the core lbadd, there will be static analysis tools (at least for the parser) that help developers develop better and more easily.
Other than analysis tools, there might be custom written code generator tools at some point, as well as a documentation homepage.
Maybe we even want to outsource certain components instead of having everything in one big repo. This would also allow other developers to re-use components that we built (like uber does with its libraries, we could outsource the parser, so that other projects have a working SQL parser (we keep the IR)).

Before doing this, we should build a sensible structure that works for us.

switch entry point architecture

Since we use raft as a protocol, here is a proposal for a new architecture.

The database can either be started in leader or in worker mode. When started in leader mode, an additional CLI starts, unless the application is started in headless mode.

start a leader in headless mode (no cli)

lbadd -leader -headless -addr ":54321"

start a worker and connect to a leader

lbadd -worker -addr ":54321"

Sample implementation as an example will follow.