cube2222/octosql

OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.

octosql's Issues

Implement CSV backend

Implement in memory filter (where)

Implement RecordStream.Close()

Design a sensible predicate notation

Implement actual optimizations.

Design streaming support.

Implement Inner Join

Doesn't show data from CSV and MySQL

In my case, I am trying to join data from a CSV and MySQL source. But in the end, it shows nothing in the output.

Platform: Windows, 64 Bit

Design physical plan

Design logical plan

Idea: The logical plan node should have a method getphysical and should, after calling on the root node, create a naive physical plan for the whole tree.

Implement ANY

Implement Union All

Implement Postgres backend

Implement data source and filter merge optimization

Implement Distinct

Implement ALL

JSON: null value causes invalid memory address or nil pointer dereference error

Platform: Windows

example.json:

[{
  "nameCache": null
}]

settings.yaml

dataSources:
  - name: t
    type: json
    config:
      path: "example.json"
      arrayFormat: true

command
octosql "select * from t as tt" -c settings.yaml

result:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0xf8 pc=0x51bce6]

goroutine 1 [running]:
github.com/cube2222/octosql.NormalizeType(0x0, 0x0, 0xc00009b988, 0x2)
        /go/src/github.com/{{ORG_NAME}}/{{REPO_NAME}}/values.go:235 +0xbb6
github.com/cube2222/octosql/storage/json.(*RecordStream).Next(0xc0001064b0, 0xc000079e60, 0x9c9e20, 0xc0001064b0)
        /go/src/github.com/{{ORG_NAME}}/{{REPO_NAME}}/storage/json/datasource.go:120 +0x32d
github.com/cube2222/octosql/app.(*App).RunPlan(0xc00009bd18, 0x9cf420, 0xc0000700b0, 0x9c4740, 0xc0000de460, 0x0, 0xc00009bd38)
        /go/src/github.com/{{ORG_NAME}}/{{REPO_NAME}}/app/app.go:45 +0x1a3
main.glob..func1(0xd31520, 0xc000079aa0, 0x1, 0x3)
        /go/src/github.com/{{ORG_NAME}}/{{REPO_NAME}}/cmd/octosql/main.go:94 +0x594
github.com/spf13/cobra.(*Command).execute(0xd31520, 0xc00006c0d0, 0x3, 0x3, 0xd31520, 0xc00006c0d0)
        /go/pkg/mod/github.com/spf13/[email protected]/command.go:830 +0x2b5
github.com/spf13/cobra.(*Command).ExecuteC(0xd31520, 0xd36a90, 0x924ae8, 0x6)
        /go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x303
github.com/spf13/cobra.(*Command).Execute(...)
        /go/pkg/mod/github.com/spf13/[email protected]/command.go:864
main.main()
        /go/src/github.com/{{ORG_NAME}}/{{REPO_NAME}}/cmd/octosql/main.go:105 +0x157

Implement logic operators

Implement ast -> logical plan conversion

Implement Redis backend

Add VictoriaMetrics support from sql to promsql

https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/Single-server-VictoriaMetrics

Design and implement arithmetics

This should probably be an extension of #26

Implement physical plan execution

Please add support for CouchDB

Please add support for CouchDB :)

Thank you

Integrate visualization for describe-like functionality

https://github.com/bradleyjkemp/memviz this with minor modifications to the code (inlining of pointers) works great.

Design and implement helper functions

Implement logi operators

Implement IN parsing

Can you add nosql scylla support?

https://github.com/scylladb/gocqlx

Implement in memory index

Implement Offset

Implement yaml data source configuration

Add Cassandra support

Write a project description note for Grześ

https://docs.google.com/document/d/1toanS54SYzr0OjeWnOvTXfY8SCU7hpqnYIEPmPWAEDk/edit?usp=sharing

Implement In Memory data source for testing purposes

Implement Left Join

Add a Schema Compare Feature

This feature will be very helpful in generating sync-scripts that can allow you to compare different aspects or all of the schema where you can run that script to match the target schema. I've used a tool that did this (which was solid), but the licensing fees were absurd.

Create a framework for scenario-matching-based optimizations.

Implement Order By

Implement JSON backend

Implement relation operators

Implement Group By

MSSQL Storage

Hi there,

Very interesting tools. Thank you in advance.
I see that MSSQL is not in the roadmap.

How could it be implemented? Adding a dedicated datastore would be enough?
I could eventually help in the process

Implement Union

Design backend interfaces

Implement Limit

Add SSH tunneling support

Would be great to be able to connect over a bastion SSH host to the actual database.

Support Excel (.xlsx) files as a Datasource

Seeing how useful is to be able to query with SQL in Excel and similar spreadsheets it would be useful to bring this functionality into a CLI tooling.

Server mode.

Think about adding server mode where queries are supplied through http calls. Would also require adding nice statistics everywhere.

Would be very useful for production usage.

Watching configuration changes in git would be amazing.

Configuration hot reload.

Implement in memory map (select)