dolthub/dolt

Dolt – Git for Data

dolt's Issues

What are the limitations of doit in terms of dataset size?

Can it handle gigabyte size data?

--ours nor --theirs useful in conflict resolution

timsehn:test timsehn$ dolt conflicts cat employees
 +  | ours   | 3 | matt | jesuele | software engineer | 12/3/2018 |        
 +  | theirs | 3 | matt | jesuele | software engineer |           | current

I really want to specify which fields to keep from our and theirs. Maybe a third option like --choose. --pk:[value] field1:ours field2:theirs....

Git-Dolt: Multiple invocations of `git dolt install` should not re-append to files

See https://github.com/liquidata-inc/ld/pull/1715#discussion_r301835971

dolt: dolt push when behind your remote returns 0 exit code. should be 1

timsehn:test-repo timsehn$ dolt push origin master
To localhost:50051/test-org/test-repo
! [rejected]          master -> remotes/origin/master (non-fast-forward)
error: failed to push some refs to 'localhost:50051/test-org/test-repo'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'dolt pull ...') before pushing again.
timsehn:test-repo timsehn$ echo $?
0

There is a failed bats test for this.

Dolt: dolt table select --hide-conflicts does not work

timsehn:bats-test timsehn$ dolt init
Successfully initialized dolt data repository.
timsehn:bats-test timsehn$ dolt table create -s 1pk5col.schema test
timsehn:bats-test timsehn$ dolt add test
timsehn:bats-test timsehn$ dolt commit -m "added test table"
timsehn:bats-test timsehn$ dolt branch test-branch
timsehn:bats-test timsehn$ dolt table put-row test pk:0 c1:1 c2:2 c3:3 c4:4 c5:5
Successfully put row.
timsehn:bats-test timsehn$ dolt add test
timsehn:bats-test timsehn$ dolt commit -m "added test row"
timsehn:bats-test timsehn$ dolt checkout test-branch
Switched to branch 'test-branch'
timsehn:bats-test timsehn$ dolt table put-row test pk:0 c1:1 c2:2 c3:3 c4:4 c5:6
Successfully put row.
timsehn:bats-test timsehn$ dolt add test
timsehn:bats-test timsehn$ dolt commit -m "added conflicting test row"
timsehn:bats-test timsehn$ dolt checkout master
Switched to branch 'master'
timsehn:bats-test timsehn$ dolt merge test-branch
Updating 5u5pr8vuge4gmskldopa5e7kr0gftv2r..nlj137rb7et8bbcintgrqjjefhladou5
Auto-merging test
CONFLICT (content): Merge conflict in test
Automatic merge failed; fix conflicts and then commit the result.
timsehn:bats-test timsehn$ dolt table select test
Cnf|pk|c1|c2|c3|c4|c5
 ! |0 |1 |2 |3 |4 |5 
timsehn:bats-test timsehn$ dolt table select --hide-conflicts test
Cnf|pk|c1|c2|c3|c4|c5
 ! |0 |1 |2 |3 |4 |5 

There's a bats test for this issue.

Dolt: Bad error on excel import of non excel file

timsehn:bats-test timsehn$ dolt table import -c employees ~/liquidata/git/ld/dolt/bats/helper/bad.xlsx
Error creating reader for /Users/timsehn/liquidata/git/ld/dolt/bats/helper/bad.xlsx.
When attempting to move data from xlsx file:/Users/timsehn/liquidata/git/ld/dolt/bats/helper/bad.xlsx to dolt table:employees, could not open a reader.
cause: zip: not a valid zip file

Should say "not a valid xlsx file".

Dolt: Weird error message on Dolt diff when changing schema

timsehn:test1 timsehn$ dolt table import -c -f -s employees.schema employees employees.csv 
timsehn:test1 timsehn$ dolt diff
diff --dolt a/employees b/employees
--- a/employees @ rr223frevrus2cefsmrld6582924bq9o
+++ b/employees @ jsrqk409vahr5h8q69nevlle06114qs8
Failed to merge schemas
timsehn:test1 timsehn$ dolt table cat employees
id|first name|last name|title            |start date|end date
0 |tim       |sehn     |ceo              |8/6/2018  |current 
1 |aaron     |son      |founder          |8/6/2018  |current 
2 |brian     |hendriks |founder          |8/6/2018  |current 
3 |matt      |jesuele  |software engineer|12/3/2018 |current 
timsehn:test1 timsehn$ dolt table schema employees 
employees @ working
{
  "columns": [
    {
      "tag": 0,
      "name": "id",
      "kind": "int",
      "is_part_of_pk": true,
      "col_constraints": [
        {
          "constraint_type": "not_null",
          "params": null
        }
      ]
    },
    {
      "tag": 1,
      "name": "first name",
      "kind": "string",
      "is_part_of_pk": false,
      "col_constraints": []
    },
    {
      "tag": 2,
      "name": "last name",
      "kind": "string",
      "is_part_of_pk": false,
      "col_constraints": []
    },
    {
      "tag": 3,
      "name": "title",
      "kind": "string",
      "is_part_of_pk": false,
      "col_constraints": []
    },
    {
      "tag": 4,
      "name": "start date",
      "kind": "string",
      "is_part_of_pk": false,
      "col_constraints": []
    },
    {
      "tag": 5,
      "name": "end date",
      "kind": "string",
      "is_part_of_pk": false,
      "col_constraints": []
    }
  ]
}

Basically export the schema and data for a table. Make a change to the type in the schema file (in this case string to int). Import the table schema and data using -c -f. Get a weird "Failed to merge schemas" on dolt diff. But when I output the schema and the table, things look fine.

'dolt login' always creates new credentials, even if they are unncesssary.

It's probably rare that the user actually wants to create new credentials and associate them with a different dolthub account. If credentials already exist and they already work with the registered remote, should we prompt for behavior, or have the user pass --new-credentails or something if they really want to re-login?

Weird dolt import bug

Table

timsehn:dolt-video timsehn$ cat /tmp/employees.csv 
id,first name,last name,title
0,tim,sehn,ceo
1,aaron,son,founder
2,brian,hendriks,founder
3,matt,jesuele,software engineer

Schema

timsehn:dolt-video timsehn$ cat /tmp/employees.schema 
{
  "fields": [
    {
      "name": "id",
      "kind": "number",
      "required": true
    },
    {
      "name": "first name",
      "kind": "string",
      "required": false
    },
    {
      "name": "last name",
      "kind": "string",
      "required": false
    },
    {
      "name": "title",
      "kind": "string",
      "required": false
    }
  ],
  "constraints": [
    {
      "constraint_type": "primary_key",
      "field_indices": [
        0
      ]
    }
  ]
}

Output of import:

timsehn:dolt-video timsehn$ dolt table import -c -f employees /tmp/employees.csv -s /tp/employees.schema 
A bad row was encountered while moving data.
Bad Row:id:"3" | first name:"matt" | last name:"jesuele" | title:"software engineer"
strconv.ParseBool: parsing "3": invalid syntax
These can be ignored using the '--continue'

Tried with the id set to integer as well. Not sure what types are supported.

Dolt: Improve error message for bad diff arguments

Git:

$ git diff badrevision
fatal: ambiguous argument 'badrevision': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

Dolt:

$ dolt diff badrevision
error: Unknown table: 'badrevision'

Dolt: allows importing of non-UTF-8 characters

Created a skipped bats test for this.

Dolt: Name a table and branch the same, dolt checkout ambiguous

Successfully initialized dolt data repository.
timsehn:bats-test timsehn$ dolt table create -s=1pk5col.schema test
timsehn:bats-test timsehn$ dolt add test
timsehn:bats-test timsehn$ dolt commit -m "create table"
timsehn:bats-test timsehn$ dolt table put-row test pk:0 c1:1 c2:2 c3:3 c4:4 c5:5
Successfully put row.
timsehn:bats-test timsehn$ dolt diff
diff --dolt a/test b/test
--- a/test @ nm261ggrdkpaitnt7g98ac0ktse89sl5
+++ b/test @ uu8j16uneeb7u0vk9ki4faa6tl53t2e8
   pk | c1 | c2 | c3 | c4 | c5
 + 0  | 1  | 2  | 3  | 4  | 5 
timsehn:bats-test timsehn$ dolt diff
diff --dolt a/test b/test
--- a/test @ nm261ggrdkpaitnt7g98ac0ktse89sl5
+++ b/test @ uu8j16uneeb7u0vk9ki4faa6tl53t2e8
   pk | c1 | c2 | c3 | c4 | c5
 + 0  | 1  | 2  | 3  | 4  | 5 
timsehn:bats-test timsehn$ dolt branch test
timsehn:bats-test timsehn$ dolt branch
* master                          
  test
timsehn:bats-test timsehn$ dolt checkout test
timsehn:bats-test timsehn$ dolt branch
* master                          
  test    

Git-Dolt: Consider changing pointer file format to JSON

Currently git-dolt pointer files use the same format as git-lfs pointer files:

key1 val1
key2 val2

...but perhaps they should just be JSON instead:

{
  "key1": "val1",
  "key2": "val2"
}

Dolt: `dolt clone` dies with ugly error when cloning a remote just created on DoltHub

At a minimum, the error should be better.

$ dolt clone mjesuele/the-new-empty
cloning https://doltremoteapi.awsdev.ld-corp.com/mjesuele/the-new-empty
error: failed to write repo state
cause: json: error calling MarshalJSON for type ref.MarshalableRef: unexpected end of JSON input

Git-Dolt: git dolt update should use the dolt HEAD revision by default

If a git-dolt pointer file is next to the corresponding dolt repository, running git dolt update without specifying a revision should result in the pointer file pointing to whatever the current HEAD revision of the dolt repo is.

Dolt: Says failed to merge schemas when changing strings to ints/floats

timsehn:bitcoin-coinbase timsehn$ dolt diff
diff --dolt a/minute_summary b/minute_summary
--- a/minute_summary @ ms4rpm098j6afj454e5uginlq6bslie0
+++ b/minute_summary @ itpfnfv7bkip2e1qt6k612v3b4p4l85f
  CREATE TABLE minute_summary (
<   Timestamp varchar not null comment 'tag:0'
>   Timestamp     int not null comment 'tag:0'
<   Open varchar comment 'tag:1'
>   Open   float comment 'tag:1'
<   High varchar comment 'tag:2'
>   High   float comment 'tag:2'
<   Low varchar comment 'tag:3'
>   Low   float comment 'tag:3'
<   Close varchar comment 'tag:4'
>   Close   float comment 'tag:4'
<   Volume_(BTC) varchar comment 'tag:5'
>   Volume_(BTC)   float comment 'tag:5'
<   Volume_(Currency) varchar comment 'tag:6'
>   Volume_(Currency)   float comment 'tag:6'
<   Weighted_Price varchar comment 'tag:7'
>   Weighted_Price   float comment 'tag:7'
  );

Failed to merge schemas

Import a table from a csv. Get all string columns. Export the schema to a file. Modify the file to have the types you want. Reimport but this time specify -s with the modified schema file.

I would expect this to show no data diff, not failed to merge schemas.

dolt: got a panic when doing a merge from a remote with conflicts

timsehn:test-repo timsehn$ dolt pull origin
Updating bl0uhig0sr1ma64sltebj20blgv7gg76..16b1487q14e8d1fp68m081336o70m0s3
clone failed
cause: runtime error: invalid memory address or nil pointer dereference
timsehn:test-repo timsehn$ dolt ls
Tables in working set:
	 test2
timsehn:test-repo timsehn$ echo $?
0
timsehn:test-repo timsehn$ dolt fetch origin master
timsehn:test-repo timsehn$ dolt merge remotes/origin/master
Updating bl0uhig0sr1ma64sltebj20blgv7gg76..16b1487q14e8d1fp68m081336o70m0s3
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x182aa40]

goroutine 1 [running]:
github.com/liquidata-inc/ld/dolt/go/libraries/doltcore/doltdb.(*Table).HashOf(0x0, 0x0, 0x0, 0x0)
	/Users/timsehn/liquidata/git/ld/dolt/go/libraries/doltcore/doltdb/table.go:184 +0x40
github.com/liquidata-inc/ld/dolt/go/libraries/doltcore/merge.(*Merger).MergeTable(0xc000370b10, 0xc000334fc8, 0x5, 0xc00032efe0, 0x2, 0x2, 0x0)
	/Users/timsehn/liquidata/git/ld/dolt/go/libraries/doltcore/merge/merge.go:53 +0x1c5
github.com/liquidata-inc/ld/dolt/go/libraries/doltcore/env/actions.MergeCommits(0xc00042c000, 0xc0004085f0, 0xc000408730, 0xaf, 0x0, 0x0, 0x0)
	/Users/timsehn/liquidata/git/ld/dolt/go/libraries/doltcore/env/actions/merge.go:22 +0x1a9
github.com/liquidata-inc/ld/dolt/go/cmd/dolt/commands.executeMerge(0xc0004160c0, 0xc0004085f0, 0xc000408730, 0x7ffeefbffb73, 0x15, 0xc0003f83a0, 0x20)
	/Users/timsehn/liquidata/git/ld/dolt/go/cmd/dolt/commands/merge.go:181 +0x5e
github.com/liquidata-inc/ld/dolt/go/cmd/dolt/commands.mergeBranch(0xc0004160c0, 0x7ffeefbffb73, 0x15, 0x0, 0x25ce780)
	/Users/timsehn/liquidata/git/ld/dolt/go/cmd/dolt/commands/merge.go:141 +0x2fd
github.com/liquidata-inc/ld/dolt/go/cmd/dolt/commands.Merge(0xc000422046, 0xa, 0xc0000b60b0, 0x1, 0x1, 0xc0004160c0, 0x5)
	/Users/timsehn/liquidata/git/ld/dolt/go/cmd/dolt/commands/merge.go:99 +0x3ae
github.com/liquidata-inc/ld/dolt/go/cmd/dolt/cli.GenSubCommandHandler.func1(0x1c25659, 0x4, 0xc0000b60a0, 0x2, 0x2, 0xc0004160c0, 0xc000370401)
	/Users/timsehn/liquidata/git/ld/dolt/go/cmd/dolt/cli/command.go:61 +0x19b
main.main()
	/Users/timsehn/liquidata/git/ld/dolt/go/cmd/dolt/dolt.go:76 +0x256

There is a skipped bats test for this.

Doing CDC with Dolt

Exploring uses cases with Dolt.

Maybe the hub already does a change to the data is just like CDC, and git Servers tell you when someone makea commit. But with a CDC you want to be able to query and get an event when the data that is part of the Query Result Set changes.

PS. Polling is cheating !

dolt: dolt config --list does not respect local overrides

timsehn:bats-test timsehn$ dolt config --global --add test global
Config successfully updated.
timsehn:bats-test timsehn$ dolt config --local --add test local
Config successfully updated.
timsehn:bats-test timsehn$ dolt config --list
test = local
testlocal = testlocal
test = global
user.email = [email protected]
user.name = bats tester
timsehn:bats-test timsehn$ dolt config --local --get test
local
timsehn:bats-test timsehn

There is a skipped bats test for this.

Should Diffs handle schema changes better

I changed the id column from a string to an int and reimported the same data. This is the diff output:

timsehn:dolt-video timsehn$ dolt diff
diff --dolt a/employees b/employees
--- a/employees @ 3suru3esi5qtd6or5e61127agr1e55s6
+++ b/employees @ 32t16pjc0112e7q2vpql8j4lpvqve0rl
   id | first name | last name | title            
 - 0  | tim        | sehn      | ceo              
 - 1  | aaron      | son       | founder          
 - 2  | brian      | hendriks  | founder          
 - 3  | matt       | jesuele   | software engineer
 + 0  | tim        | sehn      | ceo              
 + 1  | aaron      | son       | founder          
 + 2  | brian      | hendriks  | founder          
 + 3  | matt       | jesuele   | software engineer

I guess this is technically correct. I changed the meaning of the row so it thinks I removed and added the rows.

Dolt merge murdered my table?

Made schema change on a branch. Kept same data. Merged that branch to master. Committed. Table seems gone as in dolt table cat employees returns empty. dolt table schema shows the old one still there.

timsehn:dolt-video timsehn$ dolt checkout change-schema 
Switched to branch 'change-schema'
timsehn:dolt-video timsehn$ dolt table cat employees
id | first name | last name | title            
0  | tim        | sehn      | ceo              
1  | aaron      | son       | founder          
2  | brian      | hendriks  | founder          
3  | matt       | jesuele   | software engineer
timsehn:dolt-video timsehn$ dolt checkout master
Switched to branch 'master'
timsehn:dolt-video timsehn$ dolt log
commit uuhe2fgkg9pje89d74aetosg3jadjj0p
Merge: qccmdrkre0ld2mf63cpnmik5lcr32pjb c008futlhdr2du3un49a8mkvqrbi240q
Author: tim sehn <[email protected]>
Date:   Tue Feb 05 15:57:49 -0800 2019

	Merged new schema. Changed id to int and required

commit c008futlhdr2du3un49a8mkvqrbi240q
Merge: qccmdrkre0ld2mf63cpnmik5lcr32pjb ilhik5deso56fa1p67t7ugo5oimsnhje
Author: tim sehn <[email protected]>
Date:   Tue Feb 05 15:42:05 -0800 2019

	Changed schema. id from string to int and required.

commit ilhik5deso56fa1p67t7ugo5oimsnhje
Author: tim sehn <[email protected]>
Date:   Fri Feb 01 12:40:26 -0800 2019

	add start date

commit qccmdrkre0ld2mf63cpnmik5lcr32pjb
Author: tim sehn <[email protected]>
Date:   Thu Jan 31 15:51:01 -0800 2019

	Added Matt to employees

commit kldk7psjq4fttdujg65mthevpmduhd59
Author: tim sehn <[email protected]>
Date:   Thu Jan 31 15:42:24 -0800 2019

	Initial employees table

commit f7lhvs2of183hrbo1b2n81htjpifchii
Author: tim sehn <[email protected]>
Date:   Thu Jan 31 15:33:12 -0800 2019

	Data repository created.

timsehn:dolt-video timsehn$ dolt table cat employees
timsehn:dolt-video timsehn$ dolt table schema
employees @ working
idx | name       | type   | nullable | primary key
0   | id         | string | true     | true       
1   | first name | string | true     | false      
2   | last name  | string | true     | false      
3   | title      | string | true     | false  

Dolt: `dolt login` will create new credentials even when I'm already logged in

If credentials already exist, should it poll WhoAmI first and see that they currently work?

Dolt: Bad error message for invalid schema on json import

timsehn:bats-test timsehn$ dolt table import -c -s ~/liquidata/git/ld/dolt/bats/helper/employees-sch-bad.json employees ~/liquidata/git/ld/dolt/bats/helper/employees-tbl.json 
Error creating reader for /Users/timsehn/liquidata/git/ld/dolt/bats/helper/employees-tbl.json.
When attempting to move data from json file:/Users/timsehn/liquidata/git/ld/dolt/bats/helper/employees-tbl.json to dolt table:employees, could not open a reader.
cause: invalid character 'b' after object key:value pair

Mentions the valid file not the invalid file.

Dolt: Schema diff no worky for changing primary keys

timsehn:videogame-sales timsehn$ dolt diff
diff --dolt a/videogame_sales b/videogame_sales
--- a/videogame_sales @ q1s2plh3ans4d9drcei5gi4futgmovhu
+++ b/videogame_sales @ gl5087ols1lji8csbom2hfj3fall5ich
  CREATE TABLE videogame_sales (
<   Rank int comment 'tag:0'
>   Rank int comment 'tag:0'
<   Name varchar comment 'tag:1'
>   Name varchar comment 'tag:1'
<   Platform varchar comment 'tag:2'
>   Platform varchar comment 'tag:2'
    Year int comment 'tag:3'
    Genre varchar comment 'tag:4'
    Publisher varchar comment 'tag:5'
    NA_Sales float comment 'tag:6'
    EU_Sales float comment 'tag:7'
    JP_Sales float comment 'tag:8'
    Other_Sales float comment 'tag:9'
    Global_Sales float comment 'tag:10'
  );

Failed to merge schemas
timsehn:videogame-sales timsehn$ dolt schema
videogame_sales @ working
CREATE TABLE videogame_sales (
  Rank int comment 'tag:0',
  Name varchar comment 'tag:1',
  Platform varchar comment 'tag:2',
  Year int comment 'tag:3',
  Genre varchar comment 'tag:4',
  Publisher varchar comment 'tag:5',
  NA_Sales float comment 'tag:6',
  EU_Sales float comment 'tag:7',
  JP_Sales float comment 'tag:8',
  Other_Sales float comment 'tag:9',
  Global_Sales float comment 'tag:10',
  primary key (Name,Platform)
);

timsehn:videogame-sales timsehn$ dolt schema HEAD^
videogame_sales @ HEAD^
CREATE TABLE videogame_sales (
  Rank varchar comment 'tag:0',
  Name varchar comment 'tag:1',
  Platform varchar comment 'tag:2',
  Year varchar comment 'tag:3',
  Genre varchar comment 'tag:4',
  Publisher varchar comment 'tag:5',
  NA_Sales varchar comment 'tag:6',
  EU_Sales varchar comment 'tag:7',
  JP_Sales varchar comment 'tag:8',
  Other_Sales varchar comment 'tag:9',
  Global_Sales varchar comment 'tag:10',
  primary key (Rank)
);

I'll create a failing bats test for this.

Dolt: Throughput Test Fails Intermittently

https://github.com/liquidata-inc/ld/blob/master/dolt/go/libraries/utils/iohelp/read_test.go#L225

This test will unpredictably fail on Windows. On repeated runs, sometimes they all pass, sometimes they don't. In the meantime, I have a runtime check that skips this on Windows.

Dolt: Allow partial commit hashes in commands

Anywhere Git expects commit hashes, it allows you to provide just the beginning of the hash and it will attempt to match it, throwing an error if the partial hash is not enough to uniquely identify a commit.

It would be super helpful if Dolt did the same.

Dolt: Diff with two revisions is backwards

In git, if you do:

git diff older_commit newer_commit

it shows the diff with the older commit as the base. Dolt currently does the opposite. This is probably not what we want:

$ dolt diff lh7tipknpuolrahdasvflvral4n72s32 r52ivkv7jkje3pbaedcg0f6idnb93ek6
diff --dolt a/submissions b/submissions
--- a/submissions @ 2vtm8bdmg9k7k2sff8907etf596p639j
+++ b/submissions @ sjb1tlf6ot137jfn84a6eaujn7ufmh5e
+-----+--------------------------------------+--------------------------+-----------------+---------+-------------+----------------------+
|     | id                                   | email                    | name            | company | hear_source | created_at           |
+-----+--------------------------------------+--------------------------+-----------------+---------+-------------+----------------------+
|  -  | 1f8ef759-367d-4b2c-bd91-85788fa2732c | [email protected] | Alessio Fanelli | <NULL>  | <NULL>      | 2019-05-27T00:30:37Z |
+-----+--------------------------------------+--------------------------+-----------------+---------+-------------+----------------------+

dolt: dolt config --global --add currently succeeds

timsehn:bats-test timsehn$ dolt config --global --add 
Config successfully updated.
timsehn:bats-test timsehn$ dolt config --global --list
timsehn:bats-test timsehn$

This should throw a wrong number or arguments error.

There is a skipped bats test for this.

SQL conversion mode for CSV import

Provide an option during import to automatically sanitize names, e.g. s/ /_/g to make SQL queries easier to write.

Dolt schemas should use "type" instead of "kind"

Current output from dolt table import --help:

Schema definition files are json files in the format:
{
"fields": [
{"name":"FIELD_NAME", "kind":"KIND", "Required":[true|false]},
...
],
"constraints": [
{"constraint_type":"primary_key", "field_indices":[INTEGER_FIELD_INDEX]}
]
}

     where "fields" is the array of columns in each row of the table
"constraints" is a list of table constraints.  (Only primary_key constraint types are supported currently)
FIELD_NAME is the name of a column in a row and can be any valid string
KIND must be a supported noms kind (bool, string, uuid, uint, int, float)

"kind" is a noms concept, not a database concept. We should use "type" or "datatype" everywhere.

Dolt: Dolt table export does not quote lists with commas

timsehn:lunch-places timsehn$ dolt table select lunch-places
name     |type of food|visitors                     |rating
Kazu Nori|sushi       |aaron,brian,katie,osheiza,tim|8.8   
timsehn:lunch-places timsehn$ dolt table export lunch-places brian.csv
Successfully exported data.
timsehn:lunch-places timsehn$ cat brian.csv 
name,type of food,visitors,rating
Kazu Nori,sushi,aaron,brian,katie,osheiza,tim,8.8

dolt: push and pull a remote repository should say "Already up to date"

timsehn:bats-test timsehn$ remotesrv --dir /var/tmp/ &>/dev/null &
[1] 17282
timsehn:bats-test timsehn$ dolt pull test-remote
Updating nf4l8aabdndbmstk938u978epq2fjbjv..nf4l8aabdndbmstk938u978epq2fjbjv
Fast-forward

Notice the same commit numbers. Should say "Already up to date." in this case.

Failing bats test created.

Dolt: enforce SQL table naming conventions in dolt table create

timsehn:bats-test timsehn$ dolt table create -s=1pk5col.schema 1pk
timsehn:bats-test timsehn$ dolt table create -s=2pk5col.schema 2pk
timsehn:bats-test timsehn$ dolt schema
1pk @ working
CREATE TABLE 1pk (
  pk int not null comment 'tag:0',
  c1 int comment 'tag:1',
  c2 int comment 'tag:2',
  c3 int comment 'tag:3',
  c4 int comment 'tag:4',
  c5 int comment 'tag:5',
  primary key (pk)
);

2pk @ working
CREATE TABLE 2pk (
  pk1 int not null comment 'tag:0',
  pk2 int not null comment 'tag:1',
  c1 int comment 'tag:2',
  c2 int comment 'tag:3',
  c3 int comment 'tag:4',
  c4 int comment 'tag:5',
  c5 int comment 'tag:6',
  primary key (pk1,pk2)
);

tables cannot start with digits.

I'm going to create table naming bats tests.

ensure nullability for primary key columns

In all code paths that can create schemas.

Dolt: Panic on .psv import

timsehn:dictionary-english timsehn$ dolt table import -c -f -pk word,text raw ../../dolt-test-inputs/english-dictionary/parsed-dictionary.psv 
panic: runtime error: slice bounds out of range

goroutine 37 [running]:
github.com/liquidata-inc/ld/dolt/go/libraries/doltcore/table/untyped/csv.appendToken(0xc009c34000, 0x4, 0x4, 0xc009c27a80, 0x19ae, 0x7fd, 0x7fe, 0x0, 0xc009c34000, 0x4, ...)
	/Users/timsehn/liquidata/git/ld/dolt/go/libraries/doltcore/table/untyped/csv/line.go:44 +0x59a
github.com/liquidata-inc/ld/dolt/go/libraries/doltcore/table/untyped/csv.csvSplitLine(0xc009c27a80, 0x19ae, 0x10000007c, 0x0, 0x7493be8, 0x1)
	/Users/timsehn/liquidata/git/ld/dolt/go/libraries/doltcore/table/untyped/csv/line.go:11 +0xf1
github.com/liquidata-inc/ld/dolt/go/libraries/doltcore/table/untyped/csv.(*CSVReader).parseRow(0xc0003b8300, 0xc009c27a80, 0x19ae, 0x19ae, 0x0, 0x0, 0x0)
	/Users/timsehn/liquidata/git/ld/dolt/go/libraries/doltcore/table/untyped/csv/reader.go:128 +0x6a
github.com/liquidata-inc/ld/dolt/go/libraries/doltcore/table/untyped/csv.(*CSVReader).ReadRow(0xc0003b8300, 0x1dee040, 0xc0000bc010, 0xc00044e060, 0x0, 0x1894400, 0x0)
	/Users/timsehn/liquidata/git/ld/dolt/go/libraries/doltcore/table/untyped/csv/reader.go:101 +0x1e0
github.com/liquidata-inc/ld/dolt/go/libraries/doltcore/table/pipeline.ProcFuncForReader.func1(0x0, 0xc00044e060, 0xc00044e000, 0xc0003c4600, 0xc0003b2500)
	/Users/timsehn/liquidata/git/ld/dolt/go/libraries/doltcore/table/pipeline/procfunc_help.go:62 +0x41
github.com/liquidata-inc/ld/dolt/go/libraries/doltcore/table/pipeline.ProcFuncForSourceFunc.func1(0xc00044a080, 0xc0003c4660, 0xc0003c4600, 0xc00044e060)
	/Users/timsehn/liquidata/git/ld/dolt/go/libraries/doltcore/table/pipeline/procfunc_help.go:30 +0xa0
github.com/liquidata-inc/ld/dolt/go/libraries/doltcore/table/pipeline.(*Pipeline).Start.func3(0xc00044a080, 0xc0003c4660)
	/Users/timsehn/liquidata/git/ld/dolt/go/libraries/doltcore/table/pipeline/pipeline.go:214 +0x78
created by github.com/liquidata-inc/ld/dolt/go/libraries/doltcore/table/pipeline.(*Pipeline).Start
	/Users/timsehn/liquidata/git/ld/dolt/go/libraries/doltcore/table/pipeline/pipeline.go:212 +0x3b8

I uploaded the offending .psv to:

https://drive.google.com/open?id=1lJXuDSOQ0XRBQTXRbQHC5TwjFlXzPHbK

I'll try and get a reducible bats test.

Need to reserve tag numbers on drop column

Once used in any row, tag numbers cannot be reused in later revisions of a database. We don't enforce this currently.

Dolt: Can create a table with a blob type. Not supposed to be able to.

timsehn:bats-test timsehn$ cat 1pkalltypes.schema 
{
  "columns": [
    {
      "tag": 0,
      "name": "pk",
      "kind": "int",
      "is_part_of_pk": true,
      "col_constraints": [
        {
          "constraint_type": "not_null",
          "params": null
        }
      ]
    },
    {
      "tag": 1,
      "name": "int",
      "kind": "int",
      "is_part_of_pk": false,
      "col_constraints": []
    },
    {
      "tag": 2,
      "name": "string",
      "kind": "string",
      "is_part_of_pk": false,
      "col_constraints": []
    },
    {
      "tag": 3,
      "name": "boolean",
      "kind": "boolean",
      "is_part_of_pk": false,
      "col_constraints": []
    },
    {
      "tag": 4,
      "name": "float",
      "kind": "float",
      "is_part_of_pk": false,
      "col_constraints": []
    },
    {
      "tag": 5,
      "name": "uint",
      "kind": "uint",
      "is_part_of_pk": false,
      "col_constraints": []
    },
    {	
      "tag": 6,
      "name": "uuid",
      "kind": "uuid",
      "is_part_of_pk": false,
      "col_constraints": []
    },
    {
      "tag": 7,
      "name": "blob",
      "kind": "blob",
      "is_part_of_pk": false,
      "col_constraints": []
    }
  ]
}timsehn:bats-test timsehn$ dolt init
Successfully initialized dolt data repository.
timsehn:bats-test timsehn$ dolt table create -s=1pkalltypes.schema test
timsehn:bats-test timsehn$ dolt table schema
test @ working
{
  "columns": [
    {
      "tag": 0,
      "name": "pk",
      "kind": "int",
      "is_part_of_pk": true,
      "col_constraints": [
        {
          "constraint_type": "not_null",
          "params": null
        }
      ]
    },
    {
      "tag": 1,
      "name": "int",
      "kind": "int",
      "is_part_of_pk": false,
      "col_constraints": []
    },
    {
      "tag": 2,
      "name": "string",
      "kind": "string",
      "is_part_of_pk": false,
      "col_constraints": []
    },
    {
      "tag": 3,
      "name": "boolean",
      "kind": "bool",
      "is_part_of_pk": false,
      "col_constraints": []
    },
    {
      "tag": 4,
      "name": "float",
      "kind": "float",
      "is_part_of_pk": false,
      "col_constraints": []
    },
    {
      "tag": 5,
      "name": "uint",
      "kind": "uint",
      "is_part_of_pk": false,
      "col_constraints": []
    },
    {
      "tag": 6,
      "name": "uuid",
      "kind": "uuid",
      "is_part_of_pk": false,
      "col_constraints": []
    },
    {
      "tag": 7,
      "name": "blob",
      "kind": "blob",
      "is_part_of_pk": false,
      "col_constraints": []
    }
  ]
}

timsehn:bats-test timsehn$ 

Store default column values in schema

We only handle default column values when a column is added. To be consistent with other SQL databases, we need to store any default values are part of the schema and set those values on an insert as appropriate.

New file generated each time dolt table import -u is run

I attempted to update a table with a .csv file that contained the same data as the existing table, but with some of the rows missing. No updates were performed since no rows in this file were modified. However, each time this dolt table import -u is run, a new file of size 260bytes is created.

I'm not sure if this is intended behavior for a file to be created each time this command is run since no updates are made the first time this command was run, or on subsequent tries.

dolt release tag or version off

Public binary release tag is 0.9.7 but code version show 0.9.6
dolt version / dolt version 0.9.6

Dolt: Feature Request: dolt table import -c --pk=auto

This should just make assign an ascending integer primary key for each record that's imported on a newly created table. We're going to get a lot of tables that don't have a primary key and this could be a convenience so you don't have to do parsing before importing into dolt.

Dolt: Funky error message for duplicate row insert

timsehn:bats-test timsehn$ dolt sql -q "insert into test (pk,c1,c2,c3,c4,c5) values (0,6,6,6,6,6)"
Error inserting rows: [cannot insert existing row {map[0:0] map[5:6 1:6 2:6 3:6 4:6]}]

That map message is weird.

bad formatting in some help text

Hard-coded line breaks, rather than matching terminal size.

Dolt: For large imports, need some sort of progress output

Something like N/M lines imported every 100k lines? A wc -l on the csv file should not be that expensive. For other types like JSON, maybe so?

Dolt: Duplicate column name in put-row puts last. Should error?

timsehn:employees timsehn$ dolt table put-row employees id:4 "first name":faraj "last name":khasib title:"software engineer" "start date":12/3/2018 "last name":"12/5/2018" level:4 email:[email protected] nationality:palestinian
Successfully put row.
timsehn:employees timsehn$ dolt diff
diff --dolt a/employees b/employees
--- a/employees @ nkhkojanq4mlddpe7uh029bvtanfbfgj
+++ b/employees @ ocb61vb8n4g8fuofcl4rhiskh29kq3q4
   id | first name | last name | title             | start date | end date  | level | email              | nationality
 < 4  | faraj      | khasib    | software engineer | 12/3/2018  | 12/5/2018 | 4     | [email protected] | american   
 > 4  | faraj      | 12/5/2018 | software engineer | 12/3/2018  |           | 4     | [email protected] | palestinian

Dolt: Disk space gets bigger on dolt table import when you don't commit the change

See log: https://drive.google.com/open?id=1J5JQlxtQyGCBwdPT9z96FYT1fTx8gPmz

Disk gets cumulatively bigger even if you dolt table rm the table before you import. My expectation is that it should not continuously get bigger. This even happens when I run 'dolt checkout test' aafter each import which I would suspect would reset the repository back to a state where the import never happened

Dolt: `dolt status` panics on an empty repo

$ dolt status
panic: error: Unable to read HEAD commit root value

goroutine 1 [running]:
github.com/liquidata-inc/ld/dolt/go/cmd/dolt/commands.Status(0xc000037790, 0xb, 0xc00000e070, 0x0, 0x0, 0xc00009cf00, 0x6)
	/Users/mjesuele/dev/ld/dolt/go/cmd/dolt/commands/status.go:34 +0x276
github.com/liquidata-inc/ld/dolt/go/cmd/dolt/cli.GenSubCommandHandler.func1(0x1d766ae, 0x4, 0xc00000e070, 0x1, 0x1, 0xc00009cf00, 0xe)
	/Users/mjesuele/dev/ld/dolt/go/cmd/dolt/cli/command.go:61 +0x1ac
main.runMain(0x0)
	/Users/mjesuele/dev/ld/dolt/go/cmd/dolt/dolt.go:95 +0x2e0
main.main()
	/Users/mjesuele/dev/ld/dolt/go/cmd/dolt/dolt.go:56 +0x22

Not terrible, but should be more like the output of dolt log:

$ dolt log
Fatal error: cannot get HEAD commit for current branch.

Dolt: `SELECT TIMEDIFF(NOW(), UTC_TIMESTAMP())` Fails

When implementing dolt serve functionality, it was observed that Oracle's .NET MySQL connector would call SELECT TIMEDIFF(NOW(), UTC_TIMESTAMP()) immediately when connecting to dolt. This is not currently a supported SQL function, and thus an exception was thrown by the .NET client.

Specifically, the exception states that utc_timestamp was not found.

Dolt: Undetected Schema merge conflict

See tim-bug branch

I removed a column checked it in. Went to push to master. Master was ahead. Pulled down a table that still had the column. No merge conflict. No table is busted.

timsehn:lunch-places timsehn$ dolt fetch origin master
Counted chunks: 9, Buffered chunks: 5)
timsehn:lunch-places timsehn$ dolt merge remotes/origin/master
Updating gjio4j5hk58ce1mqpavg65cot43j3bmu..5tipunvdh9j6s9bmfkvfsv2bl47pjod1
lunch-places | 2 ++
1 tables changed, 2 rows added(+), 0 rows modified(*), 0 rows deleted(-)
timsehn:lunch-places timsehn$ dolt diff
diff --dolt a/lunch-places b/lunch-places
--- a/lunch-places @ 2f3bscpsg8e0snhgc8a6tj8anhjjnpd9
+++ b/lunch-places @ 1t3huhuhom9u2nf49puhs4l4qqd85bi4
   name       | type of food  | rating | best dish        | description of location                   | visitors             
 + Bibibop    | asian noodles | 7      | noodle bowl      | south east corner of 3rd and santa monica | aaron,daylon         
 + Sake House | sushi         | 85     | King Salmon Roll | Downstairs of the office                  | zach,aaron,brian,matt
timsehn:lunch-places timsehn$ dolt table select lunch-places
error: failed to transform row name:"Art's Table" | type of food:"fancy" | rating:8 | description of location:"Montana and 9th. Near Tim's house" | best dish:"short rib press".
name      |type of food |rating|best dish       |description of location                  |visitors             
Bibibop   |asian noodles|7     |noodle bowl     |south east corner of 3rd and santa monica|aaron,daylon         
Sake House|sushi        |85    |King Salmon Roll|Downstairs of the office                 |zach,aaron,brian,matt

documentation/errors regarding --schema <file>

1. Typo in description of schema file (dolt table import --schema)

{ "fields:" [ {"name":"FIELD_NAME", "kind":"KIND", "Required":[true|false]}, ... ], "constraints": [ {"constraint_type":"primary_key", "field_indices":[INTEGER_FIELD_INDEX]} ] }

"fields:" should be "fields":

2. Error when data doesn't match field type
   For example, I had the primary key field kind as "int", and when I tried to import a row that was a range "1901-2000", I got the following error:
   An error occurred moving data. Mapping transformfailed. Bad Row
   Can this error message be improved, and/or suggest using --continue to skip rows that aren't the correct datatype?

3. I suggest adding more user friendly error messaging for when brackets are added/omitted incorrectly to the schema file. The "Required" field does not require brackets but the "field_indices" field does, for example:

{ "fields": [ {"name":"Year", "kind":"int", "Required":true}, {"name":"Punxsutawney Phil", "kind":"string", "Required":false}, {"name":"February Average Temperature", "kind":"float", "Required":false}, {"name":"February Average Temperature (Northeast)", "kind":"float", "Required":false}, {"name":"February Average Temperature (Midwest)", "kind":"float", "Required":false}, {"name":"February Average Temperature (Pennsylvania)", "kind":"float", "Required":false}, {"name":"March Average Temperature", "kind":"float", "Required":false}, {"name":"March Average Temperature (Northeast)", "kind":"float", "Required":false}, {"name":"March Average Temperature (Midwest)", "kind":"float", "Required":false}, {"name":"March Average Temperature (Pennsylvania)", "kind":"float", "Required":false} ], "constraints": [ {"constraint_type":"primary_key", "field_indices":[0]} ] }

When I mistakenly added brackets to the Required field, I received this error message:
When attempting to move data from csv file:groundhog_data_master.csv to dolt table:groundhog3, could not determine the output schema. Schema File: "schema.json" explicit pk: "" cause: json: cannot unmarshal array into Go struct field jsonFieldData.required of type bool

And when omitting them for "field_indices", I received this error:
When attempting to move data from csv file:groundhog_data_master.csv to dolt table:groundhog3, could not determine the output schema. Schema File: "schema.json" explicit pk: "" cause: json: cannot unmarshal number into Go struct field jsonConstraint.field_indices of type []int