Bad Cmp result

[ericlagergren]

Bad Cmp result

[ericlagergren]

ahhh I see the problem, thanks.

[rif]

I printed the internals if it helps:

RESULT: 9223372036854775807, scale: 5, ctx: {precision:0 mode:0}, form: 1, unscaled: {false [270100000]}
EXPECTED: compact: 2701, scale: 0, ctx: {precision:0 mode:0}, form: 1, unscaled: {false []}

Some other values with similar problem:

RESULT: compact: 9223372036854775807, scale: 5, ctx: {precision:0 mode:0}, form: 1, unscaled: {false [181050000]}
EXPECTED: compact: 18105, scale: 1, ctx: {precision:0 mode:0}, form: 1, unscaled: {false []}

[ericlagergren]

can you reproduce this on the dev branch?

[ericlagergren]

I ask because I think the fix is on the dev branch atm.

[rif]

I confirm that the dev branch fixes the first problem (the one with 2701) but the second one with 1810.5 is NOT fixed. Also I got two other tests that fail: before the result was 0.3 after a Round(9) and now it's 0.300000001

[ericlagergren]

could you post minimal repros for the bugs?

[rif]

unfortunately not, I will try to push a test on your dev branch.

[ericlagergren]

okay. I can't begin debugging this until I have a couple test cases to look at.

[rif]

Would this type of test help?

func TestBig_CmpUnscale(t *testing.T) {
	b1 := &Big{compact: 9223372036854775807, scale: 5, ctx: Context{precision: 0, mode: RoundingMode(0)}, form: 1, unscaled: *new(big.Int).SetInt64(181050000)}
	b2 := &Big{compact: 18105, scale: 1, ctx: Context{precision: 0, mode: RoundingMode(0)}, form: 1, unscaled: *new(big.Int).SetInt64(0)}
	if b1.Cmp(b2) != 0 {
		t.Errorf("failed comparing %v with %v: %v", b1, b2, b1.Cmp(b2))
	}
}

[rif]

So I added this test on the dev branch and on my machine there are three tests failing:

--- FAIL: TestBig_Cmp (0.00s)
	decimal_test.go:170: #10: wanted 0, got -1
--- FAIL: TestBig_IsBig (0.00s)
	decimal_test.go:191: #4: wanted true, got false
--- FAIL: TestBig_CmpUnscale (0.00s)
	decimal_test.go:569: failed comparing 1810.5 with 1810.5: 1

[rif]

I confirm that the above issues are fixed but now I encountered:

t.Log(x.String(), y.String(), x.Cmp(y))
0.1         0.016666666666666666          -1

[rif]

x = compact: 1000000000000000, scale: 16, ctx: {16 0}, form: form(1), unscaled: {false []}
y = compact: 16666666666666666, scale: 18, ctx: {0 0}, form: form(1), unscaled: {false []}
x.String() = 0.1
y.String() =  0.016666666666666666
x.Cmp(y) = -1

[ericlagergren]

I actually think a bad test case is why this is goofed.

[ericlagergren]

yeah, this 63aaff2#diff-e035ddc6d9eb583a74a989c446aad535R168 was causing my tests to be wrong so the resulting code was wrong.

[ericlagergren]

@rif if this is still an issue feel free to reopen.

[rif]

All good on my side now. Thanks again!

[ericlagergren]

Thank you!

…

On Feb 22, 2017, at 12:06 PM, Radu Ioan Fericean ***@***.***> wrote: All good on my side now. Thanks again! — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.