Can u fix sum function precision overflow bug when double addition integer

[shukaizhe]

Can u fix sum function precision overflow bug when double addition integer

[shukaizhe]

This bug like this

There is my customization to fix

public class SumPrecision implements Function {

    @Override
    public String getName() {
        return "psum";
    }

    @Override
    public int getMinArguments() {
        return 1;
    }

    @Override
    public int getMaxArguments() {
        return 1;
    }

    @Override
    public JsonNode call(JsonNode input, JsonNode[] arguments) {
        JsonNode array = arguments[0];
        if (array.isNull()) {
            return NullNode.instance;
        } else if (!array.isArray()) {
            throw new JsltException("psum(): argument must be array, was " + array);
        }

        BigDecimal sum = new BigDecimal("0.0");
        boolean integral = true;
        for (int ix = 0; ix < array.size(); ix++) {
            JsonNode value = array.get(ix);
            if (!value.isNumber()) {
                throw new JsltException("psum(): array must contain numbers, found " + value);
            }
            integral &= value.isIntegralNumber();
            sum = sum.add(value.decimalValue());
        }
        if (integral) {
            return new LongNode(sum.longValue());
        } else {
            return new DecimalNode(sum);
        }
    }
}

[larsga]

Not sure I understand what the bug is. Can you give an example with JSON input + JSLT code and wrong output?

[shukaizhe]

This is JSON input

[
    {
      "taskSeqno": "100000265920",
      "taskSeqnoFk": "100000396209",
      "ramiCode": "11",
      "ramiCodeDesc": "22",
      "tranStatus": "05",
      "tranStatusDesc": "已放款",
      "tranDate": "20230101",
      "actualAmount": "23000.123",
      "actualAmount2": 23000.123,
      "lastOperation": "已放款",
      "lastOperationCode": "05",
      "refundFlag": "N",
      "totalReturnAmount": "0",
      "refundDate": "20221102",
      "tranDesc": "点错了",
      "returnTranDesc": "无效果",
      "returnAmount": 4000,
      "retractStatus": "00",
      "retper": "01",
      "totTranAmount": 1000.55,
      "tranAmount": 1050,
      "downPayAmount": 1122.33,
      "paymentDProp": 18,
      "loanTerm": 12,
      "ramiLafratvl": 3.11,
      "lafratvl": 1.22,
      "discountRate": 5.44,
      "factoryRate": 6.55,
      "agencyRate": 7.88,
      "agencyGroupRate": 8.77,
      "ramiCmsn": 4455.66,
      "cstCmsn": 3355.55,
      "mchtCmsn": 4433.22,
      "finalPayment": 7777.55,
      "paymentFProp": 45.88,
      "isFlexRepymt": "1",
      "sxfstage": "1",
      "discountFlg": "0",
      "discountId": "001"
    },
    {
      "taskSeqno": "100000265920",
      "taskSeqnoFk": "100000396209",
      "ramiCode": "11",
      "ramiCodeDesc": "22",
      "tranStatus": "05",
      "tranStatusDesc": "已放款",
      "tranDate": "20230104",
      "actualAmount": "23000",
      "actualAmount2": 23000.123,
      "lastOperation": "已放款",
      "lastOperationCode": "05",
      "refundFlag": "N",
      "totalReturnAmount": "0",
      "refundDate": "20221102",
      "tranDesc": "点错了",
      "returnTranDesc": "无效果",
      "returnAmount": 4000,
      "retractStatus": "00",
      "retper": "01",
      "totTranAmount": 1000.55,
      "tranAmount": 1050,
      "downPayAmount": 1122.33,
      "paymentDProp": 18,
      "loanTerm": 12,
      "ramiLafratvl": 3.11,
      "lafratvl": 1.22,
      "discountRate": 5.44,
      "factoryRate": 6.55,
      "agencyRate": 7.88,
      "agencyGroupRate": 8.77,
      "ramiCmsn": 4455.66,
      "cstCmsn": 3355.55,
      "mchtCmsn": 4433.22,
      "finalPayment": 7777.55,
      "paymentFProp": 45.88,
      "isFlexRepymt": "1",
      "sxfstage": "1",
      "discountFlg": "0",
      "discountId": "001"
    },
    {
      "taskSeqno": "100000265920",
      "taskSeqnoFk": "100000396209",
      "ramiCode": "11",
      "ramiCodeDesc": "22",
      "tranStatus": "05",
      "tranStatusDesc": "已放款",
      "tranDate": "20230108",
      "actualAmount": "23000",
      "actualAmount2": 23000.123,
      "lastOperation": "已放款",
      "lastOperationCode": "05",
      "refundFlag": "N",
      "totalReturnAmount": "0",
      "refundDate": "20221102",
      "tranDesc": "点错了",
      "returnTranDesc": "无效果",
      "returnAmount": 4000,
      "retractStatus": "00",
      "retper": "01",
      "totTranAmount": 1000.55,
      "tranAmount": 1050,
      "downPayAmount": 1122.33,
      "paymentDProp": 18,
      "loanTerm": 12,
      "ramiLafratvl": 3.11,
      "lafratvl": 1.22,
      "discountRate": 5.44,
      "factoryRate": 6.55,
      "agencyRate": 7.88,
      "agencyGroupRate": 8.77,
      "ramiCmsn": 4455.66,
      "cstCmsn": 3355.55,
      "mchtCmsn": 4433.22,
      "finalPayment": 7777.55,
      "paymentFProp": 45.88,
      "isFlexRepymt": "1",
      "sxfstage": "1",
      "discountFlg": "0",
      "discountId": "001"
    }
  ]

This is JSON result

{
  "tranDate" : "20230108",
  "customerRate" : 1.22,
  "discountAmount" : 6877.5,
  "firstPaymentRatio" : 18,
  "finalPaymentRatio" : 45.88,
  "finalPayment" : 7777.55,
  "actualAmount" : 69000.12299999999,
  "actualAmount2" : 69000.369,
  "loanTerm" : 12,
  "totTranAmount" : 1000.55,
  "totalRate" : 1000.55,
  "totalHandlingFee" : 4455.66
}

Such as JSLT script

{
    "tranDate" : .[size(.) - 1].tranDate,
    "customerRate" : .[0].lafratvl,
    "discountAmount" : (.[0].tranAmount * .[0].factoryRate),
    "firstPaymentRatio" : .[0].paymentDProp,
    "finalPaymentRatio" : .[0].paymentFProp,
    "finalPayment" : .[0].finalPayment,
    "actualAmount": sum([for(.)
     number(.actualAmount)]),
    "actualAmount2": sum([for(.)
     number(.actualAmount2)]),
    "loanTerm" : .[0].loanTerm,
    "totTranAmount" : .[0].totTranAmount,
    "totalRate" : .[0].totTranAmount,
    "totalHandlingFee" : .[0].ramiCmsn
}

Look at

"actualAmount"

[larsga]

Okay, so your complaint is that

sum([ 23000.123, 23000, 23000 ])

produces 69000.12299999999 instead of 69000.123. Which is how decimal numbers work.

If I try this with Python I get:

Python 2.7.18 (v2.7.18:8d21aa21f2, Apr 19 2020, 20:48:48)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> sum([ 23000.123, 23000, 23000 ])
69000.12299999999

If I do it in JavaScript I get:

Welcome to Node.js v19.3.0.
Type ".help" for more information.
> 23000.123 + 23000 + 23000
69000.12299999999

Your proposed solution is to use BigDecimal instead. That will be slower and more memory-intensive, but perhaps it's worth it. People have asked for the same thing before (issue #68).

What I wonder is: why do people consider this a problem? Is it because you're just not comfortable with floating numbers behaving they way they do, or is it because there is a real, practical problem involved?

[shukaizhe]

Think for u solution,This kind of scenario happens in which us want to process the amount field using JSLT script。
Script calculations will be used in API transfer，lowcode System entity transformations(only based on Spring or EJB).

[larsga]

I wanted to understand what this would look like if the JSON were loaded into a Java process, so I wrote this small program:

public class Test {
  public static void main(String[] argv) {
    System.out.println("float:  " + Float.parseFloat("69000.123"));
    System.out.println("double: " + Double.parseDouble("69000.123"));
  }
}

The output is:

float:  69000.125
double: 69000.123

If I add a few more zeroes the float loses the digits after the decimal completely, but the double keeps working.

[larsga]

The problem is not limited to the sum() function, because 23000.123 + 23000 + 23000 produces 69000.12299999999.

[catull]

You could change the JSTL to do this:

{
  "actualAmount": round (1000 * sum ([ for (.)  number (.actualAmount) ] )) / 1000
}

It would still have rounding error.

A slow alternative would be

def calculateActualAmount (amounts)
    let int = floor ($amounts / 1000)
    let fraction = $amounts - (1000 * $int)
    let result = join ([ string (floor ($fraction / 100)), string (floor (mod ($fraction,  100) / 10)), string (mod ($fraction, 10)) ], "")
    join ([ $int, $result], ".")

{
  "actualAmount": calculateActualAmount (sum ([ for (.) floor (1000 * number (.actualAmount)) ] ))
}

Ugly, I admit.