[Medium] Poor convergence with Tweedie

[quantco]

[Medium] Poor convergence with Tweedie

[ElizabethSantorellaQC]

See also #91 -- we don't know how well we are doing with Tweedie compared to other libraries.

[jtilly]

h2o can do it. That could be our reference point.

[ElizabethSantorellaQC]

@MarcAntoineSchmidtQC I just ran through all of the Tweedie benchmarks on 100k rows to see how representative the above examples are. Convergence was uniformly terrible for the "wide" problems, with inner_tol sometimes increasing by multiple orders of magnitude.

Performance was OK but not great on the "narrow" problems, with few inner cycles and the objective function decreasing by a factor of about 2 per iteration. I find it concerning that it isn't decreasing faster near the optimum, as we see on other distributions and with solvers that use second-order info in general.

We don't have these diagnostics for the l2 problems since they use the IRLS solver and we haven't produced diagnostics for that.

[ElizabethSantorellaQC]

I think it's fine, it just takes a few iterations to get going. Something is fishy here in that tweedie performance is uniquely bad, but it does converge.

(base) root@docker-desktop:/app# glm_benchmarks_run --library_names sklearn_fork --num_rows 1000 --problem_names wide_insurance_net_tweedie_p=1.5
running problem=wide_insurance_net_tweedie_p=1.5 library=sklearn_fork
/app/src/glm_benchmarks/sklearn_fork/_distribution.py:561: RuntimeWarning: divide by zero encountered in power
  + np.power(mu, 2 - p) / (2 - p)
/app/src/glm_benchmarks/sklearn_fork/_distribution.py:561: RuntimeWarning: invalid value encountered in multiply
  + np.power(mu, 2 - p) / (2 - p)
/app/src/glm_benchmarks/sklearn_fork/_glm.py:839: ConvergenceWarning: Coordinate descent failed to converge. Increase the maximum number of iterations max_iter (currently 40)
  ConvergenceWarning,
diagnostics:
            inner_tol  n_cycles   runtime  intercept
n_iter                                              
1       365389.888437         2  0.016654  -0.819840
2        98056.337796      1002  1.847528  -0.580055
3       106493.440884      1003  0.006540  -0.579109
4        95987.098967      1004  0.003800  -0.601489
5        57712.454837      1005  0.003972  -0.628642
6        34324.676895      1006  0.004078  -0.706518
7        20184.544783      1007  0.003942  -0.787785
8        11776.064658      1008  0.003850  -0.893754
9         6816.629191      1011  0.007069  -1.286337
10        3957.108968      1014  0.009310  -1.699831
11        2183.340010      1016  0.007624  -2.002786
12        1072.103115      1019  0.008746  -2.399153
13         405.621434      1025  0.013723  -2.956529
14         101.611623      1035  0.020699  -3.538236
15          11.128647      1087  0.103677  -4.373168
16           9.924435      1104  0.034469  -4.861617
17           1.907059      1382  0.544556  -5.671337
18           4.584891      1447  0.128483  -6.316777
19           1.252109      1563  0.229147  -7.004861
20           0.918424      1652  0.176536  -7.505081
21           0.215880      2020  0.725671  -8.201636
22           0.231203      2417  0.780489  -8.765148
23           0.140629      2853  0.867661  -9.181382
24           0.097148      3406  1.101057  -9.479619
25           0.060599      4001  1.165088  -9.668562
26           0.034878      4921  1.820172  -9.788430
27           0.028822      5307  0.799155  -9.838086
28           0.014474      6102  1.602430  -9.877824
29           0.009755      6578  0.945621  -9.895930
30           0.005009      7191  1.173336  -9.907228
31           0.003046      7910  1.421607  -9.913693
32           0.002544      8489  1.150602  -9.916745
33           0.001685      9489  1.987918  -9.918593
34           0.002479      9752  0.518716  -9.919159
35           0.000554     10752  1.970677  -9.919580
36           0.003511     10791  0.078427  -9.919626
37           0.000343     11791  1.971841  -9.919779
38           0.000845     12791  1.969032  -9.919813
39           0.001835     12868  0.158382  -9.919833
40           0.000154     13868  1.968844  -9.919810