Short positions lead to negative number of shares to buy back

[edtechre]

Short positions lead to negative number of shares to buy back

[edtechre]

Hi @Pirat83,

Thanks for providing your code and insights, it has been very helpful. The problem is with this line:

ctx.sell_shares = ctx.calc_target_shares(1 / len(signals))

If I print ctx.sell_shares, I can see it growing very rapidly:

...
1241787311188310
2487107915319820
4973029069724121
9944626969576674
2260623410049139
3388679891662696
5924987718324092
11607585552702381
23202327536260160
46404926525674633
92750640633824432
185547839541663600
371084823380017587
-19976084585449627

Eventually, the value becomes too large and causes integer overflow and becomes negative, which then triggers the error you see.

If I remove that section of code then the strategy runs without throwing the error.

The error is thrown because of the below:

I think ctx.calc_target_shares() takes the available cash to deterimante the position sizing. Since the total margin is also added to the cash the pybroker can buy as much shares as possible. And the margin is not limited (like in the real world) or ever payed back and eliminated from the books.

Yes, you are correct. The margin in PyBroker is unlimited, and only "paid back" when the shares are covered (bought back) by the strategy. The portfolio is immediately credited with cash when shorting shares. I believe the reason you see the cash grow exponentially is because the number of shares being shorted increases exponentially.

I can add an extra configuration that would allow you to set a margin limit, if that would also be helpful.

[Pirat83]

So when I take only long positions the numbers of the backtest TestResult look quite promissing:

IVV   2023-01-04 13:30:00    66: O:  384.8400 H:  384.8400 L:  384.6500 C:  384.8900 V:      2487 | TE:    10000.00 TMA:        0.00 TMV:    10000.00 | Cash:    10000.00 Long:        0.00 Short:        0.00
IWM   2023-01-04 13:30:00    82: O:  174.8110 H:  174.8110 L:  173.4000 C:  174.8300 V:    109277 | TE:    10000.00 TMA:        0.00 TMV:    10000.00 | Cash:    10000.00 Long:        0.00 Short:        0.00
IVV   2023-01-04 13:45:00    67: O:  384.9300 H:  384.9300 L:  384.4500 C:  384.7200 V:      9771 | TE:     9989.17 TMA:        0.00 TMV:     9989.17 | Cash:       32.98 Long:        0.00 Short:        0.00
IWM   2023-01-04 13:45:00    83: O:  174.8600 H:  174.8600 L:  174.5800 C:  174.6700 V:     39667 | TE:     9989.17 TMA:        0.00 TMV:     9989.17 | Cash:       32.98 Long:     9956.19 Short:        0.00
IVV   2023-01-04 14:00:00    68: O:  384.7400 H:  384.7400 L:  384.2500 C:  384.3300 V:      7539 | TE:     9981.76 TMA:        0.00 TMV:     9981.76 | Cash:       32.98 Long:        0.00 Short:        0.00
IWM   2023-01-04 14:00:00    84: O:  174.7200 H:  174.7200 L:  174.4400 C:  174.5400 V:     47367 | TE:     9981.76 TMA:        0.00 TMV:     9981.76 | Cash:       32.98 Long:     9948.78 Short:        0.00
IVV   2023-01-04 14:15:00    69: O:  384.3000 H:  384.3000 L:  384.1200 C:  384.9700 V:     28054 | TE:     9986.32 TMA:        0.00 TMV:     9986.32 | Cash:       32.98 Long:        0.00 Short:        0.00
IWM   2023-01-04 14:15:00    85: O:  174.5000 H:  174.5000 L:  174.3800 C:  174.6200 V:     45146 | TE:     9986.32 TMA:        0.00 TMV:     9986.32 | Cash:       32.98 Long:     9953.34 Short:        0.00
IVV   2023-01-04 14:30:00    70: O:  384.9400 H:  384.9400 L:  383.6900 C:  384.4000 V:    336823 | TE:    10029.63 TMA:        0.00 TMV:    10029.63 | Cash:       32.98 Long:        0.00 Short:        0.00
IWM   2023-01-04 14:30:00    86: O:  174.6200 H:  174.6200 L:  174.4500 C:  175.3799 V:   2432679 | TE:    10029.63 TMA:        0.00 TMV:    10029.63 | Cash:       32.98 Long:     9996.65 Short:        0.00
IVV   2023-01-04 14:45:00    71: O:  384.3500 H:  384.3500 L:  383.9600 C:  384.9700 V:     91591 | TE:    10042.18 TMA:        0.00 TMV:    10042.18 | Cash:       32.98 Long:        0.00 Short:        0.00
IWM   2023-01-04 14:45:00    87: O:  175.4000 H:  175.4000 L:  175.2200 C:  175.6000 V:    960340 | TE:    10042.18 TMA:        0.00 TMV:    10042.18 | Cash:       32.98 Long:    10009.20 Short:        0.00

At some point the pybroker starts buying assets and the long positions are added to the uninvested cash, we take no margin and I can life with the results. You can see the whole log file, the portfolio, the positions, the orders and the trades in the csv files attached to this comment.

long_only_log.txt

[Pirat83]

When I take short positions only or short and long positions the numbers of the backtest TestResult start to confuse me:

IWM   2023-01-04 00:15:00    61: O:  173.3400 H:  173.3400 L:  173.3400 C:  173.4100 V:      4351 | TE:    10000.00 TMA:        0.00 TMV:    10000.00 | Cash:    10000.00 Long:        0.00 Short:        0.00
IWM   2023-01-04 00:30:00    62: O:  173.4800 H:  173.4800 L:  173.2700 C:  173.3000 V:      2566 | TE:    10000.00 TMA:        0.00 TMV:    10000.00 | Cash:    10000.00 Long:        0.00 Short:        0.00
IWM   2023-01-04 00:45:00    63: O:  173.2700 H:  173.2700 L:  173.2300 C:  173.3200 V:     18905 | TE:    10000.00 TMA:        0.00 TMV:    10000.00 | Cash:    10000.00 Long:        0.00 Short:        0.00
IVV   2023-01-04 00:45:00    55: O:  382.4700 H:  382.4700 L:  382.4700 C:  382.5200 V:       700 | TE:    10000.00 TMA:        0.00 TMV:    10000.00 | Cash:    10000.00 Long:        0.00 Short:        0.00
IWM   2023-01-04 09:00:00    64: O:  174.0700 H:  174.0700 L:  173.9400 C:  174.1000 V:     15532 | TE:    19921.99 TMA:     9923.70 TMV:     9998.29 | Cash:    19921.99 Long:        0.00 Short:     9921.99
IWM   2023-01-04 09:15:00    65: O:  174.0600 H:  174.0600 L:  174.0500 C:  174.1000 V:      7266 | TE:    15046.07 TMA:     5048.90 TMV:     9997.17 | Cash:    15046.07 Long:        0.00 Short:     5048.03
IWM   2023-01-04 09:30:00    66: O:  174.0500 H:  174.0500 L:  174.0500 C:  174.2500 V:     16145 | TE:    12607.69 TMA:     2613.75 TMV:     9993.94 | Cash:    12607.69 Long:        0.00 Short:     2611.05
IWM   2023-01-04 09:45:00    67: O:  174.3100 H:  174.3100 L:  174.2200 C:  174.3500 V:      6576 | TE:    11387.52 TMA:     1394.80 TMV:     9992.72 | Cash:    11387.52 Long:        0.00 Short:     1392.56
IWM   2023-01-04 10:00:00    68: O:  174.3500 H:  174.3500 L:  174.3500 C:  174.5300 V:      6361 | TE:    10689.68 TMA:      698.12 TMV:     9991.56 | Cash:    10689.68 Long:        0.00 Short:      696.28
IWM   2023-01-04 10:15:00    69: O:  174.4400 H:  174.4400 L:  174.4400 C:  174.4400 V:       192 | TE:    10340.80 TMA:      348.88 TMV:     9991.92 | Cash:    10340.80 Long:        0.00 Short:      348.14
IWM   2023-01-04 10:30:00    70: O:  174.4600 H:  174.4600 L:  174.0000 C:  174.1300 V:     19604 | TE:     9991.88 TMA:        0.00 TMV:     9991.88 | Cash:     9991.88 Long:        0.00 Short:        0.00
IVV   2023-01-04 10:30:00    56: O:  383.6000 H:  383.6000 L:  383.5300 C:  383.5300 V:       600 | TE:     9991.88 TMA:        0.00 TMV:     9991.88 | Cash:     9991.88 Long:        0.00 Short:        0.00
IWM   2023-01-04 10:45:00    71: O:  174.1700 H:  174.1700 L:  174.1600 C:  174.2100 V:      8685 | TE:     9991.88 TMA:        0.00 TMV:     9991.88 | Cash:     9991.88 Long:        0.00 Short:        0.00
IWM   2023-01-04 11:00:00    72: O:  174.2400 H:  174.2400 L:  174.2400 C:  174.5100 V:      4215 | TE:    19978.48 TMA:     9989.20 TMV:     9989.28 | Cash:    19978.48 Long:        0.00 Short:        0.00
IVV   2023-01-04 11:00:00    57: O:  384.1000 H:  384.1000 L:  384.1000 C:  384.2000 V:       650 | TE:    19978.48 TMA:     9989.20 TMV:     9989.28 | Cash:    19978.48 Long:        0.00 Short:     9986.60
IWM   2023-01-04 11:15:00    73: O:  174.5000 H:  174.5000 L:  174.4000 C:  174.4000 V:      3500 | TE:    19978.48 TMA:     9989.20 TMV:     9989.28 | Cash:    19978.48 Long:        0.00 Short:        0.00
IWM   2023-01-04 11:30:00    74: O:  174.4000 H:  174.4000 L:  174.4000 C:  174.5000 V:      1928 | TE:    39969.36 TMA:    29994.12 TMV:     9975.24 | Cash:    39969.36 Long:        0.00 Short:        0.00
IVV   2023-01-04 11:30:00    58: O:  384.4400 H:  384.4400 L:  384.4400 C:  384.5400 V:       622 | TE:    39969.36 TMA:    29994.12 TMV:     9975.24 | Cash:    39969.36 Long:        0.00 Short:    29977.48
IWM   2023-01-04 11:45:00    75: O:  174.3900 H:  174.3900 L:  174.3900 C:  174.4500 V:       943 | TE:    79574.92 TMA:    69598.12 TMV:     9976.80 | Cash:    79574.92 Long:        0.00 Short:        0.00
IVV   2023-01-04 11:45:00    59: O:  384.5200 H:  384.5200 L:  384.5200 C:  384.5200 V:       301 | TE:    79574.92 TMA:    69598.12 TMV:     9976.80 | Cash:    79574.92 Long:        0.00 Short:    69583.04
IWM   2023-01-04 12:00:00    76: O:  174.1800 H:  174.1800 L:  174.0300 C:  174.2300 V:     10625 | TE:   158705.70 TMA:   148608.00 TMV:    10097.70 | Cash:   158705.70 Long:        0.00 Short:        0.00
IVV   2023-01-04 12:00:00    60: O:  384.1300 H:  384.1300 L:  383.7000 C:  384.0000 V:      2597 | TE:   158705.70 TMA:   148608.00 TMV:    10097.70 | Cash:   158705.70 Long:        0.00 Short:   148713.82
IWM   2023-01-04 12:15:00    77: O:  174.1500 H:  174.1500 L:  174.0000 C:  174.0500 V:     43075 | TE:   243131.96 TMA:   232840.76 TMV:    10291.20 | Cash:   243131.96 Long:        0.00 Short:   158476.50
IVV   2023-01-04 12:15:00    61: O:  383.7300 H:  383.7300 L:  383.5700 C:  383.7900 V:      1461 | TE:   243131.96 TMA:   232840.76 TMV:    10291.20 | Cash:   243131.96 Long:        0.00 Short:    74521.22

At some point the pybroker doubles the cash and it is growing exponentially very fast (And I am pretty sure that my strategy is so good :-P)

I think ctx.calc_target_shares() takes the available cash to deterimante the position sizing. Since the total margin is also added to the cash the pybroker can buy as much shares as possible. And the margin is not limited (like in the real world) or ever payed back and eliminated from the books. The margin is also not taken into account when taking long positions. But this is just my first impression. Maybe I am completely wrong.

long_short_log.txt

[Pirat83]

Yes this option to limit the margin was also my preferred solution, while only guessing the root cause. This limit could also apply for long positions, so people could simulate margin in their portfolios.

Thank you very much. I appreciate your work.

[edtechre]

Note, that you can check the number of short shares outstanding and use that to bound the value set on ctx.sell_shares. You may want to do that even after I add the configuration change, otherwise your strategy will then continue to short shares only until the margin limit is exhausted.

[edtechre]

After giving this more thought, I believe the main problem is with the way ctx.calc_target_shares is being used, which is different from its intended use.

From the previous code, it looks like you want to continue to short shares without covering them. Meanwhile, calc_target_shares by default uses the total portfolio equity to calculate the number of shares given a target ratio. So, when len(signals) == 1, then your code ends up shorting an amount equal to the entire portfolio equity every time you set ctx.sell_shares.

What you want to do instead is to be explicit and use the cash argument to restrict the amount that you can short.

ctx.sell_shares = ctx.calc_target_shares(
   1 / len(signals), cash=SHORT_LIMIT - ctx.total_margin,
)

Where SHORT_LIMIT is the limit for short positions. You can adjust SHORT_LIMIT yourself based on current ctx.total_market_value etc. (total_market_value excludes margin).

I have also added a check to guarantee that ctx.calc_target_shares never returns a negative number, and instead will return 0 shares even if called erroneously with negative values. (de49cf1)

I will consider adding margin limits as a feature in the future. However, I have been hesitant to do so because PyBroker will not simulate maintenance margin or margin calls, and the main goal of the framework is to identify effective trading signals, which are not affected by the use of leverage.

Let me know if you need anything else!

[edtechre]

A few updates to share:

• To better simulate margin, short selling proceeds are no longer credited to portfolio cash.
• Portfolio cash is no longer used to cover short positions.
• I have added an example for limiting portfolio margin to the FAQs.

[Pirat83]

Hi @edtechre. Thank you for your work and your time. I appretierte your work very much.

Just one last question:
Can I use the same logic for margin trades in long direction?

[edtechre]

No, since margin is only used for short positions. You can increase the initial_cash for the strategy to simulate more buying power.