memory leak when compiling collective + view + wait()

[bdhirsh]

memory leak when compiling collective + view + wait()

[eellison]

We should fix this asap but there is a hotfix already landed.. arguably should still be high pri, will look soon

[vkuzo]

here is a smaller repro: https://github.com/pytorch-labs/float8_experimental/pull/260/files

[bdhirsh]

pytorch-labs/float8_experimental#262 is a hotfix for float8, although from talking to @eellison it sounds like we can potentially fix this in inductor by having inductor not convert views into non-views when we know they came directly from a collective

[drisspg]

I have some confusion regarding the necessity of the kernel # triton_poi_fused_view_3. The code responsible for generating this kernel can be found here.

This optorch.ops.aten.view.dtype(all_gather_into_tensor_2, torch.float8_e5m2) is a view but it should be a no-op since it merely converts between a uint8 and an fp8 dtype, both of which have the same bit width. This code was used to workaround the fact that nccl doesn't currently support comms for fp8 dtype (or at least our bindings to nccl in pytorch). Perhaps we need to allow fp8 dtypes to all_gather and to the uint trick down there?