[bug]: Extremely low image quality on some models

[JantDarvus]

[bug]: Extremely low image quality on some models

[psychedelicious]

Yikes!

For example, this is one using the yiffymix_v43 two days ago.

To clarify: Were you still on 3.x two days ago?

I believe this is the model: https://civitai.com/models/129996/easyfluff

Based on the description there, this model uses v-prediction. I noticed a couple issues in our handling:

1. We aren't correctly determining checkpoint scheduler prediction on import, so the model is set as epsilon.
2. When you go and change it to v-pred, it doesn't change the config file path, which is required to convert the model to diffusers and use it.
3. Even if you manually fix the config file path, if the model was converted once before, it will not be converted again. The diffusers conversion cache must be emptied to trigger the conversion again.

Once I manually fix the config file path (stable-diffusion/v1-inference-v.yaml is the correct value) and delete the model conversion cache, I get better images. Using CLIP Skip 2 and CFG Rescale 0.7 per the model page:

So no longer totally broken, but still not right. They all have this grainy look. I tried enabling upcast_attention, which I think is related to v-pred and re-converting - this didn't seem to change anything.

At this point, I'm not sure how to proceed further.

@lstein Any ideas for fixing the root issue?

I think there are clear solutions for each of the 3 issues I noted:

1. Figure out how to determine the prediction type from the state dict for SD1. Right now, it doesn't try, jsut defaults to epsilon. I don't know how to do this.
2. Do not hardcode the ckpt config file paths unless the user explicitly sets it. When we convert, if there is no config file path, we determine the right config file using the same logic the probe uses.
3. Invalidate the ckpt conversion cache if a change to a model record means it would convert differently. We could simply invalidate the conversion cache on any model config change. I think that is reasonable, though it is less efficient.

[JantDarvus]

I was still on 3.x two days ago, yeah. I had used an old installer I still had on my desktop at the time.
The model you're using is correct, but just to check, is that using the config file provided with the model? Not sure if that might be causing the grainy look there or if there's indeed still an issue.

[psychedelicious]

No, I didn't notice there was a config file provided. Unfortunately, using it doesn't change the outputs - still grainy. I made sure to re-convert the model.

In v3, we used the diffusers ckpt conversion logic with some fixes. Those fixes were upstreamed to diffusers, and in v4 we moved back to the standard diffusers conversion. I suspect we are missing an arg in the conversion calls, or maybe diffusers has a bug.