image quality degradation as generation continues

[volotat]

image quality degradation as generation continues

[alexfredo]

I found a way to remove the ghosting : https://www.reddit.com/r/StableDiffusion/comments/12lmlem/i_found_a_way_to_remove_the_ghosting_of/

[volotat]

No. I'm not sure about the exact reason but one of the things might be that you have multiple versions of python installed on your system. Try the following, run:
py -m pip freeze
Check if there is h5py in the list. If there is not, that means that when you run 'pip' command it interacts with the different version of python that is set to 'py' command. If so, you should install h5py again with the following command:
py -m pip install h5py

[alexfredo]

I made a quick test I don't have quality degradation anymore and there's less ghosting, awesome job !

[CaptnSeraph]

1 hour and 361 frames in, no color distortion and no background distortion. seems to be fixed

[Cubey42]

like its animating I guess but reality has melted

[alexfredo]

for the error you get every generation : "go to modules/api/api.py in web-ui and replace this string (it should be at 275 line)
script_args[alwayson_script.args_from:alwayson_script.args_to] = request.alwayson_scripts[alwayson_script_name]["args"]
with this:

for idx in range(0, min((alwayson_script.args_to - alwayson_script.args_from), len(request.alwayson_scripts[alwayson_script_name]["args"]))):
script_args[alwayson_script.args_from + idx] = request.alwayson_scripts[alwayson_script_name]["args"][idx]

Also in my case the image was become purple I fixed this bug by enable "Apply color correction to img2img results to match original colors." in settings>stable diffusion I don't know if that can help you, try differents resolution too.
I have the picture degraded a lot when LoRA is activated too
Hope that helps

[Cubey42]

like so?

[alexfredo]

Yes

[alexfredo]

Also I need to restart automatic1111 every time I make a test with script.py

[Cubey42]

nm that just needed another indent

[alexfredo]

It's doesn't fix the error message sorry I don't know how to solve this problem do you have the last version of automatic1111 ?

[alexfredo]

Oh ok

[alexfredo]

I have still some problem with the script too, sometime it's become more and more darker every images

[Cubey42]

I have still some problem with the script too, sometime it's become more and more darker every images

likewise, It seems like the control net is losing all the detail

[alexfredo]

I hope that will be fixed soon this script is awesome when it works

[Cubey42]

could we run another controlnet alongside hed?

[alexfredo]

I don't know but I have tried canny instead of hed on a character without environment, here the result : https://www.reddit.com/r/StableDiffusion/comments/12hopoe/titok_dancer_convert_into_jenna_ortega_using/

[Cubey42]

I don't know but I have tried canny instead of hed on a character without environment, here the result : https://www.reddit.com/r/StableDiffusion/comments/12hopoe/titok_dancer_convert_into_jenna_ortega_using/

interesting, what does using canny look like in the script?

[alexfredo]

I put this settings in script.py :
"ControlNet":{
"args": [
{
"input_image": b64_hed_img,
"module": "canny",
"model": "control_canny-fp16 [e3fe7712]",
"weight": 0.4,
"resize_mode": "Just Resize",
"lowvram": False,
"processor_res": 512,
"guidance": 1,
"guessmode": False
}

[alexfredo]

But I thinks environments will be maybe destroy with this settings

[alexfredo]

For your pictures example above have you try "w,h = 480, 720" in script.py for the resolution ?

[Cubey42]

I'm still playing around with settings, I've tried a couple resolutions but I'm exploring other parameters to see what can help

[alexfredo]

I tested a older version of script.py and I don't have image quality degradation with this version : https://raw.githubusercontent.com/volotat/SD-CN-Animation/c75ea0c5e7f44a5ae4f3323f58447beccb879a91/script.py

[alexfredo]

But the temporal consistency and probably not as good the last version

[Cubey42]

I'll give it a try. thinking of the color problem, maybe the vae isn't staying loaded with each frame inference?

[alexfredo]

I dont know, also do you have installed the requirements for RAFT maybe it's can fix the problem :
conda create --name raft
conda activate raft
conda install pytorch=1.6.0 torchvision=0.7.0 cudatoolkit=10.1 matplotlib tensorboard scipy opencv -c pytorch

[Cubey42]

yes I did that RAFT setup. Also I think the old script.py didn't use RAFT either, I think thats the reason there is more temporal consistency is RAFT

[CaptnSeraph]

is it faster/better without RAFT? i know it uses less VRAM, but is there a tradeoff when using RAFT

[rezponze]

I'm having the same problem, the image degrades after the first frame. I've made sure I had all RAFT dependencies, also tried the previous version of the script linked above, but it degrades the frames as well. Very frustrating because really want to use this!

[volotat]

Please, check out the last version of the script and tell me if the issue with quality degradation is still prevalent.

[alexfredo]

I have a error when I try to compute the flow :
File "C:\AUTOMATIC1111\SD-CN-AnimationV4\compute_flow.py", line 8, in
import h5py
ModuleNotFoundError: No module named 'h5py'

I have installed hp5py with "pip install h5py" but it's doesn't work
For the command I have write : py compute_flow.py -i {C:\AUTOMATIC1111\SD-CN-AnimationV4\1.mp4} -o {C:\AUTOMATIC1111\SD-CN-AnimationV4\output *.h5} -v -W {720} -H {480}

[volotat]

You should not put brackets at the parameters, it is just an indication that you should modify it. Try it like this
py compute_flow.py -i C:\AUTOMATIC1111\SD-CN-AnimationV4\1.mp4 -o C:\AUTOMATIC1111\SD-CN-AnimationV4\output_flow.h5 -v -W 720 -H 480

[alexfredo]

I have still the "No module named 'h5py'" error maybe I need to restart my computer ?

[alexfredo]

You have right it's fixed my problem now it's works

[CaptnSeraph]

running a longer test now, will take about 2 hours to complete (698 frames at 400x600)
question, is it possible that using Euler as opposed to DPM 2S a would cause the issue?
ive switched to Euler a incase its a lack of ancestral that caused it too.

[Cubey42]

The degradation seems to be resolved! thank you. Just to confirm, the ghosting effect is the result of too low an FPS correct? higher framerates help remove it?

[CaptnSeraph]

Yeah, you can see it in the optical flow window when you run compute_optical.py.

The more white you see the more movement per frame.

And the more ghosting.

I wonder if interpolation would help, as SD is going to re-create the images anyway, any random mid frames created might dissappear with the img2img process.

[Cubey42]

I'm still running a 30fps test, I'll try 60/120 soon

[Cubey42]

do you think open pose is worth trying as well? we are only using canny by default but I was thinking about trying a multi-controlnet scheme. also the release of controlnet1.1 is exciting, I hope they bring it to a1111 quickly

[alexfredo]

For the ghosting problem maybe if I use the super slow motion of Runwayml for generate a very high frame rate video it's can fix the problem, but obviously the render time will be extremely slow with a high numbers of images and the flow file too big...

[CaptnSeraph]

with the recent background removal update, are we happy to close this issue or does it need further work?