CivitAI direct link method: slow download speed

[etherealxx]

CivitAI direct link method: slow download speed

[etherealxx]

Download speed is slow from civitai.

Hello @faisalhr1997👋 The update is now live! It upgrades the method used to download from civitai. Now you can use aria2 and enjoy the maximum downlad speed! (it also now support gdrive links!✨)

Release note (every new feature): https://github.com/etherealxx/batchlinks-webui#release-v301

[etherealxx]

https://colab.research.google.com/drive/1wEa-tS10h4LlDykd87TF5zzpXIIQoCmq#scrollTo=8Uvqba0yKW7U This colabs download mechanism got the code you could reuse

mine is better coz it supports MEGA👀 (though for gdrive, i'll add it on the next update, it's complicated to implement that without bugs)
anyway, he's using direct download method, nothing special with it. It's literally the download method i use to download huggingface etc.
His code also can't differentiate .safetensors embedding/LoRA, his system is flawed

[x-legion]

taking forever to download from civitai

[etherealxx]

taking forever to download from civitai

if you use the direct link method for civitai, (civitai.com/api/download/models), it will use request module (to retrieve the filename) instead of the normal download method (gdown/wget/etc). That's why it uses temporary name first (20491.bdgh). You can use the cancel button and re download the thing.

However, i think i'll update the extension so that when you input the name manually, it will use the normal download method. Thank you for the report✨

[x-legion]

Naming is not the issue here . Download speed is slow from civitai.

[etherealxx]

Naming is not the issue here . Download speed is slow from civitai.

no no, you don't get it. hear me out

links with civitai.com/api/download/models doesn't have a filename in it.

request module is needed to get the filename correctly from content-disposition

downloading the entire file is a must to get that

it's a waste of time when downloading an entire file just to get the filename, then redownloading it again

so i just used the downloaded file from request

[etherealxx]

HOWEVER, after some experimenting, i found out that this

import subprocess
link = 'https://civitai.com/api/download/models/20491'
contentdis = [line for line in subprocess.getoutput(f"curl -sI {link} | grep -i content-disposition").splitlines() if line.startswith('location')][0]
cuttedcontent = contentdis.find('response-content-disposition=attachment%3B%20filename%3D%22') + 59
filename = str(contentdis[cuttedcontent:]).replace('%22&x-id=GetObject', '')
print(filename)

able to get the filename of that civit link WITHOUT downloading it. So i need to check if this piece of code works all the time. If yes, then i'll use this, and abandon the requests method, so that i am able to use other downloader (gdown/wget/etc.) to download the file faster.

[etherealxx]

I made this branch to (hopefully) fix that problem

!rm -rf /content/stable-diffusion-webui/extensions/batchlinks-webui
!git clone -b civitupgrade https://github.com/etherealxx/batchlinks-webui /content/stable-diffusion-webui/extensions/batchlinks-webui

this should use the default downloader instead of requests for downloading civit stuff.
i need to make sure this works first before merging with main.

[x-legion]

https://colab.research.google.com/drive/1wEa-tS10h4LlDykd87TF5zzpXIIQoCmq#scrollTo=8Uvqba0yKW7U
This colabs download mechanism got the code you could reuse