tropicoo/yt-dlp-bot

Self-hosted Video Download Telegram Bot 🇺🇦

Hi there 👋

My name is Taras Terletsky. I'm software engineer from Ukraine 🇺🇦 with strong interest in all things digital and beyond.

Projects

• hikvision-camera-bot — The Telegram Bot designed for Hikvision cameras with a wide range of features such as capturing images, creating video-gifs upon motion detection, and sending various alerts.
• yt-dlp-bot — Self-hosted Telegram Bot for downloading YouTube videos
• zoneh — Zone-H Cybercrime Archive Telegram Monitoring Bot
• youtube-dl-ffmpeg-updater-win — Windows youtube-dl and ffmpeg binaries updater for lazy ones (planned to be rewritten to support yt-dlp)

Support my work

• Buy me a coffee
• PayPal
• Bitcoin wallet 14kMRS8SvfD2ydMSMEyAmefHV3Yynf9kAd

Development supported by JetBrains and Syntevo

yt-dlp-bot's Issues

Video file not found

When sending a Youtube URL to the bot it returns the following error:
🛑 General error

ℹ️ Task ID: 7e4854a8-399e-4c8c-9dac-6f00765f6e41
💬 Message: Upload error
📹 Video URL: https://www.youtube.com/watch?v=C0DPdy98e4c
🌊 Source: BOT
👀 Details: Traceback (most recent call last):
File "/app/bot/core/handlers/success.py", line 28, in handle
await self._handle()
File "/app/bot/core/handlers/success.py", line 55, in _handle
self._validate_file_size_for_upload(video_path)
File "/app/bot/core/handlers/success.py", line 102, in _validate_file_size_for_upload
raise ValueError(f'Video {video_path} not found')
ValueError: Video /tmp/download_tmpfs/TEST VIDEO.mp4 not found

⬇️ yt-dlp version: 2023.01.06
🏷️ Tag: #error

Errors from the log:

yt_worker | 2023-02-05 21:30:30,961 - [INFO] - [VideoDownloader:33] - Downloading https://www.youtube.com/watch?v=C0DPdy98e4c
yt_worker | 2023-02-05 21:30:30,962 - [INFO] - [VideoDownloader:37] - Downloading to /tmp/download_tmpfs/tmp_video_dir-j_s9nguy
yt_worker | [youtube] Extracting URL: https://www.youtube.com/watch?v=C0DPdy98e4c
yt_worker | [youtube] C0DPdy98e4c: Downloading webpage
yt_worker | [youtube] C0DPdy98e4c: Downloading android player API JSON
yt_worker | [info] C0DPdy98e4c: Downloading 1 format(s): 135+140
yt_worker | [download] Destination: /tmp/download_tmpfs/tmp_video_dir-j_s9nguy/TEST VIDEO.f135.mp4
[download] 100% of 155.85KiB in 00:00:00 at 1.31MiB/s
yt_worker | [download] Destination: /tmp/download_tmpfs/tmp_video_dir-j_s9nguy/TEST VIDEO.f140.m4a
[download] 100% of 279.31KiB in 00:00:00 at 4.05MiB/s
yt_worker | [Merger] Merging formats into "/tmp/download_tmpfs/tmp_video_dir-j_s9nguy/TEST VIDEO.mp4"
yt_worker | Deleting original file /tmp/download_tmpfs/tmp_video_dir-j_s9nguy/TEST VIDEO.f140.m4a (pass -k to keep)
yt_worker | Deleting original file /tmp/download_tmpfs/tmp_video_dir-j_s9nguy/TEST VIDEO.f135.mp4 (pass -k to keep)
yt_worker | 2023-02-05 21:30:32,458 - [INFO] - [VideoDownloader:47] - Finished downloading https://www.youtube.com/watch?v=C0DPdy98e4c
yt_worker | 2023-02-05 21:30:32,458 - [INFO] - [VideoDownloader:52] - Moving /tmp/download_tmpfs/tmp_video_dir-j_s9nguy/TEST VIDEO.mp4 to /tmp/download_tmpfs
yt_worker | 2023-02-05 21:30:32,458 - [INFO] - [VideoDownloader:53] - Content of /tmp/download_tmpfs/tmp_video_dir-j_s9nguy: ['TEST VIDEO.mp4']
yt_worker | 2023-02-05 21:30:32,458 - [INFO] - [VideoDownloader:55] - Removing /tmp/download_tmpfs/tmp_video_dir-j_s9nguy
yt_worker | 2023-02-05 21:30:32,467 - [INFO] - [asyncio:1658] - run shell command 'ffmpeg -y -loglevel error -i "/tmp/download_tmpfs/TEST VIDEO.mp4" -ss 10.0 -vframes 1 -q:v 7 "/tmp/download_tmpfs/TEST VIDEO.mp4-thumb.jpg"': <_UnixSubprocessTransport pid=45 running stdout=<_UnixReadPipeTransport fd=8 polling> stderr=<_UnixReadPipeTransport fd=10 polling>>
yt_worker | 2023-02-05 21:30:32,962 - [INFO] - [asyncio:525] - <_UnixReadPipeTransport fd=10 polling> was closed by peer
yt_worker | 2023-02-05 21:30:32,962 - [INFO] - [asyncio:525] - <_UnixReadPipeTransport fd=8 polling> was closed by peer
yt_worker | 2023-02-05 21:30:32,963 - [INFO] - [asyncio:211] - <_UnixSubprocessTransport pid=45 running stdout=<_UnixReadPipeTransport closed fd=8 closed> stderr=<_UnixReadPipeTransport closed fd=10 closed>> exited with return code 0
yt_worker | 2023-02-05 21:30:32,977 - [INFO] - [sqlalchemy.engine.Engine:2666] - BEGIN (implicit)
yt_worker | 2023-02-05 21:30:32,982 - [INFO] - [sqlalchemy.engine.Engine:1842] - SELECT file.id AS file_id, file.title AS file_title, file.name AS file_name, file.thumb_name AS file_thumb_name, file.duration AS file_duration, file.width AS file_width, file.height AS file_height, file.meta AS file_meta, file.task_id AS file_task_id, file.created AS file_created, file.updated AS file_updated
yt_worker | FROM file
yt_worker | WHERE $1::UUID = file.task_id
yt_worker | 2023-02-05 21:30:32,982 - [INFO] - [sqlalchemy.engine.Engine:1842] - [generated in 0.00035s] ('7e4854a8-399e-4c8c-9dac-6f00765f6e41',)
yt_worker | 2023-02-05 21:30:32,988 - [INFO] - [sqlalchemy.engine.Engine:1842] - UPDATE task SET status=$1::taskstatus, updated=$2::TIMESTAMP WITHOUT TIME ZONE WHERE task.id = $3::UUID
yt_worker | 2023-02-05 21:30:32,988 - [INFO] - [sqlalchemy.engine.Engine:1842] - [cached since 2.035s ago] ('DONE', datetime.datetime(2023, 2, 5, 21, 30, 32, 987898), '7e4854a8-399e-4c8c-9dac-6f00765f6e41')
yt_worker | 2023-02-05 21:30:32,994 - [INFO] - [sqlalchemy.engine.Engine:1842] - INSERT INTO file (id, title, name, thumb_name, duration, width, height, meta, task_id, created, updated) VALUES ($1::UUID, $2::VARCHAR, $3::VARCHAR, $4::VARCHAR, $5::INTEGER, $6::INTEGER, $7::INTEGER, $8::JSONB, $9::UUID, $10::TIMESTAMP WITHOUT TIME ZONE, $11::TIMESTAMP WITHOUT TIME ZONE)
yt_worker | 2023-02-05 21:30:32,994 - [INFO] - [sqlalchemy.engine.Engine:1842] - [generated in 0.00243s] ('ddd8a7cc-4ee5-49e9-955b-6c96e7dfc5a8', 'TEST VIDEO', 'TEST VIDEO.mp4', 'TEST VIDEO.mp4-thumb.jpg', 18.0, 640, 480, '{"id": "C0DPdy98e4c", "title": "TEST VIDEO", "formats": [{"format_id": "sb2", "format_note": "storyboard", "ext": "mhtml", "protocol": "mhtml", "acod ... (52442 characters truncated) ... 632, "_type": "video", "_version": {"version": "2023.01.06", "current_git_head": null, "release_git_head": "6becd25", "repository": "yt-dlp/yt-dlp"}}', '7e4854a8-399e-4c8c-9dac-6f00765f6e41', datetime.datetime(2023, 2, 5, 21, 30, 32, 992086), datetime.datetime(2023, 2, 5, 21, 30, 32, 992095))
yt_worker | 2023-02-05 21:30:33,003 - [INFO] - [sqlalchemy.engine.Engine:2672] - COMMIT
yt_worker | 2023-02-05 21:30:33,015 - [INFO] - [_RMQCallbacks:32] - Processing done with payload: id=None from_chat_id=***** from_chat_type=<TelegramChatType.PRIVATE: 'private'> from_user_id=***** message_id=25 url='https://www.youtube.com/watch?v=C0DPdy98e4c' source=<TaskSource.BOT: 'BOT'> added_at=datetime.datetime(2023, 2, 5, 21, 30, 28, 139, tzinfo=datetime.timezone.utc)
yt_bot | 2023-02-05 21:30:33,016 - [INFO] - [SuccessResultWorker:53] - [x] Received message b'{"type":3,"task_id":"7e4854a8-399e-4c8c-9dac-6f00765f6e41","from_chat_id":,"from_chat_type":"private","from_user_id":,"message_id":25,"title":"TEST VIDEO","filename":"TEST VIDEO.mp4","thumb_name":"TEST VIDEO.mp4-thumb.jpg","duration":18.0,"width":640,"height":480,"context":{"id":null,"from_chat_id":,"from_chat_type":"private","from_user_id":,"message_id":25,"url":"https://www.youtube.com/watch?v=C0DPdy98e4c","source":"BOT","added_at":"2023-02-05T21:30:28.000139+00:00"},"yt_dlp_version":"2023.01.06"}'
yt_bot | 2023-02-05 21:30:33,130 - [ERROR] - [SuccessHandler:58] - Upload of "/tmp/download_tmpfs/TEST VIDEO.mp4" failed, performing cleanup
yt_bot | 2023-02-05 21:30:33,139 - [INFO] - [ErrorResultWorker:53] - [x] Received message b'{"type":2,"task_id":"7e4854a8-399e-4c8c-9dac-6f00765f6e41","from_chat_id":,"from_chat_type":"private","from_user_id":,"message_id":25,"message":"Upload error","url":"https://www.youtube.com/watch?v=C0DPdy98e4c","context":{"id":null,"from_chat_id":,"from_chat_type":"private","from_user_id":,"message_id":25,"url":"https://www.youtube.com/watch?v=C0DPdy98e4c","source":"BOT","added_at":"2023-02-05T21:30:28.000139+00:00"},"exception_msg":"Traceback (most recent call last):\n File \"/app/bot/core/handlers/success.py\", line 28, in handle\n await self._handle()\n File \"/app/bot/core/handlers/success.py\", line 55, in _handle\n self._validate_file_size_for_upload(video_path)\n File \"/app/bot/core/handlers/success.py\", line 102, in _validate_file_size_for_upload\n raise ValueError(f'Video {video_path} not found')\nValueError: Video /tmp/download_tmpfs/TEST VIDEO.mp4 not found\n","exception_type":"ValueError","yt_dlp_version":"2023.01.06"}'

The bot did not receive a response

I followed the tutorial and everything ran smoothly, but my telegram did not receive a response from the robot. This is a screenshot of the console. Can you help me find out what went wrong?

Encountered a problem while docker compose build

I tried to start docker compose build base image according to your teaching document, but returned an error: Service base image has either an image nor a build context specified At least one must be provided

How to add %(upload_date)s %(channel)s to the video description?

Is there any way, to add upload date & chanell name, to description of uploaded video?

%(upload_date>%Y-%m-%d)s %(channel)s - %(title) formula, just change file name, but not video description.

raise KeyError(f"ID not found: {peer_id}")

port_pool' with 4 processes in it
yt_rabbitmq | 2023-04-22 13:18:50.644955+00:00 [info] <0.230.0> Running boot step cluster_name defined by app rabbit
yt_rabbitmq | 2023-04-22 13:18:50.645007+00:00 [info] <0.230.0> Running boot step direct_client defined by app rabbit
yt_rabbitmq | 2023-04-22 13:18:50.645120+00:00 [info] <0.230.0> Running boot step rabbit_maintenance_mode_state defined by app rabbit
yt_rabbitmq | 2023-04-22 13:18:50.645150+00:00 [info] <0.230.0> Creating table rabbit_node_maintenance_states for maintenance mode status
yt_rabbitmq | 2023-04-22 13:18:50.645452+00:00 [info] <0.230.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management
yt_rabbitmq | 2023-04-22 13:18:50.645582+00:00 [info] <0.526.0> Resetting node maintenance status
yt_rabbitmq | 2023-04-22 13:18:50.659434+00:00 [info] <0.585.0> Management plugin: HTTP (non-TLS) listener started on port 15672
yt_rabbitmq | 2023-04-22 13:18:50.659616+00:00 [info] <0.613.0> Statistics database started.
yt_rabbitmq | 2023-04-22 13:18:50.659681+00:00 [info] <0.612.0> Starting worker pool 'management_worker_pool' with 3 processes in it
yt_rabbitmq | 2023-04-22 13:18:50.671770+00:00 [info] <0.627.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
yt_rabbitmq | 2023-04-22 13:18:50.672032+00:00 [info] <0.526.0> Ready to start client connection listeners
yt_rabbitmq | 2023-04-22 13:18:50.673980+00:00 [info] <0.671.0> started TCP listener on [::]:5672
yt_rabbitmq | completed with 4 plugins.
yt_rabbitmq | 2023-04-22 13:18:50.839267+00:00 [info] <0.526.0> Server startup complete; 4 plugins started.
yt_rabbitmq | 2023-04-22 13:18:50.839267+00:00 [info] <0.526.0> * rabbitmq_prometheus
yt_rabbitmq | 2023-04-22 13:18:50.839267+00:00 [info] <0.526.0> * rabbitmq_management
yt_rabbitmq | 2023-04-22 13:18:50.839267+00:00 [info] <0.526.0> * rabbitmq_web_dispatch
yt_rabbitmq | 2023-04-22 13:18:50.839267+00:00 [info] <0.526.0> * rabbitmq_management_agent
yt_bot | Waiting for RabbitMQ to be reachable on port 5672
yt_bot | Connection to RabbitMQ on port 5672 verified
yt_worker | Waiting for RabbitMQ to be reachable on port 5672
yt_worker | Connection to RabbitMQ on port 5672 verified
yt_api | Waiting for RabbitMQ to be reachable on port 5672
yt_api | Connection to RabbitMQ on port 5672 verified
yt_worker | INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
yt_worker | INFO [alembic.runtime.migration] Will assume transactional DDL.
yt_bot | 2023-04-22 13:18:52,332 - [INFO] - [VideoBot:27] - Initializing bot client
yt_rabbitmq | 2023-04-22 13:18:52.340062+00:00 [info] <0.689.0> accepting AMQP connection <0.689.0> (172.19.0.7:44350 -> 172.19.0.4:5672)
yt_rabbitmq | 2023-04-22 13:18:52.344179+00:00 [info] <0.689.0> connection <0.689.0> (172.19.0.7:44350 -> 172.19.0.4:5672): user 'guest' authenticated and granted access to vhost '/'
yt_bot | 2023-04-22 13:18:52,586 - [WARNING] - [asyncio:118] - Executing <Task pending name='Task-1' coro=<main() running at /app/main.py:13> wait_for=<Future pending cb=[Task.task_wakeup()] created at /usr/local/lib/python3.11/asyncio/tasks.py:456> cb=[run_until_complete..done_cb()] created at /usr/local/lib/python3.11/asyncio/runners.py:100> took 0.221 seconds
yt_worker | 2023-04-22 13:18:52,690 - [INFO] - [WorkerLauncher:24] - Starting download worker instance
yt_worker | 2023-04-22 13:18:52,691 - [INFO] - [WorkerLauncher:45] - Setting up RabbitMQ connection
yt_worker | 2023-04-22 13:18:52,692 - [INFO] - [WorkerLauncher:54] - Saving current yt-dlp version (2023.03.04) to the database
yt_worker | 2023-04-22 13:18:52,698 - [INFO] - [WorkerLauncher:68] - Creating intermediate directories ('/tmp/download_tmpfs/downloading', '/tmp/download_tmpfs/downloaded') if not exist
yt_rabbitmq | 2023-04-22 13:18:52.705434+00:00 [info] <0.705.0> accepting AMQP connection <0.705.0> (172.19.0.5:33042 -> 172.19.0.4:5672)
yt_rabbitmq | 2023-04-22 13:18:52.753455+00:00 [info] <0.705.0> connection <0.705.0> (172.19.0.5:33042 -> 172.19.0.4:5672): user 'guest' authenticated and granted access to vhost '/'
yt_api | INFO: Started server process [1]
yt_api | INFO: Waiting for application startup.
yt_api | 2023-04-22 13:18:52,824 - [WARNING] - [asyncio:118] - Executing <Task pending name='Task-1' coro=<Server.serve() running at /usr/local/lib/python3.11/site-packages/uvicorn/server.py:76> wait_for=<Future pending cb=[Task.task_wakeup()] created at /usr/local/lib/python3.11/asyncio/locks.py:210> cb=[run_until_complete..done_cb()] created at /usr/local/lib/python3.11/asyncio/runners.py:100> took 1.087 seconds
yt_rabbitmq | 2023-04-22 13:18:52.831485+00:00 [info] <0.721.0> accepting AMQP connection <0.721.0> (172.19.0.6:52698 -> 172.19.0.4:5672)
yt_rabbitmq | 2023-04-22 13:18:52.834539+00:00 [info] <0.721.0> connection <0.721.0> (172.19.0.6:52698 -> 172.19.0.4:5672): user 'guest' authenticated and granted access to vhost '/'
yt_api | INFO: Application startup complete.
yt_api | INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
yt_bot | 2023-04-22 13:18:56,548 - [INFO] - [BotLauncher:82] - Starting "zytdlbot" bot
yt_bot | 2023-04-22 13:18:56,548 - [INFO] - [VideoBot:49] - Sending welcome message
yt_rabbitmq | 2023-04-22 13:18:56.804694+00:00 [info] <0.689.0> closing AMQP connection <0.689.0> (172.19.0.7:44350 -> 172.19.0.4:5672, vhost: '/', user: 'guest')
yt_bot | 2023-04-22 13:18:56,812 - [INFO] - [aio_pika.robust_connection:77] - Connection to amqp://guest:******@yt_rabbitmq:5672/ closed. Reconnecting after 5 seconds.
yt_bot | Traceback (most recent call last):
yt_bot | File "/usr/local/lib/python3.11/site-packages/pyrogram/methods/advanced/resolve_peer.py", line 62, in resolve_peer
yt_bot | return await self.storage.get_peer_by_id(peer_id)
yt_bot | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_bot | File "/usr/local/lib/python3.11/site-packages/pyrogram/storage/sqlite_storage.py", line 142, in get_peer_by_id
yt_bot | raise KeyError(f"ID not found: {peer_id}")
yt_bot | KeyError: 'ID not found: 0'
yt_bot |
yt_bot | During handling of the above exception, another exception occurred:
yt_bot |
yt_bot | Traceback (most recent call last):
yt_bot | File "/app/main.py", line 18, in
yt_bot | asyncio.run(main())
yt_bot | File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
yt_bot | return runner.run(main)
yt_bot | ^^^^^^^^^^^^^^^^
yt_bot | File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
yt_bot | return self._loop.run_until_complete(task)
yt_bot | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_bot | File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
yt_bot | File "/app/main.py", line 13, in main
yt_bot | await BotLauncher().run()
yt_bot | File "/app/bot/core/bot/launcher.py", line 32, in run
yt_bot | await self._start_bot()
yt_bot | File "/app/bot/core/bot/launcher.py", line 83, in _start_bot
yt_bot | await self._bot.send_startup_message()
yt_bot | File "/app/bot/core/bot/bot.py", line 50, in send_startup_message
yt_bot | await self.send_message_to_users(
yt_bot | File "/app/bot/core/bot/bot.py", line 64, in send_message_to_users
yt_bot | await asyncio.gather(*coros)
yt_bot | File "/usr/local/lib/python3.11/site-packages/pyrogram/methods/messages/send_message.py", line 128, in send_message
yt_bot | peer=await self.resolve_peer(chat_id),
yt_bot | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_bot | File "/usr/local/lib/python3.11/site-packages/pyrogram/methods/advanced/resolve_peer.py", line 89, in resolve_peer
yt_bot | peer_type = utils.get_peer_type(peer_id)
yt_bot | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_bot | File "/usr/local/lib/python3.11/site-packages/pyrogram/utils.py", line 246, in get_peer_type
yt_bot | raise ValueError(f"Peer id invalid: {peer_id}")
yt_bot | ValueError: Peer id invalid: 0
yt_bot | 2023-04-22 13:18:56,958 - [ERROR] - [asyncio:1518] - Task was destroyed but it is pending!
yt_bot | source_traceback: Object created at (most recent call last):
yt_bot | File "/app/main.py", line 18, in
yt_bot | asyncio.run(main())
yt_bot | File "/usr/local/lib/python3.11/asyncio/runners.py", line 189, in run
yt_bot | with Runner(debug=debug) as runner:
yt_bot | File "/usr/local/lib/python3.11/asyncio/runners.py", line 63, in exit
yt_bot | self.close()
yt_bot | File "/usr/local/lib/python3.11/asyncio/runners.py", line 72, in close
yt_bot | loop.run_until_complete(loop.shutdown_asyncgens())
yt_bot | File "/usr/local/lib/python3.11/site-packages/aio_pika/tools.py", line 248, in call
yt_bot | self.__task = self.loop.create_task(
yt_bot | task: <Task pending name='Task-139' coro=<OneShotCallback.__task_inner() running at /usr/local/lib/python3.11/site-packages/aio_pika/tools.py:239> wait_for=<_GatheringFuture finished result=[None, None, None, None] created at /usr/local/lib/python3.11/asyncio/tasks.py:699> created at /usr/local/lib/python3.11/site-packages/aio_pika/tools.py:248>
yt_bot | Exception ignored in: <coroutine object OneShotCallback.__task_inner at 0x7fc14e30c130>
yt_bot | Traceback (most recent call last):
yt_bot | File "/usr/local/lib/python3.11/site-packages/aio_pika/tools.py", line 241, in __task_inner
yt_bot | File "uvloop/loop.pyx", line 1281, in uvloop.loop.Loop.call_soon
yt_bot | File "uvloop/loop.pyx", line 667, in uvloop.loop.Loop._call_soon
yt_bot | File "uvloop/loop.pyx", line 676, in uvloop.loop.Loop._call_soon_handle
yt_bot | File "uvloop/loop.pyx", line 671, in uvloop.loop.Loop._append_ready_handle
yt_bot | File "uvloop/loop.pyx", line 703, in uvloop.loop.Loop._check_closed
yt_bot | RuntimeError: Event loop is closed
yt_bot exited with code 0

how to download audio without transcoding?

Since Telegram support most audio formats nowadays such as opus, aac-m4a, ogg... transcoding everything to mp3 seems not necessary.
I would rather avoid a lossy to lossy conversion and get the best quality/size version of the file.

This is the default yt-dlp behavior.

However, i don't understand how should I write it down in the YTDL_OPTS config file.

If i replace FINAL_AUDIO_FORMAT by 'best' I get:
join() argument must be str, bytes, or os.PathLike object, not 'NoneType'

What should I write instead?
Thank you!!

Cannot pull yt-base-image

$ sudo docker compose build --no-cache worker
[+] Building 0.8s (3/3) FINISHED                                                                                                               docker:default
 => [worker internal] load .dockerignore                                                                                                                 0.0s
 => => transferring context: 208B                                                                                                                        0.0s
 => [worker internal] load build definition from Dockerfile                                                                                              0.0s
 => => transferring dockerfile: 409B                                                                                                                     0.0s
 => ERROR [worker internal] load metadata for docker.io/library/yt-base-image:latest                                                                     0.8s
------
 > [worker internal] load metadata for docker.io/library/yt-base-image:latest:
------
failed to solve: yt-base-image: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

Why is there no build context for base-image?

In base-image service in docker-compose, why doesn't base-image have a context?

Wouldn't adding the context make it easy to build and run the services with just:

docker compose up -d --build

Without requiring an extra build step?

Is it possbile to use without docker ?

something like python start.py

ERROR: No matching distribution found for tgcrypto-pyrofork

2.599   Installing build dependencies: started
5.494   Installing build dependencies: finished with status 'done'
5.495   Getting requirements to build wheel: started
6.691   Getting requirements to build wheel: finished with status 'done'
6.696   Preparing metadata (pyproject.toml): started
6.980   Preparing metadata (pyproject.toml): finished with status 'done'
7.012 Collecting pyrofork==2.3.21.post3 (from -r requirements.txt (line 2))
7.016   Downloading pyrofork-2.3.21.post3-py3-none-any.whl.metadata (5.1 kB)
7.031 Collecting tenacity==8.2.3 (from -r requirements.txt (line 3))
7.034   Downloading tenacity-8.2.3-py3-none-any.whl.metadata (1.0 kB)
**7.044 ERROR: Could not find a version that satisfies the requirement tgcrypto-pyrofork==1.2.6 (from versions: none)
7.044 ERROR: No matching distribution found for tgcrypto-pyrofork==1.2.6**
------
failed to solve: process "/bin/sh -c apk add --no-cache --virtual .build-deps         build-base     && MAKEFLAGS=\"-j$(nproc)\" pip install --no-cache-dir -r requirements.txt     && apk --purge del .build-deps" did not complete successfully: exit code: 1

Links not getting rendered.

Links given in groups that has bot as admin are not getting rendered. How can we add bot to different groups if this is not working?

can't running the bot

yt_bot exited with code 1
yt_bot | [RabbitMQ] Waiting to be reachable on port 5672
yt_bot | [PostgreSQL] Waiting to be reachable on port 5432
yt_bot | [RabbitMQ] Connection on port 5672 verified
yt_bot | [PostgreSQL] Connection on port 5432 verified
yt_bot | Traceback (most recent call last):
yt_bot | File "/app/main.py", line 3, in
yt_bot | from api.core.app import app # noqa: F401
yt_bot | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_bot | File "/app/api/core/app.py", line 10, in
yt_bot | from api.core.config import settings
yt_bot | File "/app/api/core/config.py", line 10, in
yt_bot | settings = ApiSettings()
yt_bot | ^^^^^^^^^^^^^
yt_bot | File "/usr/local/lib/python3.11/site-packages/pydantic_settings/main.py", line 71, in init
yt_bot | super().init(
yt_bot | File "/usr/local/lib/python3.11/site-packages/pydantic/main.py", line 171, in init
yt_bot | self.pydantic_validator.validate_python(data, self_instance=self)
yt_bot | pydantic_core.pydantic_core.ValidationError: 3 validation errors for ApiSettings
yt_bot | API_HOST
yt_bot | Field required [type=missing, input_value={'APPLICATION_NAME': 'yt...ADED_DIR': 'downloaded'}, input_type=dict]
yt_bot | For further information visit https://errors.pydantic.dev/2.6/v/missing
yt_bot | API_PORT
yt_bot | Field required [type=missing, input_value={'APPLICATION_NAME': 'yt_...ADED_DIR': 'downloaded'}, input_type=dict]
yt_bot | For further information visit https://errors.pydantic.dev/2.6/v/missing
yt_bot | API_WORKERS
yt_bot | Field required [type=missing, input_value={'APPLICATION_NAME': 'yt_...ADED_DIR': 'downloaded'}, input_type=dict]
yt_bot | For further information visit https://errors.pydantic.dev/2.6/v/missing

Feature-request - Download-Progress to TG-reply-Message

It would be amazing to have feedback of the actual downloadprogress on my Telegram-reply-message.
like
🟩🟩🟩🟩🟩🟩🟩⬜️⬜️⬜️ 73%

The cli shows the progress, but on the TG-Message it would be perfect.

Thanks

Bot should work independently in groups.

By the architecture of bot. We can send only links to the bot and the content is forwarded to a single group. There are use cases where if bot is added to a group then a link posted in the group gets rendered and video is posted in the group. By this functionality we can use bot in many groups simultaneously. I humbly request developer to include this functionality...

Unable to Fetch

I am unable to fetch latest commits.

API_HOST and API_PORT Error

Доброго Дня. Зіткнувся з такою помилкою:

yt_bot | Waiting for RabbitMQ to be reachable on port 5672
yt_bot | Waiting for PostgreSQL to be reachable on port 5432
yt_bot | Connection to RabbitMQ on port 5672 verified
yt_bot | Connection to PostgreSQL on port 5432 verified
yt_bot | Traceback (most recent call last):
yt_bot | File "/app/main.py", line 3, in
yt_bot | from api.core.app import app # noqa: F401
yt_bot | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_bot | File "/app/api/core/app.py", line 10, in
yt_bot | from api.core.config import settings
yt_bot | File "/app/api/core/config.py", line 9, in
yt_bot | settings = ApiSettings()
yt_bot | ^^^^^^^^^^^^^
yt_bot | File "pydantic/env_settings.py", line 39, in pydantic.env_settings.BaseSettings.init
yt_bot | File "pydantic/main.py", line 341, in pydantic.main.BaseModel.init
yt_bot | pydantic.error_wrappers.ValidationError: 2 validation errors for ApiSettings
yt_bot | API_HOST
yt_bot | field required (type=value_error.missing)
yt_bot | API_PORT
yt_bot | field required (type=value_error.missing)

Інші контейнери працюють добре. Пробував через API скачати відео, все супер)
А сам бот не працює. Мабуть, це пов'язано з цією помилкою. Чи можу я це якось виправити? Дякую

[enhancement] Heroku setup support

Hello, can you make a profile for deloy heroku?

Multiple users

Is there a way to allow multiple users to use the bot? Apart from the group method. i want to be able to share the bot with other users and let them DM the bot directly.

error in app_bot/config.yml -> Fix suggested

in app_bot/config.yml there was a line added
"is_admin: !! bool True"

the space between ! and bool causes an error.

Line should lok like this
"is_admin: !!bool True"

asynco problem

for some videos

yt_api | 2023-06-13 07:38:03,319 - [WARNING] - [asyncio:118] - Executing <Task pending name='Task-1' coro=<Server.serve() running at /usr/local/lib/python3.11/site-packages/uvicorn/server.py:81> wait_for=<Future pending cb=[Task.task_wakeup()] created at /usr/local/lib/python3.11/asyncio/tasks.py:634> cb=[run_until_complete..done_cb()] created at /usr/local/lib/python3.11/asyncio/runners.py:100> took 0.126 seconds

Waiting to be reachable

See it. and please help mee

How to start bot

Start bot command

Error on building docker image

ERROR: The Compose file is invalid because:
Service base-image has neither an image nor a build context specified. At least one must be provided.

logfile path

where can i find the log-files within the docker containers?

Feature Request: Upload Audio

Hi,
i really like using your Bot!
Could you add the Option to upload Audio to the Chat ?
I already tried it using the ytdl options, but it seems the bot doesn't want to send two files.
Thanks.

Feature: Filename specification

There's different m3u8 files that will be saved with the same name as "index-1" and this doesn't let me use concurrent downloads cuz the files got replaced... so should be great an argument specification for the Filename to avoid this, like "file1.mp4 " (the extension maybe it's better to be resolved automatically)

pyrogram.errors.exceptions.bad_request_400.MediaCaptionTooLong

the bot went error when error with the video filename is too long.
below is the yt_bot logs output

2023-08-16 18:35:24,658 - [ERROR] - [SuccessHandler:68] - Upload of "/tmp/download_tmpfs/downloaded/dkft/366997176_1010603666742273_4729472621625809870_n.mp4" failed
2023-08-16 18:35:24 Traceback (most recent call last):
2023-08-16 18:35:24   File "/app/bot/core/handlers/success.py", line 61, in _handle
2023-08-16 18:35:24     await self._create_upload_task(media_object)
2023-08-16 18:35:24   File "/app/bot/core/handlers/success.py", line 86, in _create_upload_task
2023-08-16 18:35:24     await create_task(
2023-08-16 18:35:24 pyrogram.errors.exceptions.bad_request_400.MediaCaptionTooLong: Telegram says: [400 MEDIA_CAPTION_TOO_LONG] - The media caption is too long (caused by "messages.SendMedia")
2023-08-16 18:35:24 2023-08-16 18:35:24,662 - [INFO] - [SuccessHandler:73] - Cleaning up task "48f86114-ebb2-4d33-9dd2-3ea8df547a49": removing download content directory "/tmp/download_tmpfs/downloaded/dkft" with files ['366997176_1010603666742273_4729472621625809870_n.mp4-thumb.jpg', '366997176_1010603666742273_4729472621625809870_n.mp4']

⬇️ yt-dlp version: 2023.07.06
🤖 yt-dlp-bot version: 1.4.2

could add "image" option to the "download_media_type" parameter?

I wanna to download pictures from Instagram

[TIKTOK] Requested format is not available. Use --list-formats for a list of available formats

yt_worker    | 2023-11-29 19:32:59,013 - [INFO] - [MediaDownloader:67] - Downloading with options {'format': 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/mp4', 'format_sort': ['res', 'vcodec:h265', 'h264'], 'outtmpl': {'default': '/tmp/download_tmpfs/downloading/tmp_media_dir-l038i6d6/%(title).200B.%(ext)s', 'chapter': '%(title)s - %(section_number)03d %(section_title)s [%(id)s].%(ext)s'}, 'ignoreerrors': True, 'concurrent_fragment_downloads': 5, 'noplaylist': True, 'writethumbnail': True, 'verbose': True, 'cookiefile': '/app/cookies/cookies.txt', 'postprocessors': [{'key': 'FFmpegThumbnailsConvertor', 'format': 'jpg', 'when': 'before_dl'}], 'playlist_items': '1:1', 'compat_opts': set(), 'http_headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.20 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en-us,en;q=0.5', 'Sec-Fetch-Mode': 'navigate'}, 'forceprint': {}, 'print_to_file': {}}
yt_worker    | [vm.tiktok] Extracting URL: https://vm.tiktok.com/ZGedbPv6h/
yt_worker    | [vm.tiktok] ZGedbPv6h: Downloading webpage
yt_worker    | [TikTok] Extracting URL: https://www.tiktok.com/@nandecito/video/7306186198375075105?_d=secCgYIASAHKAESPgo8%2FlqoRN%2BuAPlwdsAVgyfQmqHJXZq9kMIVOz7WBpCCduFzZbdkr5412%2BtUA8t4Sfb1w5cJCt46KYbUrRd2GgA%3D&_r=1&aweme_type=150&checksum=5b3265fca0236a958973ed36303b1481b474696189e7ea006350d4e2dadf7f72&pic_cnt=5&preview_pb=0&sec_user_id=MS4wLjABAAAALmwkwg00XwcSzGzqEiJiZyU0dH-ALVeouNZU5L0J8ffRwgnErDpMbXEZ1U9sZML3&share_app_id=1233&share_item_id=7306186198375075105&share_link_id=4c0b59eb-28c3-4801-a4d0-5a870dcce0ab&sharer_language=es&social_share_type=0&source=h5_m&timestamp=1701286200&u_code=e21k4193emf719&ug_btm=b2878%2Cb2001&ugbiz_name=UNKNOWN&user_id=7102126970062390278&utm_campaign=client_share&utm_medium=android&utm_source=telegram
yt_worker    | [TikTok] 7306186198375075105: Downloading video feed
yt_worker    | Deprecated Feature: Using arbitrary fields (h264) for format sorting is deprecated and may be removed in a future version
yt_worker    | [debug] Sort order given by user: res, vcodec:h265, h264
yt_worker    | [debug] Sort order given by extractor: quality, codec, size, br
yt_worker    | [debug] Formats sorted by: hasvid, ie_pref, res, vcodec:h265(8), h264, quality, acodec, size, br, lang, fps, hdr:12(7), channels, asr, proto, vext, aext, hasaud, source, id
yt_worker    | ERROR: [TikTok] 7306186198375075105: Requested format is not available. Use --list-formats for a list of available formats
yt_worker    | Traceback (most recent call last):
yt_worker    |   File "/usr/local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 1567, in wrapper
yt_worker    |     return func(self, *args, **kwargs)
yt_worker    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_worker    |   File "/usr/local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 1723, in __extract_info
yt_worker    |     return self.process_ie_result(ie_result, download, extra_info)
yt_worker    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_worker    |   File "/usr/local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 1782, in process_ie_result
yt_worker    |     ie_result = self.process_video_result(ie_result, download=download)
yt_worker    |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_worker    |   File "/usr/local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 2884, in process_video_result
yt_worker    |     raise ExtractorError(
yt_worker    | yt_dlp.utils.ExtractorError: [TikTok] 7306186198375075105: Requested format is not available. Use --list-formats for a list of available formats
yt_worker    |
yt_worker    | 2023-11-29 19:33:01,412 - [ERROR] - [MediaDownloader:72] - Error during media download. Check logs. URL: "https://vm.tiktok.com/ZGedbPv6h/". Meta: None
yt_worker    | 2023-11-29 19:33:01,414 - [ERROR] - [MediaDownloader:46] - Failed to download https://vm.tiktok.com/ZGedbPv6h/
yt_worker    | 2023-11-29 19:33:01,415 - [ERROR] - [MediaService:91] - Failed to download media. Context: id=None from_chat_id=X from_chat_type=<TelegramChatType.SUPERGROUP: 'supergroup'> from_user_id=X message_id=67557 ack_message_id=67558 url='https://vm.tiktok.com/ZGedbPv6h/' original_url='https://vm.tiktok.com/ZGedbPv6h/' source=<TaskSource.BOT: 'BOT'> save_to_storage=False download_media_type=<DownMediaType.VIDEO: 'VIDEO'> added_at=datetime.datetime(2023, 11, 29, 19, 32, 58, 673548, tzinfo=TzInfo(UTC))
yt_worker    | Traceback (most recent call last):
yt_worker    |   File "/app/worker/core/media_service.py", line 83, in _start_download
yt_worker    |     return await asyncio.get_running_loop().run_in_executor(
yt_worker    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_worker    |   File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
yt_worker    |     result = self.fn(*self.args, **self.kwargs)
yt_worker    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_worker    |   File "/app/worker/core/media_service.py", line 85, in <lambda>
yt_worker    |     lambda: self._downloader.download(
yt_worker    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_worker    |   File "/app/worker/core/downloader.py", line 44, in download
yt_worker    |     return self._download(host_conf=host_conf, media_type=media_type)
yt_worker    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_worker    |   File "/app/worker/core/downloader.py", line 73, in _download
yt_worker    |     raise MediaDownloaderError(err_msg)
yt_worker    | worker.core.exceptions.MediaDownloaderError: Error during media download. Check logs. URL: "https://vm.tiktok.com/ZGedbPv6h/"

Cannot get bot up and running

Looks like there are some pydantic errors on the repo. Maybe can you freeze your python requirements again so we get correct versions of the packages we need to install.

Here is a stacktrack from the docker logs

yt_bot       | Waiting for RabbitMQ to be reachable on port 5672
yt_bot       | Waiting for PostgreSQL to be reachable on port 5432
yt_bot       | Connection to PostgreSQL on port 5432 verified
yt_bot       | Connection to RabbitMQ on port 5672 verified
yt_bot       | Traceback (most recent call last):
yt_bot       |   File "/app/main.py", line 7, in <module>
yt_bot       |     from bot.core.bot import BotLauncher
yt_bot       |   File "/app/bot/core/bot/__init__.py", line 1, in <module>
yt_bot       |     from bot.core.bot.bot import VideoBot
yt_bot       |   File "/app/bot/core/bot/bot.py", line 8, in <module>
yt_bot       |     from bot.core.config.config import get_main_config
yt_bot       |   File "/app/bot/core/config/__init__.py", line 1, in <module>
yt_bot       |     from bot.core.config.config import settings
yt_bot       |   File "/app/bot/core/config/config.py", line 9, in <module>
yt_bot       |     from yt_shared.config import Settings
yt_bot       |   File "/app/yt_shared/yt_shared/config.py", line 4, in <module>
yt_bot       |     from pydantic import BaseSettings, validator
yt_bot       |   File "/usr/local/lib/python3.11/site-packages/pydantic/__init__.py", line 206, in __getattr__
yt_bot       |     return _getattr_migration(attr_name)
yt_bot       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_bot       |   File "/usr/local/lib/python3.11/site-packages/pydantic/_migration.py", line 279, in wrapper
yt_bot       |     raise PydanticImportError(
yt_bot       | pydantic.errors.PydanticImportError: `BaseSettings` has been moved to the `pydantic-settings` package. See https://docs.pydantic.dev/2.0/migration/#basesettings-has-moved-to-pydantic-settings for more details.
yt_bot       |
yt_bot       | For further information visit https://errors.pydantic.dev/2.0/u/import-error

Another one

yt_api       | Waiting for RabbitMQ to be reachable on port 5672
yt_api       | Waiting for PostgreSQL to be reachable on port 5432
yt_api       | Connection to RabbitMQ on port 5672 verified
yt_api       | Connection to PostgreSQL on port 5432 verified
yt_api       | Executing <Task finished name='Task-1' coro=<Server.serve() done, defined at /usr/local/lib/python3.11/site-packages/uvicorn/server.py:63> exception=ImportError("cannot import name 'Undefined' from 'pydantic.fields' (/usr/local/lib/python3.11/site-packages/pydantic/fields.py)") created at /usr/local/lib/python3.11/asyncio/runners.py:100> took 0.104 seconds
yt_api       | Traceback (most recent call last):
yt_api       |   File "/usr/local/bin/uvicorn", line 8, in <module>
yt_api       |     sys.exit(main())
yt_api       |              ^^^^^^
yt_api       |   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
yt_api       |     return self.main(*args, **kwargs)
yt_api       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_api       |   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1055, in main
yt_api       |     rv = self.invoke(ctx)
yt_api       |          ^^^^^^^^^^^^^^^^
yt_api       |   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
yt_api       |     return ctx.invoke(self.callback, **ctx.params)
yt_api       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_api       |   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 760, in invoke
yt_api       |     return __callback(*args, **kwargs)
yt_api       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_api       |   File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 410, in main
yt_api       |     run(
yt_api       |   File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 578, in run
yt_api       |     server.run()
yt_api       |   File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 61, in run
yt_api       |     return asyncio.run(self.serve(sockets=sockets))
yt_api       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_api       |   File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
yt_api       |     return runner.run(main)
yt_api       |            ^^^^^^^^^^^^^^^^
yt_api       |   File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
yt_api       |     return self._loop.run_until_complete(task)
yt_api       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_api       |   File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
yt_api       |   File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 68, in serve
yt_api       |     config.load()
yt_api       |   File "/usr/local/lib/python3.11/site-packages/uvicorn/config.py", line 473, in load
yt_api       |     self.loaded_app = import_from_string(self.app)
yt_api       |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_api       |   File "/usr/local/lib/python3.11/site-packages/uvicorn/importer.py", line 24, in import_from_string
yt_api       |     raise exc from None
yt_api       |   File "/usr/local/lib/python3.11/site-packages/uvicorn/importer.py", line 21, in import_from_string
yt_api       |     module = importlib.import_module(module_str)
yt_api       |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_api       |   File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
yt_api       |     return _bootstrap._gcd_import(name[level:], package, level)
yt_api       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_api       |   File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
yt_api       |   File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
yt_api       |   File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
yt_api       |   File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
yt_api       |   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
yt_api       |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
yt_api       |   File "/app/main.py", line 3, in <module>
yt_api       |     from api.core.app import app  # noqa: F401
yt_api       |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_api       |   File "/app/api/core/app.py", line 1, in <module>
yt_api       |     from fastapi import FastAPI
yt_api       |   File "/usr/local/lib/python3.11/site-packages/fastapi/__init__.py", line 7, in <module>
yt_api       |     from .applications import FastAPI as FastAPI
yt_api       |   File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 16, in <module>
yt_api       |     from fastapi import routing
yt_api       |   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 22, in <module>
yt_api       |     from fastapi import params
yt_api       |   File "/usr/local/lib/python3.11/site-packages/fastapi/params.py", line 5, in <module>
yt_api       |     from pydantic.fields import FieldInfo, Undefined
yt_api       | ImportError: cannot import name 'Undefined' from 'pydantic.fields' (/usr/local/lib/python3.11/site-packages/pydantic/fields.py)

Another one

yt_worker    | Waiting for RabbitMQ to be reachable on port 5672
yt_worker    | Waiting for PostgreSQL to be reachable on port 5432
yt_worker    | Connection to RabbitMQ on port 5672 verified
yt_worker    | Connection to PostgreSQL on port 5432 verified
yt_worker    | Traceback (most recent call last):
yt_worker    |   File "/usr/local/bin/alembic", line 8, in <module>
yt_worker    |     sys.exit(main())
yt_worker    |              ^^^^^^
yt_worker    |   File "/usr/local/lib/python3.11/site-packages/alembic/config.py", line 632, in main
yt_worker    |     CommandLine(prog=prog).main(argv=argv)
yt_worker    |   File "/usr/local/lib/python3.11/site-packages/alembic/config.py", line 626, in main
yt_worker    |     self.run_cmd(cfg, options)
yt_worker    |   File "/usr/local/lib/python3.11/site-packages/alembic/config.py", line 603, in run_cmd
yt_worker    |     fn(
yt_worker    |   File "/usr/local/lib/python3.11/site-packages/alembic/command.py", line 385, in upgrade
yt_worker    |     script.run_env()
yt_worker    |   File "/usr/local/lib/python3.11/site-packages/alembic/script/base.py", line 582, in run_env
yt_worker    |     util.load_python_file(self.dir, "env.py")
yt_worker    |   File "/usr/local/lib/python3.11/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
yt_worker    |     module = load_module_py(module_id, path)
yt_worker    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_worker    |   File "/usr/local/lib/python3.11/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
yt_worker    |     spec.loader.exec_module(module)  # type: ignore
yt_worker    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_worker    |   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
yt_worker    |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
yt_worker    |   File "/app/alembic/env.py", line 9, in <module>
yt_worker    |     from yt_shared.config import settings
yt_worker    |   File "/app/yt_shared/yt_shared/config.py", line 4, in <module>
yt_worker    |     from pydantic import BaseSettings, validator
yt_worker    |   File "/usr/local/lib/python3.11/site-packages/pydantic/__init__.py", line 206, in __getattr__
yt_worker    |     return _getattr_migration(attr_name)
yt_worker    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_worker    |   File "/usr/local/lib/python3.11/site-packages/pydantic/_migration.py", line 279, in wrapper
yt_worker    |     raise PydanticImportError(
yt_worker    | pydantic.errors.PydanticImportError: `BaseSettings` has been moved to the `pydantic-settings` package. See https://docs.pydantic.dev/2.0/migration/#basesettings-has-moved-to-pydantic-settings for more details.
yt_worker    |
yt_worker    | For further information visit https://errors.pydantic.dev/2.0/u/import-error

How to rebuild?

I am adjusting the config.yml but changes are not reflecting, even after re-attempting the run commands:

# Build base image
docker compose build base-image

# Build and run all services in detached mode
docker compose up --build -d -t 0 && docker compose logs --tail 100 -f

# Stop all services
docker compose stop -t 0

What am I missing to use the newest config.yml?

Work From within a group?

Is there any way to get it to work from within a group?

For example I could setup a group with some friends and they can just post a link in the group and it downloads it for them rather than having to set each friend up manually?

Video has been uploaded two times on a single request

I've got an issue. Video is getting uploaded two times. I've commented out the group id at api part but it's still the same...

how to run only api without bot

I just want to use yt-dlp api without telegram bot how to run service without telegram env config

Error on Implementing Uploads to Channel

This is the error that is firing up when trying to implement uploads in telegram channel. Kindly resolve it...
yt_bot | 2023-10-04 08:40:57,530 - [ERROR] - [pyrogram.dispatcher:253] - 'NoneType' object has no attribute 'id'
yt_bot | Traceback (most recent call last):
yt_bot | File "/usr/local/lib/python3.11/site-packages/pyrogram/dispatcher.py", line 240, in handler_worker
yt_bot | await handler.callback(self.client, *args)
yt_bot | File "/app/bot/core/callbacks.py", line 42, in on_message
yt_bot | urls = self._url_parser.parse_urls(urls=urls, message=message, user=user)
yt_bot | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yt_bot | File "/app/bot/core/service.py", line 46, in parse_urls
yt_bot | return [
yt_bot | ^
yt_bot | File "/app/bot/core/service.py", line 51, in
yt_bot | from_user_id=message.from_user.id,
yt_bot | ^^^^^^^^^^^^^^^^^^^^
yt_bot | AttributeError: 'NoneType' object has no attribute 'id'

Ease of Operation

Modifying YAML file every time is cumbersome. It's better to have a UI with backend Database tables that stores id's and values of Group ID's to be passed as variables to YAML file, may be a user friendly scenario. Can we expect any enhancement features in future scenarios!?

Download aborts and error causes lost Data

I sent https://www.youtube.com/watch?v=yT4A47IFu2E to my telegram-bot.

But it was not able to download the complete File and got following Error-Message:

`🛑 Download error

ℹ️ Task ID: c3e51785-85eb-4f2a-bb83-0aa068a8712f
💬 Message: Download error
📹 Video URL: https://www.youtube.com/watch?v=yT4A47IFu2E
🌊 Source: BOT
👀 Details: 'NoneType' object is not subscriptable
⬇️ yt-dlp version: 2023.11.16
🤖 yt-dlp-bot version: 1.4.4
🏷️ Tag: #error
`

After having a look at the Log-files and into the shared-tmpfs folder, I assume that the Download aborted and restartet itself again.
The content in shared-tmpfs/downloading was empty, but in shared-tmpfs/downloaded were 11 generated folders (4 characters long) with timestamps from 03:51 to 08:42.

the 4char-Folders were partly empty and partly filled with the videofile and the thumbnail. The Videos were never fully downloaded but everytime only partly downloaded with filesizes from 11197MB, 11142MB, 11127MB, 11337MB, 11248MB, 11161MB, 11320MB.
So a huge part was downloaded but then dhe download aborted.

My problem is, that the folders were not deleted, so there are a lot of lost files lying around
and I think if the Download aborts, it is possible to restart the download into the last destinationfolder and the download resumes automatically. That seems not to be the case. Am I right?

It also seems that the downloads of the same file are running in parallel. the following CLI-log let me assume that.
Percentages are jumping around.

[download] 48.1% of ~ 10.78GiB at 1.23MiB/s ETA 01:17:46 (frag 461/961) [download] 0.9% of ~ 10.83GiB at 1.46MiB/s ETA 02:10:45 (frag 8/961)1) [download] 1.0% of ~ 11.51GiB at 1.45MiB/s ETA 02:16:33 (frag 463/961)frag 462/961) [download] 27.6% of ~ 10.76GiB at 1.16MiB/s ETA 01:45:51 (frag 264/961) [download] 27.6% of ~ 10.82GiB at 1.65MiB/s ETA 59:10 (frag 466/961)61) [download] 48.8% of ~ 10.80GiB at 1.54MiB/s ETA 01:01:03 (frag 468/961) [download] 27.8% of ~ 10.87GiB at 1.61MiB/s ETA 01:21:47 (frag 266/961)61) [download] 28.1% of ~ 10.81GiB at 1.43MiB/s ETA 01:31:29 (frag 269/961) [download] 28.3% of ~ 10.78GiB at 1.47MiB/s ETA 01:29:28 (frag 271/961) [download] 49.3% of ~ 10.82GiB at 1.72MiB/s ETA 01:19:06 (frag 273/961)1) [download] 9.4% of ~ 10.84GiB at 1.48MiB/s ETA 01:04:25 (frag 474/961) [download] 28.7% of ~ 10.81GiB at 1.56MiB/s ETA 01:25:09 (frag 275/961)) [download] 49.8% of ~ 10.81GiB at 1.22MiB/s ETA 01:16:07 (frag 278/961)61) [download] 29.0% of ~ 10.82GiB at 1.42MiB/s ETA 01:04:37 (frag 478/961) [download] 29.2% of ~ 10.80GiB at 1.46MiB/s ETA 01:29:59 (frag 280/961) [download] 2.9% of ~ 11.52GiB at 1.33MiB/s ETA 02:21:36rag 482/961)61)

in the CLI-log there are also following entries
[download] Got error: HTTPSConnectionPool(host='rr1---sn-n5a1c0gb-n1bl.googlevideo.com', port=443): Read timed out. [download] Got error: HTTPSConnectionPool(host='rr1---sn-n5a1c0gb-n1bl.googlevideo.com', port=443): Read timed out. [download] Got error: HTTPSConnectionPool(host='rr1---sn-n5a1c0gb-n1bl.googlevideo.com', port=443): Read timed out. [download] Got error: HTTPSConnectionPool(host='rr1---sn-n5a1c0gb-n1bl.googlevideo.com', port=443): Read timed out. [download] Got error: HTTPSConnectionPool(host='rr1---sn-n5a1c0gb-n1bl.googlevideo.com', port=443): Read timed out. [download] Got error: HTTPSConnectionPool(host='rr1---sn-n5a1c0gb-n1bl.googlevideo.com', port=443): Read timed out. [download] Got error: HTTPSConnectionPool(host='rr1---sn-n5a1c0gb-n1bl.googlevideo.com', port=443): Read timed out.
maybe google stops answering because of the amount of same downloads?

for this downloadlink the error in my case is reproduceable every time.

Thanks

Forward To Group and Chat Id's not working

I have deployed bot on my Windows machine ;bot is working; video is uploaded to only chat in bot "Forward to Group", "Forward to chat Id's " are not working;
And allowed users only one id is allowed
At end of every video this error persists every time...

🛑 General error

ℹ Task ID: a4f4a540-c3eb-4ac7-80de-eb8ae0077456
💬 Message: Upload error
📹 Video URL: https://www.youtube.com/watch?v=ZYaZ6Odbx_Y&pp=ygUPYWwgZmF0aWhhIHN1cmFo
🌊 Source: BOT
👀 Details: Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/pyrogram/methods/advanced/resolve_peer.py", line 62, in resolve_peer
return await self.storage.get_peer_by_id(peer_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyrogram/storage/sqlite_storage.py", line 142, in get_peer_by_id
raise KeyError(f"ID not found: {peer_id}")
KeyError: 'ID not found: -1001812523689'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/app/bot/core/handlers/success.py", line 61, in _handle
await self._create_upload_task(media_object)
File "/app/bot/core/handlers/success.py", line 86, in _create_upload_task
await create_task(
pyrogram.errors.exceptions.bad_request_400.ChannelInvalid: Telegram says: [400 CHANNEL_INVALID] - The channel parameter is invalid (caused by "channels.GetChannels")

⬇️ yt-dlp version: 2023.07.06
🤖 yt-dlp-bot version: 1.4.1
🏷️ Tag: #error

Why pyrogram?

Hi,
why are you using pyrogram that is not maintained since a year and it doesn't support catch up (aka you loose messages when the bot is offline)?

Cookie option error

I'm having an issue setting up cookies (due to the new twitter rules you can't view tweets without being logged in, so yt-dlp can't download anything)

I set the following in app_worker/ytdl_opts/user.py

Only CLI options are allowed to be stored as configuration. They are later converted to internal API options.
More here https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/options.py or 'yt-dlp --help'

Copy all content from the `default.py` here and edit values.
"""

FINAL_AUDIO_FORMAT = 'mp3'
FINAL_THUMBNAIL_FORMAT = 'jpg'

DEFAULT_YTDL_OPTS = [
    '--output',
    '%(title).200B.%(ext)s',
    '--no-playlist',
    '--playlist-items',
    '1:1',
    '--concurrent-fragments',
    '5',
    '--ignore-errors',
    '--verbose',
    '--cookies /config/cookies.txt'
]

AUDIO_YTDL_OPTS = [
    '--extract-audio',
    '--audio-quality',
    '0',
    '--audio-format',
    FINAL_AUDIO_FORMAT,
]

AUDIO_FORMAT_YTDL_OPTS = [
    '--format',
    'bestaudio/best',
]

VIDEO_YTDL_OPTS = [
    '--format',
    'bestvideo[ext=mp4]+bestaudio[ext=m4a]/mp4',
    '--write-thumbnail',
    '--convert-thumbnails',
    FINAL_THUMBNAIL_FORMAT,
]

as you can see here I used the correct option https://github.com/yt-dlp/yt-dlp#filesystem-options

added ./config:/config to the yt_worker volumes in docker-compose.yml (I can confirm it's being binded as i went inside the container to check)

I have also tried the following

--cookiefile
--cookie-file

but whenever I try to download something now I get the following error..

Any help or point in the right direction would be appreciated!

"❓ Did you send anything?" [TelegramCallback:34] - Received empty text

Every time somebody sends a picture with a caption we get this message in chat.
"❓ Did you send anything?"

In config:

  url_validation_regexes:
    - "^https?:\\/\\/(?:www\\.)?twitter\\.com\\/.*$"
    - "^https?:\\/\\/(?:www\\.)?youtube\\.com\\/.*$"
    - "^https?:\\/\\/youtu\\.be\\/.*$"
    - "^https?:\\/\\/(?:www\\.)?reddit\\.com\\/.*$"
    - "^https?:\\/\\/(?:www\\.)?tiktok\\.com\\/.*$"
    - "^https?:\\/\\/(?:www\\.)?x\\.com\\/.*$"
    - "^https?:\\/\\/(?:www\\.)?instagram\\.com\\/.*$"

Logs:

yt_bot       | 2023-11-19 18:13:01,177 - [WARNING] - [TelegramCallback:34] - Received empty text: {
yt_bot       |     "_": "Message",
yt_bot       |     "id": * redacted *,
yt_bot       |     "from_user": {
yt_bot       |         "_": "User",
yt_bot       |         "id": * redacted *,
yt_bot       |         "is_self": false,
yt_bot       |         "is_contact": false,
yt_bot       |         "is_mutual_contact": false,
yt_bot       |         "is_deleted": false,
yt_bot       |         "is_bot": false,
yt_bot       |         "is_verified": false,
yt_bot       |         "is_restricted": false,
yt_bot       |         "is_scam": false,
yt_bot       |         "is_fake": false,
yt_bot       |         "is_support": false,
yt_bot       |         "is_premium": false,
yt_bot       |         "first_name": "* redacted *",
yt_bot       |         "status": "UserStatus.RECENTLY",
yt_bot       |         "username": "* redacted *",
yt_bot       |         "language_code": "es",
yt_bot       |         "dc_id": 4,
yt_bot       |         "photo": {
yt_bot       |             "_": "ChatPhoto",
yt_bot       |             "small_file_id": "AQADBAADOsAxGxfSzgARAgADF9LOAAU-K6tEQVXvBwAEHgQ",
yt_bot       |             "small_photo_unique_id": "AgADOsAxGxfSzgAB",
yt_bot       |             "big_file_id": "AQADBAADOsAxGxfSzgARAwADF9LOAAU-K6tEQVXvBwAEHgQ",
yt_bot       |             "big_photo_unique_id": "AgADOsAxGxfSzgAB"
yt_bot       |         }
yt_bot       |     },
yt_bot       |     "date": "2023-11-19 18:13:01",
yt_bot       |     "chat": {
yt_bot       |         "_": "Chat",
yt_bot       |         "id": * redacted *,
yt_bot       |         "type": "ChatType.SUPERGROUP",
yt_bot       |         "is_verified": false,
yt_bot       |         "is_restricted": false,
yt_bot       |         "is_creator": false,
yt_bot       |         "is_scam": false,
yt_bot       |         "is_fake": false,
yt_bot       |         "title": "* redacted *",
yt_bot       |         "photo": {
yt_bot       |             "_": "ChatPhoto",
yt_bot       |             "small_file_id": "AQADBAADYL4xG1ICqFEAEAIAA0LnT9MW____M_yvJ383XdUABB4E",
yt_bot       |             "small_photo_unique_id": "AgADYL4xG1ICqFE",
yt_bot       |             "big_file_id": "AQADBAADYL4xG1ICqFEAEAMAA0LnT9MW____M_yvJ383XdUABB4E",
yt_bot       |             "big_photo_unique_id": "AgADYL4xG1ICqFE"
yt_bot       |         },
yt_bot       |         "dc_id": 4,
yt_bot       |         "has_protected_content": false,
yt_bot       |         "permissions": {
yt_bot       |             "_": "ChatPermissions",
yt_bot       |             "can_send_messages": true,
yt_bot       |             "can_send_media_messages": true,
yt_bot       |             "can_send_other_messages": true,
yt_bot       |             "can_send_polls": true,
yt_bot       |             "can_add_web_page_previews": true,
yt_bot       |             "can_change_info": true,
yt_bot       |             "can_invite_users": true,
yt_bot       |             "can_pin_messages": true
yt_bot       |         }
yt_bot       |     },
yt_bot       |     "mentioned": false,
yt_bot       |     "scheduled": false,
yt_bot       |     "from_scheduled": false,
yt_bot       |     "media": "MessageMediaType.PHOTO",
yt_bot       |     "has_protected_content": false,
yt_bot       |     "has_media_spoiler": false,
yt_bot       |     "photo": {
yt_bot       |         "_": "Photo",
yt_bot       |         "file_id": "AgACAgQAAx0CWAsIvgABAQGDZVpQLa3LGFHObeXs1GFTBfvg8rIAAsq_MRtxiNhScCMLmuK7NawACAEAAwIAA3gABx4E",
yt_bot       |         "file_unique_id": "AgADyr8xG3GI2FI",
yt_bot       |         "width": 576,
yt_bot       |         "height": 680,
yt_bot       |         "file_size": 58204,
yt_bot       |         "date": "2023-11-19 17:52:19",
yt_bot       |         "thumbs": [
yt_bot       |             {
yt_bot       |                 "_": "Thumbnail",
yt_bot       |                 "file_id": "AgACAgQAAx0CWAsIvgABAQGDZVpQLa3LGFHObeXs1GFTBfvg8rIAAsq_MRtxiNhScCMLmuK7NawACAEAAwIAA3MABx4E",
yt_bot       |                 "file_unique_id": "AgADyr8xG3GI2FI",
yt_bot       |                 "width": 76,
yt_bot       |                 "height": 90,
yt_bot       |                 "file_size": 2073
yt_bot       |             }
yt_bot       |         ]
yt_bot       |     },
yt_bot       |     "caption": "defadsfs",
yt_bot       |     "outgoing": false,
yt_bot       |     "matches": [
yt_bot       |         "<re.Match object; span=(0, 1), match='d'>"
yt_bot       |     ]
yt_bot       | }

Feature Request: enable customization of '--format-sort'

In the latest updates, video codec option is freeze to h265, h264 at app_worker/ytdl_opts/per_host/_base.py. Usually YouTube provides better video quality on vp9(.webm) or av1 on high resolution pieces, while some user's device may not be compatible with h265. Therefore I believe the '--format-sort' is reasonable to be customizable on the user side.

Sanitize Title

Hi, depending on the title of the YouTube video, it will change the formatting.

For example: https://www.youtube.com/watch?v=mfhBM_Yay6w

Title **Original** The Dramatic Chipmunk **Original** becomes "Original The Dramatic Chipmunk Original"

how to hide the notice of new version ?

New yt-dlp version available: 2023.11.16
Current version: 2023.10.13
Rebuild worker with docker compose build --no-cache worker

hi,Boss
erveryday,at the same time,my telegram bot receive a notice of new version，if i dont update。
i want to know how to hide the notification that reminds me once a day？
thank you.

Another bot in same server

I need another bot with different name in same server. Kindly place a guide to run two bots simultaneously...

Error building docker image

When I try to build docker image I get following error:

sudo docker compose build base-image
[+] Building 11.6s (6/9)                                                                                                                                                   
 => [base-image internal] load build definition from base.Dockerfile                                                                                                  0.0s
 => => transferring dockerfile: 496B                                                                                                                                  0.0s
 => [base-image internal] load .dockerignore                                                                                                                          0.0s
 => => transferring context: 222B                                                                                                                                     0.0s
 => [base-image internal] load metadata for docker.io/library/python:3.11-alpine                                                                                      0.8s
 => CACHED [base-image 1/5] FROM docker.io/library/python:3.11-alpine@sha256:2c0e25da660a20c1e99ccd091a3fdda4c6e3525d91fe5306d89cc24b54fc6b95                         0.0s
 => [base-image internal] load build context                                                                                                                          0.0s
 => => transferring context: 84B                                                                                                                                      0.0s
 => ERROR [base-image 2/5] RUN apk add --no-cache         tzdata         htop         bash         libstdc++                                                         10.7s
------                                                                                                                                                                     
 > [base-image 2/5] RUN apk add --no-cache         tzdata         htop         bash         libstdc++:                                                                     
0.619 fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz                                                                                        
5.622 WARNING: fetching https://dl-cdn.alpinelinux.org/alpine/v3.19/main: temporary error (try again later)                                                                
5.623 fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz                                                                                   
10.63 WARNING: fetching https://dl-cdn.alpinelinux.org/alpine/v3.19/community: temporary error (try again later)                                                           
10.63 ERROR: unable to select packages:
10.63   bash (no such package):
10.63     required by: world[bash]
10.63   htop (no such package):
10.63     required by: world[htop]
10.63   libstdc++ (no such package):
10.63     required by: world[libstdc++]
------
failed to solve: process "/bin/sh -c apk add --no-cache         tzdata         htop         bash         libstdc++" did not complete successfully: exit code: 3

How can I solve this?

download youtube shows "Unknow command"

windows docker v4.28.0, WSL2

just send the URL
ex: https://youtu.be/XHTrLYShBRQ
and you see the yt_bot log with below error message

2024-03-22 17:14:32 Traceback (most recent call last):
2024-03-22 17:14:32 File "/usr/local/lib/python3.11/site-packages/pyrogram/dispatcher.py", line 268, in handler_worker
2024-03-22 17:14:32 await handler.callback(self.client, *args)
2024-03-22 17:14:32 File "/usr/local/lib/python3.11/site-packages/pyrogram/handlers/message_handler.py", line 151, in resolve_future_or_callback
2024-03-22 17:14:32 await self.original_callback(client, message, *args)
2024-03-22 17:14:32 File "/app/bot/core/callbacks.py", line 38, in on_message
2024-03-22 17:14:32 user = client.allowed_users[get_user_id(message)]
2024-03-22 17:14:32 ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
2024-03-22 17:14:32 KeyError: 6053554717

bot responds: "Unknow command"

im using the zip for the commit: 20c35b0

yt-dlp is only downloading the first video from the playlist

How to change user.py settings, to download entire playlist?

Build base image error

As I followed step 1-10 in the quick setup guide and ready all my files, I tried to build base image but I got error "please specify build context (e.g. "." for the current directory)"

searched in the issues, and docker or docker-compose command both get the same error.

Is it possible support rclone backup?