Comments (13)
I am currently working on a (pair of) script(s) that will perform the migration in a somewhat automated manner. This will include all videos, if they're watched and anything that's in the version 1 database. The first script is run with version 1 installed, and the second with v2 due to python not being able to handle importing two modules with the same name simultaneously.
from ytcc.
First approach depends on #43
from ytcc.
First approach has been implemented in 1da677b and 0d03bac
from ytcc.
I'm going to leave this issue open for now. Hoping to get more feedback.
from ytcc.
I'm currently trying to work out how to calculate the extractor_hash
from a video URL, but I'm not getting very far. Could you offer some help on this? This is the only thing (I think) holding back the entire script
from ytcc.
@EmRowlands Awesome!!
The extractor_hash is calculated as a sha256 of youtube-dl's (unprocessed) information extractor output.
Basically pseudocode for one item:
sha256(YoutubeDL(...).extract_info(..., process=False).entries[0])
Relevant lines:
Line 50 in d97eebd
Line 84 in d97eebd
from ytcc.
And important here is that the hashed entry is from a playlist, not from the video page itself. Not sure, if we can reverse it from a yt_video_id easily, which is probably what we would need when converting from v1?
from ytcc.
I knew how it was being created, I just couldn't reproduce it because I didn't have access to the playlist data. I'm also not sure this way of generating the hash makes sense, since if a video is in multiple playlists it will have multiple extractor_hash
es (unless this is intentional). I considered suggesting using the same method, but using the extractor info for the specific video instead, but that would require youtube-dl
ing the info for every single video that is being imported.
Perhaps it would be better to use something like the format provided by --download-archive
, which provides strings that look like this:
youtube dQw4w9WgXcQ
Where the first part is the name of the extractor, and the second is a site-specific string that uniquely identifies a video.
from ytcc.
Admittedly, the extractor_hash
approach has problems. I actually found cases where it won't work with the current function that relies on a Dict[str, str], which is not always the output of processors. Sometimes the values might be more complex structures.
The hash should be the same for videos of different playlists. At least, for all examples that I checked it was the same.
I have looked into the --download-archive option again. It uses _make_archive_id to create the id, which can be generated from an unprocessed result and therefore does not require more network requests than the current approach. I think using _make_archive_id is more reliable, because then we rely on existing youtube_dl internals, which should work nicer with the rest of youtube_dl.
It is possible to replace the extractor_hash()
with _make_archive_id()
. Ytcc will simply resync all playlist content on the next update. I'll commit my changes and release a second beta soon.
from ytcc.
I've done some testing, and it appears that this approach will work with my scripts in a drop-in way. Since all of the videos from v1 will be from youtube, it's trivial to reimplement _make_archive_id()
to not require network access. I'm also not sure how to contribute these scripts, since they require v1 to be installed, with v2 code sitting in a different directory (or vice-versa for the price of a trivial change)
from ytcc.
I have finished my implementation, but there are some caveats:
- It requires version 1 to be installed
- It requires a copy of the source code of version 2
- It exists entirely "out of tree"
As such, I'm not really sure how to submit it for review. It could sit in the scripts
directory if it was only a single file, but it includes 5 files (a common file, an export and import script, a config file, and a migration shell script which runs them all).
If you think it would be acceptable to put them in a subdirectory of scripts
, I'm happy to submit a PR.
from ytcc.
@EmRowlands, Im not sure how to handle it. Is it public somewhere for me to see? Can you maybe push it to a new branch on your fork in a new subfolder of scripts/
? Then we can still decide where to put it when we merge it. Maybe, we create an orphan branch (git checkout --orphan ...
).
from ytcc.
I have added them in #50
from ytcc.
Related Issues (20)
- Support for yt-dlp as youtube-dl is being actively throttled by google HOT 5
- Ability to "ytcc unsubscribe" in bulk by -y, assume yes, or batch/script mode HOT 4
- ytcc list / ytcc ls not returning any data after initial subscribe and update HOT 5
- ytcc broken on debian HOT 3
- Do not truncate ytccf.sh table output when thumbnails are disabled HOT 5
- Update erroneously ignores videos due to age limit HOT 2
- Use short URLs to free space in the terminal HOT 2
- Show which subcription an update error belongs to HOT 1
- [Request] Portable configuration HOT 3
- there are some bugs i dont know how to call them HOT 1
- Cannot connect to the database or query failed unexpectedly HOT 6
- Playlist updates with short max backlog HOT 13
- TUI doesnโt start anymore HOT 5
- TUI shortcut for unmarking the last video played HOT 1
- Please issue a new release HOT 2
- Truncate long titles HOT 1
- filtering ? HOT 5
- youtube subscription export doesn't work anymore HOT 4
- Thumbnails HOT 2
- bug in v2.4.1: 'ytcc tui' throws error "unknown encoding: locale" HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ytcc.