njlaprell/gek

Watch subscriptions for new videos and add them to playlists based on rules.

Gek

• Gek
  • What is Gek?
  • What do I do with it?
    • Setup Rules
    • Sort Videos
    • Unsorted Videos
    • Profit
    • Tablet Remote?
• Project Setup
  • Pre-requisites
    • Domain
    • Node.js
    • Docker
  • Environment
  • The Stack
  • SSL Certificates
  • Google Cloud Project Setup.
• Deploy
  • Steps:

What is Gek?

Gek watches your subscriptions for new videos, then sorts them to your playlists based on your rules.

Set up playlist rules based on the channel that created the video, the description, and the title. When a video matches the rule, it is added to the playlist. Any video that does not match a rule is put into an unsorted list where you can remove anything you aren't interested in watching, or manually sort it to a playlist.

What do I do with it?

Log in with your Google YouTube account to grant access to your subscriptions, playlists, and the ability to like or dislike videos from within the app. Once signed in, you will see your playlists on the left.

Setup Rules

When one of your subscribed channels creates a new video, your rules will determine which playlist the video is added to.

• Type - Set type to and if you want to match on everything (such as the channel and the title). Set to or if you want to match on anything.
• Channel Match - Select a channel from your list. Often, this may be the only match you want to set if you want to sort all videos from a particular channel to a specific playlist.
• Title Match - Text to match the title of a video. This can be a regular expression for more complicated matching.
• Description Match - Text to match the description of a video. This can be a regular expression for more complicated matching.
• Playlist - This is where the real magic happens! 🪄 Select a playlist for matching videos to be saved to and click the green check to save the rule.

Lather, rinse, repeat! Create as many rules as you like. They are currently run in the order they appear, which means that once a rule matches a video to a playlist, no other rules are applied to that video. Additional abilities will be added to the rules at a later time.

Sort Videos

Once you have rules setup, you don't have to do anything else. Once an hour, Gek will check your subscriptions for new videos and sort them to your playlist. Can't wait that long? You can set how frequently videos are sorted in the Preferences or even click on the Sort Videos button if you want to sort NOW!.

Unsorted Videos

Any video that is not automatically sorted is added to the Unsorted videos queue. This makes it easy to manually sort things yourself without missing any new videos.

For each unsorted video, you can select a playlist to sort it or just click delete to remove it from the list. Click Purge Unsorted Videos to remove everything from the unsorted list.

Profit

Select a playlist from the left to list the videos. Click the refresh button at the top if new videos have been added recently. Select a video to watch and rate it or remove it from your list at any time.

Tablet Remote?

Once you try it out on a tablet, you'll be thinking, "Gee, it sure would be nice if I could use the tablet to control the video on my computer/tv."

Then you will be excited to find out that you can do just that. Load Gek up in a browser on the TV and log in, then toggle the Viewer control in the upper right to turn it into a video viewer.

On your tablet (or any other browser, really), log in and toggle the Remote control. The two devices will pair and any videos you select on the remote will play on the viewer. The tablet will also navigate next/previous videos in your play list, like, dislike, and remove videos from a playlist, and control volume, mute, and seek functions.

Project Setup

---

Pre-requisites

Domain

For accessibility outside of the localhost, you can add an entry to /etc/hosts to point to the IP and give it a domain name to use. This domain name will then be used when setting up the OAuth config with Google.

For a more production like environment, Nginx Proxy Manager can be used as a reverse proxy from the host to the UI and Server services. It can be added to the docker compose, or you can add it's network to the docker compose to join it.

Node.js

Node Version Manager allows you to select/change version of Node/npm without effort. Once installed, nvm install lts/gallium.

Docker

Docker is used for a more production ready environment. Docker Desktop provides a GUI and includes Docker Compose, which is also used to create two services.

---

Environment

ESLint is used for code styling.

Dotenv is used to read in a .env file for sensative and envronment based settings. Create a .env.dev and .env.prod file based on the .env-example file and make sure they do not get checked in.

The UI is build with angular and uses the environments files for environment configuration. UI source is in the /src folder.

The backend is written with TypeScript and consists of an express server, a websockets server, and a service to keep the video lists up to date.

Package scripts serve the UI with a watcher and use ts-node-dev to transpile the typescript for the server components. Building generates source and places it into the gek-docker folder and zips everything up as a portable package, requiring only docker-compose up-d to run.

Use npm start for an interactive menu to select dev and build tasks.

---

The Stack

• Angular UI
• NgRx for UI state using actions and effects.
• Express for the backend APIs.
  • Passport with the Google Open Auth 2 strategy to handle user authentication and permissions for Google APIs.
  • Express sessions for session managemt (will need to change eventually)
  • Firestore for session store, because:
• Firebase for storing user rules and caching API data so you don't hit Google's quota restrictions.
• WebSockets for managing Viewer/Remote communication.

---

SSL Certificates

SSL Certificates are stored in the certs directory, but not checked into version control. Update angular.json to point to the correct SSL certificate in order to use https with the angular UI during development. If you need to create a self signed certificate, run the following from the certs directory:

./ssl.sh my.domain.com

Be sure to add the certificate and the CA to your browser/OS and set trust settings. (Thanks to: https://devopscube.com/create-self-signed-certificates-openssl)

In order to use the .env file for your certs, first, base64 encode them and save the base64 result as a string for the SSL_CERT, SSL_KEY, and SSL_CA values.

---

Google Cloud Project Setup.

The project uses Google OAuth and the Firebase Firestore. Use the Cloud Console to setup a new project:

1. Go to: https://console.cloud.google.com/
   • Add a new project.
2. Menu > APIs & Services > OAuth consent screen.
   • External User Type
   • Skip scopes for now (The sensative scope will require HTTPS to be setup)
   • Add test users (including yourself).
   • Create Credentials > OAuth Client ID
   • Web Application
   • Add origin URI (https://mydomain.com:4200)
   • Add authorized redirect URI (https://mydomain.com:4200/auth)
   • Create
   • Download JSON.
     • Copy the values of the JSON to the .env file, using .env-example as a guide. Be sure the .env file is not checked in.
3. Menu > APIs & Services > Credentials.
   • Create Credentials > Service Account
   • Create App Engine Service Admin
   • Add Email
4. Menu > APIs & Services > Credentials.
   • Create Credentials > Service Account
   • Create Firebase Admin SDK Administrator Service
   • Add Email
5. Menu > APIs & Services > Enabled APIs and Services
   • Enable APIs and Services
     • Youtube Data API v3
     • Google Cloud Firestore API
6. Menu > Firestore
   • Select Native Mode
7. Go to the firebase console: https://console.firebase.google.com/
   • Add Project
   • Quickstart documentation is here: https://firebase.google.com/docs/firestore/quickstart
8. From the firebase console: https://console.firebase.google.com
   • Gear Icon > Project Settings > Service Accounts
   • Generate a new private key
     • Update the .env file with relevant values from the JSON file.
9. Install the Firebase Admin SDK and initialize the project.
   • npm install -d firebase-admin
   • Create a folder to use as a workspace and change directories to it.
   • firebase login
   • firebase init
   • Select Realtime Database.
   • Select your project.

Deploy

Deployment is done by creating a deploy zip consisting of the required source along with a docker compose file. Docker compose builds two services and joins them to an existing Nginx Proxy Manager reverse proxy. Adjust the network config appropriately.

Steps:

1. Ensure there is a .env file in the gek-docker/server directory.
2. Run npm run build:package. This cleans and builds the project with production configurations and environment, copies everything required to the gek-docker directory, then creates /dist/gek.tar.gz.
3. Copy gek.tar.gz to the production server and unzip it (tar -xf gek.tar.gz).
4. Build the containers and start them:
   • docker-compose build
   • docker-compose up -d

gek's Issues

Auto Next for Remote

Update the remote to automatically play the next video.

Sweep: Hide the "Mode" button on the top toolbar.

Details

Update the style so the button is hidden from view.

The file to change is: top-bar.component.html

Checklist

• src/app/top-bar/top-bar.component.html ✅ Commit a243483

• Find the HTML code for the "Mode" button. It should be a button element with the text "Mode". • Add the "*ngIf" directive to the button element and bind it to a flag (e.g., "showModeButton") in the component's TypeScript file. The code should look like this: Mode

• src/app/top-bar/top-bar.component.ts ✅ Commit 88f205a

• Add a new property "showModeButton" to the component class and set its value to false. This will hide the "Mode" button.

Combine Unsorted and Error Queue lists

• Updated the data model to add error details, fail date, and retry count to the unsorted queue.
• Move logic to use the unsorted queue in place of the error queue.
• Update the UI to remove the error queue.

Auto Next Preference

Check the Auto Next preference before automatically playing the next video.

Link Google TOS

https://developers.google.com/youtube/terms/developer-policies#a.-api-client-terms-of-use-and-privacy-policies

API Clients must display a link to YouTube's Terms of Service (https://www.youtube.com/t/terms), and they must also state in their own terms of use that, by using those API Clients, users are agreeing to be bound by the YouTube Terms of Service.

Playlist Buttons

Add preference to configure shortcut buttons for playlists. Preference should link the playlistId and title with a fontawesome icon.

Buttons should be displayed in the unsorted queue. Clicking moves the video to the playlist.

Use HTTPS

Update express and angular to use https.

Rebranding

Check and alter relevant YouTube branding. (https://www.youtube.com/howyoutubeworks/resources/brand-resources/#logos-icons-and-colors)

• favicon
• header icon
• App Name

Rename the app (currently violates the branding rule that it may not contain the YouTube name or abbreviations of the name).

Order Rules

Add drag and drop to allow ordering rules execution.

Sweep: update the history store to refresh every minute.

Details

Since the sort process is run automatically at regular intervals outside of the UI, the NgRx History store becomes stale until the app is reloaded.

Update the initializer.service.ts file so that when getHistory() is dispatched, it continues to dispatch again every 1 minute.

Checklist

• src/app/initializer.service.ts ✅ Commit 0f0d17a

• Import the 'interval' function from 'rxjs' at the top of the file. • In the 'initApp' method, replace the 'concat' function with 'merge' function from 'rxjs' to combine the observables. • Inside the 'merge' function, add 'interval(60000).pipe(switchMap(() => this.getHistory()))'. This will dispatch the 'getHistory' action every minute. • Implement the 'OnDestroy' interface for the 'InitializerService' class. • Declare a private variable 'historyUpdateSubscription' of type 'Subscription' in the 'InitializerService' class. • Assign the result of the 'merge' function to 'this.historyUpdateSubscription' in the 'initApp' method. • Implement the 'ngOnDestroy' method in the 'InitializerService' class. Inside this method, call 'this.historyUpdateSubscription.unsubscribe()' to clear the interval when the service is destroyed.

Refresh lists on sort

When a sort is run, the lists should automatically be refreshed.

Create an endpoint to check if sort was run recently. It should return true/false and next expected run.

Call the endpoint on init, on manual run, and on preference changes.

Use the next expected run to set a timeout to re-call and confirm at the next expected run time.

Sort from the error queue

Update the error queue UI to allow sorting to a selected playlist.

Add additional video details.

Display which playlist it was attempted to be sorted to.

Remove the combined display from the unsorted list.

Deleting video causes nav state to shift

Steps to Reproduce:

While in the Play or Remote watching a video, click delete and confirm the video deletes.

Expected: Video is deleted from the list but continues to play normally with no effects to the UI.

Actual: The nav state shifts so that the next video in the UI is always the first in the playlist.

Missing Preferences for New User

When a new user is created, preferences should be created.

Subscription Buttons

Add subscribe/unsubscribe buttons and functionality.

Select Playlists

Allow a user to explicitly opt individual playlists in/out of being displayed in the UI.

• Create a UI to list all user playlists and allow a user to opt each one in (switch for each one)
• Include an option to automatically opt in new playlists.
• If no preferences are saved, use all playlists.
• If new playlists is not on, only use playlists explicitly on.
• If new playlists is on, only use playlists that are not explicitly off.

Mobile Styles

Update styles to display well for phone size screens.

Tablet Styles

Update styles to display well for tablet size screens.

Page Title Navigation

Page title should navigate from a video back to the playlist.

Close Almost Over Toast When Navigating

Navigating to a different video should automatically close the "Almost Over" toast if it is being displayed.

Auto Play Preference

Check the auto-play preference in the Player before automatically playing a video.

Update Readme/Screenshots

Update the README and screenshots.

Video Sort throws errors

More than 5 or so matched videos being sorted cause a generic 500 error from the Google API. It appears to happen as a result of making several calls simultaneously.

Serialize the calls to add the video to a playlist in an effort to prevent the errors.

Handle un-embeddable videos

18+ videos do not play in the embedded player.

The Player/Viewer should detect these videos and:

1. Open the video in a new tab.
2. Maximize the video (may not be possible)
3. Close the new tab when the video is finished playing.

Google Auth Revocation

Revocation

Every API Client must provide a clearly explained and easy way for users to revoke any authorization consent they have provided to an API Client to access YouTube API Services.

When a user revokes consent through this mechanism, the API Client must programmatically revoke that token right away to communicate the change in permissions to Google. For example, an API Client could use a Google API Client Library to revoke the token.

In addition, following revocation of consent through this mechanism, you and your API Clients must delete all Authorized Data that was accessed or stored pursuant to that consent. That deletion should happen as soon as possible and must take place within 7 calendar days of the revocation.

As noted in section (III.A.2.i), every API Client must include in its Privacy Policy a link to Google's security settings page (https://security.google.com/settings/security/permissions). When a user revokes consent through that page, you and your API Clients must also delete all API Data related to that user that was accessed or stored pursuant to such consent. To comply with this policy, your API Clients will need to periodically reconfirm that its authorization tokens are still valid and delete API Data associated with users whose authorization tokens cannot be refreshed.

Based on the requirements defined in section (III.E.4) regarding stored data, all such deletions should happen as soon as possible and must take place within 30 calendar days of that revocation.

History/Preferences Data Refactor

The current structure of the history and preferences docs probably won't scale well.

• Preferences stored in an array as name/value pairs cannot be queried by Firestore.
  • Preferences should be stored as simple objects instead of the using the items array like resources.
  • Both preferences and history should have the userId value within the document.
    • Firestore does not appear to support querying by docId.
  • DataStore methods to get Auto Sort information should be rewritten to use queries and indexes instead of scanning every document in the collection.

Update Privacy Policy

https://developers.google.com/youtube/terms/developer-policies#a.-api-client-terms-of-use-and-privacy-policies

Each API Client must require users to agree to a privacy policy before users can access the API Client's features and functionality. The privacy policy must:

be prominently displayed and easily accessible to users at all times,

notify users that the API Client uses YouTube API Services,

reference and link to the Google Privacy Policy at http://www.google.com/policies/privacy,

clearly and comprehensively explain to users what user information, including API Data relating to users, the API Client accesses, collects, stores and otherwise uses,

clearly and comprehensively explain how the API Client uses, processes, and shares the user information described in section (III.A.2.e), including how the information is shared with either internal or external parties,

disclose, if it does so, that the API Client allows third parties to serve content, including advertisements,

disclose, if it does so, that the API Client stores, accesses or collects (or allows third parties to do so) information directly or indirectly on or from users’ devices, including by placing, accessing or recognizing cookies or similar technology on users' devices or browsers,

if the API Client accesses or uses Authorized Data, explain that, in addition to the API Client's normal procedure for deleting stored data, users can revoke that API Client's access to their data via the Google security settings page at https://security.google.com/settings/security/permissions, and

if the API Client uses Authorized Data, explain how users can contact the API Client owner or developer with questions or complaints about the Client's privacy practices.

Add docker configs

Updated for use with docker and split ui/server packages for building a distribution zip.

Sweep (fast): Update the history store every minute

Details

Since the sort process is run automatically at regular intervals outside of the UI, the NgRx History store becomes stale until the app is reloaded.

Update the initializer.service.ts file so that when getHistory() is dispatched, it continues to dispatch again every 1 minute.

Playlist Toggle

• Remove the "Playlists" button from the toolbar.
• Add a thin arrow button to the left margin of the screen to toggle the playlists open when they are closed.
• Automatically collapse the playlists when clicking on a playlist.
• Add a preference to keep playlists open when navigating.

Keep Feature

Add a user preference to designate a playlist the "Keep" list.

Add action buttons to the playlist and video views to keep a video.

Security Audit

Audit for security:

• Check/update npm packages.
• Set and check-in firestore security settings and set:
  • tokens accessible only to admin.
  • User docs accessible only to corresponding user.
• Check session security.
• Check and set auth required routes.
• Code scans.

Suggested Videos

• Display suggested videos below the video controls.
• Add a route specific to playing suggested videos.
• Store the last playlist/video when playing a suggested video and add a nav button back to the previous location.

Refresh Playlist on Nav

Navigating to a playlist should cause the contents to automatically be refreshed with an uncached version.

Like/Dislike Does not reset

Like/Dislike button should reset when navigating to a different video.

Video Actions in Playlist View

Add a button to videos in the playlist view that can dispatch actions:

• Delete
• Move to Playlist

Sweep: Update the history store every minute.

Since the sort process is run automatically at regular intervals outside of the UI, the NgRx History store becomes stale until the app is reloaded.

Update the initializer.service.ts file so that when getHistory() is dispatched, it continues to dispatch again every 1 minute.

New Video Count

Update playlist resources to track view time and use that at the measurement for the new video count.

Explore

Add the ability to explore other videos by topic

Upgrade to Angular 15