[Workflow] Use Git LargeFileStorage

[enviraldesign]

[Workflow] Use Git LargeFileStorage

[EnviralDesign]

Thank you for the tip! I am getting this setup now.

[LimitlessGreen]

@EnviralDesign Thanks for switching to lfs.
Unfortunately, I forgot that the changes did not affect the earlier stages of the repos.

This would require to rewrite the previous commits. So be careful if you're relying on some commits SHA's. They aren't the same after the procedure.

I already tested it in my repo: https://github.com/LimitlessGreen/GeoPix

Before:

After:

The steps are:

1. There are some other binarys, worthy to track: git lfs track "*.gif" "*.toe" "*.tox" "*.zip" "*.blend" "*.obj"
2. commit it: git commit -a -m "Track more files with lfs"
3. Clone the mirror repo: git clone --mirror https://github.com/EnviralDesign/GeoPix.git
4. Download BFG cleaner: https://rtyley.github.io/bfg-repo-cleaner/
5. Rewrite the previous history to lfs: java -jar .\bfg-1.14.0.jar --convert-to-git-lfs "*.{psd,wav,mp3,hdr,afphoto,jpg,png,tiff,tif,gif,svg,mov,mp4,bclip,exe,dll,ttf,exr,toe,tox,zip,blend,obj}" --no-blob-protection .\GeoPix.git\
6. clean the repo: git reflog expire --expire=now --all && git gc --prune=now --aggressive
7. force push it back (maybe need a change in your github config): git push --force
8. remove the GeoPig.git repo
9. clean your working repo with git reflog expire --expire=now --all && git gc --prune=now --aggressive too

(Source from: https://docs.gitlab.com/ee/topics/git/lfs/migrate_to_git_lfs.html)

[EnviralDesign]

I think I got this working, thanks for the detailed steps. However now the Download Zip option downloads a very small zip file with some kind of placeholder file for all the LFS managed ones - any idea how to solve this?

[LimitlessGreen]

Ouch, never trapped into this because I always use the git command line.
I'm looking for a solution.

[EnviralDesign]

ok thanks. I think I want to offload some of these larger files anyways entirely from github and integrate them as assets that get pulled from my webserver separately by the bootstrapper bat file, but still that leaves plenty of things that ideally would be downloaded via the git zip. the .toe executable is the actual software binary, so maybe I can just leave the critical binaries and do the rest some other way, if lfs can't be downloaded via zip

[LimitlessGreen]

I would minimize the traffic on my side as much as possible. What's your reason for getting the stuff separately?
If you want an extra place for static files I would recommend GitHub pages.
LFS is fine for <= 2GB files.

Do you want to let the users download the latest state or the latest release?
Maybe a workaround with Github Actions can pack the zip together on every commit and/or release.
I can do this if needed.

[EnviralDesign]

The files in the project I'd say generally fall into three separate buckets:

a) executable .toe file is the main project file that is run in TouchDesigner, other code, py, glsl, etc are sourced from disk in the toe, they can also be embedded in the toe as well. Is there any reason why I shouldn't keep the .toe file as a normal git file? There will ever only be 1 main one, maybe 2 or 3 others in the future., and the size of toe's are usually in the kilobytes, GeoPix.toe is larger than average at ~5 megs. This will easily be the most frequently updated file in the repo.

b) necessary asset files like icon png,s sheets, and other things that would break the software visually or functionally, but don't get updated as often, but do from time to time.

c) sample content of various kinds - technically all the .obj's, .hdr's, pbr material textures, some sample videos, etc all fall into this bucket of things that the software can run with out, but user experience of things they can drop in quickly might be a little bare.

I was hoping to keep a and b in the repo, in some way that resulted in the user getting it in the zip file download as well as using the github desktop client, etc.

then in my comment above, I was mentally thinking c, which is the bulk of the file size could be stored on my web hosting server, and via wget in the START_GEOPIX.bat or even from python inside the main application, the user could have these assets download entirely separately from the github and lfs.

My other concern with lfs as I am reading more into it is hitting the bandwidth cap pretty early, and easily. I already pay for webhosting with "unlimited" storage/bandwidth as they say, so I feel like I can go much further with that.

Though I'm not too familiar with GitHub Pages or Actions - do you see either of those being better alternatives?

Thanks again! Appreciate your help.

[LimitlessGreen]

Thank you for the detailed answer.

Just keep b in the repo, it's totally fine.

a is the reason why I suggested LFS. Git itself is good at storing deltas of text files. But can't track binaries very good. For example: if you modify a 5 MB File 50 times, you'll end up with a repo of 250 MB for just a 5 MB file. But also the GeoPix.toe is kind of your main file. So keeping the history of changes is important. And that's exactly where LFS kicks is. It just gives you the latest version but can also pull older ones if needed. And yes, your toe files should stay in here.

c: It's kind of a good practice to have an extra repo for assets and/or libraries. (like FreeCad for example https://github.com/FreeCAD/FreeCAD-library)
But it's not really necessary. This files barely changes and the majority of people would use them anyway. If you really want to outsource this, please give it a very low priority.

Maybe I have a quick and dirty solution for now. Just do this at the beginning of the batch file (The use must have git installed)

1. Check if the .git dir is available
2. if not: git init
   2.1 git remote add origin https://github.com/EnviralDesign/GeoPix.git
   2.2 git fetch
   2.3 git reset --hard origin/main

At the end it recreates the git repo and lfs do the job of getting the missing files.

Wow, 1GB is nothing ... https://docs.github.com/en/github/managing-large-files/about-storage-and-bandwidth-usage
... I kinda feel bad to have suggested this :/
I don't understand this limit. Under the line GitHub can save space with more usage of lfs...

Well unlimited web storage/bandwidth changes the game.

It's late in Germany. Maybe I have an idea to fix this tomorrow.

[LimitlessGreen]

Maybe it's better to stay away from lfs for now :/ . Luckily git saves all, even unreachable commits.
That's the latest commit before the rewrite: 69bafdf

So you can reset to the point before lfs and rebase your new commits on top of that.

[LimitlessGreen]

In your case it is:

git reset --hard d055f837c48d865fda00ee1ccdbf559e0914e389
git cherry-pick 31db3afae9e8bae7864678a6e189f5235e3f245c
git cherry-pick 67c294048a5e4da3f7238dc019c154b7aa04d80b^..59074f13f2418cf4c94a7dbfd46c29c41d9a391a

On top of 59074f1 you touched lfs versioned files. So merging was not intuitively possible (just do the commits again)

[EnviralDesign]

Thanks for the instructions on rolling back of sorts - I am making good headway into offloading the really big stuff to that initial bootstrap script.

Could you explain that last sentence in more detail? I'm not sure what it means or what it implies I should do differently than the 3 commands above:

"On top of 59074f1 you touched lfs versioned files. So merging was not intuitively possible (just do the commits again)"

One last question - is it possible to just clear out the history of the main toe file from time to time to keep the git lighter? I know this is totally backwards from the intent and design of GitHub generally but I actually keep a backup of every single save file of GeoPix from the last 5 years offline anyways - so just wondering if that could be a solution to preventing huge bloat over time.

Thanks again.

[LimitlessGreen]

Each git commit has its unique SHA2 hash and also each commit is pointing to a parent commit.
You commit "deleted dev tox" is pointing to the parent 59074f "Delete getSponsors.py".
the last two commits in the picture from the server-side main branch should be redone after rolling things back.

No, pls avoid that. It's possible to clean things up. But then you have to rewrite your git history. And all SHA (as I mentioned above) changes. So for example, if someone forked your repo for a pull request, things starting to go very bad because they're pointing to the commits before the rewrite.

Git is all about things pointing to other things.

[LimitlessGreen]

There is another thing like lfs called git-annex. But not sure if it's worth it.

[EnviralDesign]

@LimitlessGreen ok so I went through the steps you outlined above, the 3 terminal commands.. but now It's unclear what I need to do next.. I'm not an expert with Git command line - still learning. Do I need to push changes from that terminal? or pull? does doing any of these steps from GitHub desktop client interfere or ignore what's happening in the command line?

I see this now in the desktop client, but not sure what to make of it:

[EnviralDesign]

for example wget.exe is still managed with lfs:
https://github.com/EnviralDesign/GeoPix/blob/main/GP/Utilities/wget.exe

[EnviralDesign]

I think I might have figured it out .. not entirely sure.. but file sizes seem back to normal, no lfs, and zip download produces a large file. going to work on getting those other file assets externalized now.

[LimitlessGreen]

I don't see the commit "broke out some code to disk" in the new main branch.

Btw, I recommend using GitKraken. In my opinion, it gives new git users a really good feel to understand how the structure of git works. I myself only use it to visualize repos.

[EnviralDesign]

I meant to ask you next what software you were using for visualizing that - It seems extremely helpful.
Ya, I manually patched that back into the repo, under a different commit from a separate backup. so all good there.