Model collapse help

[lucidrains]

Model collapse help

[lucidrains]

@iboates perfect timing! I'm about to add an experimental feature that may (or may not) help you train for longer and achieve more detail! you are my perfect guinea pig :)

[lucidrains]

leave this open, i'll leave you with instructions by end of day

[iboates]

That's great. I'm also researching some options for more efficient training - do you use AWS for training anything? I was thinking about buying some time on an EC2 GPU instance.

[lucidrains]

@iboates in the end, i found it most cost effective just to invest in a couple top-of-the-line GPUs

[lucidrains]

@iboates what resolution is your map data trained on?

[iboates]

I'm doing 256x256, that size seemed sufficient to capture the map features I wanted at an appropriate level of detail. I was hoping to one day drop some big bucks and go all-out on a 1024x1024 model, once I was sure that the process would work.

[lucidrains]

@iboates 1024x1024 is out of reach for most people, even for people with a ton of compute

256x256 is good! let me get back to you on that new feature that may or may not work lol

[lucidrains]

@iboates the feature is ready! to try it, just upgrade the package and add a --cl-reg flag. it should take slightly longer to train, but if it works as expected on your end, it should train for longer, and perhaps result in better quality

[iboates]

I am trying it and it seems to make it take approximately 5s per iteration, as opposed to (if I recall correctly) about 1.8s per iterations without the flag. I think in order to train this I will have to deploy to a paid solution or try to get my hands on a fancy gaming laptop at my office (which is hard now because of the quarantine...)

Follow up: Actually it seems to now be running at about 2.3 s per iteration which is much more manageable. I find that with Colab it almost feels like it needs to "warm up", as it will start with higher processing times but then get better after a couple hours or so. Maybe it is because I have mounted my Google Drive directly and it's not actually block storage, so it takes time to build a cache for request to my training folder. Anyway, I'm excited to see if this improves my results, I guess I'll know more in a week haha

[lucidrains]

yup! that sounds about right! let me know :)