support other update types

[craffel]

support other update types

[craffel]

Copying design here since README will change:

For a given parameter group, git-cml will store either the group's parameter's values or an update to the values in the event that it can be stored more efficiently. For example, if a sparse update is made to a parameter group, git-cml will store the sparse update in a sparse format to save storage costs. For a given parameter group, all updates made since the last time the group's values were stored are stored in an updates subdirectory within .git_cml/{checkpoint_file_name}/{parameter_name}. Whenever a dense update is made to a parameter group, the new full set of parameter values is stored and all prior updates are removed from the updates subdirectory. All parameters and updates are currently stored via git-lfs.

[craffel]

Requires making it so that you can provide hints to git-theta as to the appropriate update type to use

[blester125]

Do we want to allow a single commit to have multiple changed parameters with different update types? Like a dense update to attn weights but a sparse update to the embedding table? Or a single change changes everything the same way?

[vishalathreya]

Last time we discussed this, we decided on a single commit to have only 1 kind of update to begin with. @craffel are there scenarios wherein multiple update types are used in the same training run?

[craffel]

Ideally we could support something like that - there's no reason someone wouldn't possibly update different parameter groups with different methods. But I think it's probably a lot less common, and we could always force users to break up their changes into multiple subsequent commits according to update type.

[blester125]

With #92 merged we can now use plugins for different update types.

@vishalathreya The current "sparse" implementation is really simple and you don't get any actual gain wrt to the sparseness. It would be good to incorporate the smart sparseness features you had in #75. It would probably be via updates to the read, write, and calculate sparse methods here https://github.com/r-three/git-theta/blob/main/git_theta/updates/sparse.py#L51(it would probably also make sense to abstract part of the writing into a write_update method.

[nkandpa2]

After #114 the current pattern is that git add <model> will default to storing the updates as dense updates. In cases where a user wants to use a different update type, they can set an environment variable GIT_THETA_UPDATE_TYPE=sparse before running git add <model>. Alternatively, they can do git theta add <model> --update-type sparse which internally just sets the environment variable, calls git add <model>, and restores the environment variable to its initial value.

[blester125]

I think once #127 (low-rank) and #125 (real sparse updates) land it should probably be enough to close this.

[craffel]

Closing because it's subsumed by #127 and #125.