rconradharris/ply

Git Based Patch Management

ply - git-based patch management

Description

ply is a utility to manage a series of patches against an upstream project. These patches are stored as files in a separate git repositiory so that they can themselves be versioned. These patches can then be applied to create a patched version of the code to be used for packaging and deployment.

Concepts

The upstream project resides in the upstream-repo (UR). Your local checkout of the upstream-repo is called the working-repo and is where you'll do most of your work: you'll make changes, commit them, and then run ply save to create a new set of patches, called a patch-series.

The patches are stored in the patch-repo (PR), a separate git repo that is linked to the working-repo using the ply.patchrepo git config.

Usage

• Initialize a new patch-repo which initializes the new git repo and commits an empty series file:

  ply init .

• Link working-repo to a `patch-repo`:

  ply link ../my-patch-repo  # from within the working-repo

• Unlink working-repo from current `patch-repo`:

  ply unlink

• Check that status of a `working-repo`:

  ply status
  All patches applied

• Save set of commits to the `patch-repo`:

  # Without --since, any 'new' patches (patches that follow applied patches)
  # will be saved
  ply save
  
  # Save only the last commit into the 'foo' subdirectory
  ply save --since=HEAD^ --prefix=foo HEAD^

• Rollback working-repo to match upstream:

  ply rollback

• Restore `patch-series`:

  ply restore

• Resolve a failed merge and continue applying `patch-series`:

  ply resolve

• Skip a patch that has already merged upstream. In addition to performing a git am --skip, this will also remove the relevant patch from the `patch-repo`:

  ply skip

  Note: If the upstream patch is an exact match of the version in the patch-repo, ply will automatically remove the patch from the patch-repo.

• Perform a health-check on the patch-repo. This ensures that all of the patches in the patch-repo are accounted for in the `patch-series`:

  ply check
  OK

• Create a DOT graph representation of patch dependencies:

  ply graph

  The output of this can be piped into dot to generate a PNG file:

  ply graph | dot -Tpng > dependencies.png

ply vs X?

Tools for managing patches have existed for a while, so why create another?

The short answer is:

quilt deals in patch-files which can be versioned but doesn't understand version-control. This orthogonality, in some respect, is elegant, but is a hassle in day-to-day use. Why checkpoint files in quilt when your version control system already does that for you?

stgit (stacked-git) understands version control but stores patches as commit objects, not as patch files. This means you can't version your patches, making it impossible to rollback when things go awry.

ply blends these two tools together to create a tool that understands version-control but at the same time stores patches as files which can themselves be versioned.

ply's Issues

Ply abort when we git am --abort under the hood

rick@nova-dev:~/Documents/code/openstack/nova$ ply abort
Traceback (most recent call last):
File "/home/rick/.virtualenvs/openstack/bin/ply", line 9, in
load_entry_point('plypatch==0.3.5', 'console_scripts', 'ply')()
File "/home/rick/Documents/code/openstack/ply/plypatch/cli.py", line 271, in main
args.func(args)
File "/home/rick/Documents/code/openstack/ply/plypatch/cli.py", line 69, in do
self.working_repo.abort()
File "/home/rick/Documents/code/openstack/ply/plypatch/init.py", line 249, in abort
self._resolve_conflict('abort')
File "/home/rick/Documents/code/openstack/ply/plypatch/init.py", line 236, in _resolve_conflict
self.am(**kwargs)
File "/home/rick/Documents/code/openstack/ply/plypatch/git/init.py", line 14, in wrapper
return fn(self, _args, *_kwargs)
File "/home/rick/Documents/code/openstack/ply/plypatch/git/init.py", line 73, in am
raise exc.PatchDidNotApplyCleanly
plypatch.git.exc.PatchDidNotApplyCleanly

If we manually resolved an am we stlil a way to ply resolve or ply abort.

Looks like am --abort returns an error (since there is nothing to abort) and this raises a PatchDidNotApplyCleanly exception. Solution is to catch that error and raise NothingToAbort, then have restore handle that.

Resolve is broken in 0.4.0

The problem is that the new syncing code doesn't account for partial application of the patch series.

We need to skip patches AFTER the last patch seen (the resolved patch) so that we don't erroneously remove the rest of the patches.

Old versions of Git shorten SHA1

Looks like older versions of Git use a shorter version of the SHA1 which can lead to collisions. This causes missing BLOB breakage.

-index 12e80f78..23cd274 100644
2063
+index 12e80f7..23cd274 100644

Note that '...78' became '...7'

Make _mutate_series_file support recursive series files

If we ever want to use directories to store patch sets, we'll need this fixed.

Don't fetch remote for every restore operation

We only need to fetch on the first-restore, so if we're in the middle of resolving conflicts, for example, we shouldn't refetch canonical.

Rebase -i not preserving git-notes

We use git-notes to keep track of which patches are applied. However they seem to be lost on a rebase -i.

This needs to be replicated and fixed.

Detailed uncommited changes error message required

When showing uncommitted error message please specify the repo if it is patch_repo or working_repo.

https://github.com/rconradharris/ply/blob/master/plypatch/__init__.py#L466

Overzealous commit messages

When adding a single patch using ply, I run:

git commit -a -m "Changed so-and-so in my project."
ply save

This generates a commit message in the patch-repo like this:

commit 9eaa751e5d412a65de653e1d0ce7fbed1ad4a3e3
Author: Brian Lamar <[email protected]>
Date:   Tue Jun 4 12:54:03 2013 -0400

    Saving patches: added 1, updated 72, removed 0

    Ply-Based-On: f1d88658972f9ab79ecf3f78a40c1d3529115253

 Changed-so-and-so-in-my-project.patch | 22 ++++++++++++++++++++++
 series                                                     |  1 +
 2 files changed, 23 insertions(+)

This says 72 patches were updated, when I didn't actually update any patches. The guess is that there is a small bug in commit message generation.

Always regenerate patch names during save

Currently patches retain their patch_name across save, even if their subject line changed.

Since we want to be able to rename patches by changing the subject line using git rebase, we need to be able to disable this.

Rather than add a new option for this, it's probably better to just make this the default behavior. If we come up with a good use-case for needing to disable it, we can add then then.

Git version causing conflicts

The patch files contain the git-version in them.

This can cause conflicts when merging pull requests.

The solution here is probably to just hard code it to some value.

Ply restore fails badly if user.email and user.name are not specified

Better output

Show which patches are being applied in verbose mode. Just show the number in regular mode. Show nothing in quiet mode.

Ply is leaving patch files around in working repo

If ply detects the files in the working repo match the patch-repo, it won't move the files. However, it's not deleting the files it doesn't move, causing it to leave garbage around.

The solution is to os.unlink files that aren't saved to patch repo.

If --3way merge fails, provide more helpful message to user

Right now, if the --3way merge fails during a git-am, it drops the user back to the prompt with no information.

We should provide a more helpful error that directs them to manually apply .git/rebase-apply/patch.

Eventually it would be nice to automatically apply this patch.

Off by one error on Ply restore

Said restored 1/123 when it's really on 2.

Make "ply link" idempotent

If a "ply link" is performed for a path that the repository is already linked to, let's do "exit code 0, stdout='already linked', stderr=''" so that the operation becomes idempotent. Obviously, if the paths do not match, a non-zero error code should be returned.

ply resolve counting (minor output issue?)

When resolving a merge conflict (or I guess in this case it's a patch conflict?) I noticed some minor incorrect output:

$ git add nova/test.py
$ ply resolve
Restoring 92/72
$ ply save
Restoring 72/72

I'm not quite sure how it counted to 92, but everything looked like it worked correctly except for the output counter!

SHA1 Info missing

Restoring 54/72fatal: sha1 information is lacking or useless

May need to fallback to not doing a --3way merge in these cases.

Needs more investigation.

Better error on restore after unresolved conflict

Try harder if `am` --3way fails

Right now we just give up ungracefully when am --3way fails.

There might be ways to try harder on behalf of the user.

For example, git apply --3way seems to occasionally work even if an am fails (since it doesn't call build_fake_ancestors).

Placeholder for considering this.

A better solution would be to fix the underlying missing blob issue if we can figure out exactly what's going on here.

Patch will not be removed if it has local changes

The use case of renaming a patch won't work because:

1. The original patch will be updated with change
2. A patch will be updated (if first line of commit message changed)
3. The original patch will attempt to be deleted, but will fail because it has outstanding changes.

The solution here is to pass -f to git rm.

Use absolute path when linking to patch repo

Test warning messages

Getting this when running the tests:

Failed to merge in the changes.
error: cache entry has null sha1: README
fatal: unable to write new index file

Using: git version 1.7.12.4 (Apple Git-37)

Don't recall getting this error message before. Need to test it against different versions of git.

Whitespace issues when creating patches

This should probably become a git bug, but recording here for now.

Basically, git produces all kinds of funky whitespace when generating patches that it shouldn't.

git fetch --all can be annoying

If you have all your remotes stored in your ~/.gitconfig, then doing a fetch --all will cause everything to be synced, not what we wanted.

We should probably have the following behavior:

1. By default perform a git fetch against origin
2. Add an option which allows you to specific which repos to fetch
3. One value could be 'all'
4. Allow a ~./plyrc to specify this value permanently

Move ply state into .git/

The current ply state files .patch_conflict and .patch_stats are stored in the working tree. This mucks it up and makes it hard to do a git add ..

It'd be better if these were stored in .git/ply-state