Comments (12)
How much data are we talking? More than an in-memory map and writing to disk with gob
?
from git-lfs.
We may be good with just a simple key/value map. The key should be based on the filename. If a filename is queued again, we can just update that key's value. The last 3 items just depend on the ability to figure out what is being pushed, so we can look around for git media blobs.
from git-lfs.
One other option: Change push so it requires the remote and branch (which should be provided by the pre-push hook. The pre-push hook gets this in STDIN:
{local ref} {sha} {remote ref} {remote sha}
refs/heads/master 67890 refs/heads/foreign 12345
We can get this easily with:
$ git ls-remote -h origin master
b336a0f59a945b53259856e94cfb2440bfd5ca4e refs/heads/master
from git-lfs.
We can get this easily with:
$ git ls-remote -h origin master b336a0f59a945b53259856e94cfb2440bfd5ca4e refs/heads/master
Easy, yes, but don't forget that git ls-remote
requires a round-trip to the upstream repo so it is a bit expensive.
from git-lfs.
It seems to me that instead of recording the filename, it would be more robust and easier to handle the SHA-1 of the blob representing the pointer file. If and only if that blob has to be pushed to the git server, then the corresponding git-media blob has to be pushed to the media server.
from git-lfs.
Ok, I'm leveling up my git game here, want to check in to make sure I'm on the right track. Here's a scenario:
I've got 3 commits I want to push and sync, efd7
, da5e
, and 60a0
. When I run git push origin master
, the pre-commit hook will get: origin https://github.com/rubyist/mediarepo
on its command line, and refs/heads/master efd7 refs/head/master 9b77
on its stdin. Cool.
Now, I want to figure out what objects it's about to push so I know what to sync. I know that I could do git rev-list --objects efd7 ^9b77
and get a list of objects. Is this an accurate representation of what it's about to push? I'm thinking you just trawl through that list, ignore anything that isn't a blob
, and 💥 you've got your list of objects to sync.
Does that sound legit? Is there a better way to get that list?
from git-lfs.
s/pre-commit/pre-push/
, as you obviously know.
git rev-list --objects efd7 ^9b77
will include all of the objects being pushed, but it can also include far more objects, namely objects that already exist on the remote because they are reachable from a different branch. For example, if I create a new commit and push it:
git checkout -b mybranch master
git push origin mybranch
then the hook script will get
refs/heads/mybranch efd7 refs/heads/mybranch 0000
and your rev-list
command will list all of the commits on mybranch
even though those commits are on master, and many of them are likely already present at origin.
(Actually, your command would fail because ^0000000000000000000000000000000000000000
is not a valid argument to git rev-list
. So in this case the analogous command would be
git rev-parse efd7
. But you know what I mean.)
AFAIK there is no way to get at the information you want (namely, the list of objects being newly pushed to the remote) without either hacking git or duplicating git's negotiation with the remote server 😦 Essentially you would need to know the value of all of the remote server's references before your push.
Maybe @peff knows a trick.
from git-lfs.
Essentially you would need to know the value of all of the remote server's references before your push.
This should work, right?
# Do we want pull request refs too?
$ git ls-remote --heads --tags
From https://github.com/github/git-media
fe6bba2b69f7c18dbf274a9bb958b75f05a50468 refs/heads/ensureprefix
3dc9a647e8b6df76f4f3f5cf026262f09b78f819 refs/heads/master
d71891cb680f16246c236946f07fbe8e49f52365 refs/tags/v0.0.1
4843542e284497c1e3745eed63059f791604868e refs/tags/v0.1.0
690c8da8487c76b3db7bb12a35d9bab84be484ba refs/tags/v0.2.0
d5255d7be0d70996b8561a26998f2e0300a2710a refs/tags/v0.2.1
26fe77e8f089272f44b6968a194b3e0ae4d9a578 refs/tags/v0.2.1-p1
a3f157876e6c30a4cb9ad385feebd33dad5e1b75 refs/tags/v0.2.2
3dc9a647e8b6df76f4f3f5cf026262f09b78f819 refs/tags/v0.2.3
If we're pushing a new branch, we'd want to find the nearest ref (probably master, but it'd depend on where the push originated from).
@rubyist: Having a "dry run" option would be useful here. Have git media push --dry-run
list the media blobs it intends to push. Maybe a --debug
option could show more detailed info about the git refs it used to come up with that list.
from git-lfs.
In the case of a new branch where git really is pushing all of the objects, we can only know whether or not we should sync a git media file by asking the git media server. The git media server should already prevent uploading a full blob it already knows about, so maybe it's OK to use the full list that rev-parse
gives? We could perhaps add a remote-list
like command of our own to avoid all the roundtrips that just 201 or whatever.
We'll definitely add a --dry-run
option.
from git-lfs.
There's not really a trick here. The closest simple thing is probably git rev-list --objects $all_the_things_you_are_pushing --not --remotes=$remote_you_are_pushing_to
. That is slightly different than what git generates because we are using what's in refs/remotes
(i.e., the cache from the last time we fetched), whereas git push
will actually ask the upstream for all of its ref tips. So it falls down in a few cases:
- We might not be pushing to a named ref with tracking branches at all (e.g., git push $url).
- The tracking branches might not be up to date. Usually this means we have a subset of what the other side has, and we'll err on the side of thinking we have to send more objects. But if the other side has rewound a ref, we could err in the other direction.
- The other side may have refs not included in what we've fetched (e.g.,
refs/pull
, or even.have
tips from alternates repositories).
You can get closer with:
# Find out what the other side has, but omit any objects that we don't
# have ourselves. Give them a `^`-negation, and then use the
# result to limit our object list.
git ls-remote $remote |
cut -f1 |
git cat-file --batch-check |
perl -lne 'print "^$1" if /^([0-9a-f]{40})/' |
git rev-list --objects $stuff_you_are_pushing --stdin
That's still not quite right. The server side of a push may advertise slightly more refs than ls-remote
will report, as it will advertise tips for alternates repos it has access to. We don't currently do that ourselves, but other servers might.
Git doesn't actually do any other fancy negotiation to find common refs during a push, like it does for a fetch. The assumption is that you're typically building on what the other side has, so one of their refs should be a subset of what you already have. That could change in the future, but I don't think anybody has ever proposed to make it so.
from git-lfs.
@peff, does the server also report peeled versions of tags?
It sounds like a git ls-remote --haves $remote
command or something like that might be useful here.
Though really, it is a shame to have to run git ls-remote
separately. It costs another round-trip, and possibly opens up race conditions between the pre-push
script trying to second-guess what will be sent vs. the git push
itself which knows exactly what it will send. For this purpose it would be better to have a hook that gets, on stdin, a list of all objects that are about to be pushed.
from git-lfs.
@mhagger Yeah, I never noticed that, but we don't advertise peeled tags for pushes. Another issue: receive-pack
does not necessarily advertise the same set of refs as upload-pack
. Specifically, our servers do not advertise refs/pull
for pushes (because it confuses --mirror
pushers).
I agree that ls-remote --haves
would be useful and would solve all of these problems, but that's going to mean a dependency on a newer version of git. If we're going that route, we might as well go with a separate sent-objects
hook, which has all the advantages you noted.
from git-lfs.
Related Issues (20)
- Git LFS still pulls with no .gitattributes file causing confusion HOT 1
- How to Use `git lfs prune` on Windows Environmen HOT 1
- [QUESTION] How to get upload lfs objects when pre-receive HOT 1
- Need RPM's for RHEL in s390x, ppc64le and other machines in Package Cloud HOT 1
- Error when pushing LFS objects: missing protocol: "<unknown>" HOT 1
- Git LFS pull should provide a bit more information on what's being pulled HOT 1
- pure SSH protocol connection failed: Unable to negotiate version with remote side (unable to read capabilities): EOF HOT 2
- Doesn't work in Windows 7 HOT 4
- git lfs ls-files --all -ls should print out exact size HOT 2
- .blend file size appears to increase during git lfs upload before failing. Actual file remains unchanged, fails to push. HOT 1
- Add ability to bypass .lfsconfig in git-lfs-filter-process, git-lfs-filter-clean and git-lfs-filter-smudge HOT 2
- Impossible to control git-lfs setup on git clone process. HOT 2
- filter-process initialization: strconv.ParseInt: parsing "\ufeff0": invalid syntax HOT 2
- GitHub knows my public key but I can't use LFS, why? HOT 2
- Git push rejected due to binary files despite already installing git lfs and tracking the file HOT 3
- Can't untrack large file, and push it to github. HOT 7
- error: external filter 'git-lfs filter-process' failed HOT 1
- git lfs filee not showing after pushing HOT 2
- error: could not lock config file HOT 5
- git add adds file to LFS even when file does not match the LFS filter HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from git-lfs.