git deps with limit

[openchung]

git deps with limit

[aspiers]

@openchung commented on September 10, 2020 4:12 PM:

We use the branch strategy to merge each Feature to the SIT branch, and use git deps to find all the dependencies at each SIT Commit point, Cherry-pick to the UAT branch, but we found that when the amount of commit is quite large, git deps is not work to produce result.

What do you mean exactly here? What doesn't work?

Is there a limitation for git deps?

Yes there are limitations. It only detects textual dependencies which does not guarantee correctness, and this can also sometimes result in a very large dependency tree which can potentially be avoided by manually porting commits between branches in a way which minimises dependencies.

If not, what is the possibility that the results will not run out? Thanks

It is possible that the dependency tree will be extremely large. You should make sure you are using -e to exclude any commits which have already been ported to the new branch.

[futureweihk]

Sir,

For example, in above image, the commit point contains around 360 addition and 688681 deletion, and when try to find the git deps point, it running a very long time (over 30mins and no response), is that the limitation you mentioned? are there any method to enhance the speed/performance, many thanks.

[futureweihk]

"It only detects textual dependencies" is that mean the special character in the file will produce error?

For example, sometimes the result is like above, showing utf-8 can not decode error

[aspiers]

@futureweihk commented on September 14, 2020 10:14 AM:

Sir,

For example, in above image, the commit point contains around 360 addition and 688681 deletion, and when try to find the git deps point, it running a very long time (over 30mins and no response), is that the limitation you mentioned? are there any method to enhance the speed/performance, many thanks.

If the commit is this big, something sounds badly wrong to me. Commits must be as small as possible. git-deps will not work well with huge commits.

[aspiers]

@futureweihk commented on September 14, 2020 10:20 AM:

"It only detects textual dependencies" is that mean the special character in the file will produce error?

For example, sometimes the result is like above, showing utf-8 can not decode error

No that is a completely different issue. It is very important to understand the difference between textual and semantic dependencies, so please read https://github.com/aspiers/git-deps#textual-vs-semantic-independence

[futureweihk]

@futureweihk commented on September 14, 2020 10:14 AM:

Sir,

For example, in above image, the commit point contains around 360 addition and 688681 deletion, and when try to find the git deps point, it running a very long time (over 30mins and no response), is that the limitation you mentioned? are there any method to enhance the speed/performance, many thanks.

If the commit is this big, something sounds badly wrong to me. Commits must be as small as possible. git-deps will not work well with huge commits.

Thanks for your clarification, could you please suggest are there any suggested limit, like no more than 50 files and 10000 add or deletions?
is the performance related to buffering, I mean when the tree expands, maybe the buffered data is overed the capacity, are there any parameter(maybe in some config file?) to set? thanks

[aspiers]

@futureweihk commented on September 15, 2020 7:27 AM:

Commits must be as small as possible. git-deps will not work well with huge commits.

Thanks for your clarification, could you please suggest are there any suggested limit, like no more than 50 files and 10000 add or deletions?

If it's code then usually I'd say no more than 5 or 10 files and 50 to 100 adds or deletions.

Here are some good links to read:

• https://sethrobertson.github.io/GitBestPractices/
• https://wiki.openstack.org/wiki/GitCommitMessages#Structural_split_of_changes
• https://seesparkbox.com/foundry/atomic_commits_with_git

is the performance related to buffering, I mean when the tree expands, maybe the buffered data is overed the capacity, are there any parameter(maybe in some config file?) to set? thanks

No it's nothing to do with buffering. It's a combination of git blame being expensive and the fact that the dependency tree will often mushroom in complexity. This is absolutely expected when not following best practices for small commits.

[futureweihk]

Many thanks for your patient and detailed responses, I will learn from your reference links first.

@futureweihk commented on September 15, 2020 7:27 AM:

Commits must be as small as possible. git-deps will not work well with huge commits.

Thanks for your clarification, could you please suggest are there any suggested limit, like no more than 50 files and 10000 add or deletions?

If it's code then usually I'd say no more than 5 or 10 files and 50 to 100 adds or deletions.

Here are some good links to read:

• https://sethrobertson.github.io/GitBestPractices/
• https://wiki.openstack.org/wiki/GitCommitMessages#Structural_split_of_changes
• https://seesparkbox.com/foundry/atomic_commits_with_git

is the performance related to buffering, I mean when the tree expands, maybe the buffered data is overed the capacity, are there any parameter(maybe in some config file?) to set? thanks

No it's nothing to do with buffering. It's a combination of git blame being expensive and the fact that the dependency tree will often mushroom in complexity. This is absolutely expected when not following best practices for small commits.

[futureweihk]

2020-09-16 14:24:51,520 WARN [runner.ScriptBindingsManager]: Error executing command: sh /log/jirasoftware-home/scripts/gitdeps.sh 2659_gib 3f9d22578c5292d293b210f24c6545180616cd5b UAT_GIB.202010.01_cherrypick_refs
Traceback (most recent call last):
File "/usr/local/bin/git-deps", line 8, in
sys.exit(run())
File "/usr/local/lib/python3.6/site-packages/git_deps/cli.py", line 141, in run
main(sys.argv[1:])
File "/usr/local/lib/python3.6/site-packages/git_deps/cli.py", line 135, in main
cli(options, args)
File "/usr/local/lib/python3.6/site-packages/git_deps/cli.py", line 119, in cli
detector.find_dependencies(rev)
File "/usr/local/lib/python3.6/site-packages/git_deps/detector.py", line 122, in find_dependencies
self.find_dependencies_with_parent(dependent, parent)
File "/usr/local/lib/python3.6/site-packages/git_deps/detector.py", line 147, in find_dependencies_with_parent
self.blame_hunk(dependent, parent, path, hunk)
File "/usr/local/lib/python3.6/site-packages/git_deps/detector.py", line 172, in blame_hunk
blame = subprocess.check_output(cmd, universal_newlines=True)
File "/usr/local/lib/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/usr/local/lib/python3.6/subprocess.py", line 425, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "/usr/local/lib/python3.6/subprocess.py", line 850, in communicate
stdout = self.stdout.read()
File "/usr/local/lib/python3.6/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 498: invalid start byte

Dear Sir, have you ever seen error message like above? we can not find the reason for such error

[aspiers]

This looks like a Python 3 / UTF-8 issue which is unrelated to the discussion above. Please file a separate issue for this.

[aspiers]

As the above questions have been answered, I will close this issue now. We can cover other topics in separate issues.

[futureweihk]

OK，thanks

As the above questions have been answered, I will close this issue now. We can cover other topics in separate issues.