dfetch-org/dfetch

Dependency fetcher

dfetch's Issues

Default branch not explicitly mentioned after import

Originally reported in #61:

Branch not explicit when it is the default branch (master/trunk), this was a implicit design decision, that I'm willing to re-consider.

Recursive manifests (import other manifests)

In Zephyr West it is possible to import other manifests (see docs).

This could help in having separate manifests for different parts of the code.

dfetch check interpret tags

Make it possible to interpret semver tags.

Semantically versioned tags could be interpreted. If for instance tag v3.4 is pinned but a tag v3.5 is available, notify the user that a 'newer' version is available.

Generate changelog diff

Have to think out the flow a little bit more, but as a rough idea:

• user runs dfetch check
• notices a dependency was updated
• runs dfetch check --with-changelist
• for each project with a changes or changelist document at top-level show the textual diff between the current version and the available version.

dfetch import in subfolder breaks on windows

Given a git project with submodules
When dfetch import is done
Then on windows the a FileNotFoundError is given

This is due to that in git.py Git.submodules() smpath is relative to cwd and not toplevel.

                branch = GitRepo._guess_branch_of_sha(
                    os.path.join(toplevel, sm_path), sha, logger
                )

Should become

                branch = GitRepo._guess_branch_of_sha(
                    os.path.join(os.getcwd(), sm_path), sha, logger
                )

Don't mandate remote section

When dfetch is used for a project with only 1 project, the remote sections is useless overhead. Following should be possible

            manifest:
              version: '0.0'

              projects:
                - name: ext/test-repo-tag
                  url: https://github.com/dfetch-org/test-repo
                  tag: v1

Svn & git mandated

Dfetch assumes both git and svn are available on the system.

For example, If svn is not available in the path, dfetch import will fail.

Reported by @jgeudens

Documentation PDF build broken

Apparently the pdf build broke, see failing builds here.

For instance from build 12992713:

! TeX capacity exceeded, sorry [input stack size=5000].
\font@name ->
             \T1/phv/m/n/10 
l.1601 \end{tabulary}
                     ^^M
!  ==> Fatal error occurred, no output PDF file produced!

Checking out single file from SVN fails

Cannot check out a single file from SVN (metadata has no place to go)

Traceback (most recent call last):
  File "C:\Projects\EDNA\dfetch-git\venv\Scripts\dfetch-script.py", line 11, in <module>
    load_entry_point('dfetch', 'console_scripts', 'dfetch')()
  File "c:\projects\edna\dfetch-git\dfetch\__main__.py", line 58, in main
    args.func(args)
  File "c:\projects\edna\dfetch-git\dfetch\commands\update.py", line 41, in __call__
    dfetch.project.make(project, logger.getChild(project.name)).update()
  File "c:\projects\edna\dfetch-git\dfetch\project\vcs.py", line 39, in update
    safe_rmtree(self.local_path)
  File "c:\projects\edna\dfetch-git\dfetch\util\util.py", line 18, in safe_rmtree
    shutil.rmtree(path, onerror=_remove_readonly)
  File "C:\Users\bens\AppData\Local\Programs\Python\Python37\lib\shutil.py", line 516, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\bens\AppData\Local\Programs\Python\Python37\lib\shutil.py", line 377, in _rmtree_unsafe
    onerror(os.scandir, path, sys.exc_info())
  File "c:\projects\edna\dfetch-git\dfetch\util\util.py", line 12, in _remove_readonly
    func(path)
NotADirectoryError: [WinError 267] De mapnaam is ongeldig: 'dfetch/Programming/Binaries/Modloader.exe'

Preserve local changes during update

If #24 is implemented and it is possible to create a patch. Make it possible to automatically create a patch before a update and apply it using #22 .

Support for patch files

After a project is fetched, apply a patch to apply any local changes.

Add a new attribute patch: to the manifest which gives an absolute path or a path relative to the manifest with a patch file (unified-diff git/svn). This will be applied after fetching a project, but before hashing it.

Don't assume any tooling on the system and use a pure python library (https://libraries.io/pypi/patch)

Don't forget:

• The documentation in manifest.py explaining this new attribute.
• Feature/Unit tests
• Add patch: to the schema for validation to succeed.
• Add any external lib license to legal.rst (and make sure it is MIT compatible)

Add a dfetch remove command

Add a remove command to dfetch that removes a project from disk and from the manifest.

Use manifests found in projects

Given a project:

MyProject/

    main.c
    dfetch.yaml

With the following manifest:

dfetch:
    version: 0.0
    projects:
         name: SomeProject
         dst: ThirdParty/SomeProject

Where the other project also has a manifest for its own dependencies:

SomeProject/
    some_project.c
    dfetch.yaml

With the following manifest:

dfetch:
    version: 0.0
    projects:
         name: SomeOtherProject
         dst: ../SomeOtherProject

Referring to the following code:

SomeOtherProject/
    some_other_project.c

When dfetch update is done, this should result in:

MyProject/

    main.c
    dfetch.yaml

    ThirdParty/
          SomeProject/
              some_project.c
              dfetch.yaml

          SomeOtherProject/
              some_other_project.c

Version/name clashes

• To prevent name clashes, use scoping, imported project are prefixed with the projects name that imported them. In the top example SomeOtherProject is named SomeProject/SomeOtherProject.
• Let the toplevel manifest always have priority: If toplevel has a project with the destination, warn and ignore the 'imported' project
• If two imported manifests have clashing destinations: stop and let the user decide
• Provide a flag for disabling manifest importing

Add contribution process documentation

It is unclear how the branch / review and merge strategies are at the moment.
Add documentation to contributing.rst

Support for CI-systems

It should be easy to check if all projects are up-to-date.
CI-systems would like non-zero return code if dependencies are not up-to-date.
Optionally check for local changes (and suggest using the patch: in manifest)

Optionally a parseable output format. Such as:

• SARIF https://github.com/microsoft/sarif-tutorials + https://github.com/microsoft/sarif-python-om
• JSON https://github.com/jenkinsci/warnings-ng-plugin/blob/master/plugin/src/test/resources/io/jenkins/plugins/analysis/warnings/steps/json-issues.log

Make dfetch autocommit after update

If dfetch is used in a git repo and an update is performed allow for a flag to auto-commit it to the current repo

Add freeze command

Similar to pip freeze, create a command that takes all current (on disk) versions and adds revisions/branches or tags for each project.

Document structure

Currently the technical documentation lacks a good description of the structure.

dftech add command

Create a dfetch add command which adds a new project to the manifest. Potentially make it interactively so a user is guided.

Add dfetch diff command

Add a diff command, that can show any local changes made to the project after the fetch.

Fetch to local temp dir and do a diff. (optionally make it possible to generate a patch file)

Branch name is missing when branch isn't explicitly mentioned in manifest

When executing a check or update command and the manifest doesn't explicitly mentions a branch, no branch name is shown in the output. Dfecth will function correctly because it defaults to master, but the output is incomplete.

When the branch is added to the manifest, the master branch is mentioned correctly.

manifest:

  - name: tests/googletest
    revision: dcc92d0ab6c4ce022162a23566d44f673251eee4
    remote: github-com-google
    dst: tests/googletest
    repo-path: googletest.git

$dfetch update

Dfetch (0.0.5)
  libraries/muparser  : up-to-date (v2.3.2 - 207d5b77)
  tests/googletest    : up-to-date ( - dcc92d0a)


$ dfetch check
Dfetch (0.0.5)
  libraries/muparser  : up-to-date (v2.3.2 - 207d5b77)
  tests/googletest    : installed and pinned ( - dcc92d0a), available (273f8cb0)

check command returns short revision of latest commit

The check command shows the current and latest revision of the repo, but it only shows the short revision. As I understand, it is better to use the full revision, so at the moment I still need to manually check the latest (full) revision of a repo to put in my manifest.

It would be easier that the check command shows the full revision so it can be copied easily to the manifest.

Version equality not always correct

When tag is updated, dfetch does see this. (Problem is in equality check in dfetch/project/version.py

Below scenario fails:

    Scenario: Tag is updated in manifest
        Given the manifest 'dfetch.yaml'
            """
            manifest:
              version: '0.0'

              projects:
                - name: ext/test-repo-tag
                  url: https://github.com/dfetch-org/test-repo
                  tag: v1

            """
        And all projects are updated
        When the manifest 'dfetch.yaml' is changed to
            """
            manifest:
              version: '0.0'

              projects:
                - name: ext/test-repo-tag
                  url: https://github.com/dfetch-org/test-repo
                  tag: v2.0

            """
        And I run "dfetch update"
        Then the output shows
            """
            Dfetch (0.0.6)
              ext/test-repo-tag   : Fetched v2.0
            """

More helpfull error message during import of non-existent submodules

Originally reported in #61

When running dfetch import on git repo's with submodules that didn't git submodule update --init yet, the help text is not very helpful. Check when no submodules are found if a .gitmodules file is present.

Importlib ModuleNotFoundError python 3.7.0

dfetch uses importlib to find the reosurces in the package (template.yaml, schema).
Users with pythn 3.7.0 report:

ModuleNotFoundError: No module named 'importlib.resources'

When running dfetch nder python 3.7.0

importlib.resources should be part of 3.7.0, to be investigated.

Add gif recordings

ttyrec, asciicast and similar tools allow creating recordings (or even gifs) of commandline usage.

Find a way to automate this and add it to the documentation

Parse license of external.project

Similar to how libraries.io gives an overview of licenses, parse the license of a project using https://pypi.org/project/spdx-lookup/

SVN support for dfetch import

Currently import only supports Git.
Add support for SVN.

Add autodesk/quack to list of alternatives

www.github.com/autodesk/quack solves a similar problem in a very similar way, but only for git repositories and on *nix/osx.

Only sha of submodule instead of tag during dfetch import

I have used the dfetch import command to generate a manifest of my git submodules. One of my submodules was pinned to a tag. The generated manifest contains seemingly correct information. A tag is just a label to a specific revision on the master (in this case). So it is reasonable that the branch option isn't added because the default is master. And the correct specific revision is mentioned.

  projects:
  - name: libraries/muparser
    revision: 207d5b77c05c9111ff51ab91082701221220c477
    remote: https---github-com-beltoforion
    dst: libraries/muparser
    repo-path: muparser.git

But when using dfetch update, another file is generated. This file mentions the specific tag name and revision.

Content of .dfetch_data.yaml

dfetch:
  branch: v2.3.2
  last_fetch: 20/01/2021, 22:22:01
  remote_url: https://github.com/beltoforion/muparser.git
  revision: 207d5b77c05c9111ff51ab91082701221220c477

For me, it would make sense that the originally generated manifest also specifies the correct tag name as branch option. When using an external library, one would normally update to the next tag (release) and not follow every commit on master.

PS: the .dfetch_data.yaml doesn't seem to be mentioned in the documentation. Is this file an auto-generated file that can be ignored or a configuration file that should be added to the repo?

Guard against overwriting local changes

Before implementing #27, a mechanism to guard against overwriting local changes could improve the user experience

• After a fetch, hash the local project directory
• Add hash to metadata
• Before update, hash local directory, if hash is != stop
• Let user override with --force (or removing the directory)

Later on this mechanism can be used to trigger a new flow from #27

Add tag attribute to manifest

A separate tag: attribute can improve the code much more and make the priority rules more explicit.

svn branches or tags prefix required

Given an svn project, the branch must include either the branches or tags prefix. Below file works.
This however feels a little bit clunky.

manifest:
  version: 0.0

  remotes:
  - name: github
    url-base: https://github.com/cpputest/cpputest

  projects:
  - name: cpputest-tags
    branch: tags/v3.7                     # <<< tags explicit
    vcs: svn

  - name: cpputest-branches
    branch: branches/gh-pages            # <<< branches explicit
    vcs: svn

Add list command

Add a command that can list the current state of the projects.

For each project aggregate the metadata and list it.

Support for sparse checkouts

The manifest has a src keyword. It could be used to copy only a part of a project.
Make sure the license file is copied, to not go against license agreements.

Update command fails with specific revisions in manifest

When a specific revision is mentioned in the manifest, dfetch can't find the specific revision.

Tested with current master (31d250a)

Running git ls-remote https://github.com/google/googletest.git
Return code: 0
stdout:
...

stderr:

Running ['git', 'branch', '--contains', 'dcc92d0ab6c4ce022162a23566d44f673251eee4']
git branch --contains dcc92d0ab6c4ce022162a23566d44f673251eee4 returned 129:
error: no such commit dcc92d0ab6c4ce022162a23566d44f673251eee4

I have checked out the google test repo and ran the git branch --contains dcc92d0ab6c4ce022162a23566d44f673251eee4 command and it returned master correctly.

If required, you can use following branch to reproduce the issue: https://github.com/jgeudens/ModbusScope/tree/dfetch_try2

Describe user actions after import (transition guide)

Originally reported in #61

After dfetch import submodules or externals need to be removed. This is completly not described.
Although it is hard to keep a complete correct description always, at least document some pointers to information.

For example removing git submodules is notoriously hairy SO-1260748.

Branch guessing incorrect

Originally reported in #61

• Google test branch was incorrectly guessed (see commit 10474943)

Support for tar/zip files

Some projects are stored as zip or tar archives, support adding these to a project, maybe with sha/md5 hash verification.

Add integration tests

Recently several basic usecases failed.
Although unit testing is done and the commands are run in github actions, the checked-out contents is not verified.

• Create a test repo under the dfetch-org
• Add some popular usecases:
  • tags (with some semantic versions)
  • nested tags projA/v0.0.1 & projB/0.0.1
  • branches
  • commits in branches (not listed in refs)
• Add (feature) tests for at least update: svn + git confirming downloaded content == correct:
  • revision only
  • branch + revision (use revision --> warn if revision is not in branch)
  • branch only
  • no branch no revision
  • tag only
  • tag + revision (use revision --> warn if tag is not on revision?)
  • And above with src keyword (sparse checkouts)

'/' appended to ssh url

When using an git ssh url [email protected]:cpputest/cpputest.git, dfetch adds an extra '/' resulting in [email protected]:cpputest/cpputest.git/ which is not a legal url

Update command defaults to SVN and stops on error

I found an open source project (https://github.com/ChristopherWilks/megadepth) on Github with some git submodules to test dfetch.

It appears that the current master of dfetch (36a2e5b) prefers SVN and stops with an error when the SVN handling fails. The import command works after installing the subversion package. But the following update fails with following message

dfetch -v update
  Dfetch (0.0.5)
  Looking for manifest
  Using manifest /home/jens/Projects/megadepth/dfetch.yaml
  Running svn info https://github.com/samtools/htslib
  Return code: 0
  stdout:
  Path: htslib
  URL: https://github.com/samtools/htslib
  Relative URL: ^/
  Repository Root: https://github.com/samtools/htslib
  Repository UUID: 179373a4-1820-8b07-f7c7-ce57b7db7bff
  Revision: 5945
  Node Kind: directory
  Last Changed Author: valeriu.ohan
  Last Changed Rev: 5945
  Last Changed Date: 2020-09-22 14:15:49 +0200 (di, 22 sep 2020)
  
  stderr:
  
    - htslib              : fetching (master - a7a90fe9)
  fetching htslib a7a90fe913f8a466f32f6e284cf46653944acd6f from https://github.com/samtools/htslib
  Running svn export --force  https://github.com/samtools/htslib/branches/master htslib_ci
  svn export --force  https://github.com/samtools/htslib/branches/master htslib_ci returned 1:
  svn: E205000: Try 'svn help export' for more information
  svn: E205000: Error parsing arguments

Show line number when validation fails

When a manifest is validated, it only shows the path to the element. Not the actual line number.

Describe metadata file in documentation

Originally reported in #61

the .dfetch_data.yaml doesn't seem to be mentioned in the documentation. Is this file an auto-generated file that can be ignored > or a configuration file that should be added to the repo?

metadata file (.dfetch_data.yaml) never described in documentation (what it is and what to do with it).

Line-endings not configurable

When fetching a git repo or an svn repo on windows, the contents is the same, except for the line endings.
The user has no direct control.

Rename manifest.yaml to dfetch.yaml

Manifest.yaml is to generic leading to potential clashes, rename it to dfetch.yaml.

Add troubleshooting guide

When a user has an issue, it is a little unclear what he can do to investigate.

Add a troubleshooting guide

Support for pre-commit hooks

To guard against unwanted changes, it should be possible to create a pre-commit hook that checks if a file/folder is under control of dfetch.

Git short revisions ignored

Currently short revisions are ignored for git.

For now dfetch tries to fetch as efficiently as possible and it fetches only a specific sha/branch/tag. And in case of 'src' entry only the directory or file mentioned.

With git as of now it seems that it is only possible to fetch a sha if it is the complete 40-char version. Short revisions are not accepted.

One option would be to do an entire clone, checkout the short sha and warn the user that he should update his manifest with the long version.

Another option would be to find some magic git command that can transform a short sha to a long one on the remote repo. So something like

git rev-parse <short-sha>

Add tests

Current implementation was created as a proof-of-concept. To make the project more mature add tests to keep it working.