Giter Site home page Giter Site logo

gut's Introduction

gut

This is a Git(Hub) multirepo maintenance tool, designed specifically for Divvun. But it should be quite useful to others needing to maintain tens (or hundreds) of similarly structured github repositories.

Using the gut apply -s <script> command, one can in practice run any git command on all repos (or a suitable subset, regex-selected on the repo names), not only the commands directly provided by gut.

We think it's pretty cool.

Installation

  1. get Rust
  2. clone this repo: git clone https://github.com/divvun/gut.git
  3. cd gut
  4. cargo install --path .

Alternatively - download a precompiled binary from nightly builds:

Extract the archive, and move the binary to somewhere on your $PATH.

Setup

  1. make a personal access token in GitHub - allow everything. Make sure to store it in a safe place - the token replaces your username and password when accessing GitHub via gut.
  2. run gut init -r <root-dir> -t <token>

<token> is the token created in step 1. above. The <root-dir> is the directory where you want to store all repos processed by gut. Below the <root-dir> dir, there will be one directory for every organisation you interact with, and within the organisation directory all repos are stored.

SSH access over the git protocoll

To use the git/ssh protocol, you need to set up an ssh key for GitHub. Follow these instructions.

Usage

NB! Please note that this is a potentially very powerful tool. Some commands require organisation owner permissions, and the most dangerous ones will require an explicit confirmation. If you get an error that the operation is not permitted, you probably do not have sufficient access to the repos involved.

There are some usage instructions under development.

Then there are some use cases with example commands here.

There is also some rudimentary help text. Run gut --help to get an overview.

In version 0.1.0 it reads:

$ gut --help         
gut 0.1.0
git multirepo maintenance tool

USAGE:
    gut <SUBCOMMAND>

FLAGS:
    -h, --help       
            Prints help information

    -V, --version    
            Prints version information


SUBCOMMANDS:
    add         Add users, repos to an organisation/a team
    apply       Apply a script to all local repositories that match a pattern
    branch      Set default, set protected branch
    checkout    Checkout a branch all repositories that their name matches a pattern or a topic
    ci          
    clean       Do git clean -f for all local repositories that match a pattern
    clone       Clone all repositories that matches a pattern
    commit      Add all and then commit with the provided messages for all repositories that match a pattern or a topic
    create      Create team, discussion, repo to an organisation or create a branch for repositories
    fetch       Fetch all local repositories that match a regex
    help        Prints this message or the help of the given subcommand(s)
    hook        Create, delete hooks for all repositories that match a pattern
    init        Init configuration data
    invite      Invite users to an organisation by emails
    make        Make repositories that match a regex become public/private
    merge       Merge a branch to the current branch for all repositories that match a pattern
    pull        Pull the current branch of all local repositories that match a regex
    push        Push the provided branch to remote server for all repositories that match a pattern or a topic
    remove      Remove users, repos from an organisation/a team
    set         Set information, secret for repositories or permission for a team
    show        Show config, list of repositories or users
    status      Show git status of all repositories that match a pattern
    template    Apply changes or generate new template
    topic       Add, get, set or apply a script by topic
    transfer    Transfer repositories that match a regex to another organisation
    workflow    Run a workflow

Help text for subcommands with further details reads:

SUBCOMMANDS with additional arguments:
    add
        repos       - Add all matched repositories to a team by using team_slug
        users       - Invite users by users' usernames to an organisation
    branch
        default     - Set a branch as default for all repositories that match a pattern
        protect     - Set a branch as protected for all local repositories that match a pattern
    ci          Continuous Integration
        export      - export data file for ci generate command
        generate    - generate ci for every repositories that matches
    create      Create team, discussion, repo to an organisation or create a branch for repositories
        branch      - Create a new branch for all repositories that match a regex or a topic
        discussion  - Create a discussion for a team in an organisation
        repo        - Create new repositories in an organisation and push for existing git repositories
        team        - Create a new team for an organisation
    hook        Create, delete hooks for all repositories that match a pattern
        create      - Create web hook for repos matching regex
        delete      - Delete all web hooks for all repository that match regex
    invite      Invite users to an organisation by emails
        users       - Invite users to an organisation by emails
    make        Make repositories that match a regex become public/private
        private    
        public     
    remove      Remove users, repos from an organisation/a team
        repositories    
        users       - Remove users by users' usernames from an organisation
    set         Set information, secret for repositories or permission for a team
        info        - Set description and/or website for all repositories that match regex, plain text or using a script
                      NB! Make sure there is no trailing newline at the end! Or it will fail.
        organisation- Set default organisation name for every other command
        permission  - Set access permissions for a team, for repos matching regex; matching repos will be added if not already in the team
        secret      - Set a secret all repositories that match regex
    show        Show config, list of repositories or users
        config      - Print configuration
        repositories- Show all repos matching regex   
        users       - Show all users in an organisation
    template    Apply changes or generate new template
        apply       - Apply changes from template to all repos that match the regex
        generate    - Generate a new project from a template
    topic       Add, get, set or apply a script by topic
        add      Add topics for all repositories that match a regex
        apply    Apply a script to all repositories that has a topics that match a pattern Or to all repositories that has a specific topic
        get      Get topics for all repositories that match a regex
        set      Set topics for all repositories that match a regex
    workflow    Run a workflow
        run         - Rerun the most recent workflow or send a repository_dispatch event to trigger workflows

gut's People

Contributors

bbqsrc avatar flammie avatar hamfor avatar killercup avatar lenguyenthanh avatar phaqui avatar snomos avatar trondtr avatar zoeyr avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

gut's Issues

Process repos in alphabetical order

It would be nice if processing repos matching a regex can be done in alphabetical order. It helps in estimating how far a timeconsuming process has come, and it makes it much easier to navigate the output when searching for issues.

dadmin apply --script <script>should pass target repo argument to script

dadmin apply --script <script>should pass target repo argument to script, so that in e.g. a shell script, $1 holds the absolute path to the target repo at hand. From the absolute path it is easy to extract various bits of information useful to the script, like language code, repo name, path to repo, etc.

Without this information the script is much restricted in what it can do.

Improved summary for `gut status`

The present summary looks like the following:

| lang-sqi            develop  0        0 0 3 0 0 |
| lang-nso            develop  0        0 0 3 0 0 |
| =====                                           |
| Total                        1        112       |
+-------------------------------------------------+

It would be very useful to see the total for each column, to resemble something like the following:

| lang-sqi            develop  0        0 0   3 0 0 |
| lang-nso            develop  0        0 0   3 0 0 |
| =====                                             |
| Total: 112                   1        0 0 330 0 0 |
+---------------------------------------------------+

Each column should be space-padded to the leflt, so that it right-align, including the total.

Also, the status report would be much easier to read if it is alphabetically sorted, cf #91 .

Setting website makes repo public

I ran the following command:

$ dadmin set info -o giellalttmp -r "lang-" -w https://giellalt.uit.no

It returned the following info:

 2020-05-04T08:30:02.797Z DEBUG dadmin > Arguments: Args { command: Set(Info(InfoArgs { organisation: "giellalttmp", regex: Filter { regex: lang- }, description: None, website: Some("https://giellalt.uit.no") })) }
 2020-05-04T08:30:02.798Z INFO  dadmin::user > User path: Some("/Users/smo036/Library/Preferences/dadmin/user.toml")
 2020-05-04T08:30:07.222Z DEBUG dadmin::github::rest > Patch: https://api.github.com/repos/giellalttmp/lang-sma
Set info for repo lang-sma successfully

Everything looks ok. But then some events where fired for each repo:

image

Adding a website URL does not imply that the repo is public.

create repo from svn-cloned git repo fails on push

Steps to reproduce:

  1. PERL_UNICODE=D git svn clone https://gtsvn.uit.no/langtech/trunk/langs/sme --authors-file=svn2git-authors.txt lang-sme/
  2. target/debug/dadmin create repo -o giellalttmp -r lang-.* -d /Users/smo036/svn2git/

The file svn2git-authors.txt is attached. /Users/smo036/svn2git/ is the local dir in which lang-sme is located. Replace with the actual path from step 1.

Step 1. stops with an error at svn rev 85005:

r85005 = 5ffc6c641376643c9feeea02a4abb3fea7e0f3ad (refs/remotes/git-svn)
tools/grammarcheckers/errordata/error-áidna.txt was not found in commit 5ffc6c641376643c9feeea02a4abb3fea7e0f3ad (r85005)

The resulting git repo still seems to be just fine (just lacking a 100k commits), and in size it should be good enough for real-world testing.

The result is the following:

$ target/debug/dadmin create repo -o giellalttmp -r lang-.* -d /Users/smo036/svn2git/
 2020-04-07T06:42:07.265Z DEBUG dadmin > Arguments: Args { command: Create(Repo(CreateRepoArgs { organisation: "giellalttmp", regex: Filter { regex: lang-.* }, dir: Some(Directory { path: "/Users/smo036/svn2git/" }), public: false, use_https: false, no_push: false })) }
 2020-04-07T06:42:07.265Z DEBUG dadmin::commands::create_repo > Create Repo CreateRepoArgs { organisation: "giellalttmp", regex: Filter { regex: lang-.* }, dir: Some(Directory { path: "/Users/smo036/svn2git/" }), public: false, use_https: false, no_push: false }
 2020-04-07T06:42:07.267Z DEBUG dadmin::commands::create_repo > Filtered sub dirs: ["/Users/smo036/svn2git/lang-sme"]
 2020-04-07T06:42:07.269Z INFO  dadmin::user                  > User path: Some("/Users/smo036/Library/Preferences/dadmin/user.toml")
 2020-04-07T06:42:07.302Z DEBUG dadmin::github::rest          > POST: https://api.github.com/orgs/giellalttmp/repos
 2020-04-07T06:42:09.039Z DEBUG dadmin::commands::create_repo > new created repo: "https://github.com/giellalttmp/lang-sme"
 2020-04-07T06:42:09.043Z DEBUG dadmin::git::push             > Branches ["git-svn"]
 2020-04-07T06:42:10.607Z DEBUG dadmin::git::push             > Push result Err(Error { code: -1, klass: 4, message: "src refspec \'refs/heads/git-svn\' does not match any existing object" })
Created repo for lang-sme successfully at: https://github.com/giellalttmp/lang-sme

The reported branch is the only one it seems:

git branch -a
  remotes/git-svn

Env. info:

  • latest dadmin source
  • macOS 10.14.6.

svn2git-authors.txt

Assign repo X to group Y

If a new group Y is a production group, repo X should automatically be added to CD for group Y

dadmin push should not push repos with no local changes

In recent work on keyboards in github.com/giellalt, I have noticed that dadmin push will push every matched repo even when there is nothing to push. While this is harmless(?), it produces noise in the form of messages for repos with no real change, which makes it harder to visually verify that all and only real commits and changes have been published.

Git repo syncing (aka template update pushing)

  • apply changes in template repo X to all repos [matchin pattern Y], for changes since Yn was last changed by the same type of update [manually override timestamp by explicit time range specification]; timestamp of changes are tracked in a specific file (the present svn solution looks at the last commit revision of the file und.timestamp)1
    • Silver Platter might be useful here, at least as inspiration
    • typical usecase: new functionality is added to template X in the form of a new dir, new files, and modifications to existing files. After the changes are pushed to the github template repo, the operation described here is used to merge the changes in the template repo with all repos [matching the pattern], thus causing the new feature to be available in all the repos.
    • see the existing merge code for how this is implemented in the present, subversion-based system.
  • initialise new language repo with data from template repo X, automatically adding CI, and stubs for CD

Define different level of output message

To make our app easy to use, we need to define and follow a structure level of output messages. I suggest we have 4 levels and their colors:

- Success - Green: Like: Cloned a repository success
- Info - White: To show information like: start to clone a repository
- Warning - Yellow : Some thing not right but still work
- Error - Red: Something went wrong, failed.

List of fixed commands:

  • create_branch
  • push

Running dadmin create repo leads to asking for ssh key

Since dadmin has been set up with a token, I would expect dadmin would use the token and push repos using https, instead of trying to push repos by ssh.

$ dadmin create repo --dir . -o giellalttmp -r lang-nio
 2020-04-07T22:09:14.274Z DEBUG dadmin > Arguments: Args { command: Create(Repo(CreateRepoArgs { organisation: "giellalttmp", regex: Filter { regex: lang-nio }, dir: Some(Directory { path: "." }), public: false, use_https: false, no_push: false, override_origin: false })) }
 2020-04-07T22:09:14.274Z DEBUG dadmin::commands::create_repo > Create Repo CreateRepoArgs { organisation: "giellalttmp", regex: Filter { regex: lang-nio }, dir: Some(Directory { path: "." }), public: false, use_https: false, no_push: false, override_origin: false }
 2020-04-07T22:09:14.274Z DEBUG dadmin::commands::create_repo > Filtered sub dirs: ["./lang-nio"]
 2020-04-07T22:09:14.274Z INFO  dadmin::user                  > User path: Some("/home/boerre/.config/dadmin/user.toml")
 2020-04-07T22:09:14.281Z DEBUG dadmin::github::rest          > POST: https://api.github.com/orgs/giellalttmp/repos
 2020-04-07T22:09:15.848Z DEBUG dadmin::commands::create_repo > new created repo: "https://github.com/giellalttmp/lang-nio"
 2020-04-07T22:09:15.852Z DEBUG dadmin::git::push             > Branches ["master"]
Enter passphrase for key '/home/boerre/.ssh/id_rsa': [hidden]

dadmin push command gives a bad error message

I have a bunch of directories starting with lang-, which I try to push with the command

dadmin push --branch master --organisation giellalttmp --regex lang-

The result of that operation is:

 2020-04-07T21:03:41.458Z DEBUG dadmin > Arguments: Args { command: Push(PushArgs { organisation: "giellalttmp", regex: Filter { regex: lang- }, branch: "master" }) }
 2020-04-07T21:03:41.458Z DEBUG dadmin::commands::push > push branch PushArgs { organisation: "giellalttmp", regex: Filter { regex: lang- }, branch: "master" }
 2020-04-07T21:03:41.458Z INFO  dadmin::user           > User path: Some("/home/boerre/.config/dadmin/user.toml")
 2020-04-07T21:03:41.458Z INFO  dadmin::config         > Conifg path: Some("/home/boerre/.config/dadmin/app.toml")
Error: No such file or directory (os error 2)
$ dadmin show config
 2020-04-07T21:03:52.467Z DEBUG dadmin > Arguments: Args { command: Show(Config) }
 2020-04-07T21:03:52.467Z INFO  dadmin::user > User path: Some("/home/boerre/.config/dadmin/user.toml")
 2020-04-07T21:03:52.467Z INFO  dadmin::config > Conifg path: Some("/home/boerre/.config/dadmin/app.toml")
Username: albbas
Github token: xxx (hidden the real value)
Root directory: /home/boerre/dadmin

Entering one of the directories show this info:

$ cd lang-sjd
lang-sjd $ git branch -a
* master
  remotes/git-svn

Delete all repos from org [matching regex]

It should be possible to delete repos from an org. But with warnings and an extra confirmation. E.g. something like:

dadmin remove repo -r <regex> -o <org>
The following repos will be removed:
repo A
repo B
repo C
Proceed? y/N

gut set webhook

To manage interaction with our Zulip discussion board (giella.zulipchat.com), we need a way to set webhooks.

The core arguments of the operation are:

  • URL (fixed or dynamic, see below)
  • delivery method (json or form-urlencoded)
  • event trigger list (push, everything, list of all possible events)

The URL should be either a fixed string, or the output of a script/command which is given the repo name as input value.

gut Set info needs parameter for language code/name/callback function

When setting info like the description or a url, typically the text will be identical for all matching repos except for either the language code or the language name. The command should be enhanced with an option for a call back function/script or similar, that will get the repo name as input, and return a string, which can then be inserted in the description or url text using a placeholder.

Assign team X to group Y

Does this mean assign team X for all repositories on group Y?
What happen if we add/remove new repositories to group later?

gut hook create does not work

$ gut hook create -m json -o giellalttmp -r 'lang-' -e "*" -s /Users/smo036/langtech/gut/giellalt/giella-core/devtools/gut-scripts/reponame2webhook.sh 
 2020-05-20T11:35:35.467Z DEBUG gut > Arguments: Args { command: Hook(Create(CreateArgs { organisation: "giellalttmp", regex: Filter { regex: lang- }, url: None, method: Json, script: Some(Script { path: "/Users/smo036/langtech/gut/giellalt/giella-core/devtools/gut-scripts/reponame2webhook.sh" }), events: ["*"] })) }
 2020-05-20T11:35:35.469Z INFO  gut::user > User path: Some("/Users/smo036/Library/Preferences/gut/user.toml")
$ 

Probably due to the same issues as with commit, push etc.

dadmin make private requires arguments in specific order

dadmin make private -o giellalttmp -r "(lang-|giella-)"
error: Found argument '-o' which wasn't expected, or isn't valid in this context

USAGE:
    dadmin make --regex <regex> private

For more information try --help

Placing the private command at the end made the command work:

dadmin make -o giellalttmp -r "(lang-|giella-)" private
 2020-05-04T08:41:03.827Z DEBUG dadmin > Arguments: Args { command: Make(MakeArgs { visibility: Private, organisation: "giellalttmp", regex: Filter { regex: (lang-|giella-) } }) }
 2020-05-04T08:41:03.828Z INFO  dadmin::user > User path: Some("/Users/smo036/Library/Preferences/dadmin/user.toml")
The following repos will be changed to private:
giellalttmp/lang-lut
giellalttmp/lang-otw
giellalttmp/lang-sma
...

Expected behavior: it should work independently of the order of the arguments.

Further improvements to gut status

The present footer looks like this:

| lang-zul-x-exp      develop           0  0  0  0  0  0 |
| lang-zxx            develop           0  0  0  0  0  0 |
| ================                                       |
| Repo Count          Dirty    fetch/push                |
| 113                 0                 0  0  0  0  0  0 |
+--------------------------------------------------------+

It would be nice to repead the column header again, with many repos matching it is hard to remember. That is, something like this:

| lang-zul-x-exp      develop           0  0  0  0  0  0 |
| lang-zxx            develop           0  0  0  0  0  0 |
| ================                                       |
| Repo Count          Dirty    fetch/push  U  D  M  C  A |
| 113                 0                 0  0  0  0  0  0 |
+--------------------------------------------------------+

Set GitHub description/website/topics for repos [matching regex]

Possibly with some placeholder/variable/stuff that can be replaced individually for each repo using an external function, like calling a script to turn the regex repo pattern into a language name.

This is useful both when moving the svn repos to github and when setting up a new language.

Clean Git directory command

Usage

dadmin clean -o <org> -r <pattern>

Effect

This command will have the effect of git clean -f -d for all local repositories that matches the pattern.

I think this command is useful because sometime we want to revert all changes after using apply command.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.