drone-plugins/drone-git

Drone plugin for cloning Git repositories

drone-git

Drone plugin to clone git repositories. For the usage information and a listing of the available options please take a look at the docs.

Build

Build the binary with the following command:

export GOOS=linux
export GOARCH=amd64
export CGO_ENABLED=0
export GO111MODULE=on

go build -v -a -tags netgo -o release/linux/amd64/drone-git

Docker

Build the Docker image with the following command:

docker build \
  --label org.label-schema.build-date=$(date -u +"%Y-%m-%dT%H:%M:%SZ") \
  --label org.label-schema.vcs-ref=$(git rev-parse --short HEAD) \
  --file docker/Dockerfile.linux.amd64 --tag plugins/git .

Usage

Clone a commit:

docker run --rm \
  -e DRONE_REMOTE_URL=https://github.com/garyburd/redigo.git \
  -e DRONE_WORKSPACE=/go/src/github.com/garyburd/redigo \
  -e DRONE_BUILD_EVENT=push \
  -e DRONE_COMMIT_SHA=d8dbe4d94f15fe89232e0402c6e8a0ddf21af3ab \
  -e DRONE_COMMIT_REF=refs/heads/master \
  plugins/git

Clone a pull request:

docker run --rm \
  -e DRONE_REMOTE_URL=https://github.com/garyburd/redigo.git \
  -e DRONE_WORKSPACE=/go/src/github.com/garyburd/redigo \
  -e DRONE_BUILD_EVENT=pull_request \
  -e DRONE_COMMIT_SHA=3b4642018d177bf5fecc5907e7f341a2b5c12b8a \
  -e DRONE_COMMIT_REF=refs/pull/74/head \
  plugins/git

Clone a tag:

docker run --rm \
  -e DRONE_REMOTE_URL=https://github.com/garyburd/redigo.git \
  -e DRONE_WORKSPACE=/go/src/github.com/garyburd/redigo \
  -e DRONE_BUILD_EVENT=tag \
  -e DRONE_COMMIT_SHA=3b4642018d177bf5fecc5907e7f341a2b5c12b8a \
  -e DRONE_COMMIT_REF=refs/tags/74/head \
  plugins/git

drone-git's Issues

Can't clone on deployment events ($DRONE_BUILD_EVENT == deployment)

This is technically GitHub's fault, but I believe we can have a workaround:
when we trigger a GitHub deployment using a tag ref, the webhook payload
doesn't state that the deployment has been created from a tag, and Drone treats
it as a branch (and GitHub makes it even worse when there's a tag and a branch
with the same name, see below).

How to reproduce

Create a GitHub deployment using a tag ref, wait for GitHub to notify Drone and
see the clone step failing.

Using plugins/git:next:

nitialized empty Git repository in /drone/src/github.com/nytm/drone-gh-deploy-exp/.git/
+ git fetch origin +refs/heads/v0.0.2:
fatal: Couldn't find remote ref refs/heads/v0.0.2

Using plugins/git:latest:

+ git init
Initialized empty Git repository in /drone/src/github.com/nytm/drone-gh-deploy-exp/.git/
+ git remote add origin https://github.com/nytm/drone-gh-deploy-exp.git
+ git fetch --no-tags origin +refs/heads/v0.0.9:
fatal: Couldn't find remote ref refs/heads/v0.0.9
retry in 5s
+ git fetch --no-tags origin +refs/heads/v0.0.9:
fatal: Couldn't find remote ref refs/heads/v0.0.9
retry in 5s
+ git fetch --no-tags origin +refs/heads/v0.0.9:
fatal: Couldn't find remote ref refs/heads/v0.0.9
retry in 5s
+ git fetch --no-tags origin +refs/heads/v0.0.9:
fatal: Couldn't find remote ref refs/heads/v0.0.9
retry in 5s
+ git fetch --no-tags origin +refs/heads/v0.0.9:
fatal: Couldn't find remote ref refs/heads/v0.0.9
retry in 5s
+ git fetch --no-tags origin +refs/heads/v0.0.9:
fatal: Couldn't find remote ref refs/heads/v0.0.9
time="2019-02-05T16:05:45Z" level=fatal msg="exit status 128"

(Notice how the plugin is looking for refs/heads/v0.0.9 when it should be
looking for refs/tags/v0.0.9).

The bug

Technically, there are 2 cases where the plugin misbehaves:

1. the one described above: deploy from a tag, plugins looks for a branch,
   can't find it, build fails
2. when there's actually a branch with the tag name, the plugin silently
   fetches the branch and deploy that instead of the tag.

One workaround we've been using is to fetch both refs/heads/ and
refs/tags/ then checking out $DRONE_COMMIT. BUT $DRONE_COMMIT is
actually incorrect for the second case, because GitHub sends the wrong commit
SHA to Drone! (may be related to
github-ref-bug). So, in that case,
we have an extra step that actually fetches the tag and use it when the
deployment was triggered from a tag event (in our case, environment: production):

clone:
  git:
    image: fsouza/drone-git
    pull: true

pipeline:
  checkout-tag-production-deployment:
    image: something-with-git
    commands:
      - git fetch origin +refs/tags/${DRONE_BRANCH}:
      - git checkout FETCH_HEAD -b production-branch
    when:
      event: deployment
      environment: production

(If you want to see what fsouza/drone-git is doing:
drone-plugins:next...fsouza:fix-tag-deployment)

That solution can't be generalized though, so I wonder what should be the
actual solution for that case. Some ideas:

• Solution 1: on deployment events, fetch both refs/heads/$DRONE_BRANCH
  and refs/tags/$DRONE_BRANCH and if both succeed, fail with an error
  message saying that having branches and tags with the same name isn't
  supported.
• Solution 2: on deployment events, fetch both
  refs/heads/${DRONE_BRANCH} and refs/tags/${DRONE_BRANCH}, checkout
  the wrong $GIT_COMMIT and print a warning or something like that? (I bet
  users would ignore the warning, or not even see it, so this is dangerous).
• Solution 3: leave it alone. Wait for GitHub to fix the issue and have users use a fork of
  drone-git (what we're currently doing). Once GitHub fixes the issue, Solution
  2 would be the way to go because then we'd be checking out the correct
  ${DRONE_COMMIT}.
• Something else?

[Documentation] DOCS.md update: "submodule_update_remote" should be "submodules_update_remote"

DOCS.md references the plugin parameter submodule_update_remote when it should really be submodules_update_remote.

• https://github.com/drone-plugins/drone-git/blob/master/DOCS.md#parameter-reference

As a result, this documentation is incorrect: http://plugins.drone.io/drone-plugins/drone-git/

Link to the docs is dead

https://github.com/drone-plugins/drone-git/blob/master/README.md#drone-git
= > "the docs" points to https://github.com/drone-plugins/drone-git/blob/master/DOCS.md which doesn't exist anymore => it was removed by this commit dd2b971

Wrong git hash being built on PR synchronization

After pushing a new commit to a branch, Drone picks up the change but checks out the wrong git hash. Notice in the attached image that the git hash in the header (it's actually the correct one) is different from the output of git log in the build step.

Support url.insteadOf configuration

instead of submodule override we might be able to set a global configuration to remap the git+ssh url to git+https url

git config --global url."https://github.com/".insteadOf [email protected]:
git config --global url."https://".insteadOf git://

HOW TO SETTING PERSONAL GITLAB HOSTS?

I create my own gitlab.
My drone-server can find my gitlab.
I got a fail when git clone my repo.
The error is it cat't find my gitlab domain.
I can't setting my hosts in this image.
Is it can be setting?

New release with updated curl?

I've been running into problems with too long passwords in netrc due to a curl issue. It's been fixed in curl/curl#2676 and released in curl 7.61.0, which is already on the alpine repos.

Could we get a release of the plugins/git image with the updated curl?

Thank you!

Preserve original branch the commit checked out is from

Branch in build environment is always "master", even if that wasn't the branch that triggered the build. Scripts that case behavior based on branch or otherwise introspect the git repository state are broken by this behavior.

Support for sparse checkout

moved from harness/gitness#1849

---

Thanks for providing such an awesome service/project! I really love your simple UI design!

As the main maintainer of a super huge/fat project - @cdnjs, I'm wondering if you would like to support git sparseCheckout. By default, git will checkout all the files from the repository to the filesystem, by sparseCheckout, we can decide to checkout partial of the files, so this will help decrease the number of files git will checkout, which will totally save both disk sapce, cpu time, build time!

There is only a small different steps between git clone and git, but would help a lot for a huge repo 😄

Reference:

• https://git-scm.com/docs/git-read-tree
• https://github.com/cdnjs/cdnjs/blob/master/documents/sparseCheckout.md

resolver does not work correctly.

Hello,

resolver on git plugin is not working correctly.

Error message:

+ git init
Initialized empty Git repository in /drone/src/git.domain.com/apl/test/.git/
+ git remote add origin https://git.domain.com/apl/test.git
+ git fetch --no-tags --depth=50 origin +refs/heads/master:
fatal: unable to access 'https://git.domain.com/apl/test.git/': Could not resolve host: git.domain.com
exit status 128

worker's logs:

{"time":"2018-04-03T06:45:16Z","level":"debug","message":"request next execution"}
go package net: GODEBUG setting forcing use of Go's resolver
go package net: hostLookupOrder(drone.domain.com) = files,dns
{"time":"2018-04-03T06:45:29Z","level":"debug","repo":"apl/test","build":"73","id":"196","message":"received execution"}
{"time":"2018-04-03T06:45:29Z","level":"debug","repo":"apl/test","build":"73","id":"196","message":"listen for cancel signal"}
{"time":"2018-04-03T06:45:29Z","level":"debug","repo":"apl/test","build":"73","id":"196","image":"plugins/git:latest","stage":"git","exit_code":0,"exited":false,"message":"update step status"}
{"time":"2018-04-03T06:45:29Z","level":"debug","repo":"apl/test","build":"73","id":"196","image":"plugins/git:latest","stage":"git","exit_code":0,"exited":false,"message":"update step status complete"}
{"time":"2018-04-03T06:45:30Z","level":"debug","repo":"apl/test","build":"73","id":"196","image":"plugins/git:latest","stage":"git","message":"log stream opened"}
{"time":"2018-04-03T06:45:35Z","level":"debug","repo":"apl/test","build":"73","id":"196","image":"plugins/git:latest","stage":"git","message":"log stream copied"}
{"time":"2018-04-03T06:45:35Z","level":"debug","repo":"apl/test","build":"73","id":"196","image":"plugins/git:latest","stage":"git","message":"log stream uploading"}
{"time":"2018-04-03T06:45:35Z","level":"debug","repo":"apl/test","build":"73","id":"196","image":"plugins/git:latest","stage":"git","message":"log stream upload complete"}
{"time":"2018-04-03T06:45:35Z","level":"debug","repo":"apl/test","build":"73","id":"196","image":"plugins/git:latest","stage":"git","message":"log stream closed"}
{"time":"2018-04-03T06:45:35Z","level":"debug","repo":"apl/test","build":"73","id":"196","image":"plugins/git:latest","stage":"git","exit_code":1,"exited":true,"message":"update step status"}
{"time":"2018-04-03T06:45:35Z","level":"debug","repo":"apl/test","build":"73","id":"196","image":"plugins/git:latest","stage":"git","exit_code":1,"exited":true,"message":"update step status complete"}
{"time":"2018-04-03T06:45:35Z","level":"debug","repo":"apl/test","build":"73","id":"196","error":"","exit_code":1,"message":"pipeline complete"}
{"time":"2018-04-03T06:45:35Z","level":"debug","repo":"apl/test","build":"73","id":"196","message":"uploading logs"}
{"time":"2018-04-03T06:45:35Z","level":"debug","repo":"apl/test","build":"73","id":"196","message":"uploading logs complete"}
{"time":"2018-04-03T06:45:35Z","level":"debug","repo":"apl/test","build":"73","id":"196","error":"","exit_code":1,"message":"updating pipeline status"}
{"time":"2018-04-03T06:45:35Z","level":"debug","repo":"apl/test","build":"73","id":"196","message":"stop listening for cancel signal"}
{"time":"2018-04-03T06:45:35Z","level":"debug","repo":"apl/test","build":"73","id":"196","message":"updating pipeline status complete"}
{"time":"2018-04-03T06:45:35Z","level":"debug","message":"request next execution"}
{"time":"2018-04-03T06:45:35Z","level":"debug","repo":"apl/test","build":"73","id":"196","message":"pipeline done"}

I can resolve any hosts from container

apl@beta:~/GIT/test$ docker exec -it drone-worker cat /etc/resolv.conf
search domain.com apl
nameserver 127.0.0.11
options ndots:0

apl@beta:~/GIT/test$ docker exec -it drone-worker nslookup git.domain.com 127.0.0.11
Server:    127.0.0.11
Address 1: 127.0.0.11

Name:      git.domain.com
Address 1: 192.168.1.7 git.domain.com

apl@beta:~/GIT/test$ docker exec -it drone-worker cat /etc/hosts
127.0.0.1       localhost
192.168.1.2     drone.domain.com
192.168.1.7     git.domain.com
172.19.1.15     5c72078eaeb0

I can clone repository using regular git inside worker container.

/ # git clone https://git.domain.com/apl/test.git
Cloning into 'test'...
remote: Counting objects: 86, done.
remote: Compressing objects: 100% (69/69), done.
remote: Total 86 (delta 17), reused 7 (delta 1)
Unpacking objects: 100% (86/86), done.

docker-compose.yml:

version: '3'

services:
  drone-server:
    image:  ap-drone-server:latest
    container_name: drone-server
    ports:
      - 8088:8000
      - 9000:9000
    volumes:
      - /Data/Drone/:/var/lib/drone/
    restart: always
    environment:
      - DRONE_OPEN=true
      - DRONE_HOST=https://drone.domain.com
      - DRONE_GOGS=true
      - DRONE_GOGS_URL=https://git.domain.com
      - DRONE_SECRET=2485a42e37b91fb033b328c333b1f6a9
      - GODEBUG=netdns=go
    dns:
      - 192.168.1.2
      - 192.168.1.4
    dns_search:
      - domain.com
      - apl

  drone-agent:
    image: ap-drone-agent
    container_name: drone-worker
    command: agent
    restart: always
    depends_on:
      - drone-server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - DRONE_SERVER=drone.domain.com:9000
      - DRONE_SECRET=2485a42e37b91fb033b328c333b1f6a9
      - GODEBUG=netdns=9+go
      - DEBUG=true
    dns:
      - 192.168.1.2
      - 192.168.1.4
    dns_search:
      - domain.com
      - apl
    extra_hosts:
      - "git.domain.com:192.168.1.2"
      - "drone.domain.com:192.168.1.2"

networks:
  default:
    external:
      name: br-pub

drone-agent version 0.8.5
drone-server version 0.8.5
Docker version 18.03.0-ce, build 0520e24

Support cloning as custom user

It would be great to be able to clone a repository as another user than root. I compile Rust projects using drone and the ekidd/rust-musl-builder image. That image runs the build process as the user rust. Since the repository is cloned as root, the build user has no write permissions, it cannot build the project without explicitly chowning first. I think it would be better to have a user or UID configurable per pipeline that will be used for cloning.

Remove default depth

There should be no default depth. See #14

Add Nsswitch and custom hosts file to container build

Similar to issue: harness/gitness#1269
and PR: harness/gitness#1306

Where container is not respecting the local hosts file for lookups.
In my case there is not always DNS available or the DNS is incorrect.
Need a way to manually set that.
Adding an /etc/nsswitch.conf and a custom hosts file seems to resolve this.

ADD contrib/docker/etc/nsswitch.conf
ADD contrib/docker/etc/hosts

plugins git clone public is unsuccessful in ARM64

Hi, I can not commit the issue in https://discourse.drone.io/

drone: 1.0
This command is ok.

docker run --rm \
    -e DRONE_REMOTE_URL=http://IP:33000/xiaozheng8187/godev.git \
    -e DRONE_WORKSPACE=/go/src/github.com/xiaozheng8187/godev \
    -e DRONE_BUILD_EVENT=push \
    -e DRONE_COMMIT_SHA=a14c724b8d00eff659d9351ec560b67519dbf226 \
    -e DRONE_COMMIT_REF=refs/heads/master \
    plugins/git:linux-arm64

But this command is not ok, git clone hang up
CI/CD is gogs 0.11

kind: pipeline
name: godev
clone:
  disable: true
platform:
  os: linux
  arch: arm64
workspace:
  base: /go
  path: src/gogs.xiaozheng.com/super/godev
steps:
- name: clone
  image: plugins/git:linux-arm64
  commands:
  - git clone http://IP:33000/xiaozheng8187/godev.git

docker logs j521jqk7ugfots7utxy05mp2u10s88mx

this log as:

+ git clone http://IP:33000/xiaozheng8187/godev.git
Cloning into 'godev'...

Better error reporting on pull requests

One of our developers encountered this error.

[info] Pulling image plugins/drone-git:latest
Drone Git Plugin built at 2016-02-02 08:48:17 UTC
Drone Git Plugin built at 2016-02-02 08:48:17 UTC
$ git init
Initialized empty Git repository in /drone/src/xxxx/xxxx/xxxx/.git/
$ git remote add origin git@xxxx:xxxx/xxxx.git
$ git fetch --no-tags --depth=50 origin +refs/pull/1985/merge:
Warning: Permanently added 'xxxx' (ECDSA) to the list of known hosts.\r
fatal: Couldn't find remote ref refs/pull/1985/merge
[info] build failed (exit code 1)

After talking with @bradrydzewski he mentioned that if the pull request could not be automatically merged then the ref wouldn't be present. Maybe a better error message would be.

[error] The ref for the pull request was not found. This is usually due to the branch being merged having conflicts with the target branch.

add --recursive flag

so that we can clone submodules

fatal: could not read Username for 'https://github.com': No such device or address exit status 128

Hello,

Since this morning, we receive this error when we use the plugin:

fatal: could not read Username for 'https://github.com': No such device or address exit status 128

Probably due to upgrade of this friday?

Best regards,

override git+ssh submodules

If we are using the .netrc to clone the repository, we probably also need to use the .netrc to clone the git submodules. If the submodule uses a git+ssh url we may see failures.

Therefore, I propose we parse the .gitmodules file and then override the submodule url's using environment variables.

So for this sample .gitsubmodule file:

[submodule "hello-world"]
    path = hello-world
    url = [email protected]:octocat/hello-world.git

We would parse and then run the following command to override:

git config 'submodule.hello-world.url' 'https://github.com/octocat/hello-world.git'

Note that if the existing url is git:// or https:// there would be no need to override. Also if the repository is public there is no need to perform this step.

For parsing the gitconfig format we can use the following packages:

https://github.com/scalingdata/gcfg
https://gopkg.in/gcfg.v1

checkout vs reset --hard

May be best to use git reset --hard to revision instead of git checkout ?
In case of checkout we have detached head, that not valid in some cases...

`git rev-parse HEAD` doesn't work because of the method of checkout

I'm trying to incorporate version information into our deployments from Drone, but the method of git checkout stymies this.

fatal: could not read Username for 'https://git.mydomain.com': No such device or address

I recently installed a server with Drone 1.x and I have a repo with submodules. I have this .drone.yml file :

---
kind: pipeline
name: default

platform:
  os: linux
  arch: amd64

clone:
  disable: true

steps:
- name: git
  pull: default
  image: plugins/git
  settings:
    depth: 1
    recursive: true
    submodule_override:
      vendor/plugins/acts_as_list: https://git.mydomain.com/ccp-v2/acts_as_list.git
// ...

When I run a build, it gives the error fatal: could not read Username for 'https://git.mydomain.com': No such device or address.

I did some tests and I was able to determine than git and ssh configs aren't shared between the host and the container. I saw #87 will partially fix the problem because we'll be able to specify the username but it still require the password. There's maybe a way to add the password in variable but this not ideal.

It seems that sharing the id_rsa key is enough to make the git clone working. I didn't found a way to do this.

It was working with drone 0.8.x.

Is there something I missed?

Add unshallow option

Add option to enable git fetch --unshallow

support git config options

support custom git config options

git config --add core.gitproxy '"proxy-command" 

as a yaml slice parameter:

git_config:
  - core.gitproxy proxy-command
  - user.email [email protected]