harness/drone

Gitness is an Open Source developer platform with Source Control management, Continuous Integration and Continuous Delivery.

Gitness

Gitness is an open source development platform packed with the power of code hosting and automated DevOps pipelines.

Overview

Gitness is an open source development platform packed with the power of code hosting and automated continuous integration pipelines.

Running Gitness locally

The latest publicly released docker image can be found on harness/gitness.

To install Gitness yourself, simply run the command below. Once the container is up, you can visit http://localhost:3000 in your browser.

docker run -d \
  -p 3000:3000 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /tmp/gitness:/data \
  --name gitness \
  --restart always \
  harness/gitness

The Gitness image uses a volume to store the database and repositories. It is highly recommended to use a bind mount or named volume as otherwise all data will be lost once the container is stopped.

See docs.gitness.com to learn how to get the most out of Gitness.

Where is Drone?

Gitness represents a massive investment in the next generation of Drone. Where Drone focused on continuous integration, Gitness adds source code hosting, bringing code management and pipelines closer together.

The goal is for Gitness to eventually be at full parity with Drone in terms of pipeline capabilities, allowing users to seemlessly migrate from Drone to Gitness.

But, we expect this to take some time, which is why we took a snapshot of Drone as a feature branch drone (README) so it can continue development.

As for Gitness, the development is taking place on the main branch.

For more information on Gitness, please visit gitness.com.

For more information on Drone, please visit drone.io.

Gitness Development

Pre-Requisites

Install the latest stable version of Node and Go version 1.20 or higher, and then install the below Go programs. Ensure the GOPATH bin directory is added to your PATH.

Install protobuf

• Check if you've already installed protobuf protoc --version
• If your version is different than v3.21.11, run brew unlink protobuf
• Get v3.21.11 curl -s https://raw.githubusercontent.com/Homebrew/homebrew-core/9de8de7a533609ebfded833480c1f7c05a3448cb/Formula/protobuf.rb > /tmp/protobuf.rb
• Install it brew install /tmp/protobuf.rb
• Check out your version protoc --version

Install protoc-gen-go and protoc-gen-go-rpc:

• Install protoc-gen-go v1.28.1 go install google.golang.org/protobuf/cmd/[email protected] (Note that this will install a binary in $GOBIN so make sure $GOBIN is in your $PATH)

• Install protoc-gen-go-grpc v1.2.0 go install google.golang.org/grpc/cmd/[email protected]

$ make dep
$ make tools

Build

First step is to build the user interface artifacts:

$ pushd web
$ yarn install
$ yarn build
$ popd

After that, you can build the gitness binary:

$ make build

Run

This project supports all operating systems and architectures supported by Go. This means you can build and run the system on your machine; docker containers are not required for local development and testing.

To start the server at localhost:3000, simply run the following command:

./gitness server .local.env

Auto-Generate Gitness API Client used by UI using Swagger

Please make sure to update the autogenerated client code used by the UI when adding new rest APIs.

To regenerate the code, please execute the following steps:

• Regenerate swagger with latest gitness binary ./gitness swagger > web/src/services/code/swagger.yaml
• navigate to the web folder and run yarn services

The latest API changes should now be reflected in web/src/services/code/index.tsx

User Interface

This project includes a full user interface for interacting with the system. When you run the application, you can access the user interface by navigating to http://localhost:3000 in your browser.

REST API

This project includes a swagger specification. When you run the application, you can access the swagger specification by navigating to http://localhost:3000/swagger in your browser (for raw yaml see http://localhost:3000/openapi.yaml).

For testing, it's simplest to just use the cli to create a token (this requires gitness server to run):

# LOGIN (user: admin, pw: changeit)
$ ./gitness login

# GENERATE PAT (1 YEAR VALIDITY)
$ ./gitness user pat "my-pat-uid" 2592000

The command outputs a valid PAT that has been granted full access as the user. The token can then be send as part of the Authorization header with Postman or curl:

$ curl http://localhost:3000/api/v1/user \
-H "Authorization: Bearer $TOKEN"

CLI

This project includes VERY basic command line tools for development and running the service. Please remember that you must start the server before you can execute commands.

For a full list of supported operations, please see

$ ./gitness --help

Contributing

Refer to CONTRIBUTING.md

License

Apache License 2.0, see LICENSE.

drone's Issues

Colours not available in build output

See the example below. This works as expected on drone.io and shows green text.

�[4mRunning "vows:all" (vows) task�[24m
�[32m·�[39m�[32m·�[39m�[32m·�[39m�[32m·�[39m �[32m·�[39m�[32m·�[39m�[32m·�[39m�[32m·�[39m  
  ✓ �[32m�[1mOK�[22m�[39m » �[1m8�[22m honored�[90m (0.129s)�[39m

Build doesn't run anything

It seems like I have everything hooked up properly but somehow, my tests don't run.

Here's my .drone.yml:

image: ruby2.0.0
script:
  - bundle install
  - bundle exec rake db:create db:schema:load spec
services:
  - mysql
  - elasticsearch
  - redis
  - memcached

And a screenshot:

Any idea?

Detect if running pull request.

Hi,

Is there any configuration so I can detect if I'm running pull requests or commits from master branch ?

I can see that deployment commands are not run if there is a pull request, but currently I cannot run capistrano deployment (I can't see any plugin for that).

Also, any chance to introduce IRC notifications ?

Include a more modern Haskell image

How can someone submit a newer Haskell build? 7.4 is very outdated. A better target would be the 2013.2.0.0 Haskell Platform (https://index.docker.io/u/zsol/haskell-platform-2013.2.0.0/) which has upgraded to GHC 7.6.3.

There's also a new Haskell Platform on its way which might even include GHC 7.8.

Check Docker version, show warning message

We should check for a minimum Docker version. The remote API changes between releases, and running an older version of Docker can cause panics and strange issues that are hard to debug.

I'd like to check when the app first starts. If Docker is not installed, or is out of date, we should not serve any of the standard handlers. Instead we should display a single page that gives instructions to upgrade.

GitHub Enterprise & http://

cc @suguru @floatdrop

this was brought up in issue #75

the github url is currently hardcoded with an https:// prefix. Let's slightly change the design so that instead of github_domain we store github_url which is the scheme+domain, and remove the hard-coded https:// prefix.

we may also need a migration script for this, to change existing records from github.com to https://github.com

external git repos

Is there a plan to integrate git repos like gitlab or any external source via public key?

Where is the log file located?

I installed Drone on Ubuntu 13.04. Where is the log file located? I'm trying to use a HipChat notification function but I can not get any notifications on HipChat. I want to check a log file to understand what is wrong.

secure build config parameters

Is what I can put in the repo's secure build params limited to just "key: value" pairs? I wanted to inject:

notify:
  email:
    recipients:
      - [email protected]

And the UI reports it saved it successfully, but if I navigate away from the page or peek at the repo's sqlite table it doesn't actually persist. Random entries like "somekey: foo" work fine though.

Bitbucket support

How do I get this working with BB?

PPA repository for install via APT

Hello, looks great!

If you've already set up your .deb building from deb-src, could you please put it up in a PPA? It would make installation much simpler, particularly for those of us who have our CI server configuration in puppet or chef scripts.

Cheers!
Michael

Dockerfile support

Is it possible to configure drone so that if your project already has a Dockerfile, you can omit the script part of .drone.yml and rely on the Dockerfile to build the image?

Hostname to container routing/Container persistence

A couple of feature suggestions. They might be out of scope of the project but I thought I'd throw them out there.

Hostname/Port to container routing:

• Routes ports to containers
• For HTTP based protocols, also route on Host header
• Use wildcard DNS entry to allow host names to be dynamically generated for containers: branch.repo.org.mytestdomain.com
• Hostname pattern specified in .drone.yml
• Would allow external access to built projects without having the containers directly exposed
• In our case this would allow Selenium Grid nodes to test projects

Container persistence:

• Keep containers running until pull request is closed or build is superseded
• Used with hostname/port routing to
  • allow for manual checking of failures
  • allow for manual testing of pull request features/bug fixes

Additional Security Measures for Login

Since drone needs to accept git hooks from github, it needs to be exposed on the public internet. It'd be nice if there were some protections on the login page:

• Two-factor auth
• Captcha after incorrect attempts originating from same IP or against same username
• IP whitelisting

Why all imports are pointing to github?

I was hacking around, to make drone work with enterprise github, but I got really frustrated, because all imports are pointed to github repo? I am newbie in go, so is it recommendation from language?

panic: template: _:51: function "eq" not defined

after enter command make deps and make and done it successfully, but get errors while entering command make run.

➜  drone git:(master) make run
bin/droned --port=":8080" --datasource="drone.sqlite"
panic: template: _:51: function "eq" not defined

goroutine 1 [running]:
html/template.Must(0x0, 0x18883040, 0x189b5770, 0x0)
    /usr/local/go/src/pkg/html/template/template.go:284 +0x44
github.com/drone/drone/pkg/template.init·3()
    /home/howard/go/src/github.com/drone/drone/pkg/template/template.go:112 +0x26b
github.com/drone/drone/pkg/template.init()
    /home/howard/go/src/github.com/drone/drone/pkg/template/template.go:143 +0xad
github.com/drone/drone/pkg/handler.init()
    /home/howard/go/src/github.com/drone/drone/pkg/handler/util.go:112 +0x3f
main.init()
    /home/howard/go/src/github.com/drone/drone/cmd/droned/drone.go:202 +0x39

goroutine 2 [syscall]:

goroutine 51 [chan send]:
text/template/parse.lexSpace(0x188e6cc0, 0x84bb454)
    /usr/local/go/src/pkg/text/template/parse/lex.go:334 +0xc6
text/template/parse.(*lexer).run(0x188e6cc0)
    /usr/local/go/src/pkg/text/template/parse/lex.go:198 +0x39
created by text/template/parse.lex
    /usr/local/go/src/pkg/text/template/parse/lex.go:191 +0xe6
make: *** [run] Error 2

.drone.yml example to deploy on Rackspace

Current examples show Heroku and Amazon deployments. It would be great if you can provide examples for the Rackspace too.

Cache-Control for Badges

Need to understand how this impacts our badge implementation:
github/markup#224

Any changes required?

Unable to complete registration after invite on clean install

Using latest deb-file and sqlite, after a user clicks the activation link and adds their password, the register post-handler fails during saving to db complaining about token not being unique.

2014/02/08 17:24:28 meddler.Insert: DB error in Exec: column token is not unique

Build CLI Idea

One thing that bothers me with lots of cloud based build systems is that you cannot easily (to my knowledge) test the yml file before you commit. With Drone being based on docker, I don't see why you couldn't be able to do:

$ drone build

And locally run all of the scripts to make sure your configuration is what you want before you push your commits. Would this be possible? Any thoughts? Does this exist with other platforms?

Matrix Builds / Sub-Builds / Parallel Builds

I'm wondering how would a .drone.yml would look if you'd want to test with multiple versions of Python, like for example, using Tox?

Drone shows me the previous build for the same commit

I have to branches on my repository, develop and master. I pushed one commit on develop, drone built it, and all specs passed.

Next, I merged develop into master and pushed it to github. Drone is now building the master branch, current status is building, but if I click on the build, the previous build (from develop) is shown.

GitHub caching badge, causing problems

see our README.md as an example

Self Hosted Git Repo Support

Hey guys,

thank you for this great project!
It would be awesome to have support for self hosted git repositories. Then drone would be the perfect solution for a enterprise project I'm working on and I could get rid of Jenkins :)

Cheers
Sebastian

More ways to register in drone.io

Hey. I really want to get drone.io into stress environment and see how it's going, but current registration scheme gets in the way. Would you accept settings for enabling open registration in drone.io? Invites working great, but I feel like people just move along, when seeing login page without any hints, how get access.

Thank you for sharing this project again. It's wonderful!

Page constantly refreshes during build

If a build is still in progress and I visit the status page, my browser continually refreshes every second until the build is done.

Unable to change user role to administrator - token error in database

Changing the role of a user to administrator fails with the following error:

Failed to update user data. meddler.Update: DB error in Exec: column token is not unique

Error in image bradrydzewski/php:5.*

In /etc/drone.d/phpenv.sh you add home variable '/home/ubuntu/phpenv/bin'. I think that should be '/home/ubuntu/.phpenv/bin' (note the . in phpenv), as I'm always getting the following error:

/etc/drone.d/phpenv.sh: line 21: phpenv: command not found

Wrong links at add repository page?

Wondering if this were suppposed to be /new/github.com instead of /github.com/drone/drone/settings?

also in github_link.html

Should be able to turn off autoscroll

I've got long verbose builds that have interesting output early on, but scrolling up just scrolls me down again. :)

GitLab HQ Support

I'm splitting this into a separate feature request, given its popularity. The good news is there's a Go API that we can use as our base, however, it still lacks certain features like adding post-commit hooks and ssh keys (for private repos).

Is someone will to contribute to to the API and add these features?
https://github.com/plouc/go-gitlab-client

I'd volunteer, however, I have my hands full with this code base :)

GitHub Enterprise Support

I don't have access to GitHub enterprise, so I'm just guessing here, but this is what I think will need to get altered. It should be a relatively straightforward change ...

step 1: alter the github client to accept a custom url

NewClient(url, token)

https://github.com/drone/go-github/blob/master/github/github.go#L4
https://github.com/drone/go-github/blob/master/github/http.go#L44

step 2: alter our github oauth code (for linking accounts) to use custom access token and authorization urls:

var oauth = oauth2.Client{
    RedirectURL:      settings.URL().String() + "/auth/login/github",
    AccessTokenURL:   "https://github.com/login/oauth/access_token",
    AuthorizationURL: "https://github.com/login/oauth/authorize",
    ClientId:         settings.GitHubKey,
    ClientSecret:     settings.GitHubSecret,
}

https://github.com/drone/drone/blob/master/pkg/handler/auth.go#L43

Local repository cache

Cloning a large repository can take a long (build) time, we should do something like:

if [ -d $repository_path ]
  cd $repository_path && git fetch $remote && git reset --hard $branch
else
  git clone $repo $repository_path

Clone other private repositories using Netrc file

It would be great to be able to inject a pre-defined SSH key into containers.

As part of our build process, we pull down dependencies that are in private GitHub repos. Currently there's no (easy) way of accessing these repos from Drone. This could be resolved by adding the ability to generate an SSH key that's used by containers (copied to ~/.ssh/id_rsa), which we could then add as the deploy key for our dependencies.

Question: What's the reason for not using a web framework?

Simply out of curiosity, what was the reason you chose not to go with one of the golang web frameworks?

Branch names with slashes are not properly handled

Examples: feature/my-new-feature, hotfix/urgent-bug, release/1.0

Dockerfile for drone itself

Would be insanely useful. I'm working on one, would you merge?

More verbose logs and server answers

This is low priority, but seems like drone doesn't like to tell user what is wrong. For example it only sends error messages like Not found or Unauthorized without context.

How can I know which commits the latest drone.deb includes without reinstalling Drone?

How can I know which commits the latest drone.deb includes without reinstalling Drone? The latest drone.deb includes merged PR as soon as possible?

Error "dial tcp 0.0.0.0:4243: connection refused" when running test

Thanks for opensourcing this, it is really amazing!

After installing on Ubuntu 12.04 in virtualbox and setting up this repo: https://github.com/tax/testdrone I get the following error when a test is run:

dial tcp 0.0.0.0:4243: connection refused

This is my .drone.yml file:

image: python:2.7
script:
  - python test.py
notify:
  email:
    recipients:
      - [email protected]

Any idea what might be the problem?

Add ability to split tests to execute in different containers

Hi! It will be very good to have ability to split tests to N containers.
For languages like ruby, it will dramatically decrease execution time.
Is it possible?

Webhooks not working from GitHub

When I push to GitHub, nothing happens. If I go to debug the webhook on GitHub, it says there was a timeout. However, if I capture the request by adding a hook to another server and submit it directly to my Drone instance, it works just fine. Any idea what this could be? My first thought is firewall issues, but I haven't had a problem accessing my server from other locations before.

Make GitHub scopes configurable

cc @suguru

this was brought up in issue #75

when GitHub enterprise is run in private mode it prevents using git:// to clone public projects. This means every repository should be treated as a private repository -- using the git@ url and adding the ssh deploy key.

Help Page not found

When clicking on the Help link in the main navigation you get redirected to /help that returns a "404 page not found".

Missing status badge in README.md

well, for the demo purposes would make sense (if you build on some public instance like drone.io).

image for D

I really like the idea of having a web service like drone.io to build projects.

I have still to find a service that provides ways to build D (http://dlang.org) projects with dedictated environments. So can drone be the first one please?

With some guidance on your ways of providing images, I'm likely able to provide an image myself. I guess a D image should be pulled into the drone/images repo right? Is this Repo also providing images for the drone.io web service?

Ability for an admin to permit a repo's builds to be privileged

I need this for testing one of my projects that's much like Docker. For now I've just forked Drone and switched it to run everything privileged, but I'm wondering what this would look like as a polished feature, so I can start putting together a PR.

For example, I'd expect pull-request-triggered builds to not run privileged, but I'm not sure what that means for the build. Perhaps it should set an environment variable so that the tests know to skip the ones that require it? Any other things to worry about?

Error when a non-admin tries to add a repo

When a non-admin goes to add a repository to their dashboard the first time they are (correctly) prompted to link their Github accounts. This always causes an error when the Github callback URL is invoked:

meddler.Update: DB error in Exec: column token is not unique

I believe this error is being thrown around line 85 of pkg/handler/auth.go but I won't be able to take a look until later tonight.

Cache Docker images

documenting suggestion from @afex on irc

add a section to the .drone.yml called setup

setup:
  - apt-get update
  - apt-get -y install libsqlite3 libsqlite3-dev

add these install scripts to the Dockerfile that we generate:
https://github.com/drone/drone/blob/master/pkg/build/build.go#L365

RUN apt-get update
RUN apt-get -y install libsqlite3 libsqlite3-dev

this would allow us to take advantage of Docker's layer caching. The build would use the cached image and avoid re-running install and setup commands. Saves time! If build script commands changed the cache would be invalidated and a new image would be generated.

question: how does this impact matrix builds?

Enable ListenAndServeTLS for https support

It is currently possible to set "http" or "https" modes but where to put SSL certificate and key to make it work?