concourse/hg-resource

Mercurial resource for Concourse

Concourse: the continuous thing-doer.

Concourse is an automation system written in Go. It is most commonly used for CI/CD, and is built to scale to any kind of automation pipeline, from simple to complex.

Concourse is very opinionated about a few things: idempotency, immutability, declarative config, stateless workers, and reproducible builds.

The road to Concourse v10

Concourse v10 is the code name for a set of features which, when used in combination, will have a massive impact on Concourse's capabilities as a generic continuous thing-doer. These features, and how they interact, are described in detail in the Core roadmap: towards v10 and Re-inventing resource types blog posts. (These posts are slightly out of date, but they get the idea across.)

Notably, v10 will make Concourse not suck for multi-branch and/or pull-request driven workflows - examples of spatial change, where the set of things to automate grows and shrinks over time.

Because v10 is really an alias for a ton of separate features, there's a lot to keep track of - here's an overview:

Feature	RFC	Status
set_pipeline step	✔ #31	✔ v5.8.0 (experimental)
Var sources for creds	✔ #39	✔ v5.8.0 (experimental), TODO: #5813
Archiving pipelines	✔ #33	✔ v6.5.0
Instanced pipelines	✔ #34	✔ v7.0.0 (experimental)
Static across step	🚧 #29	✔ v6.5.0 (experimental)
Dynamic across step	🚧 #29	✔ v7.4.0 (experimental, not released yet)
Projects	🚧 #32	🙏 RFC needs feedback!
load_var step	✔ #27	✔ v6.0.0 (experimental)
get_var step	✔ #27	🚧 #5815 in progress!
Prototypes	✔ #37	⚠ Pending first use of protocol (any of the below)
run step	🚧 #37	⚠ Pending its own RFC, but feel free to experiment
Resource prototypes	✔ #38	🙏 #5870 looking for volunteers!
Var source prototypes		🚧 #6275 planned, may lead to RFC
Notifier prototypes	🚧 #28	⚠ RFC not ready

The Concourse team at VMware will be working on these features, however in the interest of growing a healthy community of contributors we would really appreciate any volunteers. This roadmap is very easy to parallelize, as it is comprised of many orthogonal features, so the faster we can power through it, the faster we can all benefit. We want these for our own pipelines too! 😆

If you'd like to get involved, hop in Discord or leave a comment on any of the issues linked above so we can coordinate. We're more than happy to help figure things out or pick up any work that you don't feel comfortable doing (e.g. UI, unfamiliar parts, etc.).

Thanks to everyone who has contributed so far, whether in code or in the community, and thanks to everyone for their patience while we figure out how to support such common functionality the "Concoursey way!" 🙏

Installation

Concourse is distributed as a single concourse binary, available on the Releases page.

If you want to just kick the tires, jump ahead to the Quick Start.

In addition to the concourse binary, there are a few other supported formats. Consult their GitHub repos for more information:

• Docker image
• BOSH release
• Kubernetes Helm chart

Quick Start

$ wget https://concourse-ci.org/docker-compose.yml
$ docker-compose up
Creating docs_concourse-db_1 ... done
Creating docs_concourse_1    ... done

Concourse will be running at 127.0.0.1:8080. You can log in with the username/password as test/test.

⚠️ If you are using an M1 mac: M1 macs are incompatible with the containerd runtime. After downloading the docker-compose file, change CONCOURSE_WORKER_RUNTIME: "containerd" to CONCOURSE_WORKER_RUNTIME: "houdini". This feature is experimental

Next, install fly by downloading it from the web UI and target your local Concourse as the test user:

$ fly -t ci login -c http://127.0.0.1:8080 -u test -p test
logging in to team 'main'

target saved

Configuring a Pipeline

There is no GUI for configuring Concourse. Instead, pipelines are configured as declarative YAML files:

resources:
- name: booklit
  type: git
  source: {uri: "https://github.com/vito/booklit"}

jobs:
- name: unit
  plan:
  - get: booklit
    trigger: true
  - task: test
    file: booklit/ci/test.yml

Most operations are done via the accompanying fly CLI. If you've got Concourse installed, try saving the above example as booklit.yml, target your Concourse instance, and then run:

fly -t ci set-pipeline -p booklit -c booklit.yml

These pipeline files are self-contained, maximizing portability from one Concourse instance to the next.

Learn More

• The Official Site for documentation, reference material, and example pipelines (which no longer live in this repository).
• The Concourse Tutorial by Stark & Wayne is great for a guided introduction to all the core concepts.
• See Concourse in action with our production pipelines
• Hang around in the GitHub discussions or in Discord.
• See what we're working on the project board.

Contributing

Our user base is basically everyone that develops software (and wants it to work).

It's a lot of work, and we need your help! If you're interested, check out our contributing docs.

hg-resource's Issues

Support username/password same way as git-resource

Hi,

I wanted to use hg-resource, using http username+tokens for authentication, but it seems that this is not really supported by this resource.

and when i tried to do:

uri: https://{{repo_hg_user}}:{{repo_hg_password}}@some-url

I actually ended up in something like:

https://"username":"password"@some-url

Always including `--branch` does not allow for pulling full repository

I need to do some operations on the entire repository inside of a job. By setting the branch default if the branch option is unset, only changsets up to and including the tip of default are cloned and pulled. I need an argument to not include any branch. I can make the change, what is the preferred way to express branch(somebranch) | no-branch in the config?

Tag before pulling and rebasing

Tag should tag the current commit id of the working directory before pulling and rebasing. otherwise, manually triggering a job can lead to tagging the wrong commit.

Steps to reproduce:

1. Create a pipeline that gets, tags and pushes
2. Add a commit in the source repo and trigger the pipeline
3. Increment the version resource so that another commit can be tagged
4. Manually trigger the pipeline again.

Expected behavior:

The tip commit of the source repo should get tagged again.

Actual behavior:

Out pulls the tag commit from the destination repo. Since there is no new commit in the source repo, the old tag commit ("Added tag ... for changeset ...") is now at tip, and out tags it with the new version number.

"fly execute --exclude-ignored" broken for mercurial resource #1244

cc @marco-m Moved from concourse/concourse#1244

---

With a pipeline using the mercurial resource

resources:
- name: concourse-native
  type: hg
  source:
    uri: https://bitbucket.com/me/myrepo
    branch: default

The checked out repo will have a directory named .hg (the equivalent of .git).

When doing a one-off build with fly execute, it will upload the whole repo history, inside the .hg directory.

If I try to avoid uploading the .hg directory with --exclude-ignored, fly prints an error message and stays stuck uploading (have to hit Ctrl-C):

 concourse-native> fly -t tutorial execute --exclude-ignored --config ci/build-task.yml
 executing build 34
 could not determine ignored files: exit status 128
   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                  Dload  Upload   Total   Spent    Left  Speed
   0     0    0     0    0     0      0      0 --:--:--  0:00:24 --:--:--     0^C

I just double-tested the exact setup but with a git repository and the .git directory is not uploaded and the upload succeeds.

• Concourse version: 3.1.1
• Infrastructure/IaaS: Vagrant
• Did this used to work? First time I tried

Resource fails with `open /root/.ssh/config: no such file or directory`

So I can't be 100% sure, but after a upgrade the other day, all our resources started failing with:

open /root/.ssh/config: no such file or directory

Looking at the change I would guess this change is the offender:

75d0270#diff-539560c8bb7f0b5eb133bea29c18badb2943d366769fff45fb33a54120bde909R54

Looking at the previous atomicSave function it looks like it actually checks for the dir and creates it if it doesn't exists.

Are we doing anything wrong since we're using root, should we run as another user, should we have the go module create the dir if necessary or should we have the directory created in the docker file instead?

Cloning with subrepositories and other vcs

Hi!

I've been trying to set-up a concourse pipeline for a project of mine, and stumbled upon a problem: I can't clone a repository, that has git subrepositories.

error: check failed with exit status '1':
abort: couldn't find git for subrepo 'XXX'
(check git is installed and in your PATH)
Error updating working directory to tip: exit status 255

I was digging around and it seems the docker image for this resource needs git, and perhaps other vcs tools.

Can this feature be added in some reasonable timeframe? I'm willing to install a custom version of hg-resource if you could provide a hint on how to do that.

Resource receiving revisions from a different branch

I explicitly set the branch that the resource should be using, which in this case is default, but other branches which I'm creating from default are ending up on a pipeline, which then fails to use it with the following error:

abort: unknown revision 'x'!
Error checking out x: exit status 255`

I think this error is from sourceRepo.GetCurrentCommitId in out.go, but I assume the problem is in check. Should the hg log calls in hg.go be using --branch to filter results to a certain branch, e.g GetLatestCommitId and GetDescendantsOf? How would I go about debugging this?