iver-wharf/wharf-provider-azuredevops

Import Wharf projects from Azure DevOps repositories

Azure DevOps provider for Wharf

Import Wharf projects from Azure DevOps repositories. Mainly focused on importing from self hosted Azure DevOps instances, importing from dev.azure.com is not well tested.

Token scopes

When generating a Personal Access Token (PAT) to let wharf-provider-azuredevops talk to Azure DevOps, you need the following permission scopes associated with the token:

• Code
  Source code, repositories, pull requests, and notifications

  • Read

Components

• HTTP API using the gin-gonic/gin web framework.

• Swagger documentation generated using swaggo/swag and hosted using swaggo/gin-swagger

• Azure DevOps REST API accessed using good'ol net/http and encoding/json

Configuring

The wharf-provider-azuredevops program can be configured via environment variables and through optional config files. See the docs on the Config type over at: https://pkg.go.dev/github.com/iver-wharf/wharf-provider-azuredevops#Config

Development

1. Install Go 1.18 or later: https://golang.org/

2. Install dependencies using GNU Make or GNUWin32

   $ make deps

3. Generate the Swagger files (this has to be redone each time the swaggo documentation comments has been altered):

   $ make swag

4. Start hacking with your favorite tool. For example VS Code, GoLand, Vim, Emacs, or whatnot.

Linting

make deps # download linting dependencies

make lint

make lint-go # only lint Go code
make lint-md # only lint Markdown files

Some errors can be fixed automatically. Keep in mind that this updates the files in place.

make lint-fix

make lint-fix-go # only lint and fix Go files
make lint-fix-md # only lint and fix Markdown files

Releasing

Replace the "v2.0.0" in make docker version=v2.0.0 with the new version. Full documentation can be found at Releasing a new version.

Below are just how to create the Docker images using GNU Make or GNUWin32:

$ make docker version=v2.0.0
STEP 1: FROM golang:1.18 AS build
STEP 2: WORKDIR /src
--> Using cache de3476fd68836750f453d9d4e7b592549fa924c14e68c9b80069881de8aacc9b
--> de3476fd688
STEP 3: ENV GO111MODULE=on
--> Using cache 4f47a95d0642dcaf5525ee1f19113f97911b1254889c5f2ce29eb6f034bd550b
--> 4f47a95d064
STEP 4: RUN go install github.com/swaggo/swag/cmd/[email protected]
...

Push the image by running:
docker push quay.io/iver-wharf/wharf-provider-azuredevops:latest
docker push quay.io/iver-wharf/wharf-provider-azuredevops:v2.0.0

---

Maintained by Iver. Licensed under the MIT license.

wharf-provider-azuredevops's Issues

Use wharf-core/pkg/ginutil for error responses

Change error responses to make use of iver-wharf/wharf-core#9

Import on repo basis, instead of on project basis

This is based on RFC-0017: azuredevops-repos-and-not-projects.

---

In Azure DevOps:

• An organization may contain multiple projects
• A project may contain multiple repos
• A repo is what Wharf clones, finds the .wharf-ci.yml file in, and so on

While Wharf today imports each Azure DevOps project as a Wharf project. If an Azure DevOps project contains multiple repos, only the first one will be used.

Version endpoint

Follow example from iver-wharf/wharf-core#2 once it has been merged

Migrations in v2.0.0 didn't work

Expected

1. Have existing Wharf project with

   • Group name: MyGroup
   • Project name: MyProject

2. Refresh project after updating to v2.0.0 of wharf-provider-azuredevops

3. Have existing Wharf project renamed to:

   • Group name: MyGroup/MyProject
   • Project name: MyGitRepo

Actual

A new Wharf project was created with the name

• Group name: MyGroup/MyProject
• Project name: MyGitRepo

While the existing Wharf project was left untouched.

Checks integration in AzureDevOps

Add checks, or "statuses" as they're called in the Azure DevOps API. This implicitly makes a pull request not mergeable until the status is "succeeded"

API docs: https://docs.microsoft.com/en-us/rest/api/azure/devops/git/statuses/create?view=azure-devops-server-rest-5.0

On the commit:

In commit list:

---

We want to add this throughout the provider repos. So it would behoove us to figure out a way to do this in a generic manner.
Old thought was to use rabbitmq events, but that's no longer an obvious solution as it does in fact induce a lot of overhead for users trying to install Wharf. Would be better to just let the API talk with the providers directly.

Suggest writing an RFC on this: https://github.com/iver-wharf/rfcs

Change providers paths

From /import/azuredevops to /api/provider/azuredevops/import

And keep the old endpoint but mark it as deprecated, and we'll remove it some time in the future, such as on next major release

Map projects per ID: update importers code

Depends on iver-wharf/wharf-api#12, iver-wharf/wharf-api#15, & iver-wharf/wharf-api#16

Projects are currently mapped to their corresponding provider via name. Name can change, such as when a project is moved, but ID does not.

This is such a fundamentally impactful change that we need to be cautious and perhaps make a bunch of migration scripts.

Suggested change would be that the project in Wharfs DB is mapped to the git server and the project ID within that server. For example "gitlab.local" and "193".

---

There has come up a case where the current implementation caused a bug. Especially since the current GitLab importer actually maps the auth token with the project display name.

See the code (redacted link, it was outdated anyway, but pointed to somewhere in: https://github.com/iver-wharf/wharf-provider-gitlab/blob/master/import.go)
It would make more sense if it used the "path" instead of "name" property of the repo. To show the difference of the two, here a screenshot of GitLabs docs with an example project called "Diaspora Client" but with the path "dispora-client"

A project that is damaged by this is Foo Bar, which is accessed via the path "foo-bar" but has the name "Foo Bar" https://gitlab.local/default/foo-bar
When it tries to build docker images it builds them with the destination
harbor.local/default/Foo Bar:latest, which is an invalid docker image name.

---

There so much auto-magic and assumed relations tied together here. We need to find a way to migrate the data in as a painless and future-proof way as possible.

Migrating to basing projects off the project ID and server domain instead is a good step in my (@jilleJr's) opinion.
Then we can also import meta data for it such as it's full path so that we later can use that value for default docker image names instead of these home-baked spagetti relations.

Refactor: return problem.Response as error instead of -WritesProblem()

Currently to have fine-grained problem responses we have a lot of functions/methods with the suffix "-WritesProblem" and then returns true if they succeeded or false if they failed and wrote a problem to the gin context.

This would need us to add "NewProblem..." functions to the wharf-core/pkg/ginutil package, and a utility function that wrotes a returned problem response to the HTTP response, or converts a non-problem response to a problem response and then writes that instead to the HTTP response.

This way we could offload a lot of work from the endpoint methods by only using return values. Maybe we could wrap the functions as well so instead of being func(*gin.Context) to be func(*gin.Context) error where it uses that error->problem.Response that I mentioned in the above paragraph.

Just a thought. Could need an RFC to show both the wharf-core changes as well as the changes to the API repos as this would involve a lot of work and the specific implementation needs a good overview.

Need scheduled Trivy scans

Based on iver-wharf/iver-wharf.github.io#72

Tip is to look at https://github.com/iver-wharf/wharf-provider-gitlab/security/code-scanning/setup and see if there's any templates

Could probably just do make docker and inspect the result

Invalid gitURL when importing from https://dev.azure.com

Today, the gitURL is generated by a hardcoded format, as seen here:

wharf-provider-azuredevops/internal/importer/importer.go

Lines 274 to 285 in 7b63970

	func (i azureImporter) constructGitURL(groupName, projectName string) (string, error) {
	providerURL, err := url.Parse(i.provider.URL)
	if err != nil {
	fmt.Println("Unable to parse provider url: ", err)
	return "", err
	}
	const sshPort = 22
	gitURL := fmt.Sprintf("git@%s:%d/%s/%s/_git/%s", providerURL.Host, sshPort, groupName, projectName, projectName)
	return gitURL, nil
	}

It has worked nicely for self-hosted Azure DevOps instances before, but we hit a limit today when a customer tried to use it on https://dev.azure.com/ where the SSH URL uses a completely different domain.

This needs to be fixed. I suggest an RFC on this, as the approach might need us to import on an Azure DevOps repository basis instead of an Azure DevOps project basis.

Add configurable whitelist for remote provider base URLs

Summary

We want to add a configurable whitelist for the base URLs sent from iver-wharf/wharf-web to this provider plugin.

Motivation

If the base URL is provided by the end-user, there is the possibility for a security vulnerability. By whitelisting URLs we protect ourselves from this.

As an extra, having this list also opens up the possibility to provide a drop down list for the base URLs over at iver-wharf/wharf-web, should we want to have that in the future.

Example config file

# config.yaml
providerBaseURLWhitelist:
  - https://dev.azure.com/

Previous discussion

#14 (comment)

Security: Make TLSClientConfig.InsecureSkipVerify opt-in

@fredx30 See the warning in azuredevops.go InsecureSkipVerify should not be used in production code. https://github.com/iver-wharf/wharf-provider-azuredevops/security/code-scanning/5?query=ref%3Arefs%2Fpull%2F9%2Fmerge
Should probably be offloaded to an investigation.

Source: #9 (review)

What the security issue is referring to here is quite serious. We're disabling TLS verification when talking with the Azure DevOps API. Big no-no.

This code snippet is scattered around the repo:

wharf-provider-azuredevops/azuredevops.go

Line 283 in 2843ca9

http.DefaultTransport.(*http.Transport).TLSClientConfig = &tls.Config{InsecureSkipVerify: true}

As @Pikabanga said in his reply (#9 (comment)) it would be wiser to make it opt-in for those that really need it, instead of blindly having it enabled.

Need automated testing via `go test`

Based on iver-wharf/iver-wharf.github.io#75

Need to run Go tests and goimports formatting tests on commits and pull requests automatically.

As Wharf cannot do this yet, we should aim at using GitHub Actions.

Either we use the starter-workflow for Go https://github.com/actions/starter-workflows/blob/1d8891efc2151b2290b1d93e8489f9b1f41bd047/ci/go.yml which simply runs go test

Or we could look into a better integrated solution that could report failing tests as annotations inside the pull requests, such as:

• https://github.com/marketplace/actions/golang-test
• https://github.com/marketplace/actions/golang-test-annotations

Remove "Upload URL" from import modals

Continuation from iver-wharf/wharf-api#56

The Provider's upload URL is unused. It can safely be removed throught the repo, with no hindsight on backward compatibility.

We can add it back later if it turns out we can make use of it.