apache/airflow-client-go

Apache Airflow - OpenApi Client for Go

Airflow Go client

Go Airflow OpenAPI client generated from openapi spec.

Install

go get github.com/apache/airflow-client-go/airflow@latest

Usage

package main

import (
	"context"
	"fmt"
	"github.com/apache/airflow-client-go/airflow"
)

func main() {
	conf := airflow.NewConfiguration()
	conf.Host = "localhost:8080"
	conf.Scheme = "http"
	cli := airflow.NewAPIClient(conf)

	cred := airflow.BasicAuth{
		UserName: "username",
		Password: "password",
	}
	ctx := context.WithValue(context.Background(), airflow.ContextBasicAuth, cred)

	variable, _, err := cli.VariableApi.GetVariable(ctx, "foo").Execute()
	if err != nil {
		fmt.Println(err)
	} else {
		fmt.Println(variable)
	}
}

See README for full client API documentation.

airflow-client-go's Issues

Release versions?

Could you please specify release versions?
It would be great if they can match Airflow version

Request.Execute() return `undefined response type` error

➜  ~ curl -i -X GET 'https://airflow.xxx.com/api/v1/dags/xx/dagRuns/xxx' \
-H 'Content-Type: application/json' -H 'Accept: application/json' \
--user "xxx:xxx" \
-d '{}'
HTTP/2 404
date: Wed, 01 Feb 2023 07:42:55 GMT
content-type: application/problem+json
content-length: 263
server: gunicorn
x-robots-tag: noindex, nofollow

{
  "detail": "DAGRun with DAG ID: 'xxx' and DagRun ID: 'xxx' not found",
  "status": 404,
  "title": "DAGRun not found",
  "type": "https://airflow.apache.org/docs/apache-airflow/2.5.0/stable-rest-api-ref.html#section/Errors/NotFound"
}

In airflow version 2.5.0, Content-type header in error response is application/problem+json, but in this repo the json content-type is check by regexp

jsonCheck = regexp.MustCompile(`(?i:(?:application|text)/(?:vnd\.[^;]+\+)?json)`)

The regexp can't match application/problem+json, I guess that's the problem.

About problem+json

How to create dag through restful api or function？

I cant't find the api which can create dag in the restful apis. Does it support creating dag through API？

How to set variable

Background

Currently, I am using this package for interacting with airflow. I see variable component. It has the Get method. But, I didn't see the Set method for the variable.

Proprosal Solution

• Provide Set method for variable

Parsing of Schedule Interval fails (possibly always)

https://github.com/apache/airflow-client-go/blob/main/airflow/model_schedule_interval.go#L60-L114

This function tries to parse the data such as

{
  "__type": "CronExpression",
  "value": "0 * * * *"
}

But it always matches more than 1. This is because of two things

1. __type is a common field all 3 structs and parsing will never result in '{}'
2. There are other fields that are common between TimeDelta and RelativeDelta like hours, seconds etc. And rest of the fields end up being 0.

I'm sending a PR to fix this.

`go get github.com/apache/airflow-client-go@latest` yields outdated `github.com/apache/airflow-client-go/airflow` subpackage

I want to use the current version (v2.5.0), but using go get github.com/apache/airflow-client-go@latest as of today gives me v0.0.0-20230203175943-7af9875e7d4c. It definitely appears out of sync with the latest version as NewClearTaskInstance() function is missing and in addition the fields for theClearInstance struct appear out of sync with those of the current version.

go version: go1.19.5 darwin/arm64

Add tag v1.0.0 for the release version

Airflow API version is 1.0.0 (https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html)

AFAIK, all we have to do is make a tag v1.0.0 on the repo. This should set the client version for the Go module (official docs)

Of course, before that, there should be some kind of client versioning strategy in place. One simple strategy is to follow the API version, so every time when you bump the API version, you rebuild the client. However, sometimes generated library needs an update (e.g due to a bug in the client generation). If this happens often, you might simply have independent versioning of the client, while you maintain at least the major version. In that case you must have a mapping table that indicates client version to api version matching.

Either way, the first version tag should be added :)

Trigger dag_run response: Property is read-only - 'dag_id' - 400

func workoutDagTask(workout_id int64, workout_path string) {
	cli := airflow.NewAPIClient(&airflow.Configuration{
		Scheme:   "http",
		Host:     "localhost:8080",
		BasePath: "/api/v1",
	})
	cred := airflow.BasicAuth{
		UserName: "admin",
		Password: "admin",
	}
	ctx := context.WithValue(context.Background(), airflow.ContextBasicAuth, cred)
	var dagRunRequest airflow.DagRun
	var dagRunResponse airflow.DagRun
	dagRunRequest = airflow.DagRun{
		Conf: map[string]interface{}{
			"workout_id": workout_id,
			"workout_path": workout_path,
		},
	}
	dagRunResponse, httpResponse, err := cli.DAGRunApi.PostDagRun(ctx, "workout_dag", dagRunRequest)
	if err != nil {
		fmt.Println(err, httpResponse)
	} else {
		fmt.Println(dagRunResponse)
	}
}

httpResponse: 400 Bad Request

{
"detail": "Property is read-only - 'dag_id'",
"status": 400,
"title": "Bad Request",
"type": "https://airflow.apache.org/docs/2.0.1/stable-rest-api-ref.html#section/Errors/BadRequest"
}

Model DagRun : dag_id default "", but rest-api can not receive dag_id argument ?

Publish documentaiton on airflow.apache.org

Hello,

We should publish the documentation or at least link to this library on our website in the Documentation section.
http://airflow.apache.org/docs/

Now no one is aware of the existence of this library.

Best regards,
Kamil Breguła

Add LICENSE file to the repo

Currently the repo has no license file and pkg.go.dev isn't displaying the documentation for the package. Clarifying the license would also encourage the usage of the client.

The individual files say the file is available under Apache 2.0 license, so I presume this is the license we'd want to add to the repo as well.

(I've created a PR for this. I'm unsure how to notify, so creating an issue and linking it to resolve.)