axiomhq/axiom-go

Official Go bindings for the Axiom API

Axiom unlocks observability at any scale.

• Ingest with ease, store without limits: Axiom's next-generation datastore enables ingesting petabytes of data with ultimate efficiency. Ship logs from Kubernetes, AWS, Azure, Google Cloud, DigitalOcean, Nomad, and others.
• Query everything, all the time: Whether DevOps, SecOps, or EverythingOps, query all your data no matter its age. No provisioning, no moving data from cold/archive to "hot", and no worrying about slow queries. All your data, all. the. time.
• Powerful dashboards, for continuous observability: Build dashboards to collect related queries and present information that's quick and easy to digest for you and your team. Dashboards can be kept private or shared with others, and are the perfect way to bring together data from different sources.

For more information check out the official documentation and our community Discord.

Quickstart

Install using go get:

go get github.com/axiomhq/axiom-go/axiom

Import the package:

import "github.com/axiomhq/axiom-go/axiom"

If you use the Axiom CLI, run eval $(axiom config export -f) to configure your environment variables.

Otherwise create a personal token in the Axiom settings and export it as AXIOM_TOKEN. Set AXIOM_ORG_ID to the organization ID from the settings page of the organization you want to access.

Note

The organization ID is the slug below your organizations full name in the Axiom settings which resembles the full name. It has a copy button next to it. Alternatively you can just extract it from your browsers address bar when browsing the Axiom App: https://app.dev.axiomtestlabs.co/<your-org-id>/datasets.

You can also configure the client using options passed to the axiom.NewClient function:

client, err := axiom.NewClient(
    axiom.SetPersonalTokenConfig("AXIOM_TOKEN", "AXIOM_ORG_ID"),
)

Note

When only performing ingest or query operations, we recommend using an API token with minimal privileges, only! Create an API token with the appropriate scopes in the Axiom API tokens settings and export it as AXIOM_TOKEN.

Create and use a client like this:

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/axiomhq/axiom-go/axiom"
    "github.com/axiomhq/axiom-go/axiom/ingest"
)

func main() {
    ctx := context.Background()

    client, err := axiom.NewClient()
    if err != nil {
        log.Fatalln(err)
    }
    
    if _, err = client.IngestEvents(ctx, "my-dataset", []axiom.Event{
        {ingest.TimestampField: time.Now(), "foo": "bar"},
        {ingest.TimestampField: time.Now(), "bar": "foo"},
    }); err != nil {
        log.Fatalln(err)
    }

    res, err := client.Query(ctx, "['my-dataset'] | where foo == 'bar' | limit 100")
    if err != nil {
        log.Fatalln(err)
    }
    for _, match := range res.Matches {
        fmt.Println(match.Data)
    }
}

For further examples, head over to the examples directory.

If you want to use a logging package, check if there is already an adapter in the adapters directory. We happily accept contributions for new adapters.

License

Distributed under the MIT License.

axiom-go's Issues

Change Op in EntryGroupAgg to Alias

What the server returns as "op" isn't named correctly, should be an alias.

axiom-go/axiom/query/result.go

Line 216 in fd368e7

Op AggregationOp `json:"op"`

Change to: Alias string json:"op"

where to obtain `org_id`

me stupid, already figured out but...

when README.md mention about AXIOM_ORG_ID it should mention it's the simple string under your organisation's name in setting's page, otherwise I have been hunting for a long uuid with title Organisation Id on settings' page and found nothing.

when you provide me a click to copy button it doesn't indicate anything that is an org id, I only found out after I

1. ran axiom --help and in the FLAGS it told me what's my org id
2. google axiom org id and found this blog, this should be under any sdk uses or mention org_id

little thing, could save people minutes

HTTP2 errors with ingestion

http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error

Solved by setting request.GetBody(), axiom-go would need to provide an additional ingest api, something like

client.IngestStream(ctx, "my-dataset", func() (io.ReadCloser, error) {
	return os.Open("/some/file")
})

so that the go http client can retry correctly

Example should include IngestEvents and not Ingest stream since its not developer friendly

Developers by default want to create Event objects and ingest them and not create Streams. Please make this a bit more friendly.

How do I pass time to the events in the client

README is missing an example of passing events with custom timestamps!

No Client option to disable retries

Axiom go client Options doesn't include an Option to disable retries which is enabled by default.

It would be nice if you could opt out of the automatic retry feature

Support for Axiom Integrations

Support the /integrations endpoint of the Axiom API.

Accept APL time ranges on queries and shared access filters

Axiom 1.19 supports APL time ranges on query endpoints and the filters for shared access signatures. This should be addressed in this package as well.

This might be a good option to give Go Generics (coming in Go 1.18 Mid February 2022) a shoot. Instead of having interface{}/any, we could define a custom constraint for time.Time | APL where APL is basically a string based type with some sugar on it.

add queries docs to axiom-go

https://pkg.go.dev/github.com/axiomhq/axiom-go/axiom

Ability to ingest OTEL data using proto format

Is it possible to add ability to ingest to OTEL endpoint using proto format?

This is what I am currently using for one of my services to send logs:

req := plogotlp.NewExportRequestFromLogs(logs)
		data, err := req.MarshalProto()
		if err != nil {
			log.Println(err)
			return err
		}

		httpReq, err := http.NewRequest("POST", axiomUrl+"/v1/logs", bytes.NewBuffer(data))
		if err != nil {
			log.Println(err)
			return err
		}

		httpReq.Header.Set("X-Axiom-Dataset", axiomDatasetName)
		httpReq.Header.Set("X-Axiom-OrgId", axiomOrgId)
		httpReq.Header.Set("Authorization", "Bearer "+ axiomToken)
		httpReq.Header.Set("Content-Type", "application/x-protobuf")

		_, err = http.DefaultClient.Do(httpReq)
		if err != nil {
			log.Println(err)
			return err
		}