qlik-oss/enigma-go

Go library for consuming Qlik's Associative Engine.

enigma-go is a library that helps you communicate with a Qlik Associative Engine. Examples of use may be building your own analytics tools, back-end services, or other tools communicating with a Qlik Associative Engine.

---

• Installation
• Getting started
• Schemas
• Release
• Updating enigma-go to new engine schema

---

Installation

go get -u github.com/qlik-oss/enigma-go/v4

Getting started

Connecting to a Qlik Associative Engine and interacting with a document/app involves at least the following steps:

1. Create and set up a Dialer object with TLS configuration, etc.

2. Open a WebSocket to the Qlik Associative Engine using the Dial function in the Dialer.

3. Open or create a document/app using openDoc or createApp.

Refer to the examples section for more information.

Schemas

enigma-go includes generated API code that is based on the latest available Qlik Associative Engine schema. When a new schema is available, a new version of enigma-go will be made available.

Release

To release a new version of enigma-go you have to be on the master branch. From there you can run the release.sh script. The usage is:

./release.sh <major|minor|patch>

where the argument specifies what should be bumped. The release-script does a couple of things.

1. Creates a new version based on previous version-tag (if any, otherwise 0.0.0) and suffixes it with the QIX schema version as metadata. For example bumping minor when there are no previous tags will result in the version 0.1.0+12.429.0.
2. Generates a new API specification using the new version.
3. Adds the resulting api-spec.json file to a commit with the message Release: <version> for QIX schema version <qix_version>.
4. Creates a tag containing the version with the same message as in step 3.
5. Afterwards, adds another commit bumping the api-spec.json to latest again.

After the script has run, check the results. If everything looks good run:

git push --follow-tags

to push the tag and commit to master. The release-script also checks if the local repo is in a pristine state: no untracked files or uncommitted change and, you have to be up-to-date with the latest changes on master.

Updating enigma-go to new engine schema

• Generate enigma-go: Generate-Docs
• Update the API specification: Update-API-Docs

enigma-go's Issues

Improve testability and mockability in enigma-go

Description

This task includes improving the ability to mock in enigma-go for example that there no longer is a need to set id or handle etc.

Could you please share `schema.json` used for API autogeneration

Hi. Thank you for the great project.

I may be completely wrong, knowing nothing about go but I looked into /schema/generate.go sources anyway and it looks for me that you have pretty rich schema as an input for your code generation.
So is schema.json you use different from those used in enigma.js as https://github.com/qlik-oss/enigma.js/blob/master/schemas/12.67.2.json for example?
If so, could you please make it available to the community - including in this repo or by any other means?
It would be very helpful for those of us who try to use QIX in other statically typed languages (like Dart in my case)

Add example for setting cookies for a connection

In PR #29 the support for adding cookies for a connection was added. We should add an example for this as well. Useful when using enigma-go against a QS server.

SESSION_ATTACHED received during integration tests

When running the examples in Circle CI we get sporadic failures due to attaching to existing sessions instead of creating a new session. We need to investigate why this occurs and how we can mitigate it.

Example:
https://circleci.com/gh/qlik-oss/enigma-go/791

CustomType integration test fails in CCI

After the merge of #241 the integration test for custom types started failing in CCI. It does not seem to be related to the change itself since the test also fails when reverting the commit.

https://app.circleci.com/pipelines/github/qlik-oss/enigma-go/330/workflows/b1540f06-1718-4b20-9ce0-070f5e9f8c0f/jobs/2056

#!/bin/bash -eo pipefail
./examples/run_examples.sh


Starting example: basics/custom-type/custom-type.go
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x74453c]

goroutine 1 [running]:
github.com/qlik-oss/enigma-go/v3.(*Doc).CreateSessionObjectRaw(0x0, 0x850d10, 0xc00001c058, 0x7699e0, 0xc00052e000, 0x20, 0x796e60, 0x1)
	/home/circleci/project/qix_generated.go:6340 +0x7c
main.(*customDoc).CreateCustomObject(0xc00028fed8, 0x850d10, 0xc00001c058, 0xc00052e000, 0x84c6a0, 0xc000594168, 0x0)
	/home/circleci/project/examples/basics/custom-type/custom-type.go:78 +0x5c
main.main()
	/home/circleci/project/examples/basics/custom-type/custom-type.go:60 +0x554
exit status 2

Exited with code exit status 1
CircleCI received exit code 1

Include golang v1.17 as part of the integration tests in CCI

With a released v1.17 of golang we should include it in the Circle CI testing pipeline. The current way of testing different versions in the config.yml can also probably simplified using the build matrix support, which would make it easier to add future versions as well.

workflows:
  build:
    jobs:
      - build:
          matrix:
            parameters:
              go_version:
                - "1.14"
                - "1.15"
                - "1.16"
                - "1.17"

Fix go.mod for major version release v2

The v2 major release of enigma-go also required some changes to go.mod that was not part of the release.

The following intake failed for core-using-licenses:
qlik-oss/core-using-licenses#160

Command failed: docker run --rm --name=renovate_go --label=renovate_child -v "/mnt/renovate/gh/qlik-oss/core-using-licenses":"/mnt/renovate/gh/qlik-oss/core-using-licenses" -v "/tmp/renovate-cache":"/tmp/renovate-cache" -v "/tmp/renovate-cache/others/go":"/tmp/renovate-cache/others/go" -e GOPATH -e CGO_ENABLED -w "/mnt/renovate/gh/qlik-oss/core-using-licenses" docker.io/renovate/go:1.16.4 bash -l -c "git config --global url.\"https://**redacted**@github.com/\".insteadOf \"https://github.com/\" && go get -d ./... && go mod tidy && go mod tidy"
go: github.com/qlik-oss/enigma-go/[email protected]: go.mod has non-.../v2 module path "github.com/qlik-oss/enigma-go" (and .../v2/go.mod does not exist) at revision v2.0.0

According to golang module docs a major release should include:
https://golang.org/doc/modules/major-version

Opening a second Sense app, quickly after the first, sometimes results in error.

Description

Opening a second Sense app, quickly after the first, sometimes results in error.

Steps to Reproduce

Start a local Sense engine, and populate it with two apps (called app_1 and app_2)
Use engine uri "ws://localhost:9076/"

1. Connect to Sense engine and open app_1
2. Disconnect from Sense engine
3. Connect to Sense engine and open app_2

Expected behavior

app_2 opens w/o error

Actual behavior

app_2 fails to open, and I get an error like:

<*enigma.qixError | 0xc00026e510>: {
ErrorCode: 1002,
ErrorParameter: "app_2.qvf",
ErrorMessage: "App already open",
}

The error occurs frequently, but not always. Putting a small (~1s) sleep between disconnect and connect reduces the chances of encountering the error. Using a random engine url in each new connect (i.e. Dial("ws://localhost/app/[randomstr]")) seems to cause the problem to not occur.

Environment

Operating system

[ ] Windows
[ ] OSX
[X ] Linux

Versions

• Version of Go (go version):
  go1.11.1 linux/amd64

Example

global, err := dialer.Dial(ctx, "ws://localhost:9076/", nil)
doc, err := global.OpenDoc(ctx, "app_1.qvf", "", "", "", false)
global.DisconnectFromServer()

//time.Sleep(1 * time.Second)

global, err = dialer.Dial(ctx, "ws://localhost:9076/", nil)
doc, err = global.OpenDoc(ctx, "app_2.qvf", "", "", "", false)
global.DisconnectFromServer()

Sheet title not included in GenericObject Meta properties

Description

I'm trying to get the title of a sheet in QCB local dev environment this way

sheet, err := doc.GetObject(context, "2496d12e-02bc-47a5-964c-f11989d8152e")
l, err := sheet.GetLayout(context)

but l.Meta as not the object I'm expecting, it contains only one Name property (which is actually empty).

In the Qlik Sense Enterprise dev hub the GetLayout method gives me this:

and the title is there.

Expected behavior

Expecting to get a Meta object that includes the title property.

Am I missing something? Any suggestion? Thanks

Environment

QSB local dev

Operating system

[ ] Windows
[ ] OSX
[x] Linux

Create API specification usable for API governance application

Investigate how and possibly create way to generate API specification which can be parsed and used by the API Governance application.

Describe scope, estimated size and design

all available versions of enimga-go module does not contain v2.0.0 ... v3.0.1.

$>go list -m -versions github.com/qlik-oss/enigma-go
github.com/qlik-oss/enigma-go v0.1.0 v0.1.1 v1.0.0 v1.1.0 v1.1.1 v1.1.2 v1.2.0

Design: Move to a 1.0 release

Create design for removal experimental flag and move to 1.0 release

DoD:

• Ensure we have proper testing
• Update necessary deploy scripts
• Ensure we have proper API governance

Possibility to handle pushed changes/closing of objects when not using ChangedChannel.

Description

Engine can "push" changes to objects, e.g.

{"jsonrpc":"2.0","change":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23],"suspend":[1]}

When this happens and we use change channels to subscribe to changes, the changes are triggered correctly. However with the introduction of #35 and the possibly to react on changes synchronously we do not currently have a way to catch the pushed changes and closing of objects.

A suggestion is to e.g. introduce a new channel for pushed changes similar to the SessionMessageChannel.

We will also need similar logic for closed objects pushed by engine.

Steps to Reproduce

There's many was to get a pushed change message, here is one easily reproducible:

1. Connect a session to a sense app in e.g. Qlik Sense Server
2. Trigger a reload externally (e.g. through QMC task)
3. Wait until reload is finished and the enigma session should receive a push message.

Expected behavior

I expect a way to react on pushed changes when not using ChangedChannel.

Actual behavior

Currently doesn't exist.

Environment

Operating system

[x] Windows
[x] OSX
[x] Linux

Versions

Split generated code into multiple files

Description

This task includes splitting the generated code into multiple files. Currently when generating a new enigma-go based on a schema, all generated code ends up in qix_generated.go. Instead the code could be split up in e.g. global, doc etc.

Clarify bookmark creation

Description

In the SDK, CreateBookmark accepts a custom BookmarksProperties object that contains, among other things, the modified fields and current sheet. In enigma-go, the equivalent method only accepts a generic NxInfo and NxMetaDef. There is a CreateBookmarkRaw method that accepts an interface, so it's possible to pass properties dynamically.

Create a helper struct to enable full functionality using CreateBookmark or document how to use CreateBookmarkRaw with an example.

Generate enigma-go from OpenAPIv3 specification

Add support for new features in the engine such as enum

Add examples for enabling traffic logging to stdout or to a file

We should add examples for logging json traffic to file and to stdout. When debugging complex scenarios towards an engine this is crucial info.

Generate enigma-go from OpenAPI v3 specification

Currently enigma-go is generated from the JSON RPC specification. If we want enigma-go to include the enums from the API we need to generate from the OpenAPI specification instead.

The specification can be fetched on http://localhost:9076/openapi/rpc from a running container.

create a make lint target

This setup is very strange.
I suggest that we create a backlog issue to add make lint target that will be called from this file. That way we can test it in local env as well.

Originally posted by @bojanth in #280 (comment)

chartObject.GetProperties() is missing many chart properties

Description

It appears that the GetProperties and GetFullPropertyTree methods on a chart object do not return all of the properties of the chart.

Steps to Reproduce

1. Create (or find) and app that has at least one sheet and one chart
2. Make sure the chart has a title property value
3. Use a program that looks as follows to loop through all the sheets and all the charts of each sheet to obtain the properties of each chart (before calling the function, open a new enigma doc session on the app and pass the "doc" object to the function)

func GetAllChartProperties(ctx context, doc *enigma,doc) {
        sheetListObject, e := doc.CreateSessionObject(ctx, &enigma.GenericObjectProperties{
		Info:             &enigma.NxInfo{Type: "SheetList"},
		AppObjectListDef: &enigma.AppObjectListDef{Type: "sheet", Data: sheetContents},
	})
	sheetListObjectLayout, e := sheetListObject.GetLayout(ctx)
	if e != nil {
           fmt.Printf("error obtaining sheetList object:", e.Error())
           return
	}
	if sheetListObjectLayout != nil && sheetListObjectLayout.AppObjectList != nil {
		for _, sh := range sheetListObjectLayout.AppObjectList.Items {
			shID := sh.Info.Id
			shObj, err := doc.GetObject(ctx, shID)
			var shData models.QSheetData
			e := json.Unmarshal(sh.Data, &shData)
			if e != nil {
				fmt.Printf("error unmarshalling sheet data %s:", e.Error())
				continue
			}
			var sheetChartRefs []string
			for _, ch := range shData.Charts {
				// get the details of this chart
				chartObj, e := doc.GetObject(ctx, ch.Name)
				if err != nil {
                                     fmt.Printf("error getting chart object '%s': %s", ch.Name, e.Error())
                                     continue
				} else {
			            chartProps, err := chartObj.GetProperties(ctx)
                                     // EXAMINE chartProps IN DEBUGGER HERE AND NOTE THE ABSENCE OF title, subtitle, etc. properties
                                 }
                         }
                 }
        }
}

Expected behavior

It is expected to find properties like title, subtitle, footnote, etc. but these properties are not included in the results of the chartObject.GetProperties method call.

Actual behavior

The results of the chartObj.GetProperties call includes content as shown below:

Environment

MacOS Big Sur 11.6.1, enigma-go v3.2.0

Operating system

[ ] Windows
[x] OSX
[ ] Linux

Versions

• Version of Go (go version): 1.17
• enigma-go v3.2.0

The release script failed to create commits when executed in git bash Windows

Recreate:
Run the release script on windows and check if commits are created.

Fetch specs

Add a weekly scheduled github action to fetch the latest engine spec, generate code and create a pr.
This is similar to what exists in other repos, example: qlik-sdk-typescript.

Improve how GenericObject return-types are handled

Description

Improve how GenericObject return-types are handled.

Currently those return-type hare mapped in a file called schema-companion.json - there's probably a better way to handle this

Note: Preferably enough information should be included in the Open-RPC spec to determine the return type.

Revisit the release (cron.sh) pipeline

We should revisit the cron.sh pipeline that is in charge of generating code. It will probably also have to be in charge of bumping versions and generating the package specification.

DoReloadEx runs indefinitely when using context from WithReservedRequestID

Description

DoReloadEx() never returns when using the context returned from doc.WithReservedRequestID()

Steps to Reproduce

1. Call doc.WithReservedRequestID(ctx)
2. Call doc.DoReloadEx(ctxWithReservedRequestID)
3. Observe reload attempt run indefinitely without returning

Expected behavior

DoReloadEx should return once the data has been reloaded, even if the context provided is the one returned from doc.WithReservedRequestID()

Actual behavior

Reload (even for an empty app) does not finish until the caller of DoReloadEx is forcefully exited.

Passing in the existing context instead of the ctxWithReservedRequestID consistently fixes the problem.

Environment

Local QCS

Operating system

[ ] Windows
[x] OSX
[ ] Linux

Versions

• Version of Go (go version): 1.13
• [Other relevant versions]
  Engine 12.741.0

Investigate code coverage of unit and integration tests

We need to be able to measure the code coverage of the testing being done in enigma-go. Investigate if e.g. SonarCube can be used as a gatekeeper for PRs and in CCI.

Cron job for generating from schema is broken

The bash script for updating the library when a new schema is available is broken. Incorrect generated golang file is compared to master branch of the repo.

Not wrapping engine 7002 error on GetLayout for objects with large amount of data

Description

Enigma-go is not wrapping engine 7002 error on GetLayout for objects with large amount of data.
To be more specific: https://github.com/qlik-oss/enigma-go/blob/master/qix_generated.go#L3082

Steps to Reproduce

1. GetObject with the id of a pivot-table of millions of cells
2. Call ExpandLeft with all parameter set to true https://github.com/qlik-oss/enigma-go/blob/master/qix_generated.go#L10010 . No errors are returned.
3. Call GetLayout . The layout of the object is returned with an HyperCube.Size set to zero and a property Error set to 7002 with no description. No errors are returned.

Expected behavior

On point 3 I would expect to get an enigma-go error with 7002 code (https://github.com/qlik-oss/enigma-go/blob/master/qix_generated.go#L3082

Actual behavior

No errors are returned from the GetLayout call.
Causing some possible subsequent calls to fail with other error codes (like 6002 invalid layout).

Environment

Operating system

[ ] Windows
[ ] OSX
[x] Linux

Versions

• Version of Go (go version): go 1.13
• [Other relevant versions]

run enigma-go release script from github actions

Mitigates problems seen when releasing from windows

Validate that enigma-go can be built with multiple golang versions

We need to secure that we are backwards compatible with older golang versions e.g. 1.11 and 1.12 when we switch to 1.13. Set up a circle ci job that executes on multiple golang docker images.

Add CCI targets for golang 1.14 and 1.15

We currently only test golang versions 1.11 - 1.13 in the CCI pipeline.

Disable HTML-escaping data

Downstream issue qlik-oss/corectl#346

Probably due to the use of the default JSON encoder in golang.