Support non-SSL OCI registry for composition local development

[markphillips100]

Support non-SSL OCI registry for composition local development

[Peefy]

The windows path issue is fixed once again in KCL v0.8.2 and crossplane funcion-kcl v0.3.2 cc @zong-zhe

[zong-zhe]

Hi @markphillips100 😊

Your configuration of setting DefaultOciPlainHttp to true is correct. With this setting, you should be able to interact with your local registry. Additionally, you can control the activation of this feature through the respective environment variable OCI_REG_PLAIN_HTTP. After setting it, I would recommend that you try restarting the service to allow the environment variable to take effect, as this configuration is not hot-pluggable. It relies on the package management mechanism that loads these settings before each process startup

OCI_REG_PLAIN_HTTP=on go run . --insecure --debug

If it still doesn't work, it may be a bug, please @me 😊

[markphillips100]

Okay so have worked out why the missing package errors. I had Go v1.18.1 so bumped to latest 1.22.1. No longer any errors, except the go generate still won't work due to the missing rm command.

I start the function locally using go run . --insecure --debug from one powershell terminal, and then ran the example @Peefy mentioned using crossplane beta render xr.yaml composition.yaml functions.yaml -r. That renders fine.

I tried using a local path to my own composition's main.k and whilst this time the function finds the main.k file, it fails to run it due to an import models. This is a subfolder containing kcl-openapi generated schemas.

Back to oci testing, I switch back to using oci://localhost:7900 in my composition and can see in the registry docker logs that the kcl-function is indeed accessing the registry. It makes 3 GET requests; tags/list, manifests, and blobs. However, I presume the function is attempting to copy the download to a temp folder and this fails, and I'm guessing because of the port in the name perhaps?

"failed to ensure directories of the target path: mkdir C:\Users\mark_\AppData\Local\Temp\1086188234\localhost:7900: The directory name is invalid."

[zong-zhe]

Hi @markphillips100 😊

Thanks for the response. The issue you've encountered is a bug that appears on the Windows platform.

"failed to ensure directories of the target path: mkdir C:\Users\mark_\AppData\Local\Temp\1086188234\localhost:7900: The directory name is invalid."

We have acknowledged this problem and have begun working on a resolution. We greatly appreciate your feedback.

[markphillips100]

Thank you. I shall give it a try tmrw.

[zong-zhe]

@markphillips100, thanks for the feedback, I will continue to try to resolve it.

[markphillips100]

Thanks @Peefy and @zong-zhe . I'll get on it asap tmrw.

[markphillips100]

As you've opened a new issue for folder imports I'm happy this one can be closed if you are. Thanks once again @Peefy and @zong-zhe

[markphillips100]

I think I will. Only alternative is to use an Azure Container Registry resource and I really don't want to do that for innerloop testing.

Thanks for your quick answer.

[Peefy]

Yes. I think supporting it is feasible and should not have high complexity.

[Peefy]

cc @zong-zhe

[Peefy]

I've added a example for the local path (Currently, it is relative to the path where the function server is located). #44

Besides, @zong-zhe Can you help reply the non-SSL OCI registry for composition local development?

[markphillips100]

@zong-zhe I'm not sure what "service" you are referring to? My current use case is to use kcl mod push... to load the package to a local docker regisrty:2 and then crossplane beta render... to render out a composition that is using a KCLRun via an "oci://" source.

The kcl cli command works fine, the kcl crossplane function does not when used from a crossplane cli. Note, I haven't even tried applying the composition to a cluster yet but obviously will need a solution that works from a cluster.

I set OCI_REG_PLAIN_HTTP=on in my windows user environment variables, confirmed in a new powershell terminal that the env var is set, and then ran my crossplane beta render... command. Same error occurs: "http: server gave HTTP response to HTTPS client"
Keep in mind please that I am not a Go developer.

[markphillips100]

I'm wondering if I am getting confused with the usage of kcl-function when it comes to crossplane locally. I totally missed the part about running the function itself locally, which is why I presume you put go run . --insecure --debug?

Are their instructions for this setup somewhere I missed?

I cloned the function-kcl repo to my windows environment and tried the go run . --insecure --debug but get 2 errors:

C:\Users\mark_\go\pkg\mod\github.com\go-json-experiment\[email protected]\fields.go:8:2: package cmp is not in GOROOT (C:\Program Files\Go\src\cmp)
C:\Users\mark_\go\pkg\mod\github.com\go-json-experiment\[email protected]\internal\jsonwire\decode.go:10:2: package slices is not in GOROOT (C:\Program Files\Go\src\slices)

I also tried the first command from the repo's readme: go generate ./... and that at least appears to download some packages, it then fails as generate.go is looking for the rm command which does not exist on windows. So my guess is this repo cannot be developed on windows at all, if it needs to be in order to use crossplane with a local function?

[Peefy]

cc @zong-zhe

[Peefy]

There are two ways for crossplane beta rendering to run locally, using go run to start the function server to perform rendering, so it can see the environment variables you set. Another way is that it uses Docker to start the function, and it cannot see the environment variables OCI_REG_PLAIN_HTTP=on you set.

We can ensure that the folder of the function is structured like this, which ensures that the 'import models' work properly.

.
├── kcl.mod
├── main.k
└── models

[Peefy]

The windows path issue is fixed in KCL v0.8.1 and crossplane funcion-kcl v0.3.1

[markphillips100]

Still getting the directory issue:
"failed to ensure directories of the target path: mkdir C:\Users\mark_\AppData\Local\Temp\3264626002\localhost:7900: The directory name is invalid."

Steps I followed:

1. Pulled latest function-kcl repo, use "main" branch: ran go run . --insecure --debug. I can see the various v0.8.1 packages downloaded:

go: downloading k8s.io/apimachinery v0.29.3
go: downloading kcl-lang.io/krm-kcl v0.8.1
go: downloading kcl-lang.io/cli v0.8.1
go: downloading github.com/golang/protobuf v1.5.4
go: downloading kcl-lang.io/kcl-go v0.8.1
go: downloading kcl-lang.io/kpm v0.8.2-0.20240315081549-f70f6ff67ef5
go: downloading kcl-lang.io/lib v0.8.1

2. Use v0.3.1 in function yaml that's passed to render:

apiVersion: pkg.crossplane.io/v1beta1
kind: Function
metadata:
  name: kcl-function
  annotations:
    # This tells crossplane beta render to connect to the function locally.
    render.crossplane.io/runtime: Development
spec:
  package: xpkg.upbound.io/crossplane-contrib/function-kcl:v0.3.1

crossplane beta render... reports:

failed to ensure directories of the target path: mkdir C:\Users\mark_\AppData\Local\Temp\3264626002\localhost:7900: The directory name is invalid.

Function service reports image is being pulled from localhost:7900 and no other info.

[Peefy]

cc @zong-zhe

[markphillips100]

That seems to work okay. I have a separate error in my kcl code (my error) but that at least showed me the new temp path being used, "localhost_7900" in my case.

I haven't tried the local path use case where main.k imports other folders. Is the expectation that it should work also or should I wait for a fix for that?

[zong-zhe]

Hi @markphillips100 😊

You can give it a try, and if there are some other errors that you don't understand, you can provide us with more detailed error information.

[Peefy]

Make sense. The local path currently only supports single files, and we will implement it as a feature. I've opened an issue kcl-lang/krm-kcl#42 to track this feature.

[markphillips100]

Sorry @Peefy I have a further issue/question related to this. Let me know if you wish a new issue to be created, or if it's a Crossplane issue rather than a function-kcl one.

Local crossplane beta render... works perfectly fine with a local docker registry. I believe crossplane detects the render.crossplane.io/runtime: Development annotation and determines it should look for the Go service that's running. That's all fine.

However, I now want to test an actual manifest apply of my composition against my local KinD cluster (where crossplane is installed) and again use the same local docker registry. My guess is the function annotations are only supported from the crossplane cli and not the crossplane controller?

I tried these 2 annotations together to attempt to point the controller at the registry on my host but I see no activity in the Go service so think the crossplane controller is ignoring it:

  annotations:
    # This tells crossplane beta render to connect to the function locally.
    render.crossplane.io/runtime: Development
    render.crossplane.io/runtime-development-target: dns:host.docker.internal

[Peefy]

Hello @markphillips100 Do you mean to allow the crossplane controller in the cluster to access the kcl function of the local service?

[markphillips100]

Yes. So that the local service can then access the local registry using http://localhost:7900 insecurely. Basically same setup as for crossplane cli rendering except the crossplane controller now talks to the service instead of the cli. Is this possible do you think?

[Peefy]

Make sense. I have reviewed the documentation of Crossplane and did not find this point https://docs.crossplane.io/v1.15/concepts/composition-functions/#test-a-composition-that-uses-functions .

However, in my personal opinion, it is feasible, for example, to expose the service of the crossplane controller locally through methods such as kubectl port forward so that it can access local functions, which is very helpful for the development and debugging process. Perhaps you can open an issue on crosslane official repo to track or ask this question.

[markphillips100]

@Peefy I raised the issue/enhancement over at crossplane re crossplane controller access to remote function.

Just wondering though, as the KCLRun is handled by function-kcl, is the download of the source oci package handled by function-kcl also? If so then is it possible that the https download that is executed could support an insecure http download perhaps by an annotation on the KCLRun kind?

[Peefy]

cc @zong-zhe

[Peefy]

OCI Source is managed by function kcl and should be able to be set through the fields of KCLRun

[Peefy]

OCI Source is managed by function kcl and should be able to be set through the fields of KCLRun

[markphillips100]

So in theory something like this perhaps?

    functionRef:
      name: kcl-function
    input:
      apiVersion: krm.kcl.dev/v1alpha1
      kind: KCLRun
      metadata:
        name: basic
        annotations:
          kcl-something/allow-insecure-source: True
      spec:
        target: Resources
        source: oci://localhost:7900/my-composition-kcl:0.0.1

[markphillips100]

@Peefy would you prefer a new issue to track that feature or just reopen this one?

[Peefy]

Sure, I've opened a new issue. #47 ❤️