networkservicemesh / cmd-nsmgr Goto Github PK

View Code? Open in Web Editor NEW

5.0 5.0 20.0 928 KB

Network Service Manager

License: Apache License 2.0

Dockerfile 2.30% Go 97.70%

cmd-nsmgr's Introduction

Intro

This repo contains 'nsmgr' that implements the Network Service Manager.

This README will provide directions for building, testing, and debugging that container.

Build

Build nsmgr binary locally

You can build the locally by executing

go build ./...

Build Docker container

You can build the docker container by running:

docker build .

Usage

Environment config

NSM_NAME - Name of Network service manager (default: "nmgr")
NSM_LISTEN_ON - url to listen on. tcp:// one will be used a public to register NSM. (default: "unix:///var/lib/networkservicemesh/nsm.io.sock")
NSM_REGISTRY_URL - A NSE registry url to use (default: "tcp://localhost:5001")
NSM_MAX_TOKEN_LIFETIME - maximum lifetime of tokens (default: "10m")
NSM_REGISTRY_SERVER_POLICIES - paths to files and directories that contain registry server policies (default: "etc/nsm/opa/common/..rego,etc/nsm/opa/registry/..rego,etc/nsm/opa/server/.*.rego")
NSM_REGISTRY_CLIENT_POLICIES - paths to files and directories that contain registry client policies (default: "etc/nsm/opa/common/..rego,etc/nsm/opa/registry/..rego,etc/nsm/opa/client/.*.rego")
NSM_LOG_LEVEL - Log level (default: "INFO")
NSM_DIAL_TIMEOUT - Timeout for the dial the next endpoint (default: "750ms")
NSM_FORWARDER_NETWORK_SERVICE_NAME - the default service name for forwarder discovering (default: "forwarder")
NSM_OPEN_TELEMETRY_ENDPOINT - OpenTelemetry Collector Endpoint (default: "otel-collector.observability.svc.cluster.local:4317")
NSM_METRICS_EXPORT_INTERVAL - interval between mertics exports (default: "10s")

Testing

Testing Docker container

Testing is run via a Docker container. To run testing run:

docker run --rm $(docker build -q --target test .)

Debugging

Debugging the tests

If you wish to debug the test code itself, that can be achieved by running:

docker run --rm -p 40000:40000 $(docker build -q --target debug .)

This will result in the tests running under dlv. Connecting your debugger to localhost:40000 will allow you to debug.

-p 40000:40000

forwards port 40000 in the container to localhost:40000 where you can attach with your debugger.

--target debug

Runs the debug target, which is just like the test target, but starts tests with dlv listening on port 40000 inside the container.

Debugging the nsmgr

When you run 'nsmgr' you will see an early line of output that tells you:

Setting env variable DLV_LISTEN_FORWARDER to a valid dlv '--listen' value will cause the dlv debugger to execute this binary and listen as directed.

If you follow those instructions when running the Docker container:

docker run -e DLV_LISTEN_NSMGR=:50000 -p 50000:50000 --rm $(docker build -q --target test .)

-e DLV_LISTEN_NSMGR=:50000 tells docker to set the environment variable DLV_LISTEN_NSMGR to :50000 telling dlv to listen on port 50000.

-p 50000:50000 tells docker to forward port 50000 in the container to port 50000 in the host. From there, you can just connect dlv using your favorite IDE and debug nsmgr.

Debugging the tests and the nsmgr

docker run --rm -p 40000:40000 $(docker build -q --target debug .)

Please note, the tests start the nsmgr, so until you connect to port 40000 with your debugger and walk the tests through to the point of running nsmgr, you will not be able to attach a debugger on port 50000 to the nsmgr.

Build Docker image compatible with integration testing suite:

docker build . -t networkservicemeshci/cmd-nsmgr:master && kind load docker-image networkservicemeshci/cmd-nsmgr:master

cmd-nsmgr's People

Contributors

Stargazers

Watchers

cmd-nsmgr's Issues

nsmgr crashing when deploying the `floating_vl3-basic` example on 3 kind clusters

Hello!

I've tried to deploy the NSM over interdomain vL3 network example on 3 kind clusters, running on 3 separate VMs.

Unfortunately, on the first two clusters (ie: the ones containing pods that should connect to each other), one of the two replicates of nsmgr constantly crashes and is then stuck in the CrashLoopBackOff state. The logs indicate an error about too many files being opened (can not create node poller: too many open files) followed by a segfault. You can view the full logs here.

I've tried to change the system's soft nofile limit, but it didn't fix the problem. By inspecting the other instance of nsmgr, I then found out that the soft limit was already changed inside the pod to match the system's hard limit (which is 1048576... I doubt this limit of file descriptors should be exceeded under normal working conditions).

Do you have any ideas about what may cause this issue?
Thanks!

PS: I'm using MetalLB as a LoadBalancer provider. I gave each instance of MetalLB the 192.168.x.0/24 prefix, where x is the VM number (0, 1 or 2). I then manually updated the VM's routing tables to ensure the packets were correctly routed (I confirmed this setup is working by verifying the Spiffe Federation was successful, and by ensuring the NSE (un)registrations appeared in the registry's logs, on the third cluster)

Request for livenessProbe and readinessProbe functionality

Please implement some functionality that can be used as livenessProbe and readinessProbe in a helm chart, like the HTTP probes for the old-gen NSM version.

NSMgr deployed via k8s daemonset can't access node IP address

Issue

When NSMgr pods are being deployed via k8s daemonset, different pods IP addresses can't be inserted into the `ListenOn' env.

Possible solution

Use additional config files or additional envs. Ex:

- name: POD_IP
  valueFrom:
    fieldRef:
      fieldPath: status.podIP

func updateTCPURL(cfg *config.Config) {
	var tcpURL *url.URL
	for i := range cfg.ListenOn {
		if cfg.ListenOn[i].Scheme == "tcp" {
			tcpURL = &cfg.ListenOn[i]
			break
		}
	}
	if tcpURL == nil {
		return
	}

	var podIP string
	for _, env := range os.Environ() {
		split := strings.Split(env, "=")
		if split[0] == "POD_IP" {
			podIP = split[1]
			break
		}
	}
	if podIP == "" {
		return
	}

	hostPort := strings.Split(tcpURL.Host, ":")
	hostPort[0] = podIP
	tcpURL.Host = strings.Join(hostPort, ":")
}

Connectivity changes during runtime

Hi ! I have a question :

Lets say I have 9 containers running in a cluster which comprise of a Network Service.
Now after some time, I want to add a new container which needs to connect to my other 9 containers running previously.
How does NSM address connectivity during runtime when new containers are included in the network service ?

Thanks in advance !

Update NSMgr to latest SDK

Problem

After merge SDK/#383 bot can not automatically update cmd-nsmgr.

Solution

Update cmd-nsmgr to latest SDK manually

Log level cannot be set

Default log level TRACE cannot be changed.
I used the deployment-k8s 73d981eb2267617dda8fd82d113f362f369ab44a version from today. Deployed the kernel2kernel example, but changed the NSM_LOG_LEVEL to INFO in the nsmgr.yaml.
The result is:

kubectl describe pod -n nsm-system nsmgr-f7mrq
Name: nsmgr-f7mrq
Namespace: nsm-system
Priority: 0
Node: kind-worker2/172.17.0.5
Start Time: Mon, 15 Nov 2021 17:22:10 +0100
Labels: app=nsmgr
controller-revision-hash=76d87cddcb
pod-template-generation=1
spiffe.io/spiffe-id=true
Annotations:
Status: Running
IP: 10.244.2.3
IPs:
IP: 10.244.2.3
Controlled By: DaemonSet/nsmgr
Containers:
nsmgr:
Container ID: containerd://3ed1ff6a9d6573c3461ee758264ba4c273c41481b7e13579179ed5be3cffa3e5
Image: ghcr.io/networkservicemesh/ci/cmd-nsmgr:24b1df1
Image ID: ghcr.io/networkservicemesh/ci/cmd-nsmgr@sha256:711b149e281089b25002d583f6b6cc4e25ec79a4ff0fa9ecdab23e1c69844f80
Port: 5001/TCP
Host Port: 5001/TCP
State: Running
Started: Mon, 15 Nov 2021 17:23:16 +0100
Ready: True
Restart Count: 0
Limits:
cpu: 400m
memory: 100Mi
Requests:
cpu: 200m
memory: 100Mi
Liveness: exec [/bin/grpc-health-probe -spiffe -addr=:5001] delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: exec [/bin/grpc-health-probe -spiffe -addr=:5001] delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
SPIFFE_ENDPOINT_SOCKET: unix:///run/spire/sockets/agent.sock
NSM_NAME: nsmgr-f7mrq (v1:metadata.name)
NSM_LOG_LEVEL: INFO
NSM_REGISTRY_URL: registry:5002
POD_IP: (v1:status.podIP)
NSM_LISTEN_ON: unix:///var/lib/networkservicemesh/nsm.io.sock,tcp://:5001
NODE_NAME: (v1:spec.nodeName)
Mounts:
/run/spire/sockets from spire-agent-socket (ro)
/var/lib/networkservicemesh from nsm-socket (rw)
/var/lib/networkservicemesh/config/ from exclude-prefixes-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from nsmgr-sa-token-hgphx (ro)
exclude-prefixes:
Container ID: containerd://16697d16c73528d80460f8ef0921279a324908b5e70ac00f4f3dcc9c73bfa1d0
Image: ghcr.io/networkservicemesh/ci/cmd-exclude-prefixes-k8s:a363fa9
Image ID: ghcr.io/networkservicemesh/ci/cmd-exclude-prefixes-k8s@sha256:787915c1d68daddcdabd523de6134f24b7f3b8844b0a075a541b7755b4de7f4e
Port:
Host Port:
State: Running
Started: Mon, 15 Nov 2021 17:23:37 +0100
Ready: True
Restart Count: 0
Limits:
cpu: 50m
memory: 10Mi
Requests:
cpu: 50m
memory: 10Mi
Environment:
NSM_LOG_LEVEL: TRACE
Mounts:
/var/lib/networkservicemesh/config/ from exclude-prefixes-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from nsmgr-sa-token-hgphx (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
spire-agent-socket:
Type: HostPath (bare host directory volume)
Path: /run/spire/sockets
HostPathType: Directory
nsm-socket:
Type: HostPath (bare host directory volume)
Path: /var/lib/networkservicemesh
HostPathType: DirectoryOrCreate
exclude-prefixes-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
nsmgr-sa-token-hgphx:
Type: Secret (a volume populated by a Secret)
SecretName: nsmgr-sa-token-hgphx
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message

Normal Scheduled 9m7s default-scheduler Successfully assigned nsm-system/nsmgr-f7mrq to kind-worker2
Normal Pulling 9m5s kubelet Pulling image "ghcr.io/networkservicemesh/ci/cmd-nsmgr:24b1df1"
Normal Pulled 8m2s kubelet Successfully pulled image "ghcr.io/networkservicemesh/ci/cmd-nsmgr:24b1df1" in 1m2.703147324s
Normal Created 8m2s kubelet Created container nsmgr
Normal Started 8m1s kubelet Started container nsmgr
Normal Pulling 8m1s kubelet Pulling image "ghcr.io/networkservicemesh/ci/cmd-exclude-prefixes-k8s:a363fa9"
Normal Pulled 7m53s kubelet Successfully pulled image "ghcr.io/networkservicemesh/ci/cmd-exclude-prefixes-k8s:a363fa9" in 7.864644635s
Normal Created 7m53s kubelet Created container exclude-prefixes
Normal Started 7m40s kubelet Started container exclude-prefixes

And the logs:

kubectl logs -n nsm-system nsmgr-f7mrq nsmgr
Nov 15 16:23:16.840 [INFO] [cmd:[/bin/nsmgr]] Setting env variable DLV_LISTEN_NSMGR to a valid dlv '--listen' value will cause the dlv debugger to execute this binary and listen as directed.
2021/11/15 16:23:16 Initializing logging reporter
Nov 15 16:23:16.841 [TRAC] [cmd:[/bin/nsmgr]] (1) ⎆ nsmgr() span=14e3cf05ba8c6e11:14e3cf05ba8c6e11:0000000000000000:1
Nov 15 16:23:16.842 [INFO] Using configuration: &{nsmgr-f7mrq [{unix /var/lib/networkservicemesh/nsm.io.sock false } {tcp :5001 false }] {registry 5002 false } 10m0s INFO 50ms forwarder}
2021/11/15 16:23:16 Reporting span 14e3cf05ba8c6e11:14e3cf05ba8c6e11:0000000000000000:1
Nov 15 16:23:16.842 [TRAC] [cmd:[/bin/nsmgr]] (2) ⎆ start() span=14e3cf05ba8c6e11:62712eff7b144991:14e3cf05ba8c6e11:1
Nov 15 16:23:16.842 [INFO] Obtaining X509 Certificate Source
This application is configured via the environment. The following environment
variables can be used:

KEY TYPE DEFAULT REQUIRED DESCRIPTION
NSM_NAME String nmgr Name of Network service manager
NSM_LISTEN_ON Comma-separated list of URL unix:///var/lib/networkservicemesh/nsm.io.sock url to listen on. tcp:// one will be used a public to register NSM.
NSM_REGISTRY_URL URL tcp://localhost:5001 A NSE registry url to use
NSM_MAX_TOKEN_LIFETIME Duration 10m maximum lifetime of tokens
NSM_LOG_LEVEL String INFO Log level
NSM_DIAL_TIMEOUT Duration 50ms Timeout for the dial the next endpoint
NSM_FORWARDER_NETWORK_SERVICE_NAME String forwarder the default service name for forwarder discovering
Nov 15 16:23:38.501 [INFO] SVID: "spiffe://example.org/ns/nsm-system/pod/nsmgr-f7mrq"
Nov 15 16:23:38.502 [INFO] NSMGR Listening on: unix:///var/lib/networkservicemesh/nsm.io.sock
Nov 15 16:23:38.503 [INFO] NSMGR Listening on: tcp://:5001
Nov 15 16:23:38.503 [INFO] [cmd:Nsmgr] Startup completed in 21.661379005s
2021/11/15 16:23:42 Reporting span 15c3942e44173525:15c3942e44173525:0000000000000000:1
2021/11/15 16:23:42 ERROR: failed to flush Jaeger spans to server: write udp 127.0.0.1:35195->127.0.0.1:6831: write: connection refused
Nov 15 16:23:43.223 [TRAC] (1) ⎆ sdk/pkg/registry/core/next/nextNetworkServiceEndpointRegistryServer.Register() span=40d8cd5fcdc7ec9e:1adc6d577c962b3d:20972def88bf45f4:1
Nov 15 16:23:43.223 [TRAC] (1.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"p2p":"true"}}},"url":"inode://2097375/30688161"} span=40d8cd5fcdc7ec9e:1adc6d577c962b3d:20972def88bf45f4:1
Nov 15 16:23:43.224 [TRAC] (2) ⎆ sdk/pkg/registry/common/setlogoption/setNSELogOption.Register() span=40d8cd5fcdc7ec9e:28299415a4630400:1adc6d577c962b3d:1
Nov 15 16:23:43.224 [TRAC] (2.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"p2p":"true"}}},"url":"inode://2097375/30688161"} span=40d8cd5fcdc7ec9e:28299415a4630400:1adc6d577c962b3d:1
Nov 15 16:23:43.224 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (3) ⎆ sdk/pkg/registry/common/clientinfo/clientInfo.Register() span=40d8cd5fcdc7ec9e:252206972f4ceeb3:28299415a4630400:1
Nov 15 16:23:43.224 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (3.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"p2p":"true"}}},"url":"inode://2097375/30688161"} span=40d8cd5fcdc7ec9e:252206972f4ceeb3:28299415a4630400:1
Nov 15 16:23:43.224 [WARN] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (3.2) Environment variable POD_NAME is not set. Skipping. span=40d8cd5fcdc7ec9e:252206972f4ceeb3:28299415a4630400:1
Nov 15 16:23:43.224 [WARN] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (3.3) Environment variable CLUSTER_NAME is not set. Skipping. span=40d8cd5fcdc7ec9e:252206972f4ceeb3:28299415a4630400:1
Nov 15 16:23:43.224 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (4) ⎆ sdk/pkg/registry/common/serialize/serializeNSEServer.Register() span=40d8cd5fcdc7ec9e:6f2205cfba1409b5:252206972f4ceeb3:1
Nov 15 16:23:43.224 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (4.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161"} span=40d8cd5fcdc7ec9e:6f2205cfba1409b5:252206972f4ceeb3:1
Nov 15 16:23:43.224 [TRAC] (1) ⎆ sdk/pkg/registry/core/next/nextNetworkServiceEndpointRegistryServer.Find() span=40d8cd5fcdc7ec9e:4094e03d1c7b37e9:1c39a8869b08f9f5:1
Nov 15 16:23:43.224 [TRAC] (1.1) find={"network_service_endpoint":{},"watch":true} span=40d8cd5fcdc7ec9e:4094e03d1c7b37e9:1c39a8869b08f9f5:1
Nov 15 16:23:43.224 [TRAC] (2) ⎆ sdk/pkg/registry/common/setlogoption/setNSELogOption.Find() span=40d8cd5fcdc7ec9e:4b9598c3011b3a10:4094e03d1c7b37e9:1
Nov 15 16:23:43.225 [TRAC] (2.1) find={"network_service_endpoint":{},"watch":true} span=40d8cd5fcdc7ec9e:4b9598c3011b3a10:4094e03d1c7b37e9:1
Nov 15 16:23:43.225 [TRAC] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (3) ⎆ sdk/pkg/registry/common/clientinfo/clientInfo.Find() span=40d8cd5fcdc7ec9e:3b6180cf45241bfa:4b9598c3011b3a10:1
Nov 15 16:23:43.225 [TRAC] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (3.1) find={"network_service_endpoint":{},"watch":true} span=40d8cd5fcdc7ec9e:3b6180cf45241bfa:4b9598c3011b3a10:1
Nov 15 16:23:43.225 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (4) ⎆ sdk/pkg/registry/common/serialize/serializeNSEServer.Find() span=40d8cd5fcdc7ec9e:57a2918fa78f838f:3b6180cf45241bfa:1
Nov 15 16:23:43.225 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (4.1) find={"network_service_endpoint":{},"watch":true} span=40d8cd5fcdc7ec9e:57a2918fa78f838f:3b6180cf45241bfa:1
Nov 15 16:23:43.225 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (5) ⎆ sdk/pkg/registry/common/expire/expireNSEServer.Find() span=40d8cd5fcdc7ec9e:2aa9c46d680ff669:57a2918fa78f838f:1
Nov 15 16:23:43.225 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (5.1) find={"network_service_endpoint":{},"watch":true} span=40d8cd5fcdc7ec9e:2aa9c46d680ff669:57a2918fa78f838f:1
Nov 15 16:23:43.225 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (6) ⎆ sdk/pkg/registry/common/recvfd/recvfdNseServer.Find() span=40d8cd5fcdc7ec9e:6ca33ee34ee2f041:2aa9c46d680ff669:1
Nov 15 16:23:43.225 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (6.1) find={"network_service_endpoint":{},"watch":true} span=40d8cd5fcdc7ec9e:6ca33ee34ee2f041:2aa9c46d680ff669:1
Nov 15 16:23:43.225 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (7) ⎆ sdk/pkg/registry/common/sendfd/sendfdNSEServer.Find() span=40d8cd5fcdc7ec9e:7fa8873d809d299c:6ca33ee34ee2f041:1
Nov 15 16:23:43.225 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (7.1) find={"network_service_endpoint":{},"watch":true} span=40d8cd5fcdc7ec9e:7fa8873d809d299c:6ca33ee34ee2f041:1
Nov 15 16:23:43.225 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (8) ⎆ sdk/pkg/registry/common/memory/memoryNSEServer.Find() span=40d8cd5fcdc7ec9e:1598cc8b1b824306:7fa8873d809d299c:1
Nov 15 16:23:43.225 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (8.1) find={"network_service_endpoint":{},"watch":true} span=40d8cd5fcdc7ec9e:1598cc8b1b824306:7fa8873d809d299c:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (5) ⎆ sdk/pkg/registry/common/expire/expireNSEServer.Register() span=40d8cd5fcdc7ec9e:1560d080202f8646:6f2205cfba1409b5:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (5.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161"} span=40d8cd5fcdc7ec9e:1560d080202f8646:6f2205cfba1409b5:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (6) ⎆ sdk/pkg/registry/common/recvfd/recvfdNseServer.Register() span=40d8cd5fcdc7ec9e:0546db9fda14c316:1560d080202f8646:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (6.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:0546db9fda14c316:1560d080202f8646:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (7) ⎆ sdk/pkg/registry/common/sendfd/sendfdNSEServer.Register() span=40d8cd5fcdc7ec9e:01aa702cc3cfa4d5:0546db9fda14c316:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (7.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"unix:///proc/1/fd/12","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:01aa702cc3cfa4d5:0546db9fda14c316:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (8) ⎆ sdk/pkg/registry/common/memory/memoryNSEServer.Register() span=40d8cd5fcdc7ec9e:2d754a550040f959:01aa702cc3cfa4d5:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (8.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"unix:///proc/1/fd/12","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:2d754a550040f959:01aa702cc3cfa4d5:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (9) ⎆ sdk/pkg/registry/common/localbypass/localBypassNSEServer.Register() span=40d8cd5fcdc7ec9e:759c2489ed119b06:2d754a550040f959:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (9.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"unix:///proc/1/fd/12","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:759c2489ed119b06:2d754a550040f959:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (10) ⎆ sdk/pkg/registry/common/clienturl/clientURLNSEServer.Register() span=40d8cd5fcdc7ec9e:0f9febe94fd21d6a:759c2489ed119b06:1
Nov 15 16:23:43.227 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (10.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"tcp://10.244.2.3:5001","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:0f9febe94fd21d6a:759c2489ed119b06:1
Nov 15 16:23:43.227 [DEBU] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (10.2) passed clientURL: registry:5002 span=40d8cd5fcdc7ec9e:0f9febe94fd21d6a:759c2489ed119b06:1
Nov 15 16:23:43.227 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (11) ⎆ sdk/pkg/registry/common/connect/connectNSEServer.Register() span=40d8cd5fcdc7ec9e:602497796f8f91c1:0f9febe94fd21d6a:1
Nov 15 16:23:43.227 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (11.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"tcp://10.244.2.3:5001","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:602497796f8f91c1:0f9febe94fd21d6a:1
Nov 15 16:23:43.239 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (12) ⎆ sdk/pkg/registry/common/connect/grpcNSEClient.Register() span=40d8cd5fcdc7ec9e:22d5455aae1195ba:602497796f8f91c1:1
Nov 15 16:23:43.239 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (12.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"tcp://10.244.2.3:5001","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:22d5455aae1195ba:602497796f8f91c1:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:779bdc19ca795b19:22d5455aae1195ba:1
Nov 15 16:23:43.270 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (12.2) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"tcp://10.244.2.3:5001","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:22d5455aae1195ba:602497796f8f91c1:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:22d5455aae1195ba:602497796f8f91c1:1
Nov 15 16:23:43.271 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (11.2) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"tcp://10.244.2.3:5001","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:602497796f8f91c1:0f9febe94fd21d6a:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:602497796f8f91c1:0f9febe94fd21d6a:1
Nov 15 16:23:43.271 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (10.3) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"tcp://10.244.2.3:5001","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:0f9febe94fd21d6a:759c2489ed119b06:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:0f9febe94fd21d6a:759c2489ed119b06:1
Nov 15 16:23:43.271 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (9.2) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"unix:///proc/1/fd/12","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:759c2489ed119b06:2d754a550040f959:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:759c2489ed119b06:2d754a550040f959:1
Nov 15 16:23:43.271 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (8.2) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"unix:///proc/1/fd/12","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:2d754a550040f959:01aa702cc3cfa4d5:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:2d754a550040f959:01aa702cc3cfa4d5:1
Nov 15 16:23:43.271 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (7.2) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"unix:///proc/1/fd/12","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:01aa702cc3cfa4d5:0546db9fda14c316:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:01aa702cc3cfa4d5:0546db9fda14c316:1
Nov 15 16:23:43.271 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (6.2) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:0546db9fda14c316:1560d080202f8646:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:0546db9fda14c316:1560d080202f8646:1
Nov 15 16:23:43.271 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (5.2) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:1560d080202f8646:6f2205cfba1409b5:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:1560d080202f8646:6f2205cfba1409b5:1
Nov 15 16:23:43.272 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (4.2) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:6f2205cfba1409b5:252206972f4ceeb3:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:6f2205cfba1409b5:252206972f4ceeb3:1
Nov 15 16:23:43.272 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (3.4) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:252206972f4ceeb3:28299415a4630400:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:252206972f4ceeb3:28299415a4630400:1
Nov 15 16:23:43.272 [TRAC] (2.2) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:28299415a4630400:1adc6d577c962b3d:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:28299415a4630400:1adc6d577c962b3d:1
Nov 15 16:23:43.272 [TRAC] (1.2) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:1adc6d577c962b3d:20972def88bf45f4:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:1adc6d577c962b3d:20972def88bf45f4:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:20972def88bf45f4:0866bd446297583f:1
Nov 15 16:23:43.272 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (9) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=40d8cd5fcdc7ec9e:55abcf219f4997e6:1598cc8b1b824306:1
Nov 15 16:23:43.272 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (9.1) network service endpoint={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"unix:///proc/1/fd/12","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:55abcf219f4997e6:1598cc8b1b824306:1
Nov 15 16:23:43.273 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (8) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=40d8cd5fcdc7ec9e:7e9b8bd174610f0c:7fa8873d809d299c:1
Nov 15 16:23:43.273 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (8.1) network service endpoint={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:7e9b8bd174610f0c:7fa8873d809d299c:1
Nov 15 16:23:43.273 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (7) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=40d8cd5fcdc7ec9e:0c8df920bb958314:6ca33ee34ee2f041:1
Nov 15 16:23:43.273 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (7.1) network service endpoint={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:0c8df920bb958314:6ca33ee34ee2f041:1
Nov 15 16:23:43.273 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (6) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=40d8cd5fcdc7ec9e:0571e3e256abe864:2aa9c46d680ff669:1
Nov 15 16:23:43.273 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (6.1) network service endpoint={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:0571e3e256abe864:2aa9c46d680ff669:1
Nov 15 16:23:43.273 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (5) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=40d8cd5fcdc7ec9e:17840cb0841f5882:57a2918fa78f838f:1
Nov 15 16:23:43.273 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (5.1) network service endpoint={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:17840cb0841f5882:57a2918fa78f838f:1
Nov 15 16:23:43.273 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (4) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=40d8cd5fcdc7ec9e:415b11a2e86f7986:3b6180cf45241bfa:1
Nov 15 16:23:43.273 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (4.1) network service endpoint={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:415b11a2e86f7986:3b6180cf45241bfa:1
Nov 15 16:23:43.273 [TRAC] (3) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=40d8cd5fcdc7ec9e:2987c8f3c7749f24:4b9598c3011b3a10:1
Nov 15 16:23:43.273 [TRAC] (3.1) network service endpoint={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:2987c8f3c7749f24:4b9598c3011b3a10:1
Nov 15 16:23:43.273 [TRAC] (2) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=40d8cd5fcdc7ec9e:636f105c69cd0e47:4094e03d1c7b37e9:1
Nov 15 16:23:43.273 [TRAC] (2.1) network service endpoint={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:636f105c69cd0e47:4094e03d1c7b37e9:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:636f105c69cd0e47:4094e03d1c7b37e9:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:2987c8f3c7749f24:4b9598c3011b3a10:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:415b11a2e86f7986:3b6180cf45241bfa:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:17840cb0841f5882:57a2918fa78f838f:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:0571e3e256abe864:2aa9c46d680ff669:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:0c8df920bb958314:6ca33ee34ee2f041:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:7e9b8bd174610f0c:7fa8873d809d299c:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:55abcf219f4997e6:1598cc8b1b824306:1
2021/11/15 16:23:44 Reporting span 689c23c735a326b5:689c23c735a326b5:0000000000000000:1
2021/11/15 16:23:44 ERROR: failed to flush Jaeger spans to server: write udp 127.0.0.1:35195->127.0.0.1:6831: write: connection refused

NSMgr leaks memory somewhere (probably in `grpcfd`)

Current Behavior

NSMgr leaks memory somewhere (probably in grpcfd) after long time working.

nsmgr-8g8qn.pb.gz

Steps to Reproduce

Patch NSMgr with the following lines in main.go:

import _ "net/http/pprof"

go func() {
	_ = http.ListenAndServe("127.0.0.1:8080", nil)
}()

Run some example for a long time (12+ hours).
Collect heap profile with go tools pprof.

NSM_LISTEN_ON unix socket file permissions

This item is related to using nsm from an nsc app running as non-root.

Currently the unix listenOn socket gets created with file permissions that do not grant write access for others:

Defaulted container "nsmgr" out of: nsmgr, exclude-prefixes
/ # ls -l /var/lib/networkservicemesh/nsm.io.sock
srwxr-xr-x    1 root     root             0 Jun 28 15:01 /var/lib/networkservicemesh/nsm.io.sock

Since the socket gets created in a hostPath volume, nsm users mounting it can't rely for example on fsGroup to change group ownership. To workaround this, either an init container could be applied changing file permission/ownership, or CAP_DAC_OVERRIDE capability could be utilized by the nsc binary to bypass permission checks.

A more user friendly approach could be setting os.ModePerm file permissions in nsmgr when the socket is created, as it would not put additional requirements on the nsc application (to run as non-root).
This behaviour would match how Spire handles their unix listener:
https://github.com/spiffe/spire/blob/07219c7aed188820ebe5ffb67125fdbf69d00286/pkg/agent/endpoints/endpoints_posix.go#L31

What do you think?

nsmgr:

cmd-nsmgr/internal/manager/manager.go

Line 205 in 90a32f8

errChan := grpcutils.ListenAndServe(m.ctx, listenURL, server)

https://github.com/networkservicemesh/sdk/blob/main/pkg/tools/grpcutils/listen_and_serve.go#L66

nsc to connect nsmgr via tcp

Hi,

I noticed that nsmgr normally has a tcp listener specified as well besides the unix socket:
NSM_LISTEN_ON: unix:///var/lib/networkservicemesh/nsm.io.sock,tcp://:5001

I wonder if an NSC could leverage TCP to ask for a specific network service in the cluster?

I gave it a try basically relying on example basic, cmd-nsc and cmd-nse-icmp-responder.
The nsc managed to connect with the collocated nsmgr, but the request failed at the local vpp-forwarder:

Nov 14 16:21:09.637ESC[37m [TRAC] [id:8a75672a-f41b-42cd-9cdd-ea5eef310595] [type:networkService] ESC[0m(1.1)   request={"connection":{"id":"8a75672a-f41b-42cd-9cdd-ea5eef310595","network_service":"my-kernel-svc","context":{"ip_context":{"excluded_prefixes":["10.96.0.0/16","10.244.0.0/16"]}},"labels":{"nodeName":"kind-worker"},"path":{"index":1,"path_segments":[{"name":"nsc","id":"nsc-0","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9kZWZhdWx0L3NhL2RlZmF1bHQiLCJhdWQiOlsic3BpZmZlOi8vZXhhbXBsZS5vcmcvbnMvbnNtL3NhL25zbWdyLXNhIl0sImV4cCI6MTY2ODQ0MzQ2OX0.gCsW1Tm17HISOgUY0mdUHiEhaYPCYVp-Rrf6TASrlUY8voTSAPGj-w4_575tROZItytpExUj5k6mz2H1_lXRyg","expires":{"seconds":1668443469,"nanos":622149106}},{"name":"nsmgr-n4mzr","id":"8a75672a-f41b-42cd-9cdd-ea5eef310595","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20vc2EvbnNtZ3Itc2EiLCJhdWQiOlsic3BpZmZlOi8vZXhhbXBsZS5vcmcvbnMvZGVmYXVsdC9zYS9kZWZhdWx0Il0sImV4cCI6MTY2ODQ0MzQ2OX0.rpT5iD1QMv86LAcg0ySuFYGhpFReMXb3LMgT0Psh4t4vQd6diOdcsq5pLtliIE8CQ-8JmsNe0CUFBBJE2LDKjg","expires":{"seconds":1668443469,"nanos":623643475}}]}},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"inode://4/4026535708","name":"nsm-1"}}]}
...
Nov 14 16:21:09.642ESC[37m [TRAC] [id:7c5abadb-2f68-46c5-bcac-fd2efb386da6] [type:networkService] ESC[0m(9)         ⎆ sdk/pkg/networkservice/common/mechanisms/recvfd/recvFDServer.Request()
Nov 14 16:21:24.619ESC[31m [ERRO] [id:7c5abadb-2f68-46c5-bcac-fd2efb386da6] [type:networkService] ESC[0m(9.1)           timeout in recvfd waiting for inode://4/4026535708: context deadline exceeded
Nov 14 16:21:24.619ESC[37m [TRAC] [id:7c5abadb-2f68-46c5-bcac-fd2efb386da6] [type:networkService] ESC[0m(8.1)          request-response=null

NSM version: 1.6.1
Spire: 1.2.2
cmd-nsc: ghcr.io/networkservicemesh/ci/cmd-nsc:5de2e87
cmd-nse-icmp-responder: ghcr.io/networkservicemesh/ci/cmd-nse-icmp-responder:9e479a6

Reproduction:

start Kind with 2 workers and a controller
deployed nsm (with vpp-forwarder) in k8s namespace nsm.
add a service to nsm k8s namespace with internalTrafficPolicy: Local for nsmgr port 5001 to force NSC to connect the collocated nsmgr:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: nsm-nsmgr-svc
  namespace: nsm
spec:
  selector:
    app: nsmgr
  ports:
    - name: nsm-nsmgr-svc
      protocol: TCP
      port: 5001
      targetPort: 5001
  internalTrafficPolicy: Local
EOF

deploy cmd-nse-icmp-responder on node kind-worker:

cat <<EOF | kubectl apply -f -
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nse-kernel
  labels:
    app: nse-kernel
spec:
  selector:
    matchLabels:
      app: nse-kernel
  template:
    metadata:
      labels:
        app: nse-kernel
        "spiffe.io/spiffe-id": "true"
    spec:
      containers:
        - name: nse
          image: ghcr.io/networkservicemesh/ci/cmd-nse-icmp-responder:9e479a6
          imagePullPolicy: IfNotPresent
          env:
            - name: SPIFFE_ENDPOINT_SOCKET
              value: unix:///run/spire/sockets/agent.sock
            - name: NSM_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: NSM_LOG_LEVEL
              value: TRACE
            - name: NSM_CONNECT_TO
              value: unix:///var/lib/networkservicemesh/nsm.io.sock
            - name: NSM_CIDR_PREFIX
              value: 172.16.1.0/24
            - name: NSM_SERVICE_NAMES
              value: my-kernel-svc
          volumeMounts:
            - name: spire-agent-socket
              mountPath: /run/spire/sockets
              readOnly: true
            - name: nsm-socket
              mountPath: /var/lib/networkservicemesh
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 40Mi
            limits:
              memory: 80Mi
              cpu: 200m
      nodeSelector:
        kubernetes.io/hostname: kind-worker
      volumes:
        - name: spire-agent-socket
          hostPath:
            path: /run/spire/sockets
            type: Directory
        - name: nsm-socket
          hostPath:
            path: /var/lib/networkservicemesh
            type: DirectoryOrCreate
EOF

deploy cmd-nsc on the same node as the NSE requesting service via kernel mechanism:


cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: Pod
metadata:
  name: nsc
  labels:
    app: nsc  
    "spiffe.io/spiffe-id": "true"
spec:
  containers:
  - name: nsc
    image: ghcr.io/networkservicemesh/ci/cmd-nsc:5de2e87
    imagePullPolicy: Always
    env:
      - name: SPIFFE_ENDPOINT_SOCKET
        value: unix:///run/spire/sockets/agent.sock
      - name: NSM_LOG_LEVEL
        value: TRACE
      - name: NSM_NETWORK_SERVICES
        value: kernel://my-kernel-svc/nsm-1
      - name: NSM_NAME
        valueFrom:
          fieldRef:
            fieldPath: metadata.name
      - name: NSM_LIVENESSCHECKENABLED
        value: "false"
      - name: NSM_CONNECT_TO
        value: nsm-nsmgr-svc.nsm:5001
      - name: NSM_LOCALDNSSERVERENABLED
        value: "false"
    volumeMounts:
      - name: spire-agent-socket
        mountPath: /run/spire/sockets
        readOnly: true
      - name: nsm-socket
        mountPath: /var/lib/networkservicemesh
        readOnly: true
    resources:
      requests:
        cpu: 100m
        memory: 40Mi
      limits:
        memory: 80Mi
        cpu: 200m
  volumes:
    - name: spire-agent-socket
      hostPath:
        path: /run/spire/sockets
        type: Directory
    - name: nsm-socket
      hostPath:
        path: /var/lib/networkservicemesh
        type: DirectoryOrCreate
  nodeSelector:
    kubernetes.io/hostname: kind-worker
EOF

What should be the real memory consumed by nsmgr process?

This is a side question which popped up when investigating the issue #675 . It is regarding the 'real' memory consumption of nsmgr container and how this relates to the memory usage of it's single process.
When I checked the memory usage of nsmgr process different values were showed by different tools.

I did not relayed to the 'ps' command output, but checked the 'smaps' in /proc directory (same as pmap command). Another source was the go's profiling (pprof) tool which shows much lower memory used by nsmgr process.

> ps -o pid -o rss -o size -p 2753077
   PID   RSS  SIZE
2753077 28672 58940

Pmap and smaps shown a 29524 kilobytes

The pprof tool gives the following output:

 > go tool pprof -top memprofile-15:20:53-4287106497 | head -4
File: nsmgr
Type: inuse_space
Time: May 7, 2024 at 5:20pm (CEST)
Showing nodes accounting for 2581.36kB, 100% of 2581.36kB total

There are different values reported for RSS by pmap (29524 kilobytes), by 'ps' (28904 kilobytes) and by pprof (2581.36kB).
The only process running on this container is the nsmgr process. The grpc probes are removed on this deployment.
The kubectl top reported an 16 Mi bytes which almost the same as systemct and memory.current file shows:

systemctl status 2753077
● cri-containerd-fec3f5061630291f4345bfffa05cd5dc08e5542b98713a45d1d6de2afedfda66.scope - libcontainer container fec3f5061630291f4345bfffa05cd5dc08e5542b98713a45d1d6de2afedfda66
     Loaded: loaded (/run/systemd/transient/cri-containerd-fec3f5061630291f4345bfffa05cd5dc08e5542b98713a45d1d6de2afedfda66.scope; transient)
  Transient: yes
    Drop-In: /run/systemd/transient/cri-containerd-fec3f5061630291f4345bfffa05cd5dc08e5542b98713a45d1d6de2afedfda66.scope.d
             └─50-DevicePolicy.conf, 50-DeviceAllow.conf, 50-MemoryMax.conf, 50-CPUWeight.conf, 50-CPUQuotaPeriodSec.conf, 50-CPUQuota.conf
     Active: active (running) since Mon 2024-05-06 16:04:23 UTC; 23h ago
         IO: 0B read, 2.5M written
      Tasks: 21 (limit: 11475)
     Memory: 15.4M (max: 200.0M available: 184.2M)
        CPU: 9min 14.275s
     CGroup: /kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/kubelet-kubepods-burstable-pod7dec5ba0_a449_4165_b315_cc5181a4f45b.slice/cri-containerd-fec3f5061630291f4345bfffa05cd5dc08e5542b98713a45d1d6de2afedfda66.scope
             └─2753077 /bin/nsmgr

May 06 16:04:23 kind-worker4 systemd[1]: Started libcontainer container fec3f5061630291f4345bfffa05cd5dc08e5542b98713a45d1d6de2afedfda66.

> cat /sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/kubelet-kubepods-burstable-pod7dec5ba0_a449_4165_b315_cc5181a4f45b.slice/cri-containerd-fec3f5061630291f4345bfffa05cd5dc08e5542b98713a45d1d6de2afedfda66.scope/memory.current
16228352

> cat /sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/kubelet-kubepods-burstable-pod7dec5ba0_a449_4165_b315_cc5181a4f45b.slice/cri-containerd-fec3f5061630291f4345bfffa05cd5dc08e5542b98713a45d1d6de2afedfda66.scope/cgroup.procs
2753077

Later the RSS values showed by ps and pmap command showing much lower memory consumption than cgroup's memory.current and 'kubectl top'. What should cause the difference? What else can count to container memory usage than nsmgr process (CRI, kubelet)?

Memory leaks in nsmgr

Increased memory usage of nsmgr pods can be detected.
On some environment the memory usage reaches it's limit (400M) and leads to OOMKill.

Reproduction:

With the following example I was able to reproduce a considerable memory increase by scaling out and scaling in the NSCs.

https://github.com/ljkiraly/deployments-k8s/blob/1ac366dcb8cbffe52cf5428fe9bb6aa79a9511bd/examples/use-cases/2registry3nseXnsc/README.md

The test ran on kind cluster with 4 nodes and NSM v1.13.0 was deployed.

Sample output of kubectl top command

deployment.apps/nsc-kernel scaled
deployment.apps/nsc-kernel scaled
2024. 04. 23. 13:34:31 CEST
NAME          CPU(cores)   MEMORY(bytes)   
nsmgr-5s65v   102m         32Mi            
NAME          CPU(cores)   MEMORY(bytes)   
nsmgr-fms9q   16m          26Mi            
NAME          CPU(cores)   MEMORY(bytes)   
nsmgr-s7gz8   16m          27Mi            
NAME          CPU(cores)   MEMORY(bytes)   
nsmgr-vtd9q   33m          25Mi 

...

2024. 04. 23. 14:09:02 CEST
NAME          CPU(cores)   MEMORY(bytes)   
nsmgr-5s65v   391m         84Mi            
NAME          CPU(cores)   MEMORY(bytes)   
nsmgr-fms9q   36m          45Mi            
NAME          CPU(cores)   MEMORY(bytes)   
nsmgr-s7gz8   126m         65Mi            
NAME          CPU(cores)   MEMORY(bytes)   
nsmgr-vtd9q   145m         42Mi

NSMgr container restarts due to concurrent map writes

We've discovered that in some cases there are several restarts of the NSMgr container. We could not filter out a single scenario for reproduction, but in most of the cases, it happened when we've installed new Endpoints (NSEs, NSCs).

my-deployment-name-nsmgr-csg65-nsmgr-prev.txt:fatal error: concurrent map writes
my-deployment-name-nsmgr-lwrl8-nsmgr-prev.txt:fatal error: concurrent map writes
my-deployment-name-nsmgr-qzrzs-nsmgr-prev.txt:panic: runtime error: invalid memory address or nil pointer dereference
my-deployment-name-nsmgr-qzrzs-nsmgr-prev.txt-[signal SIGSEGV: segmentation violation code=0x1 addr=0x1b8 pc=0x906677]
NSMgr_logs.zip

The NSMgr version that we used was 5a4b013.

TestNSmgrEndpointCallback has unexpected Errors in output.

Test TestNSmgrEndpointCallback passes but had exceptions inside output.

Partial Output:

Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<-------------------------------- request={"id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":2,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.xpc7XwOlw50t_55VvJIYvGAhRqYOTe8BaxP6WvYcXhJffLZhlX4ECKDnEV0xacB1RlJwBs4Z8ZSzpT1gQqT6ow","expires":{"seconds":1597216137,"nanos":968375000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}} span={}
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<-- response={"id":"c716a3b0-2314-44bc-956a-3a470d65f0de","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":1,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.xpc7XwOlw50t_55VvJIYvGAhRqYOTe8BaxP6WvYcXhJffLZhlX4ECKDnEV0xacB1RlJwBs4Z8ZSzpT1gQqT6ow","expires":{"seconds":1597216137,"nanos":968375000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}} span={}
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==--------------> github.com/networkservicemesh/sdk/pkg/networkservice/common/roundrobin/selectEndpointServer.Request() span:{}
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==----------------------------------------> github.com/networkservicemesh/api/pkg/api/networkservice/networkServiceClient.Request() span:{}
Aug 12 14:08:57.999 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<---------------- request={"id":"429896e9-1d39-40bb-8e9e-b2214d62ec9b","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":3,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.DW6ctJxX9jFvbXReJYMO83US1w4G1jifRABbOIAVeBpYSHVwoSRWUqXqQsiM7MFs7RuSnFubFupQx2OcQtLeKg","expires":{"seconds":1597216137,"nanos":981506000}},{"name":"test-nsm2#nsmgr","id":"429896e9-1d39-40bb-8e9e-b2214d62ec9b","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.F79QbYHyDmfnAwWAumOWbKGF9kUGxq-PNuPb9r_dgWYx6q3M88xuzywTHE5AnSfBIp9QE938-SFL_36gB8IAGQ","expires":{"seconds":1597216137,"nanos":991024000}}]}} span={}
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==------------------> github.com/networkservicemesh/sdk/pkg/networkservice/common/updatetoken/updateTokenServer.Close() span:{}
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<---------------------------------------- request={"connection":{"id":"c716a3b0-2314-44bc-956a-3a470d65f0de","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":1,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM4LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZImr67dNnSUcplbWB1H0RGYT9HSB85rIXavULDrrfInezvbPfYGYswl_LTRavs4-uOW7zzQf_UgUaCaWVwRHUw","expires":{"seconds":1597216138,"nanos":27000}}]}}} span={}
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==--------------------------> github.com/networkservicemesh/sdk/pkg/networkservice/common/authorize/authorizeServer.Request() span:{}
Aug 12 14:08:57.996 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==--> github.com/networkservicemesh/sdk/pkg/networkservice/common/authorize/authorizeServer.Request() span:{}
Aug 12 14:08:58.004 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<-------------------------- request={"connection":{"id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":2,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.xpc7XwOlw50t_55VvJIYvGAhRqYOTe8BaxP6WvYcXhJffLZhlX4ECKDnEV0xacB1RlJwBs4Z8ZSzpT1gQqT6ow","expires":{"seconds":1597216137,"nanos":968375000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}}} span={}
Aug 12 14:08:58.004 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==----------------------------> github.com/networkservicemesh/sdk/pkg/networkservice/common/updatepath/updatePathServer.Request() span:{}
Aug 12 14:08:57.996 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<---------------------------------------- request={"connection":{"id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":3,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.eDPkbSx2QSWDVRWxEl93ENo7ro1pDOFiHE9nPYG9pMVrI0EBixVCHMZ12fz9klM02gbWPELPCX3KSsm6WrnLiQ","expires":{"seconds":1597216137,"nanos":996220000}}]}}} span={}
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==----------------------------------> github.com/networkservicemesh/sdk/pkg/networkservice/common/refresh/refreshClient.Close() span:{}
Aug 12 14:08:58.004 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<---------------------------------- request={"id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":2,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.xpc7XwOlw50t_55VvJIYvGAhRqYOTe8BaxP6WvYcXhJffLZhlX4ECKDnEV0xacB1RlJwBs4Z8ZSzpT1gQqT6ow","expires":{"seconds":1597216137,"nanos":968375000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}} span={}
Aug 12 14:08:58.004 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==------------------------------------> github.com/networkservicemesh/sdk/pkg/networkservice/utils/inject/injectpeer/injectPeerClient.Close() span:{}
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<-------------- request={"connection":{"id":"429896e9-1d39-40bb-8e9e-b2214d62ec9b","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":3,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.DW6ctJxX9jFvbXReJYMO83US1w4G1jifRABbOIAVeBpYSHVwoSRWUqXqQsiM7MFs7RuSnFubFupQx2OcQtLeKg","expires":{"seconds":1597216137,"nanos":981506000}},{"name":"test-nsm2#nsmgr","id":"429896e9-1d39-40bb-8e9e-b2214d62ec9b","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.F79QbYHyDmfnAwWAumOWbKGF9kUGxq-PNuPb9r_dgWYx6q3M88xuzywTHE5AnSfBIp9QE938-SFL_36gB8IAGQ","expires":{"seconds":1597216137,"nanos":991024000}}]}}} span={}
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<------------------ request={"id":"429896e9-1d39-40bb-8e9e-b2214d62ec9b","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":3,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.DW6ctJxX9jFvbXReJYMO83US1w4G1jifRABbOIAVeBpYSHVwoSRWUqXqQsiM7MFs7RuSnFubFupQx2OcQtLeKg","expires":{"seconds":1597216137,"nanos":981506000}},{"name":"test-nsm2#nsmgr","id":"429896e9-1d39-40bb-8e9e-b2214d62ec9b","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.F79QbYHyDmfnAwWAumOWbKGF9kUGxq-PNuPb9r_dgWYx6q3M88xuzywTHE5AnSfBIp9QE938-SFL_36gB8IAGQ","expires":{"seconds":1597216137,"nanos":991024000}}]}} span={}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<------------------------------------ request={"id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":2,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.xpc7XwOlw50t_55VvJIYvGAhRqYOTe8BaxP6WvYcXhJffLZhlX4ECKDnEV0xacB1RlJwBs4Z8ZSzpT1gQqT6ow","expires":{"seconds":1597216137,"nanos":968375000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}} span={}
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<-------------- response={"id":"c716a3b0-2314-44bc-956a-3a470d65f0de","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":1,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.xpc7XwOlw50t_55VvJIYvGAhRqYOTe8BaxP6WvYcXhJffLZhlX4ECKDnEV0xacB1RlJwBs4Z8ZSzpT1gQqT6ow","expires":{"seconds":1597216137,"nanos":968375000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}} span={}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==--------------------------------------> github.com/networkservicemesh/sdk/pkg/networkservice/common/updatetoken/updateTokenClient.Close() span:{}
Aug 12 14:08:58.004 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<---------------------------- request={"connection":{"id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":2,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.xpc7XwOlw50t_55VvJIYvGAhRqYOTe8BaxP6WvYcXhJffLZhlX4ECKDnEV0xacB1RlJwBs4Z8ZSzpT1gQqT6ow","expires":{"seconds":1597216137,"nanos":968375000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}}} span={}
Aug 12 14:08:58.004 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<-- request={"connection":{"id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":3,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.HgLsl8YBK2g3bqbkYmYlX9AoLeI0MH4Lee3_CDS_VPoE1i1YAoY0w9fy-YLnKSvCHSKTMDz0LOwUlm3s1NSt4A","expires":{"seconds":1597216137,"nanos":996072000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}}} span={}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==------------------------------> github.com/networkservicemesh/sdk/pkg/networkservice/common/monitor/monitorServer.Request() span:{}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==--> github.com/networkservicemesh/sdk/pkg/networkservice/common/authorize/authorizeServer.Request() span:{}
Aug 12 14:08:58.005 [ERRO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<-------------------------------- error=rpc error: code = Canceled desc = context canceled
Error returned from github.com/networkservicemesh/api/pkg/api/networkservice/networkServiceClient.Request
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
	/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
	/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/updatetoken.(*updateTokenClient).Request
	/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/updatetoken/client.go:50
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
	/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:54
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
	/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/utils/inject/injectpeer.(*injectPeerClient).Request
	/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/utils/inject/injectpeer/client.go:61
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
	/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:54
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
	/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/common/refresh.(*refreshClient).Request
	/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/refresh/client.go:55
github.com/networkservicemesh/sdk/pkg/networkservice/common/refresh.(*refreshClient).Request.func1.1
	/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/refresh/client.go:101
runtime.goexit
	/usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/asm_amd64.s:1373 span={}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==----> github.com/networkservicemesh/sdk/pkg/networkservice/common/updatepath/updatePathServer.Request() span:{}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<---- request={"connection":{"id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":3,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.HgLsl8YBK2g3bqbkYmYlX9AoLeI0MH4Lee3_CDS_VPoE1i1YAoY0w9fy-YLnKSvCHSKTMDz0LOwUlm3s1NSt4A","expires":{"seconds":1597216137,"nanos":996072000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}}} span={}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<-- request={"connection":{"id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":3,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.eDPkbSx2QSWDVRWxEl93ENo7ro1pDOFiHE9nPYG9pMVrI0EBixVCHMZ12fz9klM02gbWPELPCX3KSsm6WrnLiQ","expires":{"seconds":1597216137,"nanos":996220000}}]}}} span={}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==----> github.com/networkservicemesh/sdk/pkg/networkservice/common/updatepath/updatePathServer.Request() span:{}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==----------------> github.com/networkservicemesh/sdk/pkg/networkservice/common/localbypass/localBypassServer.Request() span:{}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==------> github.com/networkservicemesh/sdk/pkg/networkservice/common/monitor/monitorServer.Request() span:{}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<---------------- request={"connection":{"id":"429896e9-1d39-40bb-8e9e-b2214d62ec9b","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":3,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.DW6ctJxX9jFvbXReJYMO83US1w4G1jifRABbOIAVeBpYSHVwoSRWUqXqQsiM7MFs7RuSnFubFupQx2OcQtLeKg","expires":{"seconds":1597216137,"nanos":981506000}},{"name":"test-nsm2#nsmgr","id":"429896e9-1d39-40bb-8e9e-b2214d62ec9b","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM4LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.kvIDwgZHc1ie7_3FDIkwf5WAuwjBYuqdCE_neZCtDLKfLtt7ZUcY98R2eufRUiWVy-8c0xytDBIfY2w6VwOXTw","expires":{"seconds":1597216138,"nanos":5064000}}]}}} span={}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<------------------------------ request={"connection":{"id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":2,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.xpc7XwOlw50t_55VvJIYvGAhRqYOTe8BaxP6WvYcXhJffLZhlX4ECKDnEV0xacB1RlJwBs4Z8ZSzpT1gQqT6ow","expires":{"seconds":1597216137,"nanos":968375000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}}} span={}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==------------------> github.com/networkservicemesh/sdk/pkg/networkservice/common/excludedprefixes/excludedPrefixesServer.Request() span:{}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<------ request={"connection":{"id":"08de6ccb-b256-42a0-be47-45cfec799b10","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":4,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.HgLsl8YBK2g3bqbkYmYlX9AoLeI0MH4Lee3_CDS_VPoE1i1YAoY0w9fy-YLnKSvCHSKTMDz0LOwUlm3s1NSt4A","expires":{"seconds":1597216137,"nanos":996072000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}}} span={}
Aug 12 14:08:58.009 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==--------> github.com/networkservicemesh/sdk/pkg/networkservice/common/timeout/timeoutServer.Request() span:{}
Aug 12 14:08:58.009 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<-------- request={"connection":{"id":"08de6ccb-b256-42a0-be47-45cfec799b10","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":4,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.HgLsl8YBK2g3bqbkYmYlX9AoLeI0MH4Lee3_CDS_VPoE1i1YAoY0w9fy-YLnKSvCHSKTMDz0LOwUlm3s1NSt4A","expires":{"seconds":1597216137,"nanos":996072000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}}} span={}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<---- request={"connection":{"id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":3,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.eDPkbSx2QSWDVRWxEl93ENo7ro1pDOFiHE9nPYG9pMVrI0EBixVCHMZ12fz9klM02gbWPELPCX3KSsm6WrnLiQ","expires":{"seconds":1597216137,"nanos":996220000}}]}}} span={}
Aug 12 14:08:58.005 [ERRO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<------------------------------ error=Error returned from github.com/networkservicemesh/api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Canceled desc = context canceled span={}
Aug 12 14:08:58.009 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==----------> github.com/networkservicemesh/sdk/pkg/networkservice/common/updatetoken/updateTokenServer.Request() span:{}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<-------------------------------------- request={"id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":2,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.xpc7XwOlw50t_55VvJIYvGAhRqYOTe8BaxP6WvYcXhJffLZhlX4ECKDnEV0xacB1RlJwBs4Z8ZSzpT1gQqT6ow","expires":{"seconds":1597216137,"nanos":968375000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}} span={}
Aug 12 14:08:58.009 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<---------- request={"connection":{"id":"08de6ccb-b256-42a0-be47-45cfec799b10","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":4,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.HgLsl8YBK2g3bqbkYmYlX9AoLeI0MH4Lee3_CDS_VPoE1i1YAoY0w9fy-YLnKSvCHSKTMDz0LOwUlm3s1NSt4A","expires":{"seconds":1597216137,"nanos":996072000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}}} span={}
Aug 12 14:08:58.009 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==------> github.com/networkservicemesh/sdk/pkg/networkservice/common/monitor/monitorServer.Request() span:{}
Aug 12 14:08:58.009 [ERRO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<---------------------------- error=Error returned from github.com/networkservicemesh/api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Canceled desc = context canceled span={}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<------------ response={"id":"c716a3b0-2314-44bc-956a-3a470d65f0de","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.xpc7XwOlw50t_55VvJIYvGAhRqYOTe8BaxP6WvYcXhJffLZhlX4ECKDnEV0xacB1RlJwBs4Z8ZSzpT1gQqT6ow","expires":{"seconds":1597216137,"nanos":968375000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}} span={}
Aug 12 14:08:58.009 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<------ request={"connection":{"id":"f783e183-698f-4c36-b50a-7d3f699b9995","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":4,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.eDPkbSx2QSWDVRWxEl93ENo7ro1pDOFiHE9nPYG9pMVrI0EBixVCHMZ12fz9klM02gbWPELPCX3KSsm6WrnLiQ","expires":{"seconds":1597216137,"nanos":996220000}},{"name":"nse","id":"f783e183-698f-4c36-b50a-7d3f699b9995"}]}}} span={}
Aug 12 14:08:58.009 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==----------------------------------------> github.com/networkservicemesh/api/pkg/api/networkservice/networkServiceClient.Close() span:{}
Aug 12 14:08:58.009 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==----------> github.com/networkservicemesh/sdk/pkg/networkservice/common/authorize/authorizeServer.Close() span:{}
Aug 12 14:08:58.009 [ERRO] [operation:github.com/networkservicemesh/sdk/pkg/networkservice/common/refresh/refreshClient.Request] [span:{}] Error while attempting to refresh connection 87c7ea94-b34a-4d61-8f44-cb17b2588c68: rpc error: code = Canceled desc = context canceled
Error returned from github.com/networkservicemesh/api/pkg/api/networkservice/networkServiceClient.Request
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
	/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
	/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/updatetoken.(*updateTokenClient).Request
	/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/updatetoken/client.go:50
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
	/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:54
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
	/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/utils/inject/injectpeer.(*injectPeerClient).Request
	/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/utils/inject/injectpeer/client.go:61
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
	/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:54
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
	/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/common/refresh.(*refreshClient).Request
	/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/refresh/client.go:55
github.com/networkservicemesh/sdk/pkg/networkservice/common/refresh.(*refreshClient).Request.func1.1
	/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/refresh/client.go:101
runtime.goexit
	/usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/asm_amd64.s:1373

NSMgr adds NSE in registry with wrong URL

Issue

NSMgr adds NSE in registry with ListenOn[0] URL.

cmd-nsmgr/internal/manager/manager.go

Lines 105 to 108 in 196a358

    
           nsmMgr := &registry.NetworkServiceEndpoint{ 
        
           	Name: configuration.Name + "#nsmgr", 
        
           	Url:  configuration.ListenOn[0].String(), 
        
           }

If it is a unix socket URL, NSE is unaccessible from the remote NSMgr.

Existing 'solution' of this problem is not working because it sets URL after it has been used for the NSMgr creation:

cmd-nsmgr/internal/manager/manager.go

Lines 201 to 203 in 196a358

    
           if listenURL.Scheme == tcpSchema { 
        
           	nsmMgr.Url = listenURL.String() 
        
           }

Possible solution

Use URL with tcp scheme instead of ListenOn[0] URL.

What if there are multiple public URLs for different cases?

NSMgr can use multiple public URL, example:

unix socket - for communication between pods on single node
LAN IP address - for communication in domain
WAN IP address - for interdomain communication

Currently registry doesn't provide API to register NSE with both (2, 3) URLs, so NSE will be either unaccessible from the LAN, or from the Internet.

Another way can be to implement the registry in-place for simplicity sake.

Add configuration for turning profiling on/off

cmd-nsmgr
cmd-forwarder-vpp
cmd-registry-k8s

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

	nsmMgr := &registry.NetworkServiceEndpoint{
	Name: configuration.Name + "#nsmgr",
	Url: configuration.ListenOn[0].String(),
	}

	if listenURL.Scheme == tcpSchema {
	nsmMgr.Url = listenURL.String()
	}

networkservicemesh / cmd-nsmgr Goto Github PK

cmd-nsmgr's Introduction

Intro

Build

Build nsmgr binary locally

Build Docker container

Usage

Environment config

Testing

Testing Docker container

Debugging

Debugging the tests

Debugging the nsmgr

Debugging the tests and the nsmgr

Build Docker image compatible with integration testing suite:

cmd-nsmgr's People

Contributors

Stargazers

Watchers

Forkers

cmd-nsmgr's Issues

Issue

Possible solution

Problem

Solution

And the logs:

Current Behavior

Steps to Reproduce

Issue

Possible solution

What if there are multiple public URLs for different cases?

Recommend Projects

Recommend Topics

Recommend Org