networkservicemesh / cmd-nsmgr Goto Github PK
View Code? Open in Web Editor NEWNetwork Service Manager
License: Apache License 2.0
Network Service Manager
License: Apache License 2.0
This item is related to using nsm from an nsc app running as non-root.
Currently the unix listenOn socket gets created with file permissions that do not grant write access for others:
Defaulted container "nsmgr" out of: nsmgr, exclude-prefixes
/ # ls -l /var/lib/networkservicemesh/nsm.io.sock
srwxr-xr-x 1 root root 0 Jun 28 15:01 /var/lib/networkservicemesh/nsm.io.sock
Since the socket gets created in a hostPath volume, nsm users mounting it can't rely for example on fsGroup to change group ownership. To workaround this, either an init container could be applied changing file permission/ownership, or CAP_DAC_OVERRIDE capability could be utilized by the nsc binary to bypass permission checks.
A more user friendly approach could be setting os.ModePerm file permissions in nsmgr when the socket is created, as it would not put additional requirements on the nsc application (to run as non-root).
This behaviour would match how Spire handles their unix listener:
https://github.com/spiffe/spire/blob/07219c7aed188820ebe5ffb67125fdbf69d00286/pkg/agent/endpoints/endpoints_posix.go#L31
What do you think?
nsmgr:
cmd-nsmgr/internal/manager/manager.go
Line 205 in 90a32f8
NSMgr adds NSE in registry with ListenOn[0]
URL.
cmd-nsmgr/internal/manager/manager.go
Lines 105 to 108 in 196a358
Existing 'solution' of this problem is not working because it sets URL after it has been used for the NSMgr creation:
cmd-nsmgr/internal/manager/manager.go
Lines 201 to 203 in 196a358
Use URL with tcp
scheme instead of ListenOn[0]
URL.
NSMgr can use multiple public URL, example:
Currently registry doesn't provide API to register NSE with both (2, 3) URLs, so NSE will be either unaccessible from the LAN, or from the Internet.
Hello!
I've tried to deploy the NSM over interdomain vL3 network example on 3 kind clusters, running on 3 separate VMs.
Unfortunately, on the first two clusters (ie: the ones containing pods that should connect to each other), one of the two replicates of nsmgr
constantly crashes and is then stuck in the CrashLoopBackOff
state. The logs indicate an error about too many files being opened (can not create node poller: too many open files
) followed by a segfault. You can view the full logs here.
I've tried to change the system's soft nofile
limit, but it didn't fix the problem. By inspecting the other instance of nsmgr
, I then found out that the soft limit was already changed inside the pod to match the system's hard limit (which is 1048576
... I doubt this limit of file descriptors should be exceeded under normal working conditions).
Do you have any ideas about what may cause this issue?
Thanks!
PS: I'm using MetalLB as a LoadBalancer
provider. I gave each instance of MetalLB the 192.168.x.0/24
prefix, where x
is the VM number (0, 1 or 2). I then manually updated the VM's routing tables to ensure the packets were correctly routed (I confirmed this setup is working by verifying the Spiffe Federation was successful, and by ensuring the NSE (un)registrations appeared in the registry's logs, on the third cluster)
When NSMgr pods are being deployed via k8s daemonset, different pods IP addresses can't be inserted into the `ListenOn' env.
Use additional config files or additional envs. Ex:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
func updateTCPURL(cfg *config.Config) {
var tcpURL *url.URL
for i := range cfg.ListenOn {
if cfg.ListenOn[i].Scheme == "tcp" {
tcpURL = &cfg.ListenOn[i]
break
}
}
if tcpURL == nil {
return
}
var podIP string
for _, env := range os.Environ() {
split := strings.Split(env, "=")
if split[0] == "POD_IP" {
podIP = split[1]
break
}
}
if podIP == "" {
return
}
hostPort := strings.Split(tcpURL.Host, ":")
hostPort[0] = podIP
tcpURL.Host = strings.Join(hostPort, ":")
}
Increased memory usage of nsmgr pods can be detected.
On some environment the memory usage reaches it's limit (400M) and leads to OOMKill.
Reproduction:
With the following example I was able to reproduce a considerable memory increase by scaling out and scaling in the NSCs.
The test ran on kind cluster with 4 nodes and NSM v1.13.0 was deployed.
Sample output of kubectl top
command
deployment.apps/nsc-kernel scaled
deployment.apps/nsc-kernel scaled
2024. 04. 23. 13:34:31 CEST
NAME CPU(cores) MEMORY(bytes)
nsmgr-5s65v 102m 32Mi
NAME CPU(cores) MEMORY(bytes)
nsmgr-fms9q 16m 26Mi
NAME CPU(cores) MEMORY(bytes)
nsmgr-s7gz8 16m 27Mi
NAME CPU(cores) MEMORY(bytes)
nsmgr-vtd9q 33m 25Mi
...
2024. 04. 23. 14:09:02 CEST
NAME CPU(cores) MEMORY(bytes)
nsmgr-5s65v 391m 84Mi
NAME CPU(cores) MEMORY(bytes)
nsmgr-fms9q 36m 45Mi
NAME CPU(cores) MEMORY(bytes)
nsmgr-s7gz8 126m 65Mi
NAME CPU(cores) MEMORY(bytes)
nsmgr-vtd9q 145m 42Mi
Test TestNSmgrEndpointCallback passes but had exceptions inside output.
Partial Output:
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<-------------------------------- request={"id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":2,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.xpc7XwOlw50t_55VvJIYvGAhRqYOTe8BaxP6WvYcXhJffLZhlX4ECKDnEV0xacB1RlJwBs4Z8ZSzpT1gQqT6ow","expires":{"seconds":1597216137,"nanos":968375000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}} span={}
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<-- response={"id":"c716a3b0-2314-44bc-956a-3a470d65f0de","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":1,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.xpc7XwOlw50t_55VvJIYvGAhRqYOTe8BaxP6WvYcXhJffLZhlX4ECKDnEV0xacB1RlJwBs4Z8ZSzpT1gQqT6ow","expires":{"seconds":1597216137,"nanos":968375000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}} span={}
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==--------------> github.com/networkservicemesh/sdk/pkg/networkservice/common/roundrobin/selectEndpointServer.Request() span:{}
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==----------------------------------------> github.com/networkservicemesh/api/pkg/api/networkservice/networkServiceClient.Request() span:{}
Aug 12 14:08:57.999 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<---------------- request={"id":"429896e9-1d39-40bb-8e9e-b2214d62ec9b","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":3,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.DW6ctJxX9jFvbXReJYMO83US1w4G1jifRABbOIAVeBpYSHVwoSRWUqXqQsiM7MFs7RuSnFubFupQx2OcQtLeKg","expires":{"seconds":1597216137,"nanos":981506000}},{"name":"test-nsm2#nsmgr","id":"429896e9-1d39-40bb-8e9e-b2214d62ec9b","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.F79QbYHyDmfnAwWAumOWbKGF9kUGxq-PNuPb9r_dgWYx6q3M88xuzywTHE5AnSfBIp9QE938-SFL_36gB8IAGQ","expires":{"seconds":1597216137,"nanos":991024000}}]}} span={}
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==------------------> github.com/networkservicemesh/sdk/pkg/networkservice/common/updatetoken/updateTokenServer.Close() span:{}
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<---------------------------------------- request={"connection":{"id":"c716a3b0-2314-44bc-956a-3a470d65f0de","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":1,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM4LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZImr67dNnSUcplbWB1H0RGYT9HSB85rIXavULDrrfInezvbPfYGYswl_LTRavs4-uOW7zzQf_UgUaCaWVwRHUw","expires":{"seconds":1597216138,"nanos":27000}}]}}} span={}
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==--------------------------> github.com/networkservicemesh/sdk/pkg/networkservice/common/authorize/authorizeServer.Request() span:{}
Aug 12 14:08:57.996 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==--> github.com/networkservicemesh/sdk/pkg/networkservice/common/authorize/authorizeServer.Request() span:{}
Aug 12 14:08:58.004 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<-------------------------- request={"connection":{"id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":2,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.xpc7XwOlw50t_55VvJIYvGAhRqYOTe8BaxP6WvYcXhJffLZhlX4ECKDnEV0xacB1RlJwBs4Z8ZSzpT1gQqT6ow","expires":{"seconds":1597216137,"nanos":968375000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}}} span={}
Aug 12 14:08:58.004 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==----------------------------> github.com/networkservicemesh/sdk/pkg/networkservice/common/updatepath/updatePathServer.Request() span:{}
Aug 12 14:08:57.996 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<---------------------------------------- request={"connection":{"id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":3,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.eDPkbSx2QSWDVRWxEl93ENo7ro1pDOFiHE9nPYG9pMVrI0EBixVCHMZ12fz9klM02gbWPELPCX3KSsm6WrnLiQ","expires":{"seconds":1597216137,"nanos":996220000}}]}}} span={}
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==----------------------------------> github.com/networkservicemesh/sdk/pkg/networkservice/common/refresh/refreshClient.Close() span:{}
Aug 12 14:08:58.004 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<---------------------------------- request={"id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":2,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.xpc7XwOlw50t_55VvJIYvGAhRqYOTe8BaxP6WvYcXhJffLZhlX4ECKDnEV0xacB1RlJwBs4Z8ZSzpT1gQqT6ow","expires":{"seconds":1597216137,"nanos":968375000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}} span={}
Aug 12 14:08:58.004 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==------------------------------------> github.com/networkservicemesh/sdk/pkg/networkservice/utils/inject/injectpeer/injectPeerClient.Close() span:{}
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<-------------- request={"connection":{"id":"429896e9-1d39-40bb-8e9e-b2214d62ec9b","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":3,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.DW6ctJxX9jFvbXReJYMO83US1w4G1jifRABbOIAVeBpYSHVwoSRWUqXqQsiM7MFs7RuSnFubFupQx2OcQtLeKg","expires":{"seconds":1597216137,"nanos":981506000}},{"name":"test-nsm2#nsmgr","id":"429896e9-1d39-40bb-8e9e-b2214d62ec9b","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.F79QbYHyDmfnAwWAumOWbKGF9kUGxq-PNuPb9r_dgWYx6q3M88xuzywTHE5AnSfBIp9QE938-SFL_36gB8IAGQ","expires":{"seconds":1597216137,"nanos":991024000}}]}}} span={}
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<------------------ request={"id":"429896e9-1d39-40bb-8e9e-b2214d62ec9b","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":3,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.DW6ctJxX9jFvbXReJYMO83US1w4G1jifRABbOIAVeBpYSHVwoSRWUqXqQsiM7MFs7RuSnFubFupQx2OcQtLeKg","expires":{"seconds":1597216137,"nanos":981506000}},{"name":"test-nsm2#nsmgr","id":"429896e9-1d39-40bb-8e9e-b2214d62ec9b","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.F79QbYHyDmfnAwWAumOWbKGF9kUGxq-PNuPb9r_dgWYx6q3M88xuzywTHE5AnSfBIp9QE938-SFL_36gB8IAGQ","expires":{"seconds":1597216137,"nanos":991024000}}]}} span={}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<------------------------------------ request={"id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":2,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.xpc7XwOlw50t_55VvJIYvGAhRqYOTe8BaxP6WvYcXhJffLZhlX4ECKDnEV0xacB1RlJwBs4Z8ZSzpT1gQqT6ow","expires":{"seconds":1597216137,"nanos":968375000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}} span={}
Aug 12 14:08:58.000 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<-------------- response={"id":"c716a3b0-2314-44bc-956a-3a470d65f0de","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":1,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.xpc7XwOlw50t_55VvJIYvGAhRqYOTe8BaxP6WvYcXhJffLZhlX4ECKDnEV0xacB1RlJwBs4Z8ZSzpT1gQqT6ow","expires":{"seconds":1597216137,"nanos":968375000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}} span={}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==--------------------------------------> github.com/networkservicemesh/sdk/pkg/networkservice/common/updatetoken/updateTokenClient.Close() span:{}
Aug 12 14:08:58.004 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<---------------------------- request={"connection":{"id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":2,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.xpc7XwOlw50t_55VvJIYvGAhRqYOTe8BaxP6WvYcXhJffLZhlX4ECKDnEV0xacB1RlJwBs4Z8ZSzpT1gQqT6ow","expires":{"seconds":1597216137,"nanos":968375000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}}} span={}
Aug 12 14:08:58.004 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<-- request={"connection":{"id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":3,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.HgLsl8YBK2g3bqbkYmYlX9AoLeI0MH4Lee3_CDS_VPoE1i1YAoY0w9fy-YLnKSvCHSKTMDz0LOwUlm3s1NSt4A","expires":{"seconds":1597216137,"nanos":996072000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}}} span={}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==------------------------------> github.com/networkservicemesh/sdk/pkg/networkservice/common/monitor/monitorServer.Request() span:{}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==--> github.com/networkservicemesh/sdk/pkg/networkservice/common/authorize/authorizeServer.Request() span:{}
Aug 12 14:08:58.005 [ERRO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<-------------------------------- error=rpc error: code = Canceled desc = context canceled
Error returned from github.com/networkservicemesh/api/pkg/api/networkservice/networkServiceClient.Request
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/updatetoken.(*updateTokenClient).Request
/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/updatetoken/client.go:50
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:54
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/utils/inject/injectpeer.(*injectPeerClient).Request
/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/utils/inject/injectpeer/client.go:61
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:54
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/common/refresh.(*refreshClient).Request
/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/refresh/client.go:55
github.com/networkservicemesh/sdk/pkg/networkservice/common/refresh.(*refreshClient).Request.func1.1
/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/refresh/client.go:101
runtime.goexit
/usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/asm_amd64.s:1373 span={}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==----> github.com/networkservicemesh/sdk/pkg/networkservice/common/updatepath/updatePathServer.Request() span:{}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<---- request={"connection":{"id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":3,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.HgLsl8YBK2g3bqbkYmYlX9AoLeI0MH4Lee3_CDS_VPoE1i1YAoY0w9fy-YLnKSvCHSKTMDz0LOwUlm3s1NSt4A","expires":{"seconds":1597216137,"nanos":996072000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}}} span={}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<-- request={"connection":{"id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":3,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.eDPkbSx2QSWDVRWxEl93ENo7ro1pDOFiHE9nPYG9pMVrI0EBixVCHMZ12fz9klM02gbWPELPCX3KSsm6WrnLiQ","expires":{"seconds":1597216137,"nanos":996220000}}]}}} span={}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==----> github.com/networkservicemesh/sdk/pkg/networkservice/common/updatepath/updatePathServer.Request() span:{}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==----------------> github.com/networkservicemesh/sdk/pkg/networkservice/common/localbypass/localBypassServer.Request() span:{}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==------> github.com/networkservicemesh/sdk/pkg/networkservice/common/monitor/monitorServer.Request() span:{}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<---------------- request={"connection":{"id":"429896e9-1d39-40bb-8e9e-b2214d62ec9b","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":3,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.DW6ctJxX9jFvbXReJYMO83US1w4G1jifRABbOIAVeBpYSHVwoSRWUqXqQsiM7MFs7RuSnFubFupQx2OcQtLeKg","expires":{"seconds":1597216137,"nanos":981506000}},{"name":"test-nsm2#nsmgr","id":"429896e9-1d39-40bb-8e9e-b2214d62ec9b","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM4LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.kvIDwgZHc1ie7_3FDIkwf5WAuwjBYuqdCE_neZCtDLKfLtt7ZUcY98R2eufRUiWVy-8c0xytDBIfY2w6VwOXTw","expires":{"seconds":1597216138,"nanos":5064000}}]}}} span={}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<------------------------------ request={"connection":{"id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":2,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.xpc7XwOlw50t_55VvJIYvGAhRqYOTe8BaxP6WvYcXhJffLZhlX4ECKDnEV0xacB1RlJwBs4Z8ZSzpT1gQqT6ow","expires":{"seconds":1597216137,"nanos":968375000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}}} span={}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==------------------> github.com/networkservicemesh/sdk/pkg/networkservice/common/excludedprefixes/excludedPrefixesServer.Request() span:{}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<------ request={"connection":{"id":"08de6ccb-b256-42a0-be47-45cfec799b10","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":4,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.HgLsl8YBK2g3bqbkYmYlX9AoLeI0MH4Lee3_CDS_VPoE1i1YAoY0w9fy-YLnKSvCHSKTMDz0LOwUlm3s1NSt4A","expires":{"seconds":1597216137,"nanos":996072000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}}} span={}
Aug 12 14:08:58.009 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==--------> github.com/networkservicemesh/sdk/pkg/networkservice/common/timeout/timeoutServer.Request() span:{}
Aug 12 14:08:58.009 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<-------- request={"connection":{"id":"08de6ccb-b256-42a0-be47-45cfec799b10","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":4,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.HgLsl8YBK2g3bqbkYmYlX9AoLeI0MH4Lee3_CDS_VPoE1i1YAoY0w9fy-YLnKSvCHSKTMDz0LOwUlm3s1NSt4A","expires":{"seconds":1597216137,"nanos":996072000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}}} span={}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<---- request={"connection":{"id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":3,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.eDPkbSx2QSWDVRWxEl93ENo7ro1pDOFiHE9nPYG9pMVrI0EBixVCHMZ12fz9klM02gbWPELPCX3KSsm6WrnLiQ","expires":{"seconds":1597216137,"nanos":996220000}}]}}} span={}
Aug 12 14:08:58.005 [ERRO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<------------------------------ error=Error returned from github.com/networkservicemesh/api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Canceled desc = context canceled span={}
Aug 12 14:08:58.009 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==----------> github.com/networkservicemesh/sdk/pkg/networkservice/common/updatetoken/updateTokenServer.Request() span:{}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<-------------------------------------- request={"id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":2,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.xpc7XwOlw50t_55VvJIYvGAhRqYOTe8BaxP6WvYcXhJffLZhlX4ECKDnEV0xacB1RlJwBs4Z8ZSzpT1gQqT6ow","expires":{"seconds":1597216137,"nanos":968375000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}} span={}
Aug 12 14:08:58.009 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<---------- request={"connection":{"id":"08de6ccb-b256-42a0-be47-45cfec799b10","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":4,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.HgLsl8YBK2g3bqbkYmYlX9AoLeI0MH4Lee3_CDS_VPoE1i1YAoY0w9fy-YLnKSvCHSKTMDz0LOwUlm3s1NSt4A","expires":{"seconds":1597216137,"nanos":996072000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}}} span={}
Aug 12 14:08:58.009 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==------> github.com/networkservicemesh/sdk/pkg/networkservice/common/monitor/monitorServer.Request() span:{}
Aug 12 14:08:58.009 [ERRO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<---------------------------- error=Error returned from github.com/networkservicemesh/api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Canceled desc = context canceled span={}
Aug 12 14:08:58.005 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<------------ response={"id":"c716a3b0-2314-44bc-956a-3a470d65f0de","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.xpc7XwOlw50t_55VvJIYvGAhRqYOTe8BaxP6WvYcXhJffLZhlX4ECKDnEV0xacB1RlJwBs4Z8ZSzpT1gQqT6ow","expires":{"seconds":1597216137,"nanos":968375000}},{"name":"nse","id":"08de6ccb-b256-42a0-be47-45cfec799b10","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.IjVNz7S2cof7dPO1L2_zzJAE_pXJvo0d-gc4PHbX4CX3sgauuyNK7JWs7RHbRaMPsfCUcZtNOOEEo--9aFDyHg","expires":{"seconds":1597216137,"nanos":959794000}}]}} span={}
Aug 12 14:08:58.009 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] >><<------ request={"connection":{"id":"f783e183-698f-4c36-b50a-7d3f699b9995","network_service":"my-service","context":{"ip_context":{},"extra_context":{"perform":"ok"}},"path":{"index":4,"path_segments":[{"name":"nsc-1","id":"1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTcyMTYxMzcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL3Rlc3QifQ.yLf_Y0S9-plrZ2d03T72isjmtp8_Tqon08v3vw1HlZM1IfFAmyJvM-VXpDRFkhCdlBS7qfNB1SXEPFcMNIN3kg","expires":{"seconds":1597216137,"nanos":947167000}},{"name":"test-nsm2#nsmgr","id":"c716a3b0-2314-44bc-956a-3a470d65f0de","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.ZAzeu8zhW8bTSCtpZ59z6s09m1Vr8T1zzsFO3FOPmjCGEXd0A9xeYx5THjHIZBXMKkk1UdOWNhTW3O2A3qhvvA","expires":{"seconds":1597216137,"nanos":950984000}},{"name":"cross-nse","id":"87c7ea94-b34a-4d61-8f44-cb17b2588c68","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.fuFiAkaPfDQmTeTOLMgT4yL_l_ZSK0SOTZejfkkMJ081W82z1kBeQ5WoWX142E7vnqQYR4Vp9nXYUv1idBSC9w","expires":{"seconds":1597216137,"nanos":954403000}},{"name":"test-nsm2#nsmgr","id":"7ce1fddb-e8db-4153-affb-929fef4fb8b7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0IiwiZXhwIjoxNTk3MjE2MTM3LCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy90ZXN0In0.eDPkbSx2QSWDVRWxEl93ENo7ro1pDOFiHE9nPYG9pMVrI0EBixVCHMZ12fz9klM02gbWPELPCX3KSsm6WrnLiQ","expires":{"seconds":1597216137,"nanos":996220000}},{"name":"nse","id":"f783e183-698f-4c36-b50a-7d3f699b9995"}]}}} span={}
Aug 12 14:08:58.009 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==----------------------------------------> github.com/networkservicemesh/api/pkg/api/networkservice/networkServiceClient.Close() span:{}
Aug 12 14:08:58.009 [INFO] [cmd:/var/folders/jj/xh7y86d53s59zdp9l9f279d80000gn/T/go-build709694856/b414/test.test] ==----------> github.com/networkservicemesh/sdk/pkg/networkservice/common/authorize/authorizeServer.Close() span:{}
Aug 12 14:08:58.009 [ERRO] [operation:github.com/networkservicemesh/sdk/pkg/networkservice/common/refresh/refreshClient.Request] [span:{}] Error while attempting to refresh connection 87c7ea94-b34a-4d61-8f44-cb17b2588c68: rpc error: code = Canceled desc = context canceled
Error returned from github.com/networkservicemesh/api/pkg/api/networkservice/networkServiceClient.Request
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/updatetoken.(*updateTokenClient).Request
/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/updatetoken/client.go:50
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:54
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/utils/inject/injectpeer.(*injectPeerClient).Request
/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/utils/inject/injectpeer/client.go:61
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:54
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/common/refresh.(*refreshClient).Request
/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/refresh/client.go:55
github.com/networkservicemesh/sdk/pkg/networkservice/common/refresh.(*refreshClient).Request.func1.1
/Users/haiodo/Develop/git/cisco/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/refresh/client.go:101
runtime.goexit
/usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/asm_amd64.s:1373
Default log level TRACE cannot be changed.
I used the deployment-k8s 73d981eb2267617dda8fd82d113f362f369ab44a version from today. Deployed the kernel2kernel example, but changed the NSM_LOG_LEVEL to INFO in the nsmgr.yaml.
The result is:
kubectl describe pod -n nsm-system nsmgr-f7mrq
Name: nsmgr-f7mrq
Namespace: nsm-system
Priority: 0
Node: kind-worker2/172.17.0.5
Start Time: Mon, 15 Nov 2021 17:22:10 +0100
Labels: app=nsmgr
controller-revision-hash=76d87cddcb
pod-template-generation=1
spiffe.io/spiffe-id=true
Annotations:
Status: Running
IP: 10.244.2.3
IPs:
IP: 10.244.2.3
Controlled By: DaemonSet/nsmgr
Containers:
nsmgr:
Container ID: containerd://3ed1ff6a9d6573c3461ee758264ba4c273c41481b7e13579179ed5be3cffa3e5
Image: ghcr.io/networkservicemesh/ci/cmd-nsmgr:24b1df1
Image ID: ghcr.io/networkservicemesh/ci/cmd-nsmgr@sha256:711b149e281089b25002d583f6b6cc4e25ec79a4ff0fa9ecdab23e1c69844f80
Port: 5001/TCP
Host Port: 5001/TCP
State: Running
Started: Mon, 15 Nov 2021 17:23:16 +0100
Ready: True
Restart Count: 0
Limits:
cpu: 400m
memory: 100Mi
Requests:
cpu: 200m
memory: 100Mi
Liveness: exec [/bin/grpc-health-probe -spiffe -addr=:5001] delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: exec [/bin/grpc-health-probe -spiffe -addr=:5001] delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
SPIFFE_ENDPOINT_SOCKET: unix:///run/spire/sockets/agent.sock
NSM_NAME: nsmgr-f7mrq (v1:metadata.name)
NSM_LOG_LEVEL: INFO
NSM_REGISTRY_URL: registry:5002
POD_IP: (v1:status.podIP)
NSM_LISTEN_ON: unix:///var/lib/networkservicemesh/nsm.io.sock,tcp://:5001
NODE_NAME: (v1:spec.nodeName)
Mounts:
/run/spire/sockets from spire-agent-socket (ro)
/var/lib/networkservicemesh from nsm-socket (rw)
/var/lib/networkservicemesh/config/ from exclude-prefixes-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from nsmgr-sa-token-hgphx (ro)
exclude-prefixes:
Container ID: containerd://16697d16c73528d80460f8ef0921279a324908b5e70ac00f4f3dcc9c73bfa1d0
Image: ghcr.io/networkservicemesh/ci/cmd-exclude-prefixes-k8s:a363fa9
Image ID: ghcr.io/networkservicemesh/ci/cmd-exclude-prefixes-k8s@sha256:787915c1d68daddcdabd523de6134f24b7f3b8844b0a075a541b7755b4de7f4e
Port:
Host Port:
State: Running
Started: Mon, 15 Nov 2021 17:23:37 +0100
Ready: True
Restart Count: 0
Limits:
cpu: 50m
memory: 10Mi
Requests:
cpu: 50m
memory: 10Mi
Environment:
NSM_LOG_LEVEL: TRACE
Mounts:
/var/lib/networkservicemesh/config/ from exclude-prefixes-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from nsmgr-sa-token-hgphx (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
spire-agent-socket:
Type: HostPath (bare host directory volume)
Path: /run/spire/sockets
HostPathType: Directory
nsm-socket:
Type: HostPath (bare host directory volume)
Path: /var/lib/networkservicemesh
HostPathType: DirectoryOrCreate
exclude-prefixes-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
nsmgr-sa-token-hgphx:
Type: Secret (a volume populated by a Secret)
SecretName: nsmgr-sa-token-hgphx
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
Normal Scheduled 9m7s default-scheduler Successfully assigned nsm-system/nsmgr-f7mrq to kind-worker2
Normal Pulling 9m5s kubelet Pulling image "ghcr.io/networkservicemesh/ci/cmd-nsmgr:24b1df1"
Normal Pulled 8m2s kubelet Successfully pulled image "ghcr.io/networkservicemesh/ci/cmd-nsmgr:24b1df1" in 1m2.703147324s
Normal Created 8m2s kubelet Created container nsmgr
Normal Started 8m1s kubelet Started container nsmgr
Normal Pulling 8m1s kubelet Pulling image "ghcr.io/networkservicemesh/ci/cmd-exclude-prefixes-k8s:a363fa9"
Normal Pulled 7m53s kubelet Successfully pulled image "ghcr.io/networkservicemesh/ci/cmd-exclude-prefixes-k8s:a363fa9" in 7.864644635s
Normal Created 7m53s kubelet Created container exclude-prefixes
Normal Started 7m40s kubelet Started container exclude-prefixes
kubectl logs -n nsm-system nsmgr-f7mrq nsmgr
Nov 15 16:23:16.840 [INFO] [cmd:[/bin/nsmgr]] Setting env variable DLV_LISTEN_NSMGR to a valid dlv '--listen' value will cause the dlv debugger to execute this binary and listen as directed.
2021/11/15 16:23:16 Initializing logging reporter
Nov 15 16:23:16.841 [TRAC] [cmd:[/bin/nsmgr]] (1) ⎆ nsmgr() span=14e3cf05ba8c6e11:14e3cf05ba8c6e11:0000000000000000:1
Nov 15 16:23:16.842 [INFO] Using configuration: &{nsmgr-f7mrq [{unix /var/lib/networkservicemesh/nsm.io.sock false } {tcp :5001 false }] {registry 5002 false } 10m0s INFO 50ms forwarder}
2021/11/15 16:23:16 Reporting span 14e3cf05ba8c6e11:14e3cf05ba8c6e11:0000000000000000:1
Nov 15 16:23:16.842 [TRAC] [cmd:[/bin/nsmgr]] (2) ⎆ start() span=14e3cf05ba8c6e11:62712eff7b144991:14e3cf05ba8c6e11:1
Nov 15 16:23:16.842 [INFO] Obtaining X509 Certificate Source
This application is configured via the environment. The following environment
variables can be used:
KEY TYPE DEFAULT REQUIRED DESCRIPTION
NSM_NAME String nmgr Name of Network service manager
NSM_LISTEN_ON Comma-separated list of URL unix:///var/lib/networkservicemesh/nsm.io.sock url to listen on. tcp:// one will be used a public to register NSM.
NSM_REGISTRY_URL URL tcp://localhost:5001 A NSE registry url to use
NSM_MAX_TOKEN_LIFETIME Duration 10m maximum lifetime of tokens
NSM_LOG_LEVEL String INFO Log level
NSM_DIAL_TIMEOUT Duration 50ms Timeout for the dial the next endpoint
NSM_FORWARDER_NETWORK_SERVICE_NAME String forwarder the default service name for forwarder discovering
Nov 15 16:23:38.501 [INFO] SVID: "spiffe://example.org/ns/nsm-system/pod/nsmgr-f7mrq"
Nov 15 16:23:38.502 [INFO] NSMGR Listening on: unix:///var/lib/networkservicemesh/nsm.io.sock
Nov 15 16:23:38.503 [INFO] NSMGR Listening on: tcp://:5001
Nov 15 16:23:38.503 [INFO] [cmd:Nsmgr] Startup completed in 21.661379005s
2021/11/15 16:23:42 Reporting span 15c3942e44173525:15c3942e44173525:0000000000000000:1
2021/11/15 16:23:42 ERROR: failed to flush Jaeger spans to server: write udp 127.0.0.1:35195->127.0.0.1:6831: write: connection refused
Nov 15 16:23:43.223 [TRAC] (1) ⎆ sdk/pkg/registry/core/next/nextNetworkServiceEndpointRegistryServer.Register() span=40d8cd5fcdc7ec9e:1adc6d577c962b3d:20972def88bf45f4:1
Nov 15 16:23:43.223 [TRAC] (1.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"p2p":"true"}}},"url":"inode://2097375/30688161"} span=40d8cd5fcdc7ec9e:1adc6d577c962b3d:20972def88bf45f4:1
Nov 15 16:23:43.224 [TRAC] (2) ⎆ sdk/pkg/registry/common/setlogoption/setNSELogOption.Register() span=40d8cd5fcdc7ec9e:28299415a4630400:1adc6d577c962b3d:1
Nov 15 16:23:43.224 [TRAC] (2.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"p2p":"true"}}},"url":"inode://2097375/30688161"} span=40d8cd5fcdc7ec9e:28299415a4630400:1adc6d577c962b3d:1
Nov 15 16:23:43.224 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (3) ⎆ sdk/pkg/registry/common/clientinfo/clientInfo.Register() span=40d8cd5fcdc7ec9e:252206972f4ceeb3:28299415a4630400:1
Nov 15 16:23:43.224 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (3.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"p2p":"true"}}},"url":"inode://2097375/30688161"} span=40d8cd5fcdc7ec9e:252206972f4ceeb3:28299415a4630400:1
Nov 15 16:23:43.224 [WARN] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (3.2) Environment variable POD_NAME is not set. Skipping. span=40d8cd5fcdc7ec9e:252206972f4ceeb3:28299415a4630400:1
Nov 15 16:23:43.224 [WARN] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (3.3) Environment variable CLUSTER_NAME is not set. Skipping. span=40d8cd5fcdc7ec9e:252206972f4ceeb3:28299415a4630400:1
Nov 15 16:23:43.224 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (4) ⎆ sdk/pkg/registry/common/serialize/serializeNSEServer.Register() span=40d8cd5fcdc7ec9e:6f2205cfba1409b5:252206972f4ceeb3:1
Nov 15 16:23:43.224 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (4.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161"} span=40d8cd5fcdc7ec9e:6f2205cfba1409b5:252206972f4ceeb3:1
Nov 15 16:23:43.224 [TRAC] (1) ⎆ sdk/pkg/registry/core/next/nextNetworkServiceEndpointRegistryServer.Find() span=40d8cd5fcdc7ec9e:4094e03d1c7b37e9:1c39a8869b08f9f5:1
Nov 15 16:23:43.224 [TRAC] (1.1) find={"network_service_endpoint":{},"watch":true} span=40d8cd5fcdc7ec9e:4094e03d1c7b37e9:1c39a8869b08f9f5:1
Nov 15 16:23:43.224 [TRAC] (2) ⎆ sdk/pkg/registry/common/setlogoption/setNSELogOption.Find() span=40d8cd5fcdc7ec9e:4b9598c3011b3a10:4094e03d1c7b37e9:1
Nov 15 16:23:43.225 [TRAC] (2.1) find={"network_service_endpoint":{},"watch":true} span=40d8cd5fcdc7ec9e:4b9598c3011b3a10:4094e03d1c7b37e9:1
Nov 15 16:23:43.225 [TRAC] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (3) ⎆ sdk/pkg/registry/common/clientinfo/clientInfo.Find() span=40d8cd5fcdc7ec9e:3b6180cf45241bfa:4b9598c3011b3a10:1
Nov 15 16:23:43.225 [TRAC] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (3.1) find={"network_service_endpoint":{},"watch":true} span=40d8cd5fcdc7ec9e:3b6180cf45241bfa:4b9598c3011b3a10:1
Nov 15 16:23:43.225 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (4) ⎆ sdk/pkg/registry/common/serialize/serializeNSEServer.Find() span=40d8cd5fcdc7ec9e:57a2918fa78f838f:3b6180cf45241bfa:1
Nov 15 16:23:43.225 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (4.1) find={"network_service_endpoint":{},"watch":true} span=40d8cd5fcdc7ec9e:57a2918fa78f838f:3b6180cf45241bfa:1
Nov 15 16:23:43.225 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (5) ⎆ sdk/pkg/registry/common/expire/expireNSEServer.Find() span=40d8cd5fcdc7ec9e:2aa9c46d680ff669:57a2918fa78f838f:1
Nov 15 16:23:43.225 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (5.1) find={"network_service_endpoint":{},"watch":true} span=40d8cd5fcdc7ec9e:2aa9c46d680ff669:57a2918fa78f838f:1
Nov 15 16:23:43.225 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (6) ⎆ sdk/pkg/registry/common/recvfd/recvfdNseServer.Find() span=40d8cd5fcdc7ec9e:6ca33ee34ee2f041:2aa9c46d680ff669:1
Nov 15 16:23:43.225 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (6.1) find={"network_service_endpoint":{},"watch":true} span=40d8cd5fcdc7ec9e:6ca33ee34ee2f041:2aa9c46d680ff669:1
Nov 15 16:23:43.225 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (7) ⎆ sdk/pkg/registry/common/sendfd/sendfdNSEServer.Find() span=40d8cd5fcdc7ec9e:7fa8873d809d299c:6ca33ee34ee2f041:1
Nov 15 16:23:43.225 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (7.1) find={"network_service_endpoint":{},"watch":true} span=40d8cd5fcdc7ec9e:7fa8873d809d299c:6ca33ee34ee2f041:1
Nov 15 16:23:43.225 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (8) ⎆ sdk/pkg/registry/common/memory/memoryNSEServer.Find() span=40d8cd5fcdc7ec9e:1598cc8b1b824306:7fa8873d809d299c:1
Nov 15 16:23:43.225 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (8.1) find={"network_service_endpoint":{},"watch":true} span=40d8cd5fcdc7ec9e:1598cc8b1b824306:7fa8873d809d299c:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (5) ⎆ sdk/pkg/registry/common/expire/expireNSEServer.Register() span=40d8cd5fcdc7ec9e:1560d080202f8646:6f2205cfba1409b5:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (5.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161"} span=40d8cd5fcdc7ec9e:1560d080202f8646:6f2205cfba1409b5:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (6) ⎆ sdk/pkg/registry/common/recvfd/recvfdNseServer.Register() span=40d8cd5fcdc7ec9e:0546db9fda14c316:1560d080202f8646:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (6.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:0546db9fda14c316:1560d080202f8646:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (7) ⎆ sdk/pkg/registry/common/sendfd/sendfdNSEServer.Register() span=40d8cd5fcdc7ec9e:01aa702cc3cfa4d5:0546db9fda14c316:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (7.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"unix:///proc/1/fd/12","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:01aa702cc3cfa4d5:0546db9fda14c316:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (8) ⎆ sdk/pkg/registry/common/memory/memoryNSEServer.Register() span=40d8cd5fcdc7ec9e:2d754a550040f959:01aa702cc3cfa4d5:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (8.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"unix:///proc/1/fd/12","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:2d754a550040f959:01aa702cc3cfa4d5:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (9) ⎆ sdk/pkg/registry/common/localbypass/localBypassNSEServer.Register() span=40d8cd5fcdc7ec9e:759c2489ed119b06:2d754a550040f959:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (9.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"unix:///proc/1/fd/12","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:759c2489ed119b06:2d754a550040f959:1
Nov 15 16:23:43.226 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (10) ⎆ sdk/pkg/registry/common/clienturl/clientURLNSEServer.Register() span=40d8cd5fcdc7ec9e:0f9febe94fd21d6a:759c2489ed119b06:1
Nov 15 16:23:43.227 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (10.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"tcp://10.244.2.3:5001","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:0f9febe94fd21d6a:759c2489ed119b06:1
Nov 15 16:23:43.227 [DEBU] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (10.2) passed clientURL: registry:5002 span=40d8cd5fcdc7ec9e:0f9febe94fd21d6a:759c2489ed119b06:1
Nov 15 16:23:43.227 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (11) ⎆ sdk/pkg/registry/common/connect/connectNSEServer.Register() span=40d8cd5fcdc7ec9e:602497796f8f91c1:0f9febe94fd21d6a:1
Nov 15 16:23:43.227 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (11.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"tcp://10.244.2.3:5001","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:602497796f8f91c1:0f9febe94fd21d6a:1
Nov 15 16:23:43.239 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (12) ⎆ sdk/pkg/registry/common/connect/grpcNSEClient.Register() span=40d8cd5fcdc7ec9e:22d5455aae1195ba:602497796f8f91c1:1
Nov 15 16:23:43.239 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (12.1) register={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"tcp://10.244.2.3:5001","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:22d5455aae1195ba:602497796f8f91c1:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:779bdc19ca795b19:22d5455aae1195ba:1
Nov 15 16:23:43.270 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (12.2) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"tcp://10.244.2.3:5001","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:22d5455aae1195ba:602497796f8f91c1:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:22d5455aae1195ba:602497796f8f91c1:1
Nov 15 16:23:43.271 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (11.2) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"tcp://10.244.2.3:5001","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:602497796f8f91c1:0f9febe94fd21d6a:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:602497796f8f91c1:0f9febe94fd21d6a:1
Nov 15 16:23:43.271 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (10.3) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"tcp://10.244.2.3:5001","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:0f9febe94fd21d6a:759c2489ed119b06:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:0f9febe94fd21d6a:759c2489ed119b06:1
Nov 15 16:23:43.271 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (9.2) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"unix:///proc/1/fd/12","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:759c2489ed119b06:2d754a550040f959:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:759c2489ed119b06:2d754a550040f959:1
Nov 15 16:23:43.271 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (8.2) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"unix:///proc/1/fd/12","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:2d754a550040f959:01aa702cc3cfa4d5:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:2d754a550040f959:01aa702cc3cfa4d5:1
Nov 15 16:23:43.271 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (7.2) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"unix:///proc/1/fd/12","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:01aa702cc3cfa4d5:0546db9fda14c316:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:01aa702cc3cfa4d5:0546db9fda14c316:1
Nov 15 16:23:43.271 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (6.2) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:0546db9fda14c316:1560d080202f8646:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:0546db9fda14c316:1560d080202f8646:1
Nov 15 16:23:43.271 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (5.2) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:1560d080202f8646:6f2205cfba1409b5:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:1560d080202f8646:6f2205cfba1409b5:1
Nov 15 16:23:43.272 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (4.2) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:6f2205cfba1409b5:252206972f4ceeb3:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:6f2205cfba1409b5:252206972f4ceeb3:1
Nov 15 16:23:43.272 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (3.4) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:252206972f4ceeb3:28299415a4630400:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:252206972f4ceeb3:28299415a4630400:1
Nov 15 16:23:43.272 [TRAC] (2.2) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:28299415a4630400:1adc6d577c962b3d:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:28299415a4630400:1adc6d577c962b3d:1
Nov 15 16:23:43.272 [TRAC] (1.2) register-response={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:1adc6d577c962b3d:20972def88bf45f4:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:1adc6d577c962b3d:20972def88bf45f4:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:20972def88bf45f4:0866bd446297583f:1
Nov 15 16:23:43.272 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (9) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=40d8cd5fcdc7ec9e:55abcf219f4997e6:1598cc8b1b824306:1
Nov 15 16:23:43.272 [TRAC] [id:] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (9.1) network service endpoint={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"unix:///proc/1/fd/12","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:55abcf219f4997e6:1598cc8b1b824306:1
Nov 15 16:23:43.273 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (8) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=40d8cd5fcdc7ec9e:7e9b8bd174610f0c:7fa8873d809d299c:1
Nov 15 16:23:43.273 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (8.1) network service endpoint={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:7e9b8bd174610f0c:7fa8873d809d299c:1
Nov 15 16:23:43.273 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (7) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=40d8cd5fcdc7ec9e:0c8df920bb958314:6ca33ee34ee2f041:1
Nov 15 16:23:43.273 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (7.1) network service endpoint={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:0c8df920bb958314:6ca33ee34ee2f041:1
Nov 15 16:23:43.273 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (6) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=40d8cd5fcdc7ec9e:0571e3e256abe864:2aa9c46d680ff669:1
Nov 15 16:23:43.273 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (6.1) network service endpoint={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:0571e3e256abe864:2aa9c46d680ff669:1
Nov 15 16:23:43.273 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (5) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=40d8cd5fcdc7ec9e:17840cb0841f5882:57a2918fa78f838f:1
Nov 15 16:23:43.273 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (5.1) network service endpoint={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:17840cb0841f5882:57a2918fa78f838f:1
Nov 15 16:23:43.273 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (4) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=40d8cd5fcdc7ec9e:415b11a2e86f7986:3b6180cf45241bfa:1
Nov 15 16:23:43.273 [TRAC] [id:forwarder-vpp-kz8fs] [name:NetworkServiceRegistryServer.nsmgr-f7mrq] [type:NetworkServiceEndpointRegistry] (4.1) network service endpoint={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:415b11a2e86f7986:3b6180cf45241bfa:1
Nov 15 16:23:43.273 [TRAC] (3) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=40d8cd5fcdc7ec9e:2987c8f3c7749f24:4b9598c3011b3a10:1
Nov 15 16:23:43.273 [TRAC] (3.1) network service endpoint={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:2987c8f3c7749f24:4b9598c3011b3a10:1
Nov 15 16:23:43.273 [TRAC] (2) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=40d8cd5fcdc7ec9e:636f105c69cd0e47:4094e03d1c7b37e9:1
Nov 15 16:23:43.273 [TRAC] (2.1) network service endpoint={"name":"forwarder-vpp-kz8fs","network_service_names":["forwarder"],"network_service_labels":{"forwarder":{"labels":{"nodeName":"kind-worker2","p2p":"true"}}},"url":"inode://2097375/30688161","expiration_time":{"seconds":1636993483,"nanos":226141648}} span=40d8cd5fcdc7ec9e:636f105c69cd0e47:4094e03d1c7b37e9:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:636f105c69cd0e47:4094e03d1c7b37e9:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:2987c8f3c7749f24:4b9598c3011b3a10:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:415b11a2e86f7986:3b6180cf45241bfa:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:17840cb0841f5882:57a2918fa78f838f:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:0571e3e256abe864:2aa9c46d680ff669:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:0c8df920bb958314:6ca33ee34ee2f041:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:7e9b8bd174610f0c:7fa8873d809d299c:1
2021/11/15 16:23:43 Reporting span 40d8cd5fcdc7ec9e:55abcf219f4997e6:1598cc8b1b824306:1
2021/11/15 16:23:44 Reporting span 689c23c735a326b5:689c23c735a326b5:0000000000000000:1
2021/11/15 16:23:44 ERROR: failed to flush Jaeger spans to server: write udp 127.0.0.1:35195->127.0.0.1:6831: write: connection refused
NSMgr must provide API for registering NSEs.
https://github.com/networkservicemesh/api/blob/master/pkg/api/registry/registry.proto
Current implementation (networkservicemesh/networkservicemesh repo) forwards all requests to a dedicated server running in k8s-nsmd container (via unix socket).
Another way can be to implement the registry in-place for simplicity sake.
After merge SDK/#383 bot can not automatically update cmd-nsmgr.
Update cmd-nsmgr to latest SDK manually
Hi,
I noticed that nsmgr normally has a tcp listener specified as well besides the unix socket:
NSM_LISTEN_ON: unix:///var/lib/networkservicemesh/nsm.io.sock,tcp://:5001
I wonder if an NSC could leverage TCP to ask for a specific network service in the cluster?
I gave it a try basically relying on example basic, cmd-nsc and cmd-nse-icmp-responder.
The nsc managed to connect with the collocated nsmgr, but the request failed at the local vpp-forwarder:
Nov 14 16:21:09.637ESC[37m [TRAC] [id:8a75672a-f41b-42cd-9cdd-ea5eef310595] [type:networkService] ESC[0m(1.1) request={"connection":{"id":"8a75672a-f41b-42cd-9cdd-ea5eef310595","network_service":"my-kernel-svc","context":{"ip_context":{"excluded_prefixes":["10.96.0.0/16","10.244.0.0/16"]}},"labels":{"nodeName":"kind-worker"},"path":{"index":1,"path_segments":[{"name":"nsc","id":"nsc-0","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9kZWZhdWx0L3NhL2RlZmF1bHQiLCJhdWQiOlsic3BpZmZlOi8vZXhhbXBsZS5vcmcvbnMvbnNtL3NhL25zbWdyLXNhIl0sImV4cCI6MTY2ODQ0MzQ2OX0.gCsW1Tm17HISOgUY0mdUHiEhaYPCYVp-Rrf6TASrlUY8voTSAPGj-w4_575tROZItytpExUj5k6mz2H1_lXRyg","expires":{"seconds":1668443469,"nanos":622149106}},{"name":"nsmgr-n4mzr","id":"8a75672a-f41b-42cd-9cdd-ea5eef310595","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20vc2EvbnNtZ3Itc2EiLCJhdWQiOlsic3BpZmZlOi8vZXhhbXBsZS5vcmcvbnMvZGVmYXVsdC9zYS9kZWZhdWx0Il0sImV4cCI6MTY2ODQ0MzQ2OX0.rpT5iD1QMv86LAcg0ySuFYGhpFReMXb3LMgT0Psh4t4vQd6diOdcsq5pLtliIE8CQ-8JmsNe0CUFBBJE2LDKjg","expires":{"seconds":1668443469,"nanos":623643475}}]}},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"inode://4/4026535708","name":"nsm-1"}}]}
...
Nov 14 16:21:09.642ESC[37m [TRAC] [id:7c5abadb-2f68-46c5-bcac-fd2efb386da6] [type:networkService] ESC[0m(9) ⎆ sdk/pkg/networkservice/common/mechanisms/recvfd/recvFDServer.Request()
Nov 14 16:21:24.619ESC[31m [ERRO] [id:7c5abadb-2f68-46c5-bcac-fd2efb386da6] [type:networkService] ESC[0m(9.1) timeout in recvfd waiting for inode://4/4026535708: context deadline exceeded
Nov 14 16:21:24.619ESC[37m [TRAC] [id:7c5abadb-2f68-46c5-bcac-fd2efb386da6] [type:networkService] ESC[0m(8.1) request-response=null
NSM version: 1.6.1
Spire: 1.2.2
cmd-nsc: ghcr.io/networkservicemesh/ci/cmd-nsc:5de2e87
cmd-nse-icmp-responder: ghcr.io/networkservicemesh/ci/cmd-nse-icmp-responder:9e479a6
Reproduction:
nsm
.internalTrafficPolicy: Local
for nsmgr port 5001 to force NSC to connect the collocated nsmgr:cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
name: nsm-nsmgr-svc
namespace: nsm
spec:
selector:
app: nsmgr
ports:
- name: nsm-nsmgr-svc
protocol: TCP
port: 5001
targetPort: 5001
internalTrafficPolicy: Local
EOF
kind-worker
:cat <<EOF | kubectl apply -f -
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nse-kernel
labels:
app: nse-kernel
spec:
selector:
matchLabels:
app: nse-kernel
template:
metadata:
labels:
app: nse-kernel
"spiffe.io/spiffe-id": "true"
spec:
containers:
- name: nse
image: ghcr.io/networkservicemesh/ci/cmd-nse-icmp-responder:9e479a6
imagePullPolicy: IfNotPresent
env:
- name: SPIFFE_ENDPOINT_SOCKET
value: unix:///run/spire/sockets/agent.sock
- name: NSM_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NSM_LOG_LEVEL
value: TRACE
- name: NSM_CONNECT_TO
value: unix:///var/lib/networkservicemesh/nsm.io.sock
- name: NSM_CIDR_PREFIX
value: 172.16.1.0/24
- name: NSM_SERVICE_NAMES
value: my-kernel-svc
volumeMounts:
- name: spire-agent-socket
mountPath: /run/spire/sockets
readOnly: true
- name: nsm-socket
mountPath: /var/lib/networkservicemesh
readOnly: true
resources:
requests:
cpu: 100m
memory: 40Mi
limits:
memory: 80Mi
cpu: 200m
nodeSelector:
kubernetes.io/hostname: kind-worker
volumes:
- name: spire-agent-socket
hostPath:
path: /run/spire/sockets
type: Directory
- name: nsm-socket
hostPath:
path: /var/lib/networkservicemesh
type: DirectoryOrCreate
EOF
kernel mechanism
:
cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: Pod
metadata:
name: nsc
labels:
app: nsc
"spiffe.io/spiffe-id": "true"
spec:
containers:
- name: nsc
image: ghcr.io/networkservicemesh/ci/cmd-nsc:5de2e87
imagePullPolicy: Always
env:
- name: SPIFFE_ENDPOINT_SOCKET
value: unix:///run/spire/sockets/agent.sock
- name: NSM_LOG_LEVEL
value: TRACE
- name: NSM_NETWORK_SERVICES
value: kernel://my-kernel-svc/nsm-1
- name: NSM_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NSM_LIVENESSCHECKENABLED
value: "false"
- name: NSM_CONNECT_TO
value: nsm-nsmgr-svc.nsm:5001
- name: NSM_LOCALDNSSERVERENABLED
value: "false"
volumeMounts:
- name: spire-agent-socket
mountPath: /run/spire/sockets
readOnly: true
- name: nsm-socket
mountPath: /var/lib/networkservicemesh
readOnly: true
resources:
requests:
cpu: 100m
memory: 40Mi
limits:
memory: 80Mi
cpu: 200m
volumes:
- name: spire-agent-socket
hostPath:
path: /run/spire/sockets
type: Directory
- name: nsm-socket
hostPath:
path: /var/lib/networkservicemesh
type: DirectoryOrCreate
nodeSelector:
kubernetes.io/hostname: kind-worker
EOF
We've discovered that in some cases there are several restarts of the NSMgr container. We could not filter out a single scenario for reproduction, but in most of the cases, it happened when we've installed new Endpoints (NSEs, NSCs).
my-deployment-name-nsmgr-csg65-nsmgr-prev.txt:fatal error: concurrent map writes
my-deployment-name-nsmgr-lwrl8-nsmgr-prev.txt:fatal error: concurrent map writes
my-deployment-name-nsmgr-qzrzs-nsmgr-prev.txt:panic: runtime error: invalid memory address or nil pointer dereference
my-deployment-name-nsmgr-qzrzs-nsmgr-prev.txt-[signal SIGSEGV: segmentation violation code=0x1 addr=0x1b8 pc=0x906677]
NSMgr_logs.zip
The NSMgr version that we used was 5a4b013.
NSMgr leaks memory somewhere (probably in grpcfd
) after long time working.
nsmgr-8g8qn.pb.gz
main.go
:import _ "net/http/pprof"
go func() {
_ = http.ListenAndServe("127.0.0.1:8080", nil)
}()
go tools pprof
.The function of the element is to provide forwarder selection functionality.
Forwarder as NSE diagram: https://app.diagrams.net/#G14HHm-3sY1Zkl3jkV4CRgDNhIgMvNTTkU
We need to move our NSMD to new SDK and put it into new cmd-nsmgr repository.
Assigned to Alexander.
Hi ! I have a question :
Lets say I have 9 containers running in a cluster which comprise of a Network Service.
Now after some time, I want to add a new container which needs to connect to my other 9 containers running previously.
How does NSM address connectivity during runtime when new containers are included in the network service ?
Thanks in advance !
This is a side question which popped up when investigating the issue #675 . It is regarding the 'real' memory consumption of nsmgr container and how this relates to the memory usage of it's single process.
When I checked the memory usage of nsmgr process different values were showed by different tools.
I did not relayed to the 'ps' command output, but checked the 'smaps' in /proc directory (same as pmap command). Another source was the go's profiling (pprof) tool which shows much lower memory used by nsmgr process.
> ps -o pid -o rss -o size -p 2753077
PID RSS SIZE
2753077 28672 58940
Pmap and smaps shown a 29524 kilobytes
The pprof tool gives the following output:
> go tool pprof -top memprofile-15:20:53-4287106497 | head -4
File: nsmgr
Type: inuse_space
Time: May 7, 2024 at 5:20pm (CEST)
Showing nodes accounting for 2581.36kB, 100% of 2581.36kB total
There are different values reported for RSS by pmap (29524 kilobytes), by 'ps' (28904 kilobytes) and by pprof (2581.36kB).
The only process running on this container is the nsmgr process. The grpc probes are removed on this deployment.
The kubectl top reported an 16 Mi bytes which almost the same as systemct and memory.current file shows:
systemctl status 2753077
● cri-containerd-fec3f5061630291f4345bfffa05cd5dc08e5542b98713a45d1d6de2afedfda66.scope - libcontainer container fec3f5061630291f4345bfffa05cd5dc08e5542b98713a45d1d6de2afedfda66
Loaded: loaded (/run/systemd/transient/cri-containerd-fec3f5061630291f4345bfffa05cd5dc08e5542b98713a45d1d6de2afedfda66.scope; transient)
Transient: yes
Drop-In: /run/systemd/transient/cri-containerd-fec3f5061630291f4345bfffa05cd5dc08e5542b98713a45d1d6de2afedfda66.scope.d
└─50-DevicePolicy.conf, 50-DeviceAllow.conf, 50-MemoryMax.conf, 50-CPUWeight.conf, 50-CPUQuotaPeriodSec.conf, 50-CPUQuota.conf
Active: active (running) since Mon 2024-05-06 16:04:23 UTC; 23h ago
IO: 0B read, 2.5M written
Tasks: 21 (limit: 11475)
Memory: 15.4M (max: 200.0M available: 184.2M)
CPU: 9min 14.275s
CGroup: /kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/kubelet-kubepods-burstable-pod7dec5ba0_a449_4165_b315_cc5181a4f45b.slice/cri-containerd-fec3f5061630291f4345bfffa05cd5dc08e5542b98713a45d1d6de2afedfda66.scope
└─2753077 /bin/nsmgr
May 06 16:04:23 kind-worker4 systemd[1]: Started libcontainer container fec3f5061630291f4345bfffa05cd5dc08e5542b98713a45d1d6de2afedfda66.
> cat /sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/kubelet-kubepods-burstable-pod7dec5ba0_a449_4165_b315_cc5181a4f45b.slice/cri-containerd-fec3f5061630291f4345bfffa05cd5dc08e5542b98713a45d1d6de2afedfda66.scope/memory.current
16228352
> cat /sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/kubelet-kubepods-burstable-pod7dec5ba0_a449_4165_b315_cc5181a4f45b.slice/cri-containerd-fec3f5061630291f4345bfffa05cd5dc08e5542b98713a45d1d6de2afedfda66.scope/cgroup.procs
2753077
Later the RSS values showed by ps and pmap command showing much lower memory consumption than cgroup's memory.current and 'kubectl top'. What should cause the difference? What else can count to container memory usage than nsmgr process (CRI, kubelet)?
Please implement some functionality that can be used as livenessProbe and readinessProbe in a helm chart, like the HTTP probes for the old-gen NSM version.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.