Comments (8)
pawelbaranski
commented on June 15, 2024
I checked again with default-deny-all policy removed - I still got random drops
from cilium.
pawelbaranski
commented on June 15, 2024
removing both default-deny-all and redis-locks-test-allowed-ingress seems to stop the drops but it cant be the solution here
from cilium.
adamw-linadm
commented on June 15, 2024
Hey devs any idea how to debug this issue?
from cilium.
EItanya
commented on June 15, 2024
Hello 👋
How large is the cluster in question?
I'm not super familiar with how fromEntities: cluster works under the hood, but I'm curious why you're using that as opposed to individual namespaces, or pod labels.
Also, does this continue to happen for the lifetime of the CronJobs, or does it fix itself after a few seconds?
from cilium.
pawelbaranski
commented on June 15, 2024
The cluster where I am reproducing the problem has 3 nodes , 1 worker node. Second one, our main cluster has 13 nodes, 10 worker nodes - same problem here.
I updated the logic, so that it does 5 iterations and 2 calls to redis in each one.
$i = 5;
while ($i--)
{
$lock->acquire($id, 100);
sleep(5 + rand(5, 10));
$lock->release($id);
}
with hubble observe I can confirm that in a single pod some requests were forwarded some were not, basically the same requests. Logs
Jan 12 12:03:02.696: test-cron/test-cron-action-183a-pl-28417683-k7drt:55946 (ID:15453) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) policy-verdict:L3-L4 INGRESS ALLOWED (TCP Flags: SYN)
Jan 12 12:03:02.696: test-cron/test-cron-action-183a-pl-28417683-k7drt:55946 (ID:15453) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-endpoint FORWARDED (TCP Flags: SYN)
Jan 12 12:03:02.696: test-cron/test-cron-action-183a-pl-28417683-k7drt:55946 (ID:15453) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-endpoint FORWARDED (TCP Flags: ACK)
Jan 12 12:03:02.696: test-cron/test-cron-action-183a-pl-28417683-k7drt:55946 (ID:15453) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jan 12 12:03:17.697: test-cron/test-cron-action-183a-pl-28417683-k7drt:55946 (ID:15453) <> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) policy-verdict:none EGRESS DENIED (TCP Flags: ACK, PSH)
Jan 12 12:03:17.697: test-cron/test-cron-action-183a-pl-28417683-k7drt:55946 (ID:15453) <> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) Policy denied DROPPED (TCP Flags: ACK, PSH)
Jan 12 12:03:17.904: test-cron/test-cron-action-183a-pl-28417683-k7drt:55946 (ID:15453) <> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) policy-verdict:none EGRESS DENIED (TCP Flags: ACK, PSH)
Jan 12 12:03:17.904: test-cron/test-cron-action-183a-pl-28417683-k7drt:55946 (ID:15453) <> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) Policy denied DROPPED (TCP Flags: ACK, PSH)
Jan 12 12:03:18.113: test-cron/test-cron-action-183a-pl-28417683-k7drt:55946 (ID:15453) <> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) policy-verdict:none EGRESS DENIED (TCP Flags: ACK, PSH)
Jan 12 12:03:18.113: test-cron/test-cron-action-183a-pl-28417683-k7drt:55946 (ID:15453) <> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) Policy denied DROPPED (TCP Flags: ACK, PSH)
Jan 12 12:03:18.520: test-cron/test-cron-action-183a-pl-28417683-k7drt:55946 (ID:15453) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jan 12 12:03:31.521: test-cron/test-cron-action-183a-pl-28417683-k7drt:55946 (ID:15453) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jan 12 12:03:45.521: test-cron/test-cron-action-183a-pl-28417683-k7drt:55946 (ID:15453) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jan 12 12:04:00.521: test-cron/test-cron-action-183a-pl-28417683-k7drt:55946 (ID:15453) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jan 12 12:04:15.522: test-cron/test-cron-action-183a-pl-28417683-k7drt:55946 (ID:15453) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jan 12 12:04:15.523: test-cron/test-cron-action-183a-pl-28417683-k7drt:55946 (ID:15453) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Jan 12 12:04:15.523: test-cron/test-cron-action-183a-pl-28417683-k7drt:55946 (ID:15453) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-endpoint FORWARDED (TCP Flags: ACK)
from cilium.
EItanya
commented on June 15, 2024
One thing that's particularly interesting about the log you posted is that it's policy-verdict:none EGRESS DENIED, which means that the caller is rejecting, not redis. Do you potentially have CiliumNetworkPolicy resources for the CronJob as well, and can you send those over/
from cilium.
pawelbaranski
commented on June 15, 2024
We have udated cilium to 1.14.6, problem still persists. I can't reproduce policy-verdict:none EGRESS DENIED though. As I mentioned in the first post, it seemed like an ingress policy issue and with my latest logs I can confirm it is still an ingress issue.
I run another hubble observe like this (for 3 cilium agent pods in our test cluster)
kubectl -n kube-system exec -ti cilium-9jhtl -- hubble observe -f --to-namespace cache >> .dev/cilium-9jhtl.log
with this I got 13 endpoints that got drops in about 15 minutes. About ~40 pods running at the same time with the test code.
each one looks the same, for example ID:11143
Feb 2 11:07:37.567: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) <> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
Feb 2 11:07:37.567: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) <> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) Policy denied DROPPED (TCP Flags: SYN)
Feb 2 11:07:38.587: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) policy-verdict:L3-L4 INGRESS ALLOWED (TCP Flags: SYN)
Feb 2 11:07:38.587: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-endpoint FORWARDED (TCP Flags: SYN)
Feb 2 11:07:38.587: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-endpoint FORWARDED (TCP Flags: ACK)
Feb 2 11:07:38.587: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Feb 2 11:07:52.588: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Feb 2 11:08:04.589: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Feb 2 11:08:16.590: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Feb 2 11:08:27.590: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Feb 2 11:08:38.591: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Feb 2 11:08:38.592: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Feb 2 11:08:38.592: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-endpoint FORWARDED (TCP Flags: ACK)
Feb 2 11:07:37.567: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) policy-verdict:L3-L4 EGRESS ALLOWED (TCP Flags: SYN)
Feb 2 11:07:37.567: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-overlay FORWARDED (TCP Flags: SYN)
Feb 2 11:07:38.588: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-overlay FORWARDED (TCP Flags: ACK)
Feb 2 11:07:38.588: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-overlay FORWARDED (TCP Flags: ACK, PSH)
Feb 2 11:07:52.588: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-overlay FORWARDED (TCP Flags: ACK, PSH)
Feb 2 11:08:04.589: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-overlay FORWARDED (TCP Flags: ACK, PSH)
Feb 2 11:08:16.590: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-overlay FORWARDED (TCP Flags: ACK, PSH)
Feb 2 11:08:27.591: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-overlay FORWARDED (TCP Flags: ACK, PSH)
Feb 2 11:08:38.591: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-overlay FORWARDED (TCP Flags: ACK, PSH)
Feb 2 11:08:38.592: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-overlay FORWARDED (TCP Flags: ACK, FIN)
Feb 2 11:08:38.593: test-cron/test-cron-action-55c-pl-28447867-zgmvh:46544 (ID:11143) -> cache/redis-locks-75c455b7bf-cxplr:6379 (ID:50432) to-overlay FORWARDED (TCP Flags: ACK)
In all of these logs INGRESS DENIED is right before INGRESS ALLOWED and there is EGRESS ALLOWED at some point. In 12 of cases EGRESS ALLOWED is after INGRESS ALLOWED, in 1 case it is before INGRES DENIED, but since it is ruled as ALLOWED I'm not sure if that particullar log line matters. So to sum up it looks like this
12 cases
... policy-verdict:none INGRESS DENIED ...
... Policy denied DROPPED ...
... policy-verdict:L3-L4 INGRESS ALLOWED ...
... [logs for FORWARDED] ...
... policy-verdict:L3-L4 EGRESS ALLOWED ...
... [logs for FORWARDED] ...
1 case
... policy-verdict:L3-L4 EGRESS ALLOWED ...
... policy-verdict:none INGRESS DENIED ...
... Policy denied DROPPED ...
... policy-verdict:L3-L4 INGRESS ALLOWED ...
... [logs for FORWARDED] ...
from cilium.
pawelbaranski
commented on June 15, 2024
I've tried to use fromEndpoint instead of fromEntities. I had to change resource to CiliumClusterwideNetworkPolicy to make it cross-namespace with my test label (with CiliumNetworkPolicy I had nothing but drops). Change I made
apiVersion: cilium.io/v2
-kind: CiliumNetworkPolicy
+kind: CiliumClusterwideNetworkPolicy
metadata:
name: redis-locks-test-allowed-ingress
spec:
endpointSelector:
matchLabels:
app: redis-locks
ingress:
- - fromEntities:
- - cluster
- toPorts:
- - ports:
- - port: "6379"
+ - fromEndpoints:
+ - matchLabels:
+ mylabel: "test"
and I added mylabel: "test" to test crons. Result was the same as above, basically I got about 8 drops in few minutes with output like this
Feb 5 10:10:19.626: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) policy-verdict:L3-L4 EGRESS ALLOWED (TCP Flags: SYN)
Feb 5 10:10:19.626: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) to-overlay FORWARDED (TCP Flags: SYN)
Feb 5 10:10:20.635: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) to-overlay FORWARDED (TCP Flags: ACK)
Feb 5 10:10:20.636: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) to-overlay FORWARDED (TCP Flags: ACK, PSH)
Feb 5 10:10:31.636: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) to-overlay FORWARDED (TCP Flags: ACK, PSH)
Feb 5 10:10:41.637: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) to-overlay FORWARDED (TCP Flags: ACK, PSH)
Feb 5 10:10:54.638: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) to-overlay FORWARDED (TCP Flags: ACK, PSH)
Feb 5 10:11:06.639: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) to-overlay FORWARDED (TCP Flags: ACK, PSH)
Feb 5 10:11:21.640: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) to-overlay FORWARDED (TCP Flags: ACK, PSH)
Feb 5 10:11:21.641: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) to-overlay FORWARDED (TCP Flags: ACK, FIN)
Feb 5 10:11:21.641: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) to-overlay FORWARDED (TCP Flags: ACK)
Feb 5 10:10:19.626: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) <> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
Feb 5 10:10:19.626: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) <> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) Policy denied DROPPED (TCP Flags: SYN)
Feb 5 10:10:20.635: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) policy-verdict:L3-Only INGRESS ALLOWED (TCP Flags: SYN)
Feb 5 10:10:20.635: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) to-endpoint FORWARDED (TCP Flags: SYN)
Feb 5 10:10:20.636: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) to-endpoint FORWARDED (TCP Flags: ACK)
Feb 5 10:10:20.636: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Feb 5 10:10:31.637: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Feb 5 10:10:41.637: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Feb 5 10:10:54.638: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Feb 5 10:11:06.639: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Feb 5 10:11:21.640: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Feb 5 10:11:21.641: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Feb 5 10:11:21.641: test-cron/test-cron-action-141c-com-28452130-mwtc6:54456 (ID:57350) -> cache/redis-locks-7755dd4859-lrsdm:6379 (ID:29518) to-endpoint FORWARDED (TCP Flags: ACK)
which is a pattern like that one
... policy-verdict:L3-L4 EGRESS ALLOWED ...
... [logs for FORWARDED] ...
... policy-verdict:none INGRESS DENIED ...
... Policy denied DROPPED ...
... policy-verdict:L3-L4 INGRESS ALLOWED ...
... [logs for FORWARDED] ...
from cilium.
Related Issues (20)
- CFP: Allow to override on per-node basis which IP would be advertised as next-hop with BGP Control Plane HOT 4
- Loopback `lo` device no longer considered for Direct Routing in 1.15 HOT 7
- CI: controlplane: 'Timed out waiting for pre-existing resources to be received' HOT 1
- Unify datapath configuration infrastructure
- Reduce `cDefinesMap` usage to a minimum
- CI: Conformance Runtime (privileged): TestOps: "0" is not greater than "0"
- Cilium agent removes ciliumnode.spec.eni.subnet-ids when adding CiliumInternalIP and health IP HOT 8
- CI: Conformance Cluster Mesh - check-log-errors - bind: address already in use HOT 1
- bpf: enhance FIB-driven redirect support
- KPR enhancements
- CI: Conformance AKS - Hubble Relay CrashLoopBackOff
- datapath: enhancements for direct-routing interface
- Cilium health starts, but in fact no HOT 7
- datapath: ICMP enhancements
- Gateway API not passing TLS certificate HOT 9
- Cilium doesn't send neighbor advertisements in response to neighbor solicitiations for pod IPv6 IPs HOT 1
- CI: ci-runtime: hive/job: TestTimer_ExitOnCloseFnCtx times out
- CI: Conformance Ginkgo: E2E Test (1.26, f10-agent-hubble-bandwidth): "Failed to create gRPC client" warning
- CI: pkg/alibabacloud/eni: ENISuite/TestPrepareIPAllocation is flaky HOT 1
- CFP: Simplify troubleshooting connectivity issues towards control plane components
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cilium.