fabedge/fabedge

Secure Edge Networking Solution Based On Kubernetes

FabEdge

English | 中文

FabEdge is a secure container networking solution based on Kubernetes, focusing on edge computing. It enables cloud-edge, edge-edge collaboration and solves the problems including complex configuration management, network isolation, unaware of the underlying topology, etc. It supports weak network, such as 4/5G, WiFi, etc. The main use cases are IoT, IoV, smart city, etc.

FabEdge supports the major edge computing frameworks ,like KubeEdge/SuperEdge/OpenYurt.

FabEdge not only supports edge nodes (remote nodes joined to the cluster via an edge computing framework such as KubeEdge), but also edge clusters (standalone K8S clusters).

FabEdge is a sandbox project of the Cloud Native Computing Foundation (CNCF).

Features

• Kubernetes Native: Compatible with Kubernetes, transparent to applications.

• Automatic Configuration Management: the addresses, certificates, endpoints, tunnels, etc. are automatically managed.

• Cloud-Edge/Edge-Edge Collaboration: Secure tunnels between cloud-edge, edge-edge nodes for synergy.

• Topology-aware Service Discovery: reduces service access latency, by using the nearest available service endpoint.

Advantages:

• Standard: suitable for any protocol, any application.
• Secure: Uses mature and stable IPSec technology, and a secure certificate-based authentication system.
• Easy to use: Adopts the Operator pattern to automatically manage addresses, nodes, certificates, etc., minimizing human intervention.

How it works

• The cloud can be any Kubernetes cluster with supported CNI network plug-in, including Calico, Flannel, etc.

• FabEdge builds a layer-3 data plane with tunnels in additional to the control plan managed by KubeEdge, SuperEdge, OpenYurt，etc.

• FabEdge consists of Operators, Connector, Agent, Cloud-Agent.

• Operator monitors k8s resources such as nodes, services, and endpoints in the cloud, and creates a configmap for each edge node, which contains the configuration information such as the subnet, tunnel, and load balancing rules. The operator is also responsible to manage the life cycle of agent pod for each edge node.

• Connector is responsible to terminate the tunnels from edge nodes, and forward traffic between the cloud and the edge. It relies on the cloud CNI plug-in to forward traffic to other non-connector nodes in the cloud.

• Cloud-Agent runs on the non-connector nodes in the cluster and manages the routes to remote peers.

• Each edge node runs an agent and consumes its own configmap including the following functions:

  • Manage the configuration file of the CNI plug-in of this node
  • Manage the tunnels of this node
  • Manage the load balancing rules of this node

• Fab-DNS runs in all the clusters, to provide the topology-aware service discovery capability by intercepting the DNS queries.

FabEdge vs. Calico/Flannel/etc

FabEdge is not to replace the traditional Kubernetes network plugins such as Calico/Flannel. As in the above architecture diagram, Calico/Flannel is used within the cloud for communication between cloud nodes, while FabEdge is a complement to it for the edge-cloud, edge-edge communication.

Documentation

• Getting started
• User Guide
• FAQ
• Uninstall FabEdge
• Troubleshooting

Meeting

Regular community meeting at 2nd and 4th Thursday of every month

Resources:
Meeting notes and agenda
Meeting recordings：bilibili channel

Contact

Any question, feel free to reach us in the following ways:

· Email: [email protected]
. Slack: #fabedge
· Scan the QR code to join WeChat Group

Contributing

If you're interested in being a contributor and want to get involved in developing the FabEdge code, please see CONTRIBUTING for details on submitting patches and the contribution workflow.

Please make sure to read and observe our Code of Conduct.

License

FabEdge is under the Apache 2.0 license. See the LICENSE file for details.

fabedge's Issues

[install error]

I follow the guide here, and then i try to install fabedge use helm, but i got error message below.

[root@k8s-node1 ~]# helm install fabedge --create-namespace -n fabedge -f values.yaml http://116.62.127.76/fabedge-0.3.0.tgz
W1130 11:41:51.051053   23950 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
Error: failed pre-install: timed out waiting for the condition

Is there someone take a look for me?

[CNCF] Understand the project proposal process and reqs

Parts of the CNCF onboarding issue tracked by this issue

This is the list of all bullet points from the CNCF onboarding issue that this issue will track:

• Understand the project proposal process and reqs: https://github.com/cncf/toc/blob/main/process/project_proposals.adoc#introduction

[CNCF] Understand the trademark guidelines

Parts of the CNCF onboarding issue tracked by this issue

This is the list of all bullet points from the CNCF onboarding issue that this issue will track:

• Understand the trademark guidelines: https://www.linuxfoundation.org/en/trademark-usage/

Website does not have the correct trademark disclaimer

As part of our ongoing effort to cncf/techdocs#198, we noticed that the website does not pass the trademark criteria on CLOMonitor.

To fix this:
Head to the source code of the website. In the <footer> section, add a disclaimer or link to the Linux foundation trademark disclaimer page:

Disclaimer

<footer>
   <p>The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, 
         please see our <a href="https://www.linuxfoundation.org/legal/trademark-usage">Trademark Usage page</a>.
   </p>
</footer>

Link

 <footer>
      <ul>
          <li><a href="https://www.linuxfoundation.org/legal/trademark-usage">Trademarks</a></li>
      </ul>
 </footer>

[CNCF] Maintainer list + PR

Parts of the CNCF onboarding issue tracked by this issue

This is the list of all bullet points from the CNCF onboarding issue that this issue will track:

• Create maintainer list + add to aggregated https://maintainers.cncf.io/ list by submitting a PR to it.

[CNCF] Are your slack channels migrated to the Kubernetes or CNCF Slack?

Parts of the CNCF onboarding issue tracked by this issue

This is the list of all bullet points from the CNCF onboarding issue that this issue will track:

• Slack: Are your slack channels migrated to the Kubernetes or CNCF Slack? (see https://slack.com/help/articles/217872578-Import-data-from-one-Slack-workspace-to-another for more details)

[CNCF] Add FabEdge to CNCF Contributors Website

Parts of the CNCF onboarding issue tracked by this issue

This is the list of all bullet points from the CNCF onboarding issue that this issue will track:

• Have added your project to https://github.com/cncf/contribute

[CNCF] Understand the license allowlist

Parts of the CNCF onboarding issue tracked by this issue

This is the list of all bullet points from the CNCF onboarding issue that this issue will track:

• Understand the license allowlist: https://github.com/cncf/foundation/blob/master/allowed-third-party-license-policy.md#approved-licenses-for-allowlist

[CNCF] Submitted a Pull request to add your project as a sandbox project

Parts of the CNCF onboarding issue tracked by this issue

This is the list of all bullet points from the CNCF onboarding issue that this issue will track:

• Submitted a Pull request to add your project as a sandbox project to https://landscape.cncf.io/

[Question] What kubeedge feature does fabedge depend on?

What kubeedge feature does fabedge depend on, such as Autonomic Kube-API Endpoint for list-watch, kubectl exec on the edge or others?

[CNCF] Transfer domain to the CNCF

Parts of the CNCF onboarding issue tracked by this issue

This is the list of all bullet points from the CNCF onboarding issue that this issue will track:

• Domain: transfer domain to the CNCF - https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/create/63

是否支持标准K8S部署

请问是否支持在标准K8S集群下部署使用？
集群是由边缘节点和云端节点组成的普通K8S集群，想实现云边通信的能力。

A potential risk in fabedge that could lead to takeover of the cluster

Our team has recently identified a potential security risk in the fabedge project. We would like to report it to you and provide you with the relevant details so that you can fix and improve it accordingly, we have sent the specific details to your private email [email protected] and look forward to hearing from you!

edge nodes cannot connect 同

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[cloud-edge communication failed]

I follw the guide here to install fabedge upon openyurt.
After installed fabedge seems work fine.

[root@k8s-node1 ~]# kubectl get no
NAME             STATUS   ROLES                            AGE   VERSION
centos72-k8s     Ready    <none>                           5d    v1.20.9
k8s-node1        Ready    connector,control-plane,master   22d   v1.20.9
k8s-node2        Ready    <none>                           22d   v1.20.9
openyurt-edge2   Ready    <none>                           15d   v1.20.9

[root@k8s-node1 ~]# kubectl get po -n fabedge
NAME                                READY   STATUS      RESTARTS   AGE
cert-m5rb4                          0/1     Completed   0          41m
connector-68dbbdf547-dp8sq          2/2     Running     0          41m
fabedge-agent-k8s-node2             2/2     Running     0          41m
fabedge-agent-openyurt-edge2        2/2     Running     0          41m
fabedge-operator-6544c47c56-k7htx   1/1     Running     0          41m

[root@k8s-node1 ~]# kubectl get po -n kube-system
NAME                                       READY   STATUS      RESTARTS   AGE
coredns-5897cd56c4-8pb5p                   1/1     Running     0          22d
coredns-5897cd56c4-tvm4n                   1/1     Running     0          22d
etcd-k8s-node1                             1/1     Running     5          22d
kube-apiserver-k8s-node1                   1/1     Running     9          22d
kube-controller-manager-k8s-node1          1/1     Running     6          19d
kube-flannel-ds-2mh9n                      1/1     Running     0          16m
kube-proxy-fgs6k                           1/1     Running     3          15d
kube-proxy-h2lfs                           1/1     Running     5          22d
kube-proxy-rzfc2                           1/1     Running     0          22d
kube-proxy-vvt7n                           1/1     Running     5          5d
kube-scheduler-k8s-node1                   1/1     Running     8          22d
yurt-app-manager-7864899795-2fhmh          1/1     Running     1          15d
yurt-app-manager-7864899795-4djvj          1/1     Running     1          15d
yurt-controller-manager-77b97fd47b-ctj42   1/1     Running     6          19d
yurt-hub-k8s-node1                         1/1     Running     4          19d
yurt-hub-k8s-node2                         1/1     Running     0          19d
yurt-hub-openyurt-edge2                    1/1     Running     3          15d
yurt-tunnel-agent-knm57                    1/1     Running     0          18d
yurt-tunnel-agent-x5r4z                    1/1     Running     3          15d
yurt-tunnel-server-69cb47d8fc-766lh        1/1     Running     7          18d
yurtctl-servant-convert-k8s-node1-cdl22    0/1     Completed   1          19d
yurtctl-servant-convert-k8s-node2-5kp42    0/1     Completed   1          19d

Then i deploy a pod on cloud master and openyurt edge using nginx image to experience fabedge cloud-egde communication feature, but after i enter pod use commandkubecl exec -it and use curl <edge nginx pod ip> to get response, but just timeout. edge-cloud communication the same.

[root@k8s-node1 ~]# kubectl get pod -n dev -o wide
NAME                      READY   STATUS    RESTARTS   AGE   IP             NODE             NOMINATED NODE   READINESS GATES
test-k8s-node1-pod        1/1     Running   0          33m   192.168.0.13   k8s-node1        <none>           <none>
test-k8s-node2-pod        1/1     Running   0          33m   192.168.1.3    k8s-node2        <none>           <none>
test-openyurt-edge2-pod   1/1     Running   0          32m   192.168.3.12   openyurt-edge2   <none>           <none>
[root@k8s-node1 ~]# kubectl exec -it test-k8s-node1-pod -n dev -- /bin/sh
# curl 192.168.3.12
curl: (7) Failed to connect to 192.168.3.12 port 80: Connection timed out

What's the matter?

CNCF onboarding!

FabEdge welcomes all to complete Sandbox onboarding through the community.
Please see cncf/toc#798 for more information!

As an open source community, we must come together to complete all the required elements for FabEdge to become a full-fledged Sandbox project.

We welcome all to participate in this work, and to better accomplish our goals, I suggest you follow following points.

1. Open an issue in fabEdge, describing which parts of the CNCF onboarding effort the issue will track. There are dedicated issue template and label for that, please follow them.

2. Considering that our community is based on Chinese, so, I am working on the process of translating the relevant documents into Chinese and will upload them to the relevant files in FabEdge Community for your understanding when they are completed. We welcome more people to participate, as well as we thank you for pointing out our mistakes.

Once the issue is closed, we will update the CNCF issue itself.

I'm sorry, FabEdge has actually been in the sandbox for a while, but my partner and I have been delayed by some other work.

Also, I would like to thank confidential-containers, a sandbox project, for being a reference for our sandbox processes. We are new to sandbox and in many cases, we don't know how to operate it, and their process is a good example for us.

Thank you for your contribution and help!

[CNCF] Understand the services available for your project at CNCF

Parts of the CNCF onboarding issue tracked by this issue

This is the list of all bullet points from the [CNCF onboarding issue](cncf/toc#798) that this issue will track:

• Understand the services available for your project at CNCF https://www.cncf.io/services-for-projects/

[Question] Does fabedge respect podCIDR?

I want to know that does fabedge respect podCIDR which allocated to every node by rangeAllocator in kube-controller-manager or fabedge will generate a know one?

There is the problem when pod accessing its service

Describe the bug
When using fabedge on superedge, there is a problem when pod accessing its service.

To Reproduce
Steps to reproduce the behavior:

1. Create echo-service deployment and service on superedge.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-service
spec:
  replicas: 2
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - echo
              topologyKey: "kubernetes.io/hostname"
      containers:
      - image: superedge/echoserver:2.2
        name: echo
        ports:
        - containerPort: 8080
          protocol: TCP
        env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        resources: {}
---
apiVersion: v1
kind: Service
metadata:
  name: echo-service
  namespace: default
spec:
  selector:
    app: echo
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

2. Exec into one echo-service pod and access echo-service.
3. There appears intermittent problems when access echo-service.

Expected behavior
Pod normally access its service.

to support nodes with public ip address to join community

Is your feature request related to a problem? Please describe.
After nodes are added into a community, they will try to establish tunnels with internal ip addresses. In the cases they belong to different cloud provides, it does not work anymore. We need to tell them use public ip instead of internal ip.

Cloud agent leads to infinite growth of IP rules

Describe the bug

The IP rules of cloud agent nodes have increased by approximately 400 records in 14 hours. I don't know if this will have an impact on the operating system

fabedge version: v0.8.0
os version: ubuntu 16.04

[doc] About install.md

In the install.md , root@node1:~# git clone https://github.com/fabedge/fabeedge.git the link of the fabedge is wrong

Security Slam 2023 Participation

Hello fabedge community!

In preparation for this year's Cloud Native Security Slam, we've completed a survey of CNCF end users across multiple industries, including Construction, Cybersecurity, Aerospace & Defense, Game Development & Consumer Services, Consulting, and Nuclear.

Through this survey, end users have identified their interest in seeing security improvements to the projects they use. We've asked them to share which Security Slam goals are most interesting to them— and we've compiled the results in a hope that this will help your prioritization during the upcoming event.

While some users have not authorized us to share their name, we've still included their responses in our calculation for you. We CAN tell you that the fabedge responses included Epic Games.

After calculating the responses according to the interest-weight, we've found these to be the most interesting things that fabedge end users would like to see, from the five possible Security Slam badges.

1. The Mechanizer
2. The Defender
3. The Cleaner

More information will be announced in the event kickoff webinar on October 10th, including how to register for cash & swag prizes, details about how success is measured, and resources to help achieve each of the badge goals.

If you can't make it to the webinar, a recording will be made available within 24hrs. It will be sent out to the community newsletter with any essential details you may have missed.

Join the community & sign up for the webinar here: https://community.cncf.io/cloud-native-security-slam/

---

A quick look at the 2023 Event Badges

The Chronicler

Ensure that security documentation has properly formatted data relating to software supply chain security decisions, including instructions for end users seeking to validate provenance artifacts.

The Inspector

Ensure that a security self-assessment has been completed according to TAG-Security documented standards.

The Cleaner

Bring all CLOMonitor non-security scores to 100% for the project, indirectly increasing overall supply chain security (Best Practices, Documentation, License, Legal).

The Defender

Ensure each project repo is accounted for within CLOMonitor; Ensure proper check set is assigned to each project repo; Bring security score to 100% for the project (This statistically decreases the future likelihood of vulnerabilities).

The Mechanizer

Ensure that every release has an automated mechanism to supply SBOM and provenance artifacts.

CNCF TOC annual review due

Annual review due

Sandbox projects are subject to an annual review by the TOC. This is intended to be a lightweight process to ensure that projects are on track, and getting the support they need.

CLOMonitor has detected that the annual review for this project has not been filed yet. CLOMonitor relies on the information in the annual_review_url and annual_review_date fields in the CNCF Landscape configuration file for this check. If your annual review has already been presented, please make sure this information has been correctly added.

For more information about how to file your annual review please see the Sandbox annual review documentation.

secrets "tokensecret" not found

按照https://github.com/FabEdge/fabedge/blob/main/docs/install_k8s.md 文档第二部添加k8s边缘节点时候出错 add-edge-node.sh ，debug了一下，应该是找不到tokensecret
[root@ubuntu ansible]# kubectl get secret -nkubeedge tokensecret
Error from server (NotFound): secrets "tokensecret" not found
[root@ubuntu ansible]#

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[CNCF] Review the Online Programs Guidelines

Parts of the CNCF onboarding issue tracked by this issue

This is the list of all bullet points from the CNCF onboarding issue that this issue will track:

• Review the online programs guidelines: https://github.com/cncf/foundation/blob/master/online-programs-guidelines.md

[CNCF] Is your project working on written, open governance?

Parts of the CNCF onboarding issue tracked by this issue

This is the list of all bullet points from the CNCF onboarding issue that this issue will track:

• Is your project working on written, open governance? see https://contribute.cncf.io/maintainers/governance/

[CNCF] Artwork, FabEdge logo updating

Parts of the CNCF onboarding issue tracked by this issue

This is the list of all bullet points from the CNCF onboarding issue that this issue will track:

• Artwork: Submit a pull request to https://github.com/cncf/artwork with your artwork

[CNCF] Ensure your project meets the CNCF IP Policy

Parts of the CNCF onboarding issue tracked by this issue

This is the list of all bullet points from the CNCF onboarding issue that this issue will track:

• Ensure your project meets the CNCF IP Policy: https://github.com/cncf/foundation/blob/master/charter.md#11-ip-policy

to support hairpin mode

Describe the bug
pod can not access the service backed by itself.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
pod should be able to access any service

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[Question] What features have support?

According to the https://github.com/FabEdge/fabedge/blob/main/docs/roadmap.md, what features have support now? 1. the communication between edge pods 2. the communication between edge node and cloud node

[CNCF] Is your project in its own separate neutral github organization?

Parts of the CNCF onboarding issue tracked by this issue

This is the list of all bullet points from the CNCF onboarding issue that this issue will track:

• Is your project in its own separate neutral github organization?

cloud nodes except connector cannot visit pods in edge nodes

Describe the bug
cloud nodes except connector cannot visit pods in edge nodes

environment
amd64处理器
centos7虚拟机环境
软件版本如截图所示：

安装步骤:
安装好kubeedge之后
patch掉边端的kube-proxy
执行quickstart.sh脚本
然后建立节点之间的community

错误场景
在边缘节点worker01搭建了一个nginx服务。
worker01和worker02以及connector节点都能通过pod的ip地址访问nginx服务，master节点和其他节点无法访问这个nginx服务

Flannel cni remaining on edge

Describe the bug
There is still flannel cni remaining on edge when fabedge take effects.

To Reproduce
Steps to reproduce the behavior:

1. Deploy superedge with cloud-0 and edge-0.
2. Label nodes and deploy fabedge.
3. Add another edge-1 to superedge.
4. Label edge-1 to join fabedge network.
5. There flannel cni remaining on edge-1.

$ ip r
...
192.168.1.0/24 dev cni0 proto kernel scope link src 192.168.1.1 
192.168.1.0/24 dev br-fabedge proto kernel scope link src 192.168.1.1 

and env-prepares scripts output below:

+ find /etc/cni/net.d/ -type f -not -name fabedge.conf -exec rm '{}' ';'
+ cp -f /usr/local/bin/bridge /usr/local/bin/host-local /usr/local/bin/loopback /opt/cni/bin
+ ip link delete cni0
ip: can't find device 'cni0'
+ ip link delete flannel.1
ip: can't find device 'flannel.1'
+ ip route
+ grep flannel
+ read dst via gw others
+ iptables -t nat -F POSTROUTING
+ exit 0

Environments
Centos 7.8

Expected behavior
No flannel cni remains on edge when fabedge takes over edge network.