cloud-barista/cb-larva

CB-Larva is Cloud-Barista Incubator. The incubator researches and develops essential technologies for multi-cloud.

Read this in other languages: English, 한국어

CB-Larva: Cloud-Barista Incubator

Welcome to Cloud-Barista Incubator (for short CB-Larva) 👶😆

We incubate (research and develop) the new technologies in order to “Contact to the Multi-Cloud”. Proof of concept (POC) of new technologies will be performed in this repostory. Contributions are always welcome. 😍

Note that, you can use and share useful information at Cloud-Barista's coffeehouse.

Challenges in Cloud-Barista

CB-Larva mainly considers multi-cloud network technology (cb-network) for now.

The topics below are wide open.

• cb-storage: Multi-cloud storage technology to support storage for the distributed cloud services
• cb-secret: Secret(e.g., configs, credentials, DB access information) management independently and efficiently from source repositories. It will be researched in a private repository(It has secrets :)). If you have any interest in it, feel free to contact me.
• but not limited.

We’re waiting for your creative ideas. 😊

cb-larva's Issues

Rename cb-network server to controller at the right time

It's necessary to rename the cb-network server to avoid misunderstanding.

By applying the etcd, the base structure and control mechanism of cb-network has been changed to an asynchronous way.
(The watch mechanism is applied.)

However, the terms server and agent may be misunderstood as a directly connected structure and performing synchronous operations. (Actually, the existing structure was also not directly connected :-D)

Cloud-Barista Community suggested me the cb-network controller, and I like it.

Thus, I'd like to apply it at the right time. (May be the next tagging)

Integrate cb-network agent to CB-Tumblebug

What would you like to be added
: Add a installation function of cb-network agent on CB-Tumblebug
Ref: #101 (comment)

Why is this needed
: When a user creates MCIS, the cb-network could be installed on MCIS depending on an installation option (e.g., true/false)

Create README.md of poc-cb-net initially

Add description or guidelines for each directory

Resolve lint error

The below errors were issued by 'golangci-lint'
Error: Error return value of cbnet.CBNet.Write is not checked (errcheck)
Error: Error return value of cbnet.listenConnection.WriteToUDP is not checked (errcheck)
Error: Error return value of cbnet.CBNet.Write is not checked (errcheck)
Error: Error return value of lstnConn.WriteToUDP is not checked (errcheck)
Error: SA9003: empty branch (staticcheck)
Error: SA9003: empty branch (staticcheck)
Error: SA9003: empty branch (staticcheck)
Error: S1005: unnecessary assignment to the blank identifier (gosimple)
Error: SA5003: defers in this infinite loop will never run (staticcheck)
Error: issues found

Create README.KR.md initially

Issue to retrieve a host's public IP address from the networking rule

What happened
: When retrieving public IP address, the agent sometimes access an index out of range

What you expected to happen
: It makes the creation of the cb-network fail.

How to reproduce it (as minimally and precisely as possible)
: I will add a new function for that.

Update branching and releasing strategy

Add dashboard for cb-net server

It will graphically show a network structure drawn by nodes and edges like a graph data structure.

Add lint workflow

Apply cb-log to clearly trace execution processes

Now, poc-cb-net has many prints to check the status.

poc-cb-net has a plan to apply cb-log for a better trace of the execution processes.

Add mqtt-based CB-Net agent and server

Improve dashboard UI

Need tunneling for TCP packets

Current CB-Net (PoC) makes a common subnet.

VMs using the common subnet can communicate with each other by tunneling.

Now, CB-Net supports UDP packets. For example,

1. Read UDP packet (Later, TCP packet could be read)
2. Encapsulate the packet by UDP header (No change this)
3. Send it to the destination

It's not easy for me to decide to add tunneling for TCP packets.

If necessary, it will be added later.

Check LICENSE of the used packages

Provide a function/interface for users to configure a network(like VPC).

What would you like to be added
: A function/interface to configure a CIDR block of the cb-network by users
(CIDR: Classless Inter-Domain Routing)

Why is this needed
: User-customized network control
: Create and control the network users want
Note - Users (administrators) can assign a CIDR block of the cb-network for their VMs.

An example

• Reference: Configuring aliases for IP addresses and network address
  

Change URL getting public IP

Refactor project to apply the Golang package style

The current directory structure and naming should be updated to apply the Golang package style.

I hope it is helpful to provide better reusability.

Replace MQTT with etcd

Add a description of branching rule

Ignore some of on-PR event triggered by markdown-like files

To the updates of markdown-like files, running workflows is not necessary.

For example, built-test workflow is not necessary to the update of README.md.

Thus, events triggered by markdown-like files will be ignored.

Please, refer to: paths-ignore

Archive scripts to create a bridge with eth0 and cbnet0

Handle multiple networking rules of CLADNet

What would you like to be added
: I'd like to handle multiple networking rules of CLADNet.

Why is this needed
: It's needed to show multiple CLADNets on AdminWeb.

Containerize cb-network controller

golangci-lint not to print line number

[Proposal]
Update workflow

    steps:
    - name: Checkout source code
      uses: actions/checkout@v2
    
    - name: Install golangci-lint
      run: curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.34.1      
    
    - name: Run golangci-lint
      run: $(go env GOPATH)/bin/golangci-lint run

Inquire network interface information of a virtual machine

The network interface information of all VMs is necessary to create a common subnet without conflict.

Update lint-on-push.yml to show file name and line number of errors

ref) "lint" section in golang-ci-demo-by-github-actions

Create scripts to install Open vSwitch on Debian

• It will be used for CB-Net PoC to make a tunnel between peers in different network domains.

Archive files related to test and study

Create Git branching and releasing strategy for CB-Larva

Improve README as Go module is added

README.md and README.KR.md will be improved.

Assign VM ID (host ID) to cb-network agent

What would you like to be added
: A method to assign VM ID to the cb-network agent

Why is this needed
: It's because a host needs a unique VM ID to join the CLADNet and obtain an IP address.

Make a copy for the development of etcd-based cb-network.

Refactor READMD.md

Final test and debug for a demonstration of the 3rd conference

Need to restructure this project

Standard Go Project Layout is a basic layout for Go application projects.

According to Standard Go Project Layout, CB-Larva's directories and packages should be restructured.

Improve README of poc-cb-net

Improve the way to allocate IP to a network interface

I'm thinking of improving the IP allocation method.

I tried to assign an IP address for the cb-network to a virtual machine (VM) as part of the networking rule update.

The above process can skip one step, but I have seen several cases that IP is not assigned.

After analyzing the cause, I will improve it.

A plan to construct a MCIS-based cb-network test environment

1. MCIS 생성 및 삭제 테스트 수행

1. 각 CSP(최소 3개)의 crediential 생성 가이드 작성 - 참고
   • 추후 CB-Tumblebug's Test README 개선에 활용 예정
2. CB-Spider 컨테이너 구동 by CB-TB's setcbsp.sh (컨테이너 구동)
3. CB-Tumblebug 구동 by setcbtb.sh (소스코드 빌드)
4. credentials.conf 작성
5. Test 스크립트를 활용한 MCIS 생성 및 삭제 테스트 수행

./testAll-mcis-mcir-ns-cloud.sh aws 2 kimy
./cleanAll-mcis-mcir-ns-cloud.sh aws 2 kimy

./testAll-mcis-mcir-ns-cloud.sh gcp 2 kimy
./cleanAll-mcis-mcir-ns-cloud.sh gcp 2 kimy

./testAll-mcis-mcir-ns-cloud.sh azure 2 kimy
./cleanAll-mcis-mcir-ns-cloud.sh azure 2 kimy

2. MCIS에 cb-network controller 및 agent 배치

[TBD]
cb-network 설치 스크립트 작성
deploy-xxx.sh 를 활용하여 배치

3. cb-network 테스트 수행

[TBD]

Improve the update mechanism of host network information and networking rule

The current mechanism can send host network information and continuously update the networking rule of a certain group by watching the function of the etcd.

Every time the agent is executed, the agent tries to update its host network information even if the host information is not changed.

By this update event, the watch work of other agents is unnecessarily operated.

Thus, I'd like to improve this mechanism.

[To-Be]

1. Watch "networking-rule/{group-id}"
2. Get "networking-rule/{group-id}/{host-id}"
3. If exist, set the networking rule to the host
4. Compare-And-Swap (CAS) "host-network-information/{group-id}/{host-id}" periodically
   Note - the code block to update the networking rule is a critical section.

[As-Is]

1. Watch "networking-rule/{group-id}"
2. Put "host-network-information/{group-id}/{host-id}"

Add scripts to create virtual ethernet interface

• Environment setup
• cb-net setup

Add unit test code

A mechanism to utilize CLADNet ID created by users

What would you like to be added
: A mechanism to utilize CLADNet ID

Why is this needed
: It is needed to support multi-CLADNet for multi-MCIS.

A user (administrator) can request and get a network CIDR block for CLADNet.

Thus, a mechanism to utilize the network CIDR block in cb-network controller and agent.

Add installation scripts of K8s

As a use case of cb-network, K8s will be installed based on cb-network.

For this, installation scripts are needed.

Add go module

Tunneling between VMs by TUN interface configuration

A network tunnel between VMs could be created by the TUN interface configuration.
After this, a common subnet would be configured in multiple VMs.

Watch function runs many times

What happened
: Watch function runs many times.

What you expected to happen
: The issue occurs from "watch with revision"

Containerize the cb-network controller

What would you like to be added
: Dockerfile to create a container image of the cb-network controller (mentioned as the cb-network server previously)

Why is this needed
: I want to keep running the cb-network controller by the "--restart" option.

Reference:

• Start containers automatically

Improve cb-network's AdminWeb

• Enable Websocket on AdminWeb (Backend)
• Watch changes of cb-network by using the etcd client
• Remove MQTT from AdminWeb (Backend and frontend)

Prepare demonstration for the 3rd conference