lgirdk/boardfarm

Automated testing with python

boardfarm's Issues

Add support for exporting data to MongoDB

tests/lib : added logging.py to log assertions with log message

We see that when a test fails, we just get the assertion error and its difficult to track down at which step the failure is.
We have come up with a generic function that does the assertions based upon condition. It sends log message to the .txt file under results folder and assertion error with the user-defined message in the execution console.

mattsm/boardfarm#16

Add keepalive feature

If a test runs a long time, it will possibly not send anything on the console and it will close due to inactivity, so add a feature to keep the console alive if a test is doing something. Also, handle multiple consoles.

-M

Multiple Lan Containers with same VLAN (To Simulate Multiple LAN Clients)

Hello All,

We have scenarios which needs multiple LAN Clients, so if we need to automate them, we need more than 1 Docker LAN Containers with same VLAN.

I understand, we are working on this but having some hiccups. Just wanted to document the ask so we can keep work on it and get a solution.

Switch from tinyproxy to ssh dynamic proxy

We already ssh for the most part, we can still ssh and add a dynamic proxy option so we can use that instead of tinyproxy. The advantage is all traffic will go over ssh (and the various jump hosts) and traffic will come out ipv4 or ipv6 and don't require an ipv4 route directly to the LAN container.

Feasibility to have Folder structure in boardfarm to manage testcase files

Hello,

As we have huge number of testcases, its not good to write the code in single file as it will grow in space and also, its creating "Merge Conflict" if atleast 2 engineers are working on the same file in parallel.

Another idea would be to create multiple files (Ex: management_snmp_p1.py, management_snmp_p2.py, etc..), but still we have lot many P2 and P3 cases and hence these files will also grow in space and chances of Merge Conflict is still there.

So, shall we please think of having folders, so that folks can create single file of code per testcase which shall be saved in a particular folder path (Example: Management--> snmp --> P1 --> Testcase_1.py; Management--> snmp --> P1 --> Testcase_2.py).

By this way, we shall avoid having the conflict because every engineer is working on a unique file and hence it wont create any conflict.

Dynamic Test Suite File for WiFi Matrix

Hello Matthew,

Request you to help the team in making Dynamic Test Suite file depends on the number of combinations generated by the WiFi Matrix Combination Generator Script.

Using this Dynamic Test Suite, the Base class shall be called and the execution shall happen.

Add option to fail if we can't parse a test case

Right now we skip a test if we can't parse the test case but there are times when we really want to fail if this occurs (like for example when we are testing the tests) so let's add an option for that.

Running more than 1 bft scripts from the same directory at the same time will overwrite the results

make a BFT_RESULTS_DIR flag that will use a timestamp format or something to save to a different directory

Convert all ip address to use ipaddress library and mac addresses to use netaddr library

deploy-boardfarm-nodes.sh can't loop to create_container_eth1_vlan

I want build lan interface and run as below:
~/cbn/boardfarm$ ./deploy-boardfarm-nodes.sh eth0 101 104
I see 'create_container_eth1_vlan' 101 done and stop(no create next container) if pull new commit (commit fac4b98).

log message :

~/cbn/boardfarm$ ./deploy-boardfarm-nodes.sh eth0 101 104
+ IFACE=eth0
+ START_VLAN=101
+ END_VLAN=104
+ OPTS=both
+ BRINT=br-bft
+ '[' eth0 = undefined ']'
+ echo 'Creating nodes starting on vlan 101 to 104 on iface eth0'
Creating nodes starting on vlan 101 to 104 on iface eth0
++ seq 101 104
+ for vlan in '$(seq $START_VLAN $END_VLAN)'
+ echo 'Creating node on vlan 101'
Creating node on vlan 101
+ create_container_eth1_vlan 101
+ local vlan=101
+ cname=bft-node-eth0-101
+ docker stop bft-node-eth0-101
Error response from daemon: No such container: bft-node-eth0-101
+ docker run --name bft-node-eth0-101 --privileged -h bft-node-eth0-101 --restart=always -p 5101:22 -p 8101:8080 -d bft:n
ode /usr/sbin/sshd -D
e51fc24d84b4324d3c40c6aa0b39664d62cb777984a42431c2814ff5028cfa24
+ sudo ip link del eth0.101
Cannot find device "eth0.101"
+ true
+ sudo ip link add link eth0 name eth0.101 type vlan id 101
++ docker inspect --format '{{.State.Pid}}' bft-node-eth0-101
+ cspace=12900
+ sudo ip link set netns 12900 dev eth0.101
+ docker exec bft-node-eth0-101 ip link set eth0.101 name eth1
++ random_private_mac 101
++ echo 101101101101101101
++ od -An -N6 -tx1
++ sed -e 's/^  *//' -e 's/  */:/g' -e 's/:$//' -e 's/^\(.\)[13579bdf]/\10/'
+ docker exec bft-node-eth0-101 ip link set dev eth1 address 30:30:31:31:30:31
+ '[' both = both ']'
+ local_route
++ ip route list
++ grep '^default'
++ awk '{print $5}'
+ host_dev=eth1
++ ip route
++ grep 'dev eth1'
++ grep src
++ awk '{print $1}'
++ head -n1
+ local_route=172.16.1.0/24
+ docker exec bft-node-eth0-101 ip route add 172.16.1.0/24 dev eth0 via 192.168.101.1
RTNETLINK answers: Network is unreachable
+ IFACE=eth1 create_container_eth1_static wan-shared 172.16.1.220/24 172.16.1.1
+ local name=wan-shared
+ local ip=172.16.1.220/24
+ local default_route=172.16.1.1
+ cname=bft-node-eth1-wan-shared
+ docker stop bft-node-eth1-wan-shared
Error response from daemon: No such container: bft-node-eth1-wan-shared
+ docker run --name bft-node-eth1-wan-shared --privileged -h bft-node-eth1-wan-shared --restart=always -d --network=none
bft:node /usr/sbin/sshd -D
1ae4bea09a7e743bead5c21901704a0c027da775b4eb6d191963c34e55df8f40
+ docker inspect --format {{.State.Pid}} bft-node-eth1-wan-shared
+ cspace=13109
+ sudo ip link add tempfoo link eth1 type macvlan mode bridge
+ sudo ip link set dev tempfoo up
+ sudo ip link set netns 13109 dev tempfoo
+ docker exec bft-node-eth1-wan-shared ip link set tempfoo name eth1
+ docker exec bft-node-eth1-wan-shared ip link set eth1 up
+ docker exec bft-node-eth1-wan-shared ip addr add 172.16.1.220/24 dev eth1
+ docker exec bft-node-eth1-wan-shared ip route add default via 172.16.1.1 dev eth1
+ docker exec bft-node-eth1-wan-shared ping 172.16.1.1 -c3
PING 172.16.1.1 (172.16.1.1) 56(84) bytes of data.
64 bytes from 172.16.1.1: icmp_seq=1 ttl=255 time=5.25 ms
64 bytes from 172.16.1.1: icmp_seq=2 ttl=255 time=8.77 ms
64 bytes from 172.16.1.1: icmp_seq=3 ttl=255 time=1.00 ms

--- 172.16.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 1.002/5.010/8.775/3.179 ms
+ set +x
~/cbn/boardfarm$

Add option for overlay to build zephyr config dynamically

• build dynamically
• extra info from test run for vars
• upload more logs

Fix printing device info and other debug message at end of boardfarm run

Results run=0 failures=3 skipped=19
[Errno 9] Bad file descriptor
For some reason, could not close a connection.
  board_type: qemux86
  conn_cmd: sudo qemu-system-i386 -m 256 -nographic --enable-kvm -netdev tap,id=wan,ifname=${wan_iface} -device e1000,netdev=wan -netdev tap,id=lan,ifname=${lan_iface} -device e1000,netdev=lan
  connection_type: local_serial
  devices: [{'lan_network': IPv4Network(u'10.0.0.0/24'), u'name': u'wan', 'env': {'lan_iface': 'lan6d3e0e5c4aea', 'wan_iface': 'wan6d3e0e5c4aea', 'uniq_id': '6d3e0e5c4aeac4e'}, u'color': u'cyan', u'cmd': u'docker run --name wan-${uniq_id} --privileged -it bft:node /bin/bash', u'post_cmd_host': u"sudo ip link set netns $(docker inspect --format '{{.State.Pid}}' wan-${uniq_id}) dev ${wan_iface}", 'reboot': False, 'lan_gateway': IPv4Address(u'10.0.0.1'), u'cleanup_cmd': u'docker stop wan-${uniq_id}; docker rm wan-${uniq_id}', u'post_cmd': u'ip link set ${wan_iface} name eth1', u'type': u'debian', u'options': u'tftpd-server', u'pre_cmd_host': u'docker build -t bft:node bft-node'}, {u'name': u'lan', 'env': {'lan_iface': 'lan6d3e0e5c4aea', 'wan_iface': 'wan6d3e0e5c4aea', 'uniq_id': '6d3e0e5c4aeac4e'}, u'color': u'blue', u'cmd': u'docker run --name lan-${uniq_id} --privileged -it bft:node /bin/bash', u'post_cmd_host': u"sudo ip link set netns $(docker inspect --format '{{.State.Pid}}' lan-${uniq_id}) dev ${lan_iface}", 'reboot': False, 'lan_gateway': IPv4Address(u'10.0.0.1'), u'cleanup_cmd': u'docker stop lan-${uniq_id}; docker rm lan-${uniq_id}', u'post_cmd': u'ip link set ${lan_iface} name eth1', u'type': u'debian', 'lan_network': IPv4Network(u'10.0.0.0/24'), u'pre_cmd_host': u'docker build -t bft:node bft-node'}]
  notes: QEMU emulated devices
  station: qemux86-1
({'LanDevPingInternet-result': 'SKIP', 'RootFSBootTest-boot_time': 34.62, '@timestamp': '2018-09-27T15:47:44.000Z', 'build_url': 'http://chippy2.lgirdk.com/jenkins/job/boardfarm/8271/', 'MemoryUse-result': 'SKIP', 'LanDevPingRouter-result': 'SKIP', 'duration': 152, u'connection_type': u'local_serial', 'RandomWebBrowse-result': 'SKIP', 'KernelModules-result': 'SKIP', 'Nmap_LAN-result': 'SKIP', 'hostname': 'chippy2', 'tests_pass': 0, 'tests_fail': 0, 'InterfacesShow-result': 'SKIP', 'station': u'qemux86-1', u'conn_cmd': u'sudo qemu-system-i386 -m 256 -nographic --enable-kvm -netdev tap,id=wan,ifname=${wan_iface} -device e1000,netdev=wan -netdev tap,id=lan,ifname=${lan_iface} -device e1000,netdev=lan', 'ProcVmstat-result': 'SKIP', 'TopCheck-result': 'SKIP', 'DiskUse-result': 'FAIL', 'Nmap_WAN-result': 'SKIP', 'tdkbComponentTestBasic-result': 'SKIP', 'OEVersion-result': 'FAIL', 'LanDevPingWanDev-result': 'SKIP', 'RouterPingInternetName-result': 'SKIP', 'user': 'jenkins', 'unexpected_pass': 0, 'iPerf3RTest-result': 'SKIP', 'iPerf3Test-result': 'SKIP', 'SystemdCheckServices-result': 'SKIP', u'board_type': u'qemux86', 'RouterPingInternet-result': 'SKIP', 'RootFSBootTest-result': 'FAIL', 'RouterPingWanDev-result': 'SKIP', u'notes': u'QEMU emulated devices', 'change_list': 'None', u'devices': [{'lan_network': IPv4Network(u'10.0.0.0/24'), u'name': u'wan', 'env': {'lan_iface': 'lan6d3e0e5c4aea', 'wan_iface': 'wan6d3e0e5c4aea', 'uniq_id': '6d3e0e5c4aeac4e'}, u'color': u'cyan', u'cmd': u'docker run --name wan-${uniq_id} --privileged -it bft:node /bin/bash', u'post_cmd_host': u"sudo ip link set netns $(docker inspect --format '{{.State.Pid}}' wan-${uniq_id}) dev ${wan_iface}", 'reboot': False, 'lan_gateway': IPv4Address(u'10.0.0.1'), u'cleanup_cmd': u'docker stop wan-${uniq_id}; docker rm wan-${uniq_id}', u'post_cmd': u'ip link set ${wan_iface} name eth1', u'type': u'debian', u'options': u'tftpd-server', u'pre_cmd_host': u'docker build -t bft:node bft-node'}, {u'name': u'lan', 'env': {'lan_iface': 'lan6d3e0e5c4aea', 'wan_iface': 'wan6d3e0e5c4aea', 'uniq_id': '6d3e0e5c4aeac4e'}, u'color': u'blue', u'cmd': u'docker run --name lan-${uniq_id} --privileged -it bft:node /bin/bash', u'post_cmd_host': u"sudo ip link set netns $(docker inspect --format '{{.State.Pid}}' lan-${uniq_id}) dev ${lan_iface}", 'reboot': False, 'lan_gateway': IPv4Address(u'10.0.0.1'), u'cleanup_cmd': u'docker stop lan-${uniq_id}; docker rm lan-${uniq_id}', u'post_cmd': u'ip link set ${lan_iface} name eth1', u'type': u'debian', 'lan_network': IPv4Network(u'10.0.0.0/24'), u'pre_cmd_host': u'docker build -t bft:node bft-node'}], 'manifest': 'None', 'Uname-result': 'SKIP', 'test_suite': 'lgirdk', 'apss': 'None', 'tests_total': 22, 'tests_skip': 19, 'unexpected_fail': 3}, TypeError("Unable to serialize IPv4Network(u'10.0.0.0/24') (type: <class 'ipaddress.IPv4Network'>)",))
Unable to store results to elasticsearch_server specified in config. Results stored locally.
Starting Zephyr Execution....
list index out of range
Unable to Update results in Zephyr
Created /local/jenkins_builds/workspace/boardfarm/boardfarm/results/results.html
Running cleanup_cmd on </bin/bash -c docker run --name lan-${uniq_id} --privileged -it bft:node /bin/bash>...cleanup_cmd done.
Running cleanup_cmd on </bin/bash -c docker run --name wan-${uniq_id} --privileged -it bft:node /bin/bash>...cleanup_cmd done.

Make bft work in python3

Investigate the use of Kermit instead of telnet

Investigate the use of Kermit instead of telnet, as it seems that telnet grabs 100% CPU and make the laptop fan loud,

Add test network namespace to LAN container so it can access internet via that namespace and bypass CM

This is useful for installing packages etc after the LAN has been configured to route all traffic through the DUT

How to make a WLAN Docker Container to operate in a particular 802.11 Mode?

In order to execute the WiFi Test Matrix for all combinations, we need to force WLAN Client also to operate in specific 802.11 Mode. But right now via Docker Container, we are not able to achieve this.

@ketantewari @mattsm , Can you please help us on this?

Switch to is ip command for all network commands

visual results in real time

https://github.com/BushnevYuri/DockerGrafanaInfluxKit

We should be able to post realtime results and graph them... this would be in tandem with something ELK

CM Cfg (docsis class) improvements

• download CM cfg from provisioner to import current config
• store all old CM cfgs for looking (for quirks for example)
• generate config from a barebones CM cfg, and start replacing all calls to random CM cfgs to a baseline config + known changes listed in test (ask and can add more details)
• update CM config with site specific config items (ACS, sip, etc. servers)

Allow bft to be run from any directory

There are current a few issues when running boardfarm (bft) command from a different directory than it resides it. This issue is to track that problem and hopefully fix it.

Creating Docker Containers with all Required services installed by default

Hello All,

During Automation, as we dont have dedicated Boardfarms allocated for engieers, they will be using a BF which is free. But the Containers associate to that BF might not have all required services (Example: SNMP) and hence the script fails.

Can we please ensure all the Docker containers have basic services (we can come up with the list of Basic services) during the creation of containers itself?

Thanks,
Pranesh

How to create Test Suite files to perform Robustness Check

For the Automated testcases, we need to perform Robustness Check to ensure the quality of the scripts are good enough to provide it to Manual Test Team.

Please help us how we shall make a Dynamic Test Suite with possible testcases to perform Robustness check on the newly Automated testcases.

Stale telnet using 100% cpu

These seem to be stick around using CPU need to figure out why and kill them.

Test log and result.html should be put into folder with DUT name.

Our test team using SSH to trigger boardfarm to test not using Jenkisn.

If mv1-1-1 and mv1-1-2 are test on same time, it's too hard to search test log which is mv1-1-1 or mv1-1-2
The result.html will be covered by last test case.
Can you help to add new request for test log and result?
ex:
The log and result of mv1-1-1 shoud be put into /results/mv1-1-1/
The log and result of mv1-1-2 shoud be put into /results/mv1-1-2/
Thanks.

How to make a consolidated Log file for the WiFi Matrix Results

Hello All,

This issue is created to track the activities on how to make an consolidated Report as a result of the WiFi Matrix Execution.

At the moment, we are getting log files equal to as many number of Combinations we create which will not be helpful.

Generic CMTS implementation

The docsis most common commands on a CMTS can be at times similar in syntax (across manufacturer) and completely different in output (totally different in layout, info returned, etc).
Within the same brand different CMTS sw versions may or may not support certain commands (although the same information may be obtainable via a sequence of commands)
A common solution needs to be implemented in order to use "any" CMTS. This is to allow the test cases to be as generic and portable as possible.

Support posting all logs in bft run to hastebin type server

Also, add hooks for posting a link elsewhere (gerrit review comments?)

Test running python on remote device nodes

This is one example of a library that will help you do it?

https://pythonhosted.org/Pyro4/

In addition to pexpect and a shell we can try to look at using python directly.

Ideally we could run pysnmp on the WAN node as an example.

Add helpers to get the right device for handling particular tasks

Maybe one core / shell on the device is handling firewall routing, while another is handling bridging..

We should make an abstract way to handle this...

dev1 = get_bridged_dev()
dev2 = get_fw_dev()

dev1.sendline('brctl show')
dev2.sendline('cat /proc/net/nf_conntrack')

Create example boardfarm overlay

Change hard coded dut iface "eth1" to device.dut_iface

If a device attached to the DUT has a DUT facing iface, we should use device.dut_iface instead of "eth1" everywhere.

lan.dut_iface
wan.dut_iface
etc

-M

Make CDrouter use device model

It's a standalone thing in one test now, let's refactor and make it it's own device

Make testsuites.py work for overlays

Split OE and OpenWrt classes

Allow for an overlay to change the default layout of the testlog.txt files

The contents of the attribute log_to_file are written to a <test_case_name>.txt file in in the results directory. This is common across all overlays.
Functionality should be added so that an overlay can override the default layout.

Clean up tftp mess

It's used inconsistently, different things the values from different places

Starting it is confusing... who does what...

ScreenshotGUI from lan side to into web

When Run "./bft -b mv1 -n mv1-2-2 -e ScreenshotGUI -x connect" success.

But lan_portal.png shows "Unable to connect", did you have some advice how to deep to check for that?

Not sure if connection success or not.

Add example Jenkins pipeline groovy script to run a "boardfarm" job

Add option for standalone provisioner and separate WAN devices

Right now we assume these are the same device, should not be much effort to split the two and run a machine for the WAN and a separate machine for the provisioner with the provisioner being shared among multiple devices

Fix html email to show proper ip addr of wan/lan and http proxy

Fix board not found issue

This should print a much better error message:

$ ./bft -n board_not_in_inventory -e lan2_ping_lan
Traceback (most recent call last):
  File "./bft", line 432, in <module>
    main()
  File "./bft", line 88, in main
    config.BOARD_NAMES = sorted(config.BOARD_NAMES, key=lambda x: 'feature' in config.boardfarm_config[x])
  File "./bft", line 88, in <lambda>
    config.BOARD_NAMES = sorted(config.BOARD_NAMES, key=lambda x: 'feature' in config.boardfarm_config[x])
KeyError: 'mv1-4'

Site specific network config

For LAN / WAN to reach the internet- we should apply proxy settings from the global boardfarm config.

And where to fetch files from for upgrading / downgrading images

Not able to view the GUI of Cable Modems

Add function to decide DUT test scope

Due to we have several Boardfarm stations and different function test.

We need to add filter function to parser which DUT runs specific environment.

Creating <Testcase Name>_Failure_Log.txt file ONLY incase of any Testcase Failure

Hello All,

We see that, in the results folder, there is a single file named "Console.log" getting created which dumps all the console output of all executions. This will make the debugging job difficult to see which section is for our particular execution.

Can we think of a way to create a individual log file per testcase, BUT only incase of Failure with name )_Failure_log.txt?

Also, I would like the Automation script to collect as much relative log information as possible incase of Failure so that, we shall avoid Re-run or Manual execution of the scenario. Please let me know your thoughts.

Automation Code Unique Structure Discussion

Hello All,

How about having a unified structure as below for all the Automation going forward?

Shall we discuss some ideas and freeze on a common smart format please?

class TEST_NAME(...)
def testSetup(self):

some setup steps

whatever the test needs

def runTest(self):

do the test

def testCleanUp(self):

cleans the setup, leaves the CM into a known/clean state

this will be called if the test has passed

(uncaught exceptions should be scooped up by recover()

If we fail to parse a test and skip it, show it somehow... but don't stop the test from running

Make bft work on non Linux OSs

Add notion of release images

This would be defined per board, and we would be able to have a test request that image be flashed. Also, it would have a site mirror that contained an http server that hosted the images. So we could go and grab them.