esnet / enos Goto Github PK

ESnet Network Operating System (NOT ACTIVELY MAINTAINED)

License: Other

Java 32.65% Python 66.62% Shell 0.14% HTML 0.59%

enos's Introduction

IMPORTANT NOTE: ENOS IS NO LONGER ACTIVELY MAINTAINED.

enos

The ESnet Network Operating System (ENOS) is a software system that provides a secure execution environment for network functions, as well as some that can be useful in building network control plane applications.

Quickstart

This section documents a set of steps to install ENOS onto an existing host, with OpenDaylight and Corsa SDX3 integration. It is possible, via a slightly different workflow, to get an ENOS deployment with a subset of this functionality, should that not be needed or desired. These instructions are believed to work for Linux (CentOS 7 with Oracle Java 7) and MacOS (Yosemite and El Capitan with Oracle Java 7).

A more comprehensive document, which covers installation of ENOS into a fresh virtual machine, can be found at:

https://docs.google.com/document/d/1_RzUkPTbHVtEnj_dBALiAvT9aHxL1ms6Pw6mdnaBM3c

Installation

The first step is to obtain a suitable Karaf container. If using Corsa SDX3 integration, a customized install of Open Daylight (ODL) is available that contains both ODL and the SDX3 driver (as of this writing, a distribution based on ODL Lithium-SR1 has been successfully tested). Instructions on obtaining the driver distribution are beyond the scope of this document, but it will be assumed that the Karaf container has been obtained and unpacked.
Install Oracle Java 7, Maven, and rabbitmq-server.

Check out both the netshell and enos software repositories from Github:

git clone https://github.com/esnet/netshell
git clone https://github.com/esnet/enos

From the top-level directory of the ODL Karaf distribution, execute the distribution/karaf/fixup-karaf.sh script found in the top-level directory of the netshell source tree. This change restores the default search behavior for finding bundles in Maven repositories (in particular it's needed to read the NetShell bundles from the local Maven repository / cache).
ODL does not play well with the ENOS security manager (the exact conditions are not completely known). To work around this problem, the ENOS security manager must be disabled. Create a file named netshell.json in the top level directory of the Karaf, with these contents:
```
{
    "global": {
	    "securityManagerDisabled":	1
    }
}
```

Create the root directory where ENOS will store its state. By default this is /var/netshell. Also create some initialization files needed for ENOS's Python interpreter.

mkdir /var/netshell
chown $USER /var/netshell     # if needed
mkdir /var/netshell/etc
touch /var/netshell/etc/init.py
cp netshell/distribution/netshell-root/etc/profile.py /var/netshell/etc

Compile and install the netshell and enos sources to the local Maven cache:
```
(cd netshell && mvn install)
(cd enos && mvn install)
```
Start up the ODL Karaf container from the top-level directory of the ODL Karaf installation with bin/karaf.
Within the Karaf instance, load the ODL features of interest, such as OpenFlow support and the DLUX GUI:
```
feature:install odl-dlux-all odl-openflowplugin-all
```
It is possible (but not required) to test the ODL DLUX WebUI by going to the following URL:
```
http://localhost:8181/index.html
```

Features necessary for NetShell integration can be loaded as follows:

feature:install odl-openflowplugin-adsal-compatibility odl-nsf-managers

To make the embedded SSH server start up correctly, it is necessary to refresh the bindings of one of the bundles.
```
bundle:refresh -f org.apache.sshd.core
```
This is necessary so that the org.apache.sshd.core contains correct bindings for the org.apache.mina.service package. These bindings are necessary for NetShell's embedded SSH server; failure to get this right results in a a runtime exception at NetShell startup time.

Execute the following command to make the NetShell feature repository available:

feature:repo-add mvn:net.es.netshell/netshell-features/1.0.0-SNAPSHOT/xml/features

Execute the following commands as applicable to start NetShell for the first time:
```
feature:install netshell-kernel
feature:install netshell-python
```
The feature installation of netshell-python may generate some exceptions and warnings, which can (probably) be ignored.

To load the NetShell OpenDaylight MD-SAL bundles:

feature:install netshell-odl-mdsal netshell-odl-corsa-intf
feature:install netshell-odl-corsa
feature:install netshell-controller

Add the feature repository and install the feature:

feature:repo-add mvn:net.es.enos/enos-features/1.0.0-SNAPSHOT/xml/features

feature:install enos-esnet

Optionally, execute the following commands (needed the first time only) to initialize the ESnet topology. Future runs will have this topology cached:
```
python

from net.es.enos.esnet import ESnetTopology
ESnetTopology.registerToFactory()
```

Simulated ESnet 100G SDN testbed using Mininet

(NOTE: Instructions in this section have not been tested recently.)

ESnet 100G SDN testbed can be simulated within a virtual machine using Mininet:

1, Download the latest Mininet version from http://mininet.org/download/

Run and configure the Mininet VM making but do not run the walkthrough tutorial: it would create persistent virtual switches that would interfere with the overall simlation.
From the ENOS esnet/src/main/python/mininet/ directory, copy testbed.py and run.py into the mininet VM. Configure testbed.py to point to the IP/port address of OpendayLight. (this will be improved to use an option in the future)
On the mininet VM, run "sudo python run.py". This script will create the 8 SDN pops of the testbed, including ESnet core routers, SDN physical switches, OVS and service VM. Note that the topology is currently hard coded in the first lines of testbed.py.

Copyright

ESnet Network Operating System (ENOS) Copyright (c) 2015, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved.

If you have questions about your rights to use or distribute this software, please contact Berkeley Lab's Innovation & Partnerships Office at [email protected].

NOTICE. This Software was developed under funding from the U.S. Department of Energy and the U.S. Government consequently retains certain rights. As such, the U.S. Government has been granted for itself and others acting on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in the Software to reproduce, distribute copies to the public, prepare derivative works, and perform publicly and display publicly, and to permit other to do so.

enos's People

Contributors

Stargazers

Watchers

Forkers

modulexcite cygmris syllogism3rd

enos's Issues

OSCARS Layer 2 service logic

Add service logic into ENOS for gluing shell commands/application API into pathfinding and OSCARS provisioning operations.

Java API for Multipoint VPN application

Requires investigation java using a python API.

Postpone till later: Not needed for testbed.

Complaints when pinging

The following complaints appeared on Python's stdout after setting up a two-node MP-VPN between lbl-diskpt1 and bnl-diskpt1 with these commands:

vpn create vpn1
vpn vpn1 addpop denv
vpn vpn1 addpop aofa
vpn vpn1 addsite lbl.gov
vpn vpn1 addsite bnl.gov
vpn vpn1 addhost [email protected]
vpn vpn1 addhost [email protected]

These seem to be caused by the ARP packets from lbl-diskpt1.

admin@NetShell> port name 24 type HwToCore
key 24.1994
vpn1.denv-tb-of-1
port name 24 type HwToCore
key 24.1994
untapBroadcastEntry with k {mac:FF:FF:FF:FF:FF:FF,vlan:1994,port:24}
not in index
[410]2015-10-30 14:49:40,444-L2SwitchScope-WARNING-PacketOut(scope=vpn1.denv-tb-of-1,port=Port(name=2),src=00:60:DD:44:2B:2C,dst=FF:FF:FF:FF:FF:FF,etherType=0,vlan=1994) is not within this scope vpn1.denv-tb-of-1
[410]2015-10-30 14:49:40,466-L2SwitchScope-WARNING-PacketOut(scope=vpn1.denv-tb-of-1,port=Port(name=2),src=00:60:DD:44:2B:2C,dst=FF:FF:FF:FF:FF:FF,etherType=0,vlan=1994) is not valid in vpn1.denv-tb-of-1
port name eth11 type SwToHw.WAN
key eth11.16777215
[410]2015-10-30 14:49:41,365-SimpleController-WARNING-(eth11, 1994, FF:FF:FF:FF:FF:FF) not found in org.python.proxies.layer2.odl.client$ODLClient$245@56120417.scopeIndex
None
[410]2015-10-30 14:49:41,378-SimpleController-WARNING-No scope for PacketInEvent(inPort=Port(name=eth11),srcMac=00:60:DD:44:2B:2C,dstMac=FF:FF:FF:FF:FF:FF,vlan=1994)
port name eth11 type SwToHw.WAN
key eth11.16777215
[410]2015-10-30 14:49:42,364-SimpleController-WARNING-(eth11, 1994, FF:FF:FF:FF:FF:FF) not found in org.python.proxies.layer2.odl.client$ODLClient$245@56120417.scopeIndex
None
[410]2015-10-30 14:49:42,381-SimpleController-WARNING-No scope for PacketInEvent(inPort=Port(name=eth11),srcMac=00:60:DD:44:2B:2C,dstMac=FF:FF:FF:FF:FF:FF,vlan=1994)

It looks like we're trying to do a PACKET_OUT on port 2 of denv-tb-of-1. I'm wondering if we should be trying port 1 instead (both go to denv-ovs but the link using port 1 is of type "hw" and the link using port 2 is of type "none").

OSCARS support

This epic covers discovery of OSCARS topology, provisioning of OSCARS circuits, and the inventory of OSCARS circuits as topology.

a call to python sys.exit() cause the whole JVM to quit

When an ENOS python program calls sys.exit() to quit, this cause not only the program to quit, but the whole JVM.

Don't show login banner on non-interactive ssh login

On UNIX, an sshd login prints the motd (etc.) for an interactive shell login, but not if you're invoking sshd to run a command. IIRC this functionality is implemented in the back-end sshd daemon.

enos should do something similar, so if invoke sshd to run say a python script within enos, I don't get the "Welcome to enos" banner in my script's output.

LXC: Creating containers

SSH doesn't write hostkey properly

On the first startup of ENOS on a fresh install, ${ENOS_ROOT}/etc doesn't exist. This prevents sshd from writing a new host key at startup time:

83688 [pool-2-thread-1] INFO org.apache.sshd.server.session.ServerSession - Server session created from /0:0:0:0:0:0:0:1:55816
83690 [pool-2-thread-1] INFO org.apache.sshd.server.keyprovider.SimpleGeneratorHostKeyProvider - Generating host key...
83693 [pool-2-thread-1] INFO net.es.enos.kernel.security.KernelSecurityManager - checkWrite allows /var/enos/etc/hostkey.ser because thread is privileged
83693 [pool-2-thread-1] WARN org.apache.sshd.server.keyprovider.SimpleGeneratorHostKeyProvider - Unable to write key /var/enos/etc/hostkey.ser: java.io.FileNotFoundException: /var/enos/etc/hostkey.ser (No such file or directory)
83695 [pool-2-thread-1] INFO org.apache.sshd.server.session.ServerSession - Kex: server->client aes128-ctr hmac-md5 none
83695 [pool-2-thread-1] INFO org.apache.sshd.server.session.ServerSession - Kex: client->server aes128-ctr hmac-md5 none
83697 [pool-2-thread-1] INFO net.es.enos.kernel.security.KernelSecurityManager - checkRead ok /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/jre/lib/ext/sunec.jar not an ENOS file. Rely on system access
83697 [pool-2-thread-1] INFO net.es.enos.kernel.security.KernelSecurityManager - checkRead ok /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/jre/lib/ext/sunec.jar not an ENOS file. Rely on system access
100899 [pool-2-thread-3] INFO org.apache.sshd.server.session.ServerSession - Server session created from /0:0:0:0:0:0:0:1:55819
100902 [pool-2-thread-4] INFO org.apache.sshd.server.session.ServerSession - Kex: server->client aes128-ctr hmac-md5 none
100902 [pool-2-thread-4] INFO org.apache.sshd.server.session.ServerSession - Kex: client->server aes128-ctr hmac-md5 none

Despite this problem, ENOS continues to operate fairly normally. On the first user login, ${ENOS_ROOT}/etc gets created. ENOS will continue using the same (unsaved) host key until ENOS is stopped and restarted.

On the second bootup, ENOS again fails to find a saved host key and generates a different one than before. This time, ${ENOS_ROOT}/etc exists, so it can save the key, and all works well after any users who logged in on the first boot clear out the first host key from their SSH known_hosts file.

Filing this minor but annoying bug so I don't forget to fix it. Should be pretty easy, just need to make sure that ${ENOS_ROOT}/etc exists before we initialize SSH.

Audit hard-coded passwords

The class net.es.enos.api.SSHConnection has hard-coded usernames and passwords. These need to be audited and/or/removed.

perfSONAR Agent

Purpose: detection of hard and software failure on an within a VPN / OSCARS circuit.

ENOS management of Linux Kernel virtual machines (KVM) and Linux containers (LXC)

Feature will utilize ProxMox VE (https://www.proxmox.com/en/proxmox-ve) to dynamically create, delete, and generally manage Linux-based containers. This feature is required for NFV support, however, before this, it'll be used to create containers for user controllers, fake hosts, etc. Also required for site management.

This task should wait until deployment of proxmox on the testbed if complete.

Design: Intelligent MPVPN circuit routing

Create design documentation for feature.

Default admin user doesn't get a default container

When we initialize the default admin user in do_authUser(), we only create a new profile for them. This was sufficient when all that was needed was an entry in the password file, but recent code changes have added quite a few other actions to do_createUser(), including creating a default container for a user, ACLs, etc.

We need to make creation of the default admin user use do_createUser() so that these other actions get done as well.

cd / does not work

cd / seems to remain in the same directory

OSCARS topology integration

Build a topology adaptor to import layer 2 OSCARS topology into ENOS.

Unprivileged user cannot login

Steps to reproduce:

Start enos
Attempt to log-in as an unprivileged user with correct password.

Expected result: User will login successfully.
Actual result: enos will display a NullPointerException and the user shell will not allow any further inputs.

Possible cause:
With this addition in KernelSecurityManager (commit 82627ed)...:

public void checkExec(String cmd) {
        if (KernelThread.getCurrentKernelThread().isPrivileged()) {
            return;
        }
        throw new ExitSecurityException("Cannot execute UNIX processes");

An unprivileged user will be thrown the exception and the startup sequence cannot continue. A privileged user (such as admin) will still be able to login successfully.

Traceback as follows:

net.es.enos.kernel.security.ExitSecurityException: Cannot execute UNIX processes
    at net.es.enos.kernel.security.KernelSecurityManager.checkExec(KernelSecurityManager.java:292)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1016)
    at java.lang.Runtime.exec(Runtime.java:617)
    at java.lang.Runtime.exec(Runtime.java:485)
    at jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:183)
    at jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:173)
    at jline.internal.TerminalLineSettings.stty(TerminalLineSettings.java:168)
    at jline.internal.TerminalLineSettings.get(TerminalLineSettings.java:72)
    at jline.internal.TerminalLineSettings.<init>(TerminalLineSettings.java:52)
    at jline.UnixTerminal.<init>(UnixTerminal.java:31)
    at net.es.enos.shell.Shell.startShell(Shell.java:99)
    at net.es.enos.sshd.SshdShell.run(SshdShell.java:119)
    at java.lang.Thread.run(Thread.java:745)
Exception in thread "ENOS Shell User= user1" java.lang.NullPointerException
    at net.es.enos.shell.Shell.startShell(Shell.java:119)
    at net.es.enos.sshd.SshdShell.run(SshdShell.java:119)
    at java.lang.Thread.run(Thread.java:745)

Provide an OSCARS Java API compatible with Netshell types

Mostly data type translation/adaption.

MPVPN: install broadcast flows

LXC: Container management tasks

Model OSCARS circuits as topology

Model any created/discovered OSCARS circuits as topology within the ENOS topology service.

End-user controlled OVS: Design abstractions / APIs

Write design documentation describing how to: Create OVS instances for users, attach them to hardware switches.

Intelligent MPVPN circuit routing (aka Scale up to full testbed implementation)

Instead of creating a mesh of OSCARS circuits, create circuits along the physical topology and implement forwarding across multiple POP's.

Stretch goal: this could slip.

Don't put ENOS root directory under /tmp

The ENOS root directory is defined, by default, as /tmp/enos. However, on some operating systems (MacOS in particular), /tmp gets cleared on a reboot. On some other operating systems, it's common practice for /tmp to be a RAM disk for increased performance (which has a similar effect).

Losing the contents of the ENOS root directory is a Bad Thing (TM).

We should pick a better place for this.

SSH session hang for unprivileged user

If the first login session to enos is by an unprivileged user, the login session hangs after printing the banner and before the command-line prompt.

Server-side stack traces as follows:

Setting ENOS_ROOTDIR to /Users/bmah/IdeaProjects/enos/base
Setting ENOS_HOME to /Users/bmah/IdeaProjects/enos/base
JarClassLoader: Warning: jline/Terminal.class in lib/jython-standalone-2.5.2.jar is hidden by lib/jline-2.11.jar (with different bytecode)
JarClassLoader: Warning: jline/UnixTerminal.class in lib/jython-standalone-2.5.2.jar is hidden by lib/jline-2.11.jar (with different bytecode)
JarClassLoader: Warning: jline/UnsupportedTerminal.class in lib/jython-standalone-2.5.2.jar is hidden by lib/jline-2.11.jar (with different bytecode)
JarClassLoader: Warning: jline/WindowsTerminal$1.class in lib/jython-standalone-2.5.2.jar is hidden by lib/jline-2.11.jar (with different bytecode)
JarClassLoader: Warning: jline/WindowsTerminal.class in lib/jython-standalone-2.5.2.jar is hidden by lib/jline-2.11.jar (with different bytecode)
JarClassLoader: Warning: Null manifest from input stream associated with: lib/maven-toolchain-1.0.jar
JarClassLoader: Warning: org/apache/commons/logging/Log.class in lib/commons-logging-api-1.1.jar is hidden by lib/commons-logging-1.0.4.jar (with different bytecode)
JarClassLoader: Warning: org/apache/commons/logging/LogConfigurationException.class in lib/commons-logging-api-1.1.jar is hidden by lib/commons-logging-1.0.4.jar (with different bytecode)
JarClassLoader: Warning: org/apache/commons/logging/LogFactory$1.class in lib/commons-logging-api-1.1.jar is hidden by lib/commons-logging-1.0.4.jar (with different bytecode)
JarClassLoader: Warning: org/apache/commons/logging/LogFactory$2.class in lib/commons-logging-api-1.1.jar is hidden by lib/commons-logging-1.0.4.jar (with different bytecode)
JarClassLoader: Warning: org/apache/commons/logging/LogFactory$3.class in lib/commons-logging-api-1.1.jar is hidden by lib/commons-logging-1.0.4.jar (with different bytecode)
JarClassLoader: Warning: org/apache/commons/logging/LogFactory.class in lib/commons-logging-api-1.1.jar is hidden by lib/commons-logging-1.0.4.jar (with different bytecode)
JarClassLoader: Warning: org/apache/commons/logging/LogSource.class in lib/commons-logging-api-1.1.jar is hidden by lib/commons-logging-1.0.4.jar (with different bytecode)
JarClassLoader: Warning: org/apache/commons/logging/impl/Jdk14Logger.class in lib/commons-logging-api-1.1.jar is hidden by lib/commons-logging-1.0.4.jar (with different bytecode)
JarClassLoader: Warning: org/apache/commons/logging/impl/LogFactoryImpl.class in lib/commons-logging-api-1.1.jar is hidden by lib/commons-logging-1.0.4.jar (with different bytecode)
JarClassLoader: Warning: org/apache/commons/logging/impl/NoOpLog.class in lib/commons-logging-api-1.1.jar is hidden by lib/commons-logging-1.0.4.jar (with different bytecode)
JarClassLoader: Warning: org/apache/commons/logging/impl/SimpleLog$1.class in lib/commons-logging-api-1.1.jar is hidden by lib/commons-logging-1.0.4.jar (with different bytecode)
JarClassLoader: Warning: org/apache/commons/logging/impl/SimpleLog.class in lib/commons-logging-api-1.1.jar is hidden by lib/commons-logging-1.0.4.jar (with different bytecode)
JarClassLoader: Warning: com/google/common/collect/package-info.class in lib/google-collections-1.0.jar is hidden by lib/jython-standalone-2.5.2.jar (with different bytecode)
0 [main] INFO net.es.enos.boot.BootStrap - Starting ENOS
0 [main] INFO net.es.enos.boot.BootStrap - Starting BootStrap thread
1 [ENOS Bootstrap] INFO net.es.enos.boot.BootStrap - Starting services
33 [ENOS Bootstrap] INFO org.apache.sshd.common.util.SecurityUtils - BouncyCastle not registered, using the default JCE provider
387 [ENOS Bootstrap] INFO net.es.enos.shell.ShellCommandsFactory - Adding Shell command help
387 [ENOS Bootstrap] INFO net.es.enos.shell.ShellCommandsFactory - Adding Shell command exit
388 [ENOS Bootstrap] INFO net.es.enos.shell.ShellCommandsFactory - Adding Shell command test
389 [ENOS Bootstrap] INFO net.es.enos.shell.ShellCommandsFactory - Adding Shell command python
389 [ENOS Bootstrap] INFO net.es.enos.shell.ShellCommandsFactory - Adding Shell command adduser
389 [ENOS Bootstrap] INFO net.es.enos.shell.ShellCommandsFactory - Adding Shell command passwd
3361 [pool-2-thread-1] INFO org.apache.sshd.server.session.ServerSession - Session created from /0:0:0:0:0:0:0:1:52609
4690 [pool-2-thread-5] WARN net.es.enos.kernel.users.Users - authUser entry
4690 [pool-2-thread-5] WARN net.es.enos.kernel.users.Users - looking for key for bmah2
4804 [pool-2-thread-5] WARN net.es.enos.kernel.users.Users - bmah2 has entered correct password
4805 [pool-2-thread-5] INFO org.apache.sshd.server.session.ServerSession - Session bmah2@/0:0:0:0:0:0:0:1:52609 authenticated
Shell Starting
Exception in thread "ENOS Shell User= bmah2" java.lang.ExceptionInInitializerError
    at net.es.enos.shell.Shell.run(Shell.java:104)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.MissingResourceException: Can't find bundle for base name jline.console.completer.CandidateListCompletionHandler, locale en_US
    at java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:1499)
    at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1322)
    at java.util.ResourceBundle.getBundle(ResourceBundle.java:721)
    at jline.console.ConsoleReader.<clinit>(ConsoleReader.java:93)
    ... 2 more
Caused by: java.lang.ClassCastException: jline.console.completer.CandidateListCompletionHandler cannot be cast to ResourceBundle
    at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2574)
    at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1436)
    at java.util.ResourceBundle.findBundle(ResourceBundle.java:1400)
    at java.util.ResourceBundle.findBundle(ResourceBundle.java:1354)
    at java.util.ResourceBundle.findBundle(ResourceBundle.java:1354)
    at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1296)
    ... 4 more

Complete core MP-VPN functionality

This includes ARP/broadcast and MAC learning support, ODL integration for both OVS and Corsa switches.

MPVPN: install unicast flows

Make MP-VPN use epipes

For a number of reasons, we need to make the MP-VPN feature use epipes instead of pre-configured OSCARS circuits. This is primarily because OSCARS is implemented on VPLS, and contains some MAC learning and other LAN emulation features that do not interact well with the MP-VPN MAC address translation that is needed to ensure isolation between multiple VPNs.

The practical implication is VPNs running over OSCARS circuits will experience degraded connectivity (due to MAC learning timeouts) and throughput (due to flooding bandwidth limits). These problems were first seen during the SC15 demo prep and resulting troubleshooting.

This issue depends on #42.

MPVPN path computation engine

Extended existing path computation engine to handle intelligent routing of links utilized for a MPVPN instance.

LXC: Dynamic Linux container control investigation and design

Investigate ProxMox programmable API. Design ENOS management of Linux Kernel virtual machines (KVM) and Linux containers (LXC).

No STDERR output when a python program fails

This is only when executing a program in non interactive mode, i.e.

lomax@enos> python test.py

End-user controlled OVS: Implementation and designer test

This activity covers the implementation, unit test writing, designer test, and documentation of the End-user controlled OVS feature.

Inconsistent symlink resolution

In some places related to the FileACL class, symlinks in pathnames are sometimes resolved and sometimes not. This is easy to observe on MacOS because /tmp is a symbolic link to /private/tmp. It causes some problems when trying to compare pathnames (in particular to see if a given pathname is within the ENOS root directory hierarchy or not).

Assigning to @davidhua for preliminary investigation.

MPVN CLI (slight) revamp / simplication

Description:
This task is not purely required for either SENSE or testbed opening.

Netshell support for CORSA CLI (for QoS)

This is required in order to support testbed researchers who do need access to the Corsa CLI.

Note that CLI changed recently, and may change again. May want to delay this.

adduser does not error with unprivileged account

user1@enos> adduser user2 123 user

student2:enos davidhua$ ls users
admin   user1   user2
student2:enos davidhua$ cat etc/enos.users
user2:$6$76n9.pVm$GvyXYtoa.4NHpQGlZduOpLpmKt5xLwRxS9JXNyLfKAkR1qS6d6jZFA1lpdfLF.VZd2cKjGgwfuBnLOOt2.1wO1:user
admin:$6$uYcVQ/dk$XZM6aobum8wFc/etdcawiRqYZI4f6ctulJrtlTULLPAE/KytbrFYfcqHHg9fCPxofPrEBtsc6RI0T2Wf33EHw1:root
user1:$6$L4yxQeoZ$tLoIGpDyiNn55x0FIq9iRhGzx4ojeviK734iR.H0ayywEQliEbdjZKm9KkhQTLa6X/fbUUWgQgkH0YZMtWdav0:user

Problems when deleting a VPN site

admin@NetShell> reload 

admin@NetShell> demo 
VPN environment is ready.

admin@NetShell> vpn create vpn1
VPN vpn1 is created successfully.

admin@NetShell> vpn vpn1 addpop denv
Pop denv is added into VPN vpn1 successfully.

admin@NetShell> vpn vpn1 addpop aofa
Pop aofa is added into VPN vpn1 successfully.

admin@NetShell> vpn vpn1 addsite lbl.gov
The site lbl.gov is added into VPN vpn1 successfully

admin@NetShell> vpn vpn1 addsite bnl.gov
The site bnl.gov is added into VPN vpn1 successfully

admin@NetShell> vpn vpn1 delsite bnl.gov 
Traceback (most recent call last):
  File "/home/enos/distro/netshell-root/lib/layer2/vpn/vpn.py", line 398, in <module>
    main()
  File "/home/enos/distro/netshell-root/lib/layer2/vpn/vpn.py", line 354, in main
    delsite(vpn, site)
  File "/home/enos/distro/netshell-root/lib/layer2/vpn/vpn.py", line 276, in delsite
    popsRenderer.delSite(site)
  File "/home/enos/distro/netshell-root/lib/layer2/vpn/l2vpn.py", line 840, in delSite
    swSwitchScope.delEndpoint(swSwitch.props['vmPort'], siteVlan)
  File "/home/enos/distro/netshell-root/lib/layer2/common/openflow.py", line 696, in delEndpoint
    if not port.name in self.props['endpoints']:
AttributeError: 'NoneType' object has no attribute 'name'
Traceback (most recent call last):
  File "/home/enos/distro/netshell-root/lib/layer2/vpn/vpn.py", line 398, in <module>
    main()
  File "/home/enos/distro/netshell-root/lib/layer2/vpn/vpn.py", line 354, in main
    delsite(vpn, site)
  File "/home/enos/distro/netshell-root/lib/layer2/vpn/vpn.py", line 276, in delsite
    popsRenderer.delSite(site)
  File "/home/enos/distro/netshell-root/lib/layer2/vpn/l2vpn.py", line 840, in delSite
    swSwitchScope.delEndpoint(swSwitch.props['vmPort'], siteVlan)
  File "/home/enos/distro/netshell-root/lib/layer2/common/openflow.py", line 696, in delEndpoint
    if not port.name in self.props['endpoints']:
AttributeError: 'NoneType' object has no attribute 'name'

Race condition with finding OdlMdsalImpl and OdlCorsaImpl

There can be a race condition where netshell-controller comes up before netshell-odl-mdsal and netshell-odl-corsa. If this happens, the Controller class might try to retrieve (and cache) the OdlMdsalImpl and OdlCorsaImpl singleton objects before they are created, which is bad.

There is a workaround, which is to call Controller.getInstance().reinit() to re-fetch the objects, but that's a manual hack.

It's also possible in some cases that we might not have OdlCorsalImpl if that module is never loaded.

We should try to handle this better. We should probably have all code use the getters in Controller for finding the other objects, and no downstream consumer of this (currently just client.py I think) should cache these directly. The getters might be able to cache non-null values for performance, but it'll always be correct to ask the classes for the instances.

Source files containing code that's affected by this: Controller.java, client.py.

Peer testing: MVPN feature testing

MP-VPN Multiplexer (aka The Slicer)

Resource partitioning provides virtualized resource views on a per "user" basis.

LXC: Deleting containers

Cannot create privileged threads after an un-privileged thread ran

Scenario to reproduce:

log as normal user
log out
log as admin or other privileged user: the thread will be unprivileged.

The problem is that Shell's threads come from a thread pool and after running for a user, it needs to be reset.

Integrate OSCARS Java Client

Package OSCARS Java Client into ENOS bundle as a new OSGi service. Add appropriate configuration support allow for provisioning of remote OSCARS instance.

SSH session crash for unprivileged user

During the login of an unprivileged user, the enos backend can sometimes generate an exception related to permissions. This doesn't happen all of the time. Back-end stack trace:

163434 [pool-2-thread-5] WARN net.es.enos.kernel.users.Users - authUser entry
163435 [pool-2-thread-5] WARN net.es.enos.kernel.users.Users - looking for key for bmah2
163450 [pool-2-thread-5] WARN net.es.enos.kernel.users.Users - bmah2 has entered correct password
163451 [pool-2-thread-5] INFO org.apache.sshd.server.session.ServerSession - Session bmah2@/0:0:0:0:0:0:0:1:53058 authenticated
163457 [pool-2-thread-7] WARN org.apache.sshd.server.session.ServerSession - Exception caught
java.lang.SecurityException: Illegal Thread access from pool-2-thread-7 onto ENOS Shell User= bmah2
    at net.es.enos.kernel.security.KernelSecurityManager.checkAccess(KernelSecurityManager.java:75)
    at java.lang.Thread.checkAccess(Thread.java:1418)
    at java.lang.Thread.setPriority(Thread.java:1122)
    at java.lang.Thread.init(Thread.java:413)
    at java.lang.Thread.init(Thread.java:349)
    at java.lang.Thread.<init>(Thread.java:595)
    at net.es.enos.sshd.SshdShell.start(SshdShell.java:78)
    at org.apache.sshd.server.channel.ChannelSession.handleShell(ChannelSession.java:433)
    at org.apache.sshd.server.channel.ChannelSession.handleRequest(ChannelSession.java:284)
    at org.apache.sshd.server.channel.ChannelSession.handleRequest(ChannelSession.java:238)
    at org.apache.sshd.common.session.AbstractSession.channelRequest(AbstractSession.java:1083)
    at org.apache.sshd.server.session.ServerSession.running(ServerSession.java:240)
    at org.apache.sshd.server.session.ServerSession.handleMessage(ServerSession.java:216)
    at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:587)
    at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:253)
    at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54)
    at org.apache.sshd.common.io.nio2.Nio2Session$2.completed(Nio2Session.java:231)
    at org.apache.sshd.common.io.nio2.Nio2Session$2.completed(Nio2Session.java:217)
    at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
    at sun.nio.ch.Invoker.invokeDirect(Invoker.java:145)
    at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
    at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:275)
    at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:296)
    at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:407)
    at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:217)
    at org.apache.sshd.common.io.nio2.Nio2Session$2.completed(Nio2Session.java:232)
    at org.apache.sshd.common.io.nio2.Nio2Session$2.completed(Nio2Session.java:217)
    at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
    at sun.nio.ch.Invoker$2.run(Invoker.java:206)
    at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
163459 [pool-2-thread-7] INFO org.apache.sshd.server.session.ServerSession - Session bmah2@/0:0:0:0:0:0:0:1:53058 closed

LXC: Linux container ACL

Increase ssh session timeouts

10 minutes idle session timeout is too small. We should either make this longer or configurable or both.

Support for end user controlled OVS

Create OVS instances for users, attach them to hardware switches. This supports OpenFlow experiments on the SDN testbed, where we don't want to give (too much) Corsa access. This task, automation, should wait until the deployment of proxmox is done or planned.

Implement persistent storage to store tests.
Daemon/background timer thread that runs the tests.