refactor noports code for modularity, readability and testability

[gkc]

refactor noports code for modularity, readability and testability

[gkc]

@XavierChanth sgtm

[gkc]

This is in progress but is slow going. sshnp is almost complete after about 6 hours effort with a couple of hours remaining. sshnpd will take less time, maybe 4 hours, and sshrvd is simpler again, estimate 3 hours . About 1.5 days effort to complete the refactoring I guess

Then a round of reviews and testing, probably will combine to around 1.5 days

[JeremyTubongbanua]

cc @gkc - related to #201

I use 2 of my own docker containers. I compile two sets of binaries in each container (bin/ which holds compiled binaries from trunk and binrefactor/ which holds the compiled binaries from gkc-refactor-sshnp)

Video (2 minuts) of me reproducing the bug: https://drive.google.com/file/d/1wRjl1Bt4VS9W5FdL2BiueqR0wBK7Yqhz/view?usp=sharing

What I use to test: https://github.com/JeremyTubongbanua/sshnp_docker

sshnp refactored (error)

atsign@f8ce464850b1:~/.local/binrefactor$ ./sshnp -f @jeremy_0 -t @smoothalligator -d docker -h @rv_am -s id_ed25519.pub -v
INFO|2023-06-13 18:38:30.360114|AtClientManager|setCurrentAtSign called with atSign @jeremy_0 

INFO|2023-06-13 18:38:30.360214|AtClientManager|Switching atSigns from null to @jeremy_0 

INFO|2023-06-13 18:38:30.364314|HiveBase|commit_log_3c074c7d3712aa56c1d3e81561c463442f282dfda4ddcc62a724ea0844c2f0e0 initialized successfully 

INFO|2023-06-13 18:38:30.370008|HiveBase|3c074c7d3712aa56c1d3e81561c463442f282dfda4ddcc62a724ea0844c2f0e0 initialized successfully 

INFO|2023-06-13 18:38:30.370107|AtClientCommitLogCompaction (@jeremy_0)|Starting commit log compaction job running for every 11 minute(s) 

INFO|2023-06-13 18:38:30.372038|AtClientManager|setCurrentAtSign complete 

INFO|2023-06-13 18:38:30.372217|AtLookup|Creating new connection 

INFO|2023-06-13 18:38:30.731057|AtLookup|New connection created OK 

INFO|2023-06-13 18:38:30.878335|AtLookup|auth success 

INFO|2023-06-13 18:38:30.886763| sshnp |Subscribing to notifications on 43a9d9b6-c17b-46cd-8140-77ea3a50961b.docker.sshnp@ 

INFO|2023-06-13 18:38:31.133374|Monitor (@jeremy_0)|monitor started for @jeremy_0 with last notification time: null 

INFO|2023-06-13 18:38:31.200145|AbstractAtKeyEncryption (@jeremy_0)|Encrypted shared symmetric key for @jeremy_0 not found in local storage 

INFO|2023-06-13 18:38:31.200283|AbstractAtKeyEncryption (@jeremy_0)|Deleting @rv_am:shared_key@jeremy_0 from LocalSecondary 

INFO|2023-06-13 18:38:31.200713|AbstractAtKeyEncryption (@jeremy_0)|Fetching shared symmetric key for @jeremy_0 from atServer 

INFO|2023-06-13 18:38:31.260553|AbstractAtKeyEncryption (@jeremy_0)|Retrieved my encrypted copy of shared symmetric key for @rv_am from atServer - saving to local storage 

INFO|2023-06-13 18:38:31.269115|AbstractAtKeyEncryption (@jeremy_0)|'Their' copy of shared symmetric key for @rv_am not found in local storage - will check atServer 

INFO|2023-06-13 18:38:31.329979|AbstractAtKeyEncryption (@jeremy_0)|Found 'their' copy of shared symmetric key for @rv_am in atServer - saving to local storage 

INFO|2023-06-13 18:38:32.029133| sshnp |SUCCESS:id: a0200e24-3fa9-49cc-b1ea-ed39d5aa0770 status: NotificationStatusEnum.delivered  

INFO|2023-06-13 18:38:42.338449| sshnp |Tidying up files 

sshnp: connection timeout to sshrvd @rv_am service
atsign@f8ce464850b1:~/.local/binrefactor$ ./sshnp -f @jeremy_0 -t @smoothalligator -d docker -h @rv_am -s id_ed25519.pub -v
INFO|2023-06-13 18:38:45.890613|AtClientManager|setCurrentAtSign called with atSign @jeremy_0 

INFO|2023-06-13 18:38:45.890698|AtClientManager|Switching atSigns from null to @jeremy_0 

INFO|2023-06-13 18:38:45.895231|HiveBase|commit_log_3c074c7d3712aa56c1d3e81561c463442f282dfda4ddcc62a724ea0844c2f0e0 initialized successfully 

INFO|2023-06-13 18:38:45.900072|HiveBase|3c074c7d3712aa56c1d3e81561c463442f282dfda4ddcc62a724ea0844c2f0e0 initialized successfully 

INFO|2023-06-13 18:38:45.900183|AtClientCommitLogCompaction (@jeremy_0)|Starting commit log compaction job running for every 11 minute(s) 

INFO|2023-06-13 18:38:45.901743|AtClientManager|setCurrentAtSign complete 

INFO|2023-06-13 18:38:45.901852|AtLookup|Creating new connection 

INFO|2023-06-13 18:38:46.261364|AtLookup|New connection created OK 

INFO|2023-06-13 18:38:46.406355|AtLookup|auth success 

INFO|2023-06-13 18:38:46.414874| sshnp |Subscribing to notifications on 4c2d5c53-8236-4db6-b365-c0e642538830.docker.sshnp@ 

INFO|2023-06-13 18:38:46.696385|Monitor (@jeremy_0)|monitor started for @jeremy_0 with last notification time: null 

INFO|2023-06-13 18:38:46.748652|AbstractAtKeyEncryption (@jeremy_0)|Encrypted shared symmetric key for @jeremy_0 not found in local storage 

INFO|2023-06-13 18:38:46.748740|AbstractAtKeyEncryption (@jeremy_0)|Deleting @rv_am:shared_key@jeremy_0 from LocalSecondary 

INFO|2023-06-13 18:38:46.749183|AbstractAtKeyEncryption (@jeremy_0)|Fetching shared symmetric key for @jeremy_0 from atServer 

INFO|2023-06-13 18:38:46.805840|AbstractAtKeyEncryption (@jeremy_0)|Retrieved my encrypted copy of shared symmetric key for @rv_am from atServer - saving to local storage 

INFO|2023-06-13 18:38:46.812015|AbstractAtKeyEncryption (@jeremy_0)|'Their' copy of shared symmetric key for @rv_am not found in local storage - will check atServer 

INFO|2023-06-13 18:38:46.868772|AbstractAtKeyEncryption (@jeremy_0)|Found 'their' copy of shared symmetric key for @rv_am in atServer - saving to local storage 

INFO|2023-06-13 18:38:47.553470| sshnp |SUCCESS:id: ed6fd0e7-4986-459b-b89b-aeb2b335e32b status: NotificationStatusEnum.delivered  

INFO|2023-06-13 18:38:57.824368| sshnp |Tidying up files 

sshnp: connection timeout to sshrvd @rv_am service

sshnp trunk (working)

atsign@f8ce464850b1:~$ ~/.local/bin/sshnp -f @jeremy_0 -t @smoothalligator -d docker -h @rv_am -s id_ed25519.pub -v
INFO|2023-06-13 18:37:51.317767|AtClientManager|setCurrentAtSign called with atSign @jeremy_0 

INFO|2023-06-13 18:37:51.317906|AtClientManager|Switching atSigns from null to @jeremy_0 

INFO|2023-06-13 18:37:51.349300|HiveBase|commit_log_3c074c7d3712aa56c1d3e81561c463442f282dfda4ddcc62a724ea0844c2f0e0 initialized successfully 

INFO|2023-06-13 18:37:51.389530|HiveBase|3c074c7d3712aa56c1d3e81561c463442f282dfda4ddcc62a724ea0844c2f0e0 initialized successfully 

INFO|2023-06-13 18:37:51.389652|AtClientCommitLogCompaction (@jeremy_0)|Starting commit log compaction job running for every 11 minute(s) 

INFO|2023-06-13 18:37:51.391709|AtClientManager|setCurrentAtSign complete 

INFO|2023-06-13 18:37:51.391750|AtLookup|Creating new connection 

INFO|2023-06-13 18:37:53.118177|AtLookup|New connection created OK 

INFO|2023-06-13 18:37:53.282991|AtLookup|auth success 

INFO|2023-06-13 18:37:53.620010|AbstractAtKeyEncryption (@jeremy_0)|Encrypted shared symmetric key for @jeremy_0 not found in local storage 

INFO|2023-06-13 18:37:53.620075|AbstractAtKeyEncryption (@jeremy_0)|Deleting @rv_am:shared_key@jeremy_0 from LocalSecondary 

INFO|2023-06-13 18:37:53.620532|AbstractAtKeyEncryption (@jeremy_0)|Fetching shared symmetric key for @jeremy_0 from atServer 

INFO|2023-06-13 18:37:53.645636|Monitor (@jeremy_0)|monitor started for @jeremy_0 with last notification time: null 

INFO|2023-06-13 18:37:53.705453|AbstractAtKeyEncryption (@jeremy_0)|Retrieved my encrypted copy of shared symmetric key for @rv_am from atServer - saving to local storage 

INFO|2023-06-13 18:37:53.718238|AbstractAtKeyEncryption (@jeremy_0)|'Their' copy of shared symmetric key for @rv_am not found in local storage - will check atServer 

INFO|2023-06-13 18:37:53.788302|AbstractAtKeyEncryption (@jeremy_0)|Found 'their' copy of shared symmetric key for @rv_am in atServer - saving to local storage 

INFO|2023-06-13 18:37:56.572312| sshnp |SUCCESS:id: 715c0aec-a3f4-4855-b232-5febccd2befc status: NotificationStatusEnum.delivered  

INFO|2023-06-13 18:37:56.793098|AbstractAtKeyEncryption (@jeremy_0)|Encrypted shared symmetric key for @jeremy_0 not found in local storage 

INFO|2023-06-13 18:37:56.793542|AbstractAtKeyEncryption (@jeremy_0)|Deleting @smoothalligator:shared_key@jeremy_0 from LocalSecondary 

INFO|2023-06-13 18:37:56.798742|AbstractAtKeyEncryption (@jeremy_0)|Fetching shared symmetric key for @jeremy_0 from atServer 

INFO|2023-06-13 18:37:56.882780|AbstractAtKeyEncryption (@jeremy_0)|Retrieved my encrypted copy of shared symmetric key for @smoothalligator from atServer - saving to local storage 

INFO|2023-06-13 18:37:56.900090|AbstractAtKeyEncryption (@jeremy_0)|'Their' copy of shared symmetric key for @smoothalligator not found in local storage - will check atServer 

INFO|2023-06-13 18:37:56.985095|AbstractAtKeyEncryption (@jeremy_0)|Found 'their' copy of shared symmetric key for @smoothalligator in atServer - saving to local storage 

INFO|2023-06-13 18:37:57.866809| sshnp |SUCCESS:id: c26376fa-c4e5-4ef6-9e74-c4274c91e015 status: NotificationStatusEnum.delivered 

INFO|2023-06-13 18:37:58.590170| sshnp |SUCCESS:id: 2338dda2-6c90-4f8d-849f-eef0625a0d3e status: NotificationStatusEnum.delivered 

INFO|2023-06-13 18:37:59.299698| sshnp |SUCCESS:id: 28aa0abb-e71b-4f50-ba3b-3dac33eadb48 status: NotificationStatusEnum.delivered 43935 41129 atsign 85.239.53.170 9115b372-b023-4313-ba57-da82b0d2f619 

INFO|2023-06-13 18:38:00.022717|AtLookup|Creating new connection 

INFO|2023-06-13 18:38:00.173533|AtLookup|New connection created OK 

INFO|2023-06-13 18:38:00.283911|AtLookup|auth success 

INFO|2023-06-13 18:38:00.329700|SyncService (@jeremy_0)|823562380|Returning serverCommitId 890 

INFO|2023-06-13 18:38:00.386902|SyncService (@jeremy_0)|823562380|Returning serverCommitId 890 

INFO|2023-06-13 18:38:01.103665| sshnp |Received 9115b372-b023-4313-ba57-da82b0d2f619.docker.sshnp@smoothalligator notification 

INFO|2023-06-13 18:38:01.103725| sshnp |Session 9115b372-b023-4313-ba57-da82b0d2f619 connected successfully 

INFO|2023-06-13 18:38:01.188836| sshnp |Tidying up files 

ssh -p 43935 atsign@localhost -i /atsign/.ssh/id_ed25519

[JeremyTubongbanua]

More context:

The Dockerfile Base image I build

FROM dart:latest AS buildimage

RUN mkdir -p /app/bin /app/binrefactor /app/repo

RUN apt-get update
RUN apt-get install -y git

WORKDIR /app/repo 

RUN git clone https://github.com/atsign-foundation/sshnoports .

RUN git fetch origin ; \
    git checkout gkc-refactor-sshnp ; \
    dart pub get ; \
    dart compile exe /app/repo/bin/sshnp.dart -o /app/binrefactor/sshnp ; \
    dart compile exe /app/repo/bin/sshrv.dart -o /app/binrefactor/sshrv ; \
    dart compile exe /app/repo/bin/sshnpd.dart -o /app/binrefactor/sshnpd ; \
    dart compile exe /app/repo/bin/sshrvd.dart -o /app/binrefactor/sshrvd ; \
    dart compile exe /app/repo/bin/activate_cli.dart -o /app/binrefactor/at_activate ; 

RUN git fetch origin ; \
    git checkout trunk ; \
    dart pub get ; \
    dart compile exe /app/repo/bin/sshnp.dart -o /app/bin/sshnp ; \
    dart compile exe /app/repo/bin/sshrv.dart -o /app/bin/sshrv ; \
    dart compile exe /app/repo/bin/sshnpd.dart -o /app/bin/sshnpd ; \
    dart compile exe /app/repo/bin/sshrvd.dart -o /app/bin/sshrvd ; \
    dart compile exe /app/repo/bin/activate_cli.dart -o /app/bin/at_activate ; 

FROM ubuntu:latest
# FROM dart:latest

ENV USER=atsign
ENV HOMEDIR=/$USER
ENV USER_ID=1024
ENV GROUP_ID=1024

# install necessities
RUN apt-get update ; \
    apt-get install -y sudo wget curl openssh-server nano vim iproute2 nmap tmux git;

# set up user
RUN groupadd --gid ${GROUP_ID} ${USER} ; \
    useradd --system --shell /bin/bash --home ${HOMEDIR} --uid ${USER_ID} --gid ${GROUP_ID} ${USER} ; \
    usermod -aG sudo ${USER} ;

# set up other files/folders, sshnp/sshnpd needs access to ~/.ssh, ~/.sshnp and especially the ~/.ssh/authorized_keys file
RUN mkdir -p ${HOMEDIR}/.sshnp ${HOMEDIR}/.ssh ; \
    touch ${HOMEDIR}/.ssh/authorized_keys ; \
    chown -R ${USER}:${USER} ${HOMEDIR} ;

# disable sudo password, requires vim
RUN ex +"%s/^%sudo.*$/%sudo ALL=(ALL:ALL) NOPASSWD:ALL/g" -scwq! /etc/sudoers ;

# "PasswordAuthentication no" in /etc/ssh/sshd_config
RUN sed -E -i 's|^#?(PasswordAuthentication)\s.*|\1 no|' /etc/ssh/sshd_config

# "ListenAddress 127.0.0.1" in /etc/ssh/sshd_config
RUN sed -i 's/#ListenAddress 0.0.0.0/ListenAddress 127.0.0.1/g' /etc/ssh/sshd_config

# copy binaries from buildimage
RUN mkdir -p ${HOMEDIR}/.local/bin ${HOMEDIR}/.local/binrefactor ${HOMEDIR}/repo ;
COPY --from=buildimage --chown=${USER}:${USER} /app/bin/* ${HOMEDIR}/.local/bin
COPY --from=buildimage --chown=${USER}:${USER} /app/binrefactor/* ${HOMEDIR}/.local/binrefactor

WORKDIR ${HOMEDIR}

# RUN echo '${USER}:123' | sudo chpasswd 
    
USER ${USER}

ENTRYPOINT sudo service ssh start && bash

Then the Dockerfile image that uses the base image

FROM jeremy_demo-base 

COPY keys/*.atKeys ${HOMEDIR}/.atsign/keys/

sshnp commands I ran

sudo ssh-keygen -A ; ssh-keygen -o -a 100 -t ed25519 -f ~/.ssh/id_ed25519 -C "[email protected]"
~/.local/bin/sshnp -f @jeremy_0 -t @smoothalligator -d docker -h @rv_am -s id_ed25519.pub -v
~/.local/binrefactor/sshnp -f @jeremy_0 -t @smoothalligator -d docker -h @rv_am -s id_ed25519.pub -v

sshnpd commands I ran

~/.local/bin/sshnpd -a @smoothalligator -m @jeremy_0 -d docker -s -u -v
~/.local/binrefactor/sshnpd -a @smoothalligator -m @jeremy_0 -d docker -s -u -v

[gkc]

Thanks @JeremyTubongbanua - could you possibly run a REPL for all three atSigns, then run your test with the trunk binaries and email me the REPL outputs?

[gkc]

@JeremyTubongbanua Think I've figured out what the problem was, have pushed a couple of commits (fix one, fix two)

[JeremyTubongbanua]

working @gkc :)

atsign@eb741870652c:~$ ~/.local/binrefactor/sshnp -f @jeremy_0 -t @smoothalligator -d docker -h @rv_am -s id_ed25519.pub -v
INFO|2023-06-13 19:44:08.841896|AtClientManager|setCurrentAtSign called with atSign @jeremy_0 

INFO|2023-06-13 19:44:08.841996|AtClientManager|Switching atSigns from null to @jeremy_0 

INFO|2023-06-13 19:44:08.846532|HiveBase|commit_log_3c074c7d3712aa56c1d3e81561c463442f282dfda4ddcc62a724ea0844c2f0e0 initialized successfully 

INFO|2023-06-13 19:44:08.851330|HiveBase|3c074c7d3712aa56c1d3e81561c463442f282dfda4ddcc62a724ea0844c2f0e0 initialized successfully 

INFO|2023-06-13 19:44:08.851563|AtClientCommitLogCompaction (@jeremy_0)|Starting commit log compaction job running for every 11 minute(s) 

INFO|2023-06-13 19:44:08.852799|AtClientManager|setCurrentAtSign complete 

INFO|2023-06-13 19:44:08.852974|AtLookup|Creating new connection 

INFO|2023-06-13 19:44:09.165009|AtLookup|New connection created OK 

INFO|2023-06-13 19:44:09.282558|AtLookup|auth success 

INFO|2023-06-13 19:44:09.291532| sshnp |Subscribing to notifications on ecb9e730-ec62-43df-b865-6fd911198305.docker.sshnp@ 

INFO|2023-06-13 19:44:09.624312|Monitor (@jeremy_0)|monitor started for @jeremy_0 with last notification time: null 

INFO|2023-06-13 19:44:09.650497|AbstractAtKeyEncryption (@jeremy_0)|Encrypted shared symmetric key for @jeremy_0 not found in local storage 

INFO|2023-06-13 19:44:09.650532|AbstractAtKeyEncryption (@jeremy_0)|Deleting @rv_am:shared_key@jeremy_0 from LocalSecondary 

INFO|2023-06-13 19:44:09.650697|AbstractAtKeyEncryption (@jeremy_0)|Fetching shared symmetric key for @jeremy_0 from atServer 

INFO|2023-06-13 19:44:09.696309|AbstractAtKeyEncryption (@jeremy_0)|Retrieved my encrypted copy of shared symmetric key for @rv_am from atServer - saving to local storage 

INFO|2023-06-13 19:44:09.700632|AbstractAtKeyEncryption (@jeremy_0)|'Their' copy of shared symmetric key for @rv_am not found in local storage - will check atServer 

INFO|2023-06-13 19:44:09.757818|AbstractAtKeyEncryption (@jeremy_0)|Found 'their' copy of shared symmetric key for @rv_am in atServer - saving to local storage 

LateInitializationError: Field 'sshString' has not been initialized.
INFO|2023-06-13 19:44:12.083838|AbstractAtKeyEncryption (@jeremy_0)|Encrypted shared symmetric key for @jeremy_0 not found in local storage 

INFO|2023-06-13 19:44:12.083871|AbstractAtKeyEncryption (@jeremy_0)|Deleting @smoothalligator:shared_key@jeremy_0 from LocalSecondary 

INFO|2023-06-13 19:44:12.084005|AbstractAtKeyEncryption (@jeremy_0)|Fetching shared symmetric key for @jeremy_0 from atServer 

INFO|2023-06-13 19:44:12.127459|AbstractAtKeyEncryption (@jeremy_0)|Retrieved my encrypted copy of shared symmetric key for @smoothalligator from atServer - saving to local storage 

INFO|2023-06-13 19:44:12.131807|AbstractAtKeyEncryption (@jeremy_0)|'Their' copy of shared symmetric key for @smoothalligator not found in local storage - will check atServer 

INFO|2023-06-13 19:44:12.222539|AbstractAtKeyEncryption (@jeremy_0)|Saving 'their' copy of shared symmetric key for @smoothalligator to atServer 

INFO|2023-06-13 19:44:12.279256|AbstractAtKeyEncryption (@jeremy_0)|Saving 'their' copy of shared symmetric key for @smoothalligator to local storage 

INFO|2023-06-13 19:44:14.984218| sshnp |SUCCESS:id: 03140349-e58a-4b81-8d6e-9c06b6e0e19b status: NotificationStatusEnum.delivered 

INFO|2023-06-13 19:44:15.883478| sshnp |SUCCESS:id: f5819203-3f81-435f-b74a-1e87f5baefd3 status: NotificationStatusEnum.delivered 

INFO|2023-06-13 19:44:16.539964| sshnp |SUCCESS:id: c61f4e36-87a3-49e6-880a-718d7f6cec0f status: NotificationStatusEnum.delivered 36029 40355 atsign 85.239.53.170 ecb9e730-ec62-43df-b865-6fd911198305 

INFO|2023-06-13 19:44:16.951715| sshnp |Received ecb9e730-ec62-43df-b865-6fd911198305 notification 

INFO|2023-06-13 19:44:16.951748| sshnp |Session ecb9e730-ec62-43df-b865-6fd911198305 connected successfully 

INFO|2023-06-13 19:44:17.047112| sshnp |Tidying up files 

ssh -p 36029 atsign@localhost -i /atsign/.ssh/id_ed25519

[gkc]

@JeremyTubongbanua Saw this in your logs above, LateInitializationError: Field 'sshString' has not been initialized. ... have fixed that in the most recent push, please confirm with another re-test when you get a chance. Thanks again for all the testing!

[JeremyTubongbanua]

@JeremyTubongbanua Saw this in your logs above, LateInitializationError: Field 'sshString' has not been initialized. ... have fixed that in the most recent push, please confirm with another re-test when you get a chance. Thanks again for all the testing!

Sorry for late reply, my container wasn't working all day yesterday and I realized it was because I didn't start sshd D:

I can confirm that it's working good for me as of commit 3c6c31d

[gkc]

sshrv needs no refactoring, it's a very short program

Although sshrvd is also a short program, it needs a little bit of refactoring just so it's similar to sshnp and sshnpd. Estimating 2SP for this

Dropping priority to P1 as sshnp and sshnpd were the big ones and their refactoring is complete

[XavierChanth]

@gkc I can take on refactoring sshrvd

I'd also like to do a little more refactoring of sshnp and sshnpd to make some of the shared logic more visible

[gkc]

I think we can call this a wrap