Problem reproducing playback in the tutorial

[jennieli421]

Problem reproducing playback in the tutorial

[mattloose]

Hi - you need to change your command - in the python -c you are trying to connect to port 5557 - but your guppy server is on port 5555 - so try this:

python -c 'from pyguppy_client_lib.pyclient import PyGuppyClient as PGC; \ c = PGC("ipc:///tmp/.guppy/5555", "dna_r9.4.1_450bps_fast.cfg"); \ c.connect(); print(c)'

[Adoni5]

Hi @jennieli421

Sorry about the slow response on this. I'm afraid I can't really help without a bit more information. How have you installed minKNOW? If you are running on Ubuntu and installed by adding the ONT repository and sudo apt-get install minion-nc (or something similar), a systemd service file willl have been created, meaning that guppy_basecall_server will automatically start. This can then be managed by a set of controls sudo systemctl <start|stop|restart|status> guppyd . You can check this by using the status option in the terminal and seeing the output.

sudo systemctl status guppy

The problem with this is that guppy is run under a different user and group - one called minknow. This means that you need to alter the permissions if you are to work with readfish - discussed here: #221 (comment)

This shouldn't be your issue here though. I saw in your issue on Squiggulator hasindu2008/slow5tools#92 that you chmod'd the bulk FAST5 file. This is a necessary - but can you try redownloading the file and trying again?

Cheers,
Rory

[jennieli421]

When I installed MInKNOW on Ubuntu 22, I followed this tutorial.

The commands I run:

sudo apt update
sudo apt install wget
wget -O- https://cdn.oxfordnanoportal.com/apt/ont-repo.pub | sudo apt-key add -
echo "deb http://cdn.oxfordnanoportal.com/apt focal-stable non-free" | sudo tee /etc/apt/sources.list.d/nanoporetech.sources.list
sudo apt update
sudo apt install ont-standalone-minknow-gpu-release

So I haven't use commands similar to sudo apt-get install minion-nc. WHen I run this, I got an error:

The following packages have unmet dependencies:
 minknow-nc : Depends: minion-nc (= 21.06.2-1~focal) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

Not quite sure how to fix that.

Additionally, by running sudo systemctl status guppy I got "Unit guppy.service could not be found."

[Adoni5]

Apologies, for guppy that should be

sudo systemctl status guppyd

The minion-nc command I believe may be old , and officially I don't think that ONT support Ubuntu 22 (Jammy Jellyfish) yet!

I did have a similar issue recently following the tutorial - but I believe the fix was the chmod on the BULK fast5 file.

Can you try starting playback again and post the MinKNOW control server logs here (/var/log/minknow/<flowcell id>/control_server_log-0.txt and /var/log/minknow/mk_manager_svc_log-0.txt)?

[jennieli421]

Outputs from running sudo systemctl status guppyd:

● guppyd.service - Service to manage the guppy basecall server.
     Loaded: loaded (/lib/systemd/system/guppyd.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-06-15 14:35:39 EDT; 19h ago
       Docs: https://community.nanoporetech.com/protocols/Guppy-protocol/v/GPB_2003_v1_revQ_14Dec2018
   Main PID: 860 (guppy_basecall_)
      Tasks: 78 (limit: 76625)
     Memory: 301.1M (high: 8.0G limit: 8.0G available: 7.7G)
        CPU: 40min 2.964s
     CGroup: /system.slice/guppyd.service
             └─860 /opt/ont/guppy/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bp>

Jun 15 14:35:39 LX310327 guppy_basecall_server[860]: runners per device:  8
Jun 15 14:35:39 LX310327 guppy_basecall_server[860]: Use of this software is permitted solely under the terms o>
Jun 15 14:35:39 LX310327 guppy_basecall_server[860]: By running, copying or accessing this software, you are de>
Jun 15 14:35:39 LX310327 guppy_basecall_server[860]: The EULA may be found in /opt/ont/guppy/bin
Jun 15 14:35:41 LX310327 guppy_basecall_server[860]: Config loaded:
Jun 15 14:35:41 LX310327 guppy_basecall_server[860]: config file:               /opt/ont/guppy/data/dna_r9.4.1_>
Jun 15 14:35:41 LX310327 guppy_basecall_server[860]: model file:                /opt/ont/guppy/data/template_r9>
Jun 15 14:35:41 LX310327 guppy_basecall_server[860]: model version id           2021-05-17_dna_r9.4.1_minion_96>
Jun 15 14:35:41 LX310327 guppy_basecall_server[860]: adapter scaler model file: None
Jun 15 14:35:41 LX310327 guppy_basecall_server[860]: Starting server on port: ipc:///tmp/.guppy/5555

I redo the steps as mentioned in the first post. still having "Run error". Here are the log files:

control_server_log-0.txt
mk_manager_svc_log-0.txt

[jennieli421]

Another important question:
Will readfish be compatible with the new version of flow cell R10.4.1? We would like to test readfish (real-time enrichment) using the new kit and flow cell once they come in next week.

[Adoni5]

So looking at the control-server-log-0(1).txt, I can see

2023-06-16 10:08:22.746498 WARNING: hdf5_error (fast5)
    cause: unable to open file: name = '/home/tilgnerlab/readuntil/readfish_files/fast5/PLSP57501_20170308_FNFAF14035_MN16458_sequencing_run_NOTT_Hum_wh1rs2_60428.fast5', errno = 13, error message = 'Permission denied', flags = 0, o_flags = 0
    function: H5Fopen
    message: unable to open file
    stack_trace: 

Which suggest to me that you need to chmod the bulk FAST5 file again.

What is the output of
ls /home/tilgnerlab/readuntil/readfish_files/fast5/PLSP57501_20170308_FNFAF14035_MN16458_sequencing_run_NOTT_Hum_wh1rs2_60428.fast5?

ReadFish is indeed compatible with R10.4.1 - you will just have to change the base caller model in the TOML file.

[jennieli421]

~$ ls -l /home/tilgnerlab/readuntil/readfish_files/fast5/PLSP57501_20170308_FNFAF14035_MN16458_sequencing_run_NOTT_Hum_wh1rs2_60428.fast5
-rwxrwxrwx 1 tilgnerlab tilgnerlab 22633323381 Apr 23  2018

[jennieli421]

The playback permission issue is resolved. In addition to chmod 777, we also sudo usermod -a -G admin minknow to give permission to the right group. After that playback is fixed.

The unblock-all is working.

However, the targets command gives error:

2023-06-16 13:27:29,636 ru.ru_gen Region 'select_chr_12_start' (control=False) has 2 contigs of which 0 are in the reference. There are 4 targets (including +/- strand) representing 4.58% of the reference. Reads will be unblocked when classed as single_off or multi_off; sequenced when classed as single_on or multi_on; and polled for more data when classed as no_map or no_seq.
[guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [timed_out] Timeout waiting for reply to request: LOAD_CONFIG

I run guppy in a separate terminal, then started playback in minknow, then run targets command.

I couldn't tell if it was working from the histogram. The run report is attached.
MinKNOW Run Report-16-06-2023-sim.pdf

[Adoni5]

Hey @jennieli421 - great news you have gotten it working! 🍾 This LOAD_CONFIG is a problem a lot of people get caught out by, when running on a non ONT box, due to all the fun that comes with minknow group permission errors. We're working on putting some notifications in the code for this, but I believe that if you have a look at this comment on another issue you should be alright! #221 (comment)

I'm surprised that sudo usermod -a -G admin minknow hasn't fixed it though, so let me know if that link fixes it or not!

[jennieli421]

Still getting the LOAD_CONFIG error. This is what I've done:

sudo usermod -a -G minknow  $USER

groups tilgnerlab
tilgnerlab : tilgnerlab adm cdrom sudo dip plugdev lpadmin lxd sambashare minknow

groups minknow
minknow : minknow tilgnerlab

sudo chmod 775 /tmp/.guppy/5555

ls -al /tmp/.guppy/
total 8
drwxr-xr-x  2 minknow minknow 4096 Jun 16 12:25 .
drwxrwxrwt 24 root    root    4096 Jun 20 11:05 ..
srwxrwxr-x  1 minknow minknow    0 Jun 16 12:25 5555

[Adoni5]

So the umask for files in linux is User/Group/Other. So as a part of the minknow group, you should have read write and execute on the socket file, but either there or is something else going wrong here or you don't.

Can you try the yolo approach, just to see if it works and chmod 777 /tmp/.guppy/5555?

You can test the connection using the snippet at the following comment, if you aren't already - #221 (comment)

[jennieli421]

Operation not permitted:

$ chmod 777 /tmp/.guppy/5555
chmod: changing permissions of '/tmp/.guppy/5555': Operation not permitted
$ ls -ltrh /tmp/.guppy
total 0
srwxrwxr-x 1 minknow minknow 0 Jun 16 12:25 5555
$ python -c 'from pyguppy_client_lib.pyclient import PyGuppyClient as PGC; \
           c = PGC("ipc:///tmp/.guppy/5557", "dna_r9.4.1_450bps_fast.cfg"); \
           c.connect(); print(c)'
[guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [timed_out] Timeout waiting for reply to request: LOAD_CONFIG
[guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [timed_out] Timeout waiting for reply to request: LOAD_CONFIG

I've also tried this:

$ sudo chown tilgnerlab:tilgnerlab /tmp/.guppy/5555
l$ ls -ltrh /tmp/.guppy
total 0
srwxrwxr-x 1 tilgnerlab tilgnerlab 0 Jun 16 12:25 5555
$ python -c 'from pyguppy_client_lib.pyclient import PyGuppyClient as PGC; \
           c = PGC("ipc:///tmp/.guppy/5557", "dna_r9.4.1_450bps_fast.cfg"); \
           c.connect(); print(c)'
[guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [timed_out] Timeout waiting for reply to request: LOAD_CONFIG

Additionally, would be happy to have a zoom call regarding this issue. If this sounds plausible my contact is listed in my profile.

[jennieli421]

Sorry for the mistake.

$ python -c 'from pyguppy_client_lib.pyclient import PyGuppyClient as PGC; \
           c = PGC("ipc:///tmp/.guppy/5555", "dna_r9.4.1_450bps_fast.cfg"); \
           c.connect(); print(c)'
PyGuppyClient(address='ipc:///tmp/.guppy/5555', config='dna_r9.4.1_450bps_fast', align_ref=None, bed_file=None, barcodes=None, status.connected, )

However, with playback running, the readfish target command is still giving error:

$ readfish targets --device MN19362 \
              --experiment-name "RU_test" \
              --toml /home/tilgnerlab/readuntil/readfish_files/human_chr_selection.toml \
              --log-file ru_test.log
2023-06-21 13:02:28,310 ru.ru_gen /home/tilgnerlab/miniconda3/envs/readfish/bin/readfish targets --device MN19362 --experiment-name RU_test --toml /home/tilgnerlab/readuntil/readfish_files/human_chr_selection.toml --log-file ru_test.log
2023-06-21 13:02:28,310 ru.ru_gen batch_size=512
2023-06-21 13:02:28,310 ru.ru_gen cache_size=512
2023-06-21 13:02:28,310 ru.ru_gen channels=[1, 512]
2023-06-21 13:02:28,310 ru.ru_gen chunk_log=None
2023-06-21 13:02:28,310 ru.ru_gen command=targets
2023-06-21 13:02:28,310 ru.ru_gen device=MN19362
2023-06-21 13:02:28,310 ru.ru_gen dry_run=False
2023-06-21 13:02:28,310 ru.ru_gen experiment_name=RU_test
2023-06-21 13:02:28,310 ru.ru_gen func=<function run at 0x7fb9c453ad30>
2023-06-21 13:02:28,310 ru.ru_gen host=127.0.0.1
2023-06-21 13:02:28,310 ru.ru_gen log_file=ru_test.log
2023-06-21 13:02:28,310 ru.ru_gen log_format=%(asctime)s %(name)s %(message)s
2023-06-21 13:02:28,310 ru.ru_gen log_level=info
2023-06-21 13:02:28,310 ru.ru_gen max_unblock_read_length_seconds=5
2023-06-21 13:02:28,311 ru.ru_gen paf_log=None
2023-06-21 13:02:28,311 ru.ru_gen port=None
2023-06-21 13:02:28,311 ru.ru_gen run_time=172800
2023-06-21 13:02:28,311 ru.ru_gen throttle=0.4
2023-06-21 13:02:28,311 ru.ru_gen toml=/home/tilgnerlab/readuntil/readfish_files/human_chr_selection.toml
2023-06-21 13:02:28,311 ru.ru_gen unblock_duration=0.1
2023-06-21 13:02:28,311 ru.ru_gen workers=1
2023-06-21 13:02:28,317 ru.ru_gen Initialising minimap2 mapper
2023-06-21 13:02:30,629 ru.ru_gen Mapper initialised
2023-06-21 13:02:30,773 ru.ru_gen This experiment has 1 region on the flowcell
2023-06-21 13:02:30,774 ru.ru_gen Using reference: /home/tilgnerlab/readuntil/hg38-ont.mmi
2023-06-21 13:02:55,851 ru.ru_gen Region 'select_chr_12_start' (control=False) has 2 contigs of which 0 are in the reference. There are 4 targets (including +/- strand) representing 4.58% of the reference. Reads will be unblocked when classed as single_off or multi_off; sequenced when classed as single_on or multi_on; and polled for more data when classed as no_map or no_seq.
[guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [timed_out] Timeout waiting for reply to request: LOAD_CONFIG

[Adoni5]

Hi @jennieli421 - last effort then happy to arrange a zoom call! What is the host set to under caller_settings in the TOML file that you are passing to readfish?

If it isn't 'ipc:///tmp/.guppy/' that might be the issue? Port should remain 5555

Also apologies but the chmod command above should be

sudo chmod 777 /tmp/.guppy/5555

Can you try that, just to see if it solves the issue?

Thanks,
Rory

[jennieli421]

Here is the TOML file:

[caller_settings]
config_name = "dna_r9.4.1_450bps_fast"
host = "ipc:///tmp/.guppy/5555"
port = 5555

[conditions]
reference = "/home/tilgnerlab/readuntil/hg38-ont.mmi"

[conditions.0]
name = "select_chr_1_10"
control = false
min_chunks = 0
max_chunks = 2
targets = '/home/tilgnerlab/readuntil/readfish_files/ReadFish_v7_gene_targets.collapsed.hg38.txt'
single_on = "stop_receiving"
multi_on = "stop_receiving"
single_off = "unblock"
multi_off = "unblock"
no_seq = "proceed"
no_map = "proceed"

Also changed permission:

ls -ltrh /tmp/.guppy
total 0
srwxrwxrwx 1 tilgnerlab tilgnerlab 0 Jun 16 12:25 5555

Still have the LOAD_CONFIG error.

[Adoni5]

So the host in the toml file should be just ipc:///tmp/.guppy/ and not ipc:///tmp/.guppy/5555. can you try running after changing that?

[jennieli421]

That resolved the problem!

To summarize, to run readfish target the key steps are:

sudo chmod 777 /tmp/.guppy/5555

in TOML

[caller_settings]
config_name = "dna_r9.4.1_450bps_fast"
host = "ipc:///tmp/.guppy/"
port = 5555

Connect to guppy in a separate terminal:

guppy_basecall_server --config dna_r9.4.1_450bps_fast.cfg -p 5555 -l /tmp/guppy -x 'cuda:0'

Start playback.
In a separate terminal:

readfish targets --device MN19362 \
              --experiment-name "RU_test" \
              --toml /home/tilgnerlab/readuntil/readfish_files/human_chr_selection.toml \
              --log-file ru_test.log

[Adoni5]

Yes. You actually can also add your own User to the Minknow group and simply run
sudo chmod 774 /tmp/.guppy/5555. This is slightly more secure - allowing any users in the minknow group to use the 5555 socket file.

Connect to guppy in a separate terminal:
guppy_basecall_server --config dna_r9.4.1_450bps_fast.cfg -p 5555 -l /tmp/guppy -x 'cuda:0'

If you have a standard MinKNOW setup, guppy should be started automatically when you boot your computer.

But otherwise yes!

I'm going to close this issue for now.