nanoporetech/minknow_api

Protobuf and gRPC specifications for the MinKNOW API

---

API Specifications for MinKNOW

Overview

MinKNOW is software from Oxford Nanopore Technologies plc to control its single-modecule (eg: DNA or RNA) sequencing devices. It comes embedded in GridION, PromethION and MinION Mk1C devices, and can also be installed on user PCs to run devices such as the MinION Mk1B/Mk1D and the P2 Solo. MinKNOW drives the sequencing hardware, analyses the data and provides the user with the results (which can be the raw data samples from the hardware, a DNA or RNA sequence, or the answer to a biological question).

The files and code in this repository provide a way of automating interactions with MinKNOW, including gathering information about running or finished experiments, starting and stopping those experiments, and even exerting more fine-grained control. Anything that can be done via MinKNOW's user interface (and more) can be done using the APIs in this project.

This project is targetted at LIMS developers or developers of other tools that have a need to integrate with MinKNOW. Some familiarity with Python is expected (although the APIs can be used from other languages), as the examples are written in Python. If you are not familiar with gRPC, it is worth reading through some of the gRPC documentation to get a feel for it.

---

Getting Started

Dependencies

The first thing you will need is an installation of MinKNOW to communicate with. This can be obtained from the Oxford Nanopore Community download pages if you want a local installation, although the installation on a GridION, PromethION, etc will also work.

Note that you will need a compatible version of MinKNOW for the version of minknow_api you are using - see the FAQs below.

Python

For the Python module and examples, you will also need Python 3.7 or later. The Python module depends on a few other packages, such as grpcio and numpy, but these should be handled as part of package installation.

See python/README.md for more information about the Python module.

Other Languages

MinKNOW's API is based on gRPC and can be used from any language supported by gRPC. This includes Go, Java, Ruby, JavaScript (although see the note below about JavaScript) and many more. The gRPC documentation describes how to generate client libraries from the API descriptions in this repository, and how to use those libraries. The .proto files themselves contain documentation about what the various RPCs do, and a more general overview is given below.

The Manager interface is available on port 9502 - see the FAQs for more details. From there, APIs are available to get the ports that other services operate on (including the services for each flow cell position).

JavaScript

How to use gRPC from JavaScript depends on whether you're using it from Node (in which case it works much like any other language - see above) or from a web browser environment. For web browsers (including Electron), you need to use the Web variant, which may also involve connecting to a different port.

Secure connections still start at 9502 for the Manager service. When using the Manager APIs to get ports for other services, you should use fields with grpc_web in the name, like secure_grpc_web.

Authentication

See AUTH.md for information on authenticating with the MinKNOW API.

API Highlights

The files in proto/minknow_api describe MinKNOW's APIs. Each file describes a single service, which covers a specific area of MinKNOW's functionality.

There are two global services, the manager service and basecaller service. There is only one instance of each of these services: see below for how to connect to them. All other services are provided by each flow cell position independently. For example, if you are using a GridION X5, which has 5 flow cell positions, there will be 5 ports (or sets of ports - secure, gRPC Web, etc), each of which will provide all the other services.

manager.proto

manager.proto is the entry point of MinKNOW's APIs. It is always available on a specific port 9502.

The most important method is flow_cell_positions, which provides information about how to connect to the services for each flow cell position. From there you can access all the flow cell position-specific services.

Other methods on the manager service provide general information about the MinKNOW installation and its high-level state, as well as port information for the basecaller service service, which can be used to basecall data from previous experiments.

instance.proto

instance.proto provides general information about the flow cell position. Of particular interest is the get_output_directories method, which indicates where data files will be written to. set_output_directories can be used to change this.

stream_instance_activity may also be useful, as it provides a continuously-updated summary of the state of the position.

This can be accessed via the ports reported by flow_cell_positions on the manager service.

protocol.proto

protocol.proto allows starting and stopping experiment protocols, as well as providing information about the current and previous protocol runs. Note that information about protocol runs from before the last restart is not available via this API.

See the start_protocol example for an example of how to use this service to start a protocol.

acquisition.proto

The main work of a protocol is acquiring data, and this is managed in acquisition.proto. While most of the methods in acquisition.proto will not be useful to most external tools, get_acquisition_info is helpful for access detailed information about what was done by a protocol run reported by protocol.proto.

device.proto

device.proto provides more detailed information about the hardware of the flow cell position and the inserted flow cell. get_device_info provides some constant information about the position, while get_flow_cell_info provides information about the flow cell (a streaming version that provides updates about changes to the flow cell, such as it being removed, is also available). It is also possible to override the flow cell identifier and product code via this service (although this is not generally recommended).

statistics.proto

statistics.proto provides statistics about current and previous protocol runs, including duty time and temperature information. This is useful for generating reports or tables of data describing how well an experiment has performed.

---

Help

Licence and Copyright

© 2023 Oxford Nanopore Technologies PLC.

API Specifications for MinKNOW is distributed under the Terms and Conditions of the Nanopore Community.

FAQs

What MinKNOW versions will this work with?

MinKNOW's API changes over time, to support new features and occasionally for other reasons, such as improving security. The important version is that of MinKNOW Core - this version looks like 3.6.5 or 4.0.1 - rather than the date-based version of the entire MinKNOW release (which looks more like 19.12.5).

Every minor release of MinKNOW Core (the 4.0 in 4.0.1) has a fixed API, which is described by the corresponding minor releases of minknow_api. So you should use the latest 4.0.x release of minknow_api to talk to MinKNOW Core 4.0.1.

Note that while minknow_api does not strictly use semantic versioning, we aim to keep backwards-incompatible changes to a minimum, and to deprecate anything we are planning to remove for at least one minor version before we actually remove it. We also aim to only make incompatible changes that are likely to have an impact on third-party systems when we bump the major version.

What port should I connect to?

Since MinKNOW Core 5.6, there are two "entry" ports, providing manager service:

• 9501 can only be used with a gRPC secure channel. If you want to use client certificate authentication, you need to use this port.
• 9502 is for gRPC-Web connections, but can also be used with a gRPC secure channel. However, it cannot be used with client certificate authentication.

Versions prior to 5.6 only provided the 9502 port.

Ports reported by manager RPCs follow a similar pattern: there are two fields. For example, the basecaller_api RPC returns a response with two fields:

• secure is a port that can be connected to with a gRPC secure channel, and supports client certificates
• secure_grpc_web is a port that can be connected to with a gRPC-Web secure channel (but does not support client certificates)

How do I set up a gRPC secure channel?

MinKNOW provides TLS transport security for its APIs, but uses a self-signed certificate. This means that the client library you use has to trust this certificate.

If you are using the minknow_api Python package, this is all handled for you. If you are using the gRPC client libraries directly (for example, if you are connecting from a language other than Python), you will need to tell the library about MinKNOW's TLS certificates.

Within the MinKNOW installation, you can find the CA certificate at conf/rpc-certs/ca.crt. This can be passed to most gRPC client libraries as part of creating a secure/SSL channel.

Note that this certificate is only valid for the "localhost" name - connecting to 127.0.0.1 directly will not work, nor will connecting across the network. You can work around this by setting the grpc.ssl_target_name_override channel option to localhost.

Glossary

Acquisition

An acquisition period or acquisition run is a period in which data was being actively read from the flow cell. A typical protocol will have two acquisition periods - a short one with minimal analysis to calibrate the flow cell followed by a much longer one to actually gather sequence data.

Device

When the APIs refer to a device, this is the same as a flow cell position. Newer APIs use the more descriptive flow cell position, but there are plenty of older APIs that use device instead.

Experiment

In the user interface, an experiment name is another name for a protocol group ID (see Protocol). Note that what a user considers an "experiment" may not map onto a consistent concept in MinKNOW, and the APIs generally avoid this term in favour of a more precise term like "protocol run" or "protocol group".

Flow Cell

The flow cell is a consumable that plugs into the sequencing unit and contains the chemistry required to sequence the sample. Often, but not always, contains some core electronics.

The flow cell position is the location that a flow cell can be plugged in to. It is synonymous with device in the APIs.

Protocol

A protocol is a description of how to perform an experiment. This takes the form of some configuration and a Python script. A protocol run is a specific execution of that protocol.

A protocol group is a set of protocol runs given the same name (referred to as a "protocol group ID" in the API, and an "experiment name" in the user interface).

minknow_api's Issues

Using this API for run until (not read until)

Where do I start and how do I use this API for run until? Can this be captured in the documentation or may I get some quick hints on this?

how can we modify the start_protocol.py to obtain zlib_compres fast5

We did not find a switch to define zlib_compres fast5 as output option and do seem to obtain (by default?) vbz_compres fast5.
Essentially we would like to obtain fast5 identical to those generated by starting a run with MinKnow UI. Currently we do use the SQK-RBK004 kit.
Looking forward to your reply.
Best wishes
Claus

Protobuff misspelled in "About"

In the about of the minknow_api, the text is "Protobuff and gRPC specifications for the MinKNOW API". It should be "Protobuf", not "Protobuff"

https://en.wikipedia.org/wiki/Protocol_Buffers

Question Regarding Development with Simulation

Hi,

I'm currently working on a driver for the device and do not have access to the physical machine.
Is there documentation around using a simulated device? I'm having issues with the method call.

Best,
Simon

Setting min_qscore for live basecalling with the current minknow_api version

In a previous version of minknow (21.11.9, running in Ubuntu 18.04), the following configuration for read_filtering appeared in the sequencing config file /opt/ont/minknow/conf/package/sequencing/sequencing_MIN106_DNA.toml:

[basecaller_configuration.read_filtering]
min_qscore = 7

This configuration is not present in the default installation of minknow version 22.12.7 on Ubuntu 20.04.6, so calls to start_protocol.py with the minknow_api invaraibly default to min_qscore of 0 with version 22.12.7, regardless of the default settings in /opt/ont/guppy/data/dna_r9.4.1_450bps_fast.cfg

It turns out that the default for basecaller_configuration.read_filtering.min_qscore can be added back into sequencing_MIN106_DNA.toml and still works with the current minknow_api, but I was wondering:

How can code be added to start.protocol.py to allow modifying the min_qscore setting dynamically when starting a sequencing run? I am assuming there is a way to do this and I can just not figure out the syntax.

Any help would be appreciated.

grpc status UNAVAILABLE

Hi!

we are developing a dashboard which shows the status of all our promethion 48 machines, it was working before but unfortunately after the MinKnow update it shows this error.

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{"created":"@1617778828.410866000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":5397,"referenced_errors":[{"created":"@1617778828.410862000","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":398,"grpc_status":14}]}"

can we specify toml for a sequencing run via the API?

Is there a way to specify toml for a sequencing run via the API? We could specify a particular toml for a playback / simulation run.
Looking forward to your reply.
Best wishes
Claus

start_protocol.py - argument experiment-duration acccepted but not used downstream

I tried to reduce the runtime from your example from 24 hrs to 1 hour.
In suitable python environment I do run:
$ python start_protocol.py \

--host localhost --position MN32621
--sample-id "2021_04_28b_cmd_run_01"
--experiment-group "2021_04_28b_cmd_run_01"
--experiment-duration 1
--basecalling
--fastq --bam
--verbose
--kit SQK-RBK004
--barcoding
--barcode-kits "SQK-RBK004"

but in the started protocols I do find the default runtime of 72 hours:
<...>
Starting protocol on 1 positions
Starting protocol sequencing/sequencing_MIN106_DNA:FLO-MIN106:SQK-RBK004 on position MN32621
DEBUG:minknow_api.tools.protocols:Built protocol arguments: --base_calling=on --barcoding barcoding_kits=['SQK-RBK004',] --experiment_time=72 --fast5=off --fastq=on --fastq_data compress --fastq_reads_per_file=4000 --bam=on --active_channel_selection=on --mux_scan_period=1.5
Started protocol e39a452b-ee3e-49af-8d68-39d9b0b0f9dc

How can I correct this?

Thanks
Claus

user provided name through api

I have played a bit with the demo provided, but so far have not been successful; Is it possible to check on a remote machine if a run has started and if so, what the name the user has provided ?

My initial attempt is to connect to the manager to get all the positions, then connect to device to see if has_flow_cell is true; I then attempted to find through the protocol but beside the fact that its stored in /data I couldn't find the name the user has provided in the interface; I could find "acquisition_run_ids" so perhaps I should query those with something ?

I'm not familiar with gRPC but i'm happy to learn new stuff; if there is however a easier stable method I'm all ear !

Any help would be greatly appreciated :)

Read Until update?

Hello @RDokos & team!

I couldn't find anything related to the new adaptive sampling feature in this repo. Any plans to release it? I assume there's already an internal updated version of this repo with it. My team and I need it a lot!

I understand read until development was carried out in the read_until_api repo. However, I am not sure if this is still the case. From using the MinKNOW GUI it seems reasonable there would be a way to start a read until protocol from the minkow_api.

Could you either confirm these assumptions or point me in the right direction? I'm trying to start multiple adaptive sampling runs programatically.

Thanks!

Why is internet required when sequencing is ongoing?

I need to be able to sequence and basecall with no internet connection. What do I need to do?

Bad metadata key - Error 16 on Instance Service

Hi, I've successfully connected to the Manager service on 9502 (c++) and supplied the certificate and managed to get basic details and watch the flow cells posistion stream to get updates on the connected flow cells and their ports. But when i come to call an Instance service on one of those ports I get Bad Metadata Key even though Im passing the same certificate. I have tried setting the MINKNOW_API_USE_LOCAL_TOKEN to 1 as per the FAQ and it makes no difference, do i need to restart anything after setting that variable?

Thanks

Jay

restart position without the MiKNOW UI

Hello ONT,

We are trying to run our minion directly from the command line. For this we are using the start_protocol.py.
We initially had an authentication issue (Issues #34), however we set "--no-tls True" as a workaround.
Now when running we receive the following error:

`No flow cell present in position MN39033 (running)`

However, the flow cell is definitely inserted in the minion. I can see that "position_connection.device.get_flow_cell_info()" is used to obtain the flow cell information which is defined in minknow_api.device_service.DeviceService.
Notably, using "position_connection = spec.position.connect()" I can see that the device is running:

	[FlowCellPosition('localhost', {name: "MN39033"
	state: STATE_RUNNING
	rpc_ports {
	  secure: 8001
	  insecure: 8000
	  secure_grpc_web: 8002
	}
	})]

Also, using "position_connection.device.get_device_info()" I can return information on the device;

	device_info device_id: "MN39033"
	max_channel_count: 512
	max_wells_per_channel: 4
	digitisation: 8192

So cool, I can see the device, just not the flow cell. If restart the position using the MiKNOW UI the start_protocol.py script progresses (alas to another issue);

	Failed to find protocol for position MN39033 (running)
	Requested protocol:
	  product-code: None
	  kit: SQK-LSK109
	  basecalling: False
	  basecall_config: None
	  barcode-kits: None
	  barcoding: False

However, I can now retrieve the flow cell info using "position_connection.device.get_flow_cell_info()" after restarting the position with MiKNOW UI

          has_flow_cell: true
          channel_count: 126
          wells_per_channel: 1
          has_adapter: true
          adapter_id: "FA-05846"
          temperature_offset: 327.6700134277344
          asic_version: "IA02D"
          asic_id_str: "7735955"

So this leads to my question, how do we restart the position without the MiKNOW UI? I am probably missing something obvious and I apologize in advance. Thank you for any help.

how to turn off basecalling and also set run_id etc.

We are starting and stopping the run from the terminal using the API's protocols. We would like to set the run's initial configurations through the terminal using the protocols instead of using the MinKNOW GUI. We also want to turn off the basecalling by default. Please let us know how to do this, which protocols to call etc? @iiSeymour @jorj1988

Please release under a F/OSS license

This will allow packaging of minknow_api for Linux distributions like Debian, (bio)conda, and others.

As for the current LICENSE.txt, I can find no mention of a "Terms and Conditions of the Nanopore Community" document at https://nanoporetech.com/

I see that https://myaccount.nanoporetech.com/registration/users/details links to https://nanoporetech.com/commercial_information with the text "Nanopore Community Terms & Conditions", but that page lists 10 different documents, none of them named "Nanopore Community Terms & Conditions"

The closest match is the "Nanopore Product Terms & Conditions" at https://nanoporetech.com/sites/default/files/s3/terms/Nanopore-product-terms-and-conditions-nov2018-v2.pdf , but that gives no right of (re)distribution for any software, and under section 2.2.2m it specifically excludes the right to

redistribute, encumber, sell, rent, lease, sublicense or otherwise transfer rights to Goods or Software

Without a clear license that at least allows for (re)distribution, even https://pypi.org/project/minknow-api/ does not have to right to (re)distribute the minknow_api.

Thanks!

minknow_api cannot be imported

The latest release of minknow_api (version 5.0.0) cannot be imported.

Steps to reproduce:

In a clean python3 environment run:

pip install minknow_api
python -c "import minknow_api"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/matt/testenv/lib/python3.7/site-packages/minknow_api/__init__.py", line 87, in <module>
    from minknow_api.manager import get_local_authentication_token_file
  File "/Users/matt/testenv/lib/python3.7/site-packages/minknow_api/manager.py", line 21, in <module>
    import minknow_api.basecaller_service
ModuleNotFoundError: No module named 'minknow_api.basecaller_service'

broken file in /data/persistence

Hey,

I had a corrupt file in /data/persistence/3A/acquisitions/* this blocked my script that monitors runs, so I removed it; sadly now it fails cause the file is missing. We rebooted the device so I'm not sure how it remembers this run or how I can circumvent this. I don't have the original corruption error anymore. But currently my script fails with (only on this position) :

fails on this line :

# for pos.name == 3A
# this works
position_connection = pos.connect()
# here it fails -->
position_connection.protocol.list_protocol_runs()

the error :

INFO: connecting to pos : 3A
Traceback (most recent call last):
  File "get_runs.py", line 264, in <module>
    main()
  File "get_runs.py", line 261, in main
    create_acquisition_info(position_connection, run, pos)
  File "get_runs.py", line 115, in create_acquisition_info
    aq_info = connection.acquisition.get_acquisition_info(run_id=run)
  File "/usr/local/lib/python3.8/dist-packages/minknow_api/acquisition_service.py", line 523, in get_acquisition_info
    return run_with_retry(self._stub.get_acquisition_info,
  File "/usr/local/lib/python3.8/dist-packages/minknow_api/acquisition_service.py", line 74, in run_with_retry
    result = MessageWrapper(method(message, timeout=timeout), unwraps=unwraps)
  File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.NOT_FOUND
        details = "Error opening Acquisition Record file: File does not exist.
filename: /data/persistence/3A/acquisitions/08e5fbb53d8349ca8397a2de6eb567b818919780"
        debug_error_string = "{"created":"@1643103325.367472983","description":"Error received from peer ipv4:127.0.0.1:8048","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Error opening Acquisition Record file: File does not exist.\nfilename: /data/persistence/3A/acquisitions/08e5fbb53d8349ca8397a2de6eb567b818919780","grpc_status":5}"
>

I tried copying another file to the location, but then it fails on the fact that its not the right run_id. We restarted the both devices, so its stored somewhere. Can I somehow fix this ? For now, I just ignore this position but we can't track the runs any longer, which is a bit annoying.

Any help / ideas welcome !

minknow_api RPC error

Hi,

I have been running the program uncalled with minknow successfully for the past several months on Ubuntu 20.04. After a round of updates, I now get the following error message:

uncalled realtime reference_genomes/Ames/genome.fasta -t 10 --enrich --full -c 3 | tee test.paf
[15:23:21 - ReadUntil] Creating many chunk client with ReadCache data queue filtering to strand2, strand1, and strand read chunks.
[15:23:21 - ReadUntil] Using pre-defined read classification map.
[15:23:21 - ReadUntil] Creating rpc connection on port 8000.
[15:23:21 - minknow_api] Error received from rpc
[15:23:21 - minknow_api] Failed to connect to minknow instance (retry 1/5): failed to connect to all addresses
[15:23:22 - minknow_api] Error received from rpc
[15:23:22 - minknow_api] Failed to connect to minknow instance (retry 2/5): failed to connect to all addresses
[15:23:24 - minknow_api] Error received from rpc
[15:23:24 - minknow_api] Failed to connect to minknow instance (retry 3/5): failed to connect to all addresses
[15:23:26 - minknow_api] Error received from rpc
[15:23:26 - minknow_api] Failed to connect to minknow instance (retry 4/5): failed to connect to all addresses
[15:23:29 - minknow_api] Error received from rpc
[15:23:29 - minknow_api] Failed to connect to minknow instance (retry 5/5): failed to connect to all addresses
Traceback (most recent call last):
File "/home/jason/.local/lib/python3.8/site-packages/uncalled-2.2-py3.8-linux-x86_64.egg/EGG-INFO/scripts/uncalled", line 195, in realtime_cmd
client = unc.minknow_client.Client(conf.host, conf.port, conf.chunk_time, conf.num_channels)
File "/home/jason/.local/lib/python3.8/site-packages/uncalled-2.2-py3.8-linux-x86_64.egg/uncalled/minknow_client.py", line 43, in init
read_until.ReadUntilClient.init(
File "/home/jason/anaconda3/envs/pima/lib/python3.8/site-packages/read_until/base.py", line 255, in init
self.connection = minknow_api.Connection(host=self.mk_host, port=self.grpc_port)
File "/home/jason/anaconda3/envs/pima/lib/python3.8/site-packages/minknow_api/init.py", line 327, in init
raise error
File "/home/jason/anaconda3/envs/pima/lib/python3.8/site-packages/minknow_api/init.py", line 299, in init
self.instance.get_version_info()
File "/home/jason/anaconda3/envs/pima/lib/python3.8/site-packages/minknow_api/instance_service.py", line 93, in get_version_info
return run_with_retry(self._stub.get_version_info,
File "/home/jason/anaconda3/envs/pima/lib/python3.8/site-packages/minknow_api/instance_service.py", line 37, in run_with_retry
result = MessageWrapper(method(message, timeout=timeout), unwraps=unwraps)
File "/home/jason/anaconda3/envs/pima/lib/python3.8/site-packages/grpc/_channel.py", line 946, in call
return _end_unary_response_blocking(state, call, False, None)
File "/home/jason/anaconda3/envs/pima/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{"created":"@1652912609.827260489","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3008,"referenced_errors":[{"created":"@1652912609.827259173","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":397,"grpc_status":14}]}"

It has been working great so I think the latest update is making it crash. Any ideas what I need to change? Thanks,

Chandler

setting break_reads_after_seconds parameter

Hi,

I sucessfully could set the break_reads_after_seconds parameter via Analysis_Configuration. After starting a playback run the behaviour of the sequencing suggests that the parameter is overruled by the value set in the sequencing TOML file. Is that correct? And if so, would it be possible to not overrule?

Thanks
Jens

Stopping a run

Hello, I'm trying to stop a run using the api and I'm having some trouble.

I'm trying:

from minknow_api.manager import Manager
from minknow_api.acquisition_service import AcquisitionService

manager = Manager(host='localhost', port=None, use_tls=False)
aqs = AcquisitionService(manager.channel)

aqs.stop()

Which gets me an error:

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNIMPLEMENTED
	details = ""
	debug_error_string = "{"created":"@1624894312.205359526","description":"Error received from peer ipv6:[::1]:9501","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"","grpc_status":12}"

Am I doing something wrong in my code, do I not have something setup on the machine correctly, or something else?

Thanks for the help!

Is there a list of args for the start protocol method within the protocol.proto?

Minknow_api in web with javascript

Hi, I want to use the minknow_api in web using javascript. There is no tutorial or example, and I can't seem to get it work on my own. Maybe you have some example I can learn from, or make a tutorial how to start using minknow_api on web?
Also, I would like to get the right command for compiling the .proto files to javascript, as I think I miss some files because it outputs less files than the python files have.
Hope to get an answer soon, and thanks in advance.

Cors and minknow_api

Hello, I wanted to know if the web version of minknow_api supports CORS or is it not possible to use the minknow_api through different origins?

Query pore activity

Is there a way to query the pore activity? For instance, in this example I would like a way to get the attributes for [sequencing, pore, recovering, inactive, unclassified]. The goal is to be able to record the pore occupancy percentage, which should be (sequencing+recovering)/(sequencing+recovering+pore), so I can use a route if that is implemented elsewhere.

Set minimum chunk size for adaptive sequencing

I would like to perform adaptive sequencing ("Read Until") on a dataset with relatively short reads. As a result, if I want to gain significant benefit from selective sequencing, reads need to be ejected as soon as possible. In order to do so, I would like to reduce MinKnow's default read chunk size.

According to the comments in this repository, the configuration is stored in minknow/conf/app_conf. This is incorrect and should be updated; this information is now stored in minknow/conf/tuning_params.toml.

Would you additionally be able to explain the difference between the following configuration options in minknow/conf/tuning_params.toml:

• raw_data_intermediate
• raw_meta_data_intermediate
• read_data_intermediate
• event_data_intermediate

Are there any limitations I should be aware of in terms of how low I can set these parameters? I assume once the data packets are too fragmented it becomes difficult to write them in real time. It looks like the default values are already significantly lower than they used to be (400 samples, not 2000). Is that correct?

How to use the extra args?

My requirement is to make fast5 include fastq and trace table. By looking up the instructions, I understand that the fast5_out should be set out to True. Which of the following expression is correct in syntax?

extra_args=["--fast5_out=True"]
extra_args=["fast5_out=True"]
extra_args=["fast5_out=on"]
extra_args=["--fast5_out=on"]
extra_args=["--fast5_out"]

extra_args would be passed to protocol_args.extend(args).

I have failed many times . I would appreciate your help.

Messages with a `%' sign fail to send

Unescaped percentage signs (%) don't seem to be receivable by the MinKNOW API. We can send them without error but they don't appear in the UI, so some kind of silent failure?

from minknow_api.manager import Manager

m = Manager(host="localhost", port=9501, use_tls=False)
for pos in m.flow_cell_positions():
    print(f"Connecting to {pos.name}")
    conn = pos.connect()
    break

incoming_msgs = conn.log.get_user_messages()

for send in ["Hello, World!", "A \%", "B 100%", "C 100\%"]:
    resp = conn.log.send_user_message(severity=1, user_message=send)

for recv in incoming_msgs:
    print(recv)

Output:

time {
  seconds: 1625064325
  nanos: 677229000
}
severity: MESSAGE_SEVERITY_INFO
identifier: "script_user_info"
user_message: "Hello, World!"

time {
  seconds: 1625064325
  nanos: 677472000
}
severity: MESSAGE_SEVERITY_INFO
identifier: "script_user_info"
user_message: "A %"

time {
  seconds: 1625064325
  nanos: 677860000
}
severity: MESSAGE_SEVERITY_INFO
identifier: "script_user_info"
user_message: "C 100%"

---

System details:

MinKNOW: 21.05.8
minknow-core-gridion: 4.3.4
minknow-api: 4.2.4
python: 3.7.10

Example for stop_protocol

To start a protocol, it is sufficient that we do the following:

//Find a protocol to start from the available list
protocol_to_start = find_protocol(
    available_protocols,
    flow_cell="FLO-MIN106",
    kit="SQK-LSK108",
    experiment_type="sequencing")
print("Starting protocol: %s" % protocol_to_start.name)

// Construct a user info message to control the protocol group id
protocol_user_info = protocol.ProtocolRunUserInfo(
    # We need to specify StringValue as the group_id uses a nullable wrapper type
    protocol_group_id=StringValue(value="My Experiment Group")
)

// Construct a start protocol message to send to MinKNOW
start_request = protocol.StartProtocolRequest()

// Set the protocol identifier in the start message
/// Start the first protocol in the returned list

start_request.identifier = protocol_to_start.identifier
start_request.user_info.CopyFrom(protocol_user_info)

// Send the start protocol and wait for the response
started_protocol = stub.start_protocol(start_request)
print("Started protocol run_id: %s" % started_protocol.run_id)

Could you kindly give an example of how to stop the same protocol?
@jorj1988 @iiSeymour

Pausing a run?

Hi,

I just have a short question. Is it possible to pause a MinION run, let's say for 1 minute, and than resume the run by using API calls?

Thanks
Jens

read until examples

Could you please tell me the commands to operate read until? Any examples would also help. Also, should guppy be run on a CPU for read until for a minion?

How is Q-score calculated

Can you describe how the Q-score is calculated for the minimum Q-score filter and the Q-score plots?

I set the Q-score to 10, but I don't see how that shows up in the data. I would expect the Q-score for the reads to show a sharp cutoff at 10, but the distribution doesn't show a cutoff. I would also like to be able to calculate if the data passes Q20.

fastq_mean = numpy.mean(fastq.letter_annotations["phred_quality"])
fastq_median = numpy.median(fastq.letter_annotations["phred_quality"])
fastq_mode = scipy.stats.mode(fastq.letter_annotations["phred_quality"])[0][0]

Thanks!

remotely monitor the status of minknow protocol and run a custom script

Hi,

Thank you developing the those useful APIs for Miniknow. I have computer A connected with Minknow and a computer B used to process the data. I am interested in monitoring the sequencing status on computer B and run a custom script located in computer A to cleaned the data and transfer the data to computer B once the sequencing is completed. Then the data will be processed automatically on computer B with a pipeline. I was wondering which APIs I should use to implement the whole process?

Best,
ZL

Incomplete shutdown of MinKNOW experiment when initiating the call from the API (Windows)

Calling the shutdown function (with any of the three mode options) of the MinKNOW C++ API appears to cause the device to stop (no fan noise, red light on the device), yet in the latest UI the state is still "sequencing", and the voltage chart still shows a voltage being applied.

1. Is this a known issue?
2. How can we check if it's stopped for real, other than the UI?
3. Is there a code table for the device lights?

Thank you!

starting a run on a local MinION failed to find protocol

On a local machine (all changes suggested the other post were undone) I do have a Minon with ID MN32638 running. It contains a flowcell with the ID FAO64252 loaded with an DNA sample.
I tried to follow the code given in the header of start_protocol.py

The Minion is detected:
$python list_sequencing_positions.py --localhost --port 9501
usage: list_sequencing_positions.py [-h] [--host HOST] [--port PORT]
list_sequencing_positions.py: error: unrecognized arguments: --localhost
(gRPC) user@meqneuropat15:~/minknow_api/python/examples$ python list_sequencing_positions.py --host localhost --port 9501
Available sequencing positions on localhost:9501:
MN32638: running
secure: 8001
insecure: 8000

However adapting the code in the header does prompt "Failed to find protocol for position MN32638"

~/minknow_api/python/examples$ python start_protocol.py
--host localhost
--position MN32638
--sample-id "my_sample" --experiment-group "my_group"
--experiment-duration 1
--kit SQK-RBK004
--basecalling
--fastq --bam
Failed to find protocol for position MN32638
Requested protocol:
product-code: None
kit: SQK-RBK004
basecalling: True
basecall_config: None
barcode-kits: None
barcoding: False

Replacing the position with the flowcell id does not help.
Looking in start_protocol.py
I should provide additional info
protocol_info = protocols.find_protocol(
position_connection,
product_code=product_code,
kit=args.kit,
basecalling=args.basecalling,
basecall_config=args.basecall_config,
barcoding=args.barcoding,
barcoding_kits=args.barcode_kits,

I am not clear where to barcode ID code and other parameters requested. Or how to provide the requested protocol.

thanks
claus

The FAST5 is not dumped until I turn ON basecalling

WHen i sequence using the start and stop protocols via python, i do not see any FAST5's being dumped. But, when I navigate to the GUI and turn on basecalling, the FAST5's appear. How do I see the FAST5's when I run using python interface? @iiSeymour @jorj1988

Distinguish "Mux_Scan" and "Sequencing" state

Hi,

I just have a short question: How can I distinguish the current state of the sequencing run using the API? I would like to determine if the device is currently performing a "MUX_SCAN" or if the status has changed to "Sequencing". I'm actually using the Acquisition::MinknowStatus to see if the run has been started, is processing or is finished. But the status "Processing" is returned when the MinION performs a "MUX_SCAN". Any suggestions for a workaround are much appreciated.

Best regards
Jens

Question regarding stopped unblock calls

Hi,

I'm currently evaluating the adaptive sampling feature by using the minknow_api to send unblock messages to the MinKNOW GRPC server when running a bulk FAST5 playback. I recognized that some reads, for which I was sending unblock calls to MinKNOW, have not been unblocked. I also see some messages in the bream log file like the following:

[bream4.toolkit.procedure_components.sequencing_features.progressive_unblock: 105] - 2021-06-22 11:26:29,531 - INFO - Stopped 32 unblock calls

Is there a log file or a possibility to have a look at specific unblock calls and why they were stopped by MinKNOW? I would like to understand this issue. Is there something like a maximum number of unblock messages accepted by the GRPC server in MinKNOW?

Thanks
Jens

minknow_api onremote machines implemented?

Hello,
Many thanks for forwarding me here.
First step on the local linux machine worked fine and prompted:

~/minknow_api$ python ./python/examples/list_sequencing_positions.py --host localhost --port 9501
Available sequencing positions on localhost:9501:
MN32638: running
secure: 8001
insecure: 8000

However, issues started with the attempt to obtain a similar readout form an identical second machine in the same subnet running under 192.18.01.18.
On this machine MinKNOW GUI
Installed version: 4.1.22 is running and a Minion is attached to it via USB 3. This unit is “detected” by the local MinKNOW GUI.

Running on the first linux machine
~/minknow_api$ python ./python/examples/list_sequencing_positions.py --host 192.168.1.18 --port 9501

yields:
Traceback (most recent call last):
File "./python/examples/list_sequencing_positions.py", line 41, in
main()
File "./python/examples/list_sequencing_positions.py", line 23, in main
manager = Manager(host=args.host, port=args.port, use_tls=False)
File "/home/user/minknow_api/gRPC/lib/python3.7/site-packages/minknow_api/manager.py", line 108, in init
minknow_api.manager_service.GetVersionInfoRequest()
File "/home/user/minknow_api/gRPC/lib/python3.7/site-packages/grpc/_channel.py", line 946, in call
return _end_unary_response_blocking(state, call, False, None)
File "/home/user/minknow_api/gRPC/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{"created":"@1618939002.979941782","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":5419,"referenced_errors":[{"created":"@1618939002.979938522","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":397,"grpc_status":14}]}"

ping to 192.168.1.18 yields an immediate reply.

Different attempts and the content of manager.py and manager_service.py imply to me that remote hosts are (so far ?) not supported.

Could you please let me know if gRPC calls to remote machine via the minknow_api are supported?
Many thanks
Claus

possible to derive ID of Flowcell?

Hello,
is it currently possible to pass the ID of an inserted Flowcell into a variable?
Looking forward to your reply & best wishes
Claus

How to have a buik fast5 generated through the MinKNOW API?

Is there a way to activate the bulk fast5 output through this supplied version of the MinKNOW API? If not (yet) - where would we have to modify the code to have this generated by default? Any feedback would be highly appreciated.

How to get alignment info which is used in latest version of Minknow for visualzation of guppy real time alignment

Hi,
I would like to retrieve more info like alignemnt info as the visualization of guppy alignment in latest version of Minknow via minknow_api. I assume ideally that the design of Minknow follows the SoC (Separation of Concern) which makes Minknow GUI also access data via minknow_api, kind of 'eat your own dog food'. While I couldn't find such info in the code, maybe I miss something. Could someone help me on that? BTW, I am working on the integration some metrics to our own application and thanks a lot in advance.

longle flowcell not detectable

Dear Nanoporetech,

I am a MinION user. Recently, I have been trying flongle flow cells in our lab.
Our MinKNOW software (v22.05.5) always cannot detect the flongle flowcells (I've tried several flongles). However, I tried the adapter and the flongles with another GridION with another workstation (MinKNOW v.22.12.5) and that worked very well. The user of the working workstation told me that she met with the same issue before and it was solved by restarting MinKNOW. However, I failed to copy the troubleshooting method. Would you please let me know your suggestions?

Btw, I wondered if it is because of the minor difference of the MinKNOW version. I tried to update our MinKNOW version by the command "sudo apt install minion-nc", but it reported that minion-nc is already the newest version (22.05.5-1~bionic).

Looking forward to hearing from you.

Best,
Yingzi

Manager procedures not implemented?

Hi,

I see a strange behaviour when trying to use the Manager Service. I always get the status code 12 (=UNIMPLEMENTED) when executing list_devices as well as get_version_info. It seems they are not implemented on the server side. For Instance and Device I have no problems executing remote procedure calls. Is the Manager also deprecated?

Thanks in advance
Jens

ModuleNotFoundError: No module named 'minknow_api.examples'

Hi, I am new to the MinKNOW API. So far, I am able to list positions etc. but whenever I attempt to start a sequencing run with the start_protocol.py, I get this something like this:

Traceback (most recent call last):
  File "/home/rasmus/Desktop/minknow_api-master/python/minknow_api/examples/start_protocol.py", line 25, in <module>
    from minknow_api.examples.load_sample_sheet import (
ModuleNotFoundError: No module named 'minknow_api.examples'

What am I missing wrt the Module error?

Error when I run the example for start_protocol.py

My MinION-106 is connected to the machine and is detected when I use list_devices.py. I am trying to see if I can establish a communication channel to do a run until feedback to MinION. But, when I run start_protocol.py, I get the following error when I try to set sample id (note: the minion is loaded only with the configuration test cell):

Setting sample id:
Traceback (most recent call last):
  File "start_protocol.py", line 118, in <module>
    stub.set_sample_id(set_sample_id_request)
  File "/home/hariss/.local/lib/python3.6/site-packages/grpc/_channel.py", line 565, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/hariss/.local/lib/python3.6/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
        status = StatusCode.FAILED_PRECONDITION
        details = "Cannot set sample id during an acquisition period"
        debug_error_string = "{"created":"@1562026760.843265438","description":"Error received from peer ipv6:[::1]:8004","file":"src/core/lib/surface/call.cc","file_line":1046,"grpc_message":"Cannot set sample id during an acquisition period","grpc_status":9}"

But, even if I comment the sample id set-up line and proceed, the run fails to start and I get an unknown status back. Following is the error in this case:

Starting protocol: NC_48Hr_sequencing_FLO-MIN106_SQK-LSK108_plus_Basecaller
Traceback (most recent call last):
  File "start_protocol.py", line 148, in <module>
    started_protocol = stub.start_protocol(start_request)
  File "/home/hariss/.local/lib/python3.6/site-packages/grpc/_channel.py", line 565, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/hariss/.local/lib/python3.6/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Script '33fc31f4de8ed9c19afdd41af8069cf7d0477e8a-35c5ca6ae8f74dd9ac677c3d248847ce62fb89c8' failed to start"
        debug_error_string = "{"created":"@1562027313.360182885","description":"Error received from peer ipv6:[::1]:8004","file":"src/core/lib/surface/call.cc","file_line":1046,"grpc_message":"Script '33fc31f4de8ed9c19afdd41af8069cf7d0477e8a-35c5ca6ae8f74dd9ac677c3d248847ce62fb89c8' failed to start","grpc_status":2}"

I am wondering if this will be resolved when I load the flowcell? I want to do a stop_protocol after starting to see that my run until mini-model is successful.
@jorj1988 @iiSeymour

How to set initial "bias_voltage" or adapt it during a run?

Hi! I would like to set the bias voltage when launching a run from the API or - alternatively - use the option connection.device.set_bias_voltage(XXXX) for an ongoing run.

How can I do this? I have tried various ways but think I am not formatting my instruction properly. Any help or hint (in particular where to look into the code of the API) is highly appreciated.

SSL certificate unauthenticated

Hi ONT,

I am having an issue supplying th SSL certificate to start_protocol.py. I have set the `MINKNOW_TRUSTED_CA' environment variable within start_protocol.py to the certificate used by the MinKNOW app.

os.environ['MINKNOW_TRUSTED_CA'] = "/Applications/MinKNOW.app/Contents/Resources/conf/rpc-certs/ca.crt"

However when running start_protocol.py I receive the following error;

    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAUTHENTICATED
        details = "Bad metadata key"
        debug_error_string = "{"created":"@1643932207.912947000","description":"Error received from peer ipv6:[::1]:8001","file":"src/core/lib/surface/call.cc","file_line":1075,"grpc_message":"Bad metadata key","grpc_status":16}"

I am able to use the minion through the MiKNOW app which would be using the same certificate so I am unsure the reason for the discrepancy. I am running minknow_api 4.5.0 and MinKNOW 21.11.18. Any help/ insights would be greatly appreciated

Unable to connect to the RPC while using proxy

Dear @rmp @jorj1988 @glenveegee @cjw85 @sagrudd , first of all thank you for making such a beautiful API with great documentation. I am currently running MinKNOW behind a proxy server and trying to fetch the data using MinKNOW API. But the gRPC is throwing a proxy error and I am stuck at this stage.
So, I request you to please help me resolve this issue I will be very grateful.
For reference, I am adding the error message below:

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.10.78.61:3128: HTTP proxy returned response code 302"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:10.10.78.61:3128: HTTP proxy returned response code 302 {created_time:"2023-05-26T13:08:35.585583704+05:30", grpc_status:14}"

Access to flow cell check data

It would be very helpful if we can access (or control entirely) the flow cells check via API. Is that feasible? Even being able to scrape the historical data from the computer or via a web protocol would be great.

More examples, phisical device simulator

Hello,

I'm working with minknow_api right now, but unfortunately I don't currently have access to a physical sequencer.

I have a request if it would be possible to expand the example section, such as test_extract_run_statistics.py Such files are very helpful in understanding how the api works.

Alternatively, would there be an option to add a physical device simulator that could be queried using minknow_api and would return a set of statistics, e.g. random temperature values?

Best,
Wiktor