scilifelab/genologics

Python interface to the GenoLogics LIMS server via its REST API.

Scilifelab modules

Installation

Installation is as simple as

python setup.py install

If you are running several virtual environments, where one (e.g. devel) is used for development, you can install a development version by running

workon devel
python setup.py develop

Documentation

Docs are located in the doc directory. To install, cd to doc and run

make html

Documentation output is found in build.

Running the tests

The modules are shipped with a number of unit tests, located in the tests directory. To run a test, issue the command

python setup.py nosetests

or if you want to run individual tests, cd to tests and run (for example)

nosetests -v -s test_db.py

genologics's Issues

Split entities in two

Hi
I want to split entities.py into entities and descriptors
I don't think it should affect anybody as the descriptors should not be used outside of the entities.
I'm not sure what to do with SampleHistory: should it stay in entities?

Thank

"demux" end point

Hi. I need to use the multiplexing information end point "artifacts//demux". From going through the code it appears this is not yet present, unless I have missed something. Is this supported?

Consider switching to logbook

"Logbook should meet your needs and could better integrate with the rest of packages/scilifelab software. Please, give it a second look/read on the docs and compare it with the bcbio-nextgen logging module/code.". @brainstorm, I agree to this argument, you were just using different arguments before. I think the major argument is the integration with other scilifelab packages together with e-mail support. The code in EPP_logger will look rather similar.

versioning

Hi,

it's great to see some updated to this package! Good job 👍

I'm having some trouble understanding your versioning tho..

• In the package I see a reference to version "0.3.9" (https://github.com/SciLifeLab/genologics/blob/master/genologics/version.py)
• On PyPI the latest release (matching master I believe) is "0.3.11.post0"
• When I install the package my system tells me it's "0.0.0.post0"

So if I want to depend on the latest version or above, how should I do?

Update udfs with dates.

The way dates are handled by the udfdictionary is strange. I don't know how to update a date contained in a udf:

one_process.udf['Finish Date'] = one_process.udf['Finish Date']

gives:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-58-fc853b8bab69> in <module>()
----> 1 one_process.udf['Finish Date'] = one_process.udf['Finish Date']

/Users/johannesalneberg/.virtualenvs/genologics-lims/lib/python2.7/site-packages/genologics-v0.2.3_202_gd058acf-py2.7.egg/genologics/entities.pyc in     __get__(self, instance, cls)
    321     def __get__(self, instance, cls):
    322         instance.get()
--> 323         self.value = UdfDictionary(instance, udt=self._UDT)
    324         return self.value
    325 

/Users/johannesalneberg/.virtualenvs/genologics-lims/lib/python2.7/site-packages/genologics-v0.2.3_202_gd058acf-py2.7.egg/genologics/entities.pyc in __init__(self, instance, udt)
    171         self._udt = udt
    172         self._update_elems()
--> 173         self._prepare_lookup()
    174 
    175     def get_udt(self):

/Users/johannesalneberg/.virtualenvs/genologics-lims/lib/python2.7/site-packages/genologics-v0.2.3_202_gd058acf-py2.7.egg/genologics/entities.pyc in _prepare_lookup(self)
    218                 value = value == 'true'
    219             elif type == 'date':
--> 220                 value = datetime.date(*time.strptime(value, "%Y-%m-%d")[:3])
    221             self._lookup[elem.attrib['name']] = value
    222 

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_strptime.pyc in _strptime_time(data_string, format)
    452 
    453 def _strptime_time(data_string, format="%a %b %d %H:%M:%S %Y"):
--> 454     return _strptime(data_string, format)[0]

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_strptime.pyc in _strptime(data_string, format)
    326     if len(data_string) != found.end():
    327         raise ValueError("unconverted data remains: %s" %
--> 328                           data_string[found.end():])
    329     year = 1900
    330     month = day = 1

ValueError: unconverted data remains:  00:00:00

Using a tuple or a plain string does not work either.

Custom configuration path does not work

In #146 a flexible way declaring the configuration file location was implemented. Unfortunately, I can't get it to work.

This is how my (very simple) test script looks like:

from genologics import config

When I run this it immediately starts complaining about the missing config files.

/venv/lib/python2.7/site-packages/genologics/config.py:52: UserWarning: Please make sure you've created or indicated your own Genologics configuration file (i.e: ~/.genologicsrc) as stated in README.md
  warnings.warn("Please make sure you've created or indicated your own Genologics configuration file (i.e: ~/.genologicsrc) as stated in README.md")

I am using the latest (pip) version of the package. Any idea how to get this to work?

problem i copy_reference_genome.py

Får ett felmeddelande när jag försöker köra sciptet på clinical instansen av clarity lims.

Den hittar inte log-filen och anger sökvägen utan / i början

.genologicsrc filens innehåll
[genologics]
BASEURI=https://clinical-lims.scilifelab.se
USERNAME=anändare
PASSWORD=lösenord
[logging]
MAIN_LOG=/home/glsai/clinical_main_log.txt

Using python's "in" syntax with udf dictionaries

I tried using python's "in" syntax, i.e.

print 'Amount (ng)' in artifact.udf

This should return True or False, but results in the following error:

KeyError                                  Traceback (most recent call last)
<ipython-input-52-488d06a8dc5e> in <module>()
----> 1 print 'Amount (ng)' in artifact.udf

/Users/johannesalneberg/.virtualenvs/genologics-lims/lib/python2.7/site-packages/genologics-0.2.3-py2.7.egg/genologics/entities.pyc in __getitem__(self, key)
    222
    223     def __getitem__(self, key):
--> 224         return self._lookup[key]
    225
    226     def __setitem__(self, key, value):

KeyError: 0

I don't know how the 0 ends up there.. This would be a nice feature to have, especially since the LIMS seems to erase the entire field from an artifact if the user erases the value of it... Might be something to contact Genologics about as well?

Edit UDF config: "Field is required and must be filled in"

Hej,

is it possible to edit the option "Field is required and must be filled in" through the Python API?

We couldn't figure out which property on the Udfconfig class...

migration from python2 to python3

Hi I just stumbled upon your clarity REST client interface and thought it could be very useful.
However all our code is in python3
I was wondering if there was any interest in using a python3 version if we make the migration effort

Cheers
Tim

versionning issue

Hi,

We have installed genologics 0.3.8 on all our servers (same as my computer now).

On our test us :
pip freeze | grep genologics ====> 0.3.8.post0
Line 21 of /usr/lib/python2.6/site-packages/genologics/lims.py ====>if version_info[0]
and that works fine
(same as my computer)

On our prod uk :
pip freeze | grep genologics ===> genologics==0.3.8.post0
line 21 of file /usr/lib/python2.6/site-packages/genologics/lims.py ===> if version_info.major == 2:
There when I run my script I get the following error :

get the following error with the generate_metadata.py script :
File "/usr/lib/python2.6/site-packages/genologics/lims.py", line 21, in
if version_info.major == 2:
AttributeError: 'tuple' object has no attribute 'major'

Would you have any idea why that is?

Thanks a lot for your help.
Catherine

Installing from GitHub doesn't work

It seems to have something to do with how you are reading in the version of package during install

(develop)vagrant@vagrant-ubuntu-trusty-64:~$ pip install https://github.com/SciLifeLab/genologics/tarball/master
Collecting https://github.com/SciLifeLab/genologics/tarball/master
  Downloading https://github.com/SciLifeLab/genologics/tarball/master
     / 65kB 351kB/s
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 20, in <module>
      File "/tmp/pip-RMxJwu-build/setup.py", line 13, in <module>
        with open(version_py, 'r') as fh:
    IOError: [Errno 2] No such file or directory: '/tmp/pip-RMxJwu-build/version.py'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-RMxJwu-build

Mailing list for errors in EPP scripts

Add possibility to send out emails when errors occur in EPP scripts.

Caching issues

I'm having issues with specifically UDF caching in persistent sessions
It seems when I'm fetching information on the first sample everything works as expected. Then when I get information for subsequent samples the interface retains the UDF:s for the first sample.

Directly accessing the API returns the expected values

Caching try/except mechanism in entities.py

@mayabrandi pointed out that without commenting the try/except statement in:

https://github.com/SciLifeLab/genologics/blob/master/genologics/entities.py#L413

Her scripts do not work properly. I guess that this is another bit of over-defensive programming... if the element is not found, should it be handled further downstream, @pekrau?

Same issue in:

https://github.com/SciLifeLab/genologics/blob/master/genologics/entities.py#L309

Don't allow instance creation if id doesn't exists

I've found that in many entities, you can create an instance of the class even if the id that you pass to the constructor doesn't exists. It's better to see an example to understand what I mean:

from genologics.lims import *
from genologics.config import BASEURI, USERNAME, PASSWORD

lims = Lims(BASEURI, USERNAME, PASSWORD)
p = Process(lims, id='this_id_does_not_exist_for_sure')

This does not raise any exception. Instead, an exception is thrown anytime I want to check an attribute.

In [6]: p.technician
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-6-2025ab4047ff> in <module>()
----> 1 p.technician

/Users/guillem/.virtualenvs/master/lib/python2.7/site-packages/genologics-0.2.1-py2.7.egg/genologics/entities.pyc in __get__(self, instance, cls)
    361
    362     def __get__(self, instance, cls):
--> 363         instance.get()
    364         node = instance.root.find(self.tag)
    365         return self.klass(instance.lims, uri=node.attrib['uri'])

/Users/guillem/.virtualenvs/master/lib/python2.7/site-packages/genologics-0.2.1-py2.7.egg/genologics/entities.pyc in get(self, force)
    485         "Get the XML data for this instance."
    486         if not force and self.root is not None: return
--> 487         self.root = self.lims.get(self.uri)
    488
    489     def put(self):

/Users/guillem/.virtualenvs/master/lib/python2.7/site-packages/genologics-0.2.1-py2.7.egg/genologics/lims.pyc in get(self, uri, params)
     50                          auth=(self.username, self.password),
     51                          headers=dict(accept='application/xml'))
---> 52         return self.parse_response(r)
     53
     54     def put(self, uri, data, params=dict()):

/Users/guillem/.virtualenvs/master/lib/python2.7/site-packages/genologics-0.2.1-py2.7.egg/genologics/lims.pyc in parse_response(self, response)
     98             if node is not None:
     99                 message += ' ' + node.text
--> 100             raise requests.exceptions.HTTPError(message)
    101         return root
    102

HTTPError: 404: Process Not Found

Shouldn't the exception be thrown in the moment of the creation of the class?

Log is cluttered with requests 'starting new https connection'

Should be able to reuse the connection internally and thus not restart the connection for every request.

Batch create samples

Do you support the batch creation of samples via: /{version}/samples/batch/create?

Sample creation

When creating a new sample, how can you pass initial UDFs?

We have set up LIMS to require some UDFs on sample level so I can't create the sample before setting the correct UDFs.

I tried to look into setting up a UdfDictionary but it requires an instance of the Sample which is what I'm trying to create 😅

[Function request] A way to download tarfiles.

Hello and thanks for your immensely useful library!

I've found .get_file_contents() to be especially useful. Do you have any plans to make a way to download non-string type data. (i.e. contents for tarfiles?)

I'm a novice, but have tried tweaking .get_file_contents() to work on tarfiles.
The only difference with my code (at end of this messsage) is that

• it makes a request (r) and returns r.raw

whereas your code, .get_file_contents():

• returns r.text

I find that my code successfully writes .tar.gz archives most of the time, but is also prone to silent failure (e.g. occasionally writes archives that are corrupted)--for reasons that I do not understand. If you have any advice on how to download tar.gz files from Clarity, I would greatly appreciate it!

~Greg

The code:

from urllib.parse import urljoin
import tarfile

PATH_tgz = "example.tar.gz"
DIR_download = "./directory/"

make a request

r = API.request_session.get(URL, auth=(API.username, API.password), timeout=15, stream=True)
API.validate_response(r)

Write tgz and extract it

with open(PATH_tgz, 'wb') as f: f.write(r.raw.read())
FILE_tgz = tarfile.open(PATH_tgz)
FILE_tgz.extractall(DIR_download)
FILE_tgz.close()

Should "required" empty fields crash the client?

@pekrau, Validations are a bit too stringent?:

https://github.com/SciLifeLab/genologics/blob/master/genologics/lims.py#L100

$ python examples/attach_delivery_report.py

Traceback (most recent call last):
  File "attach_delivery_report.py", line 31, in <module>
    project.put()
  File "/Users/roman/.virtualenvs/genologics/lib/python2.7/site-packages/genologics-0.2-py2.7.egg/genologics/entities.py", line 492, in put
    self.lims.put(self.uri, data)
  File "/Users/roman/.virtualenvs/genologics/lib/python2.7/site-packages/genologics-0.2-py2.7.egg/genologics/lims.py", line 62, in put
    return self.parse_response(r)
  File "/Users/roman/.virtualenvs/genologics/lib/python2.7/site-packages/genologics-0.2-py2.7.egg/genologics/lims.py", line 100, in parse_response
    raise requests.exceptions.HTTPError(message)
requests.exceptions.HTTPError: 400: Invalid project. Portal ID is a required field and must have a value

Create new samples

Hi guys!

I was wondering how it would be possible to create new samples using the REST API. I got it working using this package but it required some hacking. Perhaps I'm missing something.. Are you using the API to do something like that?

This is briefly my approach:

1. The API needs a "location" reference with a container on sample level so I borrow some code from create and add the extra stuff in the XML code:

lims_sample = Sample(lims=lims, _create_new=True)
uhmmm = "{}:{}".format(Sample._PREFIX, Sample.__name__.lower())
lims_sample.root = ElementTree.Element(nsmap(uhmmm))

lims_sample.name = name
lims_sample.project = lims_project
location = ElementTree.SubElement(lims_sample.root, 'location')
ElementTree.SubElement(location, 'container', dict(uri=lims_container.uri))
position_element = ElementTree.SubElement(location, 'value')
position_element.text = position

2. To save a sample you need to use the "smp:samplecreation" tag (or whatever it's called) so I have to modify the data string before posting:

data = lims.tostring(ElementTree.ElementTree(lims_sample.root))
data = data.decode('utf-8').replace('smp:sample', 'smp:samplecreation')
lims_sample.root = lims.post(uri=lims.get_uri(Sample._URI), data=data)
lims_sample._uri = lims_sample.root.attrib['uri']

Just wanted to share the research I did to make this work. If it's something you are interested in adding support for I'd be happy to work with you. Right now I don't feel like I understand the package well enough to suggest how best to add it tho.

Bug: AttributeError: 'ReadTimeout' object has no attribute 'message'

I think I found something to fix that would make your great library even better :)

Last part of my stacktrace:

File "/Users/patrik.grenfeldt/Documents/GitHub/cg/cg/apps/lims/order.py", line 32, in save_containers
    container_map[lims_container.name] = lims_container
  File "/Users/patrik.grenfeldt/opt/anaconda3/envs/dev38/lib/python3.8/site-packages/genologics/descriptors.py", line 54, in __get__
    instance.get()
  File "/Users/patrik.grenfeldt/opt/anaconda3/envs/dev38/lib/python3.8/site-packages/genologics/entities.py", line 289, in get
    self.root = self.lims.get(self.uri)
  File "/Users/patrik.grenfeldt/opt/anaconda3/envs/dev38/lib/python3.8/site-packages/genologics/lims.py", line 87, in get
    raise type(e)("{0}, Error trying to reach {1}".format(e.message, uri))
AttributeError: 'ReadTimeout' object has no attribute 'message'

Best wishes
Patrik

Main log file for all EPP scripts.

Add a main log file for all EPP scripts stored on the genologics server. To simplify debugging and tracking. Maybe an email when errors occur?

SSL error

We are using LIMS with encrypted (https) traffic. Does the Python wrapper support this?

The wrapper/API works fine when fetching the initial resource with e.g. classes Project/Researcher. But in the next step, when trying to retrieve additional information about a specific resource I get this error:

SSLError: hostname 'clinical-lims.scilifelab.se' doesn't match either of '*.claritylims.com', 'claritylims.com'

I've traced the issue to this:
When performing GET requests without a specific port everything works. But the API then appends the port (e.g. https://clinical-lims.scilifelab.se:8443) which yields the error above.

I'm using the following configs:

[genologics]
BASEURI=https://clinical-lims.scilifelab.se
USERNAME=<REDACTED>
PASSWORD=<REDACTED>
[logging]
MAIN_LOG=~/Projects/lims/my_main_log_file

I guess a quick-and-dirty fix would be to always ignore/remove the port-number internally. Or is this a more basic Genologics API issue that they should fix?

separating core API from site specific logic

I was wondering what people though about the fact that the current repo contains the core library in the genologics directory and some scripts that are specific to SciLifeLab in the scripts.
My preference would be to separate the core code from any logic specific to a site.
This would allow the module to be more stable and easier to understand.
However I realise that there might be lots of things depending on the scripts and it might be non trivial to move them somewhere else.

Is it something you'd be willing to do?

python 3 compatibility

Hi--

I'm testing this in a python3 environment and am running into some issues.

I was wondering if this package is currently intended to be compatible with python3 and if not, if you intend to make it so? (I noticed this line: https://github.com/SciLifeLab/genologics/blob/master/genologics/lims.py#L15)

Thanks!

Make this repository the main one

Hi,
The repository that this repository was forked from only has two commits. Would it be possible to make this repository the "main" one. For CG this would mean fewer warnings about using a fork-repository as a dependency and for you it might raise the "importance" of this repository and make it easier to find, now it looks like a fork from something original.

UDF handling broken for Python 3.9+ by xml.etree API change

The getchildren() method has been removed from xml.etree from python 3.9 (https://docs.python.org/3.8/library/xml.etree.elementtree.html#xml.etree.ElementTree.Element.getchildren)
resulting in the error

File ".../lib/python3.10/site-packages/genologics/descriptors.py", line 200, in _update_elems
for elem in self.rootnode.getchildren():
AttributeError: 'xml.etree.ElementTree.Element' object has no attribute 'getchildren'

Seems to be an easy fix as getchildren() is used in only two places

/Users/egow/.pyenv/versions/3.10.9/envs/LIMS-15796/lib/python3.10/site-packages/genologics
descriptors.py
for node2 in node.getchildren():
for elem in self.rootnode.getchildren():

PR to follow

Adding a numeric UDF returns ElementTree error

I'm encountering an ElementTree Error when adding a numeric UDF to an analyte that prevents me from being able to "put" numeric udfs onto analytes in Clarity. To demonstrate:
Input:

p = Process(lims, id = ppid)
c = p.all_inputs()[19]
c.udf['Concentration'] = 9999
print(tostring(c.root))

Returns:

//anaconda/envs/py3/lib/python3.5/xml/etree/ElementTree.py in _raise_serialization_error(text)
   1054 def _raise_serialization_error(text):
   1055     raise TypeError(
-> 1056         "cannot serialize %r (type %s)" % (text, type(text).__name__)
   1057         )
   1058 

TypeError: cannot serialize b'9999' (type bytes)

I'm able to add text string UDFs without a problem. When I try to enter the numeric udf as a text string, I of course get a TypeError from descriptors.py:

c = p.all_inputs()[19]
c.udf['Concentration'] = "9999"

Returns:

//anaconda/envs/py3/lib/python3.5/site-packages/genologics/descriptors.py in __setitem__(self, key, value)
    233             elif vtype == 'numeric':
    234                 if not isinstance(value, (int, float)):
--> 235                     raise TypeError('Numeric UDF requires int or float value')
    236                 value = str(value)
    237             elif vtype == 'boolean':

TypeError: Numeric UDF requires int or float value

Am I doing something wrong here? Many thanks for your help, this is a great package!

Change to StepActions

Hi,
I need to rework the way Step stores actions and that might results in backward incompatible changes to the API.
I was wondering how much this would affect you and the potential users of the API as it seems to be still work in progress
Cheers
Tim

questions about BASEURL in config file

Hi all,
I apologize if this is not the place to ask questions. If so, please direct me to the right place. I found my BASEURL has an extra "clarity/profile" for some reason. It returns

In [3]: BASEURI
Out[3]: 'https://mako.claritylims.com/clarity/profile'

where it should be just
https://mako.claritylims.com/

My genologics.conf file looks ok. Here's what it looks like.

[genologics]
BASEURI=https://mako.claritylims.com:8443
USERNAME=xxxx
PASSWORD=XXXXXXXXXXXX
[logging]
MAIN_LOG=/Users/jack/Applications/log

Has any of you seen this yet? Any comments are very appreciated.

Logger causes previous log to be overwritten.

1. Check how the previous log can be loaded.
2. Append the new log to this file.

Testing support

I'm trying to figure out the best way to test/mock e.g. enteties from the package

I want to be able to run the tests without connecting to a LIMS.
Can I for example create a Sample instance that doesn't require a connection?

error installing genologics on windows machine

My colleague Bradley wants to learn using the Sci Life api.
He's installed python on his windows machine, and he's done pip install genologics but gets the following error :

C:\Python27\Scripts>pip install genologics
Collecting genologics
Using cached genologics-0.3.9.post0.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "c:\users\bradleyd\appdata\local\temp\pip-build-krqpil\genologics\setup.py", line 11, in
version = subprocess.Popen(["git", "describe", "--abbrev=0"],stdout=subprocess.PIPE, universal_newlines=True).communicate()[0].rstrip()
File "c:\python27\lib\subprocess.py", line 710, in init
errread, errwrite)
File "c:\python27\lib\subprocess.py", line 958, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified

Could it be that if system is windows then setup.py script need to give subprocess.Popen the argument :
shell = True
See : http://stackoverflow.com/questions/15573054/cant-make-django-less-work-on-windows

I'm just asking as Bradley seem to be able to install other python libraries just fine (eg. paramiko)

I sent you an email about it and this was your answer :

This line assumes that you have git installed, as it is part of our version control.
As we do not use Windows anywhere in our lab, this wasn't a problem. Please fill a github issue so i can take care of it properly.

Thanks a lot! :-)

New release on pip?

Are you guys planning a new release on the cheese shop? 😃

I'm using some new functionality and it would be nice to just point to a pip release, it's fine if you just upload it under a beta tag

Create new lab

Hi,
I'm trying to use the Lab.create() function to create a new lab (account). The new account is being made but all the billing-address & shipping-address fields don't get filled in. Am I missing something obvious or is creating a new lab not yet fully supported ? Below the code I'm using:

    account = Lab.create(
        lims,
        name=account_info['account_name'],
        website=account_info['account_website'],
        udf={'BudgetNrs' : account_info['account_BudgetNrs']},
        billing_address={
            'street' : account_info['billing_street'],
            'city' : account_info['billing_city'],
            'state' : account_info['billing_state'],
            'country' : account_info['billing_country'],
            'postalCode' : account_info['billing_postalCode'],
            'institution' : account_info['billing_institution'],
            'department' : account_info['billing_department']
        },
        shipping_address={
            'street' : account_info['shipping_street'],
            'city' : account_info['shipping_city'],
            'state' : account_info['shipping_state'],
            'country' : account_info['shipping_country'],
            'postalCode' : account_info['shipping_postalCode'],
            'institution' : account_info['shipping_institution'],
            'department' : account_info['shipping_department']
        }
    )

account_info <- a dictionary containing all necessary info

Best,
Sander

Meetup at Genologics UGM in barcelona

I'm not sure it is the best place to discuss this but a colleague and I will be in the UGM in May and I think this is a good opportunity to meet with developer and users of this API and discuss future developments.
I'll be in Barcelona from the 18th to the 21th of May and am happy to meet during or outside the UGM

Cheers
Tim