diamondlightsource/synchweb

ISPyB web application

synchweb's Issues

Auto-processing ticks and crosses absent in some views, not others

In certain views, the ticks, crosses and spinning cogs that signify success/failure/progress of auto-processing jobs are missing and replaced with placeholder text. In other views, the icons are present as expected.

On beamline I19 at DLS, we have recently changed the auto-processing recipes that we run. In certain circumstances, there are now more auto-processing jobs than there were before. This bug seems coincident with our changes, though I can see no reason why our changes should have caused this change in behaviour.

As an example, here's the landing page for a recent visit, showing the missing ticks/crosses at the bottom-right:

Here's a data collection group view from that visit, showing the same problem:

On the other hand, here's the sample changer screen, with the first sample selected (I can't find a way of generating a permalink for this), showing the ticks and crosses as expected:

And here's the auto-processing status summary, which is also showing ticks and crosses:

Better use of browser real-estate?

Would it be possible to enlarge the used area for

• the main overview page: it seems to be hard-wired to use only 83.333% of the width with empty grey areas to the left and right
• the "View" pop-up window (which is both limited and fixed in width/height): it could easily use more width

I would find it even better, if the View pop-up windows would allow re-sizing when encountering HTML files (while starting from the currently computed size as a sensible initial default). I find the strict and enforced fixedwidth property rather limiting and odd in the context of looking at a HTML file.

Should document ispyb-database version requirement

SynchWeb should clearly document which version of ispyb-database it requires (e.g., "ispyb-database 1.20.0 or later"). This should be done in the same way that most programs and libraries document their dependencies. Without this, I'm left guessing and trying things like using the latest from the master branch of the SynchWeb and ispyb-database repos, or using snapshots from the SynchWeb and ispyb-database repos that are approximately from the same point in time.

Magnifying glass icon should mean search, not view

In the Dewar Details section of the Shipment view, each container in a dewar is listed with three icons: a printer, a magnifying glass, and an arrow cross. The magnifying glass has a tool tip that says, "Click to View Container". This is confusing UI design. A magnifying glass always means search, not view. The magnifying glass icon should be changed in this view, and in any other place where it is used to mean view, to something else. In this case, perhaps it should be changed to an icon that looks like a container. Or maybe the icon should just be removed, and instead the user should click on the list item or the name of the container to view it.

Bulk sample update is difficult to use

Chrome, Version 112.0.5615.140 (Official Build) (64-bit)

In the Container view, e.g., https://ispyb.diamond.ac.uk/containers/cid/260277, there is a table of samples within the puck. There is a bulk-populate form above this table. It has some usability issues.

Possible to update in intermediate state

There can be a delay of several seconds between selecting a column and having it highlighted and populated with the new values. This is easily enough time to select a column, select a value, and hit Update Samples. It is possible to submit the update form during this delay, but it does not update the underlying records, despite the "successfully updated" message appearing. I assume that it's "updating" with the values that are in the table when the AJAX request is sent, rather than with those selected in the bulk update form.

It's also possible to change and submit the bulk update form after the AJAX update has been sent, but before it completes, further adding to the confusion.

Ideally, the table would be populated with the updated values so quickly as to render it impossible to submit the form before this happens. Failing that, it should not be possible to submit the form until the table updates, and certainly not while the AJAX update is in progress. It'd be nice to have some visual indication, whether a spinner or just disabling the form, to show that UI and AJAX updates are in progress.

Singular v plural

The success message refers to "sample", singular, when all samples in the puck were (or should have been) updated.

Insufficiently clear notification of errors

"Successfully updated" appears but the fields are not updated, and it's not obvious that there was an error. For example, "Screening Method" can't be set to "better than" without a minimum resolution value having been set:

To reproduce:
Open the Unattended Data Collection tab of the table. Note that "Minimum resolution" column contains no values.
Select "Screening Method" and "Better than" in the bulk update form.
Wait for table fields to be highlighted.
Click "Update samples".

Expected:
Fields are saved.
Green "successfully updated" bar appears at top of window.
(or)
Fields are not saved
Red error message appears at top of window
Green "successfully updated" does not appear

Actual:
Fields are not saved.
Green "successfully updated" bar appears at top of window.
"Update samples" button gains a thin red border.
Red "minimum resolution must be set" errors appear out of sight below table.
Error box disappears, often before you can scroll down to read it (if you know it's there)

PHP requirements?

Could you please make clear in the README.md the requirements for PHP?

5.x and 7.0 are EOL as of Jan 2019.
7.1 is currently only receiving security fixes.
7.2 and 7.3 are the actively supported versions.

Thanks

Clicking non-mx Proposal results in no Proposal being selected

Steps to Reproduce

1. Create a staff user who has only a cm Proposal.
2. For a regular user with no Proposals, create two: one with code sw and the other with code mx.
3. Log in as the staff user.
4. Click the Home button at the top.
5. Click Proposals.
6. Click the sw Proposal you just created for the regular user.

Expected Behavior

The Visit List view displays "No visits found", but the Proposal is selected, and you can click the Proposal's "sw[number]" drop-down menu to get the View All Data, Visits, Calendar, etc., menu items.

Actual Behavior

The Visit List view displays "No visits found", but the Proposal is not selected, and the selected Proposal drop-down menu that would have been labeled "sw[number]" instead is labeled "No Proposal" and does not display a drop-down menu when clicked.

Notes

Clicking the regular user's mx Proposal works fine; the problem only occurs when the user's non-mx Proposal is clicked.

Reprocessing - `POST` to `api/process/sweeps` fails

A POST to api/process looks to work correctly.

Request: {"DATACOLLECTIONID":"6037520","DISPLAYNAME":"Xia2 3dii","RECIPE":"xia2-3dii","prop":"mx21035"}
Response: {"PROCESSINGJOBID": 80}

However, the subsequent POST to api/process/sweeps fails.

Request: [{"PROCESSINGJOBID":80,"DATACOLLECTIONID":"6037520","STARTIMAGE":1,"ENDIMAGE":3600,"prop":"mx21035"}]
Response: {"status":400,"message":"No processing job specified"}

So either something is wrong with the format of the Request? Maybe the extra square brackets? Or maybe something else is happening in the php code on the SynchWeb server that is not properly parsing the response and adding a new sweep of data for reprocessing.

Any help in debugging this would be greatly appreciated.

Container > Extra Fields button shows '+' icon even when toggling off

In the Shipments > [Shipment Name] > Containers > Add Container view, there is an Extra Fields button that shows extra fields for the samples in the container:

But after pressing it to show the extra fields, you can press it again to hide the extra fields, but the icon is still the plus sign ('+') when you press it to hide the extra fields which is weird.

The plus icon ('+') should change to a minus icon ('-') when the extra fields are displayed.

Or the button could be changed to a "Show extra fields" checkbox, or perhaps a "Show extra fields" switch (if the UI toolkit supports it).

Enabling PHP DB debugging breaks Shipment view

If I enable DB debugging in the _get_all_containers function in api/Page/Shipment.php by uncommenting the line

//$this->db->set_debug(True);

the Shipment view breaks: selecting a dewar in the Shipment Contents section results in no containers being displayed in the Dewar Details view even though the selected dewar has containers.

Can assign >1 container to same location in scanned Assign Containers view

It's possible to assign more than one container to the same sample changer location in the scanned Assign Containers view (e.g., assign/scan/IMCA-CAT), but that should not be allowed. Server-side validation should be added to prevent this. Here's one way to reproduce the problem:

1. Open two web browser windows.
2. Log in to SynchWeb in both windows, and open the scanned Assign Containers view in both.
3. In the first window, assign a container to an empty sample changer location.
4. In the second window, assign a different container to the same sample changer location.
5. (Now there are two containers assigned to the same sample changer location in the DB.)

I didn't test other assign-containers views, but it seems likely that they could have a similar problem.

Editing Registered Container field in Container view is pain when list is large

In the Container view, when editing the Registered Container field, the drop-down menu is very long for some users with many registered containers, and it is not sorted alphanumerically. One user has 171 registered containers, and it's difficult to find the right container in the drop-down because there are so many containers and they are not in alphanumeric order. It would be nice if the drop-down were instead a combo box so that the user could type in part, or all, of the barcode to search for the registered container they want to select. Also, the list should be sorted alphanumerically in either case (i.e., drop-down or combo box).

User Office doesn't know whether Dewar booked in iSpyB for session

The UO expenses procedure states that:

Diamond will cover the cost of shipping up to two dewars of your samples (per 8 hour shift) to beamlines and eBIC microscopes if your establishment is in the UK or EU. Details of the shipping arrangements can be found here.

  Where you ask Diamond to cover the costs of shipping samples and this is authorised by us, one      person fewer (than outlined in section 2 above) will be financially supported by us.

Currently UO can't see whether Dewar shipping is being booked to ensure limits to investigator T&S is applied.

Incorrect client/src/index.php HtmlWebpackPlugin template

It is my understanding that HtmlWebpackPlugin is used at build time to generate client/index.php from client/src/index.php. If I'm understanding this correctly, then it looks like the HtmlWebpackPlugin template in client/src/index.php is wrong in that it uses the "evaluate" delimiter <% in some places which results in evaluation at build time but not at run time. For example, if ga_ident is not the empty string at build time, the Google Analytics functionality will always be active even if ga_ident is later changed at run time to the empty string in the client's config:

<% if (htmlWebpackPlugin.options.jsonConfig.ga_ident) { %>
<script type="text/javascript">
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
 
  ga('create', '<%= htmlWebpackPlugin.options.jsonConfig.ga_ident %>', 'auto');
  ga('send', 'pageview');
</script>
<% } %>

Similarly, if the client config has a maintenance message enabled at build time, it will always be enabled with no way to disable it at run time:

<% if ( htmlWebpackPlugin.options.jsonConfig.maintenance ) { %>
    <a class="icon"><i class="fa fa-2x fa-home"></i>&nbsp;</a>
<% } %>

and:

<% if ( htmlWebpackPlugin.options.jsonConfig.maintenance ) { %>
    <h1>Scheduled Maintenance</h1>
    <p><%= htmlWebpackPlugin.options.jsonConfig.maintenance_message %></p>
    <br />
<% } %>

Login page has Logout button

The login page (e.g., https://ispyb.diamond.ac.uk) has a Logout button, but that's weird because the user hasn't logged in yet. The Logout button should only be displayed if the user is logged in, and it should not be displayed on the login page.

making code portable

I'm sure you are already aware of this, and I don't expect that it is super high on your todo list, but there are a few hardcoded DLS-specific things in the synchweb code. This is something you might want to address in order to make synchweb more portable.

For example the _get_archive function in class.page.php has a call to what looks like your facility EPICS server (archiver.pri.diamond.ac.uk) to get beam parameters etc. Some of these calls cause php errors:

ErrorException: sizeof(): Parameter must be an array or an object that implements Countable in /var/www/sites/synchweb/api/includes/pages/class.vstat.php:430
Stack trace:
#0 [internal function]: Slim\Slim::handleErrors(2, 'sizeof(): Param...', '/var/www/sites/...', 430, Array)
#1 /var/www/sites/synchweb/api/includes/pages/class.vstat.php(430): sizeof(NULL)
#2 [internal function]: Vstat->_pies('cm1234-1')
#3 /var/www/sites/synchweb/api/lib/Slim/Route.php(462): call_user_func_array(Array, Array)
#4 /var/www/sites/synchweb/api/lib/Slim/Slim.php(1326): Slim\Route->dispatch()
#5 /var/www/sites/synchweb/api/lib/Slim/Middleware/Flash.php(85): Slim\Slim->call()
#6 /var/www/sites/synchweb/api/lib/Slim/Middleware/MethodOverride.php(92): Slim\Middleware\Flash->call()
#7 /var/www/sites/synchweb/api/lib/Slim/Middleware/PrettyExceptions.php(67): Slim\Middleware\MethodOverride->call()
#8 /var/www/sites/synchweb/api/lib/Slim/Slim.php(1271): Slim\Middleware\PrettyExceptions->call()
#9 /var/www/sites/synchweb/api/includes/class.dispatch.php(64): Slim\Slim->run()
#10 /var/www/sites/synchweb/api/index.php(95): Dispatch->dispatch()
#11 {main}

Clicking Proposal from Proposals view with session timeout does not resume correctly

Steps to Reproduce

1. Go to the Proposals view and wait for your session to time out such that whatever you do next will redirect you to the SynchWeb log-in view.
2. Click a Proposal. Since your session has timed out, you will be taken to the log-in view.
3. Enter your log-in credentials and click Login.
4. You are taken to the Visit List view.

Expected Behavior

The Visit List view displays the list of visits.

Actual Behavior

The Visit List view displays a busy indicator and does not display the list of visits.

Would it be possible to remove `client/src/js/config.json` ?

I use Ansible to deploy SynchWeb to our server and I make changes to client/src/js/config.json during the process. Since this file is tracked in git, this causes all subsequent Ansible tasks to fail.

The client/src/js/config_sample.json file is still there for people to have a reference to start from, but maybe we could remove the DLS-specific client/src/js/config.json version?

This would follow the pattern in api directory where there is no api/config.php. One is expected to make a copy of api/config_sample.php and customize it to your site.

I can always use "force" during the git checkout step so this isn't a huge deal.

Add New Container view allows creating samples without protein

I've been told that a sample must always have a protein assigned, but the "Add New Container" view (i.e., /containers/add/did/<id>) allows creating samples without a protein.

Steps to reproduce:

1. Click the "Add Container" button from the Shipment view.
2. Select a container from the Registered Container drop-down menu.
3. Enter x1 in the Name column of sample location 1.
4. Click the "Clone from First Sample" button.
5. Click the "Add Container" button.

Actual Result:

It creates the container with the samples with no proteins assigned.

Expected Result:

It should produce a validation error that no protein is selected from the Protein Acronym drop-down for the samples.

Provide the full path on the disk for autoprocessing output

Hi,

I would like to ask if it is possible to add the autoprocessing full path (including the uuid from the cluster pid) with in the API

queryURL: https://ispyb.diamond.ac.uk/api/processing/?prop=

Current response:

{
    "TYPE": "autoPROC+STARANISO",
    "IMAGESWEEPCOUNT": "1",
    "PROCESSINGSTATUS": "1",
    "PROCESSINGMESSAGE": "processing successful",
    "DCCOUNT": "1",
    "PROCESSINGJOBID": "########",
    "BEAM": {
        "XBEAM": "159.03",
        "YBEAM": "166.58",
        "REFINEDXBEAM": "166.64",
        "REFINEDYBEAM": "159.10"
    },
    "SHELLS": {
        "outerShell": {
            "CCHALF": "0.7",
            "CCANOMALOUS": "0.1",
            "ANOMALOUS": null,
            "TYPE": "autoPROC+STARANISO",
            "NTOBS": "8253",
            "NUOBS": "184",
            "RLOW": "2.79",
            "RHIGH": "2.52",
            "RMEAS": "8.357",
            "RMERGE": "8.263",
            "COMPLETENESS": "63.7",
            "ANOMCOMPLETENESS": "64.5",
            "ANOMMULTIPLICITY": "25.8",
            "MULTIPLICITY": "44.9",
            "ISIGI": "0.7",
            "RESISIGI": "0.00",
            "PROCESSINGSTATUS": "1",
            "PROCESSINGMESSAGE": "processing successful",
            "DCCOUNT": "1",
            "AUTOMATIC": "1",
            "INDEX": 0
        },
        "innerShell": {
            "CCHALF": "1.0",
            "CCANOMALOUS": "0.1",
            "ANOMALOUS": null,
            "TYPE": "autoPROC+STARANISO",
            "NTOBS": "7492",
            "NUOBS": "184",
            "RLOW": "28.59",
            "RHIGH": "4.97",
            "RMEAS": "0.063",
            "RMERGE": "0.062",
            "COMPLETENESS": "99.5",
            "ANOMCOMPLETENESS": "100.0",
            "ANOMMULTIPLICITY": "29.3",
            "MULTIPLICITY": "40.7",
            "ISIGI": "38.5",
            "RESISIGI": "0.00",
            "PROCESSINGSTATUS": "1",
            "PROCESSINGMESSAGE": "processing successful",
            "DCCOUNT": "1",
            "AUTOMATIC": "1",
            "INDEX": 1
        },
        "overall": {
            "CCHALF": "1.0",
            "CCANOMALOUS": "0.1",
            "ANOMALOUS": null,
            "TYPE": "autoPROC+STARANISO",
            "NTOBS": "50951",
            "NUOBS": "1106",
            "RLOW": "28.59",
            "RHIGH": "2.52",
            "RMEAS": "0.220",
            "RMERGE": "0.217",
            "COMPLETENESS": "91.1",
            "ANOMCOMPLETENESS": "90.1",
            "ANOMMULTIPLICITY": "28.4",
            "MULTIPLICITY": "46.1",
            "ISIGI": "14.4",
            "RESISIGI": "0.00",
            "PROCESSINGSTATUS": "1",
            "PROCESSINGMESSAGE": "processing successful",
            "DCCOUNT": "1",
            "AUTOMATIC": "1",
            "INDEX": 2
        }
    },
    "CELL": {
        "CELL_A": "####",
        "CELL_B": "####",
        "CELL_C": "####",
        "CELL_AL": "####",
        "CELL_BE": "####",
        "CELL_GA": "####"
    },
    "SG": "########",
    "AID": "#####",
    "MESSAGES": [],
    "AUTOMATIC": true
}

Addition

PROCESS_PATH: "/dls/##/data/2023/######/processed/#/#######/###/<pid#>/xia2-dials"

That would reduce the checks to programatically get the download folder for each samepl/autoprocess.

Best,
Gustavo

Clicking X button for edited Sample in Container view doesn't cancel edit

Steps to Reproduce

1. Select a Proposal.
2. In the [current-proposal] menu, click Containers.
3. Click a Container in the list.
4. Click the Pencil button to edit an empty Location.
5. Select a Protein Acronym from the drop-down menu. This changes the background of the Name textfield to red.
6. Click the X button of the Location being edited to cancel the edit.

Expected Behavior

The edit of the Location should be canceled, and the Check and X buttons should be replaced with the Pencil button.

Actual Behavior

Nothing happens. The Location remains in edit mode. The only way to recover is to reload the page or enter a valid Name and press the Check button to save it.

Request for better "View" features in "Logs & Files"

I would love to be able to open the "View" of log files not in a pop-up window, but actually in either a new tab or a new window of my browser (like the middle-mouse-click in Firefox/Chrome). But this doesn't work due to some authentication token not being present - right?

Such a feature would enable me to see different processing results in parallel (cycling through tabs) instead of having to open/close different results, forgetting things, going back etc.

There is a similar issue with the Download feature I think: right-mouse click and "Save link as" doesn't work. So one has to just click on it - which is fine for filetypes the browser doesn't know what to do with (they will be saved), but e.g. for HTML files it will then display/render that file (and I need to do another File -> Save action to actually save it). Furthermore, after going back from the rendered HTLM view the overview page in ISPyB/Synchweb has reverted to the initial view and I need to scroll again to the collection I was looking at before (where the tab expansion has now disappeared).

Not sure the above explanations describe well the experience I have ...

API GET /proposal/visits/{visit} endpoint requires prop param but not documented

The GET /proposal/visits/{visit} endpoint requires the prop parameter, but the Swagger API spec doc does not list it as a parameter nor indicate that it is required. For example, the GET request

/api/proposal/visits/sw12345-1

fails with the following exception:

exception 'ErrorException' with message 'Undefined index: prop' in /www/api/src/Page.php:518
Stack trace:
#0 /www/api/src/Page.php(518): Slim\Slim::handleErrors(8, 'Undefined index...', '/www/api/src/P....', 518, Array)
#1 /www/api/src/Page/Proposal.php(291): SynchWeb\Page->arg('prop')
#2 [internal function]: SynchWeb\Page\Proposal->_get_visits('sw12345-1')
#3 /www/api/vendor/slim/slim/Slim/Route.php(462): call_user_func_array(Array, Array)
#4 /www/api/vendor/slim/slim/Slim/Slim.php(1326): Slim\Route->dispatch()
#5 /www/api/vendor/slim/slim/Slim/Middleware/Flash.php(85): Slim\Slim->call()
#6 /www/api/vendor/slim/slim/Slim/Middleware/MethodOverride.php(92): Slim\Middleware\Flash->call()
#7 /www/api/vendor/slim/slim/Slim/Slim.php(1271): Slim\Middleware\MethodOverride->call()
#8 /www/api/src/Dispatch.php(47): Slim\Slim->run()
#9 /www/api/index.php(85): SynchWeb\Dispatch->dispatch()
#10 {main}

However, it succeeds if the prop parameter is specified in the GET request:

/api/proposal/visits/sw12345-1?prop=sw12345

"Unknown column" DB error on sample page

Navigate to sample page, e.g., https://ispyb.diamond.ac.uk/samples/sid/3909536
Scroll to "Data Collections"
Click "Processing Errors"
See error at top of page:

A database error has occurred
Unknown column 'app.datacollectionid' in 'on clause'359

Request URI: https://ispyb.diamond.ac.uk/api/dc?page=1&per_page=15&total_pages=0&total_entries=0&sid=3909536&pp=5&t=err&prop=mx26794
Responds with 503 Service Unavailable, body {"title":"There was an error with MySQL","msg":"Unknown column 'app.datacollectionid' in 'on clause'359"}

AMPQ vs STOMP for reprocessing

It seems that efforts are underway over in python-zocalo land to start using RabbitMQ with the AMQP protocol rather than ActiveMQ with the STOMP protocol. To this end I have installed RabbitMQ in the hopes of triggering reprocessing jobs from SynchWeb. However, SynchWeb is still using STOMP protocol as evidenced by the dependency on

"stomp-php/stomp-php": "3.0.6"

in the composer.json and the use of Stomp in api/src/Queue.php

So this should be simple enough... just switch over to the php-ampqlib library and rewrite api/src/Queue.php. Unfortunately it appears that php-ampqlib does not support PHP5, which at this point seems to be a requirement for running SynchWeb..... so what to do?

Maybe there is a way to configure RabbitMQ to accept Stomp messages? OR perhaps making the changes in SynchWeb that would be needed to allow use of PHP>=7.4?

Data Collection presentation is very SmarGon-centric

I understand that the main air-bearing axis of the SmarGon goniometer is labelled as Omega and therefore is the primary axis for data collection, but this isn't necessarily the case at other beamlines.

The DataCollection table in ISPyB has a column rotationAxis. I assumed this column should indicate the axis of rotation used during the data collection. Even if I have "Phi" in this column SyncWeb shows my rotation axis as Omega.

Judicious grepping has revealed that SynchWeb has hardcoded Omega (in the form of Ω) for various Data Collection parameters:

I can of course just do a global search and replace to make things look proper for our beamline, but maybe there is a better way to do this? Extract it to the config file?

I will see if I can figure out something and maybe even contribute a PR, but I must admit my javascript/marionette/webby skills are...under development.

SynchWeb/client/src/js/templates/imaging/queuexfe.html:		<td class="renderable">&Omega; Range</td>
SynchWeb/client/src/js/templates/imaging/ssdiffractionplan.html:            <span class="label">Omega Start</span>
SynchWeb/client/src/js/templates/imaging/ssdiffractionplan.html:            <span class="label">Omega Range</span>
SynchWeb/client/src/js/templates/imaging/blparamlimitsstatic.html:        <td class="renderable">&Omega;: </td>
SynchWeb/client/src/js/templates/imaging/queuegrid.html:		<td class="renderable">&Omega; Range</td>
SynchWeb/client/src/js/templates/imaging/blparamlimits.html:        <td class="renderable">&Omega;: </td>
SynchWeb/client/src/js/templates/imaging/queuepoint.html:		<td class="renderable">&Omega; Start</td>
SynchWeb/client/src/js/templates/imaging/queuepoint.html:		<td class="renderable">&Omega; Range</td>
SynchWeb/client/src/js/templates/imaging/queuepoint.html:		<td class="renderable">&Omega; End</td>
SynchWeb/client/src/js/templates/imaging/savepreset.html:            <span class="label">Omega Start</span>
SynchWeb/client/src/js/templates/imaging/savepreset.html:            <span class="label">Omega Range</span>
SynchWeb/client/src/js/templates/cell/item.html:            <li>&Omega; Start: <%-AXISSTART%>&deg;</li>
SynchWeb/client/src/js/templates/cell/item.html:            <li>&Omega; Osc: <%-AXISRANGE%>&deg;</li>
SynchWeb/client/src/js/templates/dc/grid.html:        <li>&Omega; Start: <%-AXISSTART%>&deg;</li>
SynchWeb/client/src/js/templates/dc/reprocess_dc.html:    <li>&Omega; Start: <%-AXISSTART%>&deg;, Osc: <%-AXISRANGE%>&deg;</li>
SynchWeb/client/src/js/templates/dc/dc.html:        <li>&Omega; Start: <%-AXISSTART%>&deg;</li>
SynchWeb/client/src/js/templates/dc/dc.html:        <li>&Omega; Osc: <%-AXISRANGE%>&deg;</li>
SynchWeb/client/src/js/templates/dc/dc.html:        <li>&Omega; Overlap: <%-OVERLAP%>&deg;</li>
SynchWeb/client/src/js/templates/dc/dc_strategy.html:            <th>&Omega; Start</th>
SynchWeb/client/src/js/templates/dc/dc_strategy.html:            <th>&Omega; Osc</th>
SynchWeb/client/src/js/templates/mc/datacollection.html:<span>&Omega; Start: <%-AXISSTART%>&deg; &Omega; Osc: <%-AXISRANGE%>&deg; | <a title="Click to view diffraction images for this data collection" class="images" href="/dc/view/id/<%-ID%>">Images</a></span>

composer install fails

composer install seems to be attempting to install the DHL-API package from stufisher and is bumping up against a GitHub rAPI rate limit.

bash-4.2$ composer install
Cannot create cache directory /var/www/.cache/composer/repo/https---repo.packagist.org/, or directory is not writable. Proceeding without cache
Cannot create cache directory /var/www/.cache/composer/files/, or directory is not writable. Proceeding without cache
No lock file found. Updating dependencies instead of installing from lock file. Use composer update over composer install if you do not have a lock file.
Loading composer repositories with package information
Cannot create cache directory /var/www/.cache/composer/repo/github.com/stufisher/DHL-API/, or directory is not writable. Proceeding without cache
GitHub API limit (0 calls/hr) is exhausted, could not fetch https://api.github.com/repos/stufisher/DHL-API. Create a GitHub OAuth token to go over the API rate limit. You can also wait until ? for the rate limit to reset.

Head to https://github.com/settings/tokens/new?scopes=repo&description=Composer+on+als-enable.bl1231.als.lbl.gov+2020-12-03+0026
to retrieve a token. It will be stored in "/var/www/.config/composer/auth.json" for future use by Composer.
Token (hidden): 

Is there anyway for to bypass this?

Refined Beam Positions from `fast_dp` are flipped and `SynchWeb` is wrongly flipping them back.

If you look at lines 37-66 of client/src/js/templates/dc/dc_autoproc.html you will notice some pretty dubious coding.

 <table class="reflow bc">
        <thead>
            <tr>
                <th>Beam Centre</th>
                <th>X</th>
                <th>Y</th>
            </tr>
        </thead>
        <tbody>
            <tr>
                <td>Start</td>
                <td><%- BEAM.XBEAM ? BEAM.XBEAM : '- -' %></td>
                <td><%- BEAM.YBEAM ? BEAM.YBEAM : '- -' %></td>
            </tr>
            <!-- these are flipped, not sure why? -->
            <tr>
                <td>Refined</td>
                <td><%- BEAM.REFINEDYBEAM ? BEAM.REFINEDYBEAM : '- -' %></td>
                <td><%- BEAM.REFINEDXBEAM ? BEAM.REFINEDXBEAM : '- -' %></td>
            </tr>
            <tr>
                <td>&Delta;</td>
                <td><%- BEAM.REFINEDYBEAM && BEAM.XBEAM ? (BEAM.XBEAM - BEAM.REFINEDYBEAM).toFixed(2) : '- -' %></td>
                <td><%- BEAM.YBEAM && BEAM.REFINEDXBEAM ? (BEAM.YBEAM - BEAM.REFINEDXBEAM).toFixed(2) : '- -' %></td>
            </tr>
        </tbody>
    </table>

    <% if (BEAM.REFINEDYBEAM > 0 && (Math.abs(BEAM.XBEAM-BEAM.REFINEDYBEAM) > 0.5 || Math.abs(BEAM.YBEAM-BEAM.REFINEDXBEAM) > 0.5)) { %>

The table headers and the starting values look OK:

        <thead>
            <tr>
                <th>Beam Centre</th>
                <th>X</th>
                <th>Y</th>
            </tr>
        </thead>
        <tbody>
            <tr>
                <td>Start</td>
                <td><%- BEAM.XBEAM ? BEAM.XBEAM : '- -' %></td>
                <td><%- BEAM.YBEAM ? BEAM.YBEAM : '- -' %></td>
            </tr>

But the refined position is backwards... with refined beams switched from x,y to y,x:

<td><%- BEAM.REFINEDYBEAM ? BEAM.REFINEDYBEAM : '- -' %></td>
<td><%- BEAM.REFINEDXBEAM ? BEAM.REFINEDXBEAM : '- -' %></td>

and there is even a comment in the code from @stufisher from 7 years ago wondering why:

 <!-- these are flipped, not sure why? -->

The delta calculations are also messed up:

<td>&Delta;</td>
<td><%- BEAM.REFINEDYBEAM && BEAM.XBEAM ? (BEAM.XBEAM - BEAM.REFINEDYBEAM).toFixed(2) : '- -' %></td>
<td><%- BEAM.YBEAM && BEAM.REFINEDXBEAM ? (BEAM.YBEAM - BEAM.REFINEDXBEAM).toFixed(2) : '- -' %></td>

I think the root of the problem comes from fast_dp swapping the refined beam positions from the initial starting ones.

So it appears that at some point instead of hunting down the root of the problem, someone just swapped the variable in the SynchWeb code.

I first noticed this oddity when I started upserting autoPROC.xml results and saw that the start and refined beam_x/y positions were swapped. I thought maybe autoPROC was somehow misinterpreting our image headers etc. etc. but I now think autoPROC is doing things correctly and fast_dp has flipped the refined beam_x/y and that SynchWeb then flips them back in order to display the results to the users.

New (?) "Resolution I/sig(I)=2" item in "Auto Processing" table?

Just discovered a new column headed "Resolution I/sig(I)=2" at the DLS SynchWeb interface. I guess this means the "resolution where the <I/sigI> of merged reflections (in isotropic shells) drops below 2 when analysed with increasing resolution"? It seems to be populated only by "xia dials" pipelines

Can you please clarify what it exactly means and how other pipeline developers should ensure the value is transferred into ISPyB?

Is this done via the XML file?

Non-mx proposals not listed for user

Steps to Reproduce

1. For a regular user with no Proposals, create three: one with code sw, one with mx, and one with in.
2. Create a Visit on the sw and mx Proposals, but not the in Proposal.
3. Log in as the regular user.
4. Click the Home button at the top.
5. Click Proposals.

Expected Behavior

The Proposals list should include the sw and mx Proposals. (I wish it included the in Proposal as well, but I understand that it's expected that it is not displayed because it has no Visits.)

Actual Behavior

The Proposals list only shows the mx Proposal. The sw Proposal is not listed.

Assumes one-to-one mapping of beamline to PV prefix

In api/config_sample.php, there is a variable $bl_pv_map that assumes a one-to-one mapping between a beamline and a PV prefix:

# Map of beamlinename to pv prefix
$bl_pv_map = array(
  'i02' => 'BL02I',
  'i03' => 'BL03I',
);

At some sites (e.g., IMCA-CAT), there is not a one-to-one between a beamline and a PV prefix, so it would be better to just list PVs explicitly for a given beamline rather than trying to build them dynamically with a PV prefix based on the beamline.

Configurable date format

The date format is hardcoded as dd-mm-yy. Can this be made configurable so I can display the date as mm-dd-yy instead?

Eurpoe/London timezone is hardcoded

There are at least 4 occurrences of Europe/London hardcoded into the app.

api/index.php:date_default_timezone_set('Europe/London');
api/formulatrix/index.php:date_default_timezone_set('Europe/London');
api/src/Page.php:                    $inputTZ = new \DateTimeZone('Europe/London');
client/src/js/modules/stats/views/breakdown.js:                        timezone: 'Europe/London',
client/src/js/modules/stats/views/breakdown.js:                        timezone: 'Europe/London',

It might be good to somehow extract these out to a config file.

Give Name text field default focus in Add New Shipment view

When I click the "Add Shipment" button in the Shipments view, I'm taken to the "Add New Shipment" view, and the first field I always want to enter is the Name field, but I can't just start typing; I have to first click in the Name text field to give it focus, and then I can start typing. I think every user will enter the shipment name first, so it seems like it would be good to give focus to this text field immediately on page load.

EM relion processing broken

Starting from https://ispyb.diamond.ac.uk/dc/visit/cm28212-1/ I click on ⚙️ Relion processing

which supposedly goes to https://ispyb.diamond.ac.uk/em/process/relion/session/cm28212-1

but instead ends up on:

Shipment "Mark as Sent" button can break scanned Assign Containers view

If a user clicks the "Mark as Sent" button in the Shipment view for a shipment when a container from a dewar from that shipment has been assigned to a sample changer location, it will cause the status of the dewar to be changed from processing to sent to facility! This is disastrous because it will make the container disappear from the scanned Assign Containers view because the dewar status is no longer processing which is one of the conditions for querying which containers are in the sample changer! Any other database queries that rely on the dewar status being processing as part of finding which containers are in the sample changer will also break. It should not be possible for a user to break things like this simply by clicking the "Mark as Sent" button of a shipment.

Fatal error when downloading or viewing autoproc attachments

I've upserted several auto processing results for a particular dataset. Now when I attempt to view or download any of the attachment files I get this error:

Fatal error: Uncaught Error: Class 'Symfony\Component\Filesystem\Filesystem' not found in /var/www/sites/synchweb/api/src/Page/Download.php:379 Stack trace: #0 /var/www/sites/synchweb/api/src/Page/Download.php(280): SynchWeb\Page\Download->_get_file('56986688', Array) #1 [internal function]: SynchWeb\Page\Download->_get_autoproc_attachments('1037218', '1') #2 /var/www/sites/synchweb/api/vendor/slim/slim/Slim/Route.php(462): call_user_func_array(Array, Array) #3 /var/www/sites/synchweb/api/vendor/slim/slim/Slim/Slim.php(1326): Slim\Route->dispatch() #4 /var/www/sites/synchweb/api/vendor/slim/slim/Slim/Middleware/Flash.php(85): Slim\Slim->call() #5 /var/www/sites/synchweb/api/vendor/slim/slim/Slim/Middleware/MethodOverride.php(92): Slim\Middleware\Flash->call() #6 /var/www/sites/synchweb/api/vendor/slim/slim/Slim/Middleware/PrettyExceptions.php(67): Slim\Middleware\MethodOverride->call() #7 /var/www/sites/synchweb/api/vendor/slim/slim/Slim/Slim.php(1271): Slim\Middleware\PrettyExceptions->call() #8 /var/www/sites/synchweb/a in /var/www/sites/synchweb/api/src/Page/Download.php on line 379

SynchWeb website should redirect to https!

http://diamondlightsource.github.io/SynchWeb/ should redirect to https://diamondlightsource.github.io/SynchWeb/

There should be a setting for this.

Add Container button in Shipment view is too big

The Add Container button in the Shipment view is too big. It appears to be taking on the height of the "Dewar Details" label on the left. The button should be a normal size just like the "Add Dewar" button above it.

References to DLS beamlines outside of config files

There are references to DLS beamlines outside of the config files. For example:

$ fgrep -Ilr i02 . | grep -Ev '(config|README)'
./api/includes/pages/class.fault.php
./api/includes/pages/class.pdf.php
./api/includes/pages/class.stats.php
./api/includes/pages/class.status.php
./api/includes/pages/class.vstat.php
./client/js/templates/status.html
./client/js/templates/status/status.html

Searching for i03 and i04 find similar locations.

This makes it harder to deploy at another site because changes need to be made in more places than just the config files.

LDAP authentication does not support start_tls connections.

The following options are configured in the api/config.php file (obfuscated):

$authentication_type = 'ldap';
$ldap_server = 'ldap://myldap-server.gov';
$ldap_search = 'ou=people,dc=example,dc=gov';

I am using the current master branch c2b8e95

But it appears that store.js is unable to find/configure an LDAP authentication method.