taw10/crystfel

Data processing for serial crystallography

CrystFEL - Data processing for serial crystallography

Overview

CrystFEL is a suite of programs for processing data from serial crystallography experiments, performed at synchrotron and X-ray free-electron laser facilities, as well as in your home lab using an electron microscope.

Getting started

See INSTALL.md for installation instructions, including our container registry, installation via package manager and details of pre-existing installations at X-ray facilities around the world.

CrystFEL can be used from the command line or via a graphical user interface. To start the graphical user interface, run crystfel.

There is a video tutorial, as well as a text tutorial to get you started with processing via the GUI.

For command-line use, standard man pages are available. Start with man crystfel. The manual pages are also available on the web.

Documentation

• Basic Tutorial
• How to choose the right point group for merging
• How to increase data processing speed
• Real-time data processing
• Processing electron diffraction data
• Symmetry classification for serial crystallography
• Matrix conventions used in CrystFEL code - for developers, written mostly for my own benefit.
• Hit rate graph
• Examples folder - contains some template input files.
• Contributing to CrystFEL - including how to cite CrystFEL and how to find good first issues to work on.
• Citation list - please send us details of your paper, if it's missing!
• Scripts folder - a miscellany of smaller programs to help at various stages of data processing.

Journal articles and book chapters

• Processing serial crystallography data with CrystFEL: a step-by-step guide - covers command-line processing only (pre-dates the GUI).
• Recent developments in CrystFEL - now somewhat out of date, but contains some useful information about the algorithms used.
• Crystallography and Molecular Imaging using X-ray Lasers - an introduction to the biological aspects and possibilities, written for physicists (in contrast to most other articles, which introduce the physical aspects for biologists!).
• Original paper about CrystFEL from 2012. Not open access, but a "reprint" is available here.
• Climbing the Data Mountain: Processing of SFX Data - emphasizes data volume issues for XFELs. Unfortunately not open access.
• Processing of XFEL Data - describes the entire processing pipeline. Unfortunately not open access.

Awards

In 2017, the development of CrystFEL was recognised with the Max von Laue Prize from the German Society for Crystallography (DGK).

Citing CrystFEL

Please see CONTRIBUTING.md for citation instructions.

Funding acknowledgements

Development of CrystFEL is primarily funded by the Helmholtz Association via DESY.

Partial funding for CrystFEL has been provided by:

• The consortium DAPHNE4NFDI in the context of the work of the NFDI e.V. The consortium is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - project number 460248799 (2022-).

• European Union’s Horizon 2020 research and innovation programme under grant agreement No 857641 (ExPaNDS) (2019-2023).

• X-Probe, a project of the European Union's 2020 Research and Innovation Program Under the Marie Skłodowska-Curie grant agreement 637295 (2015-2018).

• The BMBF German-Russian Cooperation SyncFELMed, grant 05K14CHA (2014-2017).

• BioStruct-X, a project funded by the Seventh Framework Programme (FP7) of the European Commission (2011-2016).

Licence

Copyright © 2012-2024 Deutsches Elektronen-Synchrotron DESY, a research centre of the Helmholtz Association.

See AUTHORS as well as individual source code files for full details of contributors.

CrystFEL is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with CrystFEL. If not, see http://www.gnu.org/licenses/.

crystfel's Issues

Settings button crashes the GUI

When I press "settings" button in the freshly oppened GUI it crashes with "segmentation fault" error.

Filling path to data manually in CrystFEL GUI

It would be great if one can fill the path to data by copy-paste instead of navigating through folders.

indexamajig --input=i to read file lists from STDIN is broken

indexamajig --input=i to read file lists from STDIN is broken. We got Couldn't open '-'.

The following patch will fix this problem.

index aa0db21..0dfc295 100644
--- a/src/indexamajig.c
+++ b/src/indexamajig.c
@@ -1241,7 +1241,7 @@ int main(int argc, char *argv[])
                return 1;
        }
 
-       if ( (args.filename != NULL) && is_hdf5_file(args.filename, &err) ) {
+       if ( (args.filename != NULL) && (strcmp(args.filename, "-") != 0) && is_hdf5_file(args.filename, &err) ) {
                ERROR("Your input file appears to be an HDF5 file.\n");
                ERROR("The input file should be a list of data files, not the "
                      "data file itself.\n");

Garbage calibration data is written when there is no hierarchy information

When there is no detector hierarchy information in a geometry file, indexamajig --mille rightly complains:

Cannot write calibration data (--mille) because geometry does not contain hierarchy information (group 'all' not found)

However, non zero-byte mille files are actually written. Passing this to align-detector causes a segmentation fault.

If the detector consists of only one panel, CrystFEL will automatically create the all group containing it.
https://gitlab.desy.de/thomas.white/crystfel/-/blob/master/doc/man/crystfel_geometry.5.md

How about assuming all panels belong to the all group if there is no definition?

align_offset does not write coffset

Even with --out-of-plane, refined coffset is not written to the output geometry file.

The cause is this check:

crystfel/libcrystfel/src/datatemplate.c

Lines 2564 to 2566 in 16f4ffd

	if ( !p->cnz_offset_default ) {
	fprintf(fh, "%s/coffset = %f\n", p->name, p->cnz_offset);
	}

Group definitions must come after panel definitions

If groups are defined before its panels are defined, we get this error:

Unknown panel group 'q1'
Unknown panel group 'q2'
...
Invalid top-level line 'group_all'

Rigid body definitions did not have this order requirement.

Is this by design? If so, it would be nice to document this.

align_detector corrupts bad region definitions

In our SACLA MPCCD geometry file, we have this for the second panel (q2):

data = /%/data

q2/adu_per_eV = 0.000999
q2/min_fs = 0
q2/min_ss = 1024
q2/max_fs = 511
q2/max_ss = 2047
q2/fs = 0.006215x -0.999981y
q2/ss = 0.999981x +0.006215y
q2/corner_x = 36.444600
q2/corner_y = 497.856016

badq2l1/min_fs = 0
badq2l1/max_fs = 4
badq2l1/min_ss = 1024
badq2l1/max_ss = 2047
badq2l1/panel  = q2

Note that eight 512 x 1024 px panels are stacked along the slow (1024 px) axis into a HDF5 dataset.

align_detector rewrites this bad region definition as follows:

bad_badq2l1/panel = q2
bad_badq2l1/min_fs = 0
bad_badq2l1/max_fs = 4
bad_badq2l1/min_ss = 0
bad_badq2l1/max_ss = 1023

It seems that min_ss and max_ss have been rewritten in terms of the panel's local data coordinate.

However, indexamajig still interprets them in terms of the global data coordinate system.

Bad region 'bad_badq2l1' is outside the panel bounds (q2) as presented in data (0 1024, 511 2047 inclusive): Bad region 0,0 to 4, 1023 inclusive

I don't know which is intended. Earlier versions worked in the same way as indexamajig, which I prefer.

Ambiguity Error

I was working with SLAC/mfxp17318_0015 data. After all indexing with determined unit cell file, I got Subprocess exited with status 512 error with no message while running Indexing Ambiguity worker.

I couldn't debug it yet for the reason of error, but, there should be a log of exception.

Is there any known possible reason of the error?

Unnecessary %s in align_detector

In align_detector, you are dereferencing %s without a variable.

crystfel/src/align_detector.c

Line 230 in 3438527

ERROR("Couldn't find %s - falling back on shell path\n");