tjlane / pypad Goto Github PK
View Code? Open in Web Editor NEWIntuitive, high-level interface to the CSPAD
License: Other
Intuitive, high-level interface to the CSPAD
License: Other
in cspad.py, I commented out this method
Then, you should be able to turn those mysterious parameters into a CSPad
geometry by doing something like this:
>>> geom = CSPad.from_dir('my_params', run=0)
>>> assembled_image = geom(raw_image)
>>> imshow(assembled_image)
where is this from_dir
method ?
Would be nice if we could fix this ASAP, as I would like to try and optimize my sample-detector distance for the ice hits with PyPad. I'll implement support for the hexagonal unit cell in score.py
The ugly fix is to write a separate script that converts cheetah
metrology to cspad
object.. Do we already have this?
[pypad:612 22:00:23] score score_params.yaml
--- SCORING CSPAD GEOMETRY ---
Figure Explanation:
Left Panel: The assembled image, superimposed with white circles where
theoretically there should be powder rings. The red dot
visualizes where the forward beam intersects the detector plane.
Right Panel: The radial projection for each individual quad, dotted
lines are theoretical peak predictions.
To plot additional samples:
Press: l for previous sample
Press: n for next sample
Loaded: score_params.yaml
Loading: gold-minus490mm.h5
found: 10 peaks
Loading: gold-minus470mm.h5
found: 8 peaks
--- Optimized Energy & Detector Distance ---
Detector offest: 582.00 mm
Energy: 9.690 keV
Total Residuals: 0.007827 inv. Angstroms
Residuals for each peak:
[ 0.00046626 0.0008697 0.00415692 0.00233418]
Plotting calibration sample: 0
(may take a moment)
distance: 92.00 mm
Loading: gold-minus490mm.h5
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/Current/bin/score", line 5, in
pkg_resources.run_script('pypad==0.0.1', 'score')
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pkg_resources.py", line 467, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pkg_resources.py", line 1200, in run_script
execfile(script_filename, namespace, namespace)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/EGG-INFO/scripts/score", line 54, in
sref.display()
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/autogeom/score.py", line 416, in display
self._plot_cal_sample(self._current_image)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/autogeom/score.py", line 445, in _plot_cal_sample
plot.imshow_cspad( self.cspad(img), vmin=0, ax=self._axL )
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/cspad.py", line 461, in call
return self._assemble_image(raw_image)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/cspad.py", line 1115, in _assemble_image
quad_index_image = self._assemble_quad( raw_image[quad_index], quad_index )
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/cspad.py", line 1086, in _assemble_quad
raise ValueError('rs: out of bounds in rows')
ValueError: rs: out of bounds in rows
I don't think I'm doing this now.
should all be one file
this should improve the optimization as it decouples the optimization of the variables
@rkirian got ya :)
need to test this
could save some head ache from having to interpret the arbitrary psana parameters, requires an exporter to psana parameters though
[pypad:609 11:24:57] genfilter cxi64813-r0135-Au.h5
--- INTERACTIVE PARAMETER OPTIMIZATION ENVIRONMENT ---
Welcome to autogeom's interactive image pre-processing environment.
Adjust the sliders to optimize your image -- the goal is to get one
or more sharp powder rings on the image to show up above the noise.
Instructions:
-- Right Click: (Left Panel) Sets the guess of the center of the
image.
(Right Panel) Set a 'peak limit', the algorithm
only optimizes peaks between these
limits. You can set many such regions.
-- Threshold: Sets a binary intensity threshold.
-- Median: A median filter. You can thinks of this as a voting
scheme, where a pixel gets set to be the same as a
consensus of those around it. This parameter sets
the size of that consensus window.
-- Minimum: A minimum filter. If any pixel in this window size
is zero, all pixels in the window become zero.
Once you are done, the final parameters you have in your window will
be written to disk in a file that can be interpreted by the
'optgeom' script, which will give you a final, optimized geometry.
PS. The rendering can be a bit slow... sorry. Please be a bit
patient!
Loading: cxi64813-r0135-Au.h5
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/Current/bin/genfilter", line 5, in
pkg_resources.run_script('pypad==0.0.1', 'genfilter')
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pkg_resources.py", line 467, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pkg_resources.py", line 1200, in run_script
execfile(script_filename, namespace, namespace)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/EGG-INFO/scripts/genfilter", line 294, in
main()
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/EGG-INFO/scripts/genfilter", line 284, in main
image, output_fn = parse_n_load()
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/EGG-INFO/scripts/genfilter", line 44, in parse_n_load
image = geom(raw_image)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/autogeom/cspad.py", line 149, in call
return self._assemble_image(raw_image)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/autogeom/cspad.py", line 560, in _assemble_image
self.beam_location[i] -= min_offset
AttributeError: 'CSPad' object has no attribute 'beam_location'
We should be able to speed up the optimization algorithm by first decouple the various degrees of freedom for a rough alignment, the to the fine optimization in the whole multi-dimensional space.
Suggestion: align gap roughly (rely on pixel center from genfilter
), the align pixel center, then rotate quads, then optimize all
this scheme could most likely be improved over time..
I get this error when I try to run assemble:
Traceback (most recent call last):4] assemble ~/NML-2013/ftc74613/cleaned_hdf5/r0028/LCLS_2013_Jan18_r0028_203203_c99c_cspad.h5
File "/reg/neh/home3/sellberg/local/scripts/assemble", line 5, in
pkg_resources.run_script('pypad==0.0.1', 'assemble')
File "/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python/pyextra/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 489, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python/pyextra/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 1214, in run_script
exec script_code in namespace, namespace
File "/reg/neh/home3/sellberg/local/site-packages/pypad-0.0.1-py2.7.egg/EGG-INFO/scripts/assemble", line 20, in
File "build/bdist.linux-x86_64/egg/pypad/cspad.py", line 30, in
ImportError: No module named nxutils
Does nxutils still exist in matplotlib???
@dermen this is your error assembling on Odin's dtc. The metrology seems to be fine, but pypad's assembler can't deal with it right now. You can assemble w/o trouble using .cspad files and optical metrologies.
I need to re-work the image assembler to be as precise as possible. Right now we're using Mikhal's version which could yield small errors in the visualization. I've opened a new branch assemble to tackle this.
https://github.com/tjlane/thor
thor contains the old x-ray scattering parts of odin
note that pypad will be deprecated in ~6 months when the ePix detector comes online -- hopefully at that time we won't even need pypad!
the current optimization uses 0th order derivative
Nothing happens to the 2D pattern when I click on 'Apply filter'. I'm running python on psexport that is using:
Python 2.7.2 (default, Jan 14 2013, 21:09:22)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-54)] on linux2
In [2]: pkg_resources.get_distribution('matplotlib').version
Out[2]: '1.1.1'
When center is way off initially, as for the AgBe below, the optimization algorithm gets stuck in local minima. Can be solved by adding a rough center algorithm. The easiest way is to include right-clicking on the 2D panel again in prepgeom
, a more fancy alternative is to have a button or pre-optimization step where one calculates the center form the hough transform.
Requires modifying the angular average code in cspad.py
, passing masks through the scripts.
if radius_range: None
is specified in filter_params.yaml
, which it is if no range is selected, optgeom gives the following error:
Set parameter : minf_size --> 3
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/Current/bin/optgeom", line 5, in
pkg_resources.run_script('pypad==0.0.1', 'optgeom')
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pkg_resources.py", line 467, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pkg_resources.py", line 1200, in run_script
execfile(script_filename, namespace, namespace)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/EGG-INFO/scripts/optgeom", line 102, in
main()
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/EGG-INFO/scripts/optgeom", line 92, in main
optizor = opt.Optimizer(geometry, **optimization_params)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/autogeom/optimize.py", line 112, in init
self.radius_range = np.sort(np.array(self.radius_range, dtype=np.float))
ValueError: setting an array element with a sequence.
it's sweet to be able to continue to work on a mask that has already been saved once
doesn't really matter but it makes it more convenient to view (at least for me, since I've used it as a convention with the origin in the lower left corner)
When trying out score
I found that the assembled output had a compressed Q-scale and that the angular average showed to high momentum transfers when using the proper sample-to-detector distance and photon energy (given in https://github.com/tjlane/pypad/blob/master/pypad/autogeom/README.md), see below compared to the output of cheetah (r0135_angavg). This results in that the scoring fails, since the Bragg peaks are in the wrong positions. There must be a bug in the internal calculation of the scattering angle and/or Q-scale in score
The following need to get done:
Second to last I will probably not do unless someone bugs me.
The last is long term. I'm not really capable of doing it, but Mikhal could if need be.
@sellberg :)
I'll start with implementing support for the hexagonal unit cell in score.py
The current algorithm tilts each ASIC around its own center -- the rotation should really be around the quad center!
This may just be in assemble.
I can fix this later, just to tired to rerun the gold right now...
[pypad(master):610 21:31:23] assemble examples/gold-minus490mm.h5 -m examples/my_cspad.cspad
Loading: examples/gold-minus490mm.h5
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/Current/bin/assemble", line 5, in
pkg_resources.run_script('pypad==0.0.1', 'assemble')
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pkg_resources.py", line 467, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pkg_resources.py", line 1200, in run_script
execfile(script_filename, namespace, namespace)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/EGG-INFO/scripts/assemble", line 55, in
imshow_cspad( cs1(raw_image), ax=ax, scrollable=True )
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/cspad.py", line 461, in call
return self._assemble_image(raw_image)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/cspad.py", line 1115, in _assemble_image
quad_index_image = self._assemble_quad( raw_image[quad_index], quad_index )
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/cspad.py", line 1077, in _assemble_quad
two_by_one = interp.rotate(two_by_one, -self._asic_rotation(quad_index, i),
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/cspad.py", line 996, in _asic_rotation
p, s, f, shape = self._metrology_basis.get_grid( self._asic_index(quad_index, i, j) )
AttributeError: 'CSPad' object has no attribute '_metrology_basis'
We should probably make sure there are no errors if one clicks on the image and nothing happens... It seems like we should add an ignore attribute.
Exception in Tkinter callback
Traceback (most recent call last):
File "/reg/g/psdm/sw/external/python/2.7.2/x86_64-rhel5-gcc41-opt/lib/python2.7/lib-tk/Tkinter.py", line 1410, in call
return self.func(_args)
File "/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python/matplotlib/backends/backend_tkagg.py", line 301, in button_release_event
FigureCanvasBase.button_release_event(self, x, y, num, guiEvent=event)
File "/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python/matplotlib/backend_bases.py", line 1655, in button_release_event
self.callbacks.process(s, event)
File "/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python/matplotlib/cbook.py", line 262, in process
proxy(_args, *_kwargs)
File "/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python/matplotlib/cbook.py", line 192, in call
return mtd(_args, **kwargs)
File "build/bdist.linux-x86_64/egg/pypad/plot.py", line 150, in _release
if self.ignore(event):
AttributeError: 'ToggleButton' object has no attribute 'ignore'
This turns out to be a good thing to have during beam times. Right now the interface is terrible. @rkirian if you have ideas let me know.
This could be useful since the distance between the quads is quite small in the initial configuration and we don't want the optimization to go into a virtual minimum where the ASICs are overlapping
it usual shifts the image a few pixels up/left
I tried to run assemble with the default metrology and it works fine, but when I try and run it with the output from optgeom, it crashes:
[psana0210:pypad:529 14:12:51] optgeom ~/source/pypad/examples/gold-minus490mm.h5 filter_params.yaml
--- ASSEMBLING GEOMETRY FROM CALIBRATION STANDARD ---
...
objective value: -1.5385e+21
objective value: -1.5385e+21
Optimization terminated successfully.
Current function value: -1538453417345293484032.000000
Iterations: 436
Function evaluations: 794
Wrote my_cspad.cspad to disk.
[psana0210:pypad:535 15:46:52] assemble ~/source/pypad/examples/gold-minus490mm.h5 --metrology my_cspad.cspad
Loading: /reg/neh/home3/sellberg/source/pypad/examples/gold-minus490mm.h5
Traceback (most recent call last):
File "/reg/neh/home3/sellberg/local/scripts/assemble", line 5, in
pkg_resources.run_script('pypad==0.0.1', 'assemble')
File "/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python/pyextra/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 489, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python/pyextra/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 1214, in run_script
exec script_code in namespace, namespace
File "/reg/neh/home3/sellberg/local/site-packages/pypad-0.0.1-py2.7.egg/EGG-INFO/scripts/assemble", line 52, in
File "build/bdist.linux-x86_64/egg/pypad/cspad.py", line 439, in call
File "build/bdist.linux-x86_64/egg/pypad/cspad.py", line 1031, in _assemble_image
ValueError: qoff_col: -2 out of bounds [0,1900)
_match_peaks(self, sample_index)
calculation time is currently Nobserved_peaks ^ Nmiller_indices => 4 miller indices are doable, anything beyond that takes forever. will see if I can find a clever workaround for this...
This warning pops up when I run the standard geometry, if possible we should change so that the metrology does not have these issues or alternatively overrides the non-orthogonality if it occurs.
Loading: examples/gold-minus490mm.h5
Parsing: quad 0
WARNING: Metrology quality control failed for 2x1: 1
--> s/f vectors are not orthogonal
Val: -0.000013 // tol: 0.000010
Parsing: quad 1
Parsing: quad 2
WARNING: Metrology quality control failed for 2x1: 0
--> s/f vectors are not orthogonal
Val: 0.000101 // tol: 0.000010
Parsing: quad 3
WARNING: Metrology quality control failed for 2x1: 3
--> s/f vectors are not orthogonal
Val: -0.000010 // tol: 0.000010
the objective function currently calculates the angular average for each quad and maximizes the product, it's the hard to control the scaling... if an error-around-the-mean objective function is implemented, where each variable is separated, we can control the RMS scaling for each variable. The scaling may affect the optimization significantly
See last (Aug 1, 2013) comment in #21 , 14375 negative pixels were masked automatically but do not appear masked when running it in the cheetah...
pypad(master):575 11:42:55] genmask examples/gold-minus470mm.h5 -o test.h5 -f cheetah
Loading: examples/gold-minus470mm.h5
--- WELCOME TO PYPAD's INTERACTIVE MASKING ENVIRONMENT ---
Green pixels are masked.
Keystrokes
m : mask u : unmask r : reset
x : clear selection w : save & exit t : toggle pixel
q : exit w/o saving
Mouse
Right click on three or more points to draw a polygon around a
set of pixels. Then press m
or u
to mask or unmask that area.
You can also mask/unmask single pixels by clicking on them with
the mouse and pressing t
to toggle the mask state.
Toggle Buttons (left)
nonbonded : Mask nonbonded pixels, and their nearest neighbours.
These pixels aren't even connected to the detector.
row 13 : In some old experiments, row 13 on one ASIC was
busted -- mask that (will be clear if this is needed)
threshold : You will be prompted for an upper and lower limit --
pixels outside that range are masked. Units are ADUs
and you can set only a lower/upper limit by passing
'None' for one option.
borders : Mask the borders of each ASIC. These often give
anomoulous responses. Recommended to mask one pixel
borders at least.
----- // -----
Masked: 14375 negative pixels
Selected: (226, 289)
Masking convex area...
Wrote: test.h5
It would be nice if one could optionally enter a number rather than clicking on sliders, e.g. for choosing thresholds, dilation, etc.
Looking a bit closer at the scripts and their outputs there are inconsistencies between the coordinate system chosen for the various scripts. I propose we stick to the CXI convention, i.e. +X towards hutch door, +Y towards ceiling, and +Z downstream. This is what cheetah uses (i.e. https://github.com/sellberg/cheetah but this should still be true for https://github.com/antonbarty/cheetah) and outputs for it's assembled images. To view the front of the detector with a simple image viewer (which usually insists to put the origin in the upper left or lower left corner, such as plt.imshow()
), this requires to mirror the X-axis prior to assembly, but this should only be done by the assembly script viewing the image and not saved in any files using an assembled coordinate system.
Below I compare the output of pypad with that of cheetah (marked r0135) and comment on differences (look for noisy ASIC):
X-Y plane rotated 90 deg clockwise when using genfilter
. Needs to transpose and mirror rows (i.e. Y-axis after transpose) in internally assembled array.
X-Y axes flipped and mirrored in X as well as Y when using score
. Needs to transpose and mirror both rows and cols in internally assembled array.
X-Y plane rotated 90 deg clockwise when using assemble
. Needs to transpose and mirror row index (i.e. Y-axis after transpose) in internally assembled array.
This issue may or may not be related to #1
At least warn the user it's not done yet!
@sellberg I envision a short (5 pg?) TeX document in the pypad repo that explains:
(0) A paragraph about what problem we set out to solve with pypad, and the future we envision for it
(1) The CXI convention
(2) The layout of the CSPAD and how memory is mapped onto the geometry
(3) The basis vector convention used in pypad
(4) Unit conventions
(4.1) Any other conventions?
(5) A brief description of each script
If we plan out what to include here, I can put this together.
following #35 it would also be nice to be able to save the mask object as well as export to another format. alternatively the exporting can be a separate script like geomconv
--> maskconv
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.