Giter Site home page Giter Site logo

pypad's People

Contributors

dermen avatar rkirian avatar sellberg avatar tjlane avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

Forkers

daihui

pypad's Issues

lies!

Then, you should be able to turn those mysterious parameters into a CSPad
geometry by doing something like this:
>>> geom = CSPad.from_dir('my_params', run=0)
>>> assembled_image = geom(raw_image)
>>> imshow(assembled_image)

where is this from_dir method ?

score crashes when input metrology is in `cheetah` format

Would be nice if we could fix this ASAP, as I would like to try and optimize my sample-detector distance for the ice hits with PyPad. I'll implement support for the hexagonal unit cell in score.py

The ugly fix is to write a separate script that converts cheetah metrology to cspad object.. Do we already have this?

[pypad:612 22:00:23] score score_params.yaml

--- SCORING CSPAD GEOMETRY ---  

Figure Explanation:
Left Panel: The assembled image, superimposed with white circles where
theoretically there should be powder rings. The red dot
visualizes where the forward beam intersects the detector plane.
Right Panel: The radial projection for each individual quad, dotted
lines are theoretical peak predictions.

To plot additional samples:
Press: l for previous sample
Press: n for next sample

Loaded: score_params.yaml
Loading: gold-minus490mm.h5
found: 10 peaks
Loading: gold-minus470mm.h5
found: 8 peaks

--- Optimized Energy & Detector Distance ---
Detector offest: 582.00 mm
Energy: 9.690 keV

Total Residuals: 0.007827 inv. Angstroms
Residuals for each peak:
[ 0.00046626 0.0008697 0.00415692 0.00233418]

Plotting calibration sample: 0
(may take a moment)
distance: 92.00 mm
Loading: gold-minus490mm.h5
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/Current/bin/score", line 5, in
pkg_resources.run_script('pypad==0.0.1', 'score')
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pkg_resources.py", line 467, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pkg_resources.py", line 1200, in run_script
execfile(script_filename, namespace, namespace)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/EGG-INFO/scripts/score", line 54, in
sref.display()
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/autogeom/score.py", line 416, in display
self._plot_cal_sample(self._current_image)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/autogeom/score.py", line 445, in _plot_cal_sample
plot.imshow_cspad( self.cspad(img), vmin=0, ax=self._axL )
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/cspad.py", line 461, in call
return self._assemble_image(raw_image)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/cspad.py", line 1115, in _assemble_image
quad_index_image = self._assemble_quad( raw_image[quad_index], quad_index )
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/cspad.py", line 1086, in _assemble_quad
raise ValueError('rs: out of bounds in rows')
ValueError: rs: out of bounds in rows

cspad.CSPad.default() has no attribute 'beam_location'

[pypad:609 11:24:57] genfilter cxi64813-r0135-Au.h5

  ---  INTERACTIVE PARAMETER OPTIMIZATION ENVIRONMENT  ---    

Welcome to autogeom's interactive image pre-processing environment.
Adjust the sliders to optimize your image -- the goal is to get one
or more sharp powder rings on the image to show up above the noise.

Instructions:
-- Right Click: (Left Panel) Sets the guess of the center of the
image.
(Right Panel) Set a 'peak limit', the algorithm
only optimizes peaks between these
limits. You can set many such regions.
-- Threshold: Sets a binary intensity threshold.
-- Median: A median filter. You can thinks of this as a voting
scheme, where a pixel gets set to be the same as a
consensus of those around it. This parameter sets
the size of that consensus window.
-- Minimum: A minimum filter. If any pixel in this window size
is zero, all pixels in the window become zero.

Once you are done, the final parameters you have in your window will
be written to disk in a file that can be interpreted by the
'optgeom' script, which will give you a final, optimized geometry.

PS. The rendering can be a bit slow... sorry. Please be a bit
patient!

Loading: cxi64813-r0135-Au.h5
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/Current/bin/genfilter", line 5, in
pkg_resources.run_script('pypad==0.0.1', 'genfilter')
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pkg_resources.py", line 467, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pkg_resources.py", line 1200, in run_script
execfile(script_filename, namespace, namespace)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/EGG-INFO/scripts/genfilter", line 294, in
main()
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/EGG-INFO/scripts/genfilter", line 284, in main
image, output_fn = parse_n_load()
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/EGG-INFO/scripts/genfilter", line 44, in parse_n_load
image = geom(raw_image)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/autogeom/cspad.py", line 149, in call
return self._assemble_image(raw_image)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/autogeom/cspad.py", line 560, in _assemble_image
self.beam_location[i] -= min_offset
AttributeError: 'CSPad' object has no attribute 'beam_location'

improve the optimization algorithm in accuracy and speed

We should be able to speed up the optimization algorithm by first decouple the various degrees of freedom for a rough alignment, the to the fine optimization in the whole multi-dimensional space.

Suggestion: align gap roughly (rely on pixel center from genfilter), the align pixel center, then rotate quads, then optimize all

this scheme could most likely be improved over time..

Does nxutils still exist in matplotlib?

I get this error when I try to run assemble:

Traceback (most recent call last):4] assemble ~/NML-2013/ftc74613/cleaned_hdf5/r0028/LCLS_2013_Jan18_r0028_203203_c99c_cspad.h5
File "/reg/neh/home3/sellberg/local/scripts/assemble", line 5, in
pkg_resources.run_script('pypad==0.0.1', 'assemble')
File "/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python/pyextra/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 489, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python/pyextra/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 1214, in run_script
exec script_code in namespace, namespace
File "/reg/neh/home3/sellberg/local/site-packages/pypad-0.0.1-py2.7.egg/EGG-INFO/scripts/assemble", line 20, in

File "build/bdist.linux-x86_64/egg/pypad/cspad.py", line 30, in
ImportError: No module named nxutils

Does nxutils still exist in matplotlib???

assembly not working for CSPad objects that originate from BasisGrid

@dermen this is your error assembling on Odin's dtc. The metrology seems to be fine, but pypad's assembler can't deal with it right now. You can assemble w/o trouble using .cspad files and optical metrologies.

I need to re-work the image assembler to be as precise as possible. Right now we're using Mikhal's version which could yield small errors in the visualization. I've opened a new branch assemble to tackle this.

@sellberg @rkirian be aware.

'Apply filter' button in `genfilter` broken

Nothing happens to the 2D pattern when I click on 'Apply filter'. I'm running python on psexport that is using:

Python 2.7.2 (default, Jan 14 2013, 21:09:22)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-54)] on linux2

In [2]: pkg_resources.get_distribution('matplotlib').version
Out[2]: '1.1.1'

optgeom can't find center if it's too far off initially, add rough center algorithm

When center is way off initially, as for the AgBe below, the optimization algorithm gets stuck in local minima. Can be solved by adding a rough center algorithm. The easiest way is to include right-clicking on the 2D panel again in prepgeom, a more fancy alternative is to have a button or pre-optimization step where one calculates the center form the hough transform.

BEFORE OPTIMIZATION
8281232a-e418-11e2-8b4d-94020a8d4f12

AFTER OPTIMIZATION
screen shot 2013-07-03 at 16 23 18

optgeom gives error when range in angular average is not specified

if radius_range: None is specified in filter_params.yaml, which it is if no range is selected, optgeom gives the following error:

Set parameter : minf_size --> 3
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/Current/bin/optgeom", line 5, in
pkg_resources.run_script('pypad==0.0.1', 'optgeom')
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pkg_resources.py", line 467, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pkg_resources.py", line 1200, in run_script
execfile(script_filename, namespace, namespace)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/EGG-INFO/scripts/optgeom", line 102, in
main()
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/EGG-INFO/scripts/optgeom", line 92, in main
optizor = opt.Optimizer(geometry, **optimization_params)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/autogeom/optimize.py", line 112, in init
self.radius_range = np.sort(np.array(self.radius_range, dtype=np.float))
ValueError: setting an array element with a sequence.

Angular average has wrong Q-scale in score

When trying out score I found that the assembled output had a compressed Q-scale and that the angular average showed to high momentum transfers when using the proper sample-to-detector distance and photon energy (given in https://github.com/tjlane/pypad/blob/master/pypad/autogeom/README.md), see below compared to the output of cheetah (r0135_angavg). This results in that the scoring fails, since the Bragg peaks are in the wrong positions. There must be a bug in the internal calculation of the scattering angle and/or Q-scale in score

score

change angular sum to angular average

Might not be so important when things are thresholded to zeros and ones, but it might skew the peak shape since higher-raidii are favored. Should be an easy fix.

change_angular_sum_to_angular_avg

finish format compatibilities

The following need to get done:

  • read: odin detectors
  • read: CXIdb cheetah h5s
  • write: CrystFEL geom
  • read/write: psana

Second to last I will probably not do unless someone bugs me.
The last is long term. I'm not really capable of doing it, but Mikhal could if need be.

examples/my_cspad.cspad is out of date

I can fix this later, just to tired to rerun the gold right now...

[pypad(master):610 21:31:23] assemble examples/gold-minus490mm.h5 -m examples/my_cspad.cspad
Loading: examples/gold-minus490mm.h5
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/Current/bin/assemble", line 5, in
pkg_resources.run_script('pypad==0.0.1', 'assemble')
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pkg_resources.py", line 467, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pkg_resources.py", line 1200, in run_script
execfile(script_filename, namespace, namespace)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/EGG-INFO/scripts/assemble", line 55, in
imshow_cspad( cs1(raw_image), ax=ax, scrollable=True )
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/cspad.py", line 461, in call
return self._assemble_image(raw_image)
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/cspad.py", line 1115, in _assemble_image
quad_index_image = self._assemble_quad( raw_image[quad_index], quad_index )
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/cspad.py", line 1077, in _assemble_quad
two_by_one = interp.rotate(two_by_one, -self._asic_rotation(quad_index, i),
File "/Library/Frameworks/Python.framework/Versions/6.3/lib/python2.6/site-packages/pypad-0.0.1-py2.6.egg/pypad/cspad.py", line 996, in _asic_rotation
p, s, f, shape = self._metrology_basis.get_grid( self._asic_index(quad_index, i, j) )
AttributeError: 'CSPad' object has no attribute '_metrology_basis'

error message if I right-click on 2D pattern

We should probably make sure there are no errors if one clicks on the image and nothing happens... It seems like we should add an ignore attribute.

Exception in Tkinter callback
Traceback (most recent call last):
File "/reg/g/psdm/sw/external/python/2.7.2/x86_64-rhel5-gcc41-opt/lib/python2.7/lib-tk/Tkinter.py", line 1410, in call
return self.func(_args)
File "/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python/matplotlib/backends/backend_tkagg.py", line 301, in button_release_event
FigureCanvasBase.button_release_event(self, x, y, num, guiEvent=event)
File "/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python/matplotlib/backend_bases.py", line 1655, in button_release_event
self.callbacks.process(s, event)
File "/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python/matplotlib/cbook.py", line 262, in process
proxy(_args, *_kwargs)
File "/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python/matplotlib/cbook.py", line 192, in call
return mtd(_args, **kwargs)
File "build/bdist.linux-x86_64/egg/pypad/plot.py", line 150, in _release
if self.ignore(event):
AttributeError: 'ToggleButton' object has no attribute 'ignore'

improve tweak script

This turns out to be a good thing to have during beam times. Right now the interface is terrible. @rkirian if you have ideas let me know.

Assemble crashes when using metrology output from optgeom

I tried to run assemble with the default metrology and it works fine, but when I try and run it with the output from optgeom, it crashes:

[psana0210:pypad:529 14:12:51] optgeom ~/source/pypad/examples/gold-minus490mm.h5 filter_params.yaml

--- ASSEMBLING GEOMETRY FROM CALIBRATION STANDARD ---

...

objective value: -1.5385e+21
objective value: -1.5385e+21
Optimization terminated successfully.
Current function value: -1538453417345293484032.000000
Iterations: 436
Function evaluations: 794
Wrote my_cspad.cspad to disk.

[psana0210:pypad:535 15:46:52] assemble ~/source/pypad/examples/gold-minus490mm.h5 --metrology my_cspad.cspad
Loading: /reg/neh/home3/sellberg/source/pypad/examples/gold-minus490mm.h5
Traceback (most recent call last):
File "/reg/neh/home3/sellberg/local/scripts/assemble", line 5, in
pkg_resources.run_script('pypad==0.0.1', 'assemble')
File "/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python/pyextra/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 489, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python/pyextra/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 1214, in run_script
exec script_code in namespace, namespace
File "/reg/neh/home3/sellberg/local/site-packages/pypad-0.0.1-py2.7.egg/EGG-INFO/scripts/assemble", line 52, in

File "build/bdist.linux-x86_64/egg/pypad/cspad.py", line 439, in call
File "build/bdist.linux-x86_64/egg/pypad/cspad.py", line 1031, in _assemble_image
ValueError: qoff_col: -2 out of bounds [0,1900)

speed up peak matching in `score`

_match_peaks(self, sample_index) calculation time is currently Nobserved_peaks ^ Nmiller_indices => 4 miller indices are doable, anything beyond that takes forever. will see if I can find a clever workaround for this...

non-orthogonal basis vectors in standard geometry

This warning pops up when I run the standard geometry, if possible we should change so that the metrology does not have these issues or alternatively overrides the non-orthogonality if it occurs.

Loading: examples/gold-minus490mm.h5

Parsing: quad 0
WARNING: Metrology quality control failed for 2x1: 1
--> s/f vectors are not orthogonal
Val: -0.000013 // tol: 0.000010

Parsing: quad 1

Parsing: quad 2
WARNING: Metrology quality control failed for 2x1: 0
--> s/f vectors are not orthogonal
Val: 0.000101 // tol: 0.000010

Parsing: quad 3
WARNING: Metrology quality control failed for 2x1: 3
--> s/f vectors are not orthogonal
Val: -0.000010 // tol: 0.000010

check scaling of degrees of freedom that are optimized, such as z, theta, x, and y

the objective function currently calculates the angular average for each quad and maximizes the product, it's the hard to control the scaling... if an error-around-the-mean objective function is implemented, where each variable is separated, we can control the RMS scaling for each variable. The scaling may affect the optimization significantly

written mask from `genmask` in `cheetah` format does not seem to include automatically masked pixels

See last (Aug 1, 2013) comment in #21 , 14375 negative pixels were masked automatically but do not appear masked when running it in the cheetah...

pypad(master):575 11:42:55] genmask examples/gold-minus470mm.h5 -o test.h5 -f cheetah
Loading: examples/gold-minus470mm.h5

--- WELCOME TO PYPAD's INTERACTIVE MASKING ENVIRONMENT ---

Green pixels are masked.

Keystrokes


m : mask u : unmask r : reset
x : clear selection w : save & exit t : toggle pixel
q : exit w/o saving

Mouse


Right click on three or more points to draw a polygon around a
set of pixels. Then press m or u to mask or unmask that area.

You can also mask/unmask single pixels by clicking on them with
the mouse and pressing t to toggle the mask state.

Toggle Buttons (left)


nonbonded : Mask nonbonded pixels, and their nearest neighbours.
These pixels aren't even connected to the detector.

row 13 : In some old experiments, row 13 on one ASIC was
busted -- mask that (will be clear if this is needed)

threshold : You will be prompted for an upper and lower limit --
pixels outside that range are masked. Units are ADUs
and you can set only a lower/upper limit by passing
'None' for one option.

borders : Mask the borders of each ASIC. These often give
anomoulous responses. Recommended to mask one pixel
borders at least.

                      ----- // -----

Masked: 14375 negative pixels
Selected: (226, 289)
Masking convex area...
Wrote: test.h5

Numbers not sliders

It would be nice if one could optionally enter a number rather than clicking on sliders, e.g. for choosing thresholds, dilation, etc.

assembly of CSPAD image is wrong in genfilter, score, and assemble

Looking a bit closer at the scripts and their outputs there are inconsistencies between the coordinate system chosen for the various scripts. I propose we stick to the CXI convention, i.e. +X towards hutch door, +Y towards ceiling, and +Z downstream. This is what cheetah uses (i.e. https://github.com/sellberg/cheetah but this should still be true for https://github.com/antonbarty/cheetah) and outputs for it's assembled images. To view the front of the detector with a simple image viewer (which usually insists to put the origin in the upper left or lower left corner, such as plt.imshow()), this requires to mirror the X-axis prior to assembly, but this should only be done by the assembly script viewing the image and not saved in any files using an assembled coordinate system.

Below I compare the output of pypad with that of cheetah (marked r0135) and comment on differences (look for noisy ASIC):

genfilter

X-Y plane rotated 90 deg clockwise when using genfilter. Needs to transpose and mirror rows (i.e. Y-axis after transpose) in internally assembled array.

score

X-Y axes flipped and mirrored in X as well as Y when using score. Needs to transpose and mirror both rows and cols in internally assembled array.

assembled-gold

X-Y plane rotated 90 deg clockwise when using assemble. Needs to transpose and mirror row index (i.e. Y-axis after transpose) in internally assembled array.

This issue may or may not be related to #1

document API, CLI, conventions, philosophy

@sellberg I envision a short (5 pg?) TeX document in the pypad repo that explains:

(0) A paragraph about what problem we set out to solve with pypad, and the future we envision for it
(1) The CXI convention
(2) The layout of the CSPAD and how memory is mapped onto the geometry
(3) The basis vector convention used in pypad
(4) Unit conventions
(4.1) Any other conventions?
(5) A brief description of each script

If we plan out what to include here, I can put this together.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.