nobeam/latticejson

A JSON based lattice file format

LatticeJSON

LatticeJSON is a JSON based lattice file format. JSON is able to describe complex data structures, has a human readable syntax and is available in all common programming language. It is therefore an appropriate choice to characterize the magnetic lattice of a particle accelerator.

Specification

This repository contains the Specification of LatticeJSON in form of a JSON Schema.

Example

A LatticeJSON file for a FODO lattice:

{
  "version": "2.2",
  "title": "FODO Lattice",
  "info": "This is the simplest possible strong focusing lattice.",
  "root": "ring",
  "elements": {
    "d1": ["Drift", {"length": 0.55}],
    "q1": ["Quadrupole", {"length": 0.2, "k1": 1.2}],
    "q2": ["Quadrupole", {"length": 0.4, "k1": -1.2}],
    "b1": ["Dipole", {"length": 1.5, "angle": 0.392701, "e1": 0.1963505, "e2": 0.1963505}]
  },
  "lattices": {
    "cell": ["q1", "d1", "b1", "d1", "q2", "d1", "b1", "d1", "q1"],
    "ring": ["cell", "cell", "cell", "cell", "cell", "cell", "cell", "cell"]
  }
}

LatticeJSON CLI

This repository also contains a Python based command-line tool which is able validate and convert LatticeJSON files into other common lattice file formats and vice versa.

You can install and update it using pip or pipenv:

pip install -U latticejson

Validate a LatticeJSON file:

latticejson validate /path/to/lattice.json

Convert an elegant lattice file to the LatticeJSON format:

latticejson convert --to json /path/to/lattice.lte

Autoformat one or more LatticeJSON files:

latticejson autoformat /path/to/lattice.json ...

To activate Bash completion add

eval "$(_LATTICEJSON_COMPLETE=source latticejson)"

to your .bashrc. Or, create an activation script with:

_LATTICEJSON_COMPLETE=source latticejson > latticejson-complete.sh

License

GNU General Public License v3.0

latticejson's Issues

Package structure and startup time

Currently every public object is exposed via the top-level __init__.py. This effects the startup time when importing latticejson as library but also the startup time of the CLI (because it is a submodule and the parent package gets always executed). Maybe it would be better to use the scipy-approach: Keep the Top-level __init__.py empty and use subpackages to import functionality. But this would be somewhat inconvenient for a user...

To solve the issue for the CLI, the cli.py could be installed separately.

Maybe the __getattr__ of PEP 562 can be used to lazy load the submodules into the top-level name space of the __init__.py.

See also:

• How to only import sub module without exec __init__.py in the package
• One class per file causes double names in python

Add custom type `location` to CLI

Add custom type Location: path or url

http://click.palletsprojects.com/parameters/#implementing-custom-types

Multipole Elegant vs MADx

Multipoles are very differently implemented:

When I convert from madx or elegant returns "2 different lattice json file, we should discuss which one we want to keep and adapt the converters and parsers. So converting back to elegant and madx is return false formats - in the example below elegant is correct madx is faulty:

to madx

OXX: MULTIPOLE, L=0.0, KNL=-679.686, BOE=3;
OXY: MULTIPOLE, L=0.0, KNL=1331.5866, BOE=3;
OYY: MULTIPOLE, L=0.0, KNL=-617.0892, BOE=3;

should be

oxx    : multipole, knl={0,0,0,-679.6860};
oxy    : multipole, knl={0,0,0,1331.5866};
oyy    : multipole, knl={0,0,0,-617.0892};

to elegant

OXX: MULT, L=0.0, KNL=-679.686, ORDER=3
OXY: MULT, L=0.0, KNL=1331.5866, ORDER=3
OYY: MULT, L=0.0, KNL=-617.0892, ORDER=3

Include energy to lattice file

Currently the energy is not defined for the lattice!
It is needed for radiation integrals ...

Energy should be included as global parameter, because it is not element specific... nearly all elements depend on energy

@TMsangohan @andreasfelix

Creator/editor/modifier and email?

I would like to have an extra meta data field that returns data on the creator/modifier of the lattice json file, next to the name of the lattice.

arctan madx

Math failure on arctan. Can we add arctan to the math expressions?

Problem: math package uses atan instead of arctan

Proposed solution by editing parse.py lines 44-45:
add if statement: if function == arctan: function = atan ?

`main_lattice` instead of `sub_lattices`

Paul mentioned he would find intuitive if we would rename the sub_lattices attribute into lattices. Also he would drop the current lattice attribute and put it into the newly named lattices attribute. If we go that way we would need a new attribute which would indicate the main_lattice similar to MADX's USES statement.

How this lattice file would look like this:

{
  "name": "FODO_RING",
  "description": "This is the simplest possible strong focusing lattice.",
  "elements": {
    "D1": {"type": "Drift", "length": 0.55},
    "Q1": {"type": "Quadrupole", "length": 0.2, "k1": 1.2},
    "Q2": {"type": "Quadrupole", "length": 0.4, "k1": -1.2},
    "B1": {"type": "Dipole", "length": 1.5, "angle": 0.392701, "e1": 0.1963505, "e2": 0.1963505}
  },
  "lattices": {
    "FODO": ["Q1", "D1", "B1", "D1", "Q2", "D1", "B1", "D1", "Q1"],
    "RING": ["FODO", "FODO", "FODO", "FODO", "FODO", "FODO", "FODO", "FODO"]
  },
  "main_lattice": "RING"
}

I have personally no strong opinion on that and would be open to implement it. But I also do not see the benefit over the current approach.

@PaulGoslawski Could you maybe briefly explain why you would prefer it this way?

@TMsangohan Do you have an opinion on that?

BPMs

@andreasfelix

Another one ;), urgently also need bpms.

T.

lattice versus strength files

Ok I am going to be difficult again.

I am working with Pierre on orbit bumps to scan apertures, which made me look into my old code (python and mathematica) and definition files (aperture).

To be short:
At cern the lattice is stored in madx format, but the settings of the magnets is stored in a separate strength file (a bit similar to our power supplies but in physics units). This allows to quickly add permanent orbit bumps into the lattice (to dynamically take care of sinking detectors - yes they sink in the ground they are heavy - and steer the beams around ULOs or Unidentified Lying Objects) in the sense that the user of the lattice does not see this, even when he is changing the settings of other magnets (the bump is automatically added in madx by calling the strength files - and they are also loaded this way in the LHC sequencer when the machine is run e.g. injection optics, ramping optics - the bumps need to ramp up too - collision optics).

Such things will be part of my future standard tools and are not easy to implement on the current format of lattice json or to add it simply to the exported madx file, so again I will argue to take the numerical values of magnet strengths out and load a strength file or several strength files separately.

In my opinion, strengths are not lattice, but power supplies are - they are the hardware and not the k-values.

Let us please discuss this point in the very near future as I also need this rather quickly.

Add Workflow to publish to https://test.pypi.org/

[Security] Workflow publish.yml is using vulnerable action actions/checkout

The workflow publish.yml is referencing action actions/checkout using references v1. However this reference is missing the commit a6747255bd19d7a757dbdda8c654a9f84db19839 which may contain fix to the some vulnerability.
The vulnerability fix that is missing by actions version could be related to:
(1) CVE fix
(2) upgrade of vulnerable dependency
(3) fix to secret leak and others.
Please consider to update the reference to the action.

Specification of the lattice file

Specification

To Discuss:
Markers
Angles in grad?
Multipole class to inherit from
Multipole names Quad vs Quadrupole
uppercase/lowercase

Lattice
    name
    description
    elements
    lattice
    sub_lattices
    definitions:
        Element
            type
            description
            length
            dx
            dy
            ds
            tilt
            dl
        Drift
        Dipole
            angle, radius, ps_value (oneOf)
            e1, h1 (oneOf)
            e2, h2 (oneOf)
            conversion_factor_ps (optional)
            conversion_factor_angle (optional)
        Quadrupole
            k1, ps_value (oneOf)
            conversion_factor_ps (optional)
            conversion_factor_angle (optional)
        Sextupole
            k2, ps_value (oneOf)
            conversion_factor_ps (optional)
            conversion_factor_angle (optional)
        Octupole
            k3, ps_value (oneOf)
            conversion_factor_ps (optional)
            conversion_factor_angle (optional)
        Cavity
            todo (Tom)

Should Markers be in the lattice file?

Interpretation of lattice versions

I am currently looking into versioned data storage, inspired by Felix, for more complicated data (epics databases, meta data, ...). One possible issue that I see is the following: Do you consider lattices where a single quad setting is changed as different versions of a lattice? If yes, that will lead to a large set of lattice files (in the order of millions for what I am planning). So how do you define the "version" of a lattice?

Add a "tool" key similar to pyproject.toml

@tomerten

This would allow plugins store data within latticejson

Add option to order element/lattice keys by name/type in autoformat.

Docker/singularity tool

Create a (u)Docker and Singularity container with the package installed?

Add cavity to specification

I think we should add a cavity element to the specification!

@TMsangohan, do you have some suggestions. If you could write down a short summary off the cavity element similar to #2, I would add it to the schema.json.

Corrector magnets

@andreasfelix
What about corrector magnets?

Those I also urgently need.

Sorry for the many requests...
Tom

Exact angles

Currently to there is no way to specify the exact value of an angle.

Possible solutions:

• Change angle units to multiples of PI (or new attribute: angle_pi)
• Change angle units to grads (or new attribute: angle_grad)
• Add a PI constant to the latticeJSON specification

Add injections elements/kicker to specification

@TMsangohan what exactly do you have in mind? If you could write down a short summary of the kicker element (similar to #2) I could add it to the specification by tomorrow.

Kicker Element of elegant

[Security] Workflow main.yml is using vulnerable action actions/checkout

The workflow main.yml is referencing action actions/checkout using references v1. However this reference is missing the commit a6747255bd19d7a757dbdda8c654a9f84db19839 which may contain fix to the some vulnerability.
The vulnerability fix that is missing by actions version could be related to:
(1) CVE fix
(2) upgrade of vulnerable dependency
(3) fix to secret leak and others.
Please consider to update the reference to the action.

Add MAD-X format converter

Insert/remove element

Should we include insert and remove functionality?

[Discussion] Best way to organize elements/lattices

Option A - {<name>: {"type": <type>, <attributes>}} (current structure)

"elements": {
    "D1": {"type": "Drift", "length": 0.55},
    "Q1": {"type": "Quadrupole", "length": 0.2, "k1": 1.2},
    "Q2": {"type": "Quadrupole", "length": 0.4, "k1": -1.2},
    "B1": {"type": "Dipole", "length": 1.5, "angle": 0.392701, "e1": 0.1963505, "e2": 0.1963505}
}

How to loop over all elements:

for name, definition in elements.items():
    type_ = attributes["type"].pop()   
    attributes = definition

How to access single element by name:

attributes = elements["NAME"].copy() # copy to avoid changing original dict
type_ = attributes.pop("type")

Option B - {<name>: [<type>, <attributes>]}

"elements": {
    "D1": ["Drift", {"length": 0.55}],
    "Q1": ["Quadrupole", {"length": 0.2, "k1": 1.2}],
    "Q2": ["Quadrupole", {"length": 0.4, "k1": -1.2}],
    "B1": ["Dipole", {"length": 1.5, "angle": 0.392701, "e1": 0.1963505, "e2": 0.1963505}]
}

How to loop over all elements:

for name, (type_, attributes) in elements.items():

How to access single element by name:

type_, attributes = elements["NAME"]

Option C - [[<name>, <type>, <attributes>]]

"elements": [
    ["D1", "Drift", {"length": 0.55}],
    ["Q1", "Quadrupole", {"length": 0.2, "k1": 1.2}],
    ["Q2", "Quadrupole", {"length": 0.4, "k1": -1.2}],
    ["B1", "Dipole", {"length": 1.5, "angle": 0.392701, "e1": 0.1963505, "e2": 0.1963505}]
]

How to loop over all elements: (most concise)

for name, type_, attributes in elements:

How to access single element by name: (Slower because one has to traverse list)

name, type_, attributes = next(x for x in elements if x[0] == "NAME") 

Sort LatticeJSON keys when autoformatting

include EDRIFT for elegant parser

Conversion fails on lte files containing EDRIFT.

Can you add this?

Thanks.

[Discussion] pylatticejson

Should there be a separate library to load/save/manipulate latticeJSON files in Python or should this be a part of apace?

If we decide to use a separate library, what should this library be able to do?

Scope of this library:

• ...

Open Questions

• Should this library be specific to latticeJSON or a general Python lattice development library?
• Does this library load lattice files into a Python dictionaries or get all elements/sub_lattices converted to Python object (similar to apace)?
• If we decide us for Python objects, should we include a signal system, so that it is possible register callbacks to value changes?
• Should it contain a print_tree function?

Unit testing

@andreasfelix @PaulGoslawski
We should include many more unit tests, so that if I change anything I am sure I do not break something. I lost half a day checking if previous things still work and I am still not sure if I now broke something or not (on my branch).

Testing is crucial if you are working as a team on the same code.

k1 for dipole magnets

@tomerten @PaulGoslawski Do you think it is reasonable to add the k1 attribute to the Dipole magnet? Or, would it be better to define a new element type which has an k0 and k1 attribute?

cli shows wrong version number

$latticejson --version
latticejson, version 0.0.2

isntalled 0.0.4

Nested JSON tree print

The print tree method you used in apace does not work completely properly if the nesting level is larger than 3. I have a solution, which is the implementation of the linux tree command in python but applied to dictionaries.

Should we add it?