xtuml/pyxtuml

Python library for parsing, manipulating, and generating BridgePoint xtUML models

pyxtuml

pyxtuml is a python library for parsing, manipulating, and generating BridgePoint xtUML models.

1   Documentation

Information on how to use pyxtuml is available at Read the Docs.

2   Reporting bugs

If you encounter problems with pyxtuml, please file a github issue. If you plan on sending pull request which affect more than a few lines of code, please file an issue before you start to work on you changes. This will allow us to discuss the solution properly before you commit time and effort.

3   License

pyxtuml is licensed under the LGPLv3+, see COPYING and COPYING.LESSER for more information.

pyxtuml's Issues

pyxtuml allows creation of duplicate associations

The pyxtuml defines Association as "connects two metaclasses to each other via two directed links." The define_association function will create both links, but doesn't check to see if the Association was previously defined.

An example of this can be found in the pyrsl test, test_relate_using. In this test, Association R1 is created twice using a reversed source_phrase and target_phrase in the second creation of R1. The second creation of R1 should be unnecessary, because the two links are created in the first creation of R1.

The use of "source" and "target" in the parameter names is probably confusing in the usage of define_association, because the association should be non-directional.

If the creation of duplicate Associations is required, then this is probably a bigger issue.

pre-build fails when informal diagrams exist in project

Getting lots of "schema mismatch" errors on projects which contain non-executable UML diagrams. I think it is simply a need to update bridgepoint/schema.py. I plan to submit a patch soon.

O_DBATTRs and schema_gen

I have not looked at the code, but results seem to show that dbattrs do not get fields in the schema.

Tests fail with Python 3.11

Describe the bug
Previously passing tests fail when running under Python 3.11

To Reproduce
Steps to reproduce the behavior:

1. Install Python 3.11
2. Run the tests

Expected behavior
The tests should pass

Environment (please complete the following information):

• OS: macOS
• Metamodel version (or Bridgepoint version used)
• Python version 3.11
• pyxtuml version 2.3.1

Additional context
This appears to be an issue with ply and the underlying regular expression engine used in the lexer. It may be related to this report: dabeaz/ply#282

Pyxtuml appears to have a limitation on function size.

See https://support.onefact.net/issues/12193 for reference.

Trying to build the Ciera pei example as a stand-alone project caused a recursion failure in pyxtuml when compiling 'do_creates()' function.
Moving the last 20 lines of code to an auxiliary function proved to be a work-around.
see: https://github.com/amullarney/PersistenceTest/tree/12020_unformalizedTest - do_creates, do_create2 in functions package.

CI with appveyor and travis out of date

The CI integration is currently out of order. Is there still a need for windows binaries?

updates to prebuilder

From a conversation on the xtUML Community hangout:

pyxtuml has an OAL parser that replicates the functionality of the BridgePoint prebuilder.  Levi and I have looked at this (again) recently.  It has an advantage in that it does not require BridgePoint or Eclipse or even Java.
I was curious to compare the output of both OAL parsers (BridgePoint prebuilder and pyxtuml).  There are a couple of differences.
The biggest difference is that pyxtuml is emitting ACT_SAB instead of ACT_TAB for actions on transitions.  This is probably a bug and probably easy to fix.
I will report it.
There are a couple of other smaller differences that I am exploring.  But the experience is generally positive and conducive to command line build environments.
pyxtuml does not populate some of the more dubious line and column number information in the VALUE subsystem instances.

The ACT_TAB issue will need to be fixed for parsing models having actions on transitions.
Adding set arithmetic support is the next most important change. This will occur around line 1729 of oal.py.

Here is the analysis note for when set operations were added to OAL. The note even refers to pyrsl support of set operations in RSL. Now, the OAL grammar can be updated to allow them in the action language.

https://github.com/xtuml/bridgepoint/blob/master/doc-bridgepoint/notes/5007_set_operations/5007_set_operations_ant.md

Duplicate association numbers in different components fail to compile.

A test case demonstrating this available at:
https://github.com/amullarney/AdHocTests/tree/12006_relate.
This is also: https://support.onefact.net/issues/12006.
Observations in the discussion of that issue appear to implicate pyxtuml.

RuntimeWarning running pre-build with python 3.8

In later versions of Python I am getting this warning when using the pre-builder:

/Users/levi/.pyenv/versions/3.8.2/lib/python3.8/runpy.py:126: RuntimeWarning: 'bridgepoint.prebuild' found in sys.modules after import of package 'bridgepoint', but prior to execution of 'bridgepoint.prebuild'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))

I found this StackOverflow article

I was able to get rid of the warnings by removing these two lines from bridgepoint/__init__.py (as suggested in the article):

from .prebuild import prebuild_action
from .prebuild import prebuild_model

but I'm not sure if that will break the API for other users. It passed all the unit tests with python 3.8 on my machine.

Pyxtuml fails when a derived attribute is specified in a package-referenced class.

[ERROR] File "/home/amullarney/.local/lib/python3.8/site-packages/bridgepoint/prebuild.py", line 1740, in label
[ERROR] return '::'.join([self.c_c.Name, self._o_obj.Name, o_attr.Name])
[ERROR] AttributeError: 'NoneType' object has no attribute 'Name'
[ERROR] pyxtuml exited with code 1

constant specification confused for enumeration

pyxtuml prebuild cannot discriminate a fully qualified constant specification from an enumeration.

Consider a constant specification TimeConversion with constants.

and the following action language.

The lines commented out cause pyxtuml.prebuild a problem.

Here is the stack trace. Notice accept_EnumNode.

[INFO] Performing pyxtuml pre-build...
[ERROR] Traceback (most recent call last):
[ERROR]   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
[ERROR]     "__main__", fname, loader, pkg_name)
[ERROR]   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
[ERROR]     exec code in run_globals
[ERROR]   File "/Library/Python/2.7/site-packages/pyxtuml-2.1a0-py2.7.egg/bridgepoint/prebuild.py", line 1914, in <module>
[ERROR]     main()
[ERROR]   File "/Library/Python/2.7/site-packages/pyxtuml-2.1a0-py2.7.egg/bridgepoint/prebuild.py", line 1908, in main
[ERROR]     prebuild_model(m)
[ERROR]   File "/Library/Python/2.7/site-packages/pyxtuml-2.1a0-py2.7.egg/bridgepoint/prebuild.py", line 1873, in prebuild_model
[ERROR]     prebuild_action(inst)
[ERROR]   File "/Library/Python/2.7/site-packages/pyxtuml-2.1a0-py2.7.egg/bridgepoint/prebuild.py", line 1851, in prebuild_action
[ERROR]     return walker.accept(root)
[ERROR]   File "/Library/Python/2.7/site-packages/pyxtuml-2.1a0-py2.7.egg/xtuml/tools.py", line 220, in accept
[ERROR]     r = fn(node, **kwargs)
[ERROR]   File "/Library/Python/2.7/site-packages/pyxtuml-2.1a0-py2.7.egg/bridgepoint/prebuild.py", line 1562, in accept_BodyNode
[ERROR]     return ActionPrebuilder.accept_BodyNode(self, node)
[ERROR]   File "/Library/Python/2.7/site-packages/pyxtuml-2.1a0-py2.7.egg/bridgepoint/prebuild.py", line 439, in accept_BodyNode
[ERROR]     self.accept(node.block.statement_list)
[ERROR]   File "/Library/Python/2.7/site-packages/pyxtuml-2.1a0-py2.7.egg/xtuml/tools.py", line 220, in accept
[ERROR]     r = fn(node, **kwargs)
[ERROR]   File "/Library/Python/2.7/site-packages/pyxtuml-2.1a0-py2.7.egg/bridgepoint/prebuild.py", line 458, in accept_StatementListNode
[ERROR]     act_smt = self.accept(child)
[ERROR]   File "/Library/Python/2.7/site-packages/pyxtuml-2.1a0-py2.7.egg/xtuml/tools.py", line 220, in accept
[ERROR]     r = fn(node, **kwargs)
[ERROR]   File "/Library/Python/2.7/site-packages/pyxtuml-2.1a0-py2.7.egg/bridgepoint/prebuild.py", line 781, in accept_AssignmentNode
[ERROR]     v_val_r = self.accept(node.expression)
[ERROR]   File "/Library/Python/2.7/site-packages/pyxtuml-2.1a0-py2.7.egg/xtuml/tools.py", line 220, in accept
[ERROR]     r = fn(node, **kwargs)
[ERROR]   File "/Library/Python/2.7/site-packages/pyxtuml-2.1a0-py2.7.egg/bridgepoint/prebuild.py", line 1009, in accept_BinaryOperationNode
[ERROR]     v_val_r = self.accept(node.right)
[ERROR]   File "/Library/Python/2.7/site-packages/pyxtuml-2.1a0-py2.7.egg/xtuml/tools.py", line 220, in accept
[ERROR]     r = fn(node, **kwargs)
[ERROR]   File "/Library/Python/2.7/site-packages/pyxtuml-2.1a0-py2.7.egg/bridgepoint/prebuild.py", line 1009, in accept_BinaryOperationNode
[ERROR]     v_val_r = self.accept(node.right)
[ERROR]   File "/Library/Python/2.7/site-packages/pyxtuml-2.1a0-py2.7.egg/xtuml/tools.py", line 220, in accept
[ERROR]     r = fn(node, **kwargs)
[ERROR]   File "/Library/Python/2.7/site-packages/pyxtuml-2.1a0-py2.7.egg/bridgepoint/prebuild.py", line 1109, in accept_EnumNode
[ERROR]     relate(v_val, s_dt, 820)
[ERROR]   File "/Library/Python/2.7/site-packages/pyxtuml-2.1a0-py2.7.egg/bridgepoint/prebuild.py", line 43, in relate
[ERROR]     assert xtuml.relate(*args, **kwargs)
[ERROR] AssertionError
[ERROR] pyxtuml exited with code 1

Function evaluation in select...where causes assertion error.

The following select statement causes pyxtuml to exit:
select any emp from instances of Employee
where ( selected.busy() );

[ERROR] assert xtuml.relate(*args, **kwargs)
[ERROR] AssertionError
[ERROR] pyxtuml exited with code 1

Pre-build does not include label for state machine actions

The "Label" field of ACT_ACT is calculated, but not set for transition actions and state actions.

Prebuilder struggles with invocations of operations/functions via package reference

When a class-based operation is invoked through a package reference, the prebuilder confuses it with a port invocation:

ROR] File "/home/user/.local/lib/python2.7/site-packages/xtuml/tools.py", line 220, in accept
[ERROR] r = fn(node, **kwargs)
[ERROR] File "/home/user/.local/lib/python2.7/site-packages/bridgepoint/prebuild.py", line 1279, in accept_InvocationStatementNode
[ERROR] self.accept(node.invocation, act_smt=act_smt)
[ERROR] File "/home/user/.local/lib/python2.7/site-packages/xtuml/tools.py", line 220, in accept
[ERROR] r = fn(node, **kwargs)
[ERROR] File "/home/user/.local/lib/python2.7/site-packages/bridgepoint/prebuild.py", line 1291, in accept_ImplicitInvocationNode
[ERROR] return self.accept_PortInvocationNode(node, act_smt=act_smt)
[ERROR] File "/home/user/.local/lib/python2.7/site-packages/bridgepoint/prebuild.py", line 1382, in accept_PortInvocationNode
[ERROR] spr_rep = one(c_ir).C_R[4009].SPR_REP4500
[ERROR] AttributeError: 'NoneType' object has no attribute 'Id'
[ERROR] pyxtuml exited with code 1

When a function is invoked through a package reference, the prebuilder exits with an assertion error:

[ERROR] File "/home/user/.local/lib/python2.7/site-packages/xtuml/tools.py", line 220, in accept
[ERROR] r = fn(node, **kwargs)
[ERROR] File "/home/user/.local/lib/python2.7/site-packages/bridgepoint/prebuild.py", line 1279, in accept_InvocationStatementNode
[ERROR] self.accept(node.invocation, act_smt=act_smt)
[ERROR] File "/home/user/.local/lib/python2.7/site-packages/xtuml/tools.py", line 220, in accept
[ERROR] r = fn(node, **kwargs)
[ERROR] File "/home/user/.local/lib/python2.7/site-packages/bridgepoint/prebuild.py", line 1363, in accept_FunctionInvocationNode
[ERROR] relate(v_val, s_dt, 820)
[ERROR] File "/home/user/.local/lib/python2.7/site-packages/bridgepoint/prebuild.py", line 43, in relate
[ERROR] assert xtuml.relate(*args, **kwargs)
[ERROR] AssertionError
[ERROR] pyxtuml exited with code 1

A simple xtUML model illustrating the issue is attached.

PkgRefIssues.zip

Cannot import xtuml module

When importing the 'xtuml' module, the 'tools' module gets imported and there is an exception in the import of that module related to the 'collections.abc' module:

To Reproduce
Steps to reproduce the behavior:

1. Create a script "test.py" containing a single line "import xtuml"
2. Run the script: python test.py
3. See the following output:

Traceback (most recent call last):
  File "/Users/levi/git/plus2json/test2.py", line 1, in <module>
    import xtuml
  File "/Users/levi/.pyenv/versions/3.10.10/lib/python3.10/site-packages/xtuml/__init__.py", line 24, in <module>
    from .tools import IdGenerator
  File "/Users/levi/.pyenv/versions/3.10.10/lib/python3.10/site-packages/xtuml/tools.py", line 131, in <module>
    class OrderedSet(collectionsAbc.MutableSet):
AttributeError: module 'collections' has no attribute 'MutableSet'

Environment (please complete the following information):

• OS: macOS Ventura
• Python version: 3.10.10
• pyxtuml version: 2.3.1

Additional context
I found this python issue: https://bugs.python.org/issue45776.

The problem is that in tools.py, the "collections.abc" module is not being explicitly imported.

Transient arrays are not supported?

Statements of the form:
Labels[index] = "";
Labels[index] = spec.Label;
produce:
ERROR:bridgepoint.prebuild:Unable to evaluate constant expression (V_TVL)

feature request: C and Java translation of templates

A subset of RSL is actually a template language.

• plain text
• substitution variables
• string formatting
• maybe 'if'
  can be considered "template".
  It would be cool to have feature that translated this subset into C and Java. :)

Empty value in referential causes relationship to not be formalized

The process of building BridgePoint relies on CME data (e.g. org.xtuml.bp.core/sql/context_menu.pei.sql).

The build was giving me an error at org.xtuml.bp.core/arc/function_body.inc:209, saying "Specialism was not a valid attribute".

Via print statement debugging, at the top of this function I can see we are getting a W_WOF instance, but not getting a CME which has a 1-1 relationship with W_WOF. At the top of the function I have print statement debugging that is outputting:

     [exec] create_global_action.inc: 209:  INFO:  SKB: Processing CME function CL_IC_Delete
     [exec] function_body.inc: 34:  INFO:  ERROR: Could not locate associated CME instance for W_WOF instance with name CL_IC_Delete and Sync_ID 8173

From this archetype line: .print "ERROR: Could not locate associated CME instance for W_WOF instance with name ${wfl.Name} and Sync_ID ${wfl.Sync_ID}"

It is the empty referential ID problem again. CME(1)--[R2012]--(1)W_WOF relies on a set of 3 referential IDs that are formalized as attributed inside W_WOF.

All 3 attributes in the set are strings. In our CME PEI data, some of these strings are empty. For those elements, the CME to W_WOF relationship is simply not hooked up. In the cases where none of the strings are empty, the relationships are there.

Generator did not give me any error at runtime about the instances not being related. We've hit this several times and adding this issue to document the solution.

Add ooaofooa schema for BridgePoint 7

Most users have migrated from BridgePoint 6 to BridgePoint 7, and the latter relies on an updated ooaofooa schema.

Remove special case for derived attribute field initialization

The problem I hit is due to the fact that T_TPS.Prev_TPS_ID is a referential attribute and thus is a derived attribute. Our PEI data in ooaofooa_heirarch.pei.sql does not specify values for every field. When pyxtuml loads the data it sets the value of T_TPS.Prev_TPS_ID to the python value "None" rather than "0" even though the field is an INTEGER. It does this because it is a derived attribute (see https://github.com/keithbrown/pyxtuml/blob/8442_build_bp_pyrsl/xtuml/model.py#L480). Here inst.__a__ is the set of attributes and inst.__d__ is the set of derived attributes in a given class (refer to line 280 in the same file).

pyxtuml behaviour is definitely different than old generator behaviour here, so I think that pyxtuml should not treat derived attributes specially and should just use whatever type-based default is done for non-derived attributes.

We could update our PEI data to include values for all fields instead of relying to generator to do the right thing. A script or small parser could make sure it was fully filled out. Sometimes it is just cleaner to read to leave out empty fields. But a post-processing to go from readable/editable to syntactically correct is probably a good way to go (given time).

persist links

It would be really nice to be able to relate instances without the need for formalized associations using identifiers and referentials.
Then, it would be nice to be able to persist these links (instantiations of associations) and then to load them.

The schema defines associations.
Recently relate/unrelate statements (even if presently they are simply shortcuts for the copying of IDs into referentials) have been introduced.

Serialize all instances as normal (possibly annotating each instance with an index number (in a comment)). Each index is simply the offset from the first element of a particular class instance.
Serialize links in the following format (for SQL):
LINK R ( "", index, "", index [, "", index ] );