nutanix / calm-dsl Goto Github PK
View Code? Open in Web Editor NEWKeep Calm and DSL On!
Home Page: https://nutanix.github.io/calm-dsl/
License: Apache License 2.0
Keep Calm and DSL On!
Home Page: https://nutanix.github.io/calm-dsl/
License: Apache License 2.0
Useful for calm-saas setups having no host pc accounts. Some commands like calm create provider_spec
may fail for projects having remote pc registered
Need support for Calm macros in the AHV NIC property.
nics = [AhvVmNic.DirectNic.ingress("@@{macro}@@")]
Presently if we use calm watch app <app_name>
. It jumps to new window and displays the action runlogs there.
It may cause a problem when we run this command in non-interactive manner. For ex: (create a task in blueprint in which docker run will trigger this command)
, It will fail by error: fd=_sys.__stdout__.fileno()) _curses.error: setupterm: could not find terminal
Error can be resolved if we run this command in interactive command. Ex: (for above case, run docker in interactive (-it flag) way for "calm watch app" command).
Expected Behaviour: The error should not come. Command should be independent of how user is executing it.
Currently, calm describe app <name>
shows the application summary. It should also show service/deployment level details as exposed in the UI.
The workaround right now is to use calm describe app <name> --out json
to get the whole json and parse it manually.
Lay down the coding standards and process to be followed.
This is related to issue #66 too.
When launching an application from Calm DSL with calm launch
, if there are dynamic variables with predefined values or eScript/HTTP task, they don't get evaluated returning the possible values. The possible values are empty.
Additionally, when there is variable dependency during launch, for example the first variable gives you a list of predefined values, and whatever value you use will be evaluated in a second variable to present different values depending on the previous input, that will need to be cover in the CLI too.
Right now the path for the config file is $HOME/.calm/config.ini. It would be great to allow setting the config file path to a different location.
Steps to reproduce:
Seeing the error below when trying to decompile (download) a blueprint.
calm.dsl.tools.validator.validation_error: '' is not one of ['PROVISION_EXISTING_MACHINE']
Failed validating instance schema at ['type']:
"''\n"
By 'enum' validator in validating schema at ['properties']['type']:
|
default: PROVISION_EXISTING_MACHINE
enum: [PROVISION_EXISTING_MACHINE]
type: string
Related command:
calm run action <action_name> --app <app_name> --watch
when action run gets finished, odd unicode-like characters are left at the terminal cursor.
Ex:
**
class Service1(Service):
ENV = Variable.WithOptions.Predefined.string(
["DEV", "PROD"], default="DEV", is_mandatory=True, runtime=True
)
class Service2(Service1):
pass
**
In this case, attributes defined in Service1 (i.e. variables and actions) should also become the attributes of Service2 class.
This should happen for all valid CALM-DSL entities. It will help user to keep repetitive code at single place, and it will be more robust to errors
Right now we strip out secrets before creating blueprint.
Support to pass encrypted secret value and passphrase during bp create command.
Use case:
Will be helpful for decompiling secret while blueprint decompilation
Improve error handling.
+ calm create bp --file blueprint.py --name Calm_Era_0bd47dfb8d3c342860c951664edf8203d17b63be
Traceback (most recent call last):
File "/root/calm-dsl/venv/bin/calm", line 11, in <module>
load_entry_point('calm.dsl', 'console_scripts', 'calm')()
File "/root/calm-dsl/venv/lib/python3.6/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/root/calm-dsl/venv/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/root/calm-dsl/venv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/root/calm-dsl/venv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/root/calm-dsl/venv/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/root/calm-dsl/venv/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/root/calm-dsl/calm/dsl/cli/bp_commands.py", line 214, in create_blueprint_command
client, bp_file, name=name, description=description, force_create=force
File "/root/calm-dsl/calm/dsl/cli/bp_commands.py", line 168, in create_blueprint_from_dsl
bp_payload = compile_blueprint(bp_file)
File "/root/calm-dsl/calm/dsl/cli/bps.py", line 231, in compile_blueprint
bp_payload = UserBlueprintPayload.get_dict()
File "/root/calm-dsl/calm/dsl/builtins/models/entity.py", line 399, in get_dict
return json.loads(cls.json_dumps())
File "/root/calm-dsl/calm/dsl/builtins/models/entity.py", line 359, in json_dumps
separators=(",", ": ") if pprint else (",", ":"),
File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps
**kw).encode(obj)
File "/usr/lib64/python3.6/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib64/python3.6/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/root/calm-dsl/calm/dsl/builtins/models/entity.py", line 412, in default
return cls.compile()
File "/root/calm-dsl/calm/dsl/builtins/models/substrate.py", line 32, in compile
cdict = super().compile()
File "/root/calm-dsl/calm/dsl/builtins/models/entity.py", line 308, in compile
attrs = cls.get_all_attrs()
File "/root/calm-dsl/calm/dsl/builtins/models/entity.py", line 304, in get_all_attrs
return ncls.get_user_attrs()
File "/root/calm-dsl/calm/dsl/builtins/models/entity.py", line 232, in get_user_attrs
user_attrs[name] = getattr(cls, name, value)
File "/root/calm-dsl/calm/dsl/builtins/models/action.py", line 191, in __get__
node_visitor.visit(node)
File "/usr/lib64/python3.6/ast.py", line 253, in visit
return visitor(node)
File "/usr/lib64/python3.6/ast.py", line 261, in generic_visit
self.visit(item)
File "/usr/lib64/python3.6/ast.py", line 253, in visit
return visitor(node)
File "/usr/lib64/python3.6/ast.py", line 261, in generic_visit
self.visit(item)
File "/usr/lib64/python3.6/ast.py", line 253, in visit
return visitor(node)
File "/usr/lib64/python3.6/ast.py", line 263, in generic_visit
self.visit(value)
File "/usr/lib64/python3.6/ast.py", line 253, in visit
return visitor(node)
File "/root/calm-dsl/calm/dsl/builtins/models/action.py", line 67, in visit_Call
task = eval(compile(ast.Expression(node), "", "eval"), self._globals)
File "", line 78, in <module>
NameError: name 'json' is not defined
When we invoke command make test
, it runs all the test successfully.
But produces logging error at the end.
Results (157.68s):
202 passed
68 deselected
--- Logging error ---
Traceback (most recent call last):
File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/__init__.py", line 1028, in emit
stream.write(msg + self.terminator)
File "/Users/abhijeet.kaurav/calm-dsl/venv/lib/python3.7/site-packages/colorama/ansitowin32.py", line 41, in write
self.__convertor.write(text)
File "/Users/abhijeet.kaurav/calm-dsl/venv/lib/python3.7/site-packages/colorama/ansitowin32.py", line 162, in write
self.write_and_convert(text)
File "/Users/abhijeet.kaurav/calm-dsl/venv/lib/python3.7/site-packages/colorama/ansitowin32.py", line 187, in write_and_convert
self.write_plain_text(text, cursor, start)
File "/Users/abhijeet.kaurav/calm-dsl/venv/lib/python3.7/site-packages/colorama/ansitowin32.py", line 195, in write_plain_text
self.wrapped.write(text[start:end])
File "/Users/abhijeet.kaurav/calm-dsl/venv/lib/python3.7/site-packages/_pytest/capture.py", line 427, in write
self.buffer.write(obj)
ValueError: I/O operation on closed file
Call stack:
File "/Users/abhijeet.kaurav/calm-dsl/calm/dsl/db/handler.py", line 62, in close
LOG.debug("Closing connection to local DB")
File "/Users/abhijeet.kaurav/calm-dsl/calm/dsl/tools/logger.py", line 193, in debug
return logger.debug(self.__add_caller_info(msg), *args, **kwargs)
Message: ':62] Closing connection to local DB'
Arguments: ()
Findings: This is producible because atexit handler is registered with db.close()
method. And inside this method we are logging debug message Closing connection to local DB
.
When using Calm UI, a variable marked as mandatory automatically enables the runtime icon. In Calm DSL the runtime=True must be implicit when is_madatory=True
calm describe marketplace_bp/marketplace_item <bp_name> --out=json
should dump out the json for the blueprint
When running something like this:
calm run action <action_name> --app <app_name>
It seems there's no option to specify action variable values. For example, the ScaleOut actions allow specification of @@{COUNT}@@ value in the UI. This should also be an option in the DSL.
Maybe an additional @click.option for --payload?
This assumes that passing a JSON payload is the correct way to specify action variable values in the first place.
For ex:
class Service1(Service):
"""Service1 Service description"""
pass
class Default(Profile):
"""Default Profile description"""
deployments = [sample_deployment]
@action
def test_action():
"""test_action Action description"""
Service1.__create__(name="task1")
In above example, we have not override system action("create") for service "Service1". We are creating runbook task i.e. Calling service action from profile actions.
If we compile the above bp, compilation will fail with error AttributeError: type object 'Service1' has no attribute '__create__'
, untill we override these actions in bp file.
Flag:
calm create bp --force ...
Behavior:
Allow bp create with the same name. I will delete any existing bp with same name first.
So for now in ahv subnets we are using:
AhvVmNic(subnet="vlan.0"),
Apart from them we should also accept parameter cluster name for identifying subnet having same name across multiple PE.
i.e.
AhvVmNic(subnet="vlan.0", cluster_name="name"),
It would be great to have the watch option with calm launch
calm launch bp -w/--watch BLUEPRINT_NAME
Right now for launching a blueprint we have two options:
-i/--ignore_runtime_variables
to ignore runtime variablesNeed to have a way to supply those values in non-interactive mode (As approach 2 is not good for large no. of variables/entities).
Case:
Can't decompile blueprints if images under Configuration > Downloadable Image Configuration have a carriage return in the image description. Doesn't seem to matter where the carriage return is. Example:
https://cloud.centos.org/centos/7/images/
Command:
calm decompile bp blueprint1
Error:
[2020-07-14 02:30:11] calm.dsl.cli.bps [INFO] blueprint1 found
[2020-07-14 02:30:12] calm.dsl.cli.bps [INFO] Decompiling blueprint blueprint1
[2020-07-14 02:30:12] calm.dsl.decompile.decompile_render [INFO] Creating blueprint directory
[2020-07-14 02:30:12] calm.dsl.decompile.decompile_render [INFO] Rendering blueprint file template
[2020-07-14 02:30:12] calm.dsl.decompile.decompile_render [INFO] Formatting blueprint file using black
Traceback (most recent call last):
File "/home/chris/Data/solutions/nutanix/calm-dsl/venv/bin/calm", line 11, in <module>
load_entry_point('calm.dsl', 'console_scripts', 'calm')()
File "/home/chris/Data/solutions/nutanix/calm-dsl/venv/lib/python3.8/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/home/chris/Data/solutions/nutanix/calm-dsl/venv/lib/python3.8/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/home/chris/Data/solutions/nutanix/calm-dsl/venv/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/chris/Data/solutions/nutanix/calm-dsl/venv/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/chris/Data/solutions/nutanix/calm-dsl/venv/lib/python3.8/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/chris/Data/solutions/nutanix/calm-dsl/venv/lib/python3.8/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/chris/Data/solutions/nutanix/calm-dsl/calm/dsl/cli/bp_commands.py", line 114, in _decompile_bp
decompile_bp(name, with_secrets)
File "/home/chris/Data/solutions/nutanix/calm-dsl/calm/dsl/cli/bps.py", line 263, in decompile_bp
create_bp_dir(bp_cls=bp_cls, with_secrets=with_secrets)
File "/home/chris/Data/solutions/nutanix/calm-dsl/calm/dsl/decompile/decompile_render.py", line 28, in create_bp_dir
bp_data = format_str(bp_data, mode=FileMode())
File "/home/chris/Data/solutions/nutanix/calm-dsl/venv/lib/python3.8/site-packages/black.py", line 725, in format_str
src_node = lib2to3_parse(src_contents.lstrip(), mode.target_versions)
File "/home/chris/Data/solutions/nutanix/calm-dsl/venv/lib/python3.8/site-packages/black.py", line 836, in lib2to3_parse
raise exc from None
black.InvalidInput: Cannot parse: 19:64: CENTOS_IMAGE = vm_disk_package(name='CENTOS_IMAGE', description='
Image descriptions that failed:
Fix:
We need more validations on blueprint classes related to calm domain.
Need to have these validations on client side
For ex:
Blueprint.packages
(Blueprint Class attribute).Blueprint -> Profiles ->
flow and check each required class are defined in blueprint class attributes i.e Blueprint.services/packages/substrates/profiles
.calm create bp --file lamp-v4/lamp-v4.py --name jg-dsl-LAMP
[2020-04-15 14:06:11] calm.dsl.cli.bps [INFO] Syncing cache
[2020-04-15 14:06:45] calm.dsl.cli.bp_commands [INFO] Blueprint state: DRAFT
Traceback (most recent call last):
File "/root/.local/bin/calm", line 8, in
sys.exit(main())
File "/root/.local/lib/python3.7/site-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/root/.local/lib/python3.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/root/.local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/root/.local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/root/.local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/root/.local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/root/.local/lib/python3.7/site-packages/calm/dsl/cli/bp_commands.py", line 201, in create_blueprint_command
assert bp_state == "ACTIVE"
AssertionError
When running make test or make test_all, an exception is generated by an incompatibility between the installed version of pytest-sugar and pytest 5.4.0+.
Fix when running these versions together is to downgrade pytest to version 5.3.5:
(venv) pip3 uninstall pytest
(venv) pip3 install pytest==5.3.5
This requires changes in dev-requirements.txt. Confirmed after discussion with @abhijeetkaurav1st who is running pytest 5.3.4.
Related issue: Teemu/pytest-sugar#187
Old DSL implementation has readiness probe with default credential defined as below
readiness_probe["credential"] = ref(default_credential)
Same is failing with below error on latest master
File "lamp-v4.py", line 129, in MySQLSubstrate
readiness_probe["credential"] = ref(default_credential)
TypeError: 'ReadinessProbeType' object does not support item assignment
I had to change it to readiness_probe.credential = ref(default_credential) to make it work
Would be useful if we could set the description attribute similarly to that of other classes / methods - via python docstring...
Right now, DSL only supports cred of type basic
. It doesn't support by which we can give reference to existing credential in http tasks.
When using calm create provider_spec -t AHV_VM
it is missing the options for GPU, Serial and Boot.
Variables in Calm UI can have a description additional to the label. In DSL it's only the label available.
[2020-04-16 16:13:00 calm.dsl.api.connection [ERROR] Error Response: {'error': '{\n "api_version": "3.1", \n "code": 422, \n "message_list": [\n {\n "details": {}, \n "message": "Launch with \'HelloProfile\' Profile require configured accounts to be verified,\\n Please verify Account(s) from Settings page: TestProvider.", \n "reason": "INVALID ENTITY"\n }\n ], \n "state": "ERROR"\n}\n', 'code': 422}
Related command:
calm run action <action_name> --app <app_name> --watch
Dynamic terminal output displays some lines on the same lines as previous. Also, when process has finished, odd unicode-like characters are left at the terminal cursor.
Screenshot of broken lines attached.
Environment when reproducing issue:
When running commands like this:
calm validate provider_spec --file ./specs/MYSQL_AWS_provider_spec.yaml
The DSL will throw an exception such as the following:
AWS_provider_spec.yaml
[2020-06-10 04:00:36] calm.dsl.cli.main [INFO] File ./specs/MYSQL_AWS_provider_spec.yaml is invalid AHV_VM spec
Traceback (most recent call last):
File "/home/chris/Data/solutions/nutanix/calm-dsl/calm/dsl/cli/main.py", line 94, in validate_provider_spec
Provider.validate_spec(spec)
File "/home/chris/Data/solutions/nutanix/calm-dsl/calm/dsl/providers/base.py", line 68, in validate_spec
Validator.validate(spec)
File "/home/chris/Data/solutions/nutanix/calm-dsl/venv/lib/python3.8/site-packages/jsonschema/validators.py", line 353, in validate
raise error
calm.dsl.tools.validator.validation_error: Additional properties are not allowed : 'associate_public_ip_address'
Seems as if the DSL assumes AHV spec unless explicitly told that it is not AHV spec.
This error can be avoided by altering command to be as follows (note addition of --type parameter):
calm validate provider_spec --file ./specs/MYSQL_AWS_provider_spec.yaml --type AWS_VM
DSL should be able to take the provided spec and check more than just presence of "type: PROVISION_AWS_VM" or use of "--type" parameter, then validate accordingly.
Comment from @abhijeetkaurav1st as follows:
First check if type attribute is defined, else go for type in spec
This approach makes sense.
Use-Case:
When scripting, I rely almost entirely on the ability to see what API calls are made by the CLI and I literally copy these URLs into the my scripts.
Notes:
Use [API]
identifier in logs
Problem Statement:
Hi Team, in ahv Bootable disk check box getting disabled when we create a bp using decompiled bp. Discussed with Abhijeet Kaurav on the same.
Steps to reproduce:
Findings/Troubleshooting Done:
Disk index is getting changed. Attaching the screenshot.
Expectations:
Bootable check box should be enabled as it is.
When launching an application from CLI it is not evaluating first if the application name already exists. It is asking for any variable and then it fails. If you have several variables to complete and at the latest step the launch fails because the app name exists, it is then a bit disappointing.
When you launch a blueprint from the CLI, it is asking for variables that are part of day 2 actions and not in the profile.
app_profile.Default.action.Patch_Image.runbook.Default_Patch_Image_runbook.variable -> ControlVmIP.value []:
app_profile.Default.action.Patch_Image.runbook.Default_Patch_Image_runbook.variable -> IMG_URI.value []:
This must not be asked during initial launch. Only when running the action.
Would be nice to have command to create blueprint file based on no. of service, profiles provided as options.
Ex:
calm init bp --ns 2 -np 3.
It should create bp having these entities accordingly.
Options should be a little more informative to give some more flexibility over the blueprint.
calm compile/create
commands should blow a error if above behaviour is not followed inside the blueprint.
Field operation type
is introduced to merge disks and virtual disks into disks options only.
In Calm-dsl, disks and virtual disks are still exposed separately
Api response can be configured to support for pagination. For now default length for api response used is 20. So following steps may happen:
Right now, the output python file via decompile command shows the default value for various entities too.
For ex:
var9 = CalmVariable.Simple.Secret.date(
"",
label="var9_label",
regex="^((0[1-9]|[12]\d|3[01])/(0[1-9]|1[0-2])/[12]\d{3})$",
validate_regex=True,
is_mandatory=False,
is_hidden=False,
runtime=False,
)
It shows the default value for various attributes like is_mandatory, is_hidden, runtime etc. If we can just omit default values from python file, it would remove redundant code.
Two scenario we need to take care of:
Executing calm get bps -f jg
doesn't work
[2020-04-15 13:32:10] calm.dsl.api.connection [ERROR] Error Response: {'error': '{\n "api_version": "3.1", \n "code": 500, \n "message_list": [\n {\n "details": {}, \n "message": "list index out of range", \n "reason": "INTERNAL_ERROR"\n }\n ], \n "state": "ERROR"\n}\n', 'code': 500}
When a user workstation to Prism Central connection is unavailable, the Calm DSL does not throw an error.
Expected behaviour is for the DSL to throw an exception or warning when the configured Prism Central connection is unavailable.
A good example is:
calm create provider_spec
When Prism Central connection isn't available, this should fail gracefully. Observed behaviour is the DSL does "nothing" - no error or exception, just permanent CLI cursor.
calm init bp
doesn not allow to pass a name. I'd like to use something like calm init bp --name myblueprint
so I can use this command for the creation of all my new blueprints to keep consistency with the structure.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.