Giter Site home page Giter Site logo

manageiq-automation_engine's Issues

Refactor statemachine_task_status methods into mixin

The statemachine_task_status method exists across several service models with only minor differences and should be refactored into a common mixin.

Files to modify:

  • miq_ae_service_automation_task.rb
  • miq_ae_service_miq_host_provision.rb
  • miq_ae_service_miq_provision.rb
  • miq_ae_service_miq_provision_task.rb
  • miq_ae_service_service_reconfigure_task.rb
  • miq_ae_service_service_template_provision_task.rb
  • miq_ae_service_vm_migrate_task.rb

`remote_url` deprecation

  • Deprecate remote_url.

  • Make it call external_url or use the same column.

  • Add a migration that removes the URL column from the SystemConsole table (check if it's not used by the proxying piece).

  • Later on remove the remove_url mixin.

More background available in #328

vm_reconfigure_request error checking quota's

A user with only a tenant quota for memory set to 256GB is trying to reconfigure a VM and add an additional HDD to his machine. We are seeing an error in the quota used method when calling check_quota. When the quota is turned off then he is able to provision the drive without error.

We are running MIQ Fine-4 and VMware vcenter 6.5.
Gist contains the logs for the vm_reconfigure_request.
https://gist.github.com/Stromweld/d59742714eda7e732378e262217dd412

Embedded Ansible erroring on 5.10.0.20

Hey guys, Embedded Ansible is periodically failing for me on latest CFME appliance, putting following into evm.log:

[----] E, [2018-10-25T10:25:02.644473 #30085:338f88] ERROR -- : MIQ(ManageIQ::Providers::EmbeddedAnsible::AutomationManager::Refresher#refresh) EMS: [Embedded Ansible Automation Manager], id: [1000000000003] Unable to perform refre
sh for the following targets:
[----] E, [2018-10-25T10:25:02.644651 #30085:338f88] ERROR -- : MIQ(ManageIQ::Providers::EmbeddedAnsible::AutomationManager::Refresher#refresh)  --- ManageIQ::Providers::EmbeddedAnsible::AutomationManager [Embedded Ansible Automati
on Manager] id [1000000000003]
[----] I, [2018-10-25T10:25:02.657125 #30085:338f88]  INFO -- : MIQ(ManageIQ::Providers::EmbeddedAnsible::AutomationManager::Refresher#refresh) Refreshing all targets...Complete
[----] E, [2018-10-25T10:25:02.657456 #30085:338f88] ERROR -- : MIQ(MiqQueue#deliver) Message id: [1000000020918], Error: [undefined method `result_stdout' for #<AnsibleTowerClient::ProjectUpdate:0x0000000016318970>]
[----] E, [2018-10-25T10:25:02.657649 #30085:338f88] ERROR -- : [ManageIQ::Providers::BaseManager::Refresher::PartialRefreshError]: undefined method `result_stdout' for #<AnsibleTowerClient::ProjectUpdate:0x0000000016318970>  Metho
d:[block (2 levels) in <class:LogProxy>]
[----] E, [2018-10-25T10:25:02.657829 #30085:338f88] ERROR -- : /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:67:in `refresh'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:11:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:103:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:102:in `each'
/var/www/miq/vmdb/app/models/ems_refresh.rb:102:in `refresh'
/var/www/miq/vmdb/app/models/miq_queue.rb:455:in `block in dispatch_method'
/usr/share/ruby/timeout.rb:93:in `block in timeout'
/usr/share/ruby/timeout.rb:33:in `block in catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:108:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:453:in `dispatch_method'
/var/www/miq/vmdb/app/models/miq_queue.rb:430:in `block in deliver'
/var/www/miq/vmdb/app/models/user.rb:267:in `with_user_group'
/var/www/miq/vmdb/app/models/miq_queue.rb:430:in `deliver'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:104:in `deliver_queue_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:137:in `deliver_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:155:in `block in do_work'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:149:in `loop'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:149:in `do_work'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:329:in `block in do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:326:in `loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:326:in `do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:153:in `run'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:127:in `start'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:22:in `start_worker'
/var/www/miq/vmdb/app/models/miq_worker.rb:402:in `block in start_runner_via_fork'
/opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork'
/opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork'
/var/www/miq/vmdb/app/models/miq_worker.rb:400:in `start_runner_via_fork'
/var/www/miq/vmdb/app/models/miq_worker.rb:390:in `start_runner'
/var/www/miq/vmdb/app/models/miq_worker.rb:441:in `start'
/var/www/miq/vmdb/app/models/miq_worker.rb:271:in `start_worker'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:74:in `start_worker_for_ems'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:52:in `block in sync_workers'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:51:in `each'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:51:in `sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:53:in `block in sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `each'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:22:in `monitor_workers'
/var/www/miq/vmdb/app/models/miq_server.rb:339:in `block in monitor'
/opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-1ee45ac9278b/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-1ee45ac9278b/lib/gems/pending/util/extensions/miq-benchmark.rb:28:in `realtime_block'
/var/www/miq/vmdb/app/models/miq_server.rb:339:in `monitor'
/var/www/miq/vmdb/app/models/miq_server.rb:380:in `block (2 levels) in monitor_loop'
/opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-1ee45ac9278b/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-1ee45ac9278b/lib/gems/pending/util/extensions/miq-benchmark.rb:35:in `realtime_block'
/var/www/miq/vmdb/app/models/miq_server.rb:380:in `block in monitor_loop'
/var/www/miq/vmdb/app/models/miq_server.rb:379:in `loop'
/var/www/miq/vmdb/app/models/miq_server.rb:379:in `monitor_loop'
/var/www/miq/vmdb/app/models/miq_server.rb:241:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:27:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:48:in `start'
/var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>'

[bug] Git import imports all available domains, but locks only one of them

When I import from git repo with two domains, both of them get imported (not nice, but acceptable when documented). But only one of them is refreshable (assigned git repository) and locked.

When refreshing a domain, it works as expected - only that one domain is refreshed. It would be great to work in the same way when importing...

Error while loading a Service Dialog

Lately I'm getting an error in Rails console while loading a Service Dialog. Not sure if it's issue in API or automate or somewhere else.

I don't know how exactly to reproduce this issue, but it happens a lot on the Dialog attached in the BZ https://bugzilla.redhat.com/show_bug.cgi?id=1518600.

Here's the error message:

F, [2017-12-08T14:47:44.897841 #27922] FATAL -- :                                                                                    
F, [2017-12-08T14:47:44.897964 #27922] FATAL -- : SystemStackError (stack level too deep):                                           
F, [2017-12-08T14:47:44.898034 #27922] FATAL -- :                                                                                    
F, [2017-12-08T14:47:44.898185 #27922] FATAL -- : /home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-automation_engine-a71f1de4e04f/lib/miq_automation_engine/engine/miq_ae_method_service.rb:2:in `const_missing'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-automation_engine-a71f1de4e04f/lib/miq_automation_engine/engine/miq_ae_method_service.rb:3:in `const_missing'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-automation_engine-a71f1de4e04f/lib/miq_automation_engine/engine/miq_ae_method_service.rb:3:in `const_missing'
...
...
...
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-automation_engine-a71f1de4e04f/lib/miq_automation_engine/engine/miq_ae_method_service.rb:3:in `const_missing'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-automation_engine-a71f1de4e04f/lib/miq_automation_engine/engine/miq_ae_method_service.rb:3:in `const_missing'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-automation_engine-a71f1de4e04f/lib/miq_automation_engine/engine/miq_ae_method_service.rb:3:in `const_missing'
activesupport (5.0.6) lib/active_support/inflector/methods.rb:270:in `const_get'
activesupport (5.0.6) lib/active_support/inflector/methods.rb:270:in `block in constantize'
activesupport (5.0.6) lib/active_support/inflector/methods.rb:266:in `each'
activesupport (5.0.6) lib/active_support/inflector/methods.rb:266:in `inject'
activesupport (5.0.6) lib/active_support/inflector/methods.rb:266:in `constantize'
activesupport (5.0.6) lib/active_support/inflector/methods.rb:311:in `safe_constantize'
activesupport (5.0.6) lib/active_support/core_ext/string/inflections.rb:77:in `safe_constantize'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-automation_engine-a71f1de4e04f/lib/miq_automation_engine/engine/miq_ae_engine/miq_ae_object.rb:551:in `convert_value_based_on_datatype'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-automation_engine-a71f1de4e04f/lib/miq_automation_engine/engine/miq_ae_engine/miq_ae_object.rb:276:in `process_args_attribute'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-automation_engine-a71f1de4e04f/lib/miq_automation_engine/engine/miq_ae_engine/miq_ae_object.rb:259:in `block in process_args_as_attributes'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-automation_engine-a71f1de4e04f/lib/miq_automation_engine/engine/miq_ae_engine/miq_ae_object.rb:259:in `each'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-automation_engine-a71f1de4e04f/lib/miq_automation_engine/engine/miq_ae_engine/miq_ae_object.rb:259:in `process_args_as_attributes'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-automation_engine-a71f1de4e04f/lib/miq_automation_engine/engine/miq_ae_engine/miq_ae_workspace_runtime.rb:140:in `instantiate'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-automation_engine-a71f1de4e04f/lib/miq_automation_engine/engine/miq_ae_engine/miq_ae_workspace_runtime.rb:49:in `instantiate'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-automation_engine-a71f1de4e04f/lib/miq_automation_engine/engine/miq_ae_engine.rb:301:in `resolve_automation_object'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-automation_engine-a71f1de4e04f/lib/miq_automation_engine/engine/miq_ae_engine.rb:95:in `deliver'
app/models/resource_action.rb:80:in `deliver_to_automate_from_dialog_field'
app/models/dynamic_dialog_field_value_processor.rb:8:in `values_from_automate'
app/models/dynamic_dialog_field_value_processor.rb:3:in `values_from_automate'
app/models/dialog_field.rb:163:in `values_from_automate'
app/models/dialog_field_sorted_item.rb:104:in `raw_values'
app/models/dialog_field_sorted_item.rb:60:in `trigger_automate_value_updates'
app/models/dialog_field_serializer.rb:24:in `serialize'
app/models/dialog_group_serializer.rb:17:in `block in serialize_dialog_fields'
activerecord (5.0.6) lib/active_record/relation/delegation.rb:38:in `map'
activerecord (5.0.6) lib/active_record/relation/delegation.rb:38:in `map'
app/models/dialog_group_serializer.rb:16:in `serialize_dialog_fields'
app/models/dialog_group_serializer.rb:9:in `serialize'
app/models/dialog_tab_serializer.rb:16:in `block in serialize_dialog_groups'
activerecord (5.0.6) lib/active_record/relation/delegation.rb:38:in `map'
activerecord (5.0.6) lib/active_record/relation/delegation.rb:38:in `map'
app/models/dialog_tab_serializer.rb:15:in `serialize_dialog_groups'
app/models/dialog_tab_serializer.rb:9:in `serialize'
app/models/dialog_serializer.rb:16:in `block in serialize_dialog_tabs'
activerecord (5.0.6) lib/active_record/relation/delegation.rb:38:in `map'
activerecord (5.0.6) lib/active_record/relation/delegation.rb:38:in `map'
app/models/dialog_serializer.rb:15:in `serialize_dialog_tabs'
app/models/dialog_serializer.rb:22:in `block in serialize_dialogs'
app/models/dialog_serializer.rb:21:in `map'
app/models/dialog_serializer.rb:21:in `serialize_dialogs'
app/models/dialog_serializer.rb:9:in `serialize'
app/models/dialog.rb:111:in `content'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-api-462b79b228b4/app/controllers/api/service_dialogs_controller.rb:20:in `fetch_service_dialogs_content'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-api-462b79b228b4/app/controllers/api/base_controller/renderer.rb:268:in `fetch_direct_virtual_attribute'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-api-462b79b228b4/app/controllers/api/base_controller/renderer.rb:256:in `block in expand_virtual_attributes'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-api-462b79b228b4/app/controllers/api/base_controller/renderer.rb:252:in `each'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-api-462b79b228b4/app/controllers/api/base_controller/renderer.rb:252:in `expand_virtual_attributes'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-api-462b79b228b4/app/controllers/api/base_controller/renderer.rb:81:in `resource_to_jbuilder'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-api-462b79b228b4/app/controllers/api/base_controller/renderer.rb:17:in `render_resource'
/home/rblanco/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-api-462b79b228b4/app/controllers/api/base_controller.rb:91:in `show'
actionpack (5.0.6) lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'
actionpack (5.0.6) lib/abstract_controller/base.rb:188:in `process_action'
actionpack (5.0.6) lib/action_controller/metal/rendering.rb:30:in `process_action'
actionpack (5.0.6) lib/abstract_controller/callbacks.rb:20:in `block in process_action'
activesupport (5.0.6) lib/active_support/callbacks.rb:126:in `call'
activesupport (5.0.6) lib/active_support/callbacks.rb:506:in `block (2 levels) in compile'
activesupport (5.0.6) lib/active_support/callbacks.rb:455:in `call'
activesupport (5.0.6) lib/active_support/callbacks.rb:101:in `__run_callbacks__'
activesupport (5.0.6) lib/active_support/callbacks.rb:750:in `_run_process_action_callbacks'
activesupport (5.0.6) lib/active_support/callbacks.rb:90:in `run_callbacks'
actionpack (5.0.6) lib/abstract_controller/callbacks.rb:19:in `process_action'
actionpack (5.0.6) lib/action_controller/metal/rescue.rb:20:in `process_action'
actionpack (5.0.6) lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
activesupport (5.0.6) lib/active_support/notifications.rb:164:in `block in instrument'
activesupport (5.0.6) lib/active_support/notifications/instrumenter.rb:21:in `instrument'
activesupport (5.0.6) lib/active_support/notifications.rb:164:in `instrument'
actionpack (5.0.6) lib/action_controller/metal/instrumentation.rb:30:in `process_action'
actionpack (5.0.6) lib/action_controller/metal/params_wrapper.rb:248:in `process_action'
activerecord (5.0.6) lib/active_record/railties/controller_runtime.rb:18:in `process_action'
actionpack (5.0.6) lib/abstract_controller/base.rb:126:in `process'
actionpack (5.0.6) lib/action_controller/metal.rb:190:in `dispatch'
actionpack (5.0.6) lib/action_controller/metal.rb:262:in `dispatch'
actionpack (5.0.6) lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
actionpack (5.0.6) lib/action_dispatch/routing/route_set.rb:32:in `serve'
actionpack (5.0.6) lib/action_dispatch/journey/router.rb:39:in `block in serve'
actionpack (5.0.6) lib/action_dispatch/journey/router.rb:26:in `each'
actionpack (5.0.6) lib/action_dispatch/journey/router.rb:26:in `serve'
actionpack (5.0.6) lib/action_dispatch/routing/route_set.rb:727:in `call'
secure_headers (3.0.3) lib/secure_headers/middleware.rb:10:in `call'
rack (2.0.3) lib/rack/etag.rb:25:in `call'
rack (2.0.3) lib/rack/conditional_get.rb:25:in `call'
rack (2.0.3) lib/rack/head.rb:12:in `call'
rack (2.0.3) lib/rack/session/abstract/id.rb:232:in `context'
rack (2.0.3) lib/rack/session/abstract/id.rb:226:in `call'
actionpack (5.0.6) lib/action_dispatch/middleware/cookies.rb:613:in `call'
actionpack (5.0.6) lib/action_dispatch/middleware/callbacks.rb:38:in `block in call'
activesupport (5.0.6) lib/active_support/callbacks.rb:97:in `__run_callbacks__'
activesupport (5.0.6) lib/active_support/callbacks.rb:750:in `_run_call_callbacks'
activesupport (5.0.6) lib/active_support/callbacks.rb:90:in `run_callbacks'
actionpack (5.0.6) lib/action_dispatch/middleware/callbacks.rb:36:in `call'
actionpack (5.0.6) lib/action_dispatch/middleware/executor.rb:12:in `call'
actionpack (5.0.6) lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
actionpack (5.0.6) lib/action_dispatch/middleware/debug_exceptions.rb:49:in `call'
actionpack (5.0.6) lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
railties (5.0.6) lib/rails/rack/logger.rb:36:in `call_app'
railties (5.0.6) lib/rails/rack/logger.rb:26:in `call'
sprockets-rails (3.2.1) lib/sprockets/rails/quiet_assets.rb:13:in `call'
actionpack (5.0.6) lib/action_dispatch/middleware/request_id.rb:24:in `call'
rack (2.0.3) lib/rack/method_override.rb:22:in `call'
rack (2.0.3) lib/rack/runtime.rb:22:in `call'
activesupport (5.0.6) lib/active_support/cache/strategy/local_cache_middleware.rb:28:in `call'
actionpack (5.0.6) lib/action_dispatch/middleware/executor.rb:12:in `call'
actionpack (5.0.6) lib/action_dispatch/middleware/static.rb:136:in `call'
rack (2.0.3) lib/rack/sendfile.rb:111:in `call'
rack-mini-profiler (0.10.7) lib/mini_profiler/profiler.rb:282:in `call'
railties (5.0.6) lib/rails/engine.rb:522:in `call'
puma (3.7.1) lib/puma/configuration.rb:232:in `call'
puma (3.7.1) lib/puma/server.rb:578:in `handle_request'
puma (3.7.1) lib/puma/server.rb:415:in `process_client'
puma (3.7.1) lib/puma/server.rb:275:in `block in run'
puma (3.7.1) lib/puma/thread_pool.rb:120:in `block in spawn_thread'

Protected dialog fields id key incorrectly contains encrypted string

Found when fixing https://bugzilla.redhat.com/show_bug.cgi?id=1499278.

The URL shows the correct id at first but then by the time the field is being handled by Automate, the id contains the encrypted value and probably shouldn't.

Instantiating [/ImaanN/Methods/testI?MiqServer::miq_server=1&ServiceTemplate::service_template=2&User::user=1&dialog_display_result=Validate Root Password Correct!&object_name=testI&password::dialog_confirm_password=v2:%7BCIHAKUtq3XtK40mtPPHp1Q%3D%3D%7D&password::dialog_root_password=v2:%7Bb7S9YKlblb%2FPcriMeL7OJw%3D%3D%7D&vmdb_object_type=service_template]
[----] I, [2017-10-09T17:09:00.913213 #19729:3ff28ed8492c]  INFO -- : Updated namespace [/ImaanN/Methods/testI?MiqServer::miq_server=1&ServiceTemplate::service_template=2&User::user=1&dialog_display_result=Validate Root Password Correct!&object_name=testI&password::dialog_confirm_password=v2:%7BCIHAKUtq3XtK40mtPPHp1Q%3D%3D%7D&password::dialog_root_password=v2:%7Bb7S9YKlblb%2FPcriMeL7OJw%3D%3D%7D&vmdb_object_type=service_template  ImaanDomain/ImaanN]
...
[----] I, [2017-10-09T17:09:01.473837 #19729:3ff2847af22c]  INFO -- : <AEMethod testi>   Attribute - dialog_confirm_password_id: v2:{CIHAKUtq3XtK40mtPPHp1Q==}
...
[----] I, [2017-10-09T17:09:01.641641 #19729:3ff2847af22c]  INFO -- : <AEMethod testi>   Attribute - dialog_root_password_id: v2:{b7S9YKlblb/PcriMeL7OJw==} 

How to handle EmbeddedAnsible roles dependencies

With Embedded I want to use a role that depends on official MIQ automation role synchrou.manageiq_automate.

Q: Should I require synchrou.manageiq_automate at specific release/commit or should I just rely on latest master?

What we currently do is we list it in meta/main.yml as

dependencies:
  - syncrou.manageiq-automate

which means latest master gets installed when user adds Embedded repository. But what can happen - in fact, it happened today :) - is that latest master can get corrupt.

Which is a big problem for Embedded, because as far as I've tested, Embedded would install requirements only at first run, when you add the repository. It's not possible to reinstall them later via CloudForms GUI, but you have to SSH to the appliance and manually handle it. So basically if Embedded fetches a corrupt role, it cannot continue to work, but you need to remove entire repo and re-add it, which is a pain.

Looking forward to some opinion on this topic!

/cc @syncrou @mkanoor

Resolve "warning: already initialized constant" message in tests

/home/travis/build/ManageIQ/manageiq-automation_engine/lib/miq_automation_engine/engine/miq_ae_method_service/miq_ae_service_model_base.rb:115: warning: already initialized constant MiqAeMethodService::MiqAeServiceManageIQ_Providers_InfraManager
/home/travis/build/ManageIQ/manageiq-automation_engine/lib/miq_automation_engine/engine/miq_ae_method_service/miq_ae_service_model_base.rb:115: warning: previous definition of MiqAeServiceManageIQ_Providers_InfraManager was here

Narrowed down the errors to test "loads all mapped models" here:

it "loads all mapped models" do

Related to the LEGACY_MODEL_NAMES mapping here:

Expose cloud_volume to automate

Looking to expose aws cloud_volume methods to automate
create/delete/attach volume and create/delete snapshot

miq_ae_service_manageiq-providers-cloud_manager-vm.rb
add expose :cloud_volumes, :association => true

miq_ae_service_cloud_volume.rb
add expose :create_volume_snapshot, :override_return => nil

Attribute value pairs message is not considered while simulation

Original BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1753523

Description of problem:
If you put 'message' in 'attribute/value pairs' field then it does not consider while simulation.

Version-Release number of selected component (if applicable):
5.11.0.25.20190913144023_5682444

How reproducible:
100%

Steps to Reproduce:

  1. Create domain, namespace, class and instance pointing to method
  2. Navigate to automate > automation > simulation page
  3. Fill values as shown in screenshot.
  4. See automation.logs

Actual results:
[----] I, [2019-09-19T03:29:39.214008 #10804:41c3a00] INFO -- : Followed Relationship [miqaedb:/test_name/test_class/test_instance#create]
[----] I, [2019-09-19T03:29:39.214742 #10804:41c3a00] INFO -- : Followed Relationship [miqaedb:/System/Request/Call_Instance#create]

It does not consider attribute 'message' in logs with test_instance.

Expected results:
[----] I, [2019-09-19T03:29:39.214008 #10804:41c3a00] INFO -- : Followed Relationship [miqaedb:/test_name/test_class/test_instance#hello]
[----] I, [2019-09-19T03:29:39.214742 #10804:41c3a00] INFO -- : Followed Relationship [miqaedb:/System/Request/Call_Instance#create]

It should consider attribute message in logs with test_instance as per BZ fixed for custom button simulation: https://bugzilla.redhat.com/show_bug.cgi?id=1651099

Additional info:

  • This behaviour is same for simulation on 5.10 and 5.11. Hence not adding it as a regression.

image

Change automate methods to communicate via REST API

Much of this was discussed at the ManageIQ design summit for Botvinnik. However, @mkanoor and I recently sat down and brainstormed our way through many of the larger problems, and we think we've come to a cleaner design.

There may be more things I'm not thinking of, but I just wanted to start getting this information out there. Let's keep this document updated as we find more issues and/or come up with more ways to solve them.


Automate currently runs methods by

  • starting a DRb server
  • spawning the ruby method wrapped in a preamble and postamble, with the preamble starting a DRb client
  • client uses service models and helper classes like $evm to communicate with the server

This causes a number of problems and headaches, which may be able to be addressed by moving to a REST API based implementation. Some notable issues:

  • DRb requires server side object management so that the server doesn't garbage collect objects in use by the client. This is a huge headache which will just go away with REST.

  • Every time we add a new object or method we must manually expose it via automate. This is prone to error. This would instead become an exercise in exposing via the REST API, which in nearly all cases should be done anyway.

  • DRb is used to handle interactions with the automate workspace, which is in the server's memory, from the client. This can be handled by dumping the workspace into some format like JSON or YAML, and passing it on STDIN into the launched process.

    The automate method preamble would take STDIN and hold it in memory, lazy loading it on first access. If the method needs to modify the workspace, then it can be modified. $evm would no longer be special and can just be a simple object defined in the preamble that has the knowledge of manipulating the workspace in this manner. $evm.log could even just write directly to the log in question or could write to STDOUT.

    In the postamble we will dump the workspace to STDOUT. In order to prevent collision with user usage of STDOUT, we can prefix the dump with some sort of delimiter (e.g. "=" * 80). STDERR can be used to transfer any exception, or perhaps that becomes part of the workspace that's dumped into STDOUT. Again, a delimiter might be necessary.

    The server, when it detects that the child process has completed, will read from the STDOUT of the child process, find the last delimiter, and process the workspace back into memory, if changed.

    There may be optimizations here to prevent unnecessarily dumping and loading the workspace.

  • DRb is also used for database interaction. The database interaction that is currently done via service models, can be done completely through the REST API. A Ruby gem needs to be created from the cfme_client code, which additionally has more of an OO design. (I'll call it manageiq_client for reference). Once that's available, then the $evm.vmdb can just be a front for accessing those objects from the manageiq_client gem. We could even deprecate $evm.vmdb in favor of having the automate methods use the normal manageiq_client gem directly.

    For example, currently $evm.vmdb["vm", 21] returns a MiqAeServiceVmVmware instance for VM 21. If the REST API gem had it's own OO based types, I could see $evm.vmdb["vm", 21] instead calling the manageiq_client gem, ultimately returning a ManageiqClient::Vm instance for VM 21

    With this in place, I think the entirety of the service models just goes away

  • DRb is also used for built-in methods (like send_email). This can be handled by exposing them from the REST API, and passing the dumped workspace from the method as a parameter to the REST API. These may need some sort of system-only access rules, but then again, maybe not.

Other thoughts:

  • With database interaction via the REST API, and the workspace passed via STDIN/STDOUT in a common format like JSON or YAML, then the support for other languages opens up. These formats are widely supported, so they could be passed to any kind of method whether it be Ruby, Python, Shell, etc. All that would be needed from the server is possibly some preamble/postamble code in the native language to simplify setting up helper objects and catching errors.

TODO

  • Other parts of $evm object that we haven't even looked at like get/set_state_var, instance_*, etc
  • What user would automate methods run as if they are accessing the REST API. For the system things like the built-in methods, it's some system account, but what about other methods? Can they just be system too?

cc @gmcculloug @mkanoor @tinaafitz @chessbyte
cc @abellotti (re: REST API stuff and cfme_client split out into manageiq_client gem)
cc @kbrock (re: system-only access rules)


This issue was moved to this repository from ManageIQ/manageiq#2215, originally opened by @Fryguy

Incorrect error when deleting a VMware ESXi host.

When deleting a VMware ESXi host this unexpected error appears in the evm.log

INFO -- : MIQ(MiqAeEngine.deliver) Delivering {:event_id=>964368, :event_stream_id=>964368, :event_type=>"HostDisconnectedEvent", "ExtManagementSystem::ems"=>2, :ems_id=>2, "Host::host"=>23, :host_id=>23} for object [EmsEvent.964368] with state [] to Automate
ERROR -- : MIQ(MiqAeEngine.deliver) Error delivering {:event_id=>964368, :event_stream_id=>964368, :event_type=>"HostDisconnectedEvent", "ExtManagementSystem::ems"=>2, :ems_id=>2, "Host::host"=>23, :host_id=>23, "User::user"=>1, "EventStream::event_stream"=>964368} for object [EmsEvent.964368] with state [] to Automate: Service Model not found

Automate export contains empty "options" hash data which breaks backwards compatibility

Following the step in Creating a Pull Request for enhancements to ManageIQ Automate Domain lead to a large number of files being updated to include an empty options hash property.

This was initially address in ManageIQ/manageiq#17080 but no longer effects the export files.

diff --git a/content/automate/ManageIQ/AutomationManagement/AnsibleTower/Operations/Methods.class/__methods__/available_credentials.yaml b/content/automate/ManageIQ/AutomationManagement/AnsibleTower/Operations/Methods.class/__methods__/available_credentials.yaml
index 28ac98c9..63ba2474 100644
--- a/content/automate/ManageIQ/AutomationManagement/AnsibleTower/Operations/Methods.class/__methods__/available_credentials.yaml
+++ b/content/automate/ManageIQ/AutomationManagement/AnsibleTower/Operations/Methods.class/__methods__/available_credentials.yaml
@@ -9,6 +9,7 @@ object:
     scope: instance
     language: ruby
     location: inline
+    options: {}
   inputs:
   - field:
       aetype: 

Also, the steps examples should be updated to match reflect the Automate model in the https://github.com/ManageIQ/manageiq-content repo.

Allow to emit notifications for selected users from automate

If the audience of the notification is set to user, it is always being sent to the @workspace.ae_user and there's no possibility to notify e.g. the admin. A possible workaround would be to send the notification to the group of the admin user, however, this is not feasible as we're bound to notification types.

@pkomanek @d-m-u can you please take a look?

[RFE] Allow max_retries override at instance level

When one creates a state machine, the maximum number of retries for a state is set at the class level. However, a use case of state machines is to create generic state machines where the state name is 'StateX' and URI is specified only at the instance level, not the class. In this case, to stay generic, one is tempted to set the maximum number of retries to a very high number for each state, to cover whatever situation at the instance level.

Allowing to override the maximum number of retries at the instance level would help creating generic state machine classes (scaffolds) with no maximum retries set, and then let the instance specify the maximum retries based on the actual URI.

Datastore import - name source

I've just noticed that we are using the file name as a domain name inside our automate datastore import. For the rest of the objects we are looking into the YAML file and taking the value from the name attribute. I am not sure which way is better, but I think we should be consistent.

[Automate Engine] Looking up objects in the database is expensive

When executing an automation request there are a TON of database queries, and most of them are duplicated. Additionally, since the content of the automate domain doesn't change frequently during execution, we should be able to do some simple caching to eliminate a lot of the performance penalty.

Some ideas...

  • wrapping an automation request in ActiveRecord::QueryCache.cache
  • general caching of the entire domain structure, and busting the cache when an domain is updated and saved.
  • #409
  • optimizing the most common lookups

I'm sure there are other ideas, but it feels like there are likely some really quick wins in here. cc @tinaafitz

[RFE] Multi-level dependency resolving with embedded method

Currently, adding an embedded method doesn't resolve its dependencies. For example, one needs to rename a virtual machine in a method. The implementation is different for RHV and VMware, so it would be easier to have a facade method that calls the actual method for the provider.

Say that the method that needs to rename the VM is /Stuff/MyProcess/RenameVM. To be able to call a method from the utility class, it embeds it, say /Stuff/VM/Common/Rename. The actual implementation is provided by /Stuff/VM/VMware/Rename and /Stuff/VM/RHV. Naively, one would simply embed these two methods in /Stuff/VM/Common/Rename. But recursive dependency is not implemented and the methods will have to be embedded in /Stuff/MyProcess/RenameVM.

This RFE asks for recursive dependency implementation, so that embedding methods is kept simple from the user stand point.

Associated RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1609924

Cleanup the README

The README is a holdover from a previous doc, so it needs some cleanup. I'll go through and do the basics as part of adding the badges, but this will need some general updates and markdown styling as well. We may also want to rip it out of the README and perhaps have a Wiki for this.

cc @mkanoor @gmcculloug @tinaafitz

Add option for quota to be applied on group but enforced at user level

Related BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1653492

Business use-case requires that we efficiently apply a quota at the group level, which it is today, however we need this quota to map to each user.

For example, group1, has a quota of: 10 vCPU, 5GB RAM, and 1T Disk

Today: if group1 has 10 users, that quota is shared between all 10 users, so if user1 uses 8 vCPU and 4GB of RAM, there is little space left.

What we need: we need each user in group1 to have 10 vCPU, 5GB of RAM, and 1T quota associated with their user account. Additionally, it would be great to allocate instances/VMs to that user as well.

This would need to be an option to enable/disable, or even combine with the current quota implementation.

Bug with handling of Arrays in Automate

There's an issue when passing arrays of strings with commas between automate methods (indcluding output from multiselect dialog).

Current Behavior:

Saving ["test1,test2", "test3,test4"] (2 elements) under $evm (or selecting multiple strings with commas in a dialog) and then reading the value in another/different automate method returns ["\"test1", "test2\"", "\"test3", "test4\""] (4 elements)

Expected Behavior:

Saving an array of strings in one method should produce the exact same array when reading the value in another method.

Steps To Reproduce:

  1. Create two automate methods that are run one after another.
  2. Save the above array under $evm in the first method.
    $evm.root['test'] = ["test1,test2", "test3,test4"]
  3. Read the value in the second method.
    $evm.log('info', "Array: #{$evm.root['test']}")
  4. See a different array returned than was saved.

Environment:

ManageIQ Lasker-1

Anything else:

I suspect the split method here:

return value.gsub(/[\[\]]/, '').strip.split(/\s*,\s*/) if datatype == 'array' && value.class == String

splits the string (array) on every comma even if they are a part of a string and not a delimiter between elements.

Using eval() would solve this problem, such as many others (nested arrays or other comma issues). Though I must admit I don't know how the values are handled on the other end to judge if this would be a safe use of the eval method.

Slow tests: miq_ae_dialog_spec: "properly instantiates dialogs" instantiates 1000s of rows

This single example takes ~21 seconds on travis and creates loads of DB records.

https://github.com/ManageIQ/manageiq/blob/8d6d23639d12105bb6a0a69e4e79b879e3c9830f/vmdb/spec/lib/miq_automation_engine/miq_ae_dialog_spec.rb#L259

[----] I, [2014-10-02T15:49:47.315131 #30878:3ff2d185e6e4]  INFO -- : ZZZ  Starting: ./spec/lib/miq_automation_engine/miq_ae_dialog_spec.rb:259
[----] I, [2014-10-02T15:49:47.315178 #30878:3ff2d185e6e4]  INFO -- : ZZZZ From Example group: ./spec/lib/miq_automation_engine/miq_ae_dialog_spec.rb:5
[----] D, [2014-10-02T15:50:06.992935 #30878:3ff2d185e6e4] DEBUG -- :   MiqAeClass Inst (1.7ms - 142rows)
[----] D, [2014-10-02T15:50:07.010563 #30878:3ff2d185e6e4] DEBUG -- :   MiqAeField Inst (4.8ms - 424rows)
[----] D, [2014-10-02T15:50:07.029055 #30878:3ff2d185e6e4] DEBUG -- :   MiqAeInstance Inst (7.3ms - 682rows)
[----] D, [2014-10-02T15:50:07.064661 #30878:3ff2d185e6e4] DEBUG -- :   MiqAeValue Inst (10.6ms - 946rows)
[----] D, [2014-10-02T15:50:07.087512 #30878:3ff2d185e6e4] DEBUG -- :   MiqAeField Inst (3.8ms - 342rows)
[----] D, [2014-10-02T15:50:09.762381 #30878:3ff2d185e6e4] DEBUG -- :   MiqAeClass Inst (1.5ms - 142rows)
[----] D, [2014-10-02T15:50:09.779401 #30878:3ff2d185e6e4] DEBUG -- :   MiqAeField Inst (4.7ms - 424rows)
[----] D, [2014-10-02T15:50:09.797099 #30878:3ff2d185e6e4] DEBUG -- :   MiqAeInstance Inst (7.2ms - 682rows)
[----] D, [2014-10-02T15:50:09.831310 #30878:3ff2d185e6e4] DEBUG -- :   MiqAeValue Inst (10.0ms - 946rows)
[----] D, [2014-10-02T15:50:10.403363 #30878:3ff2d185e6e4] DEBUG -- :   MiqAeField Inst (554.4ms - 342rows)
[----] I, [2014-10-02T15:50:10.920074 #30878:3ff2d185e6e4]  INFO -- : ZZZ  Completed: ./spec/lib/miq_automation_engine/miq_ae_dialog_spec.rb:259

This issue was moved to this repository from ManageIQ/manageiq#760, originally opened by @jrafanie

introduce internal automate method

Currently every method in automate is configured to run in Drb and considered a user provided script. However manageiq is shipped with many automate methods which are not really user provided. We should introduce internal automate methods that run default implementations internally but allow users to override with an external script.

Custom message attribute is not considering with instance in logs in automation.log

Not getting custom message in logger file in automation.log
Below are the steps to reproduce issue.

  • setup:
  1. Create domain, namespace, class and instance pointing to method
  • testSteps:
  1. Navigate to automate > automation > simulation page
  2. Fill values for attribute/value pairs of namespace, class, instance and add message attribute with any value and click on submit.
  3. See automation.log
  • expectedResults:
  1. Custom message attribute should be considered with instance in logs

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

This repository currently has no open or pending branches.

Detected dependencies

github-actions
.github/workflows/ci.yaml
  • actions/checkout v4
  • ruby/setup-ruby v1
  • paambaati/codeclimate-action v6
  • manageiq/postgresql 13

  • Check this box to trigger a request for Renovate to run again on this repository

Embedded Ansible - Add support for ansible.cfg files in user repositories

Original BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1737149

Description of problem:
If a user adds a git repository to be used with embedded ansible and it contains an ansible.cfg file we should use that file when running playbooks.

Version-Release number of selected component (if applicable): 5.11.0.17

Steps to Reproduce:

  1. Add a repo with an ansible.cfg file that overrides some settings (like roles_path)
  2. Run a playbook

Actual results:
Config settings from the repo file are not used.

Expected results:
Repo file config is used.

Additional info:
This is a core ansible feature. The priority for finding config files to use when running a playbook is described here https://docs.ansible.com/ansible/latest/reference_appendices/config.html#the-configuration-file

As a solution for this, we could ensure that our working directory is the one where the ansible.cfg exists when trying to run a playbook.

A word of warning for this though, we install the plugin-provided roles to a custom directory then set roles_path in /root/.ansible.cfg on the appliance.

If we put a user's ansible.cfg in front of ours in that priority list then those roles will be inaccessible for that playbook run. This behavior might be worth documenting.

This issue was originally raised in a comment here ManageIQ/manageiq#19079 (comment)

Vmware "memory_reserve_expand" param cannot be set as true

Original BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1811925

Would like to provision a VM through the Vmware Provider with the " Reserve all guest memory (All locked) " box checked inside VMware. NOTE: "memory_reserve_expand" set to true is not working.

Steps to Reproduce:

  1. Create a custom domain having "memory_reserve_expand" set to true
  2. Provision a VM with Vmware provider
  3. Check if the " Reserve all guest memory (All locked) " box is checked inside Vmware.

See BZ for more details.

Automate expression method broken in jansa and kasparov

Hi,

I was checking jansa with expression methods as described in the Cloudforms / ManageIQ book [1] with user input and couldn't get it to work. It looks like variable resolution is no longer working for expression methods. Or maybe the syntax changed.

Affected versions (or what I tested):
jansa-2.20201027185742_b8d5deb
jansa-3.20210118183104_77bdc30
kasparov-1.20210203001902_15acbea

Steps to reproduce:
Follow the chapter "Expression methods" in the addendum [2]

Expected output:
Expression Method should deliver a result based on variables.

Actual result:
<None>

Additional info:
It is pretty interesting to compare the logging output in automation.log in ivanchuk and in jansa of an expression method:
Ivanchuck:
INFO -- : Invoking [expression] method [/Expression/General/my_expression_check] with inputs [{"arg1"=>"value1", "arg2"=>"foo", "arg3"=>"bar"}]

Jansa/Kasparov:
INFO -- : Invoking [expression] method [/Expression/General/my_expression_check] with inputs [{"arg1"=>"${/#dialog_my_field1}", "arg2"=>"${/#dialog_my_field2}", "arg3"=>"${/#dialog_my_field3}"}]
ERROR -- : Expression method ends

If that's important I tested expression method with tags.

Kind Regards,
phospi

Links

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.