manageiq/manageiq-providers-google

ManageIQ plugin for the Google GCE provider.

ManageIQ::Providers::Google

ManageIQ plugin for the Google provider.

Development

See the section on plugins in the ManageIQ Developer Setup

For quick local setup run bin/setup, which will clone the core ManageIQ repository under the spec directory and setup necessary config files. If you have already cloned it, you can run bin/update to bring the core ManageIQ code up to date.

License

The gem is available as open source under the terms of the Apache License 2.0.

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

manageiq-providers-google's Issues

DDF: Missing Validate button for Service Account JSON

Before and after the provider form switch:

We've lost the ability (or indeed requirement) to Validate after changing Service Account JSON.

Requested entity was not found when adding the provider

Good morning, I'm having trouble adding Google to the ManageIQ provider, and all the tutorials I'm looking for, I can't do it anyway.
Follow error logs.
--- Credential validation was not successful: Requested entity was not found.

[RFE] Support Shared VPCs

Currently the google provider refresh does not collect cloud networks / subnets from Shared VPCs (https://cloud.google.com/vpc/docs/shared-vpc) because these come from other projects.

We have to first query for the shared VPC host project (if it exists) with https://cloud.google.com/compute/docs/reference/rest/v1/projects/getXpnHost

Then we can list the networks / subnets on that project that are shared.

NOTE fog-google doesn't support these APIs so we will have to use https://github.com/googleapis/google-api-ruby-client

Upgrade to fog-google version 0.6.0

Upgrade the manageiq-providers-google to version 0.6.0 1.0 of the fog-google gem.
This will expose incompatibilities here and where we read the metric data returned in the response.

Helpful links:
https://cloud.google.com/monitoring/api/v3/migration#start-migration
https://cloud.google.com/monitoring/api/v3/migration#timeserieslist

Update - 3/9/18 - the fog-google-gem v1.0 has been released so we should upgrade to that. The above links may still be helpful, but I understand other areas of the provider will likely break as well.

Travis failing with bad authentication option

Recently Travis started failing with google auth error and unknown VCR request.
I found suspicious change in the fog-google gem: fog/fog-google#442 but probably it's only reveals bug in our own code.
There was also slight changes in our code recently, which also touches involved code files, but most probably they're unrelated.

I'm running test with bundle exec rspec spec/models/manageiq/providers/google/cloud_manager/refresher_spec.rb:57 command (code).

What I see, is that on the second iteration we has 2 authentication records (first is our predefined, second - gathered with refresh).

Predefined record has authtype attribute with value default.
That attribute is nil in the second (gathered) record.

As we're not passing auth_type option here, system tries to find default one.

Problem is that even if second record has nil in its authtype field, system compares not it, but additional method's result, which is equals to the same default value we have.

So it takes first found record, but it is not that record we need. As a result, google auth fails, and new gem's changes trying to find something else, which leads to unhanded VCR requests.

GCE with shared subnetwork gives undefined method `id' for nil:NilClass error

I'm facing an error when I add my GCE provider. Validation looks good, I click add, this also looks good, but shows nothing unless I "Refresh relationship and Power States". Then, almost instantly I see this error at UI:

Last Refresh Status Error - 5 Minutes Ago undefined method id' for nil:NilClass`

I suspect that this happening because my GCE has network and its subnetwork coming from shared VCP.
In evm.log I noticed this:

ERROR -- evm: MIQ(ManageIQ::Providers::Google::CloudManager::Refresher#refresh) EMS: [myproject], id: [34] Refresh failed
ERROR -- evm: [NoMethodError]: undefined method id' for nil:NilClass Method:[block (2 levels) in <class:LogProxy>] ERROR -- evm: /opt/manageiq/manageiq-gemset/bundler/gems/manageiq-providers-google-b49174958574/app/models/manageiq/providers/google/inventory/parser.rb:448:incloud_subnet_network_port'
...
...
ERROR -- evm: /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:67:in `refresh'

All metrics shows 0:

Same thing for a GCP project without shared network works just fine.

Google inventory parser may not set cloud volume type

Looking at the code, it looks like the CloudManager::RefreshParser is not setting the type in parse_volumes.

The other thing to point out is that Google doesn't have a CloudManager::CloudVolume class, so it's possible that not setting the type is fine.

I think it's probably better to:

1. Specifically subclass ::CloudVolume to ::ManageIQ::Providers::Google::CloudManager::CloudVolume, and
2. Update the refresh_parser to set the type to this new class.

/cc @Fryguy thoughts on this?

Allow Enable/Disable external access when provisioning a VM

Some project policies disallow external access so provisioning an instance should allow for an option to select a subnet and enable/disable public IP

• ManageIQ/manageiq-ui-classic#8388
• ManageIQ/manageiq#22020
• #232

Only save deprecated images attached to active instances

GCE keeps all images around seemingly indefinitely leading to many thousands of images while only about 50 are active.

We can cut down on the extras by only saving images that are attached to active instances.

Related to ManageIQ/manageiq#21455

Google::Apis.logger defaults to Rails.logger

The Google::Apis.logger defaults to the rails logger, filling the development.log/production.log with HTTP traffic:

[----] D, [2020-06-02T14:18:50.222949 #551545:9de4] DEBUG -- : Sending HTTP get https://www.googleapis.com/compute/v1/projects/oceanic-guard-191815/global/httpHealthChecks/a91a478a1c6af430d8ff6841bcf23eeb?
[----] D, [2020-06-02T14:18:50.391402 #551545:9de4] DEBUG -- : 200
[----] D, [2020-06-02T14:18:50.392214 #551545:9de4] DEBUG -- : #<HTTP::Message:0x0000557e37eb3d58 @http_header=#<HTTP::Message::Headers:0x0000557e37eb3d08 @http_version="1.1", @body_size=0, @chunked=false, @request_method="GET", @request_uri=#<Addressable::URI:0x211d8 URI:https://www.googleapis.com/compute/v1/projects/oceanic-guard-191815/global/httpHealthChecks/a91a478a1c6af430d8ff6841bcf23eeb?>, @request_query=nil, @request_absolute_uri=nil, @status_code=200, @reason_phrase="OK", @body_type=nil, @body_charset=nil, @body_date=nil, @body_encoding=#<Encoding:UTF-8>, @is_request=false, @header_item=[["ETag", "M66ZfrouJbC9uer_pDhTSd4y1Ps=/ZhsXVMdCOh_hxqpMSgMY_ALP6cA="], ["Content-Type", "application/json; charset=UTF-8"], ["Vary", "Origin"], ["Vary", "X-Origin"], ["Vary", "Referer"], ["Content-Encoding", "gzip"], ["Date", "Tue, 02 Jun 2020 18:18:50 GMT"], ["Server", "ESF"], ["Cache-Control", "private"], ["X-XSS-Protection", "0"], ["X-Frame-Options", "SAMEORIGIN"], ["X-Content-Type-Options", "nosniff"], ["Alt-Svc", "h3-27=\":443\"; ma=2592000,h3-25=\":443\"; ma=2592000,h3-T050=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q049=\":443\"; ma=2592000,h3-Q048=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\""], ["Transfer-Encoding", "chunked"]], @dumped=false>, @peer_cert=#<OpenSSL::X509::Certificate: subject=#<OpenSSL::X509::Name CN=upload.video.google.com,O=Google LLC,L=Mountain View,ST=California,C=US>, issuer=#<OpenSSL::X509::Name CN=GTS CA 1O1,O=Google Trust Services,C=US>, serial=#<OpenSSL::BN:0x0000557e37eb85d8>, not_before=2020-05-20 12:07:40 UTC, not_after=2020-08-12 12:07:40 UTC>, @http_body=#<HTTP::Message::Body:0x0000557e37eb3c40 @body="{\n  \"id\": \"733395921692024050\",\n  \"creationTimestamp\": \"2020-03-30T05:12:13.869-07:00\",\n  \"name\": \"a91a478a1c6af430d8ff6841bcf23eeb\",\n  \"description\": \"{\\\"kubernetes.io/service-name\\\":\\\"a91a478a1c6af430d8ff6841bcf23eeb\\\"}\",\n  \"host\": \"\",\n  \"requestPath\": \"/healthz\",\n  \"port\": 32161,\n  \"checkIntervalSec\": 8,\n  \"timeoutSec\": 1,\n  \"unhealthyThreshold\": 3,\n  \"healthyThreshold\": 1,\n  \"selfLink\": \"https://www.googleapis.com/compute/v1/projects/oceanic-guard-191815/global/httpHealthChecks/a91a478a1c6af430d8ff6841bcf23eeb\",\n  \"kind\": \"compute#httpHealthCheck\"\n}\n", @size=0, @positions=nil, @chunk_size=nil>, @previous=nil>
[----] D, [2020-06-02T14:18:50.394333 #551545:9de4] DEBUG -- : Success - #<Google::Apis::ComputeV1::HttpHealthCheck:0x0000557e37ed3b30
 @check_interval_sec=8,
 @creation_timestamp="2020-03-30T05:12:13.869-07:00",
 @description=
  "{\"kubernetes.io/service-name\":\"a91a478a1c6af430d8ff6841bcf23eeb\"}",
 @healthy_threshold=1,
 @host="",
 @id=733395921692024050,
 @kind="compute#httpHealthCheck",
 @name="a91a478a1c6af430d8ff6841bcf23eeb",
 @port=32161,
 @request_path="/healthz",
 @self_link=
  "https://www.googleapis.com/compute/v1/projects/oceanic-guard-191815/global/httpHealthChecks/a91a478a1c6af430d8ff6841bcf23eeb",
 @timeout_sec=1,
 @unhealthy_threshold=3>

Script creation of the spec test environment to automate recording new VCRs

Some providers have scripts which automate creation of a new spec test environment making it simple to re-record vcr_cassettes when necessary.

Pointing this script to an empty GCE project with proper credentials should create all objects used by the spec tests, these specs should only check for inventory by name not ems_ref (since this can change), ideally re-recording twice shouldn't change anything in the specs.

fine-3:/evm.log showing [fog][WARNING] Unrecognized arguments: google_client_options

Hi

We have a fine-3 miq instance with google providers added and it is showing lots of warning message on "google_client_options"

• miq info from ap command

<snipped>
ManageIQ Version:        fine-3
<snipped>

• lots of warning message like this

[root@miq01 log]# grep google_client_options evm.log |wc
   4337   17348  264557
[root@miq01 log]# wc evm.log
  23960  174790 2601624 evm.log
[root@miq01 log]#

• References:
  • http://talk.manageiq.org/t/unrecognized-arguments-google-client-options/2607

GCE default cloud network not discovered

For Google Cloud Engine, each new project starts with a default network. The default network is an auto mode VPC network with pre-populated firewall rules.

There is a default VPC network on GCE:

But manageiq could not find it:

As a result, you will not be able to provision instance:

Because "'Environment/Cloud Network' is required"

[WIP] Replace `fog-google` with native calls to `google-api-ruby-client`

[RFE] Support Google Kubernetes Engine (GKE)

Add support for Google Kubernetes Engine (GKE) to ManageIQ

• Provider (#195)
• Icon (ManageIQ/manageiq-decorators#51)
• Capabilities Matrix (ManageIQ/manageiq-documentation#1588)

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

This repository currently has no open or pending branches.

Detected dependencies

github-actions

.github/workflows/ci.yaml

• actions/checkout v4
• ruby/setup-ruby v1
• paambaati/codeclimate-action v6
• manageiq/postgresql 13

---

• Check this box to trigger a request for Renovate to run again on this repository