Dashboard - 401 Unauthorized

[aristotelesneto]

Dashboard - 401 Unauthorized

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/77160618.

[shalako]

Hello,

Can you confirm that your cf-release includes the cc-service-dashboards client, as described in http://docs.cloudfoundry.org/services/dashboard-sso.html?

By default, the mysql broker is configured to declare in it's catalog endpoint that it needs a client p-mysql created in the UAA. Cloud controller uses the cc-service-dashboards client to create clients for service dashboards. You can verify that the client for the MySQL dashboard is created and has the permissions it needs using the UAA CLI.

$ gem install cf-uaac
$ uaac token client get admin
secret: <secret for admin client from cf-release manifest>
$ uaac client get p-mysql

The p-mysql client should have the openid and cloud_controller_service_permissions.read scopes, as described in http://docs.cloudfoundry.org/services/dashboard-sso.html.

Let us know if this helps.

[aristotelesneto]

Thanks Shannon,

Yes, it does.

I actually have clients 'cc_service_broker_client' (pre v177) as well as 'cc-service-dashboards' (post v177).

  cc_service_broker_client
    scope: cloud_controller.read cloud_controller.write cloud_controller_service_permissions.read openid
    resource_ids: none
    authorized_grant_types: client_credentials
    authorities: clients.admin clients.write clients.read
  cc-service-dashboards
    scope: cloud_controller.read cloud_controller.write cloud_controller_service_permissions.read openid
    resource_ids: none
    authorized_grant_types: client_credentials
    authorities: clients.admin clients.write clients.read

I've used a different client name other than p-mysql but the properties are the same:

  dashboard-sso-mariadb
    scope: cloud_controller.read cloud_controller.write cloud_controller_service_permissions.read openid
    resource_ids: none
    authorized_grant_types: authorization_code refresh_token
    redirect_uri: https://mariadb.10.0.0.1.xip.io/
    authorities: uaa.none

As you can see above I also tried manually granting them cloud_controller_read/write, to no avail.

I strace'd the broker to find out which bearer token it was sending, so I could run a manual query against the API, and using the same token I also get an "Inavlid Token" message. At the moment I'm trying to find out where the token is coming from.

I take it this isn't something you guys have come across in your deployments?

[shalako]

This hasn't been an issue for us, but then we haven't tried configuring a different client name in the cf-mysql-release manifest. It's possible that the dashboard running on the broker expects the client is always p-mysql. I'll move the story created for this issue into our backlog and our engineering team can take a closer look.

[drich10]

Hello @wdneto,

Unfortunately we haven't seen this exact issue before, but I went ahead and changed only the name of the dashboard client and it seems like that in it of itself should not be an issue. As long as you changed the name in the deployment manifest and registered the broker after deploying it should be ok. If you did go through these steps we will probably need more information to help you debug this issue. Would you be willing be show us your deployment manifest (redacting any sensitive information of course)?

[aristotelesneto]

I did go through those steps.

From the broker logs, as well as the manual test I did using the Authorization token the broker is trying to use, it seems more like the broker code is using the wrong token.

I'm happy to show the entire manifest, although all we use from it is the broker part - the manifest we have includes jobs from two releases as we needed to include mariadb, before you guys changed over.

Do you need both the cf-release as well as the mariadb deployment manifest? Here is a mysql lab deployment manifest: https://gist.github.com/wdneto/90a749ea37153da7de1e

[ruthie]

@aristotelesneto

The link you posted seems to be broken. Can you try again?

@ruthie

[aristotelesneto]

Oops, I've just renamed my github account, as I'm moving on from Web Drive (the wd part of the username), but try: https://gist.github.com/aristotelesneto/90a749ea37153da7de1e

[ruthie]

Hi @aristotelesneto,

Two things to look at:

1. We noticed that the external host for your mysql is mariadb.paas.10.0.0.1.xip.io, which is different from your redirect url of mariadb.10.0.0.1.xip.io. You might not be allowed to redirect there.
2. Produce your problem again and grab the uaa.log from your UAA in cf-release. If you have the problem above, you'll see something like "invalid redirect url." Otherwise, you might see something else interesting.

@ruthie @pivotal-devin-fallak

[aristotelesneto]

Hi @ruthie

Apologies, it looks like issue 1 you refer to was caused by me manually redacting stuff - they are both the same (mariadb.paas.*) :(

Looking at the uaa.log, everything seems to complete successfully.
https://gist.github.com/aristotelesneto/098c470187008e93f915

I honestly can't see anything wrong on the uaa/login logs. Its almost like the user I'm trying to access doesn't actually have permission to manage the database (admin user, however).

Could it be related to: cloudfoundry/uaa/issues/97?

[dsabeti]

@aristotelesneto
I think the issue is that the user doesn't have permissions to view that service instance.

You'll notice that the validation that failed in the broker was :ensure_can_manage_service_instance; more importantly, it seems that the token is passing the :ensure_all_necessary_scopes_are_approved. The failing validation simply makes a request to CC to determine whether or not the given user has access to the service instance.

I saw that in your first post, you tried to view the permissions yourself with gcf. When you did the check, were you logged in as the given user, or were you an admin? (gcf curl is typically used by an admin rather than a regular user) If so, then the result is misleading because the CC actually uses the token in the request to determine permissions. In this case, try repeating the experiment, expect use a plain old curl with the token in the auth header (curl -H "Authorization: Bearer <UAA TOKEN>" ...) and take a look at the result.

[aristotelesneto]

Thanks @dsabeti

I tried using a manual telnet session previously and all I got was a 401 message. I'll give it another shot shortly and let you know.

For what its worth, I'm logged in to the dashboard as well as the cli as an admin user - created using the uaa.scim.users property in the deployment manifest.

Is there a better way to debug the dashboard app itself?

[dsabeti]

Did you create the service instance as the admin user?

[aristotelesneto]

Yup. I then tried creating it as a regular user, only to experience the same problem.

[aristotelesneto]

I just tried it again:

Broker log file:
https://gist.githubusercontent.com/aristotelesneto/098c470187008e93f915/raw/8693c73c693a67825f43f9d76f783db9fc4db9c7/broker-20140923.log

gcf curl attempt, logged in as the same user:

~# gcf curl /v2/service_instances/67a1702e-131a-434f-8aea-64e686ca4113/permissions
{
   "manage": true
}

Manual curl attempt using token displayed in broker log (tokens snipped for brevity):

~# curl -k -H "Authorization: bearer eyJhbGciOiJSUzI1NiJ9.eyJqdGkiOiI5OGU4YWZjNy02ZDE4LT<snip>m2gTYQJcOmTO38-CMBlDV0REBXbjYFHCoLZdl_5dcRcHI7xsNg" https://api.10.0.0.1.xip.io/v2/service_instances/67a1702e-131a-434f-8aea-64e686ca4113/permissions
{"code":1000,"description":"Invalid Auth Token","error_code":"CF-InvalidAuthToken"}

~# curl -k -H "Authorization: bearer eyJhbGciOiJSUzI1NiJ9.eyJqdGkiOiJjZWM2NjhmYi0xYmFlLTR<snip>oC_JznEpZUC6mZl65IkJKOPtBV-Cu0iT_HxZKHDbk5B5og" https://api.10.0.0.1.xip.io/v2/service_instances/67a1702e-131a-434f-8aea-64e686ca4113/permissions
{"code":1000,"description":"Invalid Auth Token","error_code":"CF-InvalidAuthToken"}

~# curl -k -H "Authorization: bearer eyJhbGciOiJSUzI1NiJ9.eyJqdGkiOiI5OGU4YWZjNy02ZDE4LTRm<snip>Gs7B4Vm2gTYQJcOmTO38-CMBlDV0REBXbjYFHCoLZdl_5dcRcHI7xsNg" https://api.10.0.0.1.xip.io/v2/service_instances/67a1702e-131a-434f-8aea-64e686ca4113/permissions
{"code":1000,"description":"Invalid Auth Token","error_code":"CF-InvalidAuthToken"}

Am I doing this right?

[dsabeti]

The request looks correct as long as the given token isn't expired. If you pulled it from the broker log, it probably needs to be refreshed. To get the updated token, log in again with the CLI, then run cat ~/.cf/config.json and pull the token from the AccessToken field.

[aristotelesneto]

Using the CLI access token, I receive the correct json back - {"manage":true}.

At the time I retrieved the token I was still browsing around the console with a valid session. Or do you mean the token the broker uses is invalidated after each request?

How else could I verify that the token the broker is trying to use is valid?

[aristotelesneto]

Using the access_token value from the browser cookie prior to attempting to 'manage' the database instance, I also receive manage: true.

Immediatelly following me attempting to manage via the dashboard sso, I start getting invalid token.

[dsabeti]

Did the value of the access token change?

[aristotelesneto]

The one shown to be used in the broker log is the same as I tried manually. Refreshing the console session did not require me to reauthenticate, and the cookie value remains the same.

Refreshing the dashboard sso still provides me with a "not authorised" message, however - judging by the broker logs, its not attempting to re-authenticate.

[dsabeti]

Ok, let's try an experiment. We're going to fetch a token in a very different way, using the uaac command line tool. This process will simulate the method that the broker uses to fetch a token on behalf of the user.

First, in order to run the simulation, we need to temporarily hack the dashboard client to redirect to your localhost so that the uaac running on the local host can intercept any redirects. You do this by runnining the following:

uaac token client get admin # log in as the admin
uaac client update <dashboard client name> --redirect_uri http://localhost:5000

Next, you need to initiate the auth code authorization flow:

uaac token authcode get -c <dashboard client name> -s <dashboard client secret> --port 5000

Your browser should open and take you to an unstyled web page with a login form. Log in as the user who created the service instance. After seeing a success message in the browser, close the window.

At this point, your uaac client has 'logged in' as the user using the dashboard client -- precisely the set of steps that the broker uses. Now we want to use your new token to access the permissions endpoint on the cloud controller.

Fetch your token:

uaa context
=> 
[0]*[https://uaa.10.244.0.34.xip.io]
  skip_ssl_validation: true

  [0]*[user]
      client_id: <client name>
      access_token: <some access token. Take this>
      token_type: bearer
      expires_in: 43199
      scope: cloud_controller_service_permissions.read openid
      jti: 5c69a4ff-b2cd-4e6b-97fb-272ba3e4d4e3
      user_id: a2913566-d24a-41e8-ac4b-93ade19866fb
      refresh_token: <some refresh token>

Grab the token from the access_token field, and the use that as the header in your curl request:

curl -H "Authorization: bearer <token>" api....xip.io/v2/service_instances/guid/permissions

[shalako]

Keep in mind that when you're getting the token for admin in the first step, you're getting the token for the admin client, not the admin user. You'll be asked for a secret, not a password. The admin client secret can be found in your manifest.

[aristotelesneto]

That worked successfully.

I did notice one key difference in the whole process.

After logging in, during the authorize the client to access Clouf Foundry certain functionality from my account, the manual step asked me to authorize 4 things:

• View details of your applications and services
• Push applications to your account and create and bind services
• Verify user permission to manage service instances
• Access your profile data including email address

When doing it via the Dashboard it only seems to ask me the following:

• Access your profile data including email address
• Verify user permission to manage service instances

Would that make a difference?
Edit: testing after all 4 permissions are authorised gives me the answer to this question - no change.

[shalako]

When testing manually, your token receives four scopes because that is what the client is configured in uaa to offer. That is, a client can ask for a subset, or all, of those four scopes but no more. When you get the token with the uaac CLI and do not specify the scopes you want, you will receive all four.

The MySQL dashboard is configured to only ask for the scopes it needs; it only asks for the two scopes you listed.

To get even close to simulating the dashboard sso flow using the uaac CLI, run through the same test as above, but modify the token client get step to specify only the two scopes the dashboard asks for:

uaac token authcode get -c <dashboard client name> -s <dashboard client secret> --port 5000 --scope cloud_controller_service_permissions.read openid 

[shalako]

We're going to close this issue due to inactivity. If you've tried our suggestion and have further questions, feel free to reopen.

[aristotelesneto]

Apologies for not being in touch - I've just had twin babies and they're keeping me busy. I am still having this issue, despite trying the suggested avenues.

Will try again when I get some more time, and report back.