slashdevops / idp-scim-sync Goto Github PK
View Code? Open in Web Editor NEWKeep your AWS Single Sign-On (SSO) groups and users in sync with your Google Workspace directory
License: Apache License 2.0
Keep your AWS Single Sign-On (SSO) groups and users in sync with your Google Workspace directory
License: Apache License 2.0
Describe the bug
When attempting to sync a group which contains another group as a member, the sync fails when trying to retrieve user details for that address.
When running, I get the following error:
Error: cannot sync groups and their members: error getting users from the identity provider: idp: error getting user: 019c6y181bx7798, email: accounts@<domain>, error: google: error getting user accounts@<domain>: googleapi: Error 400: Type not supported: userKey, badRequest
To Reproduce
Expected behavior
I would expect to have a configuration option that either expanded the group or ignored it.
Additional context
None
When making the installan (or update) through SAR, secrets are always asked. According e.g. #99 Workload Identity Federation should be supported, but there doesn't seem to be any documentation how to use it with the service itself.
Of course general instructions are e.g. at https://dwdraju.medium.com/calling-google-cloud-services-from-aws-using-iam-roles-without-gcp-service-account-credentials-af918c05384d, but is there some config required as sync lambda side?
Is your feature request related to a problem? Please describe.
I am trying to run idp-scim-sync in a k8s pod, but it does not work as it expects to have a credentials.json locally instead of an aws secret as when running like aws lambda.
Describe the solution you'd like
I would like for it to use environment variables to read secrets if they are set as it does when running like a aws lambda
Describe alternatives you've considered
I tried faking pod being a lambda with _LAMBDA_SERVER_PORT, but that just hangs
There is also option of creating a secret in k8s and mounting that as a file, but would go against our best practices
Additional context
Hello,
In some cases, the local "State" and the AWS state are inconsistent. We stumbled upon the case where the user was deleted in AWS but not in the local State. This generated an error 404, and the synchronization was not occurring anymore :( Would it be possible to ignore this error, and force the user deletion ?
Best,
L.
Hello,
We have been investigating this issue. When Google is facing some problems, the list of groups returned is incomplete and the idp-scim-sync deletes the group on the AWS side. In other words, the users are losing access to our AWS infra :(
We think that this is related to the callback functions (see here and here). According to google-api-go-client code, "Pages invokes f for each page of results. A non-nil error returned from f will halt the iteration." As you can see in the body of the callback functions, we always return nil, so we never stop the iteration upon receiving an error, which might cause an inconsistency ๐ค
Best,
L.
First of all many thanks for your work!
Describe the bug
Users suspended in Google Workspace are synced to AWS SSO as Enabled
To Reproduce
Expected behavior
User either (preferred) gets synced as "disabled" (this is how ssosync works) or alternatively does not get synced at all (and gets removed if exists)
Describe the bug
Version: 0.0.6
I have issue with sync the newly created group to AWS SSO. I added the new group and run idpscim and I got error:
WARN[0004] aws checkHTTPResponse: body: {"schema":["urn:ietf:params:scim:api:messages:2.0:Error"],"detail":"TENANT does not exist.","status":"404","exceptionRequestId":"93529112-dfce-4fef-82ed-ddd588a90e6b","timeStamp":"2022-02-11 06:10:54.937"} method=POST status="404 Not Found"
Error: cannot sync groups and their members: error syncing state: error reconciling groups: error creating groups in SCIM Provider: scim: error creating group: aws CreateGroup: error decoding response body: %!w(<nil>), body:
To Reproduce
Steps to reproduce the behavior:
โ ./idpscim
using config file: /idp-scim-sync/build/.idpscim.yaml
INFO[0000] starting sync groups codeVersion=main
INFO[0000] getting Identity Provider data group_filter="[name:aws-sso* email:aws-sso* email:administrators*]"
INFO[0003] getting state data
INFO[0004] syncing from state lastsync="2022-02-10T16:27:51+01:00" since=14h43m3.72218s
INFO[0004] provider groups and state groups are different
INFO[0004] reconciling groups idp=3 state=2
WARN[0004] creating groups quantity=1
WARN[0004] creating group group=aws-sso-wa
WARN[0004] aws checkHTTPResponse: body: {"schema":["urn:ietf:params:scim:api:messages:2.0:Error"],"detail":"TENANT does not exist.","status":"404","exceptionRequestId":"93529112-dfce-4fef-82ed-ddd588a90e6b","timeStamp":"2022-02-11 06:10:54.937"} method=POST status="404 Not Found"
Error: cannot sync groups and their members: error syncing state: error reconciling groups: error creating groups in SCIM Provider: scim: error creating group: aws CreateGroup: error decoding response body: %!w(<nil>), body:
Usage:
idpscim [flags]
Flags:
-k, --aws-s3-bucket-key string AWS S3 Bucket key to store the state (default "state.json")
-b, --aws-s3-bucket-name string AWS S3 Bucket name to store the state
-t, --aws-scim-access-token string AWS SSO SCIM API Access Token
-e, --aws-scim-endpoint string AWS SSO SCIM API Endpoint
-c, --config-file string configuration file (default ".idpscim.yaml")
-d, --debug fast way to set the log-level to debug
-q, --gws-groups-filter strings GWS Groups query parameter, example: --gws-groups-filter 'name:Admin* email:admin*' --gws-groups-filter 'name:Power* email:power*'
-s, --gws-service-account-file string path to Google Workspace service account file (default "credentials.json")
-u, --gws-user-email string GWS user email with allowed access to the Google Workspace Service Account
-h, --help help for idpscim
-f, --log-format string set the log format (default "text")
-l, --log-level string set the log level [panic|fatal|error|warn|info|debug|trace] (default "info")
-m, --sync-method string Sync method to use [groups] (default "groups")
-v, --version version for idpscim
Error: cannot sync groups and their members: error syncing state: error reconciling groups: error creating groups in SCIM Provider: scim: error creating group: aws CreateGroup: error decoding response body: %!w(<nil>), body:
Expected behavior
The new added group in GSuite should be created in AWS SSO
Additional context
When I remove this new group, everything is working fine and I have synced state in S3
# ./idpscimcli gws groups list
using config file: /idp-scim-sync/build/.idpscim.yaml
INFO[0000] 2 groups found
INFO[0000] [
{
"email": "[email protected]",
"etag": "\"lEblPI0s7w1i1rxkFOkQrCYLrPcC1v0pVpIQVj2CXd8/TVN0qWdZ4r4TJc2wW9-CitlYWJs\"",
"id": "00ihv6364h7nes5",
"name": "aws-sso-admins"
},
{
"email": "[email protected]",
"etag": "\"lEblPI0s7w1i1rxkFOkQrCYLrPcC1v0pVpIQVj2CXd8/WAABaGiJAOrI7AFxy3Iq4nloOLM\"",
"id": "04iylrwe3iefkh8",
"name": "aws-sso-powerusers"
}
]
# ./idpscim
using config file: /idp-scim-sync/build/.idpscim.yaml
INFO[0000] starting sync groups codeVersion=main
INFO[0000] getting Identity Provider data group_filter="[name:aws-sso* email:aws-sso* email:administrators*]"
INFO[0003] getting state data
INFO[0003] syncing from state lastsync="2022-02-10T16:27:51+01:00" since=14h50m22.429897s
INFO[0003] provider groups and state groups are the same, nothing to do with groups
INFO[0003] provider users and state users are the same, nothing to do with users
INFO[0003] provider groups-members and state groups-members are the same, nothing to do with groups-members
INFO[0003] storing the new state groups=2 lastSync="2022-02-11T07:18:13+01:00" users=5
INFO[0003] sync completed
INFO[0003] sync groups completed duration=3.796357958s
but when I check the same groups in AWS SSO still I'm getting the same error:
# ./idpscimcli aws groups list
using config file: /idp-scim-sync/build/.idpscim.yaml
WARN[0000] aws checkHTTPResponse: body: {"schema":["urn:ietf:params:scim:api:messages:2.0:Error"],"detail":"TENANT does not exist.","status":"404","exceptionRequestId":"24c1c32c-4a75-456e-97cb-5cb47f83481d","timeStamp":"2022-02-11 06:20:55.414"} method=GET status="404 Not Found"
ERRO[0000] error listing groups, error: aws ListGroups: error decoding response body: EOF
Error: aws ListGroups: error decoding response body: EOF
Usage:
idpscimcli aws groups list [flags]
Aliases:
list, l
Flags:
-q, --filter string AWS SSO SCIM API Filter, example: --filter 'displayName eq "Group Bar" and id eq "12324"'
-h, --help help for list
Global Flags:
-t, --aws-scim-access-token string AWS SSO SCIM API Access Token
-e, --aws-scim-endpoint string AWS SSO SCIM API Endpoint
-c, --config-file string configuration file (default ".idpscim.yaml")
-d, --debug enable log debug level
-f, --log-format string set the log format (default "text")
-l, --log-level string set the log level (default "info")
--output-format string output format (json|yaml) (default "json")
--timeout duration requests timeout (default 10s)
Error: aws ListGroups: error decoding response body: EOF
Please let me know if I can provide more details information / check something additionally.
Describe the bug
We previously used this tool successfully. However we had some manual users in AWS SSO that would not have groups updated (from before we enabled SCIM). We removed the users thinking they would sync via SCIM but they did not.
So I removed the state file to trigger a full sync, but we get the error in the title.
While at first I thought it similar to #62 , we do not have any groups with more than 100 users in. The state file is also not created.
I've tried removing and re deploying the lambda to no avail.
Error: cannot sync groups and their members: error doing the first sync: error reconciling groups: error updating groups from SCIM provider: scim: error updating groups: statusCode: 400, errCode: 400 Bad Request, errMsg: {
"schema": [
"urn:ietf:params:scim:api:messages:2.0:Error"
],
"schemas": [
"urn:ietf:params:scim:api:messages:2.0:Error"
],
"detail": "Request is unparsable, syntactically incorrect, or violates schema.",
"status": "400",
"exceptionRequestId": "977d19cb-b1d1-46e7-ba4d-936701dd6d3b",
"timeStamp": "2023-01-03 11:50:22.856"
}
Hi, awesome tool, thanks for making this.
Is your feature request related to a problem? Please describe.
The initial run creates the State file at the end of its process - unfortunately if it panics before getting to the end during that first run you'll wind up with no State file inside of s3. Then if you run it again you'll just get duplicate errors.
Unfortunately I have 100+ users in AWS now that failed to get put into groups so unless I wipe all of them it'll never be able to perform a sync in the future as it's got no clue on state so attempting to duplicate users.
Describe the solution you'd like
Some quick ideas I had, which may be completely off:
Additional context
I'm guessing this happened because I pulled in 100 users. Only 9 groups, but it panicked right after the very first user was added to a group. I've attached the panic in case you were curious.
`โ build git:(main) โ ./idpscim --config-file .idpscim.yaml
using config file: /Users/mikej/workspace/rxrevu/idp-scim-sync/build/.idpscim.yaml
WARN[0004] no state file found in the state repository, creating this
WARN[0004] syncing from scim service, first time syncing
WARN[0004] reconciling the SCIM data with the Identity Provider data
WARN[0004] creating groups quantity=9
WARN[0004] creating group group=aa-client-success
WARN[0004] creating group group=aa-devs
WARN[0005] creating group group=aa-ops
....
WARN[0006] creating users quantity=100
WARN[0148] joining users to groups quantity=3
WARN[0148] adding member to group email=[email protected] group=aa-client-success
panic: runtime error: index out of range [0] with length 0
goroutine 1 [running]:
github.com/slashdevops/idp-scim-sync/pkg/aws.(*SCIMService).GetUserByUserName(0xc0003dc040, {0x1caf0b0, 0xc0000b8000}, {0xc0005535a0, 0x10000c0006a69a8})
/Users/mikej/workspace/corp/idp-scim-sync/pkg/aws/scim.go:303 +0x99d
github.com/slashdevops/idp-scim-sync/internal/scim.(*Provider).CreateGroupsMembers(0xc0003cea10, {0x1caf0b0, 0xc0000b8000}, 0xc0002e29f0)
/Users/mikej/workspace/corp/idp-scim-sync/internal/scim/scim.go:413 +0x524
github.com/slashdevops/idp-scim-sync/internal/core.reconcilingGroupsMembers({0x1caf0b0, 0xc0000b8000}, {0x1cc4750, 0xc0003cea10}, 0xc0002e29f0, 0xc0002e3260)
/Users/mikej/workspace/corp/idp-scim-sync/internal/core/reconciling.go:161 +0x18d
github.com/slashdevops/idp-scim-sync/internal/core.scimSync({0x1caf0b0, 0xc0000b8000}, {0x1cc4750, 0xc0003cea10}, 0xc000539860, 0xc000555e90, 0xc0004b0720)
/Users/mikej/workspace/corp/idp-scim-sync/internal/core/actions.go:90 +0xb86
github.com/slashdevops/idp-scim-sync/internal/core.(*SyncService).SyncGroupsAndTheirMembers(0xc00047c900, {0x1caf0b0, 0xc0000b8000})
/Users/mikej/workspace/corp/idp-scim-sync/internal/core/sync.go:125 +0x705
github.com/slashdevops/idp-scim-sync/cmd/idpscim/cmd.syncGroups()
/Users/mikej/workspace/corp/idp-scim-sync/cmd/idpscim/cmd/root.go:317 +0x759
github.com/slashdevops/idp-scim-sync/cmd/idpscim/cmd.sync()
/Users/mikej/workspace/corp/idp-scim-sync/cmd/idpscim/cmd/root.go:229 +0xa5
github.com/slashdevops/idp-scim-sync/cmd/idpscim/cmd.glob..func1(0x22acda0, {0x1b3942b, 0x2, 0x2})
/Users/mikej/workspace/corp/idp-scim-sync/cmd/idpscim/cmd/root.go:43 +0x17
github.com/spf13/cobra.(*Command).execute(0x22acda0, {0xc0000b2190, 0x2, 0x2})
/Users/mikej/go/pkg/mod/github.com/spf13/[email protected]/command.go:856 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0x22acda0)
/Users/mikej/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
/Users/mikej/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
github.com/slashdevops/idp-scim-sync/cmd/idpscim/cmd.Execute()
/Users/mikej/workspace/corp/idp-scim-sync/cmd/idpscim/cmd/root.go:53 +0x85
main.main()
`
Is your feature request related to a problem? Please describe.
The go1.x lambda runtime is being deprecated
Describe the solution you'd like
The provided.al2
runtime should be used instead
Describe alternatives you've considered
I'm not aware of any better runtimes
Additional context
Email from Amazon:
We are ending support for the Go 1.x runtime in AWS Lambda, in line with Amazon Linux AMI end of maintenance support on December 31, 2023 [1]. Lambda continues to support the Go programming language, using the provided.al2 runtime. Using the provided.al2 runtime offers several benefits over the go1.x runtime, including support for AWS Graviton2 processors and a streamlined implementation with a smaller deployment package and faster function invoke path. For more information, see our blog post [2].
As described in the Lambda runtime support policy [3], end of support for language runtimes in Lambda happens in two stages. Starting December 31, 2023, Lambda will no longer apply security patches and other updates to the Go 1.x runtime used by Lambda functions, and functions using Go 1.x will no longer be eligible for technical support. In addition, you will no longer be able to create new Lambda functions using the Go 1.x runtime. Starting January 30, 2024, you will no longer be able to update existing functions using the Go 1.x runtime.
We recommend that you upgrade your existing Go 1.x functions to the provided.al2 runtime before December 31, 2023.
End of support does not impact function execution. Your functions will continue to run. However, they will be running on an unsupported runtime which is no longer maintained or patched by the AWS Lambda team.
Describe the bug
Error when trying to sync users and groups. The users import, as do the groups. However, during the add members to groups phase a fatal error is returned.
Version: v0.0.14
To Reproduce
CloudWatch trace logs for a sample request
{
"idp": 2,
"level": "info",
"msg": "reconciling groups members",
"scim": 2,
"time": "2022-09-02T17:05:53Z"
}
{
"level": "warning",
"msg": "joining users to groups",
"quantity": 2,
"time": "2022-09-02T17:05:53Z"
}
{
"body": null,
"level": "trace",
"method": "GET",
"msg": "aws newRequest: request",
"path": "/xxxxxxxxxxxxxxxxx/scim/v2/Users",
"query": "filter=userName+eq+%22first.last%40example.com%22",
"time": "2022-09-02T17:05:53Z",
"url": "https://scim.us-east-1.amazonaws.com/xxxxxxxxxxxxxxxxx/scim/v2/Users?filter=userName+eq+%22first.last%40example.com%22"
}
{
"body": "{"totalResults":1,"itemsPerPage":1,"startIndex":1,"schemas":["urn:ietf:params:scim:api:messages:2.0:ListResponse"],"Resources":[{"id":"xxxxxxxxxxxxxxxxx","externalId":"xxxxxxxxxxxxxxxxx","meta":{"resourceType":"User","created":"2022-09-02T16:56:28Z","lastModified":"2022-09-02T16:56:28Z"},"schemas":["urn:ietf:params:scim:schemas:core:2.0:User"],"userName":"[email protected]","name":{"familyName\โ:\โLast\โ,\โgivenName\โ:\โFirst "},"displayName\โ:\โFirst Last\โ,\โactive":true,"emails":[{"value":"[email protected]","type":"work","primary":true}]}]}",
"level": "trace",
"msg": "aws GetUserByUserName: response raw body data",
"time": "2022-09-02T17:05:53Z"
}
Error: cannot sync groups and their members: error doing the first sync: error reconciling groups members: error creating groups members in SCIM provider: scim: error getting user by email: aws GetUserByUserName: userName: [email protected], error decoding response body: EOF
Expected behavior
User would be added to group
Additional context
There appears to be an error when parsing the json response data in module:
pkg/aws/scim.go around Line 500
Note: This had been working earlier this week, but started throwing this error. I tried deleting the state file with no luck. (State File does not recreate due to failed run). Deleted the entire CF Stack and recreated with no resolution. I don't see anything wrong with the response JSON so I don't know why it would be throwing the EOF error suddenly unless Google has a problem suddenly.
Is your feature request related to a problem? Please describe.
I'm always frustrated when Google APIs decide to barf and exclude several groups on AWS SSO side.
Google APIs have failed multiple times in the last 5 weeks. When it does, it consistently returns 502 for several minutes (idp-scim-sync handles it nicely!) and when it returns, for another minute it wildly provide a subset of existent groups, and then it returns back to normal.
When the recovery happens, all AWS SSO groups are re-created, but the permissionSets on AWS SSO reference to previously existent groups, forcing us to remap the group to account with permission all over again. This process takes ~20 minutes via Terraform, since the permissionSet also creates new IAM roles that need to be correlated to EKS roles.
Describe the solution you'd like
Introduce a new flag, preventing created groups to be removed, only to be added or modified.
Describe alternatives you've considered
An alternate solution would be in the case of a large diverging number of groups (when comparing state file groups to returned Google groups), attempt to re-fetch the groups up to 3 times, and then proceed with operation.
Describe the bug
Error: cannot sync groups and their members: error doing the first sync: error reconciling groups: error updating groups in SCIM Provider: scim: error updating groups: aws checkHTTPResponse: error code: 400 Bad Request, body:
{
"schema": [
"urn:ietf:params:scim:api:messages:2.0:Error"
],
"detail": "Request is unparsable, syntactically incorrect, or violates schema.",
"status": "400",
"exceptionRequestId": "88d9c6f3-566d-42b3-b1ed-183c3fc1efc4",
"timeStamp": "2022-01-26 16:50:48.135"
}
To Reproduce
Migrating from ssosync fails the first time syncing.
Expected behaviour
is expected to do a transparent sync without any error
Describe the bug
If you have more than 50 users, and the attributes of the users in the SCIM side are different from the IDP side the first time, only the first 50 users are updated, the others no.
To Reproduce
Steps to reproduce the behaviour:
Expected behavior
when the users in the SCIM side have the same username that the idp email, the rest of the attributes needs to be synced with the idp after the first sycn
Describe the bug
A Google Workspace can have multiple Gmail domains, e.g. @foo.com
and @bar.com
. If a user's email domain is changed after previously synced to AWS SSO via SCIM, then a subsequent SSO sync attempts to create a new user leads to an error.
WARN[0012] creating users quantity=3
WARN[0012] aws checkHTTPResponse: body: {"schema":["urn:ietf:params:scim:api:messages:2.0:Error"],"schemas":["urn:ietf:params:scim:api:messages:2.0:Error"],"detail":"Refused to create a new, duplicate resource.","status":"409","exceptionRequestId":"1f0de3fa-8bce-49ed-ae22-8604172fcb80","timeStamp":"2022-03-22 01:39:54.152"} method=POST status="409 Conflict"
Error: cannot sync groups and their members: error syncing state: error reconciling users: error creating users in SCIM Provider: scim: error creating user: aws CreateUser: error decoding response body: EOF
Error: cannot sync groups and their members: error syncing state: error reconciling users: error creating users in SCIM Provider: scim: error creating user: aws CreateUser: error decoding response body: EOF
Afterward, if I re-run the SSO sync script, a new error appears. This time, the program attempts to duplicate the group (not the user):
INFO[0011] provider groups and state groups are different
INFO[0011] reconciling groups idp=4 state=3
WARN[0011] creating groups quantity=1
WARN[0011] creating group group="AWS SSO - Shared Sandbox - Full Admin"
WARN[0011] aws checkHTTPResponse: body: {"schema":["urn:ietf:params:scim:api:messages:2.0:Error"],"schemas":["urn:ietf:params:scim:api:messages:2.0:Error"],"detail":"Refused to create a new, duplicate resource.","status":"409","exceptionRequestId":"80922f78-7149-4cf6-bf7c-4748047072dd","timeStamp":"2022-03-22 01:40:31.832"} method=POST status="409 Conflict"
Error: cannot sync groups and their members: error syncing state: error reconciling groups: error creating groups in SCIM Provider: scim: error creating group: aws CreateGroup: error decoding response body: %!w(<nil>), body:
This is actually strange, as nothing about the group has changed other than the email of a member. Maybe that's all that's needed?
To Reproduce
Steps to reproduce the behavior:
[email protected]
aws-sso-test
and add Mat to the group[email protected]
to [email protected]
For the idpscim to be able to connect to the S3 bucket, it seems I need to populate the following environment variables:
Would it possible to get the credentials using profile? (e.g. using the AWS_PROFILE env var)
Thank you!
Describe the bug
during the reconciliation of the users when I'm migration from ssosyc to this one fails with:
runtime error: index out of range [0] with length 0: boundsError[{ "path": "github.com/aws/[email protected]/lambda/errors.go", "line": 39, "label": "lambdaPanicResponse" },{ "path": "github.com/aws/[email protected]/lambda/function.go", "line": 36, "label": "(*Function).Invoke.func1" },{ "path": "runtime/panic.go", "line": 1038, "label": "gopanic" },{ "path": "runtime/panic.go", "line": 90, "label": "goPanicIndex" },{ "path": "work/idp-scim-sync/idp-scim-sync/pkg/aws/scim.go", "line": 303, "label": "(*SCIMService).GetUserByUserName" },{ "path": "work/idp-scim-sync/idp-scim-sync/internal/scim/scim.go", "line": 413, "label": "(*Provider).CreateGroupsMembers" },{ "path": "work/idp-scim-sync/idp-scim-sync/internal/core/reconciling.go", "line": 161, "label": "reconcilingGroupsMembers" },{ "path": "work/idp-scim-sync/idp-scim-sync/internal/core/actions.go", "line": 90, "label": "scimSync" },{ "path": "work/idp-scim-sync/idp-scim-sync/internal/core/sync.go", "line": 125, "label": "(*SyncService).SyncGroupsAndTheirMembers" },{ "path": "work/idp-scim-sync/idp-scim-sync/cmd/idpscim/cmd/root.go", "line": 317, "label": "syncGroups" },{ "path": "work/idp-scim-sync/idp-scim-sync/cmd/idpscim/cmd/root.go", "line": 229, "label": "sync" },{ "path": "work/idp-scim-sync/idp-scim-sync/cmd/idpscim/cmd/root.go", "line": 43, "label": "glob..func1" },{ "path": "github.com/spf13/[email protected]/command.go", "line": 856, "label": "(*Command).execute" },{ "path": "github.com/spf13/[email protected]/command.go", "line": 974, "label": "(*Command).ExecuteC" },{ "path": "github.com/spf13/[email protected]/command.go", "line": 902, "label": "(*Command).Execute" },{ "path": "reflect/value.go", "line": 556, "label": "Value.call" },{ "path": "reflect/value.go", "line": 339, "label": "Value.Call" },{ "path": "github.com/aws/[email protected]/lambda/handler.go", "line": 124, "label": "NewHandler.func1" },{ "path": "github.com/aws/[email protected]/lambda/handler.go", "line": 24, "label": "lambdaHandler.Invoke" },{ "path": "github.com/aws/[email protected]/lambda/function.go", "line": 64, "label": "(*Function).Invoke" },{ "path": "reflect/value.go", "line": 556, "label": "Value.call" },{ "path": "reflect/value.go", "line": 339, "label": "Value.Call" },{ "path": "net/rpc/server.go", "line": 377, "label": "(*service).call" },{ "path": "runtime/asm_amd64.s", "line": 1581, "label": "goexit" }] | runtime error: index out of range [0] with length 0: boundsError [{"path":"github.com/aws/[email protected]/lambda/errors.go","line":39,"label":"lambdaPanicResponse"},{"path":"github.com/aws/[email protected]/lambda/function.go","line":36,"label":"(*Function).Invoke.func1"},{"path":"runtime/panic.go","line":1038,"label":"gopanic"},{"path":"runtime/panic.go","line":90,"label":"goPanicIndex"},{"path":"work/idp-scim-sync/idp-scim-sync/pkg/aws/scim.go","line":303,"label":"(*SCIMService).GetUserByUserName"},{"path":"work/idp-scim-sync/idp-scim-sync/internal/scim/scim.go","line":413,"label":"(*Provider).CreateGroupsMembers"},{"path":"work/idp-scim-sync/idp-scim-sync/internal/core/reconciling.go","line":161,"label":"reconcilingGroupsMembers"},{"path":"work/idp-scim-sync/idp-scim-sync/internal/core/actions.go","line":90,"label":"scimSync"},{"path":"work/idp-scim-sync/idp-scim-sync/internal/core/sync.go","line":125,"label":"(*SyncService).SyncGroupsAndTheirMembers"},{"path":"work/idp-scim-sync/idp-scim-sync/cmd/idpscim/cmd/root.go","line":317,"label":"syncGroups"},{"path":"work/idp-scim-sync/idp-scim-sync/cmd/idpscim/cmd/root.go","line":229,"label":"sync"},{"path":"work/idp-scim-sync/idp-scim-sync/cmd/idpscim/cmd/root.go","line":43,"label":"glob..func1"},{"path":"github.com/spf13/[email protected]/command.go","line":856,"label":"(*Command).execute"},{"path":"github.com/spf13/[email protected]/command.go","line":974,"label":"(*Command).ExecuteC"},{"path":"github.com/spf13/[email protected]/command.go","line":902,"label":"(*Command).Execute"},{"path":"reflect/value.go","line":556,"label":"Value.call"},{"path":"reflect/value.go","line":339,"label":"Value.Call"},{"path":"github.com/aws/[email protected]/lambda/handler.go","line":124,"label":"NewHandler.func1"},{"path":"github.com/aws/[email protected]/lambda/handler.go","line":24,"label":"lambdaHandler.Invoke"},{"path":"github.com/aws/[email protected]/lambda/function.go","line":64,"label":"(*Function).Invoke"},{"path":"reflect/value.go","line":556,"label":"Value.call"},{"path":"reflect/value.go","line":339,"label":"Value.Call"},{"path":"net/rpc/server.go","line":377,"label":"(*service).call"},{"path":"runtime/asm_amd64.s","line":1581,"label":"goexit"}]
To Reproduce
First sync under the migration from the ssosyc to this one
Expected behavior
The same groups and users
Describe the vulnerability
Log entries created from user input
Building log entries from user-controlled sources is vulnerable to insertion of forged log entries by a malicious user.
Tracking issue for:
Describe the bug
due to limitations in the AWS SSO SCIM API
, specifically the patchgroup operation when some groups have more than 100 members
...
* A maximum of 100 membership changes are allowed in a single request.
...
the ACIM API
response with:
{"schema":["urn:ietf:params:scim:api:messages:2.0:Error"],"schemas":["urn:ietf:params:scim:api:messages:2.0:Error"],"detail":"Request is unparsable, syntactically incorrect, or violates schema.","status":"400","exceptionRequestId":"f32ad558-5b05-45ea-8135-552a2f254d4f","timeStamp":"2022-04-15 14:09:03.693"}
To Reproduce
Have a group in Google Workspace
with more than 100 members
Expected behavior
do the sync without any error
Hey ๐ Firstly thank you for this open source project! I'm looking to move over from ssosync. I am using the CloudFormation template to deploy as I am managing the provision of this solution via the LZA project.
For GWSGroupsFilter
, I would like to provide a list of specific groups. I already have Groups set up in my IDP and it doesn't make sense to duplicate them just to have a prefix (e.g. cloud-
or aws-
) as then I'd be managing members in two places. So instead, I'd want to provide exact group email matches, e.g. [email protected]
, [email protected]
..
I have a couple questions:
IDPSCIM_GWS_GROUPS_FILTER
.[email protected]
has 5 members, 4 of them are Users, and 1 is another group called [email protected]
, which has other members within it.Describe the bug
even when it is not used, the cloudformation template still having the parameter DisableState
To Reproduce
when is used the AWS Serverless public repository - slashdevops/idp-scim-sync to deploy the lambda the parameter DisableState
appears in the template.
Expected behavior
parameter DisableState
doesn't have to be there
Is your feature request related to a problem? Please describe.
Just wrapped up most of the setup - things are working well so far with the manual test... thank you!
Though - I just realized that the current implementation does not support SAML user attributes, and I was really hoping to take advantage of AWS's relatively new (Nov 2020) attribute-based access control ABAC: https://aws.amazon.com/blogs/aws/new-attributes-based-access-control-with-aws-single-sign-on/
Describe the solution you'd like
Modify code to also retrieve any custom attributes that have been mapped in the Google Workspace SAML app, and include those in the state file and diff when determining what to send to the AWS SCIM endpoint.
Additional context
Add any other context or screenshots about the feature request here.
Creating custom attributes in the Workspace user settings: https://admin.google.com/ac/customschema:
Mapping custom attributes in the AWS SSO SAML app: https://admin.google.com/ac/apps/saml/XXXXXXXX/attrmapping
AWS SSO attribute-based access mappings: https://us-west-2.console.aws.amazon.com/singlesignon/identity/home?region=us-west-2#!/settings?tab=abac
Hi:
I configured the project to run it in my local environment, now I'm trying to deploy the lambda but I haven't found a way to specify the bucket where my state is.
Is there a way to use a pre-existing bucket?
Thank you!
Amazon just announced new APIs to manage SSO users and groups, I'm hoping to get a first impression if there's any interest in migrating to them?
https://docs.aws.amazon.com/sdk-for-go/api/service/identitystore/
Hi team,
I want to sync some another attribute of user in GG Workspace like department to AWS SSO, can you please show me how to do this, is there any needed configuration on idp function?
Thanks,
provider.GetGroupsMembers and scim.GetGroupsMembersBruteForce doesn't add the hashCode when the field "Resources" is empty
Hi there,
I'm currently attempting to setup your tool using the Serverless Application Repository Cloudformation stack. I've followed your documentation and setup Automated Provisioning, Google Service Account, enabled Domain-wide delegation and provided the necessary OAuth Scopes mentioned in another Issue. Upon running the tool I'm receiving the following error:
cannot sync groups and their members: error getting groups from the identity provider: idp: error listing groups: googleapi: Error 404: Domain not found., notFound: withStack
null
I'm wondering if this is something you've encountered before, and if so is there some obvious misconfiguration involved? I'm sure I've provided the correct values for the application parameters, and there's not too much more configuration to get wrong!
Thank you :)
Hi,
I'm sort of new to this, I've been doing the manual provisioning in AWS SSO with google workspace directory for a while. I decied to set up the auto provisioning, went through the steps with AWS Serverless Applications. After the first sync, I can see all the users and groups synced up, but all the groups are empty and and none of the users is in their groups. I only had 2 groups (regular users and admin users), From the logs, i can see it sees the group (AWS-SSO) has 69 users. It tried to create them, but it runs into 409 conflict saying there's duplicate users already. I didn't remove all my original users and groups before turning on auto-provisioning, is that a problem?
{
"idp": 69,
"level": "info",
"msg": "reconciling users",
"scim": 50,
"time": "2022-02-14T17:37:21Z"
}
{
"level": "warning",
"msg": "creating users",
"quantity": 23,
"time": "2022-02-14T17:37:21Z"
}
{
"level": "warning",
"method": "POST",
"msg": "aws checkHTTPResponse: body: {"schema":["urn:ietf:params:scim:api:messages:2.0:Error"],"detail":"Refused to create a new, duplicate resource.","status":"409","exceptionRequestId":"279ec396-441a-49f8-b1b6-a4e04c08348c","timeStamp":"2022-02-14 17:37:21.224"}\n",
"status": "409 Conflict",
"time": "2022-02-14T17:37:21Z"
}
Error: cannot sync groups and their members: error doing the first sync: error reconciling users: error creating users in SCIM Provider: scim: error creating user: aws CreateUser: error decoding response body: EOF
To reproduce the problem:
I created a new group, added a brand new user, still getting the same errors in the log, it created the group and user, but user isn't showing as part of the group members on AWS, and there's still no state file being created.
This issue come from a comment in issue #62
Hi there!
This bug appeared again for us; we're using last version (0.0.13).
We're syncing three groups, and one of them have more than 100 users (137 right now).
I deleted the state to force a clean run of the sync app, but it still returns that error (and doesn't recreate the state.json).
Thanks a lot!
Originally posted by @snavarro-factorial in #62 (comment)
Hey I set up AWS SSO with all the regular actions done on Google Workspace.
But when launching this app from the Serverless Application Repository, I have to enter the file contents for the service account credentails from Google.
This part in particular would be nice to be documented.
Eventhough I set up a service account and downloaded the keys as json. I get an authentication error in the lambda functio then.
Please help
Describe the bug
Error deleting user that doesn't exit in the SCIM side
Error: cannot sync groups and their members: error syncing state: error reconciling users: error deleting users in SCIM Provider: scim: error deleting user: 9067641b8c-172cc7ee-9e21-4b04-b77a-b76c01bf1929, statusCode: 404, errCode: 404 Not Found, errMsg:
{
"schema": [
"urn:ietf:params:scim:api:messages:2.0:Error"
],
"schemas": [
"urn:ietf:params:scim:api:messages:2.0:Error"
],
"detail": "USER does not exist.",
"status": "404",
"exceptionRequestId": "15a8c7b4-80bf-45da-8607-80c0a954632b",
"timeStamp": "2022-04-14 07:58:47.009"
}
To Reproduce
Steps to reproduce the behavior:
Expected behavior
generate a warning log with the message and continue with the execution removing the user from the state
Describe the bug
Whenever I try to sync with with a search query I get this error:
Error: cannot sync groups and their members: error doing the first sync: error getting groups from the SCIM service: scim: error listing groups: aws checkHTTPResponse: error code: 401 Unauthorized, body:
To Reproduce
Steps to reproduce the behavior:
./idpscim [...] -q 'name:SysAdmin* email:sysadmin*'
(I've tried too with name=SysAdmin
and many others as this doc says https://developers.google.com/admin-sdk/directory/v1/guides/search-groups)Expected behavior
It actually syncs that group.
Additional context
If I run ./idpscim
without the query parameter it actually starts reading some emails but then occurs the same error as in issue #49 so I can't find how to sync groups.
Also my Google account has full admin permissions, don't know if I'm missing something on the service account (which has domain-wide delegation permissions applied too and they work, since removing them leads to another error).
Thanks!
Describe the bug
Only 200 users of 263 members of a google group are sync'd
All appropriate users are added to Identity Center user base, but adding the members to groups seems to be where the problem lies.
To Reproduce
Sync a group containing more than 200 members. You will see two "batches" of 100 users each are retrieved. Syncing stops after the second batch with no error. If you add a single member to the large group, the behavior is to add the new user if it falls under the 200 limit, and a single user is then removed from the group if they fall over the 200 limit.
Expected behavior
all users in a group are added
Additional context
I have not found any quota limits in either the AWS SCIM, AWS or Google that would prevent this. I also do not know why the "3rd" page of users is not coming in, i.e. nextpage token not present, or hard carded value in the source). Execution traces did show that group members were done in batch size of 100 per request.
Thanks in advance
Is your feature request related to a problem? Please describe.
Support nested groups in the identity provider side (Google Workspace)
Describe the solution you'd like
It is possible using the feature of Google Workspace directory API query parameter for Method: members.list includeDerivedMembership
AWS Single Sign-On doesn't support nested groups, but the idea is to have all the users members of the first level groups (root level and the nested groups) of google be added to the groups of AWS SSO
I'm getting the following "error decoding response body: EOF" error when log-level=trace in the GetUserByUserName
method:
Error: cannot sync groups and their members: error doing the first sync: error reconciling groups members: error creating groups members in SCIM provider: scim: error getting user by email: aws GetUserByUserName: userName: [email protected], error decoding response body: EOF
Source code:
Lines 489 to 503 in 094a7fc
@snavarro-factorial mentioned it in this comment too.
Is your feature request related to a problem? Please describe.
The Lambda function that's run on AWS requires JSON credentials for a GCP service account, which includes a non-rotatable secret key, and through which has broad access to all gSuite user data.
Describe the solution you'd like
GCP has a feature Workload Identity Federation (AWS instructions), which allows AWS to use OAuth to obtain short-lived credentials for impersonating a service account. Rather than having a JSON file with a secret key, instead you only need a JSON file with non-secret information, and everything else is configured in GCP via an identity pool.
Describe alternatives you've considered
The alternative here is just to use service account credentials, which just requires being a bit more careful.
Additional context
According to the AWS instructions, the primary requirement seems to be that golang.org/x/oauth2
must be version v0.0.0-20210218202405-ba52d332ba99
or later. Beyond that, there just needs to be a way to assign an AWS IAM Role to the executing function so that it has a role it can assert during the credential exchange.
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
Is your feature request related to a problem? Please describe.
The current implementation relies on long lived credentials from google cloud.
Describe the solution you'd like
We could use Workload Identity federation described https://cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds
Describe alternatives you've considered
None
Additional context
None
Hi there,
I successfully deployed the lambda function using the AWS Serverless Repository and was able to sync all Users and Groups into my AWS account by leaving the IDPSCIM_GWS_GROUPS_FILTER blank.
However, I want to sync only a few Groups and my understanding is the only way to achieve this is filter all Groups except the ones I want to sync using IDPSCIM_GWS_GROUPS_FILTER.
I tried setting this parameter but kept running into this error:
Error: cannot sync groups and their members: error getting groups from the identity provider: idp: error listing groups: googleapi: Error 400: Invalid Input: query, invalid
I have tried the following formats:
[email protected],[email protected],[email protected]
'[email protected]','[email protected]','[email protected]'
'[email protected]','[email protected]','[email protected]'
"[email protected]","[email protected],"[email protected]"
email:[email protected],email:[email protected],email:[email protected]
'email:[email protected]','email:[email protected]','email:[email protected]'
Describe the bug
If group that matches given group filter has external (from outside organization) users, sync would fail as it tries to receive information of those user(s), too.
This causes error:
{
"errorMessage": "cannot sync groups and their members: error getting users from the identity provider: idp: error getting user: 12345, email: [email protected], error: google: error getting user [email protected]: googleapi: Error 403: Not Authorized to access this resource/api, forbidden",
"errorType": "withStack"
}
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Synchronization to pass.
It could probably print out some warning about user that wasn't able to sync, but single guest shouldn't prevent whole syncronization.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.