vmware-tanzu-labs/cf-mgmt

Go automation for managing orgs, spaces that can be driven from concourse pipeline and Git managed metadata

cf-mgmt's Issues

integrate with notifications service for key events

• user added to uaa
• user assigned to role
• user removed from role
• user invite

How do I find the password and secret for a user account?

   I am just starting to look at cf-mgmt and trying to do and do an export-config.  I am trying to find out the difference between the password and user-secret and where I find them?  This appears to require a password and secret for the same ID and the usage info, to me, is not clear on what they are nor where to find them.

Option	Description
user-id value	user id that has admin privileges [%USER_ID%]
password value	password for user account that has admin privileges [%PASSWORD%]
client-secret value	secret for user account that has admin privileges [%CLIENT_SECRET%]

Enhancement Idea: Space Defaults

I'm interested in enhancing the Space Defaults framework to allow for additional space defaults. I've never written go before. I really want to code this - are you OK with a pull request that will need a lot of correction?

Idea

Configuration

• the base spaceDefaults.yml continues to function as it does today.
• allow additional *SpaceDefaults.yml that contain an org-prefix property

Execution

• When cf-mgmt matches the org-prefix string with an ORG name, both the main spaceDefaults.yml and the additional file would be loaded and applied.
• If the ORG name doesn't match any of the prefixes, only the spaceDefaults.yml is loaded and applied.

Command "cfmgmt update-sapce-security-groups" takes a long time to complete due to spending time to create empty security groups.

Hi, Caleb,
When "cfmgmt update-sapce-security-groups" was run, the pipeline took a long time to complete. The issue is with the default empty security group when a space is added to cfmgmt's configuration. When a space is added, a security-group.json file is created with an empty array. This file is not referenced in spaceConfig.yml. But when "cfmgmt update-sapce-security-group" is run, the pipeline automatically tries to bind "orgname-spacename" (eg. testorg-dev) as the default security group for the space even though there is nothing in security-group.json and it is not referenced. We have 500+ spaces. Binding empty security groups took 15 mins to complete.
Could you please research and fix this issue? For example, don't bind when there is nothing in security-group.json.

Thanks!

Joe

Git private key

Do we want the pipeline to default to using the private key of the user running the fly command? fly -t lite set-pipeline -p cf-mgmt -c pipeline.yml --load-vars-from=vars.yml --var \"git_private_key=$(cat ~/.ssh/id_rsa)\""

Sample total-routes too small

In all the example configs, the total-routes is 10 but should be much higher, perhaps 1000.

Some security group names are not created correctly when "cf-mgmt create-security-groups" is run.

Hi, Caleb,
When command "cf-mgmt create-security-groups" is run, some security groups are not created correctly in CF. Here is the command:

cf-mgmt create-security-groups --system-domain test.company.com --user-id XXXXX --password YYYYY --client-secret ZZZZZ

 The execution created about 500+ security groups, of which 10 were created with the wrong names.

 The issue is that when security groups have names ending with s, n or o, these letters are stripped off the names from the end.  For example, sec group name dns becomes d, xxxxxopen becomes xxxxxope, yyyoss becomes yyy, etc...

   Could you please research and provide a fix?

Thanks!

Joe

adding wildcards to protected orgs for smoke tests

CF-MGMT needs to protect smoke test orgs for products such as Redis. Currently Redis creates a smoke test org with a timestamp during it's smoke test. If CF-MGMT happens to run during the smoke test it tries delete the org and fails. Adding a wildcard option to address protected orgs that products create with timestamps. See example of the Redis Org created.

redis-test-ORG-1-2017_10_04-20h06m33.481s with a timestamp

delete user feature

Is the delete user feature merged with master branch and released?

update-space-users fails when adding a new uaac user to multiple spaces

If you add a user who is not in uaac to two spaces, cf-mgmt will try to create the user twice. This will generate an error at the second attempt. The two spaces are within the same org.

Looking through the code, is the list cf-mgmt uses to check users updated when a new user is created?

splunk-nozzle-org gets deleted

The Splunk Tile adds an org called splunk-nozzle-org. Since it is a partner it doesn't get added to the system org. splunk-nozzle-org needs to be added to the protected orgs list. I will submit a PR for this work.

Support userSearchFilter/groupSearchFilter

Hello,

We have users and groups that belong to different OUs that we would like to include as part of using LDAP with cf-mgmt. Currently cf-mgmt only supports one OU path for users and one for groups. We would like to be able to have multiple OU paths for both users and groups because we enforce separation of duties at our company (think different OUs for different environments, prod vs dev for example).

Manual User permissions ovewritten

When user permission are modified in the apps manager UI they do not stay persistent through a cf-mgmt pipeline push. Can a feature flag be added so that manual additions stay in place and can be added to the pipeline at a later time. @skibum55

Clarification: You must enable ldap if you are using the examples

Docs should be updated to show where to enable ldap. Logging should indicate LDAP is disabled.

oauth with 1.10

update space-users- after we run this against 1.10, sometimes we see 403 access token denied errors whe using gradle plugin.
If this happens I manually delete uaa user, create again, cf set-role for the user and it resolves

I couldnt conclude if this is a pipeline issue or gradle issue, but thought it would be worth asking here

LDAP users are removed when they shouldn't be

Given that RemoveUser is enabled for a space after an LDAP user is first added they are removed on the next run. The behavior is the user is added, removed, added, and so on.

first run:

2017/04/14 20:18:59 I0414 20:18:59.484362 18 users.go:143] Finding LDAP user for group: pcf_operations_space_auditor
2017/04/14 20:18:59 D0414 20:18:59.484405 18 ldap.go:55] Connecting to nos.example.com:3269
2017/04/14 20:18:59 D0414 20:18:59.629489 18 ldap.go:168] Searching for group: (cn=pcf_operations_space_auditor)
2017/04/14 20:18:59 D0414 20:18:59.629614 18 ldap.go:169] Using group search base: DC=nos,DC=example,DC=com
2017/04/14 20:18:59 D0414 20:18:59.665665 18 ldap.go:88] Getting details about user: CN=Smith\, Sean B,OU=End Users,OU=Accounts,DC=nw,DC=nos,DC=example,DC=com
2017/04/14 20:18:59 D0414 20:18:59.665763 18 ldap.go:124] User DN: CN=Smith\, Sean B,OU=End Users,OU=Accounts,DC=nw,DC=nos,DC=example,DC=com
2017/04/14 20:18:59 D0414 20:18:59.665944 18 ldap.go:132] CN unescaped: CN=Smith\, Sean B
2017/04/14 20:18:59 D0414 20:18:59.665971 18 ldap.go:135] CN escaped: CN=Smith, Sean B
2017/04/14 20:18:59 D0414 20:18:59.666033 18 ldap.go:137] Searching for user: (CN=Smith, Sean B)
2017/04/14 20:18:59 D0414 20:18:59.809243 18 ldap.go:88] Getting details about user: CN=Bar\, James I,OU=End Users,OU=Accounts,DC=nw,DC=nos,DC=example,DC=com
2017/04/14 20:18:59 D0414 20:18:59.809332 18 ldap.go:124] User DN: CN=Bar\, James I,OU=End Users,OU=Accounts,DC=nw,DC=nos,DC=example,DC=com
2017/04/14 20:18:59 D0414 20:18:59.809558 18 ldap.go:132] CN unescaped: CN=Bar\, James I
2017/04/14 20:18:59 D0414 20:18:59.809625 18 ldap.go:135] CN escaped: CN=Bar, James I
2017/04/14 20:18:59 D0414 20:18:59.809663 18 ldap.go:137] Searching for user: (CN=Bar, James I)
2017/04/14 20:18:59 D0414 20:18:59.9194 18 users.go:63] LdapUsers: [{CN=Smith\, Sean B,OU=End Users,OU=Accounts,DC=nw,DC=nos,DC=example,DC=com zk174c [email protected]} {CN=Bar\, James I,OU=End Users,OU=Accounts,DC=nw,DC=nos,DC=example,DC=com ks784b [email protected]}]
2017/04/14 20:18:59 D0414 20:18:59.919503 18 users.go:118] User[[email protected]] not found in: map[[email protected]:333bb853-753f-4821-9ccb-9c473f34b934]

second run:

2017/04/14 20:20:54 I0414 20:20:54.668532 18 users.go:143] Finding LDAP user for group: pcf_operations_space_auditor
2017/04/14 20:20:54 D0414 20:20:54.668548 18 ldap.go:55] Connecting to nos.example.com:3269
2017/04/14 20:20:54 D0414 20:20:54.83809 18 ldap.go:168] Searching for group: (cn=pcf_operations_space_auditor)
2017/04/14 20:20:54 D0414 20:20:54.838189 18 ldap.go:169] Using group search base: DC=nos,DC=example,DC=com
2017/04/14 20:20:54 D0414 20:20:54.879783 18 ldap.go:88] Getting details about user: CN=Smith\, Sean B,OU=End Users,OU=Accounts,DC=nw,DC=nos,DC=example,DC=com
2017/04/14 20:20:54 D0414 20:20:54.879814 18 ldap.go:124] User DN: CN=Smith\, Sean B,OU=End Users,OU=Accounts,DC=nw,DC=nos,DC=example,DC=com
2017/04/14 20:20:54 D0414 20:20:54.879901 18 ldap.go:132] CN unescaped: CN=Smith\, Sean B
2017/04/14 20:20:54 D0414 20:20:54.879922 18 ldap.go:135] CN escaped: CN=Smith, Sean B
2017/04/14 20:20:54 D0414 20:20:54.879935 18 ldap.go:137] Searching for user: (CN=Smith, Sean B)
2017/04/14 20:20:55 D0414 20:20:55.007459 18 ldap.go:88] Getting details about user: CN=Bar\, James I,OU=End Users,OU=Accounts,DC=nw,DC=nos,DC=example,DC=com
2017/04/14 20:20:55 D0414 20:20:55.007489 18 ldap.go:124] User DN: CN=Bar\, James I,OU=End Users,OU=Accounts,DC=nw,DC=nos,DC=example,DC=com
2017/04/14 20:20:55 D0414 20:20:55.007576 18 ldap.go:132] CN unescaped: CN=Bar\, James I
2017/04/14 20:20:55 D0414 20:20:55.007601 18 ldap.go:135] CN escaped: CN=Bar, James I
2017/04/14 20:20:55 D0414 20:20:55.007782 18 ldap.go:137] Searching for user: (CN=Bar, James I)
2017/04/14 20:20:55 D0414 20:20:55.096269 18 users.go:63] LdapUsers: [{CN=Smith\, Sean B,OU=End Users,OU=Accounts,DC=nw,DC=nos,DC=example,DC=com zk174c [email protected]} {CN=Bar\, James I,OU=End Users,OU=Accounts,DC=nw,DC=nos,DC=example,DC=com ks784b [email protected]}]
2017/04/14 20:20:55 D0414 20:20:55.096308 18 users.go:118] User[[email protected]] not found in: map[[email protected]:42caa51a-756d-4783-9f18-8c9a73a9749b]

There seems to be a mismatch between the user id in ldap and uaa.

Feature Request: Dry run option

I'd like a command line option to dry run any of the create/update commands, so I can see what changes would be made without actually doing them.

Extending space and org defaults to include all options

Issue

When doing export-config or add-org-to-config or add-space-to-config it would be useful to be able to have defaults set for fields such as "enable-remove-users" for every newly created orgConfig.yml or spaceConfig.yml. At the moment we use a custom bash script to check for these settings and change them (we almost always want them set to false).

Proposed Change

Allow orgs.yml (or a new orgDefaults.yml) and spaceDefaults.yml to act as templates for each, have fields to set the defaults for all newly created orgConfig.yml and spaceConfig.yml files.
For orgConfig.yml this would include fields :

• enable-remove-users
• enable-remove-private-domains
• enable-delete-spaces
• memory-limit
• instance-memory-limit
• total-routes
• total-services
• paid-service-plans-allowed
• total_private_domains

For spaceConfig.yml this would include fields:

• allow-ssh
• enable-space-quota
• memory-limit
• instance-memory-limit
• total-routes
• total-services
• paid-service-plans-allowed
• enable-security-group
• enable-remove-users
• total_private_domains
• total_reserved_route_ports
• total_service_keys
• app_instance_limit

BUG: cf-mgmt-config-osx add-space exception. Workaround Exists.

I can't use cf-mgmt-config-osx to add a space to an org - it barfs:

./cf-mgmt-config-osx add-space  --org=new-test2 --space=s1 --enable-security-group=false  
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x118f295]

goroutine 1 [running]:
github.com/pivotalservices/cf-mgmt/configcommands.(*AddSpaceToConfigurationCommand).Execute(0x13021a0, 0xc420056780, 0x0, 0x4, 0x13021a0, 0x1)
	/tmp/build/80754af9/go/src/github.com/pivotalservices/cf-mgmt/configcommands/add_space.go:31 +0xb5
github.com/pivotalservices/cf-mgmt/vendor/github.com/jessevdk/go-flags.(*Parser).ParseArgs(0xc42005a480, 0xc42000e150, 0x4, 0x4, 0xc4200cc000, 0xc42005a480, 0x7fff5fbffac2, 0xc420078e50, 0xc420043f50)
	/tmp/build/80754af9/go/src/github.com/pivotalservices/cf-mgmt/vendor/github.com/jessevdk/go-flags/parser.go:316 +0x841
github.com/pivotalservices/cf-mgmt/vendor/github.com/jessevdk/go-flags.(*Parser).Parse(0xc42005a480, 0x1301f60, 0x2, 0xc42005a480, 0x0, 0xc420078e70)
	/tmp/build/80754af9/go/src/github.com/pivotalservices/cf-mgmt/vendor/github.com/jessevdk/go-flags/parser.go:186 +0x71
main.main()
	/tmp/build/80754af9/go/src/github.com/pivotalservices/cf-mgmt/cmd/cf-mgmt-config/main.go:15 +0x79

I can still use the old cf-mgmt command so there is a workaround ...

./cf-mgmt-osx add-space-to-config  --org new-test2 --space s1                                                                                                                                                                                1 ↵
2017/11/27 11:08:48 W1127 11:08:48.881566 26445 add_space.go:19] This command has been deprecated use lastest cf-mgmt-config cli
2017/11/27 11:08:48 I1127 11:08:48.881958 26445 yaml_config.go:317] Adding space: s1 

LDAP user search should be recursive

Hey Caleb! :)

We're having an issue with user lookup portion when updating spaces with ldap.

The tool finds the groups and gets a list of users in the groups fine, but it fails to look up that user if the userSearchBase property is not set directly to the node that they live in. We have users divided between "employee", "contractor" and "service account" OUs underneath a common "users" root. If userSearchBase is set directly to one of those sub trees everything works - for that set of users only. If set to the root it doesn't find any of them.

EOF error when connecting to ldap via SSL port

2016/12/15 18:25:40 E1215 18:25:40.850527 12 ldap.go:166] unable to read LDAP response packet: unexpected EOF
unable to read LDAP response packet: unexpected EOF

Resources Removal

Right now only positive changes are supported. Still it is not so comfortable for end users. We might think on the way how to clean up environment. I suppose it should be done in 2 steps:

1. print all resources that are going to be removed
2. remove listed resources after operator confirmation

I am going to find out the way to do it during following weeks. What do you think?

Provide a template for attaching default LDAP groups to org/space configuration

There are situations where users of cf-mgmt wants to default their org and space roles to a default LDAP groups so that they don't have to go and update the org and space config every time a new org or space is created. A possible solution would be to accept a template config file for org or spaces configurations.

Allow specify the ldap bind pwd via externalized variable vs ldap.yml

Less verbose / configurable verbosity of cf-mgmt output

Issue

At the moment, output is quite verbose and makes it difficult to see what is happening at a glance if there is a large amount of orgs/spaces being acted on. Particularly with the update-org-users and update-space-users.

Proposed change

To me, it makes more sense to reduce the amount of output to exclude messages such as:

Not removing users. Set enable-remove-users: true to orgConfig for org: {ORG_NAME}

It seems unnecessary to print this full message for every org / space. This could be changed to something smaller like :

{ORG_NAME} - No changes

Or even just output the diff of what changes are actually being made in that specific job. So the only output is :

{SPACE_NAME} added to {ORG_NAME}

ie. Only additive.

sharePrivateDomains doesn't work

@calebwashburn - the share private domains functionality does not work. I've already found the root cause of the problem and will work on another PR to fix. (This one won't take as long - I had a Pivotal Dojo in the middle of the last go-round and it delayed my testing for a while)

In orgs.go, CreatePrivateDomains and SharePrivateDomains both rely on cloudcontroller.go.ListAllPrivateDomains. One needs the domain's owning orgGUID to make sure private domains are created safely. The other needs the domain's GUID to be able to make calls to Associate Private Domain with the Organization.

I'm trying to decide if I should split ListAllPrivateDomains into two methods or if I should send in some sort of argument to allow it to return the orgGUID or domainGUID as its map value.

Your thoughts / suggestions?

Remove role for ldap user

can you confirm if the delete (remove user role) functionality will work for individual ldap users?
This seems not added
we understand that it works for ldap groups

add buildpack management to `cf-mgmt` features

Hello here,

we would like to work on a PR to add support for the buildpacks management.

Our particular use case is to de-couple CF updates from buildpack updates, so that we could enforce environment consistency across different CF deployments when using CI/CD tools.

The main idea would be to add a new configuration file, ie. called buildpacks.yml to the main config folder.
We haven't thought thoroughly about the format yet, and we are ofc open to suggestions, however we were thinking of the following syntax:

buildpacks:
  - name: golang_v1.8.6
    position: 10
    enabled: true
    locked: true
    path: https://github.com/cloudfoundry/go-buildpack/releases/download/v1.8.6/go-buildpack-v1.8.6.zip
  - name: python_v1.5.24
    position: 11
    enabled: true
    locked: true
    path: https://github.com/cloudfoundry/python-buildpack/releases/download/v1.5.24/python-buildpack-v1.5.24.zip

let us know if you support the idea or if you would mind adding such a feature ;)

Many thanks,
Claudio

cf-mgmt needs to support explicit user email addresses in addition to LDAP groups

Sometimes operators do not know the exact mapping of LDAP groups to CF space and org roles. Allow for explicit configuration of user addresses

Delete spaces job stops after first falsey check

I have a hand full of teams that would like to opt-in for automatic space deletion, but we only want to enable just those teams' cf orgs.

The problem we are seeing is that the delete-spaces job stops upon the first falsey check of the flag in the spaces.yml (either the flag doesn't exist or it does and it is not set to true). Thus it doesn't check for other orgs that may have enable-delete-spaces: true in their spaces.yml.

Here's the message I see once it hits a falsey check:
2017/07/19 00:14:12 I0719 00:14:12.927462 13 spaces.go:399] Space deletion is not enabled. Set enable-delete-spaces: true in spaces.yml

This is also the last message before the job stops.

Any help would be greatly appreciated!

Command "cf-mgmt-config add-space" exited with a runtime error.

Hi, Caleb,

We use the latest cf-mgmt-config to run the below command:

cf-mgmt-config.exe add-space --config-dir TestConfigDir --org testorg1 --space dev1

The utility exited with the below runtime error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x60 pc=0x59f4ec]

goroutine 1 [running]:
github.com/pivotalservices/cf-mgmt/configcommands.(*AddSpaceToConfigurationComma
nd).Execute(0x711fe0, 0xc0420e6540, 0x0, 0x7, 0x711fe0, 0x1)
/tmp/build/80754af9/go/src/github.com/pivotalservices/cf-mgmt/configcommands/add_space.go:31 +0xbc
github.com/pivotalservices/cf-mgmt/vendor/github.com/jessevdk/go-flags.(*Parser).ParseArgs(0xc042038360, 0xc042062010, 0x7, 0x7, 0xc0420b2000, 0xc042038360, 0xc
042008480, 0xc04206aaf0, 0xc04205df50)
/tmp/build/80754af9/go/src/github.com/pivotalservices/cf-mgmt/vendor/git
hub.com/jessevdk/go-flags/parser.go:316 +0x833
github.com/pivotalservices/cf-mgmt/vendor/github.com/jessevdk/go-flags.(*Parser)
.Parse(0xc042038360, 0x711da0, 0x2, 0xc042038360, 0x0, 0xc04206ab10)
/tmp/build/80754af9/go/src/github.com/pivotalservices/cf-mgmt/vendor/github.com/jessevdk/go-flags/parser.go:186 +0x78
main.main()
/tmp/build/80754af9/go/src/github.com/pivotalservices/cf-mgmt/cmd/cf-mgm
t-config/main.go:15 +0x80

Could you please investigate and fix?

Thanks!

Joe

consider support for renaming orgs and spaces

Cloud Foundry offers the "cf rename-org" and "cf rename-space" commands.
However, if I rename an org or space in the git repo for cf-mgmt (and run the pipeline), it simply deletes the old org/space and creates a new one.
There you loose everything (apps, services and so on) in your old spaces.

It would be nice that a renamed org/space would result in the new names with still the "old stuff" in those org/space.

cheers,
Harry

Shared Private Domains

Hi!

I'm curious if you do or plan to support shared private domains? I'm using the private domains feature but I've also shared my private domains between a few different orgs. Do you have a plan to support the sharing of private domains?

Users in the format "LastName, Firstname [Division]" are not added to PCF

When using LDAP group membership to add a user with a format of "Lastname, Firstname [Division]" the user search is failing. This is preventing the creation of user accounts via cf-mgmt. It appears the user is found but LDAP is having issues with the ","

ldap.yml config

userSearchBase: DC=some,DC=company,DC=com
userNameAttribute: sAMAccountName
userMailAttribute: mail
groupSearchBase: OU=Groups,DC=some,DC=company,DC=com
groupAttribute: member

The log shows the user was found as a member of the group but is failing on the lookup of the user

2016/11/22 17:00:16 I1122 17:00:16.421913 549 ldap.go:136] Searching for group: (cn=Group Name)
2016/11/22 17:00:16 I1122 17:00:16.42372 549 ldap.go:109] Searching for user: (CN=Lastname\5c, Firstname [Division])
2016/11/22 17:00:16 I1122 17:00:16.424265 549 ldap.go:68] User entry not found CN=Lastname\, Firstname [division],OU=Development,OU=Engineering,OU=Users,OU=Something,DC=some,DC=company,DC=com

The user in LDAP looks like this when output via the powershell command get-aduser

DistinguishedName : CN=Lastname\, Firstname [Division],OU=Development,OU=Engineering,OU=Users,OU=Something,DC=some,DC=company,DC=com
Enabled           : True    
GivenName         : Firstname
Name              :Lastname, Firstname [Division]
ObjectClass       : user
ObjectGUID        : xxxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxx
SamAccountName    : Firstname.Lastname
SID               : x-x-x-xx-xxxxxxxxxxxxx-xxxxxxxxx-xxxxxxxxxx-xxxxx
Surname           : Lastname
UserPrincipalName : [email protected]

Any help in getting around this error is appreciated.

The corresponding ASG json file is not checked for existence when a named ASG is added

When the below command is run, the utility does not validate that file config/asgs/test-asg.json exists.

cf-mgmt-config.exe update-space --org testorg --space dev --named-asg test-asg

I simply adds the named asg to spaceConfig.yml.

spaceDefaults.yml is not processed when running cf-mgmt add-space-to-config

We are using cf-mgmt version 0.0.66 and try to add spaces to the config and have them populated with values from spaceDefaults.yml.
But it looks like it does not even read the spaceDefaults.yml, consider the following scenario:

metskem@athena:~/workspace/pcf-d01-orgs-spaces$ stat config/spaceDefaults.yml 
  File: config/spaceDefaults.yml
  Size: 256             Blocks: 24         IO Block: 4096   regular file
Device: 2dh/45d Inode: 4065948     Links: 1
Access: (0755/-rwxr-xr-x)  Uid: ( 1000/ metskem)   Gid: ( 1000/ metskem)
Access: 2017-10-24 10:17:07.275836901 +0200
Modify: 2017-10-24 09:50:51.427342167 +0200
Change: 2017-10-24 10:12:29.850412715 +0200
 Birth: -
metskem@athena:~/workspace/pcf-d01-orgs-spaces$ cf-mgmt add-org-to-config --org testorg
2017/10/24 10:18:45 I1024 10:18:45.315609 3944 config.go:93] Adding org: testorg 
metskem@athena:~/workspace/pcf-d01-orgs-spaces$ cf-mgmt add-space-to-config --org testorg --space testspace
2017/10/24 10:18:48 I1024 10:18:48.796558 3949 config.go:149] Adding space: testspace 
metskem@athena:~/workspace/pcf-d01-orgs-spaces$ cat config/spaceDefaults.yml       
space-developer:
  ldap_users: []
  users:
  - userA
  - userB
  ldap_group: ""
  ldap_groups: []
space-manager:
  ldap_users: []
  users: []
  ldap_group: ""
  ldap_groups: []
space-auditor:
  ldap_users: []
  users: []
  ldap_group: ""
  ldap_groups: []
metskem@athena:~/workspace/pcf-d01-orgs-spaces$ cat config/testorg/testspace/spaceConfig.yml 
org: testorg
space: testspace
space-developer:
  ldap_users: []
  users: []
  ldap_group: ""
  ldap_groups: []
  saml_users: []
space-manager:
  ldap_users: []
  users: []
  ldap_group: ""
  ldap_groups: []
  saml_users: []
space-auditor:
  ldap_users: []
  users: []
  ldap_group: ""
  ldap_groups: []
  saml_users: []
allow-ssh: false
enable-space-quota: false
memory-limit: 10240
instance-memory-limit: -1
total-routes: 1000
total-services: -1
paid-service-plans-allowed: false
enable-security-group: false
enable-remove-users: true
total_private_domains: 0
total_reserved_route_ports: 0
total_service_keys: 0
app_instance_limit: 0
metskem@athena:~/workspace/pcf-d01-orgs-spaces$ stat config/spaceDefaults.yml 
  File: config/spaceDefaults.yml
  Size: 256             Blocks: 24         IO Block: 4096   regular file
Device: 2dh/45d Inode: 4065948     Links: 1
Access: (0755/-rwxr-xr-x)  Uid: ( 1000/ metskem)   Gid: ( 1000/ metskem)
Access: 2017-10-24 10:17:07.275836901 +0200
Modify: 2017-10-24 09:50:51.427342167 +0200
Change: 2017-10-24 10:12:29.850412715 +0200
 Birth: -

The users userA and userB are not added, and looking at the Access time of spaceDefaults.yml it looks like is not even opened for input.
I must be doing something wrong here?

Does not traverse paginated results

Ran into an issue where cf-mgmt would create an org, then wasn't able to find it after. After doing some digging, we found that the call to retrieve the orgs was not traversing multiple pages. Since the org it just created is not on the first page, it assumes that it doesn't exist and fails when it gets the error that the name is still in use.

The named ASG is not checked that it already exists in spaceConfig.yml when it is added.

Hi,

The command to run is:

cf-mgmt-config.exe update-space --org testorg --space dev --named-asg test-asg

If spaceConfig.yml already has test-asg configured, cf-mgmt-config still adds it. The result is below when the above command is run twice:

named-security-groups:

• test-asg
• test-asg

Could you please not add when the named asg already exists and return an error msg like the exception handling of adding users?

Thanks!

Joe

add support for (named) quotas, being referenced from spaceConfigs and orgConfigs

If you currently want to work with quotas, you have to define that in the spaceConfig.yml and orgConfig.yml. And you have to specify this as many time as you have orgs and spaces. You will also end up as many quotas as you have orgs and spaces.

If you (plan to) have many spaces (as we do), then I would think it would be better to have the option of specifying/defining quotas in separate files, and then in spaceConfig.yml and orgConfig.yml refer to those quotas by name.

So, for example, you define :

orgquotas/org-quota-small.yml
orgquotas/org-quota-medium.yml
orgquotas/org-quota-large.yml
spacequotas/space-quota-small.yml
spacequotas/space-quota-medium.yml
spacequotas/space-quota-large.yml

And these files hold the quota values similar to what is currently in the spaceConfig.yml and orgConfig.yml.

Then you should be able to specify (in spaceConfig.yml and orgConfig.yml) that you have an "external quota config" and specify the name of one of the files (i.e. space-quota-large.yml).
To not make it all too complex, this should be mutually exclusive with the current way of quota-config.

Documentation Cleanup: ERT UAA User vs ERT UAA Client

Referring to these two sections (and probably others):

• https://github.com/pivotalservices/cf-mgmt/blob/master/docs/README.md#the-following-operation-are-enabled-with-cf-mgmt-that-will-leverage-configuration-to-modify-your-cloud-foundry-installation
• https://github.com/pivotalservices/cf-mgmt#authentication-requirements

The documentation seems to be unclear about whether you need an ERT UAA client or an ERT UAA user. UAAC has the ability to create both. Does cf-mgmt now only require a client for user, org, space, etc creation and updates?

I'm currently passing an ERT user and a UAA client to my cf-mgmt calls.

cf create-user for adding users outside ldap

list of users (must already be a user created via cf create-user)

can we get this feature added so that this need not be done outside the tool for space users and org u

ldap groups

ldap groups are updated in spaceConfig.yml only when a new space is created
When we add an ldap group to an existing space , it wouldnt update spaceConfig.yml
Is this expected?

Use cloud controller pagination APIs to list all available orgs

Currently the app limits how many orgs can be displayed using a hard coded limit of 100. We should change that to use pagination API provided by cloud controller to list all the orgs.

Getting "Given client ID does not match authenticated client"

2017/10/04 10:16:43 E1004 10:16:43.328759 89978 main.go:600] Unable to initialize cf-mgmt. Error : cannot get UAAC token, error 401: {"error":"invalid_client","error_description":"Given client ID does not match authenticated client"}
cannot get UAAC token, error 401: {"error":"invalid_client","error_description":"Given client ID does not match authenticated client"}

Any ideas?

When I login with uaac token client get id -s secret, I get back Successfully fetched token via client credentials grant.. Therefore, both id and secret must be ok.

There is no support for global security groups

Hi, Caleb,
The current release of cf-mgmt does not have support for global security groups. These security groups are staging security groups and running security groups in Cloud Foundry configuration. They are used by all spaces. Could you please consider supporting them? It would be good if the json files of these global security groups are in a different folder than in the existing asgs folder.

Thanks!

Joe

Configuring Private Domains for Orgs

Is there a way to declare the desired private domains for an org? It doesn't appear so and was curious if I'm missing it somewhere.

Thanks.

Quotas for orgs and spaces with the same names

Hey, all.

If you enable quotas for org or space cf-mgmt creates quota with the name of correspondent org or space.

The question is what if we have several spaces with the same name, but different quotas? I think that quotas names should be uniq for all resources.

Escaped characters not being filtered correctly

It seems the Go ldap library you are using does not properly managed escaped characters in filters. This causes a problem when there are non-supported characters in the group or user name. In my case it was a dash and some parentheses that caused ldap to return an empty result set or an error. See this gist for failure examples.
There is another ldap library which comes from the same parent that's able to handle the filter escaping here.
As we are usually unable to handle the naming conventions, can we either add the proper escaping filters into this repo, or make a pull request to the downstream library.

Check if a user id exists before adding it

Hey @calebwashburn - cf-mgmt used to check if a given user id already existed in ldap_users before adding it to the .yml file.
In the latest version (approximately release 11/1/17) cf-mgmt-config will add the given user id.
This means multiple instances of the same user id can be added to ldap_users.

For example:

org: funorg
org-billingmanager:
  ldap_users: []
  users: []
  saml_users: []
  ldap_groups: []
org-manager:
  ldap_users:
  - testuser
  - testuser
  - testuser
  users: []
  saml_users: []
  ldap_groups: []
org-auditor:
  ldap_users: []
  users: []
  saml_users: []
  ldap_groups: []

User "testuser" is added multiple times.

Thanks

Adding individual ldap users to existing space

Hi

If we update spaceConfig.yml for an existing space, with additional users , the pipeline doesnt pick these users and add to the foundation
For a new space it works fine

Am I missing something?