cyberark/cloudfoundry-conjur-buildpack

Buildpack for the Conjur / Cloud Foundry integration

cloudfoundry-conjur-buildpack's Issues

Users can determine buildpack version from Cloud Foundry

Is your feature request related to a problem? Please describe.

When I have the Conjur Buildpack installed in my CF foundation, I want to be able to determine the version that's installed.

Describe the solution you would like

There is a simple, documented command to determine the version of a currently installed buildpack.

Describe alternatives you have considered

n/a

Additional context

n/a

Buildpack doesn't reference `meta-buildpack` in the upload script

Is your feature request related to a problem? Please describe.

In ./upload.sh, we use the meta-buildpack name as the POSITION value for cf create-buildpack, where the parameters are cf create-buildpack BUILDPACK PATH POSITION [--enable|--disable].

Describe the solution you would like

We should update our process to use supply build packs, as per the documentation here: https://github.com/cf-platform-eng/meta-buildpack.

NOTE: Meta-buildback is being deprecated
Changes to the core CloudFoundry lifecycle process are making it hard to guarantee on-going compatibility with meta-buildpack and decorators. Some of the use cases for decorators can now be solved by leveraging the new [supply buildpack](https://docs.cloudfoundry.org/buildpacks/understand-buildpacks.html#supply-script) functionality. Some examples can be found at the bottom of this page.

If you are using meta-buildpack today and need to find an alternative, or have a use case that would have been addressed by decorators, please open an issue on this repo and we are happy to help you look for the right way to accomplish your task.

Undocumented hard requirement for language buildpack

Summary

If you do not explicitly include the language build pack in your application manifest (because it's automatically included by PCF), the conjur build pack will not invoke. This hard requirement is not documented.

Steps to Reproduce

In you application's manifest.yml, only include the conjur build pack
Stage your app
Observe that the conjur build pack does not get invoked

Expected Results

Conjur build pack should be invoked

Actual Results (including error logs, if applicable)

Conjur build pack is not invoked

Reproducible

• [*] Always
• Sometimes
• Non-Reproducible

Version/Tag number

cf cli version 6.51
DAP 11.5
PCF 2.9 (?)

test changing component

Online buildpack failing with error "does not support multi-buildpack apps"

Summary

Using the online buildpack fails with the error:

 ERR Error: one of the buildpacks chosen to supply dependencies does not support multi-buildpack apps

Steps to Reproduce

1. Clone this project and change to the tests/integration/apps/ruby directory.

2. Add a file manifest.yml which contains

   ---
   applications:
   - name: ruby-app
     env:
       CONJUR_BUILDPACK_BYPASS_SERVICE_CHECK: true
     buildpacks:
     - https://github.com/cyberark/cloudfoundry-conjur-buildpack
     - ruby_buildpack
   

3. cf push from the Ruby app directory

Expected Results

The app should successfully push and when you visit the app in the browser you should see:

      <h1>Visit us @ www.conjur.org!</h1>

      <h3>Space-wide Secrets</h3>
      <p>Database Username: space_username</p>
      <p>Database Password: space_password</p>

Actual Results (including error logs, if applicable)

The app fails to be staged on cf push:

$ cf push
Pushing from manifest to org geri-org / space geri-space as admin...
Using manifest file /Users/gjennings/Documents/repos/pcf/buildpack/tests/integration/apps/ruby/manifest.yml
Getting app info...
Creating app with these attributes...
+ name:         ruby-app
  path:         /Users/gjennings/Documents/repos/pcf/buildpack/tests/integration/apps/ruby
  buildpacks:
+   https://github.com/cyberark/cloudfoundry-conjur-buildpack
+   ruby_buildpack
  env:
+   CONJUR_BUILDPACK_BYPASS_SERVICE_CHECK
  routes:
+   ruby-app.apps.pcf.itd.conjur.net

Creating app ruby-app...
Mapping routes...
Comparing local files to remote cache...
Packaging files to upload...
Uploading files...
 2.19 KiB / 2.19 KiB [==============================================] 100.00% 1s

Waiting for API to complete processing files...

Staging app and tracing logs...
   Downloading ruby_buildpack...
   Downloaded ruby_buildpack
   Cell f285e943-6329-44c3-a703-4c6cf25cf19b creating container for instance cff4f7e6-9269-4a54-b07e-5db1278d7dd0
   Cell f285e943-6329-44c3-a703-4c6cf25cf19b successfully created container for instance cff4f7e6-9269-4a54-b07e-5db1278d7dd0
   Downloading app package...
   Downloaded app package (2.2K)
Error staging application: Staging error: staging failed
FAILED

When you look at the app logs, you see:

ruby$ cf logs ruby-app
Retrieving logs for app ruby-app in org geri-org / space geri-space as admin...

   2020-12-21T10:50:45.14-0500 [STG/0] OUT Downloaded app package (2.2K)
   2020-12-21T10:50:45.68-0500 [STG/0] ERR Error: one of the buildpacks chosen to supply dependencies does not support multi-buildpack apps
   2020-12-21T10:50:45.70-0500 [STG/0] OUT Exit status 225
   2020-12-21T10:50:45.83-0500 [STG/0] OUT Cell f285e943-6329-44c3-a703-4c6cf25cf19b stopping instance cff4f7e6-9269-4a54-b07e-5db1278d7dd0
   2020-12-21T10:50:45.83-0500 [STG/0] OUT Cell f285e943-6329-44c3-a703-4c6cf25cf19b destroying container for instance cff4f7e6-9269-4a54-b07e-5db1278d7dd0
   2020-12-21T10:50:45.93-0500 [API/0] ERR Failed to stage build: staging failed
   2020-12-21T10:50:46.12-0500 [STG/0] OUT Cell f285e943-6329-44c3-a703-4c6cf25cf19b successfully destroyed container for instance cff4f7e6-9269-4a54-b07e-5db1278d7dd0

In preparation for #101 there is a copy of the latest release in the latest branch; if you update the manifest to refer to https://github.com/cyberark/cloudfoundry-conjur-buildpack#latest, it works as expected.

Reproducible

• Always
• Sometimes
• Non-Reproducible

Version/Tag number

The online buildpack uses the latest code on master.

Environment setup

This test case was run in TAS 2.10, but should function the same on other earlier TAS versions.

Additional Information

Add any other context about the problem here.

Bump Go version to 1.12

AC:

• Bump Go version for conjur-env to 1.12
• Convert from using mitchellh/homedir to os.UserHomeDir

There is a new release of the buildpack in preparation for a new TAS tile (2)

Items to include in the release:

• Deprecate support for Conjur Enterprise v4

Adding windows stack support

Overview

Cloud Foundry has support for running Windows workloads. There is the ability to have buildpacks that support both the cflinuxfs* and windows, but the compatibility for both needs to be maintained by the buildpack owner.

Proposed Solution

• Buildpacks that support Windows need to have an equivalent Windows Batch script for compile and supply (supply.bat, compile.bat)
• The 0001_retrieve-secrets.sh for exporting secrets as environment variables would need to be ported to a Windows Batch script as well
• The Go executable would also need to be compile for the Windows OS

Additional Considerations

• The Windows supported buildpack could either be in this repository and an entirely new repository
• Developers could use Conjur SDK's to retrieve secrets programmatically rather than relying on the buildpack. The Windows buildpack primarily serves .NET Framework users so there would need to be a .NET framework SDK.

Proof-of-Concept

• I have created a POC demonstrating how this buildpack would work on a Windows stack: Commit here
• This POC is functional, but still needs some enhancements.
• We may have a need for this in the future and I am looking to assess the possibility of this becoming a supported feature of this buildpack.

Secrets retrieval script not sourced when used with Java Buildpack

The Conjur supply buildpack expects the final buildpack to copy the scripts from the buildpack's staging directory to the app droplet's /home/vcap/profile.d/ directory.

This doesn't appear to work correctly when the Java Buildpack is the final buildpack for an application.

Temporary Workaround

An application may workaround this issue by sourcing the script retrieval directly from their application using a .profile in the apps root directory:

. /home/vcap/deps/0/profile.d/0001_retrieve-secrets.sh

NOTE: The 0 in the path above is the index of the conjur_buildpack in the buildpacks list. This example assumes the conjur_buildpack is first, but should be adjusted to match your specific manifest.

References

• This appears to be a known issue in the Java Buildpack, and is not intended to be address in the current generation of the Java Buildpack: cloudfoundry/java-buildpack#695 (comment)

  NOTE: The next generation of the buildpack contract (V3) will handle profile sourcing natively, and we'll not require the final buildpack to handle it.

secrets.yml location is configurable

As an application developer,
I want to be able to configure where the buildpack looks for the secrets.yml file
So that I can fit the buildpack into my existing workflow/project layout

See #18

Buildpack has automated tests for common final buildpacks

Our buildpack automated CI should test the buildpack functionality with common final buildpacks. For example:

• Ruby (https://github.com/cloudfoundry/ruby-buildpack)
• Java (https://github.com/cloudfoundry/java-buildpack)
• Python (https://github.com/cloudfoundry/python-buildpack)

Use single point of truth for Go version

The online and offline buildpack variants each have a different method for specifying the golang version. These should be updated to share a single configuration source.

Acceptance Criteria

• Single location to specify major.minor Go version
• Valid file hashes for online build are obtained automatically

There is a v2.2.1 release of the buildpack

The release of the buildpack should include the fix from #120.

bin/compile script is not executable

The permissions on bin/compile should be changed to 755 (instead of 644)

Some customers are reporting issues using the buildpack at compile time that look like the following:

Staging app and tracing logs...
   Downloading conjur_buildpack...
   Downloaded conjur_buildpack
   Cell 16d219f2-bd22-4901-bc87-3ba078fb87b1 creating container for instance 931d1cf8-ac3d-4562-82e9-8e429266237a
   Cell 16d219f2-bd22-4901-bc87-3ba078fb87b1 successfully created container for instance 931d1cf8-ac3d-4562-82e9-8e429266237a
   Downloading app package...
   Downloaded app package (536B)
   Failed to compile droplet: Failed to compile droplet: fork/exec /tmp/buildpacks/95a89d506a853a68bafe7a16a15a332a/bin/compile: permission denied
   Exit status 223
   Cell 16d219f2-bd22-4901-bc87-3ba078fb87b1 stopping instance 931d1cf8-ac3d-4562-82e9-8e429266237a
   Cell 16d219f2-bd22-4901-bc87-3ba078fb87b1 destroying container for instance 931d1cf8-ac3d-4562-82e9-8e429266237a
   Cell 16d219f2-bd22-4901-bc87-3ba078fb87b1 successfully destroyed container for instance 931d1cf8-ac3d-4562-82e9-8e429266237a
Error staging application: App staging failed in the buildpack compile phase
FAILED

Error from 0001_retrieve-secrets

One of our users is getting this error when pushing their app.

2020-03-13T13:51:13.26-0500 [APP/PROC/WEB/0] OUT [cyberark-conjur-buildpack]: retrieving & injecting secrets
2020-03-13T13:51:13.26-0500 [APP/PROC/WEB/0] OUT ~ ~
2020-03-13T13:51:13.43-0500 [APP/PROC/WEB/0] ERR .profile.d/0001_retrieve-secrets.sh: line 44: export: `security.config.user.name=': not a valid identifier
2020-03-13T13:51:13.43-0500 [APP/PROC/WEB/0] ERR .profile.d/0001_retrieve-secrets.sh: Error on line 44
2020-03-13T13:51:13.45-0500 [APP/PROC/WEB/0] OUT Exit status 1
2020-03-13T13:51:13.45-0500 [CELL/SSHD/0] OUT Exit status 0

This is the matching line from their secrets.yml, where all of their entries follow the same format:

security.config.user.name: !var QA-CyberArkVault/PCF/DAP_PCF_RITM0283092/security_config_username/password

SECRETS_YAML_ENVIRONMENT injects no environment variables

Summary

I was using the environment functionality and my app failed to deploy.

Steps to Reproduce

Deployed java app with this secrets.yml

dv:
  CyberArk_user: 'cyberark-api-a'
  CyberArk_pass: 'test
qa:
  CyberArk_user: 'cyberark-api-a'
  CyberArk_pass: 'test'
st:
  CyberArk_user: 'cyberark-api-a'
  CyberArk_pass: 'test'
pr:
  CyberArk_user: !var AZ/real/var/username
  CyberArk_pass: !var AZ/real/var/password

And set in the manifest SECRETS_YAML_ENVIRONMENT=dv

app failed to start, couldn't find CyberArk_pass (logs attached).

Changed Secrets.yml to

  CyberArk_user: 'cyberark-api-a'
  CyberArk_pass: 'test'

And deployed no issues (logs attached).

I also tried setting the common section, still no luck.

Expected Results

I should have had the environment vars in section dv and common get set.

Actual Results (including error logs, if applicable)

They were not set properly, and logs don't show an rror.

Reproducible

• [ X] Always
• Sometimes
• Non-Reproducible

Version/Tag number

cyberark-conjur-buildpack-v1.2.1.zip

Environment setup

Container, PCF

Additional Information

NA
healthylogs.txt
failurelogs.txt

Online buildpack uses latest branch instead of master

At current, we use master in this project as a development branch. Since this is the case, and we want to make sure users of our online buildpack have a good, consistent user experience and don't inadvertently pull in-progress changes to their production code, we should have a latest branch that is a copy of the latest tag and recommend that users reference our online buildpack using that branch. In the CF docs, it notes that you can invoke online buildpack branches by referring to the buildpack with the pattern: cf push YOUR-APP -b https://github.com/REPOSITORY/BUILDPACK.git#BRANCH.

AC:

• Add a latest branch that is a copy of our last project tag
• Update CONTRIBUTING release guide to add requirements for copying code to latest branch (or build automation to do this on releases)
• Add an entry to the change log for this change
• Update README on master to recommend users refer to the latest branch when using the online version of this buildpack

Buildpack dependencies are updated

Review the buildpack dependencies and update to more modern versions as appropriate

You may consider incorporating #41 into these changes, if it's simple enough to do.

Buildpack has scripts to start a development environment

AC:

• The buildpack has bin/start and bin/stop scripts to stand up and tear down a development environment, following the golden rules as much as possible
• The dev scripts are documented in CONTRIBUTING and basic debugging / troubleshooting flows are documented there as well

Buildpack is packaged using Packager

Is your feature request related to a problem? Please describe.

Currently, we package buildpack with a set of bash scripts that utilize docker compose to produce a .zip file. We should update this process to use Package.

Describe the solution you would like

Packager is the official cloudfoundry method for packaging one's custom build pack. We should use this over manually producing a zip file.

Online Buildpack uses wrong golang version

#38 updated the golang version for the buildpack to 1.12, including a non-backward compatible API change for obtaining the home directory.

The online buildpack, however, is still installing go 1.11.

It looks like this wasn't caught before because of a bug in the online buildpack integration test: https://github.com/cyberark/cloudfoundry-conjur-buildpack/blob/master/ci/features/integration.feature#L32

conjur-env binary uses batch secret retrieval

The conjur-env binary that performs secret retrieval appears to be looping through the required secrets and retrieving them in separate requests. For apps with a lot of secrest, this could create a lot of connections to Conjur in rapid succession, which seems like it may be prone to failure.

The Conjur API has a batch secret retrieval endpoint that we should use instead.

Buildpack Upgrade Caused Apps to not be able to Restage

Summary

We couldn't restage ANY app in our nonprod PCF instance after the team performed a regular buildpack upgrade to version cyberark-conjur-buildpack-v1.2.0.zip, they were on version cyberark-conjur-buildpack-v1.1.1.zip (side note, we need to say what gets packaged in here, otherwise tough to know changes between versions.)

Reproducible

• [ X] Always

Version/Tag number

1.1 buildpack and 1.2 buildpack

Environment setup

in a container, PCF environment (version 2.9.14 of TAS). Using on-prem instance. the app in question also used the java buildpack, which is at version 4.32.1 (java buildpack offline). the app was springboot, 2.1.0.RELEASE.

Additional Information

We could really use some help.

Secrets.yml configuration can be supplied as an environment variable

Description

We support supplying the secrets.yml config for Summon as a literal string passed to the Summon command:
https://github.com/cyberark/summon/blob/434da01c1c7ecff7a58d614048ae167f91dd0425/internal/command/flags.go#L31-L34

In this issue, we would like to add support to the buildpack for specifying the secrets.yml config as a literal string in an environment variable. This allows developers to separate their Conjur configuration from their application artifacts, which will enable them to change their Conjur config without needing to rebuild the app. It was motivated in particular by Java applications, which absent this change require building the secrets.yml into the jar file.

The way this flow would work is that you'd define a SECRETS_YAML env var in your manifest, like:

---
applications:
- name: my-app
  services:
  - conjur
  buildpacks:
  - conjur_buildpack
  - ruby_buildpack
  env:
    SECRETS_YAML: |
      AWS_ACCESS_KEY_ID: !var aws/iam/user/robot/access_key_id
      AWS_SECRET_ACCESS_KEY: !var aws/iam/user/robot/secret_access_key

where the | character is used to allow multi-line SECRETS_YAML config with line breaks preserved, as per the yaml spec.

In practice, you could set this env var in the manifest as above or with the cf env command and run cf restage to restart the app with a new secrets config. This should be implemented so that if a secrets.yml file exists, it would override the env var; for example, our logic would be something like:

• check for SECRETS_YAML_PATH env var; use the file there if it exists. if not,
• check for secrets.yml file in standard locations; use the file if one exists. if not,
• check for SECRETS_YAML env var; use the content there, if it exists. if not,
• fail to apply buildpack.

Background

From #18 (comment) as reported by @mkkeffeler.

i think the issue we are having is related to this. #18

We have a manifest where we specify path: /path/to/jar.jar

this uploads the jar. the jar DOESNT contain secrets.yml. this fails, saying it can't find secrets.yml.

We update path to path:/path/to and this directory contains jar.jar, and secrets.yml (voila, conjur buildpack works, but PCF can't start the app for some reason, seemingly because it doesn't know what to do with the jar.

We don't want to have to put secrets.yml in our jar files because that means for any config change we have to rebuild our app, which is not consistent with the "build once deploy anywhere" strategy that we see people adopting across the industry.

Thoughts on this? reccomendations?

@mkkeffeler I left the bug template below in case you want to add any more info to this card using this format - it can be a helpful way to understand clearly how the product isn't meeting your expectations right now. We'll look into this and try to get back to you shortly. Please also let me know if the title doesn't accurately encapsulate your problem. Thanks!

Buildpack is converted to supply buildpack

Since the meta-buildpack will soon be deprecated, the Conjur Buildpack needs to be converted to a supply buildpack

The expected usage of this buildpack as a supply buildpack would be that the end users will specify multiple buildpacks on cf push, either by specifying as parameters to that command or in the manifest:

• cf push -b conjur_buildpack -b language_buildpack

•  buildpacks:
     - conjur_buildpack
     - language_buildpack
  

In both cases, the last buildpack in the list would be the language-specific buildpack required by their app (eg the final buildpack, or the one with a bin/finalize script).

Our documentation should continue to support using the original decorator buildpack by referring end-users to the 1.0.0 release in GitHub if they are unable to use multi-buildpack functionality.

AC:

• Update buildpack to convert to supply format (including verifying functionality using e2e demo)
• Update README documentation to describe updated buildpack format
  • The README should note that users who prefer to use the decorator buildpack version that utilizes meta-buildpack can use the 1.0.0 release of the buildpack.
• Cut a buildpack release
• Cut a tile release

Buildpack fails when multiple services are bound to an application

when the VCAP_SERVICES environment variable includes multiple services, and any of those service credentials have field name collisions with cyberark-conjur where the value type is different, the buildpack will fail with a JSON unmarshal error, such as:

Error parsing Conjur connection information: json: cannot unmarshal string into Go struct field ConjurCredentials.version of type int

There is a new release of the buildpack in preparation for a new TAS tile (1)

Buildpack v2.2.0 is released

There is a new release of the buildpack containing the enhancement in #44

We should revise the CHANGELOG message for this fix to be more clear about the value delivered to end users with this change.

Buildpack has automated tests against a live PCF installation

The buildpack has been updated to run tests against a live PCF installation, as we are already doing for the service broker.

Conjur Buildpack doesn't fail when retrieval of secrets fails

Summary

deployed an app that had conjur secrets, saw in logs it said it was injecting the vars, then failed due to conjur being down. Since the app certainly relies on conjur for environment variables we need the push to fail with an error or the app needs to fall to startup so that we are aware of this issue and don't have to dig in logs to findout.

Steps to Reproduce

Steps to reproduce the behavior:

1. push an app with conjur secrets.yml and buiildpack
2. check logs that it failed to retrieve creds due to conjur connectivity/down issue
3. see that app reports as up and started (no visible errors other than in logs)

Expected Results

app would fail to start/faill to push because it couldn't contact conjur

Actual Results (including error logs, if applicable)

app starts up without env vars that are required.

Reproducible

• [ X] Always
• Sometimes
• Non-Reproducible

Version/Tag number

conjur_buildpack 1 true false conjur_buildpack-2.1.4.zip
and a java app running springboot

Environment setup

PCF on premise

The secrets.yml check in the supply script uses different logic than the `conjur-env` binary to find the secrets.yml

Summary

This was encountered by @whip113 when he had two secrets.yml files in the app in different directories AND specified SECRETS_YAML_PATH, but gave an invalid path.

Steps to Reproduce

Set up an app with at least one secrets.yml file and set SECRETS_YAML_PATH to an invalid path for the file.

Expected Results

The buildpack will fail fast and let me know at the supply stage that the secrets.yml isn't available at the path as expected.

Actual Results (including error logs, if applicable)

The supply stage completes successfully, since all it does is check if a secrets.yml file exists anywhere in the app (that is, it uses different logic than the conjur-env binary will use when it is run):

cloudfoundry-conjur-buildpack/bin/supply

Lines 35 to 43 in 26874de

	# Search for candidate `secrets.yml` paths and report them to the deploy output
	secrets_yaml="$(find $BUILD_DIR -name 'secrets.yml' -printf \"%P\\n\")"
	if [ -z "$secrets_yaml" ]; then
	echo " - Unable to find a secrets.yml...exiting"
	exit 1
	else
	echo " - secrets.yml file found at:"
	echo $secrets_yaml
	fi

But the buildpack errors when the conjur-env binary is run, because the file does not exist at the specified SECRETS_YAML_PATH:

cloudfoundry-conjur-buildpack/conjur-env/main.go

Line 107 in 26874de

secrets, err = secretsyml.ParseFromFile(secretsYamlPath, "", nil)

Reproducible

• Always
• Sometimes
• Non-Reproducible

Version/Tag number

The latest version is impacted.

Environment setup

This is reproducible in TAS 2.9.

Additional Information

n/a

Add support for Summon environments

Summon supports an environment flag which allows you to specify different variables for different app environments (eg dev, prod, etc)

The secrets.yml syntax in that case would look something like this:

common:
  ACCOUNT: my-account
  SHARED_PASSWORD: SuperSecret

dev:
  CLUSTER_NAME: !var dev/cluster-name
  ZONE: !var dev/zone

prod:
  CLUSTER_NAME: !var prod/cluster-name
  ZONE: !var prod/zone

where the variables in common are loaded regardless of environment and the variables in dev / prod are only loaded when the environment specified is dev or prod (respectively).

The buildpack should support an environment variable (perhaps SUMMON_ENV_KEY_NAME as here?) that allows the user to specify the value of environment the buildpack should use to load the correct variables for the app env.

Fields in error messages misplaced when 0001_retrieve-secrets.sh is called/sourced from another script

Summary

If the 0001_retrieve-secrets.sh script is called or sourced from another script, and an error
occurs while executing the script (e.g. bad secrets.yaml file, or badly formatted output from
the conjur-env binary, then the fields of the resulting error message appears scrambled,
and the error message appears repeated, e.g.:

2021-04-03T08:32:603 .profile.d/0001_retrieve-secrets.sh: Error on line .profile: Unable to export `6`; value may not be a valid identifierfoobar: Error on line : Unable to export ``; value may not be a valid identifierExit status 1

Steps to Reproduce

Steps to reproduce the behavior:

1. Set up an invalid secrets.yaml file, e.g. one with a secret key that has a dash in the name.
2. Source or call the 0001_retrieve-secrets.sh script from another script.
3. You should see an error with field content in the wrong places in the error message (see example above).

Expected Results

Error message that are generated should make sense.

Actual Results (including error logs, if applicable)

Error message fields such as line number and secret key show up in the wrong places.

Reproducible

• Always
• Sometimes
• Non-Reproducible

Version/Tag number

2.2.0

Environment setup

Additional Information

Buildpack works correctly as an online buildpack

As a PCF developer
I want to include the Conjur Buildpack by referencing its Github URL

Example:
https://github.com/cloudfoundry/java-buildpack#usage

Buildpack is smarter about where to find secrets.yml

The buildpack is currently hardcoded to search the root directory for secrets.yml, and it errors if it doesn't find it.

The buildpack can be updated to accept an environment variable from the application's environment that configures the directory to search for secrets.yml in. In addition, it can be updated so that if the environment variable is not set, it searches in root and in other common directories (like BOOT-INF/classes/ for Java apps provided as JAR files - see below) used by applications for configuration YAML, so that the buildpack will work with applications in a variety of languages out of the box without requiring any specialized config.

AC:

• Buildpack is updated to accept an environment variable to configure the secrets.yml directory
• If secrets.yml directory is not set in the application environment, the buildpack is updated to look in BOOT-INF/classes/ as well as root for secrets.yml files. The ability to search directories for the secrets.yml file is written in such a way as to make it easy to add additional directories in future updates to the buildpack.
• Buildpack README is updated with information on configuring the secrets.yml file location
• Tile docs are updated with information on secrets.yml file location when using buildpack

Java apps

Java apps using the standard buildpack may not have the secrets.yml file in the root directory of the application in CF, breaking the expectations of the buildpack here.

We can update the buildpack to also check for the secrets.yml file in BOOT-INF/classes, which is where configuration files end up once built using Spring Boot v1.4+ if they are placed in src/main/resources pre-build (more info).

An example Java app manifest:

---
applications:
- name: sample-app
  memory: 512M
  instances: 1
  path: ./build/libs/sample-1.0.jar
  env:
    SPRING_PROFILES_ACTIVE: qa,cloud,default
  services:
  - sample-service

Master branch build is failing

See here. The error looks like:

[2021-02-22T17:57:34.476Z] go install: version is required when current directory is not in a module

[2021-02-22T17:57:34.476Z] 	Try 'go install github.com/cloudfoundry/libbuildpack/packager/buildpack-packager@latest' to install the latest version

[2021-02-22T17:57:34.476Z] The command '/bin/sh -c go install github.com/cloudfoundry/libbuildpack/packager/buildpack-packager' returned a non-zero code: 1

[2021-02-22T17:57:34.476Z] Packaging the conjur-buildpack as a zip file...

[2021-02-22T17:57:34.476Z] Unable to find image 'packager:latest' locally

[2021-02-22T17:57:34.731Z] docker: Error response from daemon: pull access denied for packager, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.

Buildpack uses `profile.d` directory

There are couple of buildpack issues to address after incorporating the buildpack in the Conjur service broker demo:

• The supply script should be placed in the directory profile.d, rather than .profile.d

  cloudfoundry-conjur-buildpack/bin/supply

  Lines 30 to 38 in 7bccc30

  	# We add the lib/0001_retrieve-secrets.sh script to .profile.d so that it will
  	# be run automatically to retrieve secrets when the app starts.
  	mkdir -p .profile.d
  	cp ${BUILDPACK_DIR}/lib/0001_retrieve-secrets.sh ./.profile.d/0001_retrieve-secrets.sh
  	sed "s/__BUILDPACK_INDEX__/$INDEX_DIR/g" ./.profile.d/0001_retrieve-secrets.sh -i
  	# conjur-env reads a secrets.yml file and uses it to retrieve secrets from
  	# Conjur. We copy it to the dependency directory to make it accessible to the
  	# /.profile.d script. The /vendor subdirectory is just for convenience.

• The readme should use underscores in the buildpack names rather than dashes:

  cf push my-app -b conjur_buildpack -b final_buildpack
  

• The readme should include an example of using the buildpack in the manifest:

  ---
  applications:
  - name: my-app
    services:
    - conjur
    buildpacks:
    - conjur_buildpack
    - final_buildpack
  

Buildpack is sometimes not invoked when specified, even though secrets.yml exists

Summary

When I push a Ruby app that has a secrets.yml file in its root project dir
And I specify the conjur_buildpack and ruby_buildpack (in that order), either in the manifest using the buildpacks key or in the cf push command using the -b flag
Then only the ruby buildpack is invoked, and the conjur_buildpack does not run to inject secrets into the app env

Steps to Reproduce

To make this easy to reproduce given a TAS tile installed in a 2.7+ cluster, I've revised the CF demo here: conjurdemos/cloudfoundry-conjur-demo#60.

When you install the v1.1.1 TAS tile in a TAS 2.9 foundation and run the demo scripts, the app is published but the conjur_buildpack is never invoked, so the app doesn't have access to the secret it needs.

Expected Results

I expected the conjur_buildpack to be invoked as instructed, and for the secrets specified in secrets.yml to properly be injected into the app's runtime environment.

Actual Results (including error logs, if applicable)

The buildpack was not actually invoked.

App logs:

2020-07-20T09:04:53.821-04:00 [API/0] [OUT] Created app with guid e2a68495-9f1a-477e-a3df-37dd00412ba2
2020-07-20T09:04:55.252-04:00 [API/0] [OUT] Updated app with guid e2a68495-9f1a-477e-a3df-37dd00412ba2 ({"route"=>"db5125c0-e855-4f4f-b277-d81a0af43087", :verb=>"add", :relation=>"routes", :related_guid=>"db5125c0-e855-4f4f-b277-d81a0af43087"})
2020-07-20T09:04:55.875-04:00 [API/0] [OUT] Uploading bits for app with guid e2a68495-9f1a-477e-a3df-37dd00412ba2
2020-07-20T09:05:04.374-04:00 [API/0] [OUT] Creating build for app with guid e2a68495-9f1a-477e-a3df-37dd00412ba2
2020-07-20T09:05:04.459-04:00 [API/0] [OUT] Updated app with guid e2a68495-9f1a-477e-a3df-37dd00412ba2 ({"state"=>"STARTED"})
2020-07-20T09:05:04.472-04:00 [STG/0] [OUT] Downloading ruby_buildpack...
2020-07-20T09:05:04.478-04:00 [STG/0] [OUT] Downloaded ruby_buildpack
2020-07-20T09:05:04.481-04:00 [STG/0] [OUT] Cell c16cecde-87f8-4f99-8dac-2d7d9da7f6f7 creating container for instance bb015045-cf64-47f6-900b-d90af0db8f34
2020-07-20T09:05:05.398-04:00 [STG/0] [OUT] Cell c16cecde-87f8-4f99-8dac-2d7d9da7f6f7 successfully created container for instance bb015045-cf64-47f6-900b-d90af0db8f34
2020-07-20T09:05:06.624-04:00 [STG/0] [OUT] Downloading app package...
2020-07-20T09:05:06.682-04:00 [STG/0] [OUT] Downloaded app package (1.1K)
2020-07-20T09:05:07.714-04:00 [STG/0] [OUT] -----> Ruby Buildpack version 1.8.14
2020-07-20T09:05:07.717-04:00 [STG/0] [OUT] -----> Supplying Ruby
2020-07-20T09:05:07.720-04:00 [STG/0] [OUT] -----> Installing bundler 1.17.3
2020-07-20T09:05:07.721-04:00 [STG/0] [OUT] Copy [/tmp/buildpacks/1dcf70f9a3baff8ffa6ec1a45d2c16f8/dependencies/1b585b3a526373724c47492bbb271365/bundler-1.17.3-any-stack-b7502506.tgz]
2020-07-20T09:05:08.556-04:00 [STG/0] [OUT] -----> Installing ruby 2.7.0
2020-07-20T09:05:08.556-04:00 [STG/0] [OUT] Copy [/tmp/buildpacks/1dcf70f9a3baff8ffa6ec1a45d2c16f8/dependencies/371ca1ab9c2e9472ac99abeee817c875/ruby-2.7.0-linux-x64-cflinuxfs3-68cd2bae.tgz]
2020-07-20T09:05:16.009-04:00 [STG/0] [OUT] -----> Installing dependencies using bundler 1.17.3
2020-07-20T09:05:16.009-04:00 [STG/0] [OUT] Running: bundle install --jobs=4 --retry=4
2020-07-20T09:05:19.873-04:00 [STG/0] [OUT] Fetching gem metadata from https://rubygems.org/....
2020-07-20T09:05:19.917-04:00 [STG/0] [OUT] Resolving dependencies...
2020-07-20T09:05:19.940-04:00 [STG/0] [OUT] Using bundler 1.17.3
2020-07-20T09:05:19.943-04:00 [STG/0] [OUT] Fetching mustermann 1.0.3
2020-07-20T09:05:19.945-04:00 [STG/0] [OUT] Fetching rack 2.0.8
2020-07-20T09:05:19.947-04:00 [STG/0] [OUT] Fetching tilt 2.0.9
2020-07-20T09:05:20.071-04:00 [STG/0] [OUT] Installing mustermann 1.0.3
2020-07-20T09:05:20.073-04:00 [STG/0] [OUT] Installing tilt 2.0.9
2020-07-20T09:05:20.165-04:00 [STG/0] [OUT] Installing rack 2.0.8
2020-07-20T09:05:20.633-04:00 [STG/0] [OUT] Fetching rack-protection 2.0.5
2020-07-20T09:05:20.709-04:00 [STG/0] [OUT] Installing rack-protection 2.0.5
2020-07-20T09:05:20.770-04:00 [STG/0] [OUT] Fetching sinatra 2.0.5
2020-07-20T09:05:20.943-04:00 [STG/0] [OUT] Installing sinatra 2.0.5
2020-07-20T09:05:21.056-04:00 [STG/0] [OUT] Bundle complete! 2 Gemfile dependencies, 6 gems now installed.
2020-07-20T09:05:21.062-04:00 [STG/0] [OUT] Gems in the groups development and test were not installed.
2020-07-20T09:05:21.062-04:00 [STG/0] [OUT] Bundled gems are installed into `/tmp/contents549709963/deps/0/vendor_bundle/ruby/2.7.0`
2020-07-20T09:05:21.817-04:00 [STG/0] [OUT] -----> Regenerating bundler binstubs...
2020-07-20T09:05:23.069-04:00 [STG/0] [OUT] Cleaning up the bundler cache.
2020-07-20T09:05:25.535-04:00 [STG/0] [OUT] -----> Creating runtime environment
2020-07-20T09:05:25.667-04:00 [STG/0] [OUT] -----> Saving vendor_bundle to cache
2020-07-20T09:05:25.877-04:00 [STG/0] [OUT] -----> Finalizing Ruby
2020-07-20T09:05:26.132-04:00 [STG/0] [OUT] -----> Copy binaries to app/bin directory
2020-07-20T09:05:34.003-04:00 [STG/0] [OUT] Exit status 0
2020-07-20T09:05:34.003-04:00 [STG/0] [OUT] Uploading droplet, build artifacts cache...
2020-07-20T09:05:34.004-04:00 [STG/0] [OUT] Uploading droplet...
2020-07-20T09:05:34.004-04:00 [STG/0] [OUT] Uploading build artifacts cache...
2020-07-20T09:05:34.132-04:00 [STG/0] [OUT] Uploaded build artifacts cache (1.4M)
2020-07-20T09:05:34.478-04:00 [API/0] [OUT] Creating droplet for app with guid e2a68495-9f1a-477e-a3df-37dd00412ba2
2020-07-20T09:05:36.571-04:00 [STG/0] [OUT] Uploaded droplet (23.9M)
2020-07-20T09:05:36.579-04:00 [STG/0] [OUT] Uploading complete
2020-07-20T09:05:37.095-04:00 [STG/0] [OUT] Cell c16cecde-87f8-4f99-8dac-2d7d9da7f6f7 stopping instance bb015045-cf64-47f6-900b-d90af0db8f34
2020-07-20T09:05:37.095-04:00 [STG/0] [OUT] Cell c16cecde-87f8-4f99-8dac-2d7d9da7f6f7 destroying container for instance bb015045-cf64-47f6-900b-d90af0db8f34
2020-07-20T09:05:37.359-04:00 [CELL/0] [OUT] Cell c16cecde-87f8-4f99-8dac-2d7d9da7f6f7 creating container for instance 93bbec58-29ee-4a89-7835-c102
2020-07-20T09:05:38.060-04:00 [STG/0] [OUT] Cell c16cecde-87f8-4f99-8dac-2d7d9da7f6f7 successfully destroyed container for instance bb015045-cf64-47f6-900b-d90af0db8f34
2020-07-20T09:05:38.466-04:00 [CELL/0] [OUT] Cell c16cecde-87f8-4f99-8dac-2d7d9da7f6f7 successfully created container for instance 93bbec58-29ee-4a89-7835-c102
2020-07-20T09:05:39.342-04:00 [CELL/0] [OUT] Downloading droplet...
2020-07-20T09:05:41.602-04:00 [CELL/0] [OUT] Downloaded droplet (23.9M)
2020-07-20T09:05:42.033-04:00 [CELL/0] [OUT] Starting health monitoring of container
2020-07-20T09:05:46.456-04:00 [APP/PROC/WEB/0] [ERR] /home/vcap/deps/0/bundler/gems/bundler-1.17.3/lib/bundler/rubygems_integration.rb:200: warning: constant Gem::ConfigMap is deprecated
2020-07-20T09:05:46.948-04:00 [APP/PROC/WEB/0] [ERR] /home/vcap/deps/0/vendor_bundle/ruby/2.7.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1641: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
2020-07-20T09:05:46.948-04:00 [APP/PROC/WEB/0] [ERR] /home/vcap/deps/0/vendor_bundle/ruby/2.7.0/gems/mustermann-1.0.3/lib/mustermann.rb:62: warning: The called method `new' is defined here
2020-07-20T09:05:46.948-04:00 [APP/PROC/WEB/0] [ERR] /home/vcap/deps/0/vendor_bundle/ruby/2.7.0/gems/mustermann-1.0.3/lib/mustermann/pattern.rb:59: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
2020-07-20T09:05:46.949-04:00 [APP/PROC/WEB/0] [ERR] /home/vcap/deps/0/vendor_bundle/ruby/2.7.0/gems/mustermann-1.0.3/lib/mustermann/regular.rb:22: warning: The called method `initialize' is defined here
2020-07-20T09:05:46.951-04:00 [APP/PROC/WEB/0] [ERR] /home/vcap/deps/0/vendor_bundle/ruby/2.7.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1604: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
2020-07-20T09:05:46.951-04:00 [APP/PROC/WEB/0] [ERR] /home/vcap/deps/0/vendor_bundle/ruby/2.7.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1621: warning: The called method `compile!' is defined here
2020-07-20T09:05:46.951-04:00 [APP/PROC/WEB/0] [ERR] /home/vcap/deps/0/vendor_bundle/ruby/2.7.0/gems/mustermann-1.0.3/lib/mustermann/pattern.rb:59: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
2020-07-20T09:05:46.951-04:00 [APP/PROC/WEB/0] [ERR] /home/vcap/deps/0/vendor_bundle/ruby/2.7.0/gems/mustermann-1.0.3/lib/mustermann/regexp_based.rb:17: warning: The called method `initialize' is defined here
2020-07-20T09:05:47.019-04:00 [APP/PROC/WEB/0] [ERR] [2020-07-20 13:05:47] INFO WEBrick 1.6.0
2020-07-20T09:05:47.019-04:00 [APP/PROC/WEB/0] [ERR] [2020-07-20 13:05:47] INFO ruby 2.7.0 (2019-12-25) [x86_64-linux]
2020-07-20T09:05:47.020-04:00 [APP/PROC/WEB/0] [ERR] [2020-07-20 13:05:47] INFO WEBrick::HTTPServer#start: pid=12 port=8080
2020-07-20T09:05:48.632-04:00 [CELL/0] [OUT] Container became healthy

Reproducible

• Always
• Sometimes
• Non-Reproducible

I have not been able to run this successfully myself yet, but I am aware of others who have successfully run on TAS 2.8 - so I'm not sure that this bug is always present.

Version/Tag number

I've been able to reproduce this in TAS 2.9, and a customer reported a similar issue in TAS 2.7. Earlier versions may also be affected; we haven't seen the issue in TAS 2.4.

We've been testing with the v1.1.1 tile.

It would be interesting to also check whether the following configurations are impacted:

• v1.1.1 tile with manual upload of latest buildpack release, TAS 2.9
• v1.1.1 tile, TAS 2.4
• v1.1.1 tile, TAS 2.7 (customer reported this has the issue, but we have not yet reproduced)

Additional Information

Add any other context about the problem here.

Repo has bug and feature request issue templates

AC:
The following templates have been added to the repo, with appropriate component labels:

• Bug template
• Feature request template

Docs includes instructions for decorator buildpack

We may have users that need to use the old decorator buildpack functionality. Our docs should include instructions for doing just that.

#19 (comment)

Remove support for Conjur Enterprise v4

AC:

• Remove support for Conjur Enterprise v4 in code / tests
• Deprecate support for Conjur Enterprise v4 in the changelog

Buildpack dependencies are updated to latest versions

AC:

• Dependencies of the buildpack (including Go version) are updated to latest versions. May be completed together with #41.
• NOTICES.txt is updated

Add code coverage for cloudfoundry-conjur-buildpack

Add code coverage to this project.

Acceptance Criteria:

• Code coverage should be measured across all tests defined in this repository every time a build is run in any branch.
• Coverage report should include:
  • Line Coverage (including highlighting which lines are covered and which are not - not just giving a percentage)
  • Conditional Coverage
  • Optionally, other kinds of coverage such as Package, Class, File, or Method coverage
• Coverage report should be created in or converted to Cobertura format as part of build to allow for ingestion into coverage aggregation tools
• Coverage report should be archived in the Jenkins build using the cobertura command as shown at https://github.com/cyberark/conjur-api-ruby/blob/072d21e01e46382ee4d577e180c1269f8ff9f36e/Jenkinsfile#L27
• Coverage report should be uploaded to Code Climate with each build

standardised CHANGELOG exists, and is validated via pipeline

If the repo has a changelog that doesn't meet the standard, do try to change earlier entries to match the standard.
If the repo doesn't have a changelog use this as a starter:

# Changelog
All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).

## [Unreleased]

Acceptance criteria

• CHANGELOG.md exists and follows the KeepAChangeLog format. For reference consult https://github.com/cyberark/conjur/blob/master/CHANGELOG.md.
• Add parse-changelog validation step to Jenkinsfile. For reference consult https://github.com/cyberark/conjur/blob/master/Jenkinsfile#L27.

Buildpack has separate scripts for running unit and integration tests

AC:

• Buildpack has bin/test_unit for running unit tests and bin/test_integration for running lightweight integration tests
• Any tests that run against a remote TAS instance are isolated into their own test scripts, with the goal of eventually keeping only smoke tests in this repo and running full end-to-end tests when we build the tile.

Buildpack test suite is updated to test against v4 Conjur

The buildpack test suite should be updated so that it also runs against a v4 Conjur instance.