Allow API token to be pulled from SSM Secure String

[honeycombio]

Allow API token to be pulled from SSM Secure String

[vreynolds]

Thanks, that helps! I think this would be a reasonable enhancement. We can't make promises on our own timeline, but if you'd like to go ahead and open a PR, we'd be happy to get it over the finish line 😄

[vreynolds]

Hello @verygoodsoftwarenotvirus, thanks for the detailed write up!

It makes sense that you would want to pull the API key from SSM -- that's a pretty standard place to store such things.
Your specific pain point around Terraform is not entirely clear to me, can you help me understand what your ideal workflow would look like with the SecureString? (e.g. at which point does the plaintext API key come in -- would a human still need to manually populate that somewhere safe?)

As far as implementation goes, do you anticipate this change to incur additional lambda overhead? I'm not familiar with the SSM client, perhaps this is equivalent to opening a KMS session and decrypting the cipher blob, like the lambda does today.
Another thing to consider is backwards compatibility. We prefer to make non-breaking changes where possible, so we would probably want to support both encrypted blobs and SSM parameters.

[verygoodsoftwarenotvirus]

Your specific pain point around Terraform is not entirely clear to me, can you help me understand what your ideal workflow would look like with the SecureString? (e.g. at which point does the plaintext API key come in -- would a human still need to manually populate that somewhere safe?)

The plaintext API Key would have to be distributed to Terraform somehow, I've been doing so with variable sets in Terraform Cloud. Once there, you'd be able to have something like:

resource "aws_iam_role" "honeycomb_cloudwatch_logs" {
  name = "honeycomb-cloudwatch-logs-lambda-role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_ssm_parameter" "honeycomb_api_key" {
  name  = "HONEYCOMB_API_KEY"
  type  = "SecureString"
  value = var.HONEYCOMB_API_KEY
}

resource "aws_lambda_function" "cloudwatch_logs" {
  s3_bucket        = "honeycomb-integrations-us-east-1"
  s3_key           = "agentless-integrations-for-aws/LATEST/ingest-handlers.zip"
  function_name    = "honeycomb-cloudwatch-logs-integration"
  role             = aws_iam_role.honeycomb_cloudwatch_logs.arn
  handler          = "cloudwatch-handler"
  runtime          = "go1.x"
  memory_size      = "128"

  environment {
    variables = {
      ENVIRONMENT = "changeme"
      PARSER_TYPE = "json"
      # naming things is hard
      SSM_PARAMETER_NAME = aws_ssm_parameter.honeycomb_api_key.name
      DATASET = "cloudwatch-logs"
      SAMPLE_RATE = "1"
      TIME_FIELD_FORMAT = "%s(%L)?"
      TIME_FIELD_NAME = "start_time"
    }
  }
}

With the variable properly configured and the appropriate code changes made, I could hypothetically run the above Terraform code once from a brand-new AWS account and have a functioning log forwarder, with a key that is encrypted at rest. The "once" part is the thing I sort of care about in all this. :)

From there, the code would do something like:

func InitHoneycombFromEnvVars() error {
	sampleRate = 1
	if os.Getenv("SAMPLE_RATE") != "" {
		i, err := strconv.Atoi(os.Getenv("SAMPLE_RATE"))
		if err != nil {
			logrus.WithField("sample_rate", os.Getenv("SAMPLE_RATE")).
				Warn("Warning: unable to parse sample rate, falling back to 1.")
		}
		sampleRate = uint(i)
	}

	// If KMS_KEY_ID is supplied, we assume we're dealing with an encrypted key.
	kmsKeyID := os.Getenv("KMS_KEY_ID")
	if kmsKeyID != "" {
		encryptedWriteKey := os.Getenv("HONEYCOMB_WRITE_KEY")
		if encryptedWriteKey == "" {
			return fmt.Errorf("Value for KMS_KEY_ID but no value for HONEYCOMB_WRITE_KEY")
		} else {
			kmsSession := session.Must(session.NewSession(&aws.Config{
				Region: aws.String(os.Getenv("AWS_REGION")),
			}))

			config := &aws.Config{}
			svc := kms.New(kmsSession, config)
			cyphertext, err := base64.StdEncoding.DecodeString(encryptedWriteKey)
			if err != nil {
				logrus.WithError(err).
					Error("unable to decode ciphertext in write key")
				return fmt.Errorf("unable to decode ciphertext in write key")
			}
			resp, err := svc.Decrypt(&kms.DecryptInput{
				CiphertextBlob: cyphertext,
			})

			if err != nil {
				logrus.WithError(err).Error("unable to decrypt honeycomb write key")
				return fmt.Errorf("unable to decrypt honeycomb write key")
			}
			writeKey = string(resp.Plaintext)
		}
	} else if ssmParameterName := os.Getenv("SSM_PARAMETER_NAME"); ssmParameterName != "" {
		rawKey, err := ps.GetParameter(&ssm.GetParameterInput{
			Name:           aws.String(ssmParameterName),
			WithDecryption: aws.Bool(true),
		})
		if err != nil {
			panic(err)
		}
		
		writeKey = *rawKey.Parameter.Value
		if writeKey == "" {
			return fmt.Errorf("no value for HONEYCOMB_WRITE_KEY")
		}
	} else {
		writeKey = os.Getenv("HONEYCOMB_WRITE_KEY")
		if writeKey == "" {
			return fmt.Errorf("no value for HONEYCOMB_WRITE_KEY")
		}
	}

	apiHost = os.Getenv("API_HOST")
	if apiHost == "" {
		apiHost = "https://api.honeycomb.io"
	}

	dataset = os.Getenv("DATASET")
	if dataset == "" {
		dataset = "honeycomb-cloudwatch-logs"
	}

	errorDataset = os.Getenv("ERROR_DATASET")

	libhoney.UserAgentAddition = fmt.Sprintf("integrations-for-aws/%s", version)

	// Call Init to configure libhoney
	libhoney.Init(libhoney.Config{
		WriteKey:   writeKey,
		Dataset:    dataset,
		APIHost:    apiHost,
		SampleRate: sampleRate,
	})

	debug := envOrElseBool("HONEYCOMB_DEBUG", false)
	if debug {
		go readResponses(libhoney.TxResponses())
	}

	return nil
}

As far as implementation goes, do you anticipate this change to incur additional lambda overhead? I'm not familiar with the SSM client, perhaps this is equivalent to opening a KMS session and decrypting the cipher blob, like the lambda does today.

I'm not the world's foremost AWS expert, but I would assume it is no more costly than the KMS decryption call.

Hope this helps, happy to elaborate further. :)

[dstrelau]

Thank you for submitting this issue! In an effort to better respond to new issues, we're closing old, inactive issues like this one. If you feel this is still important to consider, please leave a comment.