[Logstash 7.16.2] S3 input plugin replaces the region in endpnt url

[glen-uc]

[Logstash 7.16.2] S3 input plugin replaces the region in endpnt url

[kaisecheng]

I can reproduce the same error, however, logstash-input-s3 does not modify the endpoint setting, see here and here. It passes to aws SDK straightly. The plugin use aws SDK to access aws service, so this could be an issue of the SDK or the SDK require more setup to make it work with vpc.
Need further investigation.

[robbavey]

@glen-uc Looks like you are encountering aws/aws-sdk-ruby#2483

Can you try setting the environment variable AWS_S3_US_EAST_1_REGIONAL_ENDPOINT to 'regional' and report back?

[Edit]
The environment variable has other issues, try setting

s3_us_east_1_regional_endpoint=regional

in your aws config file

[glen-uc]

@robbavey Thank you for your reply.

we tried adding s3_us_east_1_regional_endpoint=regional to the config file and deployed the logstash with interface endpoint, but we got this error

 hostname "<bucket_name>.<vpc_id>.s3.us-east-1.vpce.amazonaws.com" does not match the server certificate
  Exception: Seahorse::Client::NetworkingError

Looks like it's not replacing the region in the endpoint as expected, in order to solve this error we tried passing ssl_verify_peer as false in additional setting and redeployed the logstash but it again failed but this time with this error

{:exception=>Aws::S3::Errors::NoSuchBucket, :message=>"The specified bucket does not exist"

We verified that specified bucket exists and logstash has necessary permissions (it works when we use default endpoint)

[jacqclouseau]

@glen-uc, we're seeing the same issue with the s3 output plugin (logstash-output-s3 v4.3.5), our workaround is to set the endpoint value with 2 region strings, e.g.:

s3 {
   region => "us-east-1"
   endpoint =>  "https://<our_vpc_endpoint_id>.s3.us-east-1.us-east-1.vpce.amazonaws.com"
}

this "tricks" the plugin's logic to replace the first us-east-1 string, but keeping the second us-east-1 string, so that the final value contains the aws region.

[glen-uc]

@jacqclouseau Thank you for the suggestion

I tried your approach by adding one more region to the endpoint URL i.e endpoint => "https://<our_vpc_endpoint_id>.s3.us-east-1.us-east-1.vpce.amazonaws.com"

But still, I am getting

{:exception=>Aws::S3::Errors::NoSuchBucket, :message=>"The specified bucket does not exist"

Here is my full logstash s3 input configuration

                s3 {
                    bucket => <my_bucket>
                    type => <my_type>
                    sincedb_path => <my_path>
                    prefix => <my_prefix>
                    endpoint => "<vpc_ep_id>.s3.us-east-1.us-east-1.vpce.amazonaws.com"
                    region => "us-east-1"
                    additional_settings => {
                             ssl_verify_peer => false
                    }
                  }

Note: If i remove the custom endpoint logstash works again so not an issue with the bucket being missing

[jacqclouseau]

@glen-uc, we've encountered the name or service not known and hostname does not match the server certificate errors only.

We've seen the name resolution error message when the 'us-east-1' string was removed from the target's address.

The certificate validation error was seen when we set the endpoint value to endpoint => "https://<vpc endpoint>.s3.us-east-1.us-east-1.vpce.amazonaws.com". After having a look at the certificate properties we had to change the value to endpoint => "https://bucket.<vpc endpoint>.s3.us-east-1.us-east-1.vpce.amazonaws.com". I don't know if this applies to our setup only, or if that's how the AWS S3 certificates for the VPC interfaces get generally created.

We could reproduce the cert issue without Logstash by running curl -v https://<bucket name>.<vpc endpoint>.s3.us-east-1.vpce.amazonaws.com, what gave us a SSL_ERROR_BAD_CERT_DOMAIN response. We then inspected the certificate by running the following command:

fqdn='<bucket name>.<vpc interface>.s3.us-east-1.vpce.amazonaws.com'
echo | openssl s_client -showcerts -servername "${fqdn}" -connect "${fqdn}":443 2>/dev/null | openssl x509 -inform pem -noout -text

Looking at the X509v3 Subject Alternative Name values told us what names the certificate would recognise.

Running curl -v https://<bucket name>.bucket.<vpc endpoint>.s3.us-east-1.vpce.amazonaws.com was then just the confirmation we needed.

Apologies if I went off topic with the cert issue description.

[glen-uc]

@jacqclouseau Thank you for the detailed description due to which we were able to solve the problem with logstash

Here is what happened in our case

We were running logstash in a K8 cluster along with other logging components like fluentbit, when we migrated to using interface endpoints for s3 we first did changes to fluent bit so that it uses the interface endpoint by setting the endpoint to something like this https://<vpc_ep_id>.s3.us-east-1.vpce.amazonaws.com and it worked fine without any additional configuration ( hence we assumed bucket.<vpc_ep_id> not required while setting up endpoint URL )

When doing the same for logstash we encountered errors as described above but finally we are able to solve it by setting endpoint URL to something like this https://bucket.<vpc_ep_id>.s3.us-east-1.us-east-1.vpce.amazonaws.com

Note: not using bucket.<vpc_ep_id>... might also be the reason why setting s3_us_east_1_regional_endpoint=regional did not work, will try by setting this and using only one region in endpoint URL something like https://bucket.<vpc_ep_id>.s3.us-east-1.vpce.amazonaws.com to see if it works