bryanklewis/logstash-output-azure_event_hubs

Azure Event Hubs Output for Elastic Logstash

logstash-output-azure_event_hubs's Issues

Timeout issue

The below error occurs on when we try to ingest new files on Linux machine. I was able to ingest few files but after couple of files it throws Time out error. Not sure what is the reason behind it.

1. 5671 port is enabled
2. .conf file is similar to mentioned in documentation

Connection to Event Hubs failed, will attempt connection again. {:error_message=>"Opening MessagingFactory timed out.", :class=>"Java::ComMicrosoftAzureEventhubs::TimeoutException", :retry_in_seconds=>2}
[2023-01-09T05:09:25,100][INFO ][com.microsoft.azure.eventhubs.impl.MessagingFactory][main] messagingFactory[MF_0ac0b3_1673258965100], hostName[xxxxx.servicebus.windows.net], info[starting reactor instance.]

sas key problem

I have tried the plugin didn't understand what is the SAS key is.
from the token structure, I can see there are "sr", "sig", "see" and "skn".
I assumed that the sas_key_name is "skn"
but what will be the sas_key?

Is there any example I can refer to?

Now I put something like below:
sas_key => 'sig=XXXXX&se=XXXX'
but with no luck

Gem missing jar dependencies

Hi! After installing this plugin and configuring the pipeline to use it, logstash complains about being unable to load 'proton-j' and other jar dependencies. I've manually downloaded the .gem file and it seems to be missing all vendor dependencies.

I've been able to make the plugin work from the sources, but it'd be nice if you could re-upload the fixed gem so that 'logstash-plugin install' worked.

Thanks a lot, and have a happy new year!

Can you specify a consumer_group?

Sorry if I missed it but I can't find where you can specify a consumer_group if you don't wish to use $Default. Thank you.

Logstash onConnectionError

Hey @bryanklewis
following is the Warning I am receiving at the logstash end where I have enabled this plugin:
to continue the log flow I have to restart the service. Is there any possible way to automate the continuous flow and avoid this error?

[WARN ][com.microsoft.azure.eventhubs.impl.MessagingFactory][###] onConnectionError messagingFactory[MF_ef3613_1638427488081], hostname[###], error[The connection was inactive for more than the allowed 60000 milliseconds and is closed by container 'LinkTracker'. TrackingId : ###, SystemTracker:gateway5, Timestamp:2021-12-07T03:20:55]

Add batch send support

Currently the plugin sends single events. Azure Event Hubs batch send support could offer improvements where higher throughput is required.

Encryption in transit

Is it possible to add a cert parameter or a way to ensure encryption in transit

logstash-output-azure_event_hubs

Hello,

Recently i upgraded logstash to 8.2 version and after that, from the logstash-plain.log I can see the next error:

[2022-06-16T18:46:22,466][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=>"output", :name=>"azure_event_hubs"}

Can you please help? How to resolve this issue?

Thanks,
Kiran.

service_uri param is not working properly

I'm looking to send events to an Event Hub in the Azure Gov cloud with a different service uri but am running into an error:

[2020-04-24T12:17:16,376][ERROR][logstash.javapipeline ][main] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<NameError: no method 'setEndpoint' for arguments (org.jruby.RubyString) on Java::ComMicrosoftAzureEventhubs::ConnectionStringBuilder>

Looks like the setEndpoint method is looking for a URI object but the param is being mapped to a string. Is there a specific format that was used to get this working or is this a bug?

Logstash output to Event Hub throughput

Not so much an issue, but more of a question.

I was playing with using Event Hub as a broker for messaging between Logstash instances (frontend/consolidation), and I am not seeing anywhere near the throughput that I would need. I have tweaked the number of client threads, but can't seem to get more than 1.2k messages per minute. It is well below the maximum for throttling on the Event Hub side. The vm it is running in is not resource constrainted.

Are you able to see higher message rates? Any recommended tweaks to get the throughput up?

received event properties on undefined (after setting via properties_bag)

Hey thanks for this gem!

I've been developing with it fine and dandy, but when I tried to add properties via properties_bag, I get undefined.

My process is logstash -> eventhub -> function app consumes.

To make things clean, I'm ignoring my function app for now and just made an ultra simple new node project locally to subscribe to the events from the following guide:

link https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-node-get-started-send

which after being configured, successfully receives events from logstash, but the properties field per a debug is undefined

Here's my output binding in logstash, as per your example:

azure_event_hubs { service_namespace => "EVENTHUBNAMESPACE" event_hub => "EVENTHUBVALUE" sas_key_name => "RootManageSharedAccessKey" sas_key => "SASKEYVALUE" properties_bag => {"Table" => "rv_health" "Format" => "JSON" "IngestionMappingReference" => "rv_health_mapping"} }

but again, the event's I receive look like:

{
  "body": {
    "pipeline_version": "1",
    "pipeline_host": "0924febe44aa",
    "source": "RV18_VGS",
    "pipeline_path": "/usr/share/logstash/input/rg_health/rv_health_small.log",
    "pipeline_ingest_timestamp": "2022-03-02T15:36:34.535Z",
    "ts": "20200312T194132.819207Z",
    "success": true,
    "data": {
      "test": "testVal"
    },
    "elapsed": 0.006981
  },
  "offset": "4295524656",
  "sequenceNumber": 1262,
  "enqueuedTimeUtc": "2022-03-02T15:36:35.322Z",
  "systemProperties": {}
}

Any suggestions? Am I doing something silly?

Thanks!