logstash-output-azuresearch is a logstash plugin to output to Azure Search. Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite destinations. Azure Search is a managed cloud search service provided in Microsoft Azure.
You can install this plugin using the Logstash "plugin" or "logstash-plugin" (for newer versions of Logstash) command:
bin/plugin install logstash-output-azuresearch
# or
bin/logstash-plugin install logstash-output-azuresearch (Newer versions of Logstash)
Please see Logstash reference for more information.
output {
azuresearch {
endpoint => "https://<YOUR ACCOUNT>.search.windows.net"
api_key => "<AZURESEARCH API KEY>"
search_index => "<SEARCH INDEX NAME>"
column_names => ['col1','col2','col3'..] ## ## list of column names (array)
key_names => ['key1','key2','key3'..] ## list of Key names (array)
flush_items => <FLUSH_ITEMS_NUM>
flush_interval_time => <FLUSH INTERVAL TIME(sec)>
}
}
- endpoint (required) - Azure Search service endpoint URI
- api_key (required) - Azure Search API key
- search_index (required) - Azure Search Index name to insert records
- column_names (required) - List of column names (array) in a target Azure search index. 1st item in column_names should be primary key
- key_names (optional) - Default:[] (empty array). List of key names (array) in in-coming record to insert. The number of keys in key_names must be equal to the one of columns in column_names. Also the order or each item in key_names must match the one of each items in column_names.
- flush_items (optional) - Default 50. Max number of items to buffer before flushing (1 - 1000).
- flush_interval_time (optional) - Default 5. Max number of seconds to wait between flushes.
Here is an example configuration where Logstash's event source and destination are configured as standard input and Azure Search respectively.
input {
stdin {
codec => json_lines
}
}
output {
azuresearch {
endpoint => "https://<YOUR ACCOUNT>.search.windows.net"
api_key => "<AZURESEARCH API KEY>"
search_index => "<SEARCH INDEX NAME>"
column_names => ['id','user_name','message','created_at']
key_names => ['postid','user','content','posttime']
flush_items => 100
flush_interval_time => 5
}
}
You can find example configuration files in logstash-output-azuresearch/examples.
Now you run logstash with the the example configuration like this:
# Test your logstash configuration before actually running the logstash
bin/logstash -f logstash-stdin-json-to-azuresearch.conf --configtest
# run
bin/logstash -f logstash-stdin-json-to-azuresearch.conf
Here is an expected output for sample input (JSON Lines):
JSON Lines
{ "id": "a001", "user_name": "user001", "message":"msg001", "created_at":"2016-12-28T00:01:00Z" },
{ "id": "a002", "user_name": "user002", "message":"msg002", "created_at":"2016-12-28T00:02:00Z" },
{ "id": "a003", "user_name": "user003", "message":"msg003", "created_at":"2016-12-28T00:03:00Z" },
Output (Azure Search POST message)
"value": [
{ "@search.score": 1, "id": "a001", "user_name": "user001", "message": "msg001", "created_at": "2016-12-28T00:01:00Z" },
{ "@search.score": 1, "id": "a002", "user_name": "user002", "message": "msg002", "created_at": "2016-12-28T00:02:00Z" },
{ "@search.score": 1, "id": "a003", "user_name": "user003", "message": "msg003", "created_at": "2016-12-28T00:03:00Z" }
]
Bug reports and pull requests are welcome on GitHub at https://github.com/yokawasa/logstash-output-azuresearch.