elastickent/mil-std-1553-es

Docker based 1553 Chapter 10 data ingest into Elasticsearch.

mil-std-1553 to Elasticsearch

mil-std-1553 is a industrial control bus used by air and space craft.

IRIG 106 is a comprehensive telemetry standard to ensure interoperability in aeronautical telemetry application at RCC member ranges. It defines how to store 1553 data in Chapter 10 format.

IRIG 106 is developed and maintained by the Telemetry Group of the Range Commanders Council.

Prerequisites

• Docker
• Filebeat
• Elasticsearch
• Kibana

Build and run.

1. Clone this repo.

2. Edit es_curl_setup.sh and filebeat.yml files with your Elasticsearch server details. Then run it.

   ./es_curl_setup.sh

3. In your newly cloned ./mil-std-1553-es, clone https://github.com/bbaggerman/irig106lib and https://github.com/bbaggerman/irig106utils. -Thank you bbaggerman!

4. Create a data_sets directory. Your directory structure should now look like the following.

./mil-std-1553-es
├── Dockerfile
├── Readme.md
├── data_sets
├── docker_build_run.sh
├── es_curl_setup.sh
├── filebeat.yml
├── irig106lib
│   ├── gcc
│   ├── msvs6
│   ├── python
│   ├── readme.txt
│   ├── src
│   ├── vs2005
│   ├── vs2008
│   ├── vs2010
│   ├── vs2012
│   ├── vs2015
│   └── vs2017
└── irig106utils
    ├── gcc
    ├── i106utils.txt
    ├── msvs6
    ├── src
    ├── vs2005
    ├── vs2008
    ├── vs2010
    ├── vs2012
    └── vs2017

5. Add your ch10 file to your ./data_sets directory. Samples can be downloaded from here.

   cp ~/TC-1553_107_132248.ch10 ./data_sets/

6. Load the data into Elasticsearch.

   ./docker_build_run.sh /data_sets/TTC-1553_107_132248.ch10

7. Import dashboard in ./kibana/export.ndjson via Kibana's Management->Saved Objects->Import.

Note: idmp1553 ouput does not contain Year/Month/Day. The Elasticsearch ingest pipeline sets it to Jan 1st, so set your time filter in Kibana to start at Jan 1 of this year. Hour/Day/Seconds.milliseconds are preserved.

Dashboard:

mil-std-1553-es's Issues

Readme suggest

I went through the instructions created the dashboard. These are the changes I would make to the readme file

mil-std-1553 to Elasticsearch

​
mil-std-1553 is a industrial control bus used by air and space craft.
​
IRIG 106 is a comprehensive telemetry standard to ensure interoperability in aeronautical telemetry application at RCC member ranges. It defines how to store 1553 data in Chapter 10 format.
​
IRIG 106 is developed and maintained by the Telemetry Group of the Range Commanders Council.
​

Prerequisites

• Laptop/Server with the following
  • Docker
  • Filebeat
• Elasticsearch (can be on a remote system)
• Kibana (can be on a remote system)
  ​

Build and run.

​

1. Clone this repo.
2. Edit es_curl_setup.sh with:

• your Elasticsearch server details on both lines with a elastic URL
• add the -u parameter if a username is required
  Example of the #2 instructions:
  curl -k -XPUT "https://elastic.mydomain.com:9200/_template/1553" -u "kent" -H...
  curl -k -XPUT "https://elastic.mydomain.com:9200/_template/1553" -u "kent" -H
  ​
  ​

3. Run es_curl_setup.sh on the laptop/server
   ./es_curl_setup.sh
   Note: this .sh is setup for linux. Mac requires the top line to be removed, that line being "#!/usr/local/bin/bash"
4. Edit the filebeat.yml from this repo with your Elasticsearch server details. No need to run Filebeat yet

​
5. In your newly cloned ./mil-std-1553-es, clone https://github.com/bbaggerman/irig106lib and https://github.com/bbaggerman/irig106utils. -Thank you bbaggerman!
6. Create a data_sets directory. Your directory structure should now look like the following.
Note: You might need to change the irig106utils and irig106lib foldernames to the below

./mil-std-1553-es
├── Dockerfile
├── Readme.md
├── data_sets
├── docker_build_run.sh
├── es_curl_setup.sh
├── filebeat.yml
├── irig106lib
│   ├── gcc
│   ├── msvs6
│   ├── python
│   ├── readme.txt
│   ├── src
│   ├── vs2005
│   ├── vs2008
│   ├── vs2010
│   ├── vs2012
│   ├── vs2015
│   └── vs2017
└── irig106utils
    ├── gcc
    ├── i106utils.txt
    ├── msvs6
    ├── src
    ├── vs2005
    ├── vs2008
    ├── vs2010
    ├── vs2012
    └── vs2017

7. Add your ch10 file to your ./data_sets directory. Samples can be downloaded from here.
   ​
   cp ~/TC-1553_107_132248.ch10 ./data_sets/
   ​
8. In docker_build_run.sh change the filebeat parameter to the folder locatin of your filebeat.
   Example:
   Old: ..."Ch" | filebeat -c ./filebeat.yml...
   New: ..."Ch" | filebeat-7.6.2-darwin-x86_64/filebeat -c ./filebeat.yml...
9. Load the data into Elasticsearch.
   ​
   ./docker_build_run.sh /data_sets/TTC-1553_107_132248.ch10
   Note: this .sh is setup for linux. Mac requires the top line to be removed, that line being "#!/usr/local/bin/bash"
   ​
10. Import dashboard in ./kibana/export.ndjson via Kibana's Management->Saved Objects->Import.
    ​
    Note: idmp1553 ouput does not contain Year/Month/Day. The Elasticsearch ingest pipeline sets it to Jan 1st, so set your time filter in Kibana to start at Jan 1 of this year. Hour/Day/Seconds.milliseconds are preserved.
    ​

Dashboard:

​