Go ETL using Ratchet pipelines
make start
To configure, edit ./config/config.yaml to load a new pipeline.
To add a custom ETL, create a new plugin on ./plugins and add is on config.yaml.
- Add on config/config.yaml:
workers:
- schedule:
hour: 20
minute: 0
job:
name: http-requestor
code: |
package main
import (
"github.com/Henrod/go-etl/processors"
"github.com/dailyburn/ratchet"
)
type etl string
func (e etl) Extract() ratchet.DataProcessor {
return processors.NewHTTPRequestor("GET", "http://localhost:8080")
}
func (e etl) Transform() ratchet.DataProcessor {
return &processors.Logger{}
}
func (e etl) Load() ratchet.DataProcessor {
return &processors.Null{}
}
// ETL is the exported symbol of this plugin
var ETL etl
- Start:
make start
- Create a new plugin on ./plugins like this:
// ./plugins/http-requestor/main.go
package main
import (
"github.com/Henrod/go-etl/processors"
"github.com/dailyburn/ratchet"
)
type etl string
func (e etl) Extract() ratchet.DataProcessor {
return processors.NewHTTPRequestor("GET", "http://localhost:8080")
}
func (e etl) Transform() ratchet.DataProcessor {
return &processors.Logger{}
}
func (e etl) Load() ratchet.DataProcessor {
return &processors.Null{}
}
// ETL is the exported symbol of this plugin
var ETL etl
- Build the plugin binary:
make plugins
- Add on config/config.yaml:
workers:
- period: 1h
job:
name: http-requestor
- Start:
make start