Archived Data Migrator For Software Defined Buildings
adm works in a parallelized manner with go routines reading from the source and writing to channels from which other go routines read and write to specified destinations.
adm can be abstracted to three main components. adm itself as well as reader and writer APIs that conform to the Reader and Writer interface.
- adm - Responsible for parallelizing and assigning tasks to readers and writers as well as logging progress.
- Readers - Responsible for reading data from a specified source. This can be through HTTP calls or reading local files.
- Writers - Responsible for writing data to a specified destination. Destinations can be local files or a network endpoint.
- Clone the repo.
- Install dependencies:
go get
- Build adm:
go build
- Run adm:
./adm
The params.yml
file contains modifiable settings. Settings are listed below:
- source_url: Source url to query for data. (TODO: Allow for multiple sources for metadata, timeseries data, etc.)
- worker_size: Loose bound on maximum number of running go routines allowed.
- open_io: Bound on maximum number of open IOs allowed.
- metadata_dest: Destination of metadata. Can be a text file or a URL.
- timeseries_dest: Destination of timeseries data. Can be a text file or a URL.
- read_mode: Specified read mode (See Read Mode)
- write_mode: Specified write mode (See Write Mode)
- chunk_size: Rough estimate of number of timeseries tuples to process per go routine.
adm supports reading data from various sources. Modes are integers corresponding to an implementation of the Reader interface. As of now, only one read mode is implemented.
type Reader interface {
readUuids(src string) ([]string, *ProcessError)
readWindows(src string, uuids []string) ([]*Window, *ProcessError)
readMetadata(src string, uuids []string, dataChan chan *MetadataTuple) *ProcessError
readTimeseriesData(src string, slots []*TimeSlot, dataChan chan *TimeseriesTuple) *ProcessError
}
- 1 (Giles Mode) - Read from Giles endpoint.
adm supports writing data to various destinations. Modes are integers corresponding to an implementation of the Writer interface. As of now, only one write mode is implemented.
type Writer interface {
writeUuids(dest string, uuids []string) *ProcessError
writeMetadata(dest string, dataChan chan *MetadataTuple) *ProcessError
writeTimeseriesData(dest string, dataChan chan *TimeseriesTuple) *ProcessError
}
- 2 (File Mode) - Write to local files.