calvinmclean/automated-garden

weather-based smart irrigation controller and backend

automated-garden's Issues

Make config generator more interactive

Description

The config generator should ask for fields when they are missing to improve beginner experience. Also it can be further improved to directly write to files.

Requirements

• Prompts for all fields that are not already defined in config file
• Allow writing to configurable file
• Still allow print to stdout
• By default, do not overwrite existing file (but have option for it)

Open Questions

• Which library should be used for prompts? Options:
  • go-survey/survey
  • songmu/prompter
  • c-bata/go-prompt

Add metrics for clients (`storage`, `influxdb`, `weather`, etc)

It would be relatively easy to add metrics wrappers to count the number of requests, timing of requests, etc.

• weather
• influxdb
• mqtt

Add fan control to periodically turn on a fan for airflow

Control a fan on a schedule to increase airflow in a garden. Useful specifically for seed-starting setups.

Requirements

• Available as part of a Zone
• Support simple/flexible schedules like other actions, but will likely be on a smaller scale like "every 10 minutes"

Open Questions

• Will this be controlled by a relay or variable control like a mosfet? Maybe both should be options
• Should it be more similar to a WaterAction, telling the controller to run it for a specific period of time, or like a LightAction that requires the server to toggle ON/OFF states in the controller? Most likely it will be similar to LightAction because I will be running for 10+ minutes. Maybe the same should be done for WaterAction because I am adapting this to work better for a yard irrigation setup and will be doing long periods of watering
• If this is so similar to a LightAction, do I combine those types somehow?

Add more weather data in ZoneResponse

Currently, a response from GET /zones/{zoneID} returns information about the rain/moisture data and next watering time/duration. It would be useful to have additional data, such as recent temperature (used by scaling), plus some details on the scale factor currently available.

Key Features/Considerations

• Uses time period of Zone's watering interval for data
• Temperature data

Create Patch methods for other API types instead of massive one for Garden/Zone/Plant

Currently I implement a Patch method for high-level API types. I should break this up into smaller parts for sub-types because it is getting unruly

Use acorn secret for weather client details when running dev

https://docs.acorn.io/running/args-and-secrets
https://docs.acorn.io/authoring/secrets

This would allow me to use weather client easily without maintaining separate config.

Depends on #56

Consider deprecating YAML/ConfigMap storage Client

Description

After merging the changes for #13, the existing YAML storage client becomes obsolete because it can be replaced by the hord implementation. The ConfigMap, however, is not replaced so it might still make sense to keep it. On the other hand, it could be removed since the redis option is very easy to use in a K8s environment.

Use secure storage for WeatherClient secrets

Description

Ideally there will be a secure option for storing API secrets like this. Consider Hashicorp Vault or similar.

This will require a new feature for a composite StorageClient which should be pretty easy to achieve using the current interface setup.

Follow-up for #77

GenerateConfig does not work correctly with multiple Zones

When generating a config with multiple zones, there are a few issues:

• NUM_ZONES is 0 (actually this was just missing from config, but should be inferred by len of Zones)
• ZONES array is not comma-separated

Setup Loki for log aggregation

https://grafana.com/docs/loki/latest/getting-started

Requirements

• Add any required configuration and document deploy steps
• Only for K8s deployments, not worth it in Docker
• Add support for JSON logs in garden-app

Refactor moisture as a WeatherClient so I can de-couple it from a Garden and not require a Garden struct for moisture details

Re-add:

// TODO: Can I re-enable this if moisture comes from WeatherClient instead of garden? Follow up issue #95
if ws.HasSoilMoistureControl() {
	logger.Debug("getting moisture data for WaterSchedule")
	soilMoisture, err := wsr.getMoisture(ctx, garden, ws)
	if err != nil {
		logger.WithError(err).Warn("unable to get moisture data for WaterSchedule")
	} else {
		logger.Debugf("successfully got moisture data for WaterSchedule: %f", soilMoisture)
		response.WeatherData.SoilMoisturePercent = &soilMoisture
	}
}

around line 60 in water_schedule_responses.go (NewWaterScheduleResponse)

This might be a little tricky because the weather client in this case needs access to InfluxDB client

Record data about water scaling to show water savings

Description

Record data about weather-based scaling in order to display data about water savings.

Requirements

• Use InfluxDB to record watering actions from the server-side (record configured duration and scaled duration)
• Integrate with watering history API to show water savings and also can show if there is a mix-match with recordings from the controller

Getting RainData fails if WaterSchedule.Interval < 24h

When WaterSchedule.Interval is 12h, GetRainData fails with error:

unable to get rain data from weather client: unable to read response body '{\"body\":[],\"status\":\"ok\",\"time_exec\":0.02210092544555664,\"time_server\":1668203473}': json: cannot unmarshal array into Go struct field weatherDataResponse.body of type map[string][]float32

This requires 2 changes:

• Correctly handle empty data/array
• Default to larger interval (like temperature control) or do not allow rain delays with such small intervals

Add WaterSchedule feature to ensure minimum watering duration in a given interval

Description

If someone uses aggressively water-saving scaling, it could result in long periods without watering which might not be good even if the low water usage is great the rest of the time. Additionally, you might want to experiment with some low-water setups but don't want to risk killing any plants.

This feature will solve those problems by allowing the user to configure a minimum watering duration for a specified period of time. Ideally, this time should be a multiple of the watering interval since this is when it will be checked. This should definitely be a requirement because someone might mistakenly have a large watering interval and expect to be saved by minimum watering.

{
  "water_schedule": {
    "minimum_watering": {
      "duration": "2h",
      "interval": "168h"
    }
  }
}

Create WeatherClient API resource and remove from config

Description

This feature is a pre-requisite to some other features related to WeatherClient because it will assign an ID to the client.

Related features:

• #78
• #65
• #49 (kind of?)

Requirements

• Use same format as the current configuration
• Located on the root path of the API
• Assign and ID and "display name" to the clients
• Research and create issue for secure credential storage, but initial implementation will just store in YAML
• This will not affect SoilMoistureControl because that is tied to a physical garden-controller and should only work with Zone's in a Garden with moisture sensors
• Netatmo client should update and save access_token and refresh_token when using refresh token
• Implementation details:
  • WeatherControl now needs to reference a WeatherClient's ID
  • Remove from config
  • Add yaml and json tags to structs
  • Implement in CRUD in StorageClient
  • Add API CRUD endpoints
  • Add Validate method to WeatherClient so Options can be validated for specific clients create endpoint will initialize the client which should verify if it is valid
  • Add /test endpoint or something similar to test weather client configuration/auth
  • Add configmap storage client
  • OpenAPI specification and docs
  • testing

Implement “day of week” schedule and other similar schedules

This would allow you to choose “every Sunday and Wednesday” instead of time-based constant interval.

Depends on #22 for more complex data structure for schedules

I could also do something like "every month"

Since cron already supports both of these cases, is easily compatible with gocron library, and is well-known and somewhat easy to use (at least it is with all of the helpful resources available). I was originally going to make this a special case for parsing the pkg.Duration string, but this won't make sense in the cases where it is used as an actual Duration instead of a interval, so I might extend the struct to have a similar Interval struct.

Add optional scaling to RainControl

This would be an additional option for the RainControl that allows watering to just be scaled proportionally to the amount of rain, rather than just skipping.

Requirements

• Optional, default is to just skip
• Allow setting max scaling amount (100% would allow skip/delay, 50% would make sure you always do at least 50% of duration)
  • This will act as a hard cutoff value, not an input to the scaling range. For example, if max is 50% and we calculate water should be scaled to 25%, it will still do 50%. If this was used as the max end of the scaling range, it would water 37.5%
• Allow setting a point where scaling kicks in so small amounts of rain will have no impact
  • Rain Threshold: 20mm, Scaling activation: 50%:
    • 5mm rain, 100% watering
    • 10mm rain, 100% watering
    • 15mm rain, 50% watering
    • 20mm rain, 0% watering
    • 17.5mm rain, 25% watering
      • If this example also had a max scaling amount of 50%, this would do 50% of watering

Open Questions

• How will this work if we also have TemperatureControl? Will they compound? I think that makes the most sense
• I may not be able to use ScaleControl because higher inputs should scale negatively. Also I do not want to scale down when there is no rain, only scale down when threshold is exceeded

Fix deprecation warnings from Github Actions

Fix the following deprecations:

• Node.js 12 actions are deprecated. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/. Please update the following actions to use Node.js 16: actions/checkout, actions/setup-go, actions/checkout
• The set-output command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
• The save-state command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

Add support for secure MQTT

Description

MQTT broker supports authentication so I should enable it in the broker, server, and controller. Another benefit of this is that it enables using a cloud-hosted MQTT.

Fix TODO: allow ExecuteAction functions to return warnings so they can be logged without returning an error

automated-garden/garden-app/worker/zone_actions.go

Line 46 in 60e011c

// TODO: Refactor to be able to return warnings so they can be logged without returning an error

An alternative to returning warnings would be to just use the built-in logger that I forgot about... I would have preferred to keep the logger out of action execution, but it is good to log this stuff, especially as I add other data-influence behaviors that could benefit from clear logging.

Combine YAML + ConfigMap clients

These clients have a lot of duplication since they are both just reading from the same YAML files, but reading/writing from different sources. I was thinking about having the YAML client struct have replaceable functions for read/write, but realized I don't want those to be exported. I might move them into the same package and just have different sources.

Move Garden Health to main GET endpoint

The primary reason for this is to reduce extra HTTP calls. /health endpoint provides a small amount of data that could easily fit into the regular response:

{
	"status": "UP",
	"details": "last contact from Garden was 21.593868678s ago",
	"last_contact": "2023-02-13T22:23:18.299911341Z"
}

However, it could reduce the performance of this endpoint because of calls to external client (InfluxDB)

Use viper to bind all config parameters to environment variables

https://github.com/spf13/viper#working-with-environment-variables

Add support for WaterSchedules as a separate resource and allow for multiple per Zone

Description

It would be nice to be able to define more complex WaterSchedules for a Zone. This would allow for watering a Zone multiple times in one day (such as morning/evening, mid-afternoon top-off, or a multiple short soaks for even, deep watering). A complex configuration like this might be annoying to work with as a JSON object nested under a Zone. Also, after putting careful consideration into the WaterSchedule, you might want to use it in multiple Zones without reconfiguring.

Requirements

• Manage WaterSchedules as separate API resources, independent of Gardens or Zones
• Reference by ID in a Zone
• WaterSchedules run independent of Zones as their own scheduled "thing" and it will check for any eligible Zones when it executes
  • This results in fewer scheduled Jobs needing to be managed if I have multiple Zones using the same schedule
  • Manual delaying of watering can go back to the old way of keeping a counter on a Zone of the number of skips, or more likely a timestamp saying the next time it can be watered
  • Possible annoying thing or issue with this implementation is that next_watering_time and weather_data is now tied to the WaterSchedule + Zone. I will implement the response on the WaterSchedule API and then the ZoneResponse can use the same methods and compensate for any configured skips

nil pointer when getting Zone with WeatherControl and no weather client configured

worker/zone_actions.go:106

Make dynamic WaterSchedules that have different values for different times of year

Details

This would allow changing the watering intervals to water less frequently in winter and more frequently in summer, rather than just using weather data to increase/decrease durations

Depends on #22

Implementation

• Since Zones are able to have multiple WaterSchedules, this can be implemented by having multiple WaterSchedules configured for a Zone and then add "active periods" to WaterSchedules so they can be considered "active" or "inactive"
• I don't want to worry about stopping and re-enabling scheduled Jobs whenever a WaterSchedule becomes active or inactive, so instead the schedules will always be running, but will first check if they are active before executing

Add support for temperature/humidity sensors

Add the ability to read data from temperature/humidity sensors. Even though I added support for external weather APIs, this feature can still be useful for indoor gardens and for monitoring the electronics in outdoor locations.

Requirements

• Allow reading temperature/humidity data in garden-controller, publishing data to InfluxDB via Telegraf and MQTT
• Allow garden-app to read data from InfluxDB
• Initially this will not be used for WeatherControl and will just be used to have data about the controller's conditions

create common "restrictEndDatedMiddleware" using EndDateable interface

Description

This is a simple enhancement to reduce some code duplication on middlewares for restricting access to end-dated API resources

Integration testing

It would be good to introduce some integration tests that verify the results of various actions when the entire application stack is running. The mock controller could be extended with some sort of assertions/expectations for better testing.

Overall coverage before integration testing: 53.3%
Integration tests should focus on areas that are not currently tested by unit tests, plus maybe a few high-level API tests for basic peace of mind testing and end-to-end creation of resources and seeing their effects.

Specific areas that need testing (based on low coverage and fit for integration testing):

• controller/controller.go (0%)
• controller/handlers.go (0%)
• influxdb/client.go (0%)
• mqtt/mqtt.go (0%)
• yaml/yaml.go (40%, could be covered by more unit tests)
• server package code like NewGardensResource, routes, etc.
  • server/server.go (0%)
  • garden_responses.go (80%, some NextLightTime code that could be covered, maybe could be unit test?)
• LightDelay action executing (could do very short light delays, then wait)

Use JSON decodable Duration format so ActionRequest duration can use string format

Maybe I can create a struct that lets me write my own JSON Unmarshal so I can easily integrate all the error detecting that I have to do throughout the whole system. This would save a lot of pointless error checks and will improve the front-end experience.

Initially I wanted to Marshal the value as the integer representation for an action's message to the controller, but it will make more sense to have the string representation for JSON to the front-end. I will look into options to make converting to integer representation easier

Allow UI to choose display units for measurements (metric vs standard)

Currently the backend does everything in metric, but it would be nice to choose preferred units on the UI

Refactor moisture-controlled watering to be related to WeatherControl

After adding the WeatherClient functionality for reading from my Netatmo weather station, the built-in soil moisture is kind of disconnected. It would make sense to integrate this into the WeatherControl feature.

Requirements

• Preserve current functionality of using moisture data to delay watering, but implement it similar to the GetTotalRain function
• SoilMoistureControl will be part of weather.Control next to RainControl struct
• Refactor the API response showing current moisture to be part of a new WeatherData struct that also has rain data
• Do not implement as a WeatherClient because then I have to enable multiple clients

Implement real storage backend

Implement another storage.Client that uses actual database backend, whether it is SQL or NoSQL.

I recently found madflojo/hord and added support for local file storage in this PR. Now, I can switch my local storage solution to use this and will just need some additional configuration to support multiple KV and NoSQL storage solutions.

Requirements

• Implement storage.Client with type KV that can be compatible with all current hord backends
• hord will return an error if something is not found. I need to either handle this to just return nil, nil like other clients or implement a new NotFound error that can be used by all client implementations

Open Questions

• What happens to my combined configmap/yaml storage? YAML storage will be replaced by hord, but it won't support saving to a configmap. I could just remove that since I don't use it anymore, but it is also fine to keep it around
• Currently, everything relies on Zones being managed as a map under a Zone, but this doesn't work as well with a KV store. The options are requiring editing Zones by reading/writing an entire Garden, or storing Zones separately and having the Garden ID in the Key so it can be found by Garden ID

Implement oauth flow for Netatmo WeatherClient instead of requiring all secrets

Description

Instead of generating an access token and refresh token, you should be able to use the oauth flow to redirect to Netatmo website for login. This could be implemented as a CLI helper and/or as part of the process for creating a client using API.

Add data validation at startup to help reduce errors when making API constraint changes

Description

Since I am not using a database or DB migrations, whenever I change my data structures to add a new "required" field, my existing test data might not have these fields. This will cause a lot of errors that should not exist if new data was created through the API. I should have the application validate at startup and fail to run (make optional/forceable with config/CLI arg) if current data doesn't fit API constraints.

Add help info to interactive config generator

The library that I am using allows for having help tips for each prompt. This could improve the user experience for beginners that are relying on the config generator. This also helps reduce the need for documentation.

Update to latest Go version

Add basic HTTP and Go metrics + Prometheus

Add HTTP metrics and export using Prometheus

Replace storage.MockClient in tests with an in-memory storage backend

Description

After #13 adds support for hord database client, I can easily use the in-memory hashmap option in all of my tests which will increase coverage on actual storage client.

Once this is done, I can remove the MockClient which also allows me to remove the storage interface.

However, would it be better to implement direct unit tests in-package instead of relying on semi-integration tests from the server and worker packages?

Setup code coverage

Description

It would be nice to have code coverage reports on PRs and on the whole projects. Two good free options are codecov and coveralls. These should both integrate easily with GitHub

Add "Getting Started" guide in README to run with Docker Compose and/or Acorn

Requirements

• Very simple to run everything, including mock controller using Docker (or Acorn option for advanced users)
• Interact with Garden and Zone API resources using curl
• Run an action with curl
• Introduce Insomnia API Client collection for advanced usage
• Look at Grafana dashboard

Scale watering duration based on local weather conditions

With the addition of WeatherControl, it would be cool to be able to use current, or recent, temperature and humidity data to scale watering durations up/down. This feature would also be a good opportunity to implement an open weather API because it doesn't need data to be as accurate as the RainControl.

Requirements

• Scale watering up/down based on recent temperature data
  • Baseline watering duration: standard watering amount that will be used to increase/decrease
  • Baseline average temperature: this could be provided or maybe calculated from past weather data (or location-based data). Temperatures above/below this amount will trigger a variable scaling based on scale factor as a max
  • Scale factor: percentage value to control maximum baseline duration scaling (maybe have different up/down values?)
  • Use average daily high temperatures. This is more user friendly because we are often aware of expected summer highs, but would have no idea what the average temperature is for the span of a whole day
  • Use a rolling average of daily highs between now and the last watering time
    • Actually just use watering interval rather than actual last watering time, just for ease and consistency
    • What if the interval is 1 day or even less than a day? Maybe default to 3 days if that is the case

Open Questions

• After doing a bit more research about watering, it seems like it might be more important to increase watering frequency during times of increased temperature. Unfortunately, there isn't a great way to handle this without constantly reconfiguring schedules
• I originally wanted to also include humidity data in this, but I am not sure how to do that accurately...

Add another implementation of Weather Client using publicly-available weather API

Description

Add another implementation of weather.Client that uses a publicly-available weather API to remove the current hardware barrier for weather-based watering.

Open Weather Map
Wunderground

Requirements

• Allow disabling certain features of the client (maybe disable rain data if an API doesn't provide reliable data). An option for implementing this would be to have multiple interfaces and use an assertion to check if methods are available. Other option is to have empty implementation of these methods.
• Add new interfaces to Weather Client to implement support for certain features: ForecastClient and HistoricalClient

Open Questions

• Consider adding support for multiple Weather Clients because this allows using
  • This will be tricky to implement in a clean way
• Consider adding support for forecasting data instead of just historical data
  • This relates to the requirement of adding clients that are partial implementations since some services might have forecast data only (or no forecast data)
  • This also relates to multiple Weather Clients, especially if some are partial implementations

Fix API specification and docs

After some recent changes, documentation and API specification need to be updated.

Some configuration + API changes that need documentation:

• #9
  • OpenAPI
  • Docs
• #17
  • OpenAPI
  • Docs
• #24
  • OpenAPI
  • Docs
• #30
  • OpenAPI
  • Docs
• #35
  • OpenAPI
  • Docs
• #43
  • OpenAPI
  • Docs

Also need to document:

• Using config generator
• More in-depth guide for setting up Netatmo client

MQTT or WiFi disconnect on controller

MQTT logs after plugging controller in:

2023-01-26T22:55:34: New connection from 10.42.0.1:24331 on port 1883.
2023-01-26T22:55:34: New client connected from 10.42.0.1:24331 as seed-garden (p2, c0, k15).

Based on logs it looks like it was being disconnected due to conflict? Now I am no longer messing with different controllers so maybe it is a non-issue. I will keep an eye on it and check logs if it fails again.

So far this has not reproduced, probably since I haven't been configuring any other controllers, but I should still make sure the controller code supports wifi reconnect and MQTT reconnect.

Add Enable/Disable to a Zone and Garden

When a Zone/Garden is disabled, all scheduled actions and on-demand actions will be ignored.

Allow displaying weather scaling estimate based on current data

In addition to next_water_time, display what the calculated duration would be based on current weather conditions in requests to GET /zones/{zoneID}. This is useful for UI details and for debugging weather scaling.

Add metrics for Worker

It is difficult to think of useful metrics in this case. A lot of things are already covered on the InfluxDB side of things (based off actual watering and controller data). The main thing is that I do not have a lot of visibility into the scheduled jobs. Certain things like execution counts and durations are not particularly useful, although it could be interesting if there is an MQTT issue causing delays.

Useful metrics:

• Currently-schedule Jobs (Gauge)
  • Tagged with "type": "garden"/"zone" and `"id": ""
  • Adhoc light delay jobs will self-decrement the gauge when run
  • Always increment/decrement when adding/removing scheduled jobs
• If an error occurs in the background, it is hard to know about so those should be metrics

Implement separate scaling for up/down

It might be nice to have more flexible scaling configurations that have different parameters for scaling up or down.

Some possible use-cases are:

• Only scale down, never up (or reverse)
• Limit scaling down to halfway, but allow scaling up to double watering
• Allow a larger range for scaling up if there is a big range of high temperatures

Implementation:

• Separate up/down range will allow to have more extreme scaling down for colder weather, but will prevent extremely high water use in summer
• Min/max scaling value can be used similarly to prevent from scaling too high or too low

WaterActionRequest with small time unit causes default watering on the controller

I noticed that if a very small watering duration value is used for a WaterActionRequest, such as 10 nanoseconds, the value is not sent correctly to the microcontroller or is not decoded correctly and results in the default watering time. These types of values are not realistic to use and cannot accurately be done by the controller so it should not be allowed anyways.

This was discovered because the iOS app was out of date with the backend version (specifically changes from #66), so a value of 10s sent from iOS was decoded as 10 microseconds by the backend.