siafoundation/hostd

The next-generation of Sia storage providing software. More performant. More stable. Support for larger nodes.

hostd's Issues

Add frontend

Add an embedded web UI. Requires #1

Add Registry Subscriptions

The move from internal/mux to core/mux broke the registry subscription RPC. core/mux will need changes before it can be readded.

UI loading incorrect page

Current Behavior

when I hit the browser refresh button, the page refreshes to home but the URL still says whatever page you were at before you hit refresh.

Expected Behavior

Hitting refresh should show you the same page your are currently on.you

Steps to Reproduce

clicked on config
then clicked on the refresh button on the chrome browser.
URL says config but the home page is displayed. (see above image)

Version

0.1.0 Alpha 1

What operating system did the problem occur on (e.g. Ubuntu 22.04, macOS 12.0, Windows 11)?

windows 10, chrome browser

Anything else?

No response

UI metrics graphs support historic periods

Description

Add a way to set the period for the graphs to look back in history, vs periods that always end at the current day/time.

Version

No response

What operating system are you running (e.g. Ubuntu 22.04, macOS, Windows 11)?

No response

Anything else?

No response

Migrate to core

Migrate from the siad package to the core package

Show current period on graphs

Description

The current time is 9:53 AM, but looking at the graphs, I can only see up to 9:30. Displaying the current period (9:45-10:00 AM) would provide more up-to-date information.

Version

No response

What operating system are you running (e.g. Ubuntu 22.04, macOS, Windows 11)?

No response

Anything else?

No response

Remove ephemeral stores

• SQLite replacement for ephemeral registry store
• SQLite replacement for ephemeral account store
• SQLite replacement for ephemeral settings store
• SQLite replacement for ephemeral contracts store
• SQLite replacement for ephemeral storage manager
• SQLite replacement for ephemeral wallet store
• Update internal/test and cmd/hostd packages to use SQLite stores
• Remove ephemeral stores

Add Sector blocklist

Add the ability for hosts to block sectors by root, preventing them from being stored.

This change would add a new table, 4 new methods on the volume manager, and 4 new endpoints.

persist/sqlite changes

• Add a new table to init.sql
• Add a migration to create the table on existing hosts to migrations.go
• Add methods for add, list, and remove to Store

host/storage changes

• Add methods from the sqlite store to the Store interface in persist.go
• Add methods for add, list, and remove to VolumeManager.
• Keep the blocklist in memory for speed. Load it at startup.
• Check if the sector root is blocked in VolumeManager.Write and return an error.

api changes

• Add endpoints to the API for add, list, and remove. Maybe under /sectors/blocklist

Adding missing MDM instructions

New hostd MDM instructions:

• InstrAppendSectorRoot - appends a sector root to a contract. The sector must already be stored on the host with either InstrAppendSector or InstrStoreSector.
• InstrUpdateSector - updates a sector in a contract with new data
• InstrStoreSector - temporarily stores a sector on the host. Does not require a contract or collateral but is limited to a max duration of 144 blocks.

Implemented siad MDM instructions:

• InstrAppendSector
• InstrDropSectors
• InstrHasSector
• InstrReadOffset
• InstrReadSector
• InstrSwapSector

Missing siad MDM instructions:

• InstrRevision
• InstrReadRegistry
• InstrUpdateRegistry
• InstrRegistrySubscribe

It is currently impossible to implement RegistrySubscribe since migrating from hostd's internal/mux to core/mux

Sign + Notarize macOS build

macOS users cannot currently run hostd without jumping through hoops. Add workflow to sign and notarize the macOS builds. We can copy the workflow from siafoundation/embarcadero.

Add Alerts UI

Description

Add UI for global alerts https://api.sia.tech/hostd#baee3e48-fa37-4926-a21c-bf58608dccec

Version

No response

What operating system are you running (e.g. Ubuntu 22.04, macOS, Windows 11)?

No response

Anything else?

No response

Enter custom location when creating volume

Description

The "Location" in the create volume dialog is currently read-only. Make the location writable to enter custom locations. This will make it easier to access specific folders quickly.

Version

v0.1.0-alpha.1

What operating system are you running (e.g. Ubuntu 22.04, macOS, Windows 11)?

No response

Anything else?

The UX can be confusing to first-time users:
https://discord.com/channels/809849352516141067/1086016314419712020/1110479836251701269
https://discord.com/channels/809849352516141067/1086016314419712020/1110495574039474247

I'm not sure if it would be better to combine Name + Location or validate that Name does not contain / and show an error.

Add expiration height to contract sort

Description

Currently, contracts can be sorted by "timeline" (negotiationHeight) or status. Add an additional sort option for expiration height; the sort field is expirationHeight. An additional column on the graph for blocks remaining might be useful too.

Version

No response

What operating system are you running (e.g. Ubuntu 22.04, macOS, Windows 11)?

No response

Anything else?

No response

Add Tracing

Add tracing to hostd, most likely using open telemetry.

Remove `renterd` copy

Current Behavior

I'm unsure if there's anywhere else, but the "Contracts" page says: "There are currently no active contracts. Configure autopilot or manually form contracts to get started." in hostd when there are no contracts.

Expected Behavior

1. Start a new empty hostd node
2. Navigate to the contracts page

Steps to Reproduce

No response

Version

v0.1.0-alpha.2

What operating system did the problem occur on (e.g. Ubuntu 22.04, macOS 12.0, Windows 11)?

all

Anything else?

No response

Expose log over API

Tracking issue for a feature request. I'm not sure we will implement it, but getting additional feedback on some of the constraints would be good.

It would be beneficial if users could view and search their logs from the UI instead of manually parsing them on disk. Searchability might require moving the logs to the database instead of a local text file which could introduce performance concerns.

Add filtering to contracts endpoint

Add basic filters to the contracts endpoint. The contracts are already paginated, but filtering and sorting within those results is not currently possible.

Potential filters:

• Renter public key
• Status (pending, rejected, active, successful, failed)
• Expiration block (range)

Sort:

• Formation height
• Expiration height

siad migration

Enable migration from siad to hostd

Improve Test Coverage

Tracking issue for overall improvements to test coverage. PRs and additional issues should be referenced here

Packages that could use improvement:

Package	Coverage	Notes
go.sia.tech/hostd/host/contracts	50%	full contract lifecycle and cleanup tests
go.sia.tech/hostd/host/settings	40%
go.sia.tech/hostd/host/storage	64%	there are pretty good tests on migration and resizing, but they could be improved
go.sia.tech/hostd/internal/persist/sqlite	51%
go.sia.tech/hostd/rhp/v2	58.0%
go.sia.tech/hostd/rhp/v3	44.4%

Migrate to SQLite store

• Add accounts to the SQLite store
• Remove ephemeral store

Documentation

• Setup guides
• API docs (swagger?, postman?)
• Update host support docs

Contract Manager refactor

• Surface financial records in the contract store for atomicity between revisions.
• Move contract lifecycle decisions into the store
• Move storage manager out of contract manager

Show totals on the contracts page

Description

In the case of showing locked collateral, it would be nice to show the total locked collateral at the top (maybe as an option in the view option).

See the above host manager page as an example.

Version

0.1.0 Alpha 3

What operating system are you running (e.g. Ubuntu 22.04, macOS, Windows 11)?

Ubuntu 22.04, windows 10 chrome.

Anything else?

No response

Switch to structured logger

Implement RHP3 Renew RPC

Add volume manager

The Volume Manager is the default implementation of a storage
manager. It is responsible for managing the storage of sectors on a physical disk. It
stores metadata in the host's SQLite database and sector data in a flat file.

Responsibilities

• Managing volume metadata
• Managing sector metadata
• Managing volume data
• Managing temporary storage
• Reporting disk errors
• Migrating volume data

Requirements

• Stored sectors must stay consistent between the storage manager, contract manager,
  and temp storage
• Metadata must be ACID between contract revisions in RHP2 and program
  finalization in RHP3.
• Support for setting volumes to read-only
• Support for checking on-disk consistency and reporting errors
• Support for resizing and removing volumes
• During resize, sectors must be able to be moved within the same volume
• Avoid locking a folder during resize or remove operations; sectors not being
  actively moved should be readable.
• Support for multiple volumes with dynamic sizes
• Provide alerts and status information on long-running operations

Hostd: in contacts page the date should show up even without moving over the contract.

Description

In the contracts page right now there is no list of contracts with start and end date (an important feature) without moving over the contract.

The extra columns in the above example are useful. I sort by expiration when I want to know about upcoming expires and contract start when I want to see new contracts.

Version

0.1.0 Alpha 3

What operating system are you running (e.g. Ubuntu 22.04, macOS, Windows 11)?

Ubuntu 22.04, windows 10 chrome.

Anything else?

No response

RHP3 websocket support

Add web socket support to RHP3 so hosts can communicate directly with browsers. This feature was started in siad, but had some unresolved problems with the implementation preventing actual use.

Unresolved:

• How will hosts manage their SSL certificates?
  • There is a LetsEncrypt Go library; we could have cert management built-in, like Caddy. Some hosts will want to override the automatic management.
  • Require hosts to set up a proxy, but that is much less friendly.
• This would effectively require our hosts to announce using a domain instead of an IP.
  • We already recommend using a domain name over an IP, but this would change that to a requirement.

Tasks:

• Make sure SiaFoundation/mux and SiaFoundation/core transports work over a WebSocket conn.
• Add the web socket port, docker port, and CLI flag for 9984/tcp
• Start a secondary RHP3 listener on the WebSocket port.
• Cert management

Extra:

• Provide a browser RHP3 client lib. This may be possible with Go WASM without any changes, but I'm not sure this is the best option.

slider to allocate space when creating volumes.

Description

It be nice to have a slider to select the amount of space to allocate to a volume. Sia Host manager had that and it made it much easier to allocate space.

Version

0.1.0 Alpha 1

What operating system are you running (e.g. Ubuntu 22.04, macOS, Windows 11)?

Ubuntu 22.04, windows 10 chrome.

Anything else?

No response

API changes

I've made some changes to the API that break the UI.

• The JSON keys on the settings object have changed:
  • minStoragePrice -> storagePrice
  • minEgressPrice -> egressPrice
  • minIngressPrice -> ingressPrice
  • dyndns -> ddns
• [POST] /settings has been changed to [PATCH] /settings
• [PUT] /settings/dyndns/update has been changed to [PUT] /settings/ddns/update

Make the log location configurable

Description

It would be nice to be able to configure where the logs go to. More common in Linux is somewhere under /var/log rather than in the --dir location.

I have not checked but it would be nice if logging would survive a logrotate or have a postrotate option to restart logging. Otherwise you have to use the copytruncate option in logrotate which is not so neat.

Can the hostd.env be leveraged to hold this parameter rather than a bigger and bigger command line.

Version

v0.1.0-alpha.1 with fix at d4c887f

What operating system are you running (e.g. Ubuntu 22.04, macOS, Windows 11)?

Ubuntu 22.04

Anything else?

No response

Cache commonly used sectors in memory

Add a configurable LRU cache for sectors.

Edit: add implementation details

---

The cache needs to link the sector's root (types.Hash256) with the sector's data (*[rhpv2.SectorSize]byte). Older sectors should be ejected as more sectors are added. The cache should be disabled if the setting is zero.

There are a couple of moving parts here. My recommendation is to start with a static LRU cache in the volume manager, write some tests, then move on to making it configured via the host's settings. Each part can be together or separate PRs.

Part 1 - Add an LRU cache to the volume manager

1. Add the cache to the volume manager

   hostd/host/storage/storage.go

   Lines 63 to 78 in 16d5e8e

   	// A VolumeManager manages storage using local volumes.
   	VolumeManager struct {
   	a Alerts
   	vs VolumeStore
   	cm ChainManager
   	log *zap.Logger
   	recorder *sectorAccessRecorder
   	tg *threadgroup.ThreadGroup
   	mu sync.Mutex // protects the following fields
   	volumes map[int]*volume
   	// changedVolumes tracks volumes that need to be fsynced
   	changedVolumes map[int]bool
   	}
   	)

2. Check the cache for the sector root. If the sector does not exist in the cache, read it from disk, and add it.

   hostd/host/storage/storage.go

   Lines 844 to 871 in 16d5e8e

   	// Read reads the sector with the given root
   	func (vm *VolumeManager) Read(root types.Hash256) (*[rhpv2.SectorSize]byte, error) {
   	done, err := vm.tg.Add()
   	if err != nil {
   	return nil, err
   	}
   	defer done()
   	loc, release, err := vm.vs.SectorLocation(root)
   	if err != nil {
   	return nil, fmt.Errorf("failed to locate sector %v: %w", root, err)
   	}
   	defer release()
   	vm.mu.Lock()
   	v, ok := vm.volumes[loc.Volume]
   	if !ok {
   	vm.mu.Unlock()
   	return nil, fmt.Errorf("volume %v not found", loc.Volume)
   	}
   	vm.mu.Unlock()
   	sector, err := v.ReadSector(loc.Index)
   	if err != nil {
   	return nil, fmt.Errorf("failed to read sector %v: %w", root, err)
   	}
   	vm.recorder.AddRead()
   	return sector, nil
   	}

3. Add newly written sectors to the cache

   hostd/host/storage/storage.go

   Lines 901 to 927 in 16d5e8e

   	// Write writes a sector to a volume. release should only be called after the
   	// contract roots have been committed to prevent the sector from being deleted.
   	func (vm *VolumeManager) Write(root types.Hash256, data *[rhpv2.SectorSize]byte) (func() error, error) {
   	done, err := vm.tg.Add()
   	if err != nil {
   	return nil, err
   	}
   	defer done()
   	release, err := vm.vs.StoreSector(root, func(loc SectorLocation, exists bool) error {
   	if exists {
   	return nil
   	}
   	start := time.Now()
   	vol, err := vm.getVolume(loc.Volume)
   	if err != nil {
   	return fmt.Errorf("failed to get volume %v: %w", loc.Volume, err)
   	} else if err := vol.WriteSector(data, loc.Index); err != nil {
   	return fmt.Errorf("failed to write sector %v: %w", root, err)
   	}
   	vm.log.Debug("wrote sector", zap.String("root", root.String()), zap.Int("volume", loc.Volume), zap.Uint64("index", loc.Index), zap.Duration("elapsed", time.Since(start)))
   	return nil
   	})
   	if err == nil {
   	vm.recorder.AddWrite()
   	}
   	return release, err
   	}

Part 2 - Make the cache size configurable

1. Add a new setting for the number of sectors to cache. An int or uint32 is fine. Each sector is 4MiB.

   hostd/host/settings/settings.go

   Lines 39 to 76 in 16d5e8e

   	// Settings contains configuration options for the host.
   	Settings struct {
   	// Host settings
   	AcceptingContracts bool `json:"acceptingContracts"`
   	NetAddress string `json:"netAddress"`
   	MaxContractDuration uint64 `json:"maxContractDuration"`
   	WindowSize uint64 `json:"windowSize"`
   	// Pricing
   	ContractPrice types.Currency `json:"contractPrice"`
   	BaseRPCPrice types.Currency `json:"baseRPCPrice"`
   	SectorAccessPrice types.Currency `json:"sectorAccessPrice"`
   	Collateral types.Currency `json:"collateral"`
   	MaxCollateral types.Currency `json:"maxCollateral"`
   	StoragePrice types.Currency `json:"storagePrice"`
   	EgressPrice types.Currency `json:"egressPrice"`
   	IngressPrice types.Currency `json:"ingressPrice"`
   	PriceTableValidity time.Duration `json:"priceTableValidity"`
   	// Registry settings
   	MaxRegistryEntries uint64 `json:"maxRegistryEntries"`
   	// RHP3 settings
   	AccountExpiry time.Duration `json:"accountExpiry"`
   	MaxAccountBalance types.Currency `json:"maxAccountBalance"`
   	// Bandwidth limiter settings
   	IngressLimit uint64 `json:"ingressLimit"`
   	EgressLimit uint64 `json:"egressLimit"`
   	// DNS settings
   	DDNS DNSSettings `json:"ddns"`
   	Revision uint64 `json:"revision"`
   	}

2. Add the new setting to the database

   hostd/persist/sqlite/init.sql

   Lines 184 to 209 in 16d5e8e

   	CREATE TABLE host_settings (
   	id INTEGER PRIMARY KEY NOT NULL DEFAULT 0 CHECK (id = 0), -- enforce a single row
   	settings_revision INTEGER NOT NULL,
   	accepting_contracts BOOLEAN NOT NULL,
   	net_address TEXT NOT NULL,
   	contract_price BLOB NOT NULL,
   	base_rpc_price BLOB NOT NULL,
   	sector_access_price BLOB NOT NULL,
   	collateral BLOB NOT NULL,
   	max_collateral BLOB NOT NULL,
   	storage_price BLOB NOT NULL,
   	egress_price BLOB NOT NULL,
   	ingress_price BLOB NOT NULL,
   	max_account_balance BLOB NOT NULL,
   	max_account_age INTEGER NOT NULL,
   	price_table_validity INTEGER NOT NULL,
   	max_contract_duration INTEGER NOT NULL,
   	window_size INTEGER NOT NULL,
   	ingress_limit INTEGER NOT NULL,
   	egress_limit INTEGER NOT NULL,
   	ddns_provider TEXT NOT NULL,
   	ddns_update_v4 BOOLEAN NOT NULL,
   	ddns_update_v6 BOOLEAN NOT NULL,
   	ddns_opts BLOB,
   	registry_limit INTEGER NOT NULL
   	);

3. Add a migration to add the new column https://github.com/SiaFoundation/hostd/blob/master/persist/sqlite/migrations.go
4. Change the store methods to read and update the new column

   hostd/persist/sqlite/settings.go

   Lines 17 to 112 in 16d5e8e

   	func (s *Store) Settings() (config settings.Settings, err error) {
   	var dyndnsBuf []byte
   	const query = `SELECT settings_revision, accepting_contracts, net_address,
   	contract_price, base_rpc_price, sector_access_price, collateral,
   	max_collateral, storage_price, egress_price, ingress_price,
   	max_account_balance, max_account_age, price_table_validity, max_contract_duration, window_size,
   	ingress_limit, egress_limit, registry_limit, ddns_provider, ddns_update_v4, ddns_update_v6, ddns_opts
   	FROM host_settings;`
   	err = s.queryRow(query).Scan(&config.Revision, &config.AcceptingContracts,
   	&config.NetAddress, (*sqlCurrency)(&config.ContractPrice),
   	(*sqlCurrency)(&config.BaseRPCPrice), (*sqlCurrency)(&config.SectorAccessPrice),
   	(*sqlCurrency)(&config.Collateral), (*sqlCurrency)(&config.MaxCollateral),
   	(*sqlCurrency)(&config.StoragePrice), (*sqlCurrency)(&config.EgressPrice),
   	(*sqlCurrency)(&config.IngressPrice), (*sqlCurrency)(&config.MaxAccountBalance),
   	&config.AccountExpiry, &config.PriceTableValidity, &config.MaxContractDuration, &config.WindowSize,
   	&config.IngressLimit, &config.EgressLimit, &config.MaxRegistryEntries,
   	&config.DDNS.Provider, &config.DDNS.IPv4, &config.DDNS.IPv6, &dyndnsBuf)
   	if errors.Is(err, sql.ErrNoRows) {
   	return settings.Settings{}, settings.ErrNoSettings
   	}
   	if dyndnsBuf != nil {
   	err = json.Unmarshal(dyndnsBuf, &config.DDNS.Options)
   	if err != nil {
   	return settings.Settings{}, fmt.Errorf("failed to unmarshal ddns options: %w", err)
   	}
   	}
   	return
   	}
   	// UpdateSettings updates the host's stored settings.
   	func (s *Store) UpdateSettings(settings settings.Settings) error {
   	const query = `INSERT INTO host_settings (id, settings_revision,
   	accepting_contracts, net_address, contract_price, base_rpc_price,
   	sector_access_price, collateral, max_collateral, storage_price,
   	egress_price, ingress_price, max_account_balance,
   	max_account_age, price_table_validity, max_contract_duration, window_size, ingress_limit,
   	egress_limit, registry_limit, ddns_provider, ddns_update_v4, ddns_update_v6, ddns_opts)
   	VALUES (0, 0, $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22)
   	ON CONFLICT (id) DO UPDATE SET (settings_revision,
   	accepting_contracts, net_address, contract_price, base_rpc_price,
   	sector_access_price, collateral, max_collateral, storage_price,
   	egress_price, ingress_price, max_account_balance,
   	max_account_age, price_table_validity, max_contract_duration, window_size, ingress_limit,
   	egress_limit, registry_limit, ddns_provider, ddns_update_v4, ddns_update_v6, ddns_opts) = (
   	settings_revision + 1, EXCLUDED.accepting_contracts, EXCLUDED.net_address,
   	EXCLUDED.contract_price, EXCLUDED.base_rpc_price, EXCLUDED.sector_access_price,
   	EXCLUDED.collateral, EXCLUDED.max_collateral, EXCLUDED.storage_price,
   	EXCLUDED.egress_price, EXCLUDED.ingress_price, EXCLUDED.max_account_balance,
   	EXCLUDED.max_account_age, EXCLUDED.price_table_validity, EXCLUDED.max_contract_duration, EXCLUDED.window_size,
   	EXCLUDED.ingress_limit, EXCLUDED.egress_limit, EXCLUDED.registry_limit, EXCLUDED.ddns_provider,
   	EXCLUDED.ddns_update_v4, EXCLUDED.ddns_update_v6, EXCLUDED.ddns_opts);`
   	var dnsOptsBuf []byte
   	if len(settings.DDNS.Provider) > 0 {
   	var err error
   	dnsOptsBuf, err = json.Marshal(settings.DDNS.Options)
   	if err != nil {
   	return fmt.Errorf("failed to marshal ddns options: %w", err)
   	}
   	}
   	return s.transaction(func(tx txn) error {
   	_, err := tx.Exec(query, settings.AcceptingContracts,
   	settings.NetAddress, sqlCurrency(settings.ContractPrice),
   	sqlCurrency(settings.BaseRPCPrice), sqlCurrency(settings.SectorAccessPrice),
   	sqlCurrency(settings.Collateral), sqlCurrency(settings.MaxCollateral),
   	sqlCurrency(settings.StoragePrice), sqlCurrency(settings.EgressPrice),
   	sqlCurrency(settings.IngressPrice), sqlCurrency(settings.MaxAccountBalance),
   	settings.AccountExpiry, settings.PriceTableValidity, settings.MaxContractDuration, settings.WindowSize,
   	settings.IngressLimit, settings.EgressLimit, settings.MaxRegistryEntries,
   	settings.DDNS.Provider, settings.DDNS.IPv4, settings.DDNS.IPv6, dnsOptsBuf)
   	if err != nil {
   	return fmt.Errorf("failed to update settings: %w", err)
   	}
   	// update the currency stats
   	timestamp := time.Now()
   	if err := setCurrencyStat(tx, metricContractPrice, settings.ContractPrice, timestamp); err != nil {
   	return fmt.Errorf("failed to update contract price stat: %w", err)
   	} else if err := setCurrencyStat(tx, metricBaseRPCPrice, settings.BaseRPCPrice, timestamp); err != nil {
   	return fmt.Errorf("failed to update base RPC price stat: %w", err)
   	} else if err := setCurrencyStat(tx, metricSectorAccessPrice, settings.SectorAccessPrice, timestamp); err != nil {
   	return fmt.Errorf("failed to update sector access price stat: %w", err)
   	} else if err := setCurrencyStat(tx, metricCollateral, settings.Collateral, timestamp); err != nil {
   	return fmt.Errorf("failed to update collateral stat: %w", err)
   	} else if err := setCurrencyStat(tx, metricStoragePrice, settings.StoragePrice, timestamp); err != nil {
   	return fmt.Errorf("failed to update storage price stat: %w", err)
   	} else if err := setCurrencyStat(tx, metricEgressPrice, settings.EgressPrice, timestamp); err != nil {
   	return fmt.Errorf("failed to update egress price stat: %w", err)
   	} else if err := setCurrencyStat(tx, metricIngressPrice, settings.IngressPrice, timestamp); err != nil {
   	return fmt.Errorf("failed to update ingress price stat: %w", err)
   	} else if err := setNumericStat(tx, metricMaxRegistryEntries, settings.MaxRegistryEntries, timestamp); err != nil {
   	return fmt.Errorf("failed to update max registry entries stat: %w", err)
   	}
   	return nil
   	})
   	}

5. Pass the initial cache setting into the volume manager and initialize the cache

   hostd/host/storage/storage.go

   Lines 956 to 979 in 16d5e8e

   	// NewVolumeManager creates a new VolumeManager.
   	func NewVolumeManager(vs VolumeStore, a Alerts, cm ChainManager, log *zap.Logger) (*VolumeManager, error) {
   	vm := &VolumeManager{
   	vs: vs,
   	a: a,
   	cm: cm,
   	log: log,
   	recorder: &sectorAccessRecorder{
   	store: vs,
   	log: log.Named("recorder"),
   	},
   	volumes: make(map[int]*volume),
   	changedVolumes: make(map[int]bool),
   	tg: threadgroup.New(),
   	}
   	if err := vm.loadVolumes(); err != nil {
   	return nil, err
   	}
   	go vm.recorder.Run(vm.tg.Done())
   	go vm.cleanup()
   	return vm, nil
   	}

6. Resize the cache when the setting changes. There are a few ways to do this. The least invasive is probably to add a ResizeCache(sectors int) error func to the volume manager and call it from the settings API handler.

   hostd/api/endpoints.go

   Lines 116 to 153 in 16d5e8e

   	func (a *api) handlePATCHSettings(c jape.Context) {
   	buf, err := json.Marshal(a.settings.Settings())
   	if !a.checkServerError(c, "failed to marshal existing settings", err) {
   	return
   	}
   	var current map[string]any
   	err = json.Unmarshal(buf, &current)
   	if !a.checkServerError(c, "failed to unmarshal existing settings", err) {
   	return
   	}
   	var req map[string]any
   	if err := c.Decode(&req); err != nil {
   	return
   	}
   	err = patchSettings(current, req)
   	if !a.checkServerError(c, "failed to patch settings", err) {
   	return
   	}
   	buf, err = json.Marshal(current)
   	if !a.checkServerError(c, "failed to marshal patched settings", err) {
   	return
   	}
   	var settings settings.Settings
   	if err := json.Unmarshal(buf, &settings); err != nil {
   	c.Error(err, http.StatusBadRequest)
   	return
   	}
   	err = a.settings.UpdateSettings(settings)
   	if !a.checkServerError(c, "failed to update settings", err) {
   	return
   	}
   	c.Encode(a.settings.Settings())
   	}

Save selected time period

Description

The selected time period should persist between reloads, it currently resets.

Version

No response

What operating system are you running (e.g. Ubuntu 22.04, macOS, Windows 11)?

No response

Anything else?

No response

Fix RPCRenew

RPCRenew is currently commented out due to incompatibility with siad. Need to bring it inline with the rest of the RPC

Integrated DynDNS

Add DynDNS support through various DNS providers. Much of the work has already been completed in the internal/dyndns package

• Add DynDNS package for updating providers
  • Integrate No-IP
  • Integrate DuckDNS
  • Integrate AWS route53
  • Integrate Cloudflare
• Add DynDNS config to host internal settings
• Periodically check for IP changes and update DNS provider

UI Localization

Description

Currently in the hostd Overview screen, all graph use the US date format (mm/dd/yyyy). Please can they display in the localized date format. The format should be taken from either the server (better) or from the browser (best).

Guessing, but if you use the localized date format, you can also use the localised time?

Version

v0.1.0-alpha.1 with fix at d4c887f

What operating system are you running (e.g. Ubuntu 22.04, macOS, Windows 11)?

Ubuntu 22.04

Anything else?

No response

Change scales of pricing graph

Description

The values on the pricing graph should line up with the values on the configuration page:

• Ingress - SC/TB
• Egress - SC/TB
• Collateral - SC/TB/mo
• sector access - SC/million
• base rpc - SC/million

Version

No response

What operating system are you running (e.g. Ubuntu 22.04, macOS, Windows 11)?

No response

Anything else?

No response

Track storage over time

Will need updates to both the backend and the frontend to periodically grab the host's used storage

Incorrect amount of available space

Current Behavior

when adding a volume, it does not accurately indicate the amount of used space.

You will note in the screen shot that I already have 9 TB of space used in two folders on the mounted volume but it is still offering me to create a 9 tb volume.

Expected Behavior

as volume folders are created the amount of available space should drop.

Steps to Reproduce

No response

Version

0.1.0 Alpha 1

What operating system did the problem occur on (e.g. Ubuntu 22.04, macOS 12.0, Windows 11)?

Ubuntu 22.04

Anything else?

No response

Implement API

All endpoints are authenticated and prefixed by /api

Method	Path	Description
GET	/	summary of host's current state: synced height, wallet balance, earnings, collateral, storage, contracts, etc
GET	/syncer	get the current gateway state
GET	/syncer/peers	get the current list of peers
PUT	/syncer/peers/:address	connect to a peer
DELETE	/syncer/peers/:address	disconnect from a peer
POST	/announce	announce the host to the network
GET	/settings	get the current settings
PUT	/settings	update the current settings
GET	/financials/:period	financial breakdown -- daily, weekly, monthly, yearly totals
GET	/contracts	paginated list of active contracts, sorted by expiration height asc
GET	/contracts/:id	get a specific contract
DELETE	/sectors/:root	delete a sector
GET	/volumes	list of added storage volumes, stats per volume
POST	/volumes	add a new volume to the host
GET	/volumes/:id	get current state of a volume
PUT	/volumes/:id	change settings of a volume
DELETE	/volumes/:id	remove a volume from the host
PUT	/volumes/:id/resize	resize a volume
GET	/wallet	summary of wallet, including balance and address
GET	/wallet/transactions	paginated list of transactions sorted by block height desc
POST	/wallet/send	send a transaction

Use volume manager

To-Do:

• Update the interfaces in the rhp packages
• Update usage in internal/test package
• Update cmd/hostd
• Remove the ephemeral storage manager

Migrate RHP3 to contractor and volume manager

Track Host Statistics

Replace the financial manager with a statistics manager. This module will be responsible for keeping track of all time-based statistics for the host. Statistics should be saved in 15-minute intervals to limit database growth but provide sufficient drill-down. This also includes #23

The financial statistics will need to handle reorgs and contract failures. Especially when dealing with accounts. The other statistics are significantly easier to implement.

Stats to track:

• Potential Revenue:
  • RPC
  • Storage
  • Ingress
  • Egress
  • Registry Read
  • Registry Write
• Earned Revenue:
  • RPC
  • Storage
  • Ingress
  • Egress
  • Registry Read
  • Registry Write
• Collateral
  • Locked
  • Risked
  • Burnt
• Contracts
  • Pending
  • Rejected
  • Active
  • Successful
  • Failed
• Storage
  • Usage
  • Contract Sectors
  • Temporary Sectors
  • Registry Entries
• Bandwidth
  • Ingress
  • Egress
• Wallet
  • Balance

Add CAD to currencies

Description

Is CAD currency going to be added to the list (it was available in the previous version.

Version

0.1.0 Alpha 1

What operating system are you running (e.g. Ubuntu 22.04, macOS, Windows 11)?

Ubuntu 22.04

Anything else?

No response

Close create and resize dialog while initializing volumes

Description

I created a new volume of 10TB. Initially, the volume is added to /volumes with a status of "creating". The maxSectors field is updated as the volume initializes. Once the volume is finished, the status changes to "ready".

From a UX perspective, we should be able to close the volume and perform other tasks while the volume initializes in the background. Manually closing the dialog works but may hide a potential error.

I don't know if this can be done easily in the UI alone. We might need to update the API to return immediately if the volume is successfully created and continue initializing it in the background. An alerts endpoint to track errors may be necessary. Thoughts?

Add financial metrics to host statistics

Add financial metrics to the host's tracked statistics:

• Potential Revenue:
  • RPC
  • Storage
  • Ingress
  • Egress
  • Registry Read
  • Registry Write
• Earned Revenue:
  • RPC
  • Storage
  • Ingress
  • Egress
  • Registry Read
  • Registry Write

Statistics will need to be reverted in case of contract failures or rejections.

Hostd - contracts page (page no resizing very well)

Current Behavior

There is too much blank space between columns and at the same time it is not showing the full contract address when there is more than enough space on the screen for it.

Expected Behavior

It would be nice if there was an option to add more columns of data (not sure what at this point but I would suggest looking to host manager as a guide) and has full display of the contract id.

Steps to Reproduce

No response

Version

0.1.0 Alpha 3

What operating system did the problem occur on (e.g. Ubuntu 22.04, macOS 12.0, Windows 11)?

Ubuntu 22.04, windows 10 chrome.

Anything else?

No response

Automate Integrity Checks

We've added an integrity check endpoint to manually check the integrity of contracts and their stored sectors, but automating this would be helpful in detecting issues early.

An automated check needs to be very careful of additional IO overhead. We don't want to cause excessive wear on the host's disks or bottlenecks for renters trying to access their data.

• How often should the check run?
• How many sectors should be checked per iteration?
• How to determine which sectors to check? Random? Least recently accessed?

Includes #85

UI table field lengths

Current Behavior

I have a large screen, the contract id is not completely shown even though there is lots of space to do so. It seems like the website does not take full advantage of bigger screens.

Expected Behavior

I would expect on bigger screens that as much of a field is being shown as can be.

Steps to Reproduce

No response

Version

0.1.0 Alpha 1

What operating system did the problem occur on (e.g. Ubuntu 22.04, macOS 12.0, Windows 11)?

windows 10, chrome browser

Anything else?

No response

Reorder sections on the dashboard

Description

It would be nice if there was the ability to move panes on the sia hostd web windows.

Right now at the to is revenue so it would be nice to move the contract pane to the top rather than have revenue at the top.

it would also be nice if we could ad some widgets similar to the ones that show up at the bottom of the sia host manager apps.

Version

0.1.0 Alpha 1

What operating system are you running (e.g. Ubuntu 22.04, macOS, Windows 11)?

Ubuntu 22.04, windows 10 chrome.

Anything else?

No response