k0swe/forester

A web-based amateur radio logging application with fast sync, offline and rig control

forester's Issues

Use Browser Location API to fill logging station's latitude, longitude and grid square

Fix login/logout behavior

Currently when a user logs in, the QSO list does not start to populate, and when they log out the QSO list does not get emptied. This might be fixed by having a separate page for logged out users and protecting logged-in routes.

Clear WSJT-X and Hamlib agent status when they disappear

Right now the agent display does not clear out WSJT-X or Hamlib status changes to down. They're based on the last WSJT-X heartbeat and Hamlib rig state, which are not cleared out when their status is down.

Migrate away from angular/flex-layout

Per angular/flex-layout#1426, flex-layout is being deprecated and will only be maintained for a short time longer. Blog post in https://blog.angular.io/modern-css-in-angular-layouts-4a259dca9127.

Forester needs to move away from using this library.

QTH profiles

Simplify entering the logging station's details by adding QTH profiles.

Not saving WSJT-X QSOs

Right now, Kellog is not saving logged QSOs sent by kel-agent.

Turns out, that's because kel-agent is serializing multiple JSON documents in a single websocket message:

{"version":"kel-agent v0.2.2 (f03f44e)","wsjtx":{"type":"QsoLoggedMessage","payload":{"id":"WSJT-X","dateTimeOff":"2021-01-03T00:43:45Z","dxCall":"K6ESE","dxGrid":"EL09","txFrequency":7076100,"mode":"FT8","reportSent":"-15","reportReceived":"-07","txPower":"100","comments":"","name":"","dateTimeOn":"2021-01-03T00:42:45Z","operatorCall":"","myCall":"K0SWE","myGrid":"DM79LV","exchangeSent":"","exchangeReceived":""}}}
{"version":"kel-agent v0.2.2 (f03f44e)","wsjtx":{"type":"LoggedAdifMessage","payload":{"id":"WSJT-X","adif":"\n\u003cadif_ver:5\u003e3.1.0\n\u003cprogramid:6\u003eWSJT-X\n\u003cEOH\u003e\n\u003ccall:5\u003eK6ESE \u003cgridsquare:4\u003eEL09 \u003cmode:3\u003eFT8 \u003crst_sent:3\u003e-15 \u003crst_rcvd:3\u003e-07 \u003cqso_date:8\u003e20210103 \u003ctime_on:6\u003e004245 \u003cqso_date_off:8\u003e20210103 \u003ctime_off:6\u003e004345 \u003cband:3\u003e40m \u003cfreq:8\u003e7.076100 \u003cstation_callsign:5\u003eK0SWE \u003cmy_gridsquare:6\u003eDM79LV \u003ctx_pwr:3\u003e100 \u003cEOR\u003e"}}}

SyntaxError: Unexpected token { in JSON at position 419
    at JSON.parse (<anonymous>)
    ...

The correct behavior is probably for kel-agent to send only one JSON document per websocket message. As a workaround, Kellog can split on line separators.

Going haywire creating agent websockets

When forester has trouble establishing a websocket connection to the agent, it's supposed to back off and retry every ~10 seconds. Right now, it's retrying without waiting, creating dozens or hundreds of simultaneous websocket connection requests. Then if the agent does come back in this state, we get dozens or hundreds of established connections, and dozens of simultaneous notifications when WSJT-X sends something. As a result, I've been getting a lot of duplicate WSJT-X logged QSOs.

LoTW import doesn't appear to work

Access to fetch at 'https://us-central1-k0swe-kellog.cloudfunctions.net/ImportLotw?logbookId=K7CHW' from origin 'https://forester.radio' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

us-central1-k0swe-kellog.cloudfunctions.net/ImportLotw?logbookId=K7CHW:1 Failed to load resource: the server responded with a status of 504 (Gateway Timeout)

Log QSO from WSJT-X

Create a QSO contact record from WSJT-X agent event.

Log spam of "InvalidValueError" for lat/long

InvalidValueError: at index 0: not a LatLng js?key=AIzaSyDSTJMwyUK9K5Vd0I1MWtg2Gb14DylAhpE:66 InvalidValueError: at index 0: not a LatLng or LatLngLiteral with finite coordinates: in property lat: not a number
InvalidValueError: setPosition: not a LatLng js?key=:66 InvalidValueError: setPosition: not a LatLng or LatLngLiteral: in property lat: not a number

These two lines repeat ad nauseam. I don't have lat/long set. Attempted to use Forester for the first time and imported a 5k+ QSO ADIF generated by WSJT-X. App itself is basically unusable and is getting killed.

OS: Ubuntu 22.04
Browser: Chrome 101.0.4951.54

Use generated Firestore IDs instead of random

I've noticed some problems with duplicate QSOs being created, for instance during WSJT-X listening. We could curtail duplicates by using our own Firestore IDs instead of randomly assigned ones. Choosing the key is obviously important, but so far I've had good luck in deduplication logic by using the logging station callsign, contacted station callsign and the time with one minute precision. If two people talk twice in the same minute, is that really a separate QSO?

Onboarding flow

When a user logs in for the first time, we at least need to create a new logbook with their personal callsign.

Auto-import QRZ.com details for new QSOs

When a new QSO gets uploaded to Firestore, kick off a cloud function to import station details from QRZ.com.

Button to sync QSO list search with WSJT-X

It'd be great if the QSO list could filter to the callsign you're currently calling in WSJT-X to see if and when you've called them before. This should probably default to "off" but have a button/toggle next to the search bar to enable it.

Delete Logbook

I created a "Test Logbook" so I could play with ADIF imports. I discovered that I can't delete that logbook, though I can delete all the QSOs in the logbook. It would be nice to be able to delete the logbook as well to reduce clutter, ideally with a warning if it's not empty.

Customizable QSO entry fields, and "Save and Next" button

For use cases like contesting and SOTA/POTA, operators are focused on just a few QSO entry fields, and need to be able to quickly and repetitively enter similar QSOs. For these cases, there should be a subset of visible fields relevant to the event at hand, there should be an additional Save button that goes directly to a new entry, and the new entry should copy most fields like frequency and signal report from the previously saved QSO.

Suggested by Devin KN6PHZ.

Autocapitalize Callsigns

When typing one-handed or on a phone, it's challenging to capitalize all letters in a callsign. It looks like autocapitalize="characters" can do this automatically.

ADX import and export

Though ADX isn't widely supported, I feel I should at least try. Maybe I'll be the second logging application to do so.

Mobile-friendly QSO details

The QSO details editor form should be reactive for mobile form factors. It's pretty much unusable on mobile right now.

Public logbooks

I'd like to be able to mark a logbook as "publicly viewable" and allow anonymous users to view a logbook without being able to edit it.

Import SIG, SIG_INFO

I imported an ADIF file for a POTA activation. In the resulting QSOs I don't see any of the data I put in SIG, SIG_INFO, MY_SIG, and MY_SIG_INFO which are used to convey special activities (e.g. "POTA" in SIG/MY_SIG) and their details (e.g. park/summit/island number in SIG_INFO/MY_SIG_INFO).

When logging park hunter QSOs I'd been putting the park number in the comment field, but a separate field would be nicer.

ADI export fit and finish

Basic export was implemented in #6, but there are fit and finish issues that remain. This includes creating Date objects for everything that needs it, and some testing, verification and hardening.

Restructure DB to support many-to-many users-to-logbooks

Right now, the DB is structured such that there's one logbook per user. This is restrictive for users who need multiple logbooks (operating portable, travelling, licensed in multiple countries, etc.) I could just make a logbooks array under the user, but if I make logbooks a first-class entity, I can easily expand to multiple users on one logbook later (to support club logbooks, etc).

QSO detail mat-select fields don't use dirty color

I can't get the CSS selector right to make the band and continent select fields take on the accent color when the value is dirty (modified from original).

Chrome Auto-fill Gets in the Way of Contact Country Popup

Chrome seems eager to auto-fill my own address into the City, State, and Country fields in the "Contacted Station". I can mostly ignore the city and state popups, but the one for country is kind of annoying, since the Chrome auto-fill popup is above Forester's own autocomplete list of countries.

Apparently it's difficult to tell Chrome not to auto-fill a field that looks like an address, but there are some hacks people have tried. Accepting "USA" for "United States" might make this a little less annoying.

Edit QSO details

Be able to edit a QSO that's already in the database.

Flags don't render on Windows

I'm using Unicode "emoji" to display flags in the QSO list. This works fine on Linux (at least my distro) and Mac, but Windows doesn't appear to have a font installed by default that has the flag glyphs. I'll probably need to install a fallback font.

ADIF file import only works once

The "Import ADIF file" feature only works reliably the first time it's used; after that the page has to be reloaded, or maybe you can open the file selection and cancel and get it out of it's funk.

There's a JS console log message that's probably relevant:

ERROR TypeError: Failed to execute 'readAsText' on 'FileReader': parameter 1 is not of type 'Blob'.

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

File: renovate.json
Error type: Invalid JSON (parsing failed)
Message: Syntax error: expecting end of expression or separator near "] "time

ADI import

Per-logbook Station Details

I did a few POTA activations this month. In the hopes of using Forester to log contacts, I set up one logbook for each park so that ADIF exports for POTA submissions would be easy. I ended up not using Forester for "live logging" due to the challenges of quick data entry in the QSO box, which is a separate issue.

I would like to enter QSOs from my paper logs, but one barrier is that Forester pre-fills the Logging Station details section with information from my home QTH. I would like to set Callsign, location, grid square, city/state/country, and a park identifier as default values at the logbook level, not the application level. Setting frequency and mode default values (but making it easy to change them) would also be nice when you're calling CQ.

PWA version upgrade

Now that I can see the revision in the app, I can tell the PWA is not being updated regularly. I should fix that. https://angular.io/guide/service-worker-communications#swupdate-service

Invite more editors to a logbook

Now that the data model supports more than one logbook per user (#126), there should be a way for users to invite additional editors to a logbook. This would support things like club logbooks.

Can't run `npm update`

I'm getting an error when trying to run npm update at 82cb2c9 with npm 9.3.1 and node 18.14.0:

...
6387 timing reifyNode:node_modules/google-gax/node_modules/@grpc/grpc-js Completed in 412ms
6388 timing reifyNode:node_modules/jose Completed in 417ms
6389 timing reifyNode:node_modules/google-gax Completed in 445ms
6390 timing reifyNode:node_modules/caniuse-lite Completed in 455ms
6391 timing reify:unpack Completed in 462ms
6392 timing command:update Completed in 34524ms
6393 verbose stack TypeError [ERR_INVALID_ARG_TYPE]: The "from" argument must be of type string. Received undefined
6393 verbose stack     at new NodeError (node:internal/errors:399:5)
6393 verbose stack     at validateString (node:internal/validators:163:11)
6393 verbose stack     at relative (node:path:1191:5)
6393 verbose stack     at /home/chris/.nvm/versions/node/v18.14.0/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:1093:21
6393 verbose stack     at Array.map (<anonymous>)
6393 verbose stack     at /home/chris/.nvm/versions/node/v18.14.0/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:1091:66
6393 verbose stack     at Array.map (<anonymous>)
6393 verbose stack     at [rollbackMoveBackRetiredUnchanged] (/home/chris/.nvm/versions/node/v18.14.0/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:1091:8)
6393 verbose stack     at [reifyPackages] (/home/chris/.nvm/versions/node/v18.14.0/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:239:31)
6393 verbose stack     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
6394 verbose cwd /home/chris/src/k0swe/forester
6395 verbose Linux 6.0.12-76060006-generic
6396 verbose node v18.14.0
6397 verbose npm  v9.3.1
6398 error code ERR_INVALID_ARG_TYPE
6399 error The "from" argument must be of type string. Received undefined
6400 verbose exit 1
6401 timing npm Completed in 34551ms
6402 verbose unfinished npm timer reify 1676390457583
6403 verbose unfinished npm timer reify:audit 1676390491621
6404 verbose unfinished npm timer auditReport:getReport 1676390491621
6405 verbose unfinished npm timer reify:unretire 1676390492102
6406 verbose code 1
6407 error A complete log of this run can be found in:
6407 error     /home/chris/.npm/_logs/2023-02-14T16_00_57_558Z-debug-0.log

Seems like maybe this is a bug in either reify or node itself? relative (node:path:1191:5) is trying to validateString but is actually passing undefined.

This is reproducing both in GitHub actions as well as on my workstation with identical results, so I don't think it's a local installation issue.

Error in log when opening prefs

ERROR TypeError: Cannot set properties of undefined (setting 'disabled')
at i.enableSaveButton (main.670fb76c19c20a76.js:1:1320767)
at main.670fb76c19c20a76.js:1:1322229
at rv (main.670fb76c19c20a76.js:1:2085971)
at Object._ [as next] (main.670fb76c19c20a76.js:1:2086180)
at _e.next (main.670fb76c19c20a76.js:1:2006532)
at ze._next (main.670fb76c19c20a76.js:1:2006199)
at ze.next (main.670fb76c19c20a76.js:1:2005885)
at main.670fb76c19c20a76.js:1:2003756
at b (main.670fb76c19c20a76.js:1:2019718)
at Hk.next (main.670fb76c19c20a76.js:1:2003593)

Search more than callsign

I don't want a massive form with a text box for each possible ADIF field. I'd like to model this after the GCP console search fields: type a field name, then colon, then the value to search in that field, and the search term visually becomes a chip. If the field isn't specified, then use full-text search.

This needs a revamp to the QSO service search facility.

Export selected QSOs

Instead of exporting all QSOs to ADIF, only export a selected subset. This supports events like contests, POTA/SOTA, etc.

Suggested by Devin KN6PHZ.

Blue-on-black text is hard to read

When editing a QSO, the changed fields change from white text on a dark gray background to dark blue text on a dark gray background. The latter is really hard to read, and since it happens as soon as a single character is changed, it's hard to verify that you're tying the correct value.

Opening QSO details dialog changes on/off times

This seems to be a result of QsoDetailComponent formatDate and parseDate using the same underlying object as QsoListComponent. I think I'll have to clone the object before messing with dates to avoid this behavior. Luckily, it doesn't save unless you actually edit and save the QSO.

Replace generated protobuf with hand-rolled Typescript interfaces

At first I was excited by the idea of using generated protocol buffer data structures in https://github.com/k0swe/adif-json-protobuf with this Angular project, but I've decided at this point that it's just too much effort for too little benefit. I'm writing this issue for posterity to document the problems I've had.

• I want to use protobuf as my interface description language, but I actually want to use proto3 JSON as my storage format in conjunction with Cloud Firestore.
• JS protobuf has toObject, but not fromObject. This makes it difficult to use JS protobuf as a wire and storage format because there's no generated unmarshaling code; that has to be written by hand.
• JS protobuf toObject is not the same as proto3 JSON because it doesn't support well known types. This means every time I use the Timestamp type, I need to do custom post-processing on the marshalled object.
• JS protobuf AsObject types, by design, are not nullable and contain default values. This mirrors how protobufs are used in every language binding, but it makes it quite difficult to do Cloud Firestore "patch" updates that don't overwrite fields that didn't change.

Given all this, I find myself writing a lot of translation and cleanup code around what is supposed to be a "batteries included" generated library. If I were using binary protobuf as a wire format I'd probably be fine, but trying to use this for proto3 JSON is surprisingly not effective. Instead, I'm going to copy the AsObject interfaces into my project, make the fields nullable and use that as my Cloud Firestore storage structure.

Fonts weren't downloaded in PWA

I tried installing the Forester PWA on Mac and then immediately going offline, and noticed that things like the Material Icons font weren't pre-downloaded. I need to make sure all important fonts are pre-cached in the PWA. That might involve hosting them myself, which would be fine.

WebSocket connection to 'ws://localhost:8081/websocket' failed

WebSocket connection to 'ws://localhost:8081/websocket' failed: main.670fb76c19c20a76.js:1

What's this looking for? The agent? Can it be toggled off?

Hotkey for QSO OK button

When entering or editing a QSO, it would be nice to have a hotkey for the dialog's OK button. I tried cmd-enter, shift-enter, and cmd-S, and hovering over the button in the hopes of a tooltip, none of which worked.

ADIF Import Doesn't Set Logging Callsign

I created a logbook for a POTA activation and named it after the park number and name. I then imported an ADIF file I created for that activation. The callsign in the "Logging Station" section of all the QSOs was set to the name of the logbook rather than the OPERATOR field from my ADIF file.

ADIF documentation for the OPERATOR field says

if STATION_CALLSIGN is absent, OPERATOR shall be treated as both the logging station's callsign and the logging operator's callsign

Customizable QSO list columns and order

Mode reference

ADIF 3.1.1 section III.B.9 specifies the "allowable" enumeration of modes and submodes. I don't want to be extremely strict about following that to allow for adoption of new modes before they enter the ADIF spec, but modes already in there should be stored and emitted appropriately. For instance, JS8 is actually a submode of MFSK. It'd be great to export ADIF files that follow the mode spec whenever possible.

Logging station details

While QTH profiles (#131) would be ideal to have, there at least needs to be a way to modify logging station details.

ADI export

Manual new QSO

Create a QSO from the web form

Import from ADI merges existing info

Similar to k0swe/forester-func#10, when importing an ADI file in the web app, we should merge any new information from the ADI like new QSLs, etc.