friedrichfroebel/cmanager

Cache manager to synchronize cache logs between geocaching.com and opencaching.de

Cache manager

The cache manager (cmanager) is a Java-based program which is able to manage GPX files and synchronize geocache logs from Geocaching.com to Opencaching.de. It therefore loads a GPX file with the users cache founds (e.g. myfounds.gpx). After configuring an OKAPI token in the settings, the user is able to match his/her founds against caches listed on Opencaching.de.

Further information in German:

• https://wiki.opencaching.de/index.php/Cmanager
• https://forum.opencaching.de/viewtopic.php?t=4348

Note: If cmanager fails with an exception, you should try to update your local Java installation to the latest version at first. This especially holds true for SSL errors, but might cause other problems as well.

Important note: This application tries to support you with logging duplicates as best as possible. The results might be wrong due to the used heuristics, so you might end up with multiple results for one GC cache or no results at all. There is no guarantee that the results are correct, so you should be careful with logging by ensuring that this actually is a valid duplicate. (This is one of the reasons for not automating the synchronization by providing some batch processing for sending the actual logs.)

License & Source Code

cmanager is distributed under the terms of The GNU General Public License v3. The sources are available on GitHub.

This version is a fork of the original repository by Samsung1. While the original version discontinued support for the log transfer functionality itself, this fork tries to keep it working. Additionally some problems of the original version have been fixed.

Distribution/Download

Releases are published on GitHub as "Releases".

Besides the regular release, there is a pre-release as well. This will be built by GitHub Actions on every change to the master branch. Please note that these files (named cm-<hexStringOfTheCommit>.jar) are development versions which might have additional bugs or provide undocumented changes/features. Use them at your own risk!

Building from Source

Prerequisites

• Java development kit (JDK) in version >= 10. The minimum required Java version to run the application is Java 8 nevertheless, but the Compatibility.java file depends on a method introduced in Java 10.
• You need to provide API keys for compiling cmanager. See next section for details.

API keys

Request your personal API keys for the supported OpenCaching sites, currently:

• opencaching.de OKAPI signup

Copy templates/oc_okapi.properties to the root directory of the Git repository. Then edit oc_okapi.properties and insert your keys.

Building with Gradle

Run gradle build from the root directory of the Git repository (or use ./gradlew build if you do not have Gradle installed locally).

JAR

To create a JAR file, run gradle jar (or ./gradlew jar). The JAR file will be located in build/libs.

Java-independent packages

There is experimental support for bundling the application in a way that no local Java installation is needed for executing it.

To create the corresponding image, run gradle jpackageImage (or ./gradlew jpackageImage). The image will be available inside the build/jpackage/cmanager directory. You might want to put this directory into a dedicated archive file for redistribution using the jpackageImageZip task. Please note that this an incubating feature of Java 14, so at least Java 14 is required and this feature might break due to API changes.

Usage

Prerequisites

• Java in version >= 8. I recommend you to use at least Java 10. If you use the jpackage image, no own Java installation is needed.

Starting the application with Gradle

Run gradle run (or ./gradlew run) from the root directory of the Git repository.

Starting the application JAR file

Run java -jar cm-0.1.2.jar from the directory containing the JAR file. Replace 0.1.2 with the corresponding version number.

Starting the Java-independent package

Double-click on cmanager.exe (on Windows) from the directory containing this package. Please note that this version is not being distributed in the release section at the moment.

cmanager's Issues

Improve CI handling

While introducing automated master builds, I did not find a good way to replace existing releases. At the moment the old release has to be deleted first (deleting the pre-release itself, then the tag) and then Travis will be able to push the new pre-release.

Ideally we could have a workflow where the CI deletes the old tag in the first step or just updates the existing tag to point to the new commit and then replaces the asset itself.

Switch from Joda-Time to `java.time`

The code currently uses the Joda-Time library. As we are targeting at least Java 8, we should be save to move it to the built-in java.time module.

Some hints about this are shown here.

Retrieve owner name and cache size for duplicate search

When searching duplicates, the owner name and the cache size are not considered for now as Okapi.getCache (called by Okapi.getCachesAround) does not retrieve them. We already check for null values in the GeocacheComparator class, but we might want to test whether including the owner name and cache size on the initial comparison will improve the overall detection ratio.

Some problems might arise from the different naming schemas used on both sites, especially when including the cache size, so we need some careful evaluation or correct mapping. Owner names might differ as well.

OKAPI test cannot initialize OAuth client due to OC timeouts

Running the Gradle tests, especially cmanager.okapi.OkapiAuthedTest, does not work at the moment. (cmanager.okapi.OkapiRealInteractionTest has not been working for quite a while as well due to permanent internal server errors on the test server, so I am going to ignore it here.)

Apparently, fetching the OKAPI user authorization page throws some strange timeout:

[...] <h1 style='clear: both'>Anfrage abgelaufen</h1>                <p>Die Anfrage ist wegen Zeitüberschreitung abgelaufen. Bitte versuche es noch einmal.</p>                    </div>    </body></html>

According to the log, at least three requests to the OKAPI page have failed beforehand in this case:

[Test worker] INFO org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec - Recoverable I/O exception (org.apache.hc.core5.http.NoHttpResponseException) caught when processing request to {s}->https://www.opencaching.de:443
[Test worker] INFO org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec - Recoverable I/O exception (org.apache.hc.core5.http.NoHttpResponseException) caught when processing request to {s}->https://www.opencaching.de:443
[Test worker] INFO org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec - Recoverable I/O exception (org.apache.hc.core5.http.NoHttpResponseException) caught when processing request to {s}->https://www.opencaching.de:443

I am not sure whether this is related to the migration of the Apache HttpClient library from version 4 to 5, but the error message seems to indicate that this is not related?

Warn on closing unsaved lists

If a list has been modified and the program is being exited, there is no warning about it and all changes will be lost.

To reproduce, either open a larger GPX file, delete one cache from it and exit the application, or create a new list, add at least one cache from another file to it and then exit the application (or only close the list).

To avoid this, we should warn the user using a corresponding dialog box and provide a "back to list" and "ignore changes and close" button.

Allow log sync with entered username only

Synchronizing logs does not check whether a corresponding GC username has been set, but without this checking for candidates does not really make sense as no log will be shown in the "copy log" dialog for the duplicates.

Additionally we probably want to show only caches which actually have a log from the given user on synchronization (otherwise "Find on OC" does not make much sense). This should avoid empty log lists within the "copy log" dialog.

Evaluate migrating CI from Travis to GitHub Actions

Currently we are using Travis CI for automated builds. We might consider replacing it with GitHub Actions.

With this we might be able to simplify the current pre-release process (#7). Additionally this might allow us to provide Java-independent releases for Windows systems (by bundling the JRE using the jpackage tool from Java 14) iff GitHub Actions allow protecting secrets on Windows hosts as well (which Travis does not for now).

Add support for multiple OKAPI installations

We may want to support multiple OKAPI installations. In the best case they should be switchable during runtime without having to settle with one specific installation. At first this should make it easier to test some features which involve modifications (like submitting logs) by using a public/private test installation, but could be helpful for other cases as well.

For this we have to evaluate the shadow list API (or its planned successor) as well as it only supports the German OC site for now.

There is a PR for this on the original project (RoffelKartoffel#24), but this probably needs some modifications to satisfy our requirements.

Blacklist feature

Allow the user to move selected (GC or OC and pair of GC/OC) caches to a ignore list which will be ignored when syncing the logs.

Date comparison for events

Event duplicates might produce lots of false positives. To avoid this, we should compare the log date with the event dates of the matches (maybe we have to consider some threshold due to timezones and due to some late logs?) The corresponding OKAPI field is date_hidden for type == "Event".

Own class for date-specific functionality

Move the date-specific functionality to an own class with static methods. This should make changes (like #4) and automated tests for them easier. At the moment this should mostly affect cache invalidation for the map tiles and empty search requests.

Rework automated release/pre-release builds

The automated release and pre-release builds currently use the https://github.com/actions/create-release and https://github.com/actions/upload-release-asset actions which are considered unmaintained.

It should be evaluated which of the recommended replacement packages is suitable for the current workflow (simple automated messages for pre-releases, automated uploads of assets for pre-releases and regular releases).

Strip cache name on comparison

The GeocacheComparator class uses the original names. While this works in most of the cases, it fails in some cases where the owner added a whitespace after the name for example. By trimming the cache name on comparison, we should be able to improve the detection ratio while not producing false positives.

To test this behaviour, we should probably use another comparator-specific test class to improve overview (the current test class already looks rather messy).

Improve handling of empty searches

When an empty duplicate search is observed, the corresponding GC code is being cached by saving a file inside the ~/.cmanager/cache folder in a dedicated subdirectory. When checking if a GC code previously led to an empty search, we determine the file age. Currently the file age is chosen randomly for each cache access and resides in the range of 4 to 13 months.

We might want to change this. Especially cache invalidation could probably be made deterministic by choosing a fixed age.

Improve test coverage

Test coverage is rather low at the moment, so refactoring code is prone to errors at the moment. We should add more tests to make it less risky.

Handle log passwords

When logging a cache that requires a password, this will silently fail at the moment. We should either provide a corresponding input field or just report a message to the user if this is the case.

I already started working on this in c130efa to retrieve the req_passwd value for the cache and providing an inactive interface to submit the password and parse the output. As I do not want to test this in production, I did not yet finish it.

Use JSON for OKAPI everywhere

The OKAPI handlers should all use as document-based approach with GSON. This is currently not the case for the home coordinate retrieval (getHomeCoordinates).

Msg: The shadow list retrieval has been disabled temporarily as the API endpoint seems to be broken for now.

JFYI, not sure if this is an issue at all. Thanks a lot for maintaining this program!

---

$ java -jar cm-0.5.1.jar 
The shadow list retrieval has been disabled temporarily as the API endpoint seems to be broken for now.
The shadow list retrieval has been disabled temporarily as the API endpoint seems to be broken for now.

Division by zero when filtering empty lists

When trying to filter an empty list, an "Division by zero" error (ArithmeticException) is thrown. The culprit seems to be https://github.com/FriedrichFroebel/cmanager/blob/master/src/main/java/cmanager/list/filter/FilterModel.java#L20, which leads to 0 cores if the list has size 0 and therefore the division fails. We should probably check if the list is empty beforehand and return the original list if it is empty.

Steps to reproduce:

1. Create a new empty list.
2. Try to apply a filter.

Small decimal coordinate values are sent to OKAPI in scientific notation

Coordinates like E 000° 00.023 (0.000383 in GPX file) are not serialized correctly (example geocache: https://www.geocaching.com/geocache/GC6T8FW_4587-est-smx). They will be sent in scientific notation (3.83E-4) to the OKAPI in scientific notation, which responds with an error code 400.

We should make sure that these values are serialized correctly, id est we should send 0.000383 in our example. This requires a custom formatter which should be locale-independent (otherwise we get a comma instead of a period as a decimal separator on some systems, for example on German ones).

Future of the `.form` files

There currently are two .form files (XML-based layouts) for the cache panel and the log panel depending on NetBeans IDE to transform them into Java code. We should think of a proper way to handle these files in the future as I do not really like IDE-specific stuff.

Add debug logging

We might want to introduce some debug logging, especially for the duplicate search (data retrieval, similarity etc.) to make problem/restriction search easier. java.util.logging.* should be sufficient.

Search radius too small for Mystery and Multi

The search radius for mystery and multi caches seems to be too small at the moment as modified coordinates appear to cause problems with no matches being found (although the correct OC listing contains the GC link, but this does not help if the corresponding OC cache is out of range).

On GC there had been a limit for the maximal distance between the listing and actual cache coordinates of a mystery cache corresponding to 3.2 km (2 miles) (see here, thanks to j-he). For multi caches I do not know of such a limitation. At the moment the search is done with a radius of 1 km for mystery caches and 50 m for multi caches.

When changing anything about this, we should make the radius as big as useful, but should always keep in mind that the load on the OKAPI itself stays small enough.

Previously the shadow list API handled most of these caches without the need to do a distance search afterwards, but it does not work at the moment (see #5). The OKAPI does not allow searching based on the GC code as well. We therefore would need to either maintain a corresponding OKAPI dump at the user's device (which is not practical and discouraged) or provide an own API for this with regular updates (while possible, this would mean extra costs and work) - but only if we want to further use this (actually helpful) part of the application.

Program freezes on some caches when opening the log copy dialog

After synching the geocacging.com and opencaching caches, all geocaching logs were transphered to opencaching through the duplicate dialogue window except two caches. If I double-clicked on the opencaching-entry as demanded, a new window popped up and the program freezed. This occured every time I retried to double click on the caches. The concerned caches were:
OC4916 / GC86B
OC10AD2 / GC516EE

The Error was reproducable, but since I logged these caches manually, now i can't get that error anymore

Show archive log date for archived caches

To better estimate whether logging an archived cache, we might consider showing the date of the archive log. This is especially recommended when the log to post is dated after the archive event.

The OKAPI does not provide us with this information (the date of the last modification does not have to be correct), so we would probably have to search the log entries of the cache until we find an archive log if the cache actually is archived at the moment.

Provide option to keep size of CopyLogDialog

This feature request has been reported in the Opencaching.de forum:

Changing the size of the copy log dialog for being able to improve readability will not persist when pressing "Return" and opening the next geocache entry. It would be better to remember the size in this case. We could either save this for the current session only or store it inside the registry to remember it forever (then an option for disabling/enabling this behaviour might be useful).

When having submitted multiple logs, there have been reported some memory issues as well. I am not sure for now, but this might be related to calling THIS.setVisible(false) instead of closing the dialog/frame.

Future of the shadow list

The shadow list feature with not publicly shown duplicates on GC and OC has recently been disabled due to problems with the API endpoint provided by Opencaching.de, namely https://www.opencaching.de/api/gc2oc.php. The endpoint is currently returning empty files instead of valid GZip archives with the requested data.

It seems like the corresponding endpoint has been removed from the Opencaching.de code (development version) on 2020-01-12 (commit 1720ea5), so I assume that we will not be able to use this endpoint in the future anymore. According to a commit from 2016-01-18 (commit 89d813c) this endpoint had been introduced especially for cmanager, but as the original version does not further support Opencaching.de, some issues/functionalities have been discontinued there.

In fact there is an older API being an alternative to the OKAPI (in reality it probably is the predecessor) returning XML which should contain the manually set shadow entries as well. When performing a query for all caches and looking at the gccom and gccom2 attributes for the waypoints, they seemed to be identical for each entry. In fact sometimes the shadow entry gccom2 should have been different (when the gccom attribute is empty, which means that this GC code has not officially been set by the owner), but both attributes match in all cases. According to the commits, the XML API has been removed from the code (development version) on 2020-01-12 (commit e02ca45) as well.

We will have to monitor this and are probably safe to remove the corresponding code later on.

Clean up root package

The root package directory (cmanager) should be cleaned up by moving the corresponding files to suitable subpackages. This is especially true for the CacheList*.java files, but applies to the other files (besides of Main.java) as well.

Compare Event date (=hidden date) when syncing Event-Caches

In some cases the synchronization of event caches is not successful, especially if there is a series of events which always takes place at the same location.
Therefore I would suggest: in case of Event-Caches the event date (=hidden date of the cache) should be compared to prevent mis-matches.

Unreliable duplicate search in some cases

The duplicate search is unreliable in some cases as the results might differ when running it two times in a row.

This is caused by the OKAPI runtime cache. On the first run, the cache size and owner name are null. If there is a match, the details like the size and owner will be added to the geocache instance. When running the duplicate search again, the cached geocache instance (the OC cache) will be returned which corresponds to the enriched version as this is the same object which we have added the details to beforehand. Now the comparison might produce different results as the cache size and owner name are taken into account as well. One example for this behaviour is GC26VJD/OC2ED4.

We should therefore always feed the basic object into the geocache comparator when searching for duplicates. For this we might want to add a new method getBasicCopy (or similar) to the Geocache class which returns a basic copy of itself and use the result of this method for feeding it into the comparator. By doing this, we should always get the same results from the duplicate search when running it multiple times.

Especially the part with the cache size and the owner name are related to #25, but this definitely are two different issues we should tackle on its own for now.

Provide proxy settings in GUI

A proxy has to be activated using the corresponding terminal parameters at the moment. This is not very intuitive, so we might want to consider adding it to the settings dialog of the GUI. This is related to #12.

Provide link to log after submission

When a log has been submitted successfully, the corresponding button gets removed. We could use this space to display another button or a clickable hyperlink to open the newly created log entry in the browser to check that everything is okay or to allow uploading log photos. The corresponding log ID is already set inside the response document created by GSON.

Better code documentation

Most of the code is currently missing documentation and is deeply nested in some places which makes it hard to read/understand.

Exclude own caches in duplicates

Due to some circumstances with adoptions, one might get a duplicate for its own OC cache. As proposed in the OC forum we might want to exclude such caches based on the user/owner ID. Nevertheless this should be a rare case (and when logging the user should be able to spot this anyway).

Configurable threshold value for comparison

We might want to make the threshold value in GeocacheComparator configurable from the settings in the GUI as well. If this value is too low (for example less than 50 %), we should probably show a warning or even disallow this as this might produce too much false positives.

Display cache status in DuplicateDialog

Display the cache status inside the duplicate dialog, as proposed in #12 (comment).

Follow XDG base directory specifications

The application does not follow the XDG base directory specifications for now, as it saves its cache inside $HOME/.cmanager/cache. Correct would be $HOME/.cache/cmanager.

The old cache should probably be migrated, while deleting the old directory afterwards.

Specifications

• https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html
• https://wiki.archlinux.org/title/XDG_Base_Directory

Possible implementations

• https://github.com/dirs-dev/directories-jvm: Maintained, but with Windows issues due to using a PowerShell script there. As far as I am aware, our user-base is mostly using Windows.
• https://github.com/omajid/xdg-java: No recent commits, but most basic implementation without using PowerShell scripts.

Add test support for CI

There currently are no automated tests on Travis CI. While I would be able to enable these for commits to the master branch, they would not work for pull requests as I would have to expose an Opencaching.de password to the public due to Travis not allowing access to secret variables on external contributions (which perfectly makes sense to avoid exposing it due to malicious contributors).

We might avoid this by mocking network calls. There would be the option to set-up an own site with the Opencaching.de server code to use for each test, but the tests would take a lot of time afterwards. Using the real site for each commit on the master branch and for pull requests might introduce additional traffic which I am not sure either.

Additionally all tests requiring write access (or existing logs) at Opencaching.de have currently been disabled. This is the case as I do not want to do development tests on the production server. We might come up with a appropriate solution for this in the same step (or in another one). (There is a public test server, but it currently is in maintenance mode and OKAPI e-mails are not sent for now.)

Replace smileys

GC allows users to input 20 different smileys using square brackets. When transferring the log, they will not be replaced at the moment.

OC.de provides another set of smileys, being inserted into the log using

<img src="resource2/tinymce/plugins/emotions/img/smiley-cool.gif" alt="" border="0" width="18px" height="18px" />

Use the look and feel of the system

The GUI elements are using the "default Java style" at the moment. We might want to adapt this to a system-specific version according to the docs, as proposed in #12 (comment).

Sync own listings between GC and OC

Please forgive me if this is already supported but I am unaware about usage:

AFAIR the old cmanager had the possibility to sync own listings between both platforms.
I could however not find out how to use it or whether this is really (still) included.

As I consider double listings (GC+OC) meaningful to support OC also with caches I already listed on GC only at the moment, but can't afford manual sync initially and on each change, such feature would be really helpful.

I cannot support with implementation but would be willing to test such implementation.

NullPointerException within Okapi.getCachesAround and OcUtil.findSingleGeocache

There have been reports about NullPointerExceptions in the mentioned methods if there are no search results. This seems a bit strange as this was the first report about this, but I would have expected more reports in this widely used features. The reports have been about Java 15, so this might be related to changes introduced there and should be investigated further as it might affect other parts of the source code as well.

As a first step, we should probably add a test for this as there is none at the moment. Some quick fixes for the reported source positions have been committed already.

Handle OKAPI errors correctly

Currently OKAPI errors are not really handled (except status codes) which leads to NullPointerExceptions while accessing the objects serialized by GSON. This should not really be a problem as these errors should only indicate errors inside the application code, but makes debugging rather hard.

To do this, we should add a new object error to the response classes (in cases where there are no response classes, we might want to introduce them) with the error properties. The corresponding OKAPI documentation is available at https://www.opencaching.de/okapi/introduction.html#errors.