fxstein/goprox

The missing GoPro data and workflow manager for macOS

goprox's Issues

Feature: Automatic Imports

Automatic activation of the import workflow when a GoPro MicroSD card is inserted.

Leverage launchd to run goprox when a GoPro device is mounted. Since launchd has no ability to differentiate volume mounts, goprox must check if the mounted volume is a GoPro device SD card.

References

https://macresearch.org/tutorial-backups-with-launchd/

Feature: Improved testing

• Add archive processing to testing
• Add additional GoPro metadata to originals directory
• Exclude various markers and files from test repo tree

Feature: Test Storage Hierarchy before Operations

Make sure to test the availability of all storage components - especially links - before executing any operations. In complex storage environments where the library components are distributed over multiple volumes or disks, a missing volume will create runtime errors that might get caught too late.
Important to check for key folder availability before engaging in any operation that copies, moves or writes media files.

Feature: Delta patch compress optional files

The storage requirements for goprox are very significant when all layers of the library are being kept for future re-prosessing.

There are basically 3 copies of data: archive, imported and processed

Both archive and imported are optional lineage copies that allow a user to go back to the originals coming off a camera and re-process the entire workflow. With early versions of goprox this is desirable as logic changes and even bugfixes can easily be applied to the original media.

Since the delta between imported and processed media is metadata only - we are not resampling or recompressing the media files - delta patches could be leveraged to replace media file with significantly smaller delta files, while allowing to restore the original file later on.

Initial testing has resulted in a 99% reduction in storage required to hold imported delta patch files, compared to the original media files.

Example:

xdelta3 -S djw -s /Users/oratzes/goprox-test-moved/imported/2022/20221011/20221011151401_GoPro_Hero10_2442_G1541642.JPG /Users/oratzes/goprox-test-moved/processed/JPEG/2022/20221011/P_20221011151401_GoPro_Hero10_2442_G1541642.jpg P_20221011151401_GoPro_Hero10_2442_G1541642.xdelta

produces a 1.4kB delta file compared to the original 4.7MB image. This would allow users to keep the entire imported media path while consuming 99% less storage than a full copy of the data.

goprox needs to be able to generate the delta patch files, replace the original files in imported and restore them on demand as needed for future processing.

Update: Add new GoPro Labs v1.20.70 FW for Hero 11

Update the GoPro Labs firmware for the Hero11 to the latest November 2022 release.

https://community.gopro.com/s/article/GoPro-Labs?language=en_US

Feature: AWS Glacier support

In addition to #10 add the ability to archive off no longer actively used media files to AWS S3 Glacier storage leveraging the AWS CLI

brew install awscli
aws glacier help

Bug: Make test option local timezone independent

Turns out the file creation time for files without valid EXIF data (NODATA) changes dependent on in which timezone the developer machine is setup in. As such the file names change and the test fails.

[2022-12-18 10:14:47] Error: Test failed! 
[2022-12-18 10:14:47] 
[2022-12-18 10:14:48]  .../2022/20220214/20220214204307_NODATA_GH010739.MP4 | Bin 597806 -> 0 bytes
[2022-12-18 10:14:48]  .../2022/20220214/20220214204307_NODATA_GX010093.MP4 | Bin 14966973 -> 0 bytes
[2022-12-18 10:14:48]  ...20306001216_NODATA_IMG_20220306_001216_00_004.dng | Bin 18535575 -> 0 bytes
[2022-12-18 10:14:48]  ...20306001216_NODATA_IMG_20220306_001216_00_004.dng | Bin 18539894 -> 0 bytes
[2022-12-18 10:14:48]  .../20220214/P_20220214204307_NODATA_GH010739.mp4    | Bin 601364 -> 0 bytes
[2022-12-18 10:14:48]  .../20220214/P_20220214204307_NODATA_GX010093.mp4    | Bin 14970530 -> 0 bytes
[2022-12-18 10:14:48]  6 files changed, 0 insertions(+), 0 deletions(-)

Feature: Propagate and Collect Deletes

Imported and processed files of goprox are mainly imported into other image and video processing platforms. For images, a common path is to import all processed files into Apple Photos. This allows for easy browsing, filtering, and sorting (based on goprox metadata and tags). Once imported it is very common to review the images for favorites, decent shots as well as outright junk. Often bad images get deleted quickly from Photos to reduce the amount of overhead to photo libraries.

Once deleted - sometimes as much as 80-90% of imported images, the problem becomes that the goprox library continues to hold the deleted images and future imports create the problem of re-importing previously deleted images, especially after 30-40 days when the deleted items list in Photos gets purged.

However it is this list of deleted items, that is programmatically available via Photos internal SQLite database for said 30-40 days. As such it would be easy to extract the list of deleted images and media files from photos, apply some filters and in return remove them from the processed or even imported library paths. The originals would still be available in the original SDcard images if there is ever a need to recover a deleted file.

In order to make this permanent, goprox will need to keep an ever-growing list of deleted media files and eliminate them from future processing as well as from the current library storage.

The Photos SQLite store is commonly found here:

~/Pictures/Photos\ Library.photoslibrary/database/Photos.sqlite

The query to extract recently deleted media files from Photos would be like this:

SELECT 
  zAddAssetAttr.ZORIGINALFILENAME AS 'zAddAssetAttr-Original Filename'
FROM ZASSET zAsset
  JOIN ZADDITIONALASSETATTRIBUTES zAddAssetAttr ON zAddAssetAttr.Z_PK = zAsset.ZADDITIONALATTRIBUTES
WHERE zAsset.ZTRASHEDSTATE = 1
ORDER BY zAddAssetAttr.ZORIGINALFILENAME;

and results in something like this:

IMG_0922.PNG
P_20220923092015_GoPro_Max_6013_GS__4797.jpg
P_20220923092019_GoPro_Max_6013_GS__4798.jpg
P_20220923092026_GoPro_Hero11_8909_GOPR0089.jpg
P_20220923092030_GoPro_Hero11_8909_GOPR0090.jpg
P_20220923092033_GoPro_Hero11_5131_GOPR0100.jpg
P_20220923092037_GoPro_Hero11_5131_GOPR0101.jpg
P_20220923125107_GoPro_Max_6013_GS__4802.jpg
P_20220923125118_GoPro_Hero11_4632_GOPR0075.jpg
P_20220923125120_GoPro_Hero11_8909_GOPR0094.jpg
P_20220923125127_GoPro_Hero11_5131_GOPR0105.jpg
P_20220923125157_GoPro_Max_6013_GS__4803.jpg
P_20220923125209_GoPro_Hero11_4632_GOPR0076.jpg
P_20220923125210_GoPro_Hero11_8909_GOPR0095.jpg
P_20220923125217_GoPro_Hero11_5131_GOPR0106.jpg

It contains any recently deleted media files and would need to be filtered for goprox specific files.

To store the list of deleted files a new deleted substructure would need to be introduced as part of the library. Deleted images could be grouped into list files by year for ease of processing.

Update: Add new GoPro Hero 11 Firmware v1.20

Update to latest GoPro Hero 11 firmware:

https://gopro.com/en/us/update/hero11-black

Documentation: Update ReadMe with latest features and updates

Update README.md to reflect the recent changes from:

• #12 Feature: Test Storage Hierarchy before Operations
• #14 Feature: Import from Archive
• #19 Feature: exiftool error handling

In addition, correct the 360 file handling description that states incorrectly that 360 files become mp4 files after processing. This was an earlier attempt that turned out to be an unworkable solution.

Workflow: Implement git-flow Model

As the project attracts additional collaborators it will become necessary to implement a more formal contribution model and migrate away from the quick and dirty everything-happens-in-main model.

We will be implementing the git-flow model as described here: github-flow

Bug: Fix GitHub Actions shield

Fix broken build badge due to changes in API: badges/shields#8671

Feature: GPSTime

Implement gpstime feature to allow the use of the GPS timestamp plus the timezone information from the geonames feature for timestamping of the processed media files. This would enable the use of the GPS time information for timecode synchronization of multiple camera angles in apps like DaVinci Resolve.

Consider adding an additional prefix to the processed files to clearly identify GPS timed media files.

References

This would be similar to the GoPro Labs GPS timecode implementation inside the camera firmware but with the big difference that goprox is timezone aware and as such can correctly set the entire timestamp and not just minutes and seconds - which is problematic at hour and day boundaries.
https://gopro.github.io/labs/control/gpssync/

Bug: Permission Denied during --firmware processing

As of Ventura 13.4 and Homebrew 4.0.22-6-gf71294d getting the following permissions error when performing --firmware task:

➜  goprox --version
goprox v00.52.00
➜  goprox --firmware labs 
GoProX started... 
Checking firmware... 
_firmware:34: permission denied: /opt/homebrew/Cellar/goprox/00.52.00/bin/firmware.labs/HERO11 Black/H22.01.02.10.70
No firmware files found at /opt/homebrew/Cellar/goprox/00.52.00/bin/firmware.labs/HERO11 Black
GoProX processing finished. 

same with

➜  sudo goprox --firmware labs 
GoProX started... 
Checking firmware... 
_firmware:34: permission denied: /opt/homebrew/Cellar/goprox/00.52.00/bin/firmware.labs/HERO11 Black/H22.01.02.10.70
No firmware files found at /opt/homebrew/Cellar/goprox/00.52.00/bin/firmware.labs/HERO11 Black
GoProX processing finished. 

Feature: Expand relative paths in logging output

Display both relative and absolute paths in logging output to aid debugging and failure detection.

For example:

_info "Source: $source ($(realpath $source))"

Feature: Create hidden markers on GoPro cards

Create and leave hidden markers on GoPro media cards once various goprox processes have been completed successfully.

For example:

.goprox.archived
.goprox.imported
.goprox.cleaned
.goprox.fwchecked

This enables a quick and easy way to check if and when goprox has performed certain functions against the media cards. The timestamp represents the time the process was last successfully completed.

Documentation: Create initial help screen and man pages

Create an initial version of a help screen that explains the various commands and options.
Do the same with an initial man page for goprox. Also add to homebrew installer.

Feature: Multi Tier Storage Support

Currently goprox only supports a single storage hierarchy even though portions of the library can be distributed across multiple storage volumes by leveraging links for subfolders of it. For example imported can be located on one dedicated SSD while processed can be located on another SSD and archive can be mapped to an internal drive. Distributing the media file structure can help with growing libraries and leverage the full throughput of current MacBooks or similar.

However, in cases where processing is performed in multiple physical locations - for example, mobile - on-the-road capture and import vs home/office post-processing a more complex storage hierarchy might be required.

While out on set or on the road, capture from SD cards needs to happen rapidly and often involves minimal infrastructure, commonly with external HDD or SSD devices.
When back in the production studio or office, additional storage devices with much larger capacities are often used to collect all the media files for an entire or even multiple projects.

Currently, the workflow is rather manual and requires goprox configurations that are different from field/road to office/home which in turn requires manual copying and transferring of media.

Proposed solution

Add the principle of mobile vs home configuration setups to goprox and select the environment at runtime or potentially even autodetect the environment based on the availability of certain storage devices connected to the computer.
In Mobile mode archive/import and optionally process media data into staging devices eg external SSDs connected to the laptop.
Once back in the office/home migrated these staged media files into the permanent storage hierarchies.

Related

In addition, expand this capability through the support of AWS S3 Glacier storage #11 for long term archival of no longer actively required media files

Feature: Timeshift

Implement time shift feature to modify the time when a media file has been created.

This could be implemented based on GPS time and timezone see geonames feature or by time offset.

Need to determine how to best select the media files that should be shifted and how to persist that time shift across multiple goprox runs.

References

Feature: Camera firmware updates

Updated firmware from GoPro added to goprox firmware list

Feature: Cleanup parameters and add filter options

Create the option to filter processing of goprox by date and other metadata. Leverage exiftool -if and -if4 to perform the filtering

Update: Add various new GoPro firmware versions

Latest firmware updates from:
https://gopro.com/en/us/update

GoPro Hero 11 Black H22.01.02.10.00
GoPro Hero 11 Black Mini H22.03.02.30.00

Latest labs firmware updates from:
https://gopro.github.io/labs/

GoPro Max H19.03.02.00.75
GoPro Hero 11 Black H22.01.02.10.70
GoPro Hero 11 Black Mini H22.03.02.30.70

Test and verify new exiftool version 12.50

The new production release 12.50 just became available through home-brew. Initial testing reveals some minor changes in processed files output. Need to verify and make sure nothing broke.

Feature: Add basic support for DJI drone import

Enable the import from DJI drone sd-cards similar to the GoPro media.

Since there is no version text file on the sd-cards of DJI drones, it requires some directory/filename logic to identify as a DJI drone.

Won't be able to recognize the drone type or serial number, but at least the fact that it is DJI drone media.

➜  tree -d
.
├── DCIM
│   └── 100MEDIA
├── LOST.DIR
└── MISC
    ├── GIS
    ├── IDX
    └── THM
        └── 100

9 directories

➜  MISC tree   
.
├── GIS
│   └── dji.gis
├── IDX
├── IDX_BLOCK
└── THM
    └── 100
        ├── DJI_0001.SCR
        ├── DJI_0001.THM
        ├── DJI_0002.SCR
        ├── ...
        ├── DJI_0524.THM
        ├── DJI_0525.SCR
        └── DJI_0525.THM

Bug: Allow cleanup with import and without archive

Contrary to the readme, cleanup is only allowed when archiving.

Archiving and importing everything is taking a lot of space.

It would be great if cleanup was allowed for import only as well. Maybe with an additional option if import is considered dangerous and could damage file?

Bug: Fix firmware processing

Fix a few issues with firmware processing:

• Cope with situations where no firmware files are available without breaking the logic
• Fix path to /MISC/version.txt when firmware check does not run against the current directory

Feature: Enhanced logging

Add the ability to make logging configurable and apply to exiftool output dynamically.

Leverage -v and -q options to increase or decrease exiftool output in line with the overall logging level for goprox

Feature: Downsample videos

Create the ability to downscale and sample video files. Create lower resolution and time limited previews that are significantly smaller in size and can be easier shared with various applications and platforms.

Leverage ffmpeg to downscale videos to e.g. 720p, lower frame rates and quality and limit to the first 30 sec of any video. Place preview samples in dedicated preview folder structure.

Feature: Improved archive processing

Refactor goprox --archive processing to read the camera model and serial number from /MISC/version.txt rather than the first media file encountered. This creates a more consistent behavior and avoids empty archive file names in case no matching media file can be found.

Also, exclude newly created hidden .goprox markers from the archive.

Feature: Support Windows platform

Currently goprox is only being developed and tested under MacOS. With the support of zsh in Windows 10, the same workflow should be easily possible under Windows.

Looking for help from someone with Windows experience and the ability to test.

Feature: Enhance system level logging

Integrated goprox logging with macOS system logs leveraging logger. This is required to support the auto mount feature running goprox in the background whenever a new mount point is being created.

Bug: Archive uses current directory instead of source directory

The _archive_media function uses the current directory instead of source directory when checking for camera model and serial.

On line 572, find -E . should be changed to find -E $source

Bug: Incorrect file creation dates

Both import and process seem to be setting the incorrect FileCreateDate. Imported and processed files get the date the process was run and not that of the original image creation date.

Needs further analysis.

Feature: exiftool error handling

Need to implement an exiftool error handling strategy. While some errors like skipping existing files are ok in most cases, other errors are not. This applies to all calls of the exiftool but specifically where followup actions like --clean depend on the successful completion of a prior exiftool step.

This was raised by @giga in #18. At the minimum we need to capture the exit status of every exiftool run and make it available as part of other logic for decision of whether to proceed in the goprox logic or not.

Feature: Import from Archive

Imports and archives are normally performed directly from the root of the GoPro sdcard. In some cases, it is required to import an archive instead. This is especially helpful when mobile collection is limited to creating archives and further processing is then performed at home or in the office.

goprox creates archives like:

20220930092409_GoPro_Hero10_2442.tar.gz
20220930093920_GoPro_Hero10_8034.tar.gz
20220930103800_GoPro Max_6013.tar.gz
20220930103807_GoPro_Hero11_8909.tar.gz
20220930103816_GoPro_Hero11_5131.tar.gz

Currently, an archive needs to be uncompressed into a directory and then goprox can import from there.
If the source is an archive file *.tar.gz the import process should happen automatically without the need for a manual decompression step.

Bug: Check for errors when importing files

Exit status of exiftool in _import_media function is not checked. This is dangerous especially since cleanup can now be used with import.

Feature: Single file summary

Create the ability to pass a single image file to goprox (even without the path) and provide the full path and summary metadata together with warnings if e.g. system create date is not matching the EXIF create date.

goprox 20221028112412_GoPro_Hero11_5131_G0327016.JPG

Misc: Create FreeBSD Port

I wanted to run this within a FreeBSD jail to reorganize a bunch of media already on a TrueNAS appliance.

Since macOS and *BSD share some heritage this was actually much easier than I thought it would be.

# $FreeBSD$

PORTNAME=	goprox
DISTVERSION=	00.52.00
CATEGORIES=	multimedia

DISTVERSIONPREFIX=	v

MAINTAINER=	[email protected]
COMMENT=	Missing GoPro workflow manager for zsh
WWW=		https://github.com/fxstein/GoProX

LICENSE=	MIT

RUN_DEPENDS=	exiftool:graphics/p5-Image-ExifTool \
		jq:textproc/jq \
		zsh:shells/zsh

USES=		shebangfix

USE_GITHUB=	yes
GH_ACCOUNT=	fxstein
GH_PROJECT=	GoProX

SHEBANG_LANG=	zsh
SHEBANG_FILES=	goprox

zsh_OLD_CMD=	"/usr/bin/env zsh" /bin/zsh /usr/bin/zsh
zsh_CMD=	${LOCALBASE}/bin/zsh

NO_BUILD=	yes
DOCS=		README.md LICENSE

do-install:
	${MKDIR} ${STAGEDIR}${DATADIR}
	${INSTALL_SCRIPT} ${WRKSRC}/goprox ${STAGEDIR}${DATADIR}/goprox
	${LN} -fs ${DATADIR}/goprox ${STAGEDIR}${PREFIX}/bin/goprox
	${INSTALL_MAN} ${WRKSRC}/man/goprox.1 ${STAGEDIR}${PREFIX}/man/man1/
	# NOTE: Firmware isn't included to keep the package size low.
	#${MKDIR} ${STAGEDIR}${DATADIR}/firmware
	#(cd ${WRKSRC}/firmware && ${COPYTREE_SHARE} . ${STAGEDIR}${DATADIR}/firmware)
	#${MKDIR} ${STAGEDIR}${DATADIR}/firmware.labs
	#(cd ${WRKSRC}/firmware.labs && ${COPYTREE_SHARE} . ${STAGEDIR}${DATADIR}/firmware.labs)

.include <bsd.port.mk>

I have some remaining questions:

• The source archive is nearing 1GB: could firmware be downloaded on demand instead?
• Is this useful enough to submit to the FreeBSD ports tree?

Feature: Add support for new GoPro Hero 11 Mini

Add support for the new GoPro Hero 11 Mini.

https://gopro.com/en/us/shop/cameras/hero11-black-mini/CHDHF-111-master.html

https://community.gopro.com/s/topic/0TO3b000000K1RUGA0/hero11-black-mini?language=en_US