tormoder/fit

A Go package for decoding and encoding Garmin FIT files

fit

fit is a Go package that implements decoding and encoding of the Flexible and Interoperable Data Transfer (FIT) Protocol. Fit is a "compact binary format designed for storing and sharing data from sport, fitness and health devices". Fit files are created by newer GPS enabled Garmin sport watches and cycling computers, such as the Forerunner/Edge/Fenix series.

The two latest versions of Go is supported. The core decoding package has no external dependencies. The latest release of Go and a few external dependencies are required for running the full test suite and benchmarks.

Latest release: 0.15.0

Version Support

The current supported FIT SDK version is 21.115.

Developer data fields are currently only partially supported. At the moment the decoder parses Developer Data Field Descriptions, Developer Data ID Messages and Field Description Messages. The decoder currently discards developer data fields found in records.

The encoder will currently (silently) ignore anything related to Developer data fields, This also means that encoding will not fail if protocol version 2 is specified for a file header.

Developer data fields support is tracked by #21 and #64.

Features

• Supports all FIT file types.
• Accessors for scaled fields.
• Accessors for dynamic fields.
• Field components expansion.
• Go code generation for custom FIT product profiles.

Installation

Using Go modules:

$ go get github.com/tormoder/[email protected]

Using $GOPATH:

$ go get github.com/tormoder/fit

About fit

• Example Usage
• Data Types
• Main API Reference
• Custom Product Profiles
• Upcoming Features
• Contributing
• License

Contributors

• usedbytes
• colinrgodsey
• bpg
• pieterclaerhout
• beyoung
• cloudlena
• dudanian

fit's Issues

`TestGenerator` usage of `gofmt` output is fragile

gofmt changes over time, meaning that the comparison against the "golden" files in TestGenerator breaks easily, even when the code is logically equivalent.

The official recommendation is to use a specific version of gofmt to avoid such problems:

Note that these kinds of minor updates to gofmt are expected from time to time. In general, we recommend against building systems that check that source code matches the output of a specific version of gofmt. For example, a continuous integration test that fails if any code already checked into a repository is not “properly formatted” is inherently fragile and not recommended.

The alternative would be to read in the golden data, format it with the "current" version of gofmt via go/format.Source(), and compare that against the generated code. That's slightly complicated by the fact that all the files are concatenated, but should be easy to solve.

internal/profile: support new messages introduced in SDKs from 16.30 to 20.43

all: consider using vanity import paths

writer: support Developer data fields

Decoder support is tracked by #21.

Top level protocol 2 support is tracked by #20.

"use of internal package not allowed" after clone

To make it easier to contribute we should probably find a way to get rid of "use of internal package not allowed" - error messages after clone. Currently I need to sed s/tormoder/torgil/g all code.

Example:
"../../go/src/github.com/torgil/fit/reader.go:14:2: use of internal package not allowed"

changing the line "github.com/tormoder/fit/internal/types" to "github.com/torgil/fit/internal/types" removes the error for me but clutters the workspace and will make for constant merge-confilcts in every pull/rebase-iteration.

Cleanup: reader.go

wiki: Typo in https://github.com/tormoder/fit/wiki/Data-Types#execptions

cmd/stringer: fork upstream tool and convert to library

See https://godoc.org/golang.org/x/tools/cmd/stringer.

Issue Parsing

First of all, I want to thank you for this project. It is very well done and super useful. Thank you very much. I might even be able to break off a couple cycles if that helps you to keep maintaining it.

I am having an issue with a FIT file that I downloaded from garmin. Running your "Example_test.go" with my file when I parse the fit file I get...

parsing data message: missing data definition message for local message number 10
It was a fit downloaded directly from garmin. I don't fully understand your fit decoder yet to be able to debug it. Have you seen this before? I included the fit file if it helps.
Broken.zip

Cleanup: cmd/fitgen/internal/profile/transform.go

activity: structured activity file type

Currently, after decoding you basically get a flat activity file.
This closly represents the fit file protocol.
But an application might be more interested in a tree-structure of the activity, e.g.

type StructuredActivityFile {
  Sessions []StructuredSession
}
type StructuredSession{
  Lap []*StructuredLap
}
type StructuredLap {
  LapMsg // Copy or reference of original lap for summary information
  Records []*RecordMsg // Contains only data records which were during this lap.
 // maybe also others, such as events, ...
}

This could be calculated in a post-processing step as a method of ActivityFile by

 func (a *ActivityFile) StructuredActivity() StructuredActivity {...}

I guess it's only neccessary to compare timestamps to build the tree.

What do you think of this?
Or is the scope of the package to represent the fit protocol closly, leaving all higher level functionality for other packages?

json: implementation of json.MarshalJSON for fit.Fit type

internal/profile: implement generic code generation for byte components

Add "golden" decode tests

profile: support "hidden"/undocumented fields and messages?

See #61 for some context.

vendor: migrate to dep?

See https://github.com/golang/dep.

internal/types: add unit tests

Parsing data or definition message failed, When I upgrade from 16.20 to 20.43

When I call 'fit.Decode' method, I get the following error messages:

parsing definition message: validating Session failed: field 8: unknown base type 0X%!X(PANIC=runtime error: index out of range)

or
parsing data message: missing data definition message for local message number 10

or
parsing definition message: validating Session failed: field 9: type BaseSint32 is not compatible with profile type BaseUint32

writer: encoding of FIT files

reader: validating session failed

Trying to decode my fit file fails with the following error:

parsing definition message: validating Session failed: field 10: type BaseSint32 is not compatible with profile type BaseUint32

Logging output:

header decoded: size: 14 | protover: 16 | profver: 2078 | dsize: 120284 | dtype: .FIT | crc: 0x30ec
definition message parsed: local: 0 | global: FileId | arch: LittleEndian | fields: 7
parsed file_id message: {Activity Garmin 2691 3982257377 2019-06-02 09:10:36 +0000 UTC 65535 }
definition message parsed: local: 1 | global: FileCreator | arch: LittleEndian | fields: 3
definition message parsed: local: 2 | global: Event | arch: LittleEndian | fields: 7
definition message parsed: local: 3 | global: DeviceInfo | arch: LittleEndian | fields: 27
definition message parsed: local: 4 | global: MesgNum(22) | arch: LittleEndian | fields: 10
definition message parsed: local: 5 | global: MesgNum(141) | arch: LittleEndian | fields: 7
definition message parsed: local: 6 | global: DeviceSettings | arch: LittleEndian | fields: 72
definition message parsed: local: 7 | global: UserProfile | arch: LittleEndian | fields: 28
definition message parsed: local: 8 | global: MesgNum(147) | arch: LittleEndian | fields: 54
definition message parsed: local: 9 | global: MesgNum(79) | arch: LittleEndian | fields: 17
definition message parsed: local: 10 | global: Sport | arch: LittleEndian | fields: 10
definition message parsed: local: 11 | global: MesgNum(13) | arch: LittleEndian | fields: 43
definition message parsed: local: 12 | global: ZonesTarget | arch: LittleEndian | fields: 11
definition message parsed: local: 13 | global: Record | arch: LittleEndian | fields: 9
definition message parsed: local: 14 | global: MesgNum(104) | arch: LittleEndian | fields: 5
definition message parsed: local: 15 | global: Lap | arch: LittleEndian | fields: 80
definition message parsed: local: 0 | global: MesgNum(216) | arch: LittleEndian | fields: 13
definition message parsed: local: 1 | global: MesgNum(233) | arch: LittleEndian | fields: 1
definition message parsed: local: 5 | global: MesgNum(140) | arch: LittleEndian | fields: 32
illegal definition message: local: 6 | global: Session | arch: LittleEndian | fields: 93

3708692446.zip

cmd/fitgen: handle fields not enabled in profile but still needed

For example, when running fitgen on SDK 21.38, it fails with

panic: genMsgs: could not find type for ref field name "HeartRateSourceType"

For some reason this field in not enabled in the profile. Find out why and possibly enable some kind of special handling of such fields inside fitgen.

Currently new SDKs are generated locally by patching fitgen to enbale the missing field(s).

strange error message

I get the following error message when importing a fit file:

parsing definition message: validating Session failed: field 6: size (4) is greater than size of profile base type BaseSint32 (4)

which to me seems like stating 4>4

This is the file causing the problem.

cmd/fitgen: add golden generator output for 20.90 SDK

Reminder.

reader: support Developer data fields

Required for Fit 2.0 (#20).

all: support for FIT protocol 2.x

error decoding header: not supported: protocol version 2.x not supported by sdk protocol version 1.0

Would be great thx if fit supports version 2.x. _.

cmd/fitgen: parse sdk version from C code

Decode: Handle accumulated components

Wahoo fit files: missing data definition message for local message number 7

Received following error parsing a wahoo .fit file.

header decoded: size: 14 | protover: 32 | profver: 2027 | dsize: 337952 | dtype: .FIT | crc: 0x4965
definition message parsed: local: 0 | global: FileId | arch: BigEndian | fields: 5
parsed file_id message: {Activity WahooFitness 31 3107002583 2021-12-11 10:14:48 +0000 UTC 65535 }
definition message 

parsed: local: 0 | global: DeveloperDataId | arch: BigEndian | fields: 2
definition message parsed: local: 0 | global: FieldDescription | arch: BigEndian | fields: 5
definition message parsed: local: 0 | global: FieldDescription | arch: BigEndian | fields: 5
definition message parsed: local: 0 | global: FieldDescription | arch: BigEndian | fields: 5
definition message parsed: local: 0 | global: FieldDescription | arch: BigEndian | fields: 5
definition message parsed: local: 0 | global: FieldDescription | arch: BigEndian | fields: 5
definition message parsed: local: 0 | global: FieldDescription | arch: BigEndian | fields: 5
definition message parsed: local: 0 | global: FieldDescription | arch: BigEndian | fields: 5
definition message parsed: local: 0 | global: Event | arch: BigEndian | fields: 4
parsing time: seconds from device power on
parsing data message: missing data definition message for local message number 7

Archive.zip

Tried with the java FitCSVTool directly and that worked, but the Local Number seems to be 0 everywhere ( which looks like the reason the decoding actually fails. )

In code: (reader.go:213)
d.defmsgs[dm.localMsgType] = dm

In this case, dm.localMsgType is always 0.

Tried with other wahoo .fit files, but all have the same outcome.

reader: fix accumulated components handling

See test file testdata/python-fitparse/compressed-speed-distance.fit.

Compressed distance is not accumulated correctly. Possibly a roll over/overflow issue.

Handling of multiple `DeviceInfoMsg`s

Currently there is precisely one DeviceInfoMsg, and it lives in the File.
It looks like having multiple DeviceInfoMsgs is valid per the spec, and indeed a number of the testdata references include multiple DeviceInfoMsgs. This can be useful to track battery voltage over time, or for multiple devices.

The question is, how to support this? The simplest thing would be to just delete DeviceInfo from File and add a slice of them to the relevant file types (according to SDK 21.32.00 that's ActivityFile, WeightFile, BloodPressureFile, MontoringAFile, MonitoringBFile). However, that would break anyone that's relying on File.DeviceInfo.

So, I made a change which duplicates DeviceInfoMsgs into both File.DeviceInfo and a slice in the individual file types (usedbytes/fit@17b8c4f) - however this causes a problem when a file is decoded and re-encoded. The last DeviceInfoMsg ends up in both the slice and File.DeviceInfo, which means it ends up duplicated when writing out. This could be avoided by just not writing out File.DeviceInfo, but that might be "surprising" for users. Checking File.DeviceInfo against each entry in the slice and avoiding duplicates would be possible but a bit ugly, because the writer is using reflection and we'd need a special case just for DeviceInfo.

So, the simplest looks like just removing File.DeviceInfo, but I'm wary of what breakage that might cause, next easiest would be skip File.DeviceInfo on write-out. Any thoughts/opinions?

Decode: Handle field "Components"

Cleanup: cmd/fitgen/internal/profile/generator.go

writer: Revisit length of arrays when encoding

It's not obvious how the length of Arrays should be handled when encoding. In the profile xls, some arrays are tagged with "[N]", presumably indicating that they have a variable length. In some of these cases, the "EXAMPLE" column includes a number which appears to be the intended length. In other cases, the "EXAMPLE" column only has '1'.

The decoder always reads the number of elements which are encoded in the file. The encoder could also do this - but the length would need to be known at the point of generating the definition message, which means if the length is different between messages in the same file, we'd need to make a new definition message (see also, #36) or truncate/pad.

The current behaviour is to use the value from the "EXAMPLE" column, and always encode exactly that number of elements, truncating or padding as necessary. This causes the DecodeEncodeDecode test to fail in a number of cases, because this length is '1' in the xls, whereas the input file has many elements.

cmd/fitgen: Use github.com/tealeg/xlsx for xlsx-csv conversion

This would remove the Python dependecy for profile code generation.

Support chained FIT files

From SDK 16.30 (page 15):

"The FIT protocol allows for multiple FIT files to be chained together in a single FIT file. Each FIT file in the chain must be a properly formatted FIT file (header, data records, CRC)."

cmd/fitgen: add tests to verify generator output

• Add golden tests for generator output: messages.go, types.go and profile.go
• Add tests for different versions of the Fit SDK

all: use error wrapping features introduced in Go 1.13

Ref.
golang.org/x/xerrors
and
https://go.googlesource.com/proposal/+/master/design/29934-error-values.md.

reader: verification against other SDKs

Think about ways of verifying decoded data against other SDKs (most likely Java).
Implement at CSV-tool?

reader: decode message by message

Can you decode messages in a loop, instead of decoding the whole file in on call?
An application is a corrupted fit file, where you want to extract all record messages that are available.

Issue Parsing Leyzene Files

Hi- thank you for this project- works very well for all my Garmin files, but fails for leyzene.
The error is:

error: parsing definition message: validating Session failed: field 139: size 4 for BaseUint32 as base type in definition message is greater than size 2 for BaseUint16 as base type from profile

With the standard logger enabled I get the following:

header decoded: size: 14 | protover: 16 | profver: 2016 | dsize: 27297 | dtype: .FIT | crc: 0x5140
definition message parsed: local: 0 | global: FileId | arch: LittleEndian | fields: 7
parsed file_id message: {Activity Lezyne 4 2585936027 2018-03-27 07:14:43 +0000 UTC 65535 }
definition message parsed: local: 1 | global: FileCreator | arch: LittleEndian | fields: 2
definition message parsed: local: 9 | global: TimestampCorrelation | arch: LittleEndian | fields: 2
definition message parsed: local: 3 | global: DeviceInfo | arch: LittleEndian | fields: 11
definition message parsed: local: 5 | global: Record | arch: LittleEndian | fields: 9
definition message parsed: local: 2 | global: Event | arch: LittleEndian | fields: 9
definition message parsed: local: 6 | global: Lap | arch: LittleEndian | fields: 56
illegal definition message: local: 7 | global: Session | arch: LittleEndian | fields: 48

I've also tried the version from @beyoung but get the same result.
If you point me in the right direction I think I'd be able to fix it.
Thanks in advance.

Troubles creating "course" file types

I'm trying to use this library to generate fit files of the type "course".

I'm able to generate them, but they always come out as invalid.

After a lot of investigation, it turns out that the main issue is that for a "course" file, the SDK doesn't allow you to define event messages (it only allows this for "activity" files). From what I can read in the Garmin Fit SDK and the cookbook samples, the timer events are required.

Am I overlooking something or does the SDK currently doesn't allow you to do this?

reader: handle unknown messages and fields better

Include definition messages in addition to byte payload of unknown fields. Maybe make this a decode option.

Writer excludes fields that are missing from the first data message

In some real life scenarios sensors may connect / disconnect / reconnect during the recording of an activity. As the result, the activity file may have several "blocks" of records with different definition messages that have different set of fields. A common use case is when a power meter was connected to a Garmin bike computer after the activity recording has been started.

In the current fit file model we combine all messages of the same type in one collection (per type), with message fields initialized with "invalid" values for all the fields that don't have a valid data. When this file is writing back by the writer it is quite possible to have such message with missing fields as the first message in the collection, which then will be used as a template to build a definition message for all records of this type. All fields of the following records that don't have a matching definition field will be excluded from the final result.

Find timezone for local timestamps using github.com/bradfitz/latlong

Use any coordinates (if present in any messages) to find local time zone.

monitoring message fields.

A quick question - I'm trying to parse a monitoring file (monitoringB with heart rate data from a forerunner 35). The heart_rate fields fail to parse. The FIT profile shows 37 possible fields in the monitoring message (mesg_num 55), but the MonitoringMsg (messages.go) struct only has 12 of these fields defined or available. These 12 looks fairly similar to the fields shown in the FIT File Types Description document (2.2), which also doesn't show the whole list of available fields stated in the message defined in the profile. Any idea what's going on there? There seems to be a discrepancy between the profile and the document, and the code roughly aligns with the document.

Created files are invalid

Problem on Strava

The files created by Encode are somehow invalid for some platforms. For example, if I upload the newly created file to Strava, it says "The upload contains bad data.".

I tried to debug them by having a valid FIT file (e.g., https://github.com/tormoder/fit/blob/master/testdata/me/activity-small-fenix2-run.fit) and running it through the following simple program:

package main

import (
	"bytes"
	"encoding/binary"
	"log"
	"os"

	"github.com/tormoder/fit"
)

func main() {
	testData, err := os.ReadFile("activity-small-fenix2-run.fit")
	if err != nil {
		log.Fatalln(err)
	}

	file, err := fit.Decode(bytes.NewReader(testData))
	if err != nil {
		log.Fatalln(err)
	}

	f, err := os.Create("new.fit")
	if err != nil {
		log.Fatalln(err)
	}
	defer f.Close()

	err = fit.Encode(f, file, binary.LittleEndian)
	if err != nil {
		log.Fatalln(err)
	}
}

Using FitFileViewer

When uploading the new.fit file to https://www.fitfileviewer.com/, it shows empty data (whereas the original activity-small-fenix2-run.fit file showed valid content):

Using GPXSee

Weirdly enough, https://www.gpxsee.org/ shows the new.fit file correctly.

Using RUNALYZE

Analyzing both files with https://runalyze.com/tool/fit-viewer, the main differences seem to be that new.fit has a lot of enhanced_speed, enhanced_max_speed, enhanced_altitude fields on its laps and records (whereas activity-small-fenix2-run.fit doesn't have any enhanced_* fields) and that new.fit always uses 0 as its local message type index.

Update: I did another test run where I removed the code that generates the enhanced_* field generation code from this library and even without those, neither Strava nor FitFileViewer would accept the file. So I'm guessing that those aren't the problem.

writer: All messages in a slice are encoded based on the valid fields of the first message

In Fit files which consist of slices of messages (e.g. ActivityFile.Records), the writer implementation constructs the definition message based on the first element in that slice, and omits any fields which are invalid in that first element.

This means that if later messages include data in fields which were invalid in the first message, they will not be encoded because they weren't in the original definition.

I can see a few ways to improve this, with various tradeoffs:

1. Encode all fields all the time. This will waste huge amounts of file space, but it is simple. This could be added as an encoder option
2. Compare each new message against the current definition message. If they differ, encode a new definition message. This would be computationally slower, but that hasn't been quantified
3. Allow the caller to "register" a particular message template before encoding begins. This registration should include all fields which the user wants to encode. This would use less space than 1.

With 2., there would need to be considerations made about whether to use more than one local message ID. The FIT SDK suggests that local message IDs should be kept to a minimum to ensure maximum compatibility for decoders. That might be something else to add as an encode option.

data definition message missing

When I call 'fit.Decode' method, I get the following error messages:

header decoded: size: 14 | protover: 32 | profver: 2085 | dsize: 114956 | dtype: .FIT | crc: 0x773b
definition message parsed: local: 1 | global: FileId | arch: LittleEndian | fields: 7

error parsing file id message: error reading data message: missing data definition message for local message number 5

But I can decode it with FitCSVTool.jar and turn it into csv file.

00000b62-072f-4211-b8e3-26b0de4d51c6.fit.zip

reader: introduce fast path for Activity file type (if possible)

The majority of FIT files have type Activity. Consider special handling of them. Benchmark.

Decode: Handle timestamps that are seconds from device power on

Spec:

"if date_time is < 0x10000000 then it is system time (seconds from device power on)"