rolandsz/mi-fit-and-zepp-workout-exporter

A Python script which allows users to export workout data from Mi Fit and Zepp applications.

Mi Fit and Zepp workout exporter

This repository contains an example Python implementation for the article.

Environment setup

pip install -r requirements.txt

Usage

The script authenticates the user with the API then exports all workouts to the output directory using the specified file format.

python3 main.py [-h] [-e ENDPOINT] [-t TOKEN] [-f {gpx,geojson,gpkg,parquet,shp,csv,json,xlsx,sql,xml,html}] [-o OUTPUT_DIRECTORY]

Acknowledgements

The latitude/longitude parsing is based on Miroslav Bendík's MiFitDataExport project.

How to get the token manually

If the authentication does not work out of the box, you can also provide the token manually:

1. Open the GDPR page
2. Click Export data
3. Sign in to your account
4. Open the developer tools in your browser (F12)
5. Select the Network tab
6. Click on Export data again
7. Look for any request containing the apptoken header or cookie
8. Pass the token to the script using the -t argument

mi-fit-and-zepp-workout-exporter's Issues

only exports 200 workouts of 800+

Hey there,
not sure if it is an issue, but the tool is only exporting 200 workouts all out of 2022.
Id would like to export also from 2018 on, those are included inside the csv file.

Does anyone know if that is possible?

New version doesn't work

I run main.py as before and get the error:

Traceback (most recent call last):
  File "/Mi-Fit-and-Zepp-workout-exporter/main.py", line 8, in <module>
    from src.exporters.geopandas_exporter import GeoPandasExporter
  File "/Mi-Fit-and-Zepp-workout-exporter/src/exporters/geopandas_exporter.py", line 7, in <module>
    import geopandas as gpd
  File "/Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/geopandas/__init__.py", line 1, in <module>
    from geopandas._config import options  # noqa
  File "/Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/geopandas/_config.py", line 109, in <module>
    default_value=_default_use_pygeos(),
  File "/Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/geopandas/_config.py", line 95, in _default_use_pygeos
    import geopandas._compat as compat
  File "/Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/geopandas/_compat.py", line 9, in <module>
    import pyproj
  File "/Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/pyproj/__init__.py", line 49, in <module>
    import pyproj.network
  File "/Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/pyproj/network.py", line 10, in <module>
    from pyproj._network import (  # noqa: F401 pylint: disable=unused-import
ImportError: dlopen(/Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/pyproj/_network.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_proj_context_is_network_enabled'

My pip list:

Package            Version
------------------ ---------
attrs              21.4.0
black              22.3.0
certifi            2022.6.15
charset-normalizer 2.0.12
click              8.1.3
click-plugins      1.1.1
cligj              0.7.2
Fiona              1.8.21
geopandas          0.11.0
gpxpy              1.5.0
idna               3.3
isort              5.10.1
munch              2.5.0
mypy-extensions    0.4.3
nodeenv            1.7.0
numpy              1.23.1
packaging          21.3
pandas             1.4.3
pathspec           0.9.0
pip                22.1.2
platformdirs       2.5.2
pydantic           1.9.1
pyparsing          3.0.9
pyproj             3.3.1
pyright            1.1.251
python-dateutil    2.8.2
pytz               2022.1
requests           2.27.1
setuptools         57.4.0
Shapely            1.8.2
six                1.16.0
tomli              2.0.1
typing_extensions  4.3.0
urllib3            1.26.10

GPX export error - stratos 3 data

I have found one issue in \gpx_file_exporter.py

when the values have for some reason empty string in the data it will throw this error

Traceback (most recent call last): File "C:\temp\Mi-Fit-and-Zepp-workout-exporter\main.py", line 24, in <module> scraper.run() File "C:\temp\Mi-Fit-and-Zepp-workout-exporter\core\scraper.py", line 51, in run exporter.export() File "C:\temp\Mi-Fit-and-Zepp-workout-exporter\third_party\gpx_file_exporter.py", line 52, in export track_data = self.parse_track_data() File "C:\temp\Mi-Fit-and-Zepp-workout-exporter\third_party\gpx_file_exporter.py", line 95, in parse_track_data times=array.array('q', [int(val) for val in File "C:\temp\Mi-Fit-and-Zepp-workout-exporter\third_party\gpx_file_exporter.py", line 95, in <listcomp> times=array.array('q', [int(val) for val in ValueError: invalid literal for int() with base 10: ''

it can be easily fixed by filtering out the empty value like this on lines 95-113 in \gpx_file_exporter.py

list(filter(None,self.data["time"].split(";")))

Implement "format(s)" command-line argument

Issue #1 showed that non-essential features depending on untested third-party code can easily break the core functionality of the application. Users should be able to opt-in for extra functionalities (like conversion to .gpx), so at least they can dump their data in raw .json format.

Acceptance criteria:

1. Introduce a new argument in the ArgumentParser such as format or formats
2. Make sure the value of this argument is validated so it can only take values "json" or "gpx" (See https://docs.python.org/3/library/argparse.html#choices)
3. Make sure it can handle multiple values such as "json,gpx" (See https://docs.python.org/3/library/argparse.html#nargs)
4. Refactor the Scraper class so it converts & exports workouts in the user-specified format(s)

Nice to have:

1. Write unit tests for AC 4

Add speed value to GPX info

When querying a specific workout in /v1/sport/run/detail.json, aside the longitude_latitude, heart_rate, etc, there is also the speed parameter. Would it be possible to support that value as well on the exported gpx format?

If you don't have access to any workout with that value, basically its something like the following:

{
	"code": 1,
	"message": "success",
	"data": {
		[...]
		"speed": "1,4.76;0,4.65;1,4.65;0,4.60;[...];0,6.57;1,6.57;0,5.74",
		"cadence": ""
	}
}

500 server error

When running the script, I get the output

Traceback (most recent call last):
  File "XXX\main.py", line 26, in <module>
    scraper.run()
  File "XXX\core\scraper.py", line 27, in run
    history = self.api.get_history()
  File "XXX\core\api.py", line 15, in get_history
    r.raise_for_status()
  File "C:\tools\miniconda3\lib\site-packages\requests\models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api-mifit-de2.huami.com/v1/sport/run/history.json?source=run.mifit.huami.com

With another value for the API key, I get a 401 error so I assume this is the right key. Might the API have changed since this was written? Any idea how to fix this?

Thanks!

Works on linux but not windows

when using a error with the path occurs.

INFO:root:There are 200 workouts
Traceback (most recent call last):
File "E:\Dokumente\projects\Mi-Fit-and-Zepp-workout-exporter\main.py", line 67, in
scraper.run()
File "E:\Dokumente\projects\Mi-Fit-and-Zepp-workout-exporter\src\scraper.py", line 40, in run
self.exporter.export(output_file_path, summary, points)
File "E:\Dokumente\projects\Mi-Fit-and-Zepp-workout-exporter\src\exporters\gpx_exporter.py", line 35, in export
with open(output_file_path, "w") as fp:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'workouts\Workout--2022-12-31--02-17-30.gpx'
PS E:\Dokumente\projects\Mi-Fit-and-Zepp-workout-exporter>

Fixed it for now by using a linux os

Export to Garmin, heartrate missing

if you would like to import exported GPX to Garmin Connect (GC), you will notice HR is missing.
this is due to the fact GC accepts HR only as INT, not with values these are exported by default with decimals, like 100.00.

to fix this, just update the line:
f"gpxtpx:hr{int(point.heart_rate)}</gpxtpx:hr>"

Also I have indentified few more "Workout types":
if summary.type == 1:
return "running"
elif summary.type == 6:
return "walking"
elif summary.type == 9:
return "cycling"
elif summary.type == 10:
return "indoor_cycling"
elif summary.type == 16:
return "other"
elif summary.type == 23:
return "indoor_rowing"
elif summary.type == 92:
return "badminton"
elif summary.type == 8:
return "treadmill_running"

Export other data besides the workouts

Thank you very much for your application! Could you please expand the application by adding functionality to export other data besides the workouts. Such additional data worth exporting is weight measurements, etc. Thank you again!

Export of non-GPS based workout (e.g. indoor cycling)

Wonderful work setting up this repo! I am interesting in extracting both heart-rate and location information.

I was able to export all GPS based workouts to GPX. However, for the non-GPS based workouts like indoor cycling I get a more or less empty file, just containing a start date/time and a workout of type "None". This makes sense, as there is no position information.

However, if I try to export as CSV (or XSLX) instead, the script breaks the moment it encounters the first non-GPS workout, with the error at the bottom of this issue.

I have two questions:
(1) Is it possible to still export the heart-rate measurements for non-GPS workouts to e.g. CSV or XLSX?
(2) If this is not feasible, is there a way to have the script skip any non-GPS workouts when exporting as CSV/XLSX?

I hope my request is clear, but let me know if you have any questions.

PS. I make YouTube videos on www.youtube.com/thequantifiedscientist , and I'll make sure to link this repo in the description if I end up using it. Great work!

The error:
INFO:src.scraper:Downloaded /Volumes/DataAnalysisSSD/Projects/2017-07-15_trackingMyself/data/amazfit/amazfitGTS4/pythonExport/Workout--2022-11-10--09-50-24.xlsx Traceback (most recent call last): File "/Users/robterhorst/Mi-Fit-and-Zepp-workout-exporter/main.py", line 67, in <module> scraper.run() File "/Users/robterhorst/Mi-Fit-and-Zepp-workout-exporter/src/scraper.py", line 40, in run self.exporter.export(output_file_path, summary, points) File "/Users/robterhorst/Mi-Fit-and-Zepp-workout-exporter/src/exporters/geopandas_exporter.py", line 60, in export gdf = gdf[ File "/Users/robterhorst/opt/anaconda3/envs/miZeppExport/lib/python3.10/site-packages/geopandas/geodataframe.py", line 1412, in __getitem__ result = super().__getitem__(key) File "/Users/robterhorst/opt/anaconda3/envs/miZeppExport/lib/python3.10/site-packages/pandas/core/frame.py", line 3810, in __getitem__ indexer = self.columns._get_indexer_strict(key, "columns")[1] File "/Users/robterhorst/opt/anaconda3/envs/miZeppExport/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6111, in _get_indexer_strict self._raise_if_missing(keyarr, indexer, axis_name) File "/Users/robterhorst/opt/anaconda3/envs/miZeppExport/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6174, in _raise_if_missing raise KeyError(f"{not_found} not in index") KeyError: "['track_date', 'timestamp', 'latitude', 'longitude', 'altitude'] not in index"

file error

Thanks you to have writted this script.
I succeeded in authentification but I have this error :
Traceback (most recent call last):
File " main.py", line 72, in scraper.run()
File "\src\scraper.py", line 41, in run for summary in self.fetch_workout_summaries():
File "scraper.py", line 27, in fetch_workout_summaries history = self.api.get_workout_history()
File "src\api.py", line 163, in get_workout_history model = WorkoutHistory(**response)
File "\site-packages\pydantic\main.py", line 164, in init
pydantic_self.pydantic_validator.validate_python(data, self_instance=pydantic_self)
pydantic_core._pydantic_core.ValidationError: 315 validation errors for WorkoutHistory
data.summary.32.strokes
Field required [type=missing, input_value={'trackid': '1648144986',...trate_setting_type': -1}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.5/v/missing

I updated all the module but I have the same bug.
Can you help me
thanks

HTTPError: 401 Client Error

I'm getting this error when running:

requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://api-mifit-de2.huami.com/v1/sport/run/?source=run.mifit.huami.com

I'm pretty sure my token is correct (i've manage to retrive the token in a few different ways, they're all the same).

Any clue?