activitywatch/aw-core

Core library for ActivityWatch

aw-core

Core library for ActivityWatch.

Modules

• aw_core, contains basic datatypes and utilities, such as the Event class, helpers for configuration and logging, as well as schemas for buckets, events, and exports.
• aw_datastore, contains the datastore classes used by aw-server-python.
• aw_transform, all event-transforms used in queries.
• aw_query, the query-language used by ActivityWatch.

Logging

Run python with LOG_LEVEL=debug to use change the log level across all AW components

How to install

To install the latest git version directly from github without cloning, run pip install git+https://github.com/ActivityWatch/aw-core.git

To install from a cloned version, cd into the directory and run poetry install to install inside an virtualenv. If you want to install it system-wide it can be installed with pip install ., but that has the issue that it might not get the exact version of the dependencies due to not reading the poetry.lock file.

aw-core's Issues

[peewee] Change bucket_id in Event model to int

We have minor problems with database size right now, one highly redundant entry in the peewee database is the bucket_id field which is a CharField containing the entire human-readable id. It would be better to just assign an int with the int ID of the bucket, reducing the size of the field from many bytes (an ID is usually >20 chars) to something closer to 32 bits.

Add Bucket model

#71 (comment)

Creating new table in peewee database

I've created a new table in the peewee.py file. I've tried both make build and make install in aw-core, yet it doesn't seem like the peewee file ever runs the code. I've looked into the source code and couldn't find when the PeeweeStorage was initialized. Any idea on how to get the file to run manually?

Code below:

if not BucketModel.table_exists():
    BucketModel.create_table()
if not EventModel.table_exists():
    EventModel.create_table()
if not TestModel.table_exists():
    TestModel.create_table()
self.update_bucket_keys()

Size-limit JSON files

Prevent JSON files (in the file storage method) from growing larger than a certain size (which takes considerable resources deserializing, storing in memory, and then again serializing). The previous proposal about date-naming is unnecessarily complex (which timezone should the dates be in, etc.) and it's just easier to start a new file when the last one gets too big.

This would require additional logic in the get() method, where it would have to read all the files, append them together, and return the contained events.

Add endtime to event model

We currently have an issue when fetching events by starttime and endtime.

23:00                             00:00                           01:00
<----------------------------------------------------------------------
         | event start                                 event end |

If we have a case like this and we have an event which starts before 00:00 and ends after 00:00, if we query with startime=the_second_day we will not get the event even though it is within the second day.

Currently we are checking the datetimes like this

if event.starttime >= starttime and event.starttime <= endtime:
    return event

What we actually should do when fetching events with a starttime and endtime

if event.endtime >= starttime and event.starttime <= endtime:
    return event

This would however require us to add a "endtime" field to save in every datastore.
At that point we could also remove the duration field since it's essentially endtime-starttime to save on db size.

TL;DR: Our datastorages cannot filter by starttime+duration to get the endtime, we need an actual endtime field instead of duration to be able to filter after endtime and fix a few quirks when having events in between two dates.

Restructure config files

I think that a default config section title of $appname (as in the {afk, window, spotify} watchers currently) is a bad idea.

• Change to a proper default section.

We should also handle testing settings using configparsers override functionality (override some defaults when run in testing mode), not as it's own defaults.

• Testing modes should be an override of the default, not a different default.

I also want to add comments to the config files to make them easier to edit.

• Add comments to the config files, making them friendlier to edit.

It would be nice to have some kind of argparse + configparser settings union to simplify overriding parameters using the commandline.

• (extra) Nice way to use argparse output as overrides.

Do you want InfluxDB support?

I ended up adding some influxdb support to play around with custom dashboards with Grafana.

It's a working draft and needs cleaning up. But I'm not sure if it's worth pursuing since the main thing I wanted was the annotation feature as a shortcut to tagging, but I wasn't too impressed with it in the end.

So saying that I could of gone straight to tagging since I do know VueJS, just that I find UI to be a massive timesink and tried to cheat.

Side note: I think something like web-hooks/hooks might be good idea to allow custom scripts/APIs to be triggered; like when an event is inserted.

How to test own code?

I am sorry if there already is an explanation online, I couldn't find anything. How would I compile own code from aw-core, so that a test server would be able to use it, or more spefically, after I run make build from aw-server, which should include my aw-core code, where do I find the finished binary/where can I run the compiled code now?

Log file name on Windows not allowed

Log file contains forbidden character :.

Easy fix.

Top Browser Domain is empty if url is a file

I was reading a pdf on firefox, and firefox showed it in top browser domains. However, because there's no real domain, it's just an activity with a colour and a time.

I think for a case like this, the activity should either be ignored from the "top browser domains" category, or get a custom browser domain name which lets the user know it's a file.

EDIT: Another issue is that all events without a clear domain name, will get clumped together, as they all have empty strings.It's why a custom browser domain name must be identifiable from all others, i.e if i'm reading two different pdf documents on firefox, both need to look different.

Event merge from different buckets

I'm guessing this has come up many times, so apologies for not finding related issue. The closest one is in ActivityWatch but it's not exactly that :(

I found code that does the union on events but it has a note WARNING: This function strips all data from events as it cannot keep it consistent. Do you know/remember why is that?

What I'd like to do is to have a list of events that contain all data from all buckets. Something like:

buckets = aw_client.get_buckets()
all_events = [aw_client.get_events(bucket['id']) for bucket in buckets]
merged_events = aw_transform.merge_events(*all_events)

I need this, so if there's no such functionality, I'm happy to contribute, but maybe you already have one?

Implement JSON datastore

Should be used both as a datastore method on the server and for temporary caching (when server can't be reached for example) on clients.

Folder structure should be something like:

<dataroot>/{bucket id}/{yyyy-mm-dd}.json

Make Event inherit defaultdict(list)

Also, make e[attr] accessible as e.attr (every dict item an attribute). To prevent collisions check for existing attribute with the specified name when reading or writing so that it becomes an unsafe shorthand. (Essentially object attributes as accessed in JavaScript)

Remove worthless datastores

We have a lot of datastores, and some of them are currently worthless.

• TinyDB should be removed since the peewee storage is faster and is also saved to a single file
• Should we remove the SQLite datastore? Could it possibly be an improvement over peewee or should we just scrap it?

Database migrations

Me and @johan-bjareholt have had some discussion about this since it's something we'd need if we want to change the database schema between versions once we have real users.

I looked around for migration tooling and found the minimalistic caribou: https://github.com/clutchski/caribou

Last commit was 7 years ago and only seems to support Python 2.6 but the code base is really small (271 LOC) and I wouldn't mind forking it and updating it myself. Unless there is something equally simple out there.

Switch to pytest as testing framework

Nosetest is no longer developed. Both matplotlib and pandas now use pytest instead. Also includes ways to do parametrization in a neat way.

"setup.py install" doesn't work

setup.py develop does however, it's all very strange.
I get an ImportError saying there is no module named aw.core, which there obviously should be.

The same issue holds true for aw-client, but not aw-server or the aw-watcher-*'s.

Migration from peewee to sqlite loses some window titles

Migrating my own buckets through starting aw-server with an existing peewee-database and letting it auto-migrate lost some of my window title data.

This will need to be solved before (if) we migrate users. Also need to test if rust migrations have the same issue.

Blocks PR: #86

The window titles existed before the migration.

There's also some new 2020-02-19 11:56:58 [WARNING]: Gap was of negative duration but could be safely merged (-7830.091s). This message will only show once per batch. (aw_transform.flood:46) warnings too.