dapritchard/jiraworklog

Command-line application to synchronize local worklog records with a Jira server

jiraworklog's Issues

Initial error-testing checklist

Some examples of things to test for:

• error authenticating to Jira (special case WRT testing)
• config file doesn't exist
• config file not in the correct format
• worklogs file doesn't exist
• worklogs file not in the correct format
• checkedin file not in the correct format
• error reading from Jira
• error writing to Jira
• error writing to checkedin

Data structures outline

Entries data structures

• Representation of local entries
  • A dict with keys given by issueId and value a dict with the following fields:
    • comment (i.e. the description)
    • started
    • timeSpentSeconds
• Representation of remote entries
  • Same as the representation of local entries but with the following added information:
    • id
    • updated
• Representation of checked-in entries
  • Same as the representation of remote entries

Added / updated / removed entries

Use the same representation as for the regular entries lists but add a level of depth after issueId level with added, updated, and removed keys.

Basic sematics proposal

Suppose we have the following collection of work time entries

entry 1
entry 2

that we provide as an input to our routines which has the effect of uploading the entries to Jira.
Then later we add another entry so that the entries look like this:

entry 1
entry 2
entry 3

If we want to submit this collection of entries to our routines then we want this to have the effect of uploading the entry 3 to Jira, but not entry 1 nor entry 2.

Actions that can occur

We have the following actions that can occur.

1. Local side
   1. Add entry
   2. Remove entry
   3. Update entry
2. Remote side
   1. Add entry
   2. Remove entry
   3. Update entry

Note that for our routines to be able to distinguish an update from a removal and addition, a given work log entry will have to have some kind of identifier. Some time clock applications will not provide us with such information, so we will need to think about whether we will want to have different behavior based on whether or not we have that information (in other words, even if we did have that information we could treat modifications the same as a removal and addition).

Add/remove/update a local entry

Let's call for now the entries that are already logged in to Jira the checked-in entries.

1. If we add/remove/update a local entry and the checked-in entries are unchanged locally and on the remote, then we should add the entry to the remote.
2. If we remove a local entry and other checked-in entries are unchanged locally and on the remote, then we should remove the corresponding entry on the remote.
3. If we update a local entry and other checked-in entries are unchanged locally and on the remote, then we should update the corresponding entry on the remote.

Add/remove/update an entry on the remote

If we add/remove/update an entry on the remote then ideally we would have a mechanism to update the local entry. However, most time clock applications will not have an programmatic way to make such an update, and I think we should support such applications. If we accept this as a fundamental limitation of our application, then we have the following options. Note that work log entries have a unique identifier so we always do have the ability to unambiguously identify what add/remove/update has occurred.

1. We could force the remote to align with the local view of the work log entries. This would amount to "throwing out" any changes that occur on the remote. This would by far be the most straightforward approach conceptually and to implement.
2. We could try to track any changes on the remote.
   1. Suppose an entry gets added on the remote. We could mostly just ignore it, but what happens if an entry gets added for the same time? We would probably in that case want to consider the new local entry to correspond to the remote entry.
   2. Suppose an entry gets removed from the remote.
      1. If the corresponding entry gets removed from the local view then we would probably consider the local and remote to be realigned with regards to the added entry.
      2. If the corresponding entry get updated in the local view then we have two situations depending on the time clock system that constructs our input data.
         1. If the time clock system does not have the ability to uniquely identify a given entry then an update amounts to a remove and then an add. Thus we are unable to distinguish this situation from a removal and unrelated addition, as opposed to a modification. So probably the best thing to do is to is to consider the removal as having realigned the local and remote with regards to the removed entry on the remote, and to then add the new entry to the remote.
         2. If the time clock system does have the ability to uniquely identify a given entry then we are able to identify an update in the local view. I think probably we still want to do the same thing as for the case where we can't identify the update: consider the removal as having realigned the local and remote with regards to the removed entry on the remote, and then add the modified entry to the remote as a new entry.
   3. Suppose an entry gets modified on the remote.
      1. If the corresponding entry gets updated from the local view then I think probably we will want to overwrite the modification on the remote.
      2. If the corresponding entry gets removed from the local view then I think probably we will want to remove the entry on the remote. Note that effectively we will be "discarding" any changes (prompt the user before performing?)

Do we want/need a persistent view of the local and/or remote work log entries?

1. If we force the remote to align with the local view of the work log entries, then we wouldn't need to keep a persistent local view (but could it be useful, despite not being strictly necessary?).
2. There are a few examples of situations where I think we might need a persistent view of the work log entries (this is not entended to be an exhaustive list)

• If an entry gets removed from the remote then we may not want to have it added back in by our routine. We would need to know about the existence of it to realize that it was a remote removal.
• If an entry gets modified on the remote then we may not want to have it added back in by our routine. We would have to know what it's original value was (or equivalently, which work log entry it corresponds to) to know which work log entry to not add to the remote.

Malformed entries

Open questions:

• What should we do with malformed local data? For example, if we have overlapping work log entries?
• If we have identical entries will this break any assumptions that we are making?
• Will Jira let us upload such entries?

User interface proposal

Packaging and installation

The routines will be presented as a Python package that can be downloaded by pip. The installation will include a an executable script that will serve as a command-line application. Thus the user installation process will roughly be:

1. Install Python
2. Install jira-worklog via pip
3. Ensure that the Python bin directory is on $PATH.

Application user interface

The main information that the application needs as inputs are the following.

1. The worklog entries. We will probably want to take this data either from standard input or from a file pointed to by e.g. a --file command-line argument.
2. The most recent checked-in state of the worklogs. This is something that would presumable be written by the application so we could potentially read from and write to a default location like e.g. ~/.config/jira-worklog/jira-worklog-state. Or maybe we could check the current working directory for a particular file name. We should probably also take command line arguments specifying non-default locations to read from and write to.
3. Possibly information on how to parse the worklog entries. We may be able to store configuration information for a set of commonly-used time clock applications. The kind of information we will need will be things like the following. Given the quantity of the information we will probably need a configuration file for this also, with the location specified via a command-line argument.
   1. Data format: CSV, TSV, JSON, etc.
   2. If we allow for multiple tags per work log entry then for "flat" formats like CSV we will need a second separator.
   3. Field key for: description, time start, time end, tags.
   4. Mapping of tags to Jira issues.

Given these considerations, a typical invocation might look something like the following (and where the checked-in worklog state and parsing information data is stored in some standard location).

jira-upload-worklog worklog-entries.csv

The fully specified version might look something like the following.

jira-upload-worklog --checked-in-entries=path/to/file --parse-config=path/to/other-file worklog-entries.csv

Considerations of the above approach

1. Relying on having a checked-in state file makes the ability to work on multiple machines more laborious. The user would have to share the checked-in state file across the different machines. Would it be possible to store the state on the Jira server somehow? Or perhaps we could consider provide support for reading and writing to remote locations like Dropbox or SSH etc.
2. The parsing configuration file should be able to support data from multiple time clock application systems. Maybe we allow for multiple named systems (and probably a field specifying a default system).

Basic routines outline

Initial use outline

Push worklogs:

1. Fetch worklogs from the remote.
   1. Create an empty collection of checked-in worklogs.
   2. Existing worklogs on the remote are considered to be added entries.
2. Check which local entries correspond to remote entries and connect those.
3. Push the remaining local entries to the remote.
4. Add the remaining local entries to the checked-in worklogs.

General case outline

Push worklogs:

1. Fetch worklogs from the remote.
2. Compare the local view to the checked-in entries. Create a list of additions and removals.
3. Compare the remote view to the checked-in entries. Create a list of additions, removals, and updates.
4. Check modified remote entries. If there is a corresponding local removal then move them to the "added" remote category and remove the corresponding local removal entry.
5. Check which local "added" entries correspond to remote "added" entries and connect those.
6. Check which local "removed" entries correspond to remote "removed" entries and
7. Push remaining local "added" and "removed" entries to the remote.
8. Update the checked-in worklogs.

Consider switching from pytz to dateutil / zoneinfo / tzdata

See pytz: The Fastest Footgun in the West. At the top of the datetime documentation there's also a link to zoneinfo, which itself has a link to tzdata.

Authentication walkthrough

To experiment with this package:

1. Get an Atlassian API token, unless you already have one.
   See here for details.
2. Ensure that the jira package is installed and on your search path.
3. Try out some code like the following.

from jira import JIRA

user = '[email protected]'          # your email address
apikey = 'XXXXXXXX'             # your API token
server = 'https://my.host.net'  # the URL up to the host name

jira = JIRA(server, basic_auth=(user,apikey) )

If you are able to authenticate then you can do something like the following.

from pprint import pprint  # only needed for interactive noodling around

# Load the following routine from `fetch_worklogs.py`
out = fetch_worklogs_remotedata(jira, ['P0053-4'])
pprint(out)

# This is what the raw data looks like
issue_p0053_04 = jira.issue('P0053-4')
worklogs_list = jira.worklogs(issue_p0053_04)
pprint(vars(worklogs_list[0]))

Outline of target functionality

Consider the following example CSV.

task,start,end,tags
Review sample size calculations document before 2021-01-05 meeting,2021-01-04 14:52,2021-01-04 16:53,P0053_07_rpp
Update effect size (power) calculations,2021-01-07 13:01,2021-01-07 14:54,P0053_07_rpp
Update effect size (power) calculations,2021-01-07 09:59,2021-01-07 11:45,P0053_07_rpp
Review effect size calculations Slack thread,2021-01-08 09:00,2021-01-08 09:30,P0053_07_rpp
Channel Slack conversation,2021-01-12 09:00,2021-01-12 09:33,P0053_07_rpp
Effort estimate for additional sensitivity analyses,2021-01-11 22:09,2021-01-11 22:55,P0053_04_pre_contract_work
Prepare Bayesian method proposal,2021-01-14 13:10,2021-01-14 13:30,P0053_12_meetings
Add project Git configuration file to specify 'pull as a rebase',2021-02-15 16:41,2021-02-15 17:15,P0053_07_rpp
Create a list of covariates to QC,2021-02-15 12:46,2021-02-15 13:00,P0053_14_create_cohort

Suppose this output is generated from some time clock application. We process the data upload the entries to Jira using the functionality provided by this library. Then some time passes and we generate some new work time entries, and we export the data again from our application. This new data may contain all of the original entries plus some new entries. If we provide the entirety of the newly exported data as the input to the library routines, the routines should be smart enough to upload only the new work time entries to Jira.