mesozoic/gmail-yaml-filters

A quick tool for generating Gmail filters from YAML rules.

gmail-yaml-filters's Issues

Setup Instructions

gmail-yaml-filters looks very interesting. Conversion of yaml into xml seems to work fine.

For the upload feature the program fails because of missing files (client_secret.json). I couldn't find any instructions how to create this file properly. I tried to create credentials in Google console but don't know the correct redirect URI.

How to enable these features?

Thanks!

Can this import an existing set of rules?

I ask because an operation says "sync" - I don't want to lose my existing rules.

Thanks, Martin.

Add support for "Also apply filter to matching conversations" parameter

Basically subj.

When updating filters, I have to go to the web UI and reapply them. Would be great to do it automatically on sync.

Rules not generated in order

I was expecting that the rules were generated in order of the yaml file. I noticed that if my first three rules were "for_each" rules, the fourth rule, the first non-for_each rule was first in the output followed by the for_each rules. Is this expected behavior?

KeyError: 'action' in _simplify_filters, if server-side filter has no "Do this:" action

Encountered in versions: 0.6.1, 0.7.1.

Steps to reproduce:

1. Create a rule in filters.yaml: - { label: Bug demo, from: [email protected] }
2. gmail-yaml-filters --sync filters.yaml
3. Delete the "Bug demo" label via the GMail interface. (The label will be removed from the rule, so you now have a rule that matches [email protected] but has no action.)
4. gmail-yaml-filters --sync filters.yaml

Resulting stacktrace:

Traceback (most recent call last):
  File "/usr/local/google/home/marain/hack/gmail-filters/venv/bin/gmail-yaml-filters", line 11, in <module>
    sys.exit(main())
  File "/usr/local/google/home/marain/hack/gmail-filters/venv/local/lib/python2.7/site-packages/gmail_yaml_filters/main.py", line 606, in main
    upload_ruleset(ruleset, service=gmail, dry_run=args.dry_run)
  File "/usr/local/google/home/marain/hack/gmail-filters/venv/local/lib/python2.7/site-packages/gmail_yaml_filters/upload.py", line 179, in upload_ruleset
    known_filters = GmailFilters(service)
  File "/usr/local/google/home/marain/hack/gmail-filters/venv/local/lib/python2.7/site-packages/gmail_yaml_filters/upload.py", line 146, in __init__
    self.reload()
  File "/usr/local/google/home/marain/hack/gmail-filters/venv/local/lib/python2.7/site-packages/gmail_yaml_filters/upload.py", line 150, in reload
    self.matchable_filters = [_simplify_filter(existing) for existing in self.filters]
  File "/usr/local/google/home/marain/hack/gmail-filters/venv/local/lib/python2.7/site-packages/gmail_yaml_filters/upload.py", line 139, in _simplify_filter
    'action': {key: set(values) for key, values in filter_dict['action'].items()},
KeyError: 'action'

Workaround: manually delete the rule, and re-sync.

"KeyError: 'filter'" on --upload where user has no prior filters

gmail-yaml-filters 0.61.

To reproduce:

Delete all filters from GMail settings. Then:

% gmail-yaml-filters --upload filters.yaml
Traceback (most recent call last):
File "/tmp/hack/venv/bin/gmail-yaml-filters", line 11, in
sys.exit(main())
File "/tmp/hack/venv/local/lib/python2.7/site-packages/gmail_yaml_filters/main.py", line 651, in main
upload_ruleset(ruleset, dry_run=args.dry_run)
File "/tmp/hack/venv/local/lib/python2.7/site-packages/gmail_yaml_filters/upload.py", line 183, in upload_ruleset
known_filters = GmailFilters(service)
File "/tmp/hack/venv/local/lib/python2.7/site-packages/gmail_yaml_filters/upload.py", line 146, in init
self.reload()
File "/tmp/hack/venv/local/lib/python2.7/site-packages/gmail_yaml_filters/upload.py", line 149, in reload
self.filters = self.gmail.users().settings().filters().list(userId='me').execute()['filter']
KeyError: 'filter'

Reason: when the user has no filters, the GMail API behaves illogically: instead of returning "{filter: []}", it returns just "{}":

% curl -H 'Authorization: Bearer ya29.GltEBGQ8__restoftokenelided__' https://www.googleapis.com/gmail/v1/users/me/settings/filters
{}

google auth not working, no helpful error

I generated a client secret json file and I get this error when trying to use it:

❮ gmail-yaml-filters --dry-run --sync my-filters.yaml
Traceback (most recent call last):
  File "/home/mark/.local/bin/gmail-yaml-filters", line 33, in <module>
    sys.exit(load_entry_point('gmail-yaml-filters==0.9.5', 'console_scripts', 'gmail-yaml-filters')())
  File "/home/mark/.local/lib/python3.10/site-packages/gmail_yaml_filters/main.py", line 99, in main
    credentials = get_gmail_credentials(client_secret_path=args.client_secret)
  File "/home/mark/.local/lib/python3.10/site-packages/gmail_yaml_filters/upload.py", line 273, in get_gmail_credentials
    credentials = store.get()
  File "/home/mark/.local/lib/python3.10/site-packages/oauth2client/client.py", line 407, in get
    return self.locked_get()
  File "/home/mark/.local/lib/python3.10/site-packages/oauth2client/file.py", line 54, in locked_get
    credentials = client.Credentials.new_from_json(content)
  File "/home/mark/.local/lib/python3.10/site-packages/oauth2client/client.py", line 302, in new_from_json
    module_name = data['_module']
KeyError: '_module'

Supported actions Categorize

Is it possible to add some new supported actions for Categorize?

The export from Gmail generates this:

# for Primary
<apps:property name='smartLabelToApply' value='^smartlabel_personal'/>
# for Social
<apps:property name='smartLabelToApply' value='^smartlabel_social'/>
# for Updates
<apps:property name='smartLabelToApply' value='^smartlabel_notification'/>
# for Forums
<apps:property name='smartLabelToApply' value='^smartlabel_group'/>
# for Promotions
<apps:property name='smartLabelToApply' value='^smartlabel_promo'/>

Multiple Actions

I'd like to apply multiple labels, is there a syntax for this? It appears I need to use a foreach loop to do it now.

Reverse Nesting

I was wondering if there is a way to reverse the nesting structure you have. So for example, if any of the following three are true:

  more:
    -
      from:
        not:
          all:
            - [email protected]
            - [email protected]
            [email protected]
      subject:
        any:
          - "TEST"
          - "Test Test"
          - "TESTESTEST"
  more:
    -
      from:
        any:
          - "[email protected]"
          - "[email protected]"
          - "[email protected]"
          - "[email protected]"
          - "[email protected]"
          - "[email protected]"
  more:
    -
      from:
        not:
          all:
            - [email protected]
            - [email protected]
            [email protected]
      to:
        any:
          - "[email protected]"
          - "[email protected]"
          - "[email protected]"
          - "[email protected]"
          - "[email protected]"
          - "[email protected]"

So if any of the above three are true, do the following:

  important: true
  label: TEST

Handle empty actions on sync or prune

gmail-yaml-filters --sync will crash when it tries to prune a rule with an empty action.

Reproducible by:

1. Create the following rule:

from: [email protected]
label: FromJim

2. Run gmail-yaml-filters --sync to create the rule in Gmail.
3. In a web browser, load (or reload) Gmail, confirming the existence of the label FromJim.
4. In the web browser, click on the three dots next to FromJim and delete the label.
5. Go to Settings->All Settings->Filters and Blocked Addresses. Notice that the rule has no action.
6. Rerun gmail-yaml-filters --sync

Expected Results:
No errors. Ideally, the label would be recreated, the rule would be redefined, and the old rule (the one with the empty action) would be deleted.

Actual Results:
An error with a traceback:

Traceback (most recent call last):
  File "/home/user/.local/bin/gmail-yaml-filters", line 33, in <module>
    sys.exit(load_entry_point('gmail-yaml-filters==0.9.4', 'console_scripts', 'gmail-yaml-filters')())
  File "/home/user/.local/lib/python3.9/site-packages/gmail_yaml_filters/main.py", line 108, in main
    prune_filters_not_in_ruleset(ruleset, service=gmail, dry_run=args.dry_run)
  File "/home/user/.local/lib/python3.9/site-packages/gmail_yaml_filters/upload.py", line 215, in prune_filters_not_in_ruleset
    print('Deleting', prunable_filter['id'], prunable_filter['criteria'], prunable_filter['action'], file=sys.stderr)
KeyError: 'action'

AttributeError: 'dict' object has no attribute 'iteritems'

gmail-yaml-filters is failing for me, with any options, with the samples included in the documentation.

$ python3 -V
Python 3.6.5

$ pip3 list | grep yaml
gmail-yaml-filters       0.7.2

$ $ cat test-filters.yaml
# Simple example
-
  from: [email protected]
  label: news
  not_important: true

$ gmail-yaml-filters test-filters.yaml
Traceback (most recent call last):
  File "/usr/local/bin/gmail-yaml-filters", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/gmail_yaml_filters/main.py", line 595, in main
    ruleset = RuleSet.from_object(rule for rule in data if not rule.get('ignore'))
  File "/usr/local/lib/python3.6/site-packages/gmail_yaml_filters/main.py", line 487, in from_object
    return cls.from_iterable(obj, base_rule=base_rule)
  File "/usr/local/lib/python3.6/site-packages/gmail_yaml_filters/main.py", line 516, in from_iterable
    ruleset.update(cls.from_object(data, base_rule=base_rule))
  File "/usr/local/lib/python3.6/site-packages/gmail_yaml_filters/main.py", line 485, in from_object
    return cls.from_dict(obj, base_rule=base_rule)
  File "/usr/local/lib/python3.6/site-packages/gmail_yaml_filters/main.py", line 503, in from_dict
    new_rule = Rule(data, base_rule=base_rule)
  File "/usr/local/lib/python3.6/site-packages/gmail_yaml_filters/main.py", line 337, in __init__
    self.update(data)
  File "/usr/local/lib/python3.6/site-packages/gmail_yaml_filters/main.py", line 353, in update
    for key, value in dict(data).iteritems():
AttributeError: 'dict' object has no attribute 'iteritems'

Create rules in a specific order so that has:userlabels works as a filter condition

A few of my rules rely on ordering, which I understand it not guaranteed, however it seems to work. The specific requirement is that they test for the presence (or absence) or labels from other rules. This allows me to avoid giant nested structures. I can do this with 'has:userlabels' or 'has:nouserlabels'

Is there any potential for supporting this as a condition?

Applying labels to messages and threads

When we upload or sync a configuration, the filters are created but not applied to all old emails.

Is it possible to apply the filters to all conversations?

I read this but I'm not sure that is to do what I explain: https://developers.google.com/gmail/api/guides/labels#applying_labels_to_messages_and_threads

smartlabels

In the gmail filter UI, when I specify that a filter categorize something as "social", the resulting XML is:

<apps:property name='smartLabelToApply' value='^smartlabel_social'/>

gmail-yaml-filters wants to do:

<apps:property name="label" value="CATEGORY_SOCIAL"/>

And when used literally creates a label called CATEGORY_SOCIAL and does not place the email in the social tab. Seems that the smartlabel syntax is the appropriate one.

Reproducible output

Right now, the XML output that's generated contains two different things that differ from invocation to invocation: the ID (due to the use of hash) and the timestamp. I like to diff my XML against the previous version to verify the changes, and this adds a lot of diff noise.

It would be great if we could have a way to produce reproducible output. I have a branch that replaces the invocation of hash with CRC32 and allows the user to set the SOURCE_DATE_EPOCH environment variable used in reproducible builds to get a fixed timestamp.

Assuming this is okay with you, I'd like to submit that branch, unless you're aware of a different way to go about it.

sync with different gmail accounts?

Is it possible to use the sync capability with multiple gmail accounts at the same time (e.g. work account and personal account)? For example, is there a way to pass the gmail account as a CLI parameter?

feature request: forwardTo support

Hi, It would be great to have support for forwarding filter syntax.

Better detection of your own email address?

I noticed that gmail-britta has this feature:

https://github.com/antifuchs/gmail-britta#better-detection-of-your-own-email-address

Does this project have anything equivalent? If not, maybe I could hack that in if it was easy - any suggestions on the format of the YAML configuration for it? Thanks!!

No such file or directory: 'client_secret.json'

With v0.9.3, gmail-yaml-filters seems to expect client credentials even though I'm just using it to convert YAML to XML:

> pip3 install -U gmail-yaml-filters
> gmail-yaml-filters filters.yaml > filters.xml
/home/mark/.local/lib/python3.8/site-packages/oauth2client/_helpers.py:255: UserWarning: Cannot access /home/mark/.credentials/gmail_yaml_filters.json: No such file or directory
  warnings.warn(_MISSING_FILE_MESSAGE.format(filename))
Traceback (most recent call last):
  File "/home/mark/.local/lib/python3.8/site-packages/oauth2client/clientsecrets.py", line 121, in _loadfile
    with open(filename, 'r') as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'client_secret.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mark/.local/bin/gmail-yaml-filters", line 8, in <module>
    sys.exit(main())
  File "/home/mark/.local/lib/python3.8/site-packages/gmail_yaml_filters/main.py", line 93, in main
    credentials = get_gmail_credentials(client_secret_path=args.client_secret)
  File "/home/mark/.local/lib/python3.8/site-packages/gmail_yaml_filters/upload.py", line 275, in get_gmail_credentials
    flow = oauth2client.client.flow_from_clientsecrets(client_secret_path, scopes)
  File "/home/mark/.local/lib/python3.8/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/mark/.local/lib/python3.8/site-packages/oauth2client/client.py", line 2134, in flow_from_clientsecrets
    client_type, client_info = clientsecrets.loadfile(filename,
  File "/home/mark/.local/lib/python3.8/site-packages/oauth2client/clientsecrets.py", line 165, in loadfile
    return _loadfile(filename)
  File "/home/mark/.local/lib/python3.8/site-packages/oauth2client/clientsecrets.py", line 124, in _loadfile
    raise InvalidClientSecretsError('Error opening file', exc.filename,
oauth2client.clientsecrets.InvalidClientSecretsError: ('Error opening file', 'client_secret.json', 'No such file or directory', 2)

The previous version (pip3 install -U gmail-yaml-filters==0.9.2) works fine.

support for CC

Can we add a test for CC similarly to TO?

Generated XML file is missing the XML declaration

The generated XML file is missing the XML declaration:

<?xml version='1.0' encoding='UTF-8'?>

This results in an error, where gmail won't allow you to import the file. Make sure the generated file includes the XML declaration.

"to" is not mentioned in the supported conditions

"deliveredto" is mentioned but "to" is not mentioned. It is shown in the examples and it seems to work.

oauth2client is deprecated

Tried reinstalling this with both Python 2.x and Python 3.x. Both fail with:

ImportError: No module named oauth2client.client

Per https://github.com/google/oauth2client, oauth2client is deprecated anyways.

How to mark as SPAM?

Hi,

Great work with this app.

I was trying to mark something as SPAM, but adding label SPAM does not work and marking not_spam: false also doesn't. Am I missing something or it is not available yet?

Thanks,

C.