harishkrupo/oauth2ms

oauth2ms: XOAUTH2 compatible O365 token fetcher

Table of Contents

• Overview
• Dependencies
• Installation
  • XDG CONFIG HOME
• Usage
  • Encryption
• TODOS

Overview

This tool can be used to fetch oauth2 tokens from the microsoft identity endpoint. Additionally, it can encode the token in the XOAUTH2 format to be used as authentication in IMAP mail servers. Example configuration for emacs given in steps.org.

Dependencies

Requires mbsync >= 1.3. More information available at: #2 (comment)

Installation

Clone the repository and install the requirements.

pip install -r requirements.txt

Then copy the oauth2ms file to any location in your $PATH.

XDG CONFIG HOME

For the app to locate your config file, the $XDG_CONFIG_HOME environment variable must be set. It is generally set to $HOME/.config.

Usage

Create a config file with the below details at $XDG_CONFIG_HOME/oauth2ms/config.json

{
    "tenant_id": "TENANT_ID",
    "client_id": "CLIENT_ID",
    "client_secret": "CLIENT_SECRET",
    "redirect_host": "localhost",
    "redirect_port": "5000",
    "redirect_path": "/getToken/",
    "scopes": ["https://outlook.office.com/IMAP.AccessAsUser.All"]
}

If you aren’t sure how to fetch the the TENANT_ID, CLIENT_ID and CLIENT_SECRET values, read more on steps.org. After creating the config file, execute oauth2ms, it should pop up a browser window asking you to login. Once logged it, it should redirect you to a page which says “Authorization complete.”. On the terminal you should see the token printed. Subsequent fetches should use the refresh token to get the access token. Call oauth2ms with the --encode-xoauth2 to get the token in XOAUTH2 format.

oauth2ms --encode-xoauth2

Optionally, you can add “https://outlook.office.com/SMTP.Send” to the list of scopes to use the token for stmp. See steps.org for information on configuring emacs for sending mail via smtp with xoauth2.

Encryption

Encryption of the fetched token cache can be enabled using the -e (--encrypt-using-fingerprint) option. The option takes the email/fingerprint to identify the gpg key to use for encrypting and decrypting the token cache.

oauth2ms -e <gpg recipient fingerprint>

TODOS

• [x] Use gpg to encrypt the fetched data
• [ ] Add support for encrypted config.json
• [ ] Support more than one account at a time

oauth2ms's Issues

Support writing to/reading from keyring

Thanks for this tool!

I think it would be great if this supported saving client secrets and refresh tokens in the user's keyring. Perhaps the Secret Storage API (via secretstorage)could be used to interact with the keyring.

Sending Emails Fails

Hello, thanks for this package - finally able to work with email on emacs. Unfortunately, I am unable to send any emails.

I copied the code recommended in steps.org and added my smtp server details. When I try to send an email in emacs I am prompted for my password (rightly or wrongly, I was expecting to not be prompted, so I just hit return). It quickly fails after that. Here is what I can see (from the *Messages* buffer:

Sending...
530 5.7.57 Client not authenticated to send mail. [server.prod.outlook.com]
451 4.7.0 Temporary server error. Please try again later. PRX4  [server.prod.outlook.com]
221 2.0.0 Service closing transmission channel
smtpmail-send-it: Sending failed: 451 4.7.0 Temporary server error. Please try again later. PRX4 [server.prod.outlook.com] in response to AUTH

For more detail here is the trace of the SMTP session:

220 server.outlook.office365.com Microsoft ESMTP MAIL Service ready at Fri, 26 Feb 2021 22:05:56 +0000
250-server.outlook.office365.com Hello [98.217.221.170]
250-SIZE 157286400
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-STARTTLS
250-8BITMIME
250-BINARYMIME
250-CHUNKING
250 SMTPUTF8
220 2.0.0 SMTP server ready
250-server.outlook.office365.com Hello [98.217.221.170]
250-SIZE 157286400
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-AUTH LOGIN XOAUTH2
250-8BITMIME
250-BINARYMIME
250-CHUNKING
250 SMTPUTF8
AUTH XOAUTH2 [...token..]  [note: the token matches the output of the most recent `oauth2ms --encode-xoauth2`]
451 4.7.0 Temporary server error. Please try again later. PRX4  [server.prod.outlook.com]
QUIT
221 2.0.0 Service closing transmission channel

Process smtpmail connection broken by remote peer

I appreciate any guidance you can provide. Thank you!

NO AUTHENTICATE

Hi there,

I finally got a chance to try out this project on Saturday, but I've not been able to get it working with mbsync:

IMAPAccount mc
Host outlook.office365.com
Port 993
User [email protected]
PassCmd "oauth2ms"
SSLType IMAPS
AuthMechs XOAUTH2

IMAPStore mc-remote
Account mc

MaildirStore mc-local
Path ~/.mail/[email protected]/
Inbox ~/.mail/[email protected]/INBOX
# The SubFolders option allows to represent all
# IMAP subfolders as local subfolders
SubFolders Verbatim

Channel mc
Master :mc-remote:
Slave :mc-local:
Patterns *
Expunge None
CopyArrivalDate yes
Sync All
Create Slave
SyncState *

mbsync -l mc
IMAP command 'AUTHENTICATE XOAUTH2 <base64 hash>' returned an error: NO AUTHENTICATE failed.

The base64 hash looks like this when deserialized (I replaced some of the token to make it wrong):

[email protected]=Bearer eyJ0eXAiOiJKV1QiLCJub25jZSI6InJfMEU5VDU3aUh1Z21aaXXaaaaAaAaAA1aaA3AaAAa3A2AAAaA

The permissions look like this:

Adding smtp after using oauth2 for imap

Thank you for oauth2ms! This helped me continue to use emacs mbsync and mu4e when my university forced oauth2 imap this summer. At that time, smtp did not yet require oauth2 so I did not add code to my mu4e setup to use oauth2. That changed today. Therefore, I added to my .emacs based on the code in steps.org. I also made sure the config.json file had

   "scopes": ["https://outlook.office365.com/IMAP.AccessAsUser.All", "https://outlook.office365.com/SMTP.Send"]

And I had already set the smtp permissions in azure.

But attempting to send smtp results in an smtp trace like shown below (I deleted the token string). Any suggestion on where to start?

220 SN7P220CA0005.outlook.office365.com Microsoft ESMTP MAIL Service ready at Mon, 3 Oct 2022 15:50:36 +0000
250-SN7P220CA0005.outlook.office365.com Hello [<ip address>]
250-SIZE 157286400
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-STARTTLS
250-8BITMIME
250-BINARYMIME
250-CHUNKING
250 SMTPUTF8
220 2.0.0 SMTP server ready
250-SN7P220CA0005.outlook.office365.com Hello [<ip address>]
250-SIZE 157286400
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-AUTH LOGIN XOAUTH2
250-8BITMIME
250-BINARYMIME
250-CHUNKING
250 SMTPUTF8
AUTH XOAUTH2 

[TOKEN CHUNK HERE DELETED]

535 5.7.3 Authentication unsuccessful [SN7P220CA0005.NAMP220.PROD.OUTLOOK.COM]
QUIT
221 2.0.0 Service closing transmission channel

How to setup SMTP if one knows the access token already?

I obtained the token using instructions given here. I am able to download email with it. I tried to adapt the code you give for SMTP to use that token. I came up with this,

   (use-package smtpmail
      :ensure t
      :config
      (add-to-list 'smtpmail-auth-supported 'xoauth2)
      :init
      (cl-defmethod smtpmail-try-auth-method
        (process (_mech (eql xoauth2)) user password)
        (let* ((access-token "<token string here"))
          (smtpmail-command-or-throw
           process
           (concat "AUTH XOAUTH2 " access-token)
           235)))
      (setq message-send-mail-function 'smtpmail-send-it
            message-kill-buffer-on-exit t
            starttls-use-gnutls t
            smtpmail-default-smtp-server "smtp.example.com"
            smtpmail-smtp-server "smtp.example.com"
            smtpmail-stream-type  'starttls
            smtpmail-starttls-credentials
            '(("smtp.example.com" 587 nil nil))
            smtpmail-auth-credentials
            '(("smtp.example.com" 587 "[email protected]" nil))
            smtpmail-smtp-service 587))

When sending the email, the authentication still fails. Any ideas?

Mac install

Hi,

Is there an easy way to install this on a mac? I'm trying to configure apple mail to use with outlook 365 (imap disallowed). Thanks

urllib3.exceptions.LocationParseError: Failed to parse

Hi All,

I really think my config is close but I get a parse error ?

oauth2ms 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/models.py", line 379, in prepare_url
    scheme, auth, host, port, path, query, fragment = parse_url(url)
  File "/usr/lib/python3/dist-packages/urllib3/util/url.py", line 392, in parse_url
    return six.raise_from(LocationParseError(source_url), None)
  File "<string>", line 2, in raise_from
urllib3.exceptions.LocationParseError: Failed to parse: https://login.microsoftonline.com/589c76f5-ca15-41f9-884b-55ec15a0672a/v2.0/.well-known/openid-configuration

During handling of the above exception, another exception occurred:

OpenID

results in:

{"token_endpoint":"https://login.microsoftonline.com/589c76f5-ca15-41f9-884b-55ec15a0672a/oauth2/v2.0/token","token_endpoint_auth_methods_supported":["client_secret_post","private_key_jwt","client_secret_basic"],"jwks_uri":"https://login.microsoftonline.com/589c76f5-ca15-41f9-884b-55ec15a0672a/discovery/v2.0/keys","response_modes_supported":["query","fragment","form_post"],"subject_types_supported":["pairwise"],"id_token_signing_alg_values_supported":["RS256"],"response_types_supported":["code","id_token","code id_token","id_token token"],"scopes_supported":["openid","profile","email","offline_access"],"issuer":"https://login.microsoftonline.com/589c76f5-ca15-41f9-884b-55ec15a0672a/v2.0","request_uri_parameter_supported":false,"userinfo_endpoint":"https://graph.microsoft.com/oidc/userinfo","authorization_endpoint":"https://login.microsoftonline.com/589c76f5-ca15-41f9-884b-55ec15a0672a/oauth2/v2.0/authorize","device_authorization_endpoint":"https://login.microsoftonline.com/589c76f5-ca15-41f9-884b-55ec15a0672a/oauth2/v2.0/devicecode","http_logout_supported":true,"frontchannel_logout_supported":true,"end_session_endpoint":"https://login.microsoftonline.com/589c76f5-ca15-41f9-884b-55ec15a0672a/oauth2/v2.0/logout","claims_supported":["sub","iss","cloud_instance_name","cloud_instance_host_name","cloud_graph_host_name","msgraph_host","aud","exp","iat","auth_time","acr","nonce","preferred_username","name","tid","ver","at_hash","c_hash","email"],"kerberos_endpoint":"https://login.microsoftonline.com/589c76f5-ca15-41f9-884b-55ec15a0672a/kerberos","tenant_region_scope":"NA","cloud_instance_name":"microsoftonline.com","cloud_graph_host_name":"graph.windows.net","msgraph_host":"graph.microsoft.com","rbac_url":"https://pas.windows.net"}

What am I doing wrong ?

Thank you very much for writing this code.

Problem configuring URLs for Office365

While trying to configure oauth2ms, I get the following error after authenticating in my office365 server: AADSTS50011: The reply URL specified in the request does not match the reply URLs configured for the application: '20460e5d-ce91-49af-a3a5-70b6be7486d1'.

This is with the following values in the config (I'm using Evolution's client id, and it works in other scripts like mutt_oauth2):

   {
        "tenant_id": "common",
        "client_id": "20460e5d-ce91-49af-a3a5-70b6be7486d1",
        "client_secret": "",
        "redirect_host": "localhost",
        "redirect_port": "5000",
        "redirect_path": "/GetToken/",
        "scopes": ["https://outlook.office365.com/IMAP.AccessAsUser.All", "https://outlook.office365.com/SMTP.Send"]
   }

I'm confused about the redirect uri: shouldn't it be something like https://login.microsoftonline.com/common/oauth2/nativeclient? This is what mutt_oauth2 uses. But then, I don't have a port or a path...

Thanks in advance!

Sending failed SMTPclient is disabled for the tenant

Thank you very much for developing this package which allows me to receive emails from my work email account using mbsync.
However, I am unable to use it to send emails. At my work they use a O365 server and SMTP Authentication is disabled. I am not sure if this package assumes that this option must be turned on? Specifically, when I try to send an email with the suggested configuration for SMTP

Sending...
Sending via mail...
auth-source-search: found 1 backends matching (:max 1 :host "smtp.office365.com" :port "587")
Decrypting /home/myusername/.authinfo.gpg...done
auth-source-search-backend: got 1 (max 1) in netrc:~/.authinfo.gpg matching (:max 1 :host "smtp.office365.com" :port "587")
auth-source-search: found 1 results (max 1) matching (:max 1 :host "smtp.office365.com" :port "587")
auth-source-search: found 1 backends matching (:host "smtp.office365.com" :port "587" :user nil :max 1 :require nil :create nil)
auth-source-netrc-parse: using CACHED file data for ~/.authinfo.gpg
auth-source-search-backend: got 1 (max 1) in netrc:~/.authinfo.gpg matching (:host "smtp.office365.com" :port "587" :user nil :max 1 :require nil :create nil)
auth-source-search: found 1 results (max 1) matching (:host "smtp.office365.com" :port "587" :user nil :max 1 :require nil :create nil)
[mu4e] Update process returned with non-zero exit code
535 5.7.139 Authentication unsuccessful, SmtpClientAuthentication is disabled for the Tenant. Visit https://aka.ms/smtp_auth_disabled for more information. [FR3P281CA0054.DEUP281.PROD.OUTLOOK.COM]
221 2.0.0 Service closing transmission channel
smtpmail-send-it: Sending failed: 535 5.7.139 Authentication unsuccessful, SmtpClientAuthentication is disabled for the Tenant. Visit https://aka.ms/smtp_auth_disabled for more information. [FR3P281CA0054.DEUP281.PROD.OUTLOOK.COM] in response to AUTH

At a broader level, if nowadays SMTP is being disabled by MS what protocol is actually supported to send emails, and is there a way to replicate it? Thank you in advance for any help.

My msmtprc config file is

account work
host smtp.office365.com
from [email protected]
port 587
user [email protected]
passwordeval gpg2 -q --for-your-eyes-only --no-tty -d ~/.authinfo.gpg | awk '/machine smtp.office365.com login [email protected]/ {print $NF}'
auth login
tls on
tls_starttls on
tls_certcheck on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile ~/.msmtp.log

device code flow

My university seems to use the device code flow, I guess it is https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-device-code. I found https://docs.microsoft.com/en-us/python/api/msal/msal.application.publicclientapplication?view=azure-python and tried to edit your code to make it work, but I lack understanding of how everything works really. Could you help?

issue during setup after authentication

after setting up the azure app with the correct permissions, after the first run of oauth2ms I log in and grant permissions in the browser, then I get the following error after I see a blank page with "Authorization complete.".

Traceback (most recent call last):
  File "/home/japhir/bin/oauth2ms", line 226, in <module>
    app_state, token = build_new_app_state(crypt)
  File "/home/japhir/bin/oauth2ms", line 143, in build_new_app_state
    token = result["access_token"]
KeyError: 'access_token'

here's an overview of my permissions config:

what am I doing wrong?

'NoneType' object is not subscriptable

I am attempting to setup oauth2ms to work with mbsync, and while troubleshooting ran into the following error:

Traceback (most recent call last):
  File "~/.local/bin/oauth2ms", line 247, in <module>
    token = fetch_token_from_cache(app_state)
  File "~/.local/bin/oauth2ms", line 181, in fetch_token_from_cache
    return result["access_token"]
TypeError: 'NoneType' object is not subscriptable

This occurs when executing both mbsync -a and oauth2ms --encode-xoauth2. I am not sure what is producing this as it seemed to be working fine, albeit with the incorrect base64.

Token Returned, Authentication Error in mbsync

Hi @harishkrupo,

First off, thank you for writing this script and the step-by-step instructions here - they've been incredibly helpful to someone just starting out learning how this kind of authentication works. I have a slightly different error than issue #2.

When I run oauth2ms directly from the shell, I get a token that when plugged into jwt.ms seems correct (unique_id, scope, tenant and client ID all look right - but I do not know much about what I am looking at and might be missing something).

When I run mbsync -V, I receive the following output:

Opening far side store rice-remote...
Resolving outlook.office365.com... ok
Connecting to outlook.office365.com (52.96.103.18:143)...
Opening near side store rice-local...
Connection is now encrypted
Logging in...
Authenticating with SASL mechanism XOAUTH2...
Error performing SASL authentication step: SASL(-1): generic failure: Unable to find a callback: 18948

Here's the relevant part of my .mbsyncrc:

Host outlook.office365.com
User [email protected]
AuthMechs XOAUTH2
Passcmd oauth2ms

Searching for the whole or part of that error message has not gotten me to anything that has helped me. I am on macos and had to install the cyrus-sasl-xoauth2 plugin to /usr/local/lib/sasl2 instead of /usr/lib/sasl2, so it might be something with not finding the right plugin, but I was receiving a different error message before, and I think that I've resolved that piece of it. Changing to passcmd "oauth2ms --encode-xoauth2" returned the same behavior.

So my working hypothesis is that oauth2ms is properly fetching a token, mbsync is finding something to attempt xoauth2, but perhaps not the right plugin. It's also possible there's a setting in my azure that has been set by my organization's IT department that prevents this from working without an extra step (for example, I had to request permission the first time I ran oauth2ms, but that was granted).

I realize this is likely outside of the scope of issues for oauth2ms, but any troubleshooting advice you have would be much appreciated.

Alternative to obtain tenant ID and client ID and secret

Must say that what this project is set to achieve is amazing!
Unfortunately on step 3 of Azure app set up an "Access denied You don't have permission to register applications in the " message shows up. There is no point in trying to ask for permission at my workplace, they are too inflexible/unhelpful. So my question, which is probably a shot in the dark, is if there is any alternative way to obtain a tenant ID, client ID and secret?

Other information I was able to gather: In Azure's Active directory Overview section there is a "My feed" subsection which under my name displays a code 189cdec2-7acf-4b53-bb9a-9080ff602f56 (possibly this is my tenant or client ID?). Under the same section I can also see an Azure AD Connect icon and the type of license used in my company is Azure AD Premium P2. Thank you in advance for any help.

O365 Authentication Issues

I've set up oauth2ms according to the instructions provided in steps.org, and I've searched the web for any clues to what I may have done incorrectly. I've spent about 15 hours trying to troubleshoot this, I'm out of ideas, and I would be very grateful to anyone who could help! :)

Every time I execute oauth2ms it opens a browser window and returns a token there, with the message "Authorization complete." Back in the terminal I see this:

tfry@tfryX1:~$ oauth2ms Opening in existing browser session. libva error: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null) Something went wrong during authorization Server returned: {'error': 'invalid_client', 'error_description': "AADSTS700025: Client is public so neither 'client_assertion' nor 'client_secret' should be presented.\r\nTrace ID: 262be02c-ead2-4c32-a8e0-1007dc5ba700\r\nCorrelation ID: 5c9b96da-0533-46c7-b5ca-cdbfdd77f22f\r\nTimestamp: 2022-10-06 17:26:37Z", 'error_codes': [700025], 'timestamp': '2022-10-06 17:26:37Z', 'trace_id': '262be02c-ead2-4c32-a8e0-1007dc5ba700', 'correlation_id': '5c9b96da-0533-46c7-b5ca-cdbfdd77f22f'} Traceback (most recent call last): File "/usr/local/bin/oauth2ms", line 240, in <module> app_state, token = build_new_app_state(crypt) TypeError: cannot unpack non-iterable NoneType object tfry@tfryX1:~$

My config.json (the XXXs are simply obfuscating stuff):

{ "tenant_id": "9XXXXXX6-3bbd-49b8-a5bc-ecXXXXXX0b8", "client_id": "9XXXXXX5-10e4-4cdb-a13d-21XXXXXXXX71", "client_secret": "gYvXXXXXXXXXXXXXXXXXXXXXXXyfbAy", "redirect_host": "localhost", "redirect_port": "5000", "redirect_path": "/getToken/", "scopes": ["https://outlook.office.com/IMAP.AccessAsUser.All", "https://outlook.office.com/SMTP.Send"] }

What have I done wrong, and can you point me to next steps in troubleshooting this? Thank you so much in advance!!

Multi-account support implementation

Hi @harishkrupo,

Thanks for your efforts in this project! I see multi-account support is one of the TODO's and I'm willing to lend a hand if needed. I had some thoughts on how to implement this that I'd like to run by you. Would it be an option to:

• Add an optional command line parameter (e.g., --account [email protected])
• Set credentials file to whatever is passed under account (e.g., save_data_path("oauth2ms") + "/" + cmdline_args.account)

I'm not sure how this would work with the credentials cache or indeed if anything else would break. What's your take?

"AADSTS700025: Client is public so neither 'client_assertion' nor 'client_secret' should be presented.

Hi there. Great piece of code.
Is this error familiar?:
Server returned: {'error': 'invalid_client', 'error_description': "AADSTS700025: Client is public so neither 'client_assertion' nor 'client_secret' should be presented. ...

TypeError: 'NoneType' object is not subscriptable

Recently upgraded system (manjaro). Previous set up worked flawlessly. Now, running oauth2ms gives the following error:

Traceback (most recent call last):
  File "/usr/bin/oauth2ms", line 247, in <module>
    token = fetch_token_from_cache(app_state)
  File "/usr/bin/oauth2ms", line 181, in fetch_token_from_cache
    return result["access_token"]
TypeError: 'NoneType' object is not subscriptable

I am not sure what might have caused this and happy to provide any additional info. This package has been great and it appears that something else has now broken it. Thanks for the work and making this available to us

Using a public id

I have been trying to follow the instructions, but I cannot get past step one as Azure simply won't let me create an ID (I only have the free account perhaps this is a reason).

Is it possible to just use a publicly available client_id and client_secret (such as what Thunderbird uses)?

mbsync IMAP command 'NAMESPACE' returned an error: BAD User is authenticated but not connected.

Hello, after oauth2ms has been working perfectly (thank you very much @harishkrupo) for a couple of months, with the arrival of the new year it started spitting out IMAP command 'NAMESPACE' returned an error: BAD User is authenticated but not connected.. Removal and recreation of credentials.bin did not have any effect, neither logging out from outlook.com.
Any ideas what this is about?