tableau/vizalerts

Data-driven alerting for Tableau Server

What is it?

VizAlerts is a data-driven automation platform for Tableau Server. It allows any user to author dashboards that perform various actions based on any criteria you can define in your viz.

Actions currently supported are:

• Send email to any audience, with flexible control over who is sent what, and the ability to embed or attach images of your visualizations as PNGs or PDFs
• Send SMS messages to any audience using the Twilio service
• (Future) Export viz data and/or images to a file share

Once VizAlerts is installed and running, all of the work to build and manage your alerts happens in Tableau--no scripting required, and no separate interface necessary.

Got any documentation?

Do we ever! There are two files included in the \docs folder, install_guide.md and user_guide.md, intended for Tableau Server administrators and for alert authors, respectively. They're the best way to learn about what VizAlerts can do.

If you're an impatient Millennial like me, here's a video that skims the very basics.

How do I set it up?

Please see the Install Guide for installation instructions. Only the Tableau Server administrator needs to set it up. Once working, any user on Tableau Server who can publish may use VizAlerts.

What versions of Tableau are supported?

Tableau Server version 8.2.5 and higher is required (ideally version 9--if you're using version 8, some things won't work as well). Tableau Online is not currently supported, though we are looking at ways we might be able to achieve that.

How is it licensed?

Please see the LICENSE file in the root path for details.

Is VizAlerts supported?

VizAlerts is supported by the community. While Tableau's Professional Services team may be engaged to assist with the deployment and usage of this tool, VizAlerts is not officially supported by Tableau Software: It is a community-built and community-supported tool.

For general questions or issues, please bring them to the VizAlerts Group created on the Tableau Community site.

Bugs discovered, and feature aspirations will be tracked via GitHub's issue tracker.

How can I contribute?

If you're interested in contributing to VizAlerts, please email Matt about what you're interested in doing. We've found it's the easiest way for us to coordinate planned changes amongst ourselves.

What's the longer-term vision for this tool?

Initially, VizAlerts was born out of a hackathon we held at Tableau, and it was conceived of as a proof-of-concept of how data analyzed through Tableau could be used as a programming language of sorts to drive automation of various tasks. The most critical and simplest task to tackle first was email automation, so that's what has been primarily focused on to date. But more actions have been added recently, and we're interested in finding other ways that VizAlerts can be used to drive automation, such as exporting data to file shares, making changes to other content on Tableau Server, and generally hooking into APIs exposed by various other third-party services.

vizalerts's Issues

Add config option for SMTP server port

We connect to the SMTP server without specifying a port number right now. If the SMTP server uses a non-default port, this will cause VizAlerts to not work (and reportedly, through a confusing error .

Solution: add a line to the vizalerts.yaml file to support specifying a non-default SMTP server port.

Unable to cleanup temp directory fatal error when file in temp directory is open

When a file in the temp directory is open for editing (such as when someone is checking to see what was sent) VizAlerts generates an "ALERT: VizAlerts failed to execute properly" email with the message "Unable to cleanup temp directory .\temp, error:". This halts all other VizAlerts processing.

I don't think this should be a fatal error, I think it's more of a warning condition. My suggestion would be for VizAlerts to ignore the locked file while it cleans up the rest of the files and generate the warning.

Jonathan

Add ability to ignore certificate check for SSL

Currently if you configure VizAlerts to use encryption to connect to your Tableau Server instance (which is highly recommended), if for some reason your certificate isn't recognized as valid, the application will error and you will not be able to use VizAlerts. The workaround of switching to unencrypted requests is not ideal.

Instead, a new configuration line should be added which allows the administrator to disable cert checking, as in some cases the certificate may be valid / trustworthy, but technical issues can prevent it from being recognized as such.

(The same option should probably be added for SMTP server cert checking as well, but I haven't given that much thought yet)

VizAlerts trigger for IFTTT

Enable VizAlerts to be used as a trigger for IFTTT. This would likely be a new type of action.

Add support for SMS messages in Advanced Alerts

Noticed there was never a specific Issue for this, so this is mainly being created for visibility.

@jdrummey advocated for and added a feature to allow VizAlerts the ability to send SMS messages via the Twilio service, and submitted a pull request for that feature. I integrated that code into the project, with a few tweaks.

SMS actions are now possible (pending testing and integration to master) via the SMS Action fields:

SMS Action *: Represents whether or not an SMS message will be sent for this row of data. A value of "1" indicates a message will be sent. Currently, any other value will cause no messages to be sent for that row.
SMS To *: String value representing a single phone number, or a list of numbers delimited by commas or semicolons (spaces not acceptable, unlike email addresses!)
SMS From ~: String value representing a single phone number that the SMS will originate from. If left alone, the default from number in the configuration file will be used. Given that you must purchase additional "from" numbers, for Twilio, this seems like it won't be used too much.
SMS Message *: Body text for the SMS message being sent. Only the VIZ_LINK content reference will be recognized.

With the new twbconfig changes, the Administrator will have extremely flexible control over who can send SMS messages from what alerts, and what numbers are allowed to be sent to.

Lots of flexibility will be allowed in how users specify their phone numbers, as the phonenumbers Python library handles that for us. If the number isn't in E.164 format, we'll attempt to validate using the country code provided by the twbconfig (this can also be dynamically adjusted).

Note that I am not implementing any consolidation logic at this time, as we've done with Emails, and note also that you cannot use both an Email Action and an SMS Action (or any other action for that matter) when you are consolidating your emails.

Off-line Python install for Windows needed

From this post https://community.tableau.com/message/390189#390189 I was able to install the files requests_ntlm and requests. The file doesn't contain pypdf2 since this is a new file needed for this 1.0.0 release.

So for step 3.c. of "Install Python & Required Modules" I had to use the .zip file Matt supplied (above URL) because I could not get the necessary files as suggested in Appendix A of the documentation. Also, in order to use pip the user will need Python installed on the computer they are using in order to get the Python files for their Tableau Server -- that certainly doesn't make things any simpler!

I recommend having the necessary Python files pre-packaged in the .zip file (in their own folder) so they can be used for new installs and ignored for upgrades unless necessary like in my situation.

Note that steps 1, 2, and 3.a & 3.b were easy to do. I was able to get the executable files, save them locally to my laptop, paste them to my Tableau Server, and then run them as an administrator on my Tableau Server.

Export to file

A set of code already exists for this feature at https://github.com/jdrummey/VizAlerts/tree/file-export.

There are several cases for exporting a viz to a file:

• Using a Tableau viz as a data preparation tool where a CSV of the viz is used for some other process.
• Downloading snapshots of a Tableau viz as it exists at a point in time to create an archive. Some data sources do not preserve the necessary information needed to roll back to a point in time so having the archive is useful.
• Downloading PNGs of a viz that can be placed in a folder where a PowerPoint has links to those files. This enables the PowerPoint to always have the latest versions of the PNGs.
• Once Issue #20 is implemented then it's foreseeable that some users would want to export a TWBX to a share accessible by Tableau Reader users.

Support TWBX as a content reference

There are Tableau deployments where there are a number of Tableau Reader users so having the ability for VizAlerts to distribute TWBX would be useful.

The challenge here is that the logic about what gets downloaded is inside Tableau. What I do know is that if a workbook has only live connections then a TWB is downloaded, if a workbook has only static connections (including extracts) then a TWBX is downloaded, I'm not sure of the both state. I suspect that might be a TWBX (that wouldn't be fully usable in Reader).

Given the amount of effort it would take to build a process that would check all the data sources and generate extracts (and particularly the security risks that might create) I suspect that it would be better to put the burden on the viz creator to make sure that the workbook will be downloaded as a TWBX, and the VizAlert scode would error if a TWB was downloaded.

VizAlerts workflows (i.e. chaining actions, VizAlerts as IFTTT)

The idea here is to enable relatively simple chaining of actions into workflows. A couple of examples:

On condition X VizAlerts will download a workbook from dev, change data sources in the workbook to point to production, publish the workbook to production, and send out a notification on success/failure.

On condition X VizAlerts will send out advanced alert views to designated recipients, on condition !X VizAlerts will send SMS notifications to designated recipients. This is one we can do now inside a single advanced alert by playing with the Email Action & SMS Action fields, or via multiple advanced alerts.

Other workflow actions besides downloading a workbook, updating data sources in a workbook, and publishing a workbook might include running arbitrary Python/PowerShell/batch script code, updating parameters in a workbook, triggering an IFTTT action see #73, changing Tableau Server settings (or even restarting Tableau Server), etc.

Implement smart default behavior for Advanced Alerts

Now that Conditional Subscriptions are in 10.1 and released to the world (yay!), I'm thinking about how to make Advanced Alerts easier. This thought came to me as I thought about how I'd want alerting to work in Tableau Server, in an ideal world.

In my opinion, alerting should work as an extra set of filters that overlay the standard viz data. There should be a shelf much like Filters as it exists right now, where you can drag any field to it, filter it, and expect whatever meets that criteria to be "alertable". The difference is that it doesn't actually hide your viz data. This would allow you to define arbitrary criteria for your alerts, just like VizAlerts does, but without the side effect of hiding all of it when you don't want to be alerted. Rather, there should be some way to simply toggle a "highlight marks meeting alert criteria" so you can see the alert effects, while keeping the context for the data behind your alert intact.

As I was thinking about how we could do something like this with VizAlerts, it occurred to me that we almost can. Advanced Alerts look at the value for the Action Flag (Email Action, SMS Action fields) to determine whether to actually do anything now. So if you had your alert criteria defined in the Email Action calc, like so:

IF [alert condition] THEN 1
ELSE 0
END

...you could keep all the data on your viz, and emails would only be sent when your alert condition occurred. This would hold true for both Email Actions and SMS Actions. All while keeping the data visible.

The next thought is, okay--so we have our alert criteria set, so how does VizAlerts know what to do when your alert is triggered? I think we can make things easier by implementing smart defaults, rather than using "required" fields. Using this approach rather than the current strict implementation helps people "see and understand" their alert, as it was, and get progressively more advanced with their capabilities.

So, an example would be "alert me when this bar gets over a certain threshold". To do that, they'd edit the Email Action calc such that when SUM([Bar Size] > Threshold), it returned a 1. In the current world, they'd also have to add Email Subject, Email Body, Email To to the level of detail, or one of the Rows or Columns shelves, before VizAlerts would actually work. But why should they have to?

If they add no other attributes, we should assume they want to email just themselves, with the generic view name for the email Subject, and the body will contain just the VIZ_IMAGE() content reference. Basically, just a Simple Alert. Easy, right? And they can still see their data!

Okay, now they can get more advanced. They want some more stuff in the Body. To accomplish this, they'd drag Email Body to the Detail Shelf, and edit the calc (we should use VIZ_IMAGE() in the body by default, perhaps), add their HTML content. Still sends just to them, still the generic Subject, but now they have their custom Body.

But what if they add one of their data fields to the Body, and it changes the level of detail in the viz? They're going to get more emails! Well, yes, probably. But this will all be visible in their viz--with Body on the Detail shelf, the marks will now be more numerous, so that fact should become more obvious. And if they want to see which marks will be responsible for sending said emails, they can add Email Action to the Color shelf, and instantly see it. Remember that even if they end up telling VizAlerts to send duplicate emails, it won't send them, because we remove all instructions that are exact duplicates.

I have yet to build a truly compelling walkthrough for how this could work, but I think it's really worth implementing, and building some videos around. I think this is really how the alert experience should be working in VizAlerts, and hopefully down the road, Server itself as well.

Support additional formats for embedded content in alert emails (csv, pdf)

Advanced Alerts allows the embedding of a Tableau Server View, rendered as a PNG image, in the emails that are sent via the VIZ_IMAGE() placeholder text. Users have requested that CSV and PDF formats be embeddable into the emails as well. This would need to be done after Allow embedding of customized View content in Advanced Alerts, for the CSV side of things.

Thereafter, there could be three placeholder text values for these three content types:

VIZ_IMAGE()
VIZ_CSV()
VIZ_PDF()

In thinking about this feature, this set of faux-functions could also be used to customize Simple Alerts as well, simply by adding them to the level of detail in the view data. In that case, they would toggle the addition of the content to the Simple Alert email. If present, the content of that type would be embedded in / attached to the email. If left out, nothing would be sent except the email subject, a blank body, and the standard footer.

By default, Simple Alerts always send the rendered viz image and attach the CSV of the view data. I don't think we'd want to change that. So within a Simple Alert, it seems like they should work like this:

Function	Value	Result
VIZ_IMAGE()	(field missing)	View image is embedded
VIZ_IMAGE()	null	View image is embedded
VIZ_IMAGE()	1	View image is embedded
VIZ_IMAGE()	0	View image is not embedded
VIZ_CSV()	(field missing)	View CSV is embedded
VIZ_CSV()	null	View CSV is embedded
VIZ_CSV()	1	View CSV is embedded
VIZ_CSV()	0	View CSV is not embedded
VIZ_PDF()	(field missing)	View PDF is not embedded
VIZ_PDF()	null	View PDF is embedded
VIZ_PDF()	1	View PDF is embedded
VIZ_PDF()	0	View PDF is not embedded

In the case of multiple values that disagree, the most restrictive values found would trump others--in other words, having the field being included in the view with a value of "0" would trump another value found of "1" or anything else.

For Advanced Alerts, all of these should be able to accept valid paths to Tableau Server views, with customizable URL parameters as described here. See comments at that link for why we would not do this for Simple Alerts.

Add subscriber_groups and owner_groups to VizAlerts schedule viz

This is something Matt & I talked about at one point in the development of VizAlerts 2.0: The idea is to have concatenated strings (delimited by a field that is not allowed for Tableau Server group names, or some set of characters that is likely to be unique if one doesn't exist) of the groups that the subscriber and viz owner belong to.

The use case for this is that we could write more specialized formulas for filtering who can subscribe and/or create trigger views based on their group membership. For example a filter like CONTAINS([owner_groups], "VizAlertsOK") could restrict to only publishers in the VizAlertsOK group.

I believe the string_agg() function https://www.postgresql.org/docs/9.4/static/functions-aggregate.html used in a subquery that is joined to the custom SQL would work here.

Installation: Use REST API to install VizAlerts schedule viz

With VizAlerts 2.0 it's now necessary to have Tableau Desktop, the PostgreSQL driver, and access to the PostgreSQL repository for the machine that Tableau Desktop is running from. Not all VizAlerts admins have easy access to all of those so to speed up VizAlerts installation and reduce chances of error I propose using a script that leverages Tableau's REST API to do the following:

• asks for Tableau Server password
• asks for the readonly password
• asks for the site & project to publish in
• publishes workbook with embedded password for the Custom SQL data source

Then from there the user could open the published workbook via web editing and change options as needed.

|filename doesn't allow all characters in VIZ_LINK() content reference

The |filename argument is used for both actual filenames as well as the optional text of the VIZ_LINK() content reference. For filenames only -_.[a-zA-Z0-9] are allowed values, so something like "Josie's Dashboard" isn't allowed because it's got a single quote.

I've got two ideas on this:

1. use a separate argument for VIZ_LINK() for the displayed text of the URL (instead of |filename)

2. allow |filename to support more complex characters when used in the VIZ_LINK argument.

I did an implementation of the latter, here's the code starting at line 1340 in the find_viz_refs() function:

                                # looking for |exportfilename
                                if element.startswith(EXPORTFILENAME_ARGUMENT):
                                    filename = re.match(EXPORTFILENAME_ARGUMENT + u'=(.*)', element).group(1)

#~~~added these two lines
                                    # if vizrefformat.group(1) is other than LINK then go through all validation tests
                                    if vizrefformat.group(1) != 'LINK':
#~~~end of added lines next section of code is indented for the if condition
                                         # code from https://github.com/mitsuhiko...

Add support for unicode email addresses

Since unicode is supported for domain names we should support unicode email addresses.

VizAlerts Error: HTTPError generating trusted ticket:

Hi Guys,

I have configured vizalerts, however i am getting the below error in an email, can you please help me out. FYI, i have configured Trusted authentication too,any help is appreciated

HTTPError generating trusted ticket: Not Found Request details: Server: mycompany.domain.com.com, Site: , Username: _, Url: http://mycompany.domain.com/trusted, Postdata: username=_.

Below is the error, when vizalert.py is executed:

C:\VizAlerts>vizalerts.py
2016-06-16 16:34:26 - [INFO] - LoggerQuickSetup - Logging initialized, writing t
o .\logs\vizalerts.log_2016-06-16.log
2016-06-16 16:34:26 - [DEBUG] - trusted_ticket_test - testing trusted ticket: http://mycompany.domain.com, , None
2016-06-16 16:34:26 - [DEBUG] - get_trusted_ticket - Generating trusted ticket.
Request details: Server: mycompany.domain.com, Site: , Username: ***, Url: http://mycompany.domain.com/trusted, Postdata: username=**.
2016-06-16 16:34:27 - [ERROR] - get_trusted_ticket - HTTPError generating truste
d ticket: Not Found Request details: Server: mycompany.domain.com, Site: ,
Username: ***_, Url: http://mycompany.domain.com/trusted, Pos
tdata: username=_.
2016-06-16 16:34:27 - [ERROR] - trusted_ticket_test - HTTPError generating trust
ed ticket: Not Found Request details: Server: mycompany.domain.com, Site:
, Username: ***, Url: http://mycompany.domain.com/trusted, Po
stdata: username=**.
2016-06-16 16:34:27 - [INFO] - send_email - sending email: smptservername,[email protected],**@domain.com,None,None,ALER
T: VizAlerts failed to execute properly,None
2016-06-16 16:34:27 - [DEBUG] - send_email - email body: HTTPError generating tr
usted ticket: Not Found Request details: Server: mycompany.domain.com, Sit
e: , Username: ***_, Url: http://mycompany.domain.com/trusted,
Postdata: username=_****.
2016-06-16 16:34:27 - [ERROR] - - An unhandled exception occurred: Trac
eback (most recent call last):
File "C:\VizAlerts\vizalerts.py", line 1654, in
main()
File "C:\VizAlerts\vizalerts.py", line 203, in main
trusted_ticket_test()
File "C:\VizAlerts\vizalerts.py", line 365, in trusted_ticket_test
quit_script(errormessage)
File "C:\VizAlerts\vizalerts.py", line 895, in quit_script
sys.exit(1)
SystemExit: 1
Traceback (most recent call last):
File "C:\VizAlerts\vizalerts.py", line 1654, in
main()
File "C:\VizAlerts\vizalerts.py", line 203, in main
trusted_ticket_test()
File "C:\VizAlerts\vizalerts.py", line 365, in trusted_ticket_test
quit_script(errormessage)
File "C:\VizAlerts\vizalerts.py", line 895, in quit_script
sys.exit(1)
SystemExit: 1

Email Attachment ~ with commas causes Tableau Server to deliver one row for trigger view

In an Advanced Alert trigger view if the content references in the Email Attachment ~ field are separated with commas then Tableau Server will only deliver one row in downloaded trigger view CSV.

For example VIZ_PDF(workbook1/view1), VIZ_PDF(workbook1/view2) fails.

I'm not sure what we can do about this besides document it because it's Tableau Server that has the problem.

Add URL shortening option for naked VIZ_LINK references for SMS alerts

This is something that I thought about when building SMS and never got around to writing up. URLs use up a lot of characters and a unpredictable number of characters (especially when URL parameters are used) so having the ability to have something like a |shortenURL option for VIZ_LINK() content references that used a pre-configured URL shortener would be helpful to trim down the URLs into a known length.

Can VizAlerts send notification by Active Directory Group, instead of email addresses?

Export CSV to Excel XLSX

There was a question from the VizAlerts presentation at TC16 https://community.tableau.com/docs/DOC-10039 on exporting directly to Excel .XLSX format since apparently some users demand that instead of CSV.

This is a pretty common feature request in Tableau, I think VizAlerts could be a good place for it once file export is implemented. I think the way to do this is with a new content reference like VIZ_XLS().

There's a pandas.DataFrame.to_excel method that could be used for this. Here are possible features in rough order of how much I could imagine people wanting them, this issue could be split up or modified as specific functionality is implemented:

1. Barebones conversion with write to specified workbook & default worksheet name, the existing |filename option could be used for the workbook. This would overwrite any existing workbook.
2. Support naming a worksheet in a workbook, this could be a new |worksheet option. This would overwrite an existing worksheet name in the workbook.
3. Support appending a worksheet to existing workbook (create worksheet and potentially workbook if it doesn't exist). This could be a |append option on the content reference.
4. Support ordering of columns & data types. For this I'm thinking of a side-along sheet (probably an additional view rather than a file) that would have the following fields: Original Field Name, Output Field Name, Output Field Order, and Output Data Type, this could an option on the content reference like |schema=schemaworkbook\view-name
5. Support downloading data into multiple worksheets into a single workbook.
6. Support appending downloaded data to an existing worksheet.
7. (un)pivoting export of Measure Names/Values tables to make a "wide" sheet. This particular feature could be extended to the VIZ_CSV() export as well.

Several other requirements:

• The code would need to check for the # of rows output due to the 1M row Excel limitation. I'd suggest creating additional worksheets with _1, _2, etc. appended to the worksheet name in this case.
• Unicode needs to be accepted.
• Dates need to be handled well, though I'm not totally sure what "well" means.
• Leading 0's in postal/zip codes need to be handled (or well documented so they don't get truncated).
• The code would need to gracefully handle situations where the workbook was open and not writeable. My thought here is to a) (in the case of append) copy the existing workbook to the temp folder, b) do all writes on the temp version, then c) copy the updated version back to the actual location. If c) fails then an informative error message could tell the user that a prepared workbook exists in the temp folder.
• The code would need to be threadsafe. I'm not sure how this would be accomplished.

Use tabcmd for exporting PDFs instead of URL actions

This is courtesy of Bryant Howell @ Tableau: There are tabcmd options (such as page size) for PDF downloads that aren't available when grabbing PDFs using URLs. Bryant created a version of the tabhttp.py library that is object oriented and uses tabcmd instead of URL actions. The tableau_email.py file (attached as a txt file below because github is picky that way) has a TableauEmailer object that is a framework for reading out subscriptions and sending the email from them, and a TabCmd object that wraps all the tabcmd options and gets the necessary info from the repository.

The TabCmd object uses trick that Bryant figured out where the configuration file for tabcmd is modified with information from the repository to switch tabcmd onto a session that is established via trusted tickets.

Here's a code sample from Bryant using both the TableauEmailer and TabCmd objects.

`
from tableau_email import *

tableau_server_admin_user = 'bhowell'
tableau_server_admin_pw = ''
tableau_server_url = 'http://127.0.0.1'
tabcmd_dir = "C:\tabcmd\Command Line Utility"
tabcmd_config_location = 'C:\Users\bhowell\AppData\Local\Tableau\Tabcmd'
tabcmd_repository_pw = ''
smtp_server = '

schedule_name_to_keep = "-- Full PDF Every Morning"

tableau_emailer = TableauEmailer(tabcmd_dir, tabcmd_config_location, tabcmd_repository_pw, tableau_server_url,
tableau_server_admin_user, tableau_server_admin_pw, smtp_server)
tableau_emailer.tabcmd.export_type = 'fullpdf'
tableau_emailer.tabcmd.export_page_layout = 'landscape'
tableau_emailer.generate_emails_from_named_schedule_in_repository(schedule_name_to_keep, '[email protected]',
'basic_email')

tableau_emailer.generate_email_from_view('[email protected]', '[email protected]', email_subject,

'basic_email', user, view_location)

tableau_emailer.tabcmd.site = 'default'
for region in ['East', 'West', 'Central', 'South']:
print "Bulding the {} e-mail".format(region)
tableau_emailer.generate_email_from_view('[email protected]', '[email protected]',
'This viz is the {}!'.format(region), 'basic_email', 'bhowell',
'Book1/Dashboard1', view_filter_map={"Region": region},
email_content_type='csv')
`

Here's my (Jonathan's) take on this:

• Being able to support all the possible options for PDF output would be great since otherwise it forces the single PDF output option defined by the viz Page Setup settings. There's obviously a need for this because Bryant built this code for a client.
• It would limit VizAlerts to Windows-only at this time since tabcmd doesn't run on Linux. We haven't explicitly made VizAlerts to run on Linux, but there's not much to prevent that. There's been at least one public ask on the forums for Linux support.
• tabcmd would need to be installed on the same machine as VizAlerts which adds another step to the setup process.
• My experience with tabcmd through v8.3 is that it's 50-100% slower than direct URL actions, I don't know how performance is with v9+.

If we did this the anticipated code changes would be:

• Add a configuration option to tell VizAlerts where to find tabcmd
• In vizalerts.py the get_viz_ref function would need to have all the tabcmd PDF options as arguments for content references
• In tabhttp.py:
  • add the TabCmd code
  • for VIZ_PDF use tabcmd for the downloads
• Update the documentation

tableau_email.py.txt

Create an installer application for Windows environments

Setting up and configuring VizAlerts is currently a pain point. Python and several packages must be installed, the config file can be intimidating and is easy to get wrong, and there are several Tableau Server changes that need to be made. To whatever degree feasible, it'd be great to have an installer perform the work. Ideally it should:

• Copy files to a user-designated folder
• Install Python
• Collect Tableau Server information
  • Ideally, configure and test trusted tickets
• Collect / obtain Tableau Server PostgreSQL repository info
  • Ideally, test database connection
• Collect mail server information & email addresses
  • Ideally, test mail server
• Create the Scheduled Task (or Service, in the future?)
• Simple test to ensure things are working
• Ideally, publish datasources and workbooks
• Instruct user as to next steps & further customization (schedule creation, manuals, config file, help forums, etc)

All that's a lot of work, and it tends to assume that the installer is running from the Tableau Server machine.

This Issue is also related to @jdrummey 's #62

Add WhatsApp support

As an alternative (or addition?) to SMS delivery add support for WhatsApp.

PATH is an international healthcare NGO working with the Zambian Ministry of Health to eliminate malaria, with the support of the Tableau Foundation. There are community health workers who have feature phones (so email is not an option) who primarily use WhatsApp rather than SMS.

There is no official WhatsApp API, there is a Python library at https://github.com/tgalal/yowsup.

Given the risk that the community-built library could be made invalid by a change to WhatsApp any deployment needs to have a fallback, so I propose having the ability for VizAlerts in the same installation to support both an SMS delivery (such as via the Twilio integration) and the WhatsApp integration. I'm thinking that we'd add a new Email Action value of 3 for WhatsApp, then if a user needed to fall back they could change their trigger views to use 2 instead. Otherwise the code would look much like the Twilio integration.

Some other notes:

• This is meant to be one-way outbound messaging only.
• The yowsup library has support for more logging and checking delivery status, we'd use the minimal amount in this first implementation.
• A requirement for WhatsApp is a usable cellphone number so there would need to be a number that is not attached to an install of the WhatsApp application on a device.

Add multi-threading to support timely alert processing

When a set of alerts are scheduled to run at the same time, VizAlerts currently processes them one at a time. This isn't ideal, since people subscribing to alerts at popular times can cause a large backlog of to process, which in term causes them to run later than they should. That's not always a huge concern in and of itself for many situations, but it becomes a bigger problem when so many alerts will be relying on relative date filters to prevent sending the same alert twice. Timely alert processing is critical for that technique to work, so it's important that we build that feature into VizAlerts.

Furthermore, this feature is important for enabling us to build a new feature for dynamic view rendering in Advanced Alerts, as a single Advanced Alert would then be processing an arbitrary number of views to render for the multiple emails it sends.

It also probably makes sense to have a Queue for alert processing, and a separate Queue for email processing, with config values setting the thread limit for both, since exporting View data from Tableau Server takes much longer than processing emails, and a single Advanced Alert can send multiple emails. But it may be easier to focus on a single queue for alert processing overall for the time being. Either way, the admin should have control of the number of concurrent operations being run.

Change CSV output to use LF instead of CR

In tabhttpy.py the export_view() function has the following line #211:
f.write(response.content.replace('\r\n', '\r')) # remove extra carriage returns

The CR written by \r is a rarely used form of newline, for a single newline character LF (\n) is the much more common character (see https://en.wikipedia.org/wiki/Newline for a discussion). This can cause problem for downstream scripts that are expecting CRLF or LF but not CR.

I propose replacing the \r with a \n for the newline, so the line would be this:
f.write(response.content.replace('\r\n', '\n')) # remove extra carriage returns

SMTP email sent over SSL due to Unicode characters

This should be easy to fix.

From an astute user:

I ran into a small but irritating problem when emailing with a provider that uses SSL (Strato.com to be precise) .
I got the error that the character mapping was not correct:
2016-07-27 00:30:21 - [ERROR] - send_email - Email failed to send: character mapping must return integer, None or unicode
2016-07-27 00:30:21 - [ERROR] - process_csv - Alert was triggered, but encountered a failure rendering data/image:
character mapping must return integer, None or unicode
2016-07-27 00:30:21 - [ERROR] - process_views - Unable to process data from viewname -redacted-, error:
Alert was triggered, but encountered a failure rendering data/image:
character mapping must return integer, None or Unicode

After some trials and errors and googles I found this:
http://stackoverflow.com/questions/14754301/issues-with-smtplib-unicode-and-openerp-core

And after changing
if configs["smtp.user"]:
server.login(configs["smtp.user"], configs["smtp.password"])
to
if configs["smtp.user"]:
server.login(str(configs["smtp.user"]), str(configs["smtp.password"]))

the problem was solved.

"-1" trusted ticket failure due to bad domain value passed into get_trusted_ticket

https://community.tableau.com/message/503588

tabUtil\tabhttp.py, line 180 needs to be:

ticket = get_trusted_ticket(server, sitename, username, encrypt, logger, configs["server.certcheck"], subscriberdomain, clientip)

Right now it's passing in the clientip for the user domain because it's not sending the certcheck information in like it should be.

Derive alert list to process from a Tableau Server viz rather than a direct PostgreSQL query

Rather than a large and ungainly SQL query in the vizalerts.yaml file being used to pull the list of alerts for VizAlerts to process, the CSV data from a Tableau Server view should be used instead. There are multiple reasons why this is a good idea, but the primary one is that certain config values that are currently global in nature can be customized much more flexibly on an alert-by-alert basis.

Take, for example, viz processing timeouts. Right now, they are fairly flexible, but apply to all vizzes universally. There is no way to grant exceptions to extremely critical alerts that you may wish to.

Another example is the allowed email domain list. Currently you must whitelist domains that apply to ALL alerts. Opening up access does so for everyone building any alert, which is not desirable. There may be specific authors that you trust to be careful, or certain vizzes that must be able to send email to ANY domain rather than a small set of them.

Pulling the data from a Tableau viz means that calcs can be used to define configurable values that override defaults set by parameters, like this example for a [Timeout (s)] calc:

IF [Owner] = 'mcoles' OR [Project] = 'Zambia' THEN 3600
ELSE [Default Timeout (s)] // default set via parameter]
END

This has the additional benefit of supporting an ExportCSV type function, as the shares a given user or alert are able to push data to can be dynamically derived much more easily than with the static SQL query in the config file.

Another side benefit is that other sheets in the same workbook could be used for Admin views showing the list of alerts to process, without any additional work.

Possible risks to the approach are that it's more resource-intensive, and prone to failure, to pull data from a Tableau Server viz rather than a PostgreSQL query directly.

So the implementation here would be:

• Take the existing SQL query out of vizalerts.yaml and create a connection out of it, and save it as a Tableau workbook.
• The admin would then publish that workbook to Tableau Server (permissions would need to be severely restricted--ADmins only!)
• VizAlerts.py, rather than calling PostgreSQL directly, would instead use the tabhttp module to pull down the CSV data
• The CSV would then be converted to a list of dictionaries in Python, which would be returned by the get_views() function.

Capture VizAlerts operational data for analysis / reporting / alerting

Currently VizAlerts does not store information on what it is doing in a structured way. Text logs are written to, but they're not built in a structured way that one could do analysis on.

VizAlerts should be given the ability to describe the actions it has taken, how long they took, and whether the outcome was a success or failure to a relational database such as PostgreSQL so that the administrator can monitor the health of alerts, and alert authors can see the history of their own alerts.

This would allow for many possible use cases:

• Build custom alerts for alert failures (currently VizAlerts can send an email to the admin and subscriber should an alert fail, but it's not customizable)
• Monitor alert trends
  • How many unique alerts are there?
  • How many emails were sent to who, when? How large are they?
  • How many data rows are being exported to CSV? How large are they?
  • Which are Simple Alerts, and which are Advanced Alerts?
  • Which alerts are triggered all the time? Which are never triggered and may need to be retired?
  • A user complains that they did not receive the email. Did VizAlerts send it successfully?

Some of this data can be obtained from the Tableau Server repository database, but it is not retained for long enough and does not contain enough information to answer all of these questions.

Once this item is complete, ideally a Tableau Datasource file (.tds) would be added to the release which allows admins and users to connect to the database this info is stored in. User filtering would provide row-level security so that users could not see others' information.

Locales defaulting to semicolon delimiters cannot use Advanced Alerts

In some countries, the standard text delimiter used is ";" rather than ",". This (apparently) results in Tableau Server exporting a CSV using that delimiter rather than a comma, but VizAlerts is still expecting a comma.

We should find a way to adjust the behavior to whatever the appropriate setting is for the locale, and use that.

See https://community.tableau.com/message/475397 for more details.

Add support for animated GIFs

Paul Chapman had a blog post recently about putting animated GIFs of sets of views in a Tableau Server Project as a graphical way to show what's in the Project: https://thatdatavizguy.com/2016/04/27/how-can-you-make-tableau-server-project-folders-more-interesting/

I'm thinking this would be a "bling" feature, it would be done by:

1. making it an optional install for the extra required libraries
2. add the necessary arguments for content references
3. using the same logic as for merging PDFs, only this time it would be merging image attachments
4. enabling file export of the animated GIFs
5. I'm presuming users would want to email them as well and the way the logic is right now that would be more complicated to implement.
   #5 in particular is an argument for having a more object-oriented approach in the code.

Documentation corrections

@AirCooledNut has some feedback on a few minor corrections to the user_guide based on testing v1.1.0-to-be. I'll fix these.

Compatability of Vizalerts with Tableau server 9.3

Hi Team,

I have a question, is the latest version of VizAlerts (v1.1.1) compatible with Tableau server 9.3.

I downloaded VizAlerts v1.1.1 and trying to configure, I'm seeing error message which refers to config.yml file.

Thanks,
Shoba

Minor fixes for install_guide.docx

Paul Hardman had some suggestions for updates to the docs in this post:

• In the 'Tableau Server Repository Settings' table on the db.query row, you should probably refer to the fact that if you change from the default "_lerts%" your string should all be in lower case - I set up my schedules with a prefix of VizAlerts and had to use "vizalerts%" in order to get it to work.

• In the 'Put VizAlerts through its paces' section, in point 6, the reference should be to the 'Advanced Alerts Test' worksheet rather than 'Advanced Alerts Demo'.

• When setting up a scheduled task, you may need to check "Run with Highest Priviledges", as this is the only way we could get the scheduled task to complete successfully, even when using a service account.

  The last might have been due to permissions issues, so we might want to mention that.

Relax field name requirements for Advanced Alerts

As of today, VizAlerts looks for specific field names in the data exported from a View to tell it what the properties of the email it's trying to send should be. They need to be exact matches to map correctly. The current field names are:

" Email To *"
" Email From ~"
" Email CC ~"
" Email BCC ~"
" Email Subject *"
" Email Header ~"
" Email Body *"
" Email Footer ~"
" Email Action *"
" Email Consolidate ~"
" Email Footer ~"

This strict requirement creates a few problems for usability:

1. The preceding spaces are easy to miss if fields are created manually, and can sometimes get erased by certain tools that auto-trim them. This causes them not to be recognized, and cause unintended behavior.
2. Field names must be unique within a single data connection in Tableau, so if you wanted to build two different alerts with different content in them, but which are based on the same data connection, you can't really do it right now. The workaround is to duplicate the connection entirely, which allows you to customize the field values for the new alert but keep the original logic in the old alert. That's pretty awkward to perform, duplicates work should the data connection logic need to change in the future, and can inflate workbook size.

For those reasons and probably more, VizAlerts should switch from an exact-match requirement to a pattern-match requirement on the field names, while retaining backward-compatibility with the old style. I imagine it would be something like a regex:

"\s{0,1}Email\sTo\s*.*"

...which, translated, is intended to mean "a single space or no space, then "Email", then a single space, then "To", then a space, then an asterisk, then anything else you want to tack on the end".

That should be sufficiently strict to ensure that these special fields never conflict with any normal field names someone would use in their data connection, but also allow enough flexibility that they can create new ones for other alerts using the same datasource.

New logic will need to be added to VizAlerts that will check for "duplicate" fields in the data based on searches in this pattern, and handle the error appropriately if found.

Default config values for timeouts and retries don't always apply as expected

The default values in vizalerts.yaml for as timeouts and retry counts are as follows:

viz.data.timeout:
- [0, 3600, 30]
- [3601, 7200, 60]
- [7201, 14400, 120]
- [14401, 315360000, 300]
viz.data.retrieval_tries:
- [0, 3600, 1]
- [3601, 7200, 2]
- [7201, 14400, 2]
- [14401, 315360000, 3]

If it has been 3600.1 seconds since a view was last checked for data, no rule will apply, since the default range ends at 3600 and starts again at 3601.

The fix will be simply to change the default timeouts in the config file so that we end at 3600 and pick up again in the next rule at 3600 as well. In vizalerts.py, only the first matching rule applies, so overlaps like this introduce no issues.

The issue can be manually fixed the same way for any existing installation of VizAlerts (no need to wait for the later version).

Incorrect error message for log directory cleanup

In vizalerts.py, lines 194 - 198, the incorrect verbiage and config value is being reported to the user--should be log dir, not temp dir.

Trailing delimiters in recipients list cause "missing" error

If you specify a delimited list of email addresses in the To field as follows:

[email protected];[email protected];[email protected];

...VizAlerts will throw a "Missing Email To *" failure email backatcha, because it's looking for the last email address in the list where it doesn't exist. We should be okay with empty addresses between delimiters so long as we at least have one valid address we've derived from the full string. Perhaps tokenize the string, iterate through the recips, then perform a check at the end for having parsed 0 valid emails out of it, returning an error in that case...

Add MMS support to SMS Advanced Alerts

There are two ways to send an SMS message using VizAlerts:

1. Use the email-to-SMS gateways that some mobile providers have, for example [email protected] in the USA. AT&T also has an email-to-MMS gateway at [email protected] where inline images in emails are automatically converted for MMS.

2. Use an Advanced Alert with an SMS action. This enhancement is about adding MMS support to the SMS Advanced Alerts. This is pretty straightforward for the Twilio integration, any VIZ_IMAGE() references in an SMS alert would be attached to the message object.

Troubleshooting PDF rendering - missing headers and columns

This needs to go in the troubleshooting section of the User Guide. When there are text tables built out using all discrete pills on Rows and the view is rendered to PDF the output can end up with one or more of the following problems:

• font becomes smaller
• row headers are missing
• right-hand columns missing
• right-most column is partially cut off

See the all discretes text table problem.png for an example screenshot of a PDF with this problem.

This problem is due to a bug in Tableau Server's rendering that crops up when the PDF is directly downloaded by VizAlerts, it doesn't happen when printing to PDF from Tableau Desktop nor when downloading the PDF manually from Tableau Server.

There are two workarounds:

1. Place the worksheet in a dashboard where the dashboard has the right size for the desired page layout. This can create a lot more work.
2. In the VIZ_PDF() content reference add the :size=width,height URL parameter where [width],[height] are appropriate for the desired page layout, for example :size=1100,850 for a landscape 8 1/2"x11" page.

See the all discretes text table fixed.png for an example screenshot of a corrected PDF.

Updates for 10.0 support

There shouldn't be much that needs to change for this, so this is really just about doc updates and minor tweaks.

• The most notable change is actually positive: Subscribe Others will work for Simple Alerts just as you'd expect. So if you build a user-customized Simple Alert, you can opt others in without needing them to subscribe themselves to the right schedule, or go to the trouble of building an Advanced Alert. And user filters will work! So we mainly need to add that to the documentation.
• Right now we require the admin to specify the Tableau Server version in the config file: 8 or 9. We need 10 in there, not because anything changed, but because people will enter it and expect it to work. 10 should be equivalent to 9, nothing is changing. Ideally we'd remove this value and simply derive it via an API call. Probably worth doing and dropping v8 support anyway, but perhaps in v1.2.0 or somesuch.

Allow Advanced Alert author to specify whether emails should be atomic

Advanced Alerts can send multiple emails at once to dynamic sets of individuals / distribution lists. However, data or workbook issues can cause valid data to be present in one email, but missing from another (say, the TO: field is left NULL on one email, and is populated properly on another). Currently, if any fields in any of the emails set to be generated during a single trigger of the alert are invalid, the entire set of emails will not be sent.

Instead, it would be better to allow the author to specify whether emails should be considered atomic for a given alert trigger--either partially successful results could still be processed (e.g., 4 out of 5 emails), or if one fails, they all do. This could be done via another Measure field being added to the data. The default should be that the email set be atomic.

In the event that one or more emails out of the set DO fail, and the emails are not being considered atomic, the author should still be alerted via email.

Allow embedding of customized View content in Advanced Alerts

Currently, the only way to receive an alert from a view that dynamically adjusts its content to your own user account on Tableau Server is via a Simple Alert--you must subscribe to it yourself, and everyone else who wants the same thing must also do the same.

Advanced Alerts can push HTML emails with embedded View images in them to any email address(es) desired, but they cannot dynamically render arbitrary View information into them and customize it for the audience it pushes the email to.

This feature would increase the flexibility of an Advanced Alert by allowing the author to instruct VizAlerts to embed not just the Advanced Alert view itself in them, but any view whose data the author owns (or if possible, whose data / image the author is entitled to export). The implementation would use the VIZ_IMAGE() placeholder text to do this, but allow passing in a path to the view in question, e.g.:

VIZ_IMAGE('myworkbook/myview')

This feature should also support customizing the view via Filter values passed in via URL parameters, e.g.:

VIZ_IMAGE('myworkbook/myview?team=avengers')

Ideally, this would be built in such a way that we allowed it to be used for other content types as well (PDF, CSV), but those might need to be a new feature request.

Twilio support - multiple outbound numbers

This suggestion is to move the Twilio smsaction.account_id and smsaction.auth_token options from the vizalerts.yaml to the VizAlerts schedule viz where they could be parameters/calculated fields. The reason why is that Twilio enables pretty fine-grained controls over the phone numbers and each outbound phone number can have a different auth_token.

Jonathan

Add gitignore file

With soon-to-be v1.1.0 inadvertently including .pyc files, which is not generally desirable, and others joining in contributing, need to add a gitignore file to filter undesirable filetypes out of our repository automatically. There's probably some pre-baked ones we can pillage for this.

Tableau Cloud Support

There are at least three challenges for Tableau Online support:

1. VizAlerts directly queries the PosgreSQL database and there aren't equivalent REST API functions. This would require a new scheduling mechanism for the trigger views to be built.

2)Tableau Online doesn't support trusted authentication. This could be worked around with the REST API.

3. The REST API does not support downloading images/PDFs/etc., it only supports thumbnails. Theoretically there's a workaround for this where the authentication token is used, but that hasn't been tested on Tableau Online.

Alerts processed for unlicensed users

VizAlerts will still attempt to process an alert based on an Unlicensed user who is still subscribed to a view on an Alerts schedule. This causes a failure email to be sent, because Tableau Server will refuse to provide the CSV data for a view on behalf of an unlicensed user.

My thought is that the correct behavior should be the same as standard Subscriptions functionality, which is that any Subscriptions for an unlicensed user are simply ignored. We could just exclude that data in the SQL query in the config file, but it'd be better to set a flag for those rows then have VizAlerts skip them but log that it did so. That way there's more visibility for the Admin on what's going on.

No reason provided for most trusted ticket errors

The code is not currently checking the right property for URLErrors thrown by get_trusted_ticket in tabhttp.py, line 77. It should be pulling in the reason, not the code. Code is for HTTPError. As a result, issues with trusted tickets are harder to diagnose, as we get no lower level of detail on what went wrong.

Document Linux/MacOS setup

Document what's required for VizAlerts to run on Linux & MacOS, which is really about file paths.

Initial test of an alert applies a 30 second timeout rather than configured timeout

When a user subscribes to a View on an Alert schedule, the first time VizAlerts tests it on the schedule, the timeout value used is 30 seconds. This can cause problems for longer running views, and is especially unhelpful when the alert is to be run daily or weekly, as these can be alotted longer timeouts than normal, and the subscriber will need to go longer with no good information from their initial alert.

The fix should probably be to derive a fake "last ran at" time from the schedules table in the PostgreSQL repository, which would provide a more-valid value to apply the timeout rules by. For example, for an alert that runs daily, but has never run before, we can provide now() minus 24 hours as the last_ran_at time, which would cause it to use a rule matching 86400 seconds, instead of 1800 (30 minutes). That'd allow it a 5 minute timeout under the default config, instead of 30 seconds.