psteiwer/analyzethis

Easily transform a CSV file/Table/SQL query into a personalized preview of DeepSee/InterSystems IRIS BI

AnalyzeThis

After selecting a Data Source (CSV file, Caché Class, or SQL Query), your data is easily and quickly transformed into a personalized preview of DeepSee

Accessing

Once installation is complete, a new "Analyze This" Link will be added to the User Portal. The User Portal can be found from the Management Portal -> Analytics -> User Portal

Usage

Provide a Data Source (CSV file, Caché Class, or SQL Query) and assign dimensions/measures as necessary. Once ready, click the import button. At this point, a new Caché class is created based on your Data Source and the data is imported. This new class is used as the source for the DeepSee cube. A DeepSee cube is also generated based on the properties in the source class. Once the cube is created and the records are built, a sample dashboard can be viewed. A scoring system is used to evaluate and rank the possible charts and tables that are displayed on this dashboard. After getting a visual idea of what DeepSee can offer, users can begin exploring the model within Architect or the data within Analyzer.

Installation

1. Use the Download ZIP option for this project
2. Extract the files and copy path
   • This is the path to the directory that contains README.md and LICENSE
3. Open terminal and ZN to desired namespace
4. Run the following commands:

   Set path="PATH FROM STEP 2"
   Do $system.OBJ.Load(path_"/AnalyzeThis/Installer.cls","ck")
   Do ##class(AnalyzeThis.Installer).RunInstaller(path)

5. From the Management Portal, navigate to DeepSee/Analytics->User Portal
   • Find new "AnalyzeThis" Link with custom Cover Image

analyzethis's Issues

Pasting in file path does not enable next button

If a user does not use the Browse button and simply pastes a file path into the text box, the Next button is not enabled

Read final row even if it does not have line feed

Currently, the last line of a file is not read if there is no final line feed

UnitTests.TestRegexLine.TestNoFinalLineFeed Fails

Create an API to generate a sample dashboard and a scorecard from an existing DeepSee Cube

Hello,
It would be nice to have an API to generate sample dashboard and scorecard from an existing DeepSee Cube.
If one already exists, would you be able to document it?

Thanks in advance!
Asaf

Remove horizontal scroll bar in CSVImport dialog

Width is slightly off due to some padding. This is causing a horizontal scroll bar

Accept $h time formats

$h is not a date format option on the property check page, this should be made available

Switching Source Type throws error

Error invoking instance method: SwitchSourceType:
zSwitchSourceType+13^AnalyzeThis.UI.Dialog.CSVImport.1 *sourceClassFile
Set %ret = %obj.SwitchSourceType()
%obj='[email protected]'

Rows are silently skipped if they contain non-numeric data for an integer measure

If a row in a source file contains non-numeric data for a field that is being used as an integer measure, that row is not included in the generated source table (and therefore isn't included in the cube either). The Import Data dialog gives no indication that any errors occurred.

The following source data from https://www.baseball-reference.com/teams/BOS/ reproduces this issue when the "GB" (games back) column is included as an integer measure - 19 rows are skipped because they contain "--" to represent 0 in that field:
redsox_seasons.csv

Numeric values with , separator get errors

Numeric values formatted with , are causing issues when trying to import them as measures.

Minor enhancement request to the AnalyzeThis installation process

It would be nice to be able to install AnalyzeThis using the following structure:

Step 0: Download the zip file and extract it to
Step 1: s path="\AnalyzeThis-master"
Step 2: do $system.OBJ.Load(path_"/AnalyzeThis/Installer.cls","ck")
Step 3: do ##class(AnalyzeThis.Installer).RunInstaller(path)

This way, you can copy and paste steps 2 and 3.
In addition to the Readme file change, the RunInstaller function will need to add a path parameter.

Add Cover Image for User Portal Link

As title says, make user portal link pretty

Using a source class that contains an SQL reserved word causes an error

I removed [ SqlTableName = SalesTransaction ] from HoleFoods.Transaction and tried to select it as my source class. I then received this error.

ERROR #5001: SQLToJSON Error. SQLCODE: Invalid SQL statement

SOURCE ELEMENT: %ZEN.Component.html (htmlContent)

Import on IRIS 2019.3 CE doesn't work

Imported the file which goes with repo - inpatient.csv
Import never ends.
IRIS for UNIX (Ubuntu Server LTS for x86-64 Containers) 2019.3 (Build 309U) Wed Sep 11 2019 16:20:03 EDT

Remove temporary streams when finished

Currently all temporary streams are saved and never removed

Can not compile when IRISLIB mounted read only

Compile errors on initial load when IRISLIB is mounted as read only

Line breaks in cells do not work

If a cell has a line break in it, the Regex reading in the CSV file does not work properly and splits this cell into two CSV lines instead of ignoring the line break and treating it as data

UnitTests.TestRegexLine.TestCRLFadv fails

No link after install

Installed via zpm on a mac, IRIS 2019.34, but cannot see any link in the analytics user portal. Install reported no errors and AnalyzeThis_ThirdParty.js is in the csp/user/ directory.

I'm not really familiar with Zen so not sure how it loads js files, but inspecting the loading in developer tools in Chrome it looks like that is just not happening

Update footer buttons

The UI is now separate from the generate methods. Previously the generate methods were with the UI and they updated the buttons depending on what was generated. In particular, the show scorecard button no longer appears even when a scorecard is generated. I also believe that the dashboard button does not work. The footer buttons need to be evaluated and possibly refactored

Dates can be imported as 19XX instead of 20XX

When a date uses MM/DD/YY format, the system assumes it is 19YY instead of 20YY. Perhaps there should be two options to replace the current MM/DD/[YY]YY with:
MM/DD/19YY or
MM/DD/20YY
using 20YY as the default.

Import fails if property name is "Type"

This is a property used to store metadata about the file being imported, this overwrites the data node called "Type" if defined

Uniqueness checking not correct

I am getting this error while trying to test:

First 16 characters of CubeName must be unique from existing CubeNames. Inpatients is not unique from Inpatients2

Handle null column values

Trying to import a "bad csv file" with null headers causes an error. This is the most common "bad csv file" issue I have seen.

If an empty column header is detected, we should ignore that column. If there is a row with data in that column, we should ignore the row. This generally happens because a row is formatted incorrectly and a single cell's data is split across more than 1 cell. This shifts the rest of the cells over and will create a new column if there is data in the last column

Add timestamp support

Currently, %DeepSee.WizardUtils does not handle "TIMESTAMP" properties. Once this is implemented, AnalyzeThis can take advantage

Remove mini-analyzer from generated widgets

Mini-analyzer option should be removed from the generated widgets

Running AnalyzeThis.Generator:GenerateDash() 2x gives error

Since dashboard exists, section of code is skipped and undefined error is hit:
zGenerateSampleDashboards+374^AnalyzeThis.Dashboard.AutoPivot.1 *scores(13)

Local file does not use Cube Name

Temporary file name of "importfromlocal" is used instead of the Cube Name property

Make the prediction on cube measures for integer values

Coul you autodetect for the measures upon the values? e.g. it's a measure if first 10 values are integers
Would be great

Column with SQLReserved word causes listing error

When trying to view a listing, I am seeing this error:
ERROR #5540: SQLCODE: -1 Message: ( expected, , found^ SELECT TOP ? ID , DATE , (4)

The following SQL was generated:
SELECT TOP 1000 ID, date, location, newcases, newdeaths, totalcases, totaldeaths FROM AnalyzeThis_Generated.coronavirus source WHERE source.%ID IN (SELECT _DSsourceId FROM AnalyzeThis_Generated_coronavirusCube.Listing WHERE _DSqueryKey = 'en1665844046')

This is because date is a reserved word and should be escaped

Allow user defined line terminators

Currently Carriage Return+Line Feed is the only accepted line terminator
Change this to default and allow users to define their own

add the option to use URL based CSV

to let me put the URL for CSV in the cloud: on Github, in AWS S3, Google Bucket, etc

Store line limit for refresh

Currently line limit is not stored so refreshing the data from the UI will pull in all data from the data source

Included option not working

This is due to recent changes, probably need to check for the checked property instead of value

Use file size limitation

Allow restriction of file size based on input value

Clean up Score calculations in GenerateSampleDashboards

AnalyzeThis.Dashboard.AutoPivot:GenerateSampleDashboards() contains code that logs the "scores" of different factors. If these scores are then within a certain range, a pre-defined widget type is used and it dynamically adds the factors with correct scores.

At this time, adding new chart types and modifying factors is not a simple process. This process should be broken down into easier to manage stages.

If Regex matcher text ends before closing quotes, no match found

UnitTests.TestRegexLine.TestNoEndQuote Fails

dev - Import button always disabled

It appears that the status is always empty in AnalyzeThis.UI.CSVImport:drawDone(), this is causing the Import button to always be disabled

Automate date format

Automatically detect the date format when a user selects the date type

Replace Cube Info With Metadata

After the import and generation is completed, there is an option to view some cube info. This includes some null statistics. This should be replaced with the full metadata table information

é imported as é

Imported a value of Saint Barthélemy, it was imported as Saint Barthélemy. This is how the string appears in the data global