apache/flagon-useralejs

Apache Flagon UserALE is a comprehensive, thin-client behavioral logging tool

flagon-useralejs's Issues

Add xmlHttpRequest logging example to examples

worked up something in my dev environment. Need to add it into /example README until can refactor our example page.

Address new package vulnerabilities

Update devdependencies in package.js to remove package vulnerabilities in build pipeline

How to deploy in a a load balancing environment?

We have a proxy server that redirects user requests to different servers for load balancing. On the proxy side we have our UI application. Some analytics do need to track information about the host. That being said what is the advice wrt:

1. Capturing every user interaction with the application would eventually end up creating large volume of data relatively quickly. Is there a way to purge data that is “x” days old or some retention/deletion policy of some sort?
2. Is there a need to have a cluster of ELK or is one sufficient?

Consider Cypress tests to flex wide range of .options API behavior

Recent bug reports show, for example, behavior where reducing the latency between log shipments, and trimming the package size results in dropping logs that should be sent before a page unloads. Currently, unit tests and integration tests dont flex the range of options like this (string options are easier to test for --logging endpoint, userId, sessionID, etc.). Would be good test test for these edge-cases.

Finalize Doap and other artifacts in preparation for 2.1.1 RC-1

Update versions, dates, etc., etc.

Update Rollup to v2.0.0

Requires updating rollup-plugin(s)

Update detect-browser method for Browser Meta Data in Client Logs

current method is 'detect-browser'. Adds run-time dependencies (overhead).

explore more modern fingerprinting methods as alternative:

example of such a library:
fingerprintjs

Update WebExtension packageBrowserLog funx to resolve missing data values in logs

looks like some fields are set to null that don't need to be (e.g., toolversion, toolname, useraleversion). Should pull in info from globals.

Update Changelog for UserALE.js 2.2.0 Release

Update changelog.md

Add partial Flagon logo to UserALE.js browser extension icon

for branding

Page Load Stats

Received a user request for page load stats printed on logs. Will investigate.

Next.js Users having difficulty with UserALE.js

Next.js doesn't have a concept of 'window'. If embedded as a script-tag, UserALE.js attaches as an object to the window object. If embedded as an npm module, it will throw errors given that some of the listeners and properties UserALE.js references depend on window.

There are a few simple workarounds to this issue: https://frontend-digest.com/why-is-window-not-defined-in-nextjs-44daf7b4604e

Some users have difficult with solution three at the above link.

Would be good to find the best approach and include in documentation

Explore OpenTelemetry User-Interaction Libraries

OpenTelemetry has a plugin that instruments the client for Click data, and attaches a traceID to logs. Should explore the project and see if we can't bring similar functionality into UserALE.js. OpenTelementry is ALv2.

sessionID not written properly

Fresh loaded UserALE creates sessionIDs that are malformed. Until the page is refreshed these sessionID looks like htis in JSON:

"sessionID":""session_1613853205259""

When the page is reloaded:

"sessionID":"session_1613853205259"

At page load, getInitialSettings checks for a value in 'data-session' (userale.options). If it doesn't find it will do one of two things:

1. it will check sessionStorage for a userAleSessionId key and if it finds one it will pass that to getInitialSettings
2. if it doesn't find that key it will create a new one and set the key and return the key that it sets.

Looks like the returned value for the case where no key exists is JSON.stringify(value). When a key exists, the returned value is JSON.parse(window.sessionStorage.getItem(sessionKey)). Either fresh Session ID's need to not be stringified or need to be parsed after strigified.

Fix Typos in README

Still some dangling 2.1.0.s...

Update nodaemon example server to accept JSON from navigator.sendbeacon

On https://github.com/apache/incubator-flagon-useralejs/tree/flagon-userale-50

sendOnClose is now executed through navigator.sendbeacon (only) triggered on visibility state change to 'hidden'. This appears to reliably break the example test server

TypeError: req.body.forEach is not a function
    at /Users/jpoore/Documents/Apache_Flagon/test/incubator-flagon-useralejs/example/server.js:77:12
    at Layer.handle [as handle_request] (/Users/jpoore/Documents/Apache_Flagon/test/incubator-flagon-useralejs/node_modules/express/lib/router/layer.js:95:5)
    at next (/Users/jpoore/Documents/Apache_Flagon/test/incubator-flagon-useralejs/node_modules/express/lib/router/route.js:137:13)
    at Route.dispatch (/Users/jpoore/Documents/Apache_Flagon/test/incubator-flagon-useralejs/node_modules/express/lib/router/route.js:112:3)
    at Layer.handle [as handle_request] (/Users/jpoore/Documents/Apache_Flagon/test/incubator-flagon-useralejs/node_modules/express/lib/router/layer.js:95:5)
    at /Users/jpoore/Documents/Apache_Flagon/test/incubator-flagon-useralejs/node_modules/express/lib/router/index.js:281:22
    at Function.process_params (/Users/jpoore/Documents/Apache_Flagon/test/incubator-flagon-useralejs/node_modules/express/lib/router/index.js:335:12)
    at next (/Users/jpoore/Documents/Apache_Flagon/test/incubator-flagon-useralejs/node_modules/express/lib/router/index.js:275:10)
    at jsonParser (/Users/jpoore/Documents/Apache_Flagon/test/incubator-flagon-useralejs/node_modules/body-parser/lib/types/json.js:119:7)
    at Layer.handle [as handle_request] (/Users/jpoore/Documents/Apache_Flagon/test/incubator-flagon-useralejs/node_modules/express/lib/router/layer.js:95:5)

see #50 for more detail

Update npm packages to eliminate vulnerabilities

rollup
nodemon
gulp-eslint
babel-register

React.js Test Page Included in UserALE.js testing framework

Would be good to figure out how to incorporate testing against react frameworks in unit tests. might require development of a react.js test page.

Add configurable filter for different log 'types' in browser extension options

Currently the browser extension has limited configurability through /UserAleWebExtension/options.js--userAleHost, userAleScript, toolUser, toolName, toolVersion. As such, the embedded userale-X.x.x.js script logs everything. It would be good to embed a filter in ./background.js that receives user inputs through opstionsPage.js and options.js. A nice example example of a filter can be found in our /example dir:

window.userale.filter(function (log) { var type_array = ['mouseup', 'mouseover', 'mousedown', 'keydown', 'dblclick', 'blur', 'focus', 'input', 'wheel']; var logType_array = ['interval']; return !type_array.includes(log.type) && !logType_array.includes(log.logType); });
Would be nice to include a form object that would allow users to add (through drop down or simple text input) event 'types' (e.g., mousedown, mouseup, mouseover) and log 'types' (e.g., 'raw', 'interval') (aside: maybe toggle for 'interval') to exclude from the browser extension log stream.

Convert library to typescript

Are you open to the idea of converting the library to typescript? If so I was thinking of giving it a go.

Here's some information on how one might convert a library to type script and why one might want to do it.
https://www.typescriptlang.org/docs/handbook/migrating-from-javascript.html

update Gulp dependencies (Chokiadar, fsevents) when available

Gulp has not yet released to address dependency conflcts with node v13

Gulp sub-dependency 'minimist' has prototype pollution vulnerability

Update Package.json file to include entire Major version for Node 13 (compatability)

NPM throws a polite error message about node compatability. Yarn just wont do it. Being too conservative with compatability in package.json is costing us yarn users.

Index.js is not found when running the example page

Currently, when running npm run example:run on a fresh build and navigating tolocalhost:8000, the browser console throws the following error:

GET http://localhost:8000/index.js net::ERR_ABORTED 404 (Not Found)

This is because in the Express app we are not using the directory that index.js is located in. This results in the index.js not being loaded and none our custom Userale settings from that file are being used

Update Readme to be consistent with Incubator NPM Release Policies

https://incubator.apache.org/guides/distribution.html#npm

our release process is compliant, save the "full disclaimer" at the bottom of ReadME

UncleGedd to add Apache Keys to Keys file

UncleGedd to add Apache Keys to Keys file

Update rollup-plugin-commonjs to rollup/plugin-commonjs

Rollup refactored its plugin constellation. Need to move over this dev-dependency to new rollup/plugin-commonjs. rollup-plugin-commonjs is deprecated (not presently worried about it).

Browser Extension builds Intervals Differently was: Poorly Written Intervals logs

We are seeing multiple interval logs written with same start, but different end-times and counts.

{
"@timestamp": "2018-03-23T20:31:39.542Z",
"count": 2,
"duration": 3818557,
"endTime": 1521837223993,
"host": "172.18.0.1",
"logType": "interval",
"major_ver": "1",
"minor_ver": "0",
"pageUrl": "https://confluence.draper.com/display/TF/Jedi+Slide+Decks+and+Write+Ups",
"patch_ver": "0",
"path": [
"div#main. aui-page-panel",
"div.ia-splitter",
"div#full-height-container",
"div#page",
"body#com-atlassian-confluence.theme-default aui-layout aui-theme-default synchrony-active",
"shadow",
"#document-fragment",
"html",
"#document",
"Window"
],
"sessionID": "session_1521544103912",
"startTime": 1521833405436,
"target": "div#main. aui-page-panel",
"targetChange": true,
"toolName": "test_app",
"toolVersion": null,
"type": "click",
"typeChange": true,
"userAction": false,
"userId": "bqw0205",
"useraleVersion": "1.0.0"
}

{
"@timestamp": "2018-03-23T20:31:39.542Z",
"count": 4,
"duration": 3819958,
"endTime": 1521837225394,
"host": "172.18.0.1",
"logType": "interval",
"major_ver": "1",
"minor_ver": "0",
"pageUrl": "https://confluence.draper.com/display/TF/Jedi+Slide+Decks+and+Write+Ups",
"patch_ver": "0",
"path": [
"div#main. aui-page-panel",
"div.ia-splitter",
"div#full-height-container",
"div#page",
"body#com-atlassian-confluence.theme-default aui-layout aui-theme-default synchrony-active",
"shadow",
"#document-fragment",
"html",
"#document",
"Window"
],
"sessionID": "session_1521544103912",
"startTime": 1521833405436,
"target": "div#main. aui-page-panel",
"targetChange": true,
"toolName": "test_app",
"toolVersion": null,
"type": "click",
"typeChange": true,
"userAction": false,
"userId": "bqw0205",
"useraleVersion": "1.0.0"
}

sendOnClose function broken -- logs dropped when users navigate away from page

Navigating from a page to another should trigger "sendOnClose" which flushes the log queue when a page is closed.

user report that at transmittalInterval >50ms and logcountthreshold=1, the last log (e.g., 'click') isn't being transmitted.

Add new initial setting for customizing a log package to be used by packageLog

Might be nice if instead of requiring users to use our native log schema, users to specify their own in options API and use it with automated logging and custom logs.

Imagine new field in options API. If is not null, then packageLog function uses some other object instead. May require exporting more functions for clientTime, etc., etc. to make it convenient for users to populate fields.

Add Contribution Guide

Should notify future contributors to submit PRs to Test branch.

Update Package.json for bumps in Package-Lock.json

Update Package.json for bumps in Package-Lock.json

add LogType = Raw and Type = load to PageLoad Custom Log

minor mods to make this easier to index and search in back end solution (e.g., ELK)

Distill related question

Is there any updated version of Distill? Documentation says it is not maintained. WHat is the alternative?

Decorator Syntax Examples

Users are interested in applying custom labels to their logs.

There is an easy way to do this through the .log .map and .packageCustomLog exports.

Some good examples in https://github.com/apache/incubator-flagon-useralejs/blob/master/example/index.js

We should probably start with a new .md to capture, contextualize, and decompose some of these examples.

Here's one:

document.addEventListener('click', function(e){ if (e.target.innerHTML === 'Click Me!') { **window.userale.map(function (log) { return Object.assign({}, log, { logType: 'custom', customLabel: 'map & packageLog Example' }); });** window.userale.packageLog(e, window.userale.details(window.userale.options(),e.type)); /**you'll want to reset the map callback function, or set a conditional (e.g., return log), else * the callback may be applied to other events of the same class (e.g., click) */ window.userale.map(); } else { return false } });

Here' another
`document.addEventListener('change', function(e) { if (e.target.value === 'packageCustomLog') { window.userale.packageCustomLog({ customLabel: 'packageCustomLog Example', customField1: 'foo', customField2: 'bar'}, function(){return {'foo': 'bar', 'bar': 'foo'}}, true ); } else { return false } });

Refactor Gulp Build Pipeline

Gulp is reaching obsolescence. Might consider webpack or new npm functionality as an alternative.

Opinion: webpack may be a better option given that relying on newer npm functionality will affect the projects compatability with earlier versions of npm (12x-14x), which many of our users still rely on.

Update Dependencies

Update Dev Dependencies where Possible

NPM deployment automatically starts regardless of usage

The flagon-userale NPM package has a handful of issues. The most egregious (in my eyes) is that, even if I don't explicitly start it, it will begin collecting/sending logs as soon as it is imported. Maybe there is some argument that this is intended behavior, but having it do this is probably a surprise for developers. For example, this React project results in log collection/sending:

// in index.js
import React from 'react';
import ReactDOM from 'react-dom';
import App from './App';

ReactDOM.render(
  <React.StrictMode>
    <App />
  </React.StrictMode>,
  document.getElementById('root')
);

// in App.jsx
import { packageCustomLog } from "flagon-userale";
import { useCallback, useEffect, useState } from "react";

function App() {
  const [secondsElapsed, setElapsed] = useState(0);
  const resetTimer = useCallback(() => {
    setElapsed(0);
    packageCustomLog({ elapsed: secondsElapsed, logType: "timerReset" }, undefined, false);
  }, [secondsElapsed, setElapsed]);

  useEffect(() => {
    const timerId = setTimeout(() => {
      setElapsed(secondsElapsed + 1);
      packageCustomLog({ elapsed: secondsElapsed, logType: "timerUpdate" }, undefined, false);
    }, 1000);

    return () => {
      clearTimeout(timerId);
    };
  }, [secondsElapsed, setElapsed]);

  return (
    <main className="App">
      <p>App has been running for {secondsElapsed} seconds.</p>
      <button onClick={resetTimer}>Reset</button>
    </main>
  );
}

export default App;

This surprises me, since it means that importing any part of userale will automatically start the logging process. I'd imagine we ought to figure out a way to have it not start by default when imported via NPM. Of course, this is a breaking change.

Update JSDOM for Unit Testing/or consider Jekyll

Unit testing framework is a little dated, though wholly sufficient. Should at minimum upgrade to JSDOM 10, or consider refactoring in Jekyll.

Update Webpack Example

Webpack and dependencies in demonstration are likely out of date.

Update README for 2.2.0

Update Readme text for 2.2.0 release. Update changelog entires. Add links to new examples.

Push TraceID into UserALE and Zipkin data model

Looks like there may be a way to force a traceID that can be passed into the Request context. This could allow us to pace the same trace id into a 'submit' or 'click' log and a zipkin trace using a custom log.

openzipkin/zipkin-js#128

Use UserALE.js example page as template for Cypress journey tests?

Still coming up to speed with Cypress, but if we're looking to mock user interactions with userale and flex it from an integration standpoint the example page (index.html) in ./example might be a good starting point. Index.js has all the features you'd want to flex userale's entire API (.options, .filter, .map, .log, .packageLog, .packageCustomLog). Worth investigating.

Moderate Severity Vulnerability in Dependency Tree

one of our packages was recently flagged by npm. Should update, if dependabot doesn't get to it first.

ws 5.0.0 - 7.4.5
Severity: moderate
Regular Expression Denial of Service - https://npmjs.com/advisories/1748
fix available via npm audit fix
node_modules/ws

Journey tests for sendOnClose mods

On https://github.com/apache/incubator-flagon-useralejs/tree/flagon-userale-50

Unit tests for sendOnClose report as passing, however, sendOnClose breaks on live web (triggering off of unload events is no longer recommended). #50.

This is a great use case for cypress journey tests.

Likely best to merge flagon-userale-50 with test branch after cypress tests are merged.

Capture what tab# page is rendered in

We get this via the browser plugin, however, that is a property of browser, which the page doesn't have access to. Will look into it. Likely requires a server-side solution

Add Uniform Quick Start Guide on Top of All READMEs

Each readme should have a quickstart guide that includes setting up client and logging endpoint for UserALE.js integration.

Log Call Back function from userale demo, might be interesting in mainline UserALE.js

**Log Call Back function from userale demo on website, might be interesting in mainline UserALE.js

users might want to do additional logging or client side work, leveraging callback logs – last 5 logs.

Add JSDOM back into Dependabot Automation Workflow

Add JSDOM back into Dependabot Automation Workflow

Explore Local Storage SessionID Solution for Multi-Tab Applications

Currently, userale will assign a new sessionID unless it detects an existing sessionID stored in 'session storage". New pages (tabs) spawned from the current page is in the same browsing context, and will have access to sessionStorage. However, sessionStorage doesn't persist across different browsing contexts. Meaning if user opened new tab, then navigated to same application or page a new sessionID would be assigned. These two sessions should be the same (unless there is some edge case I'm not thinking about).

Option 1: pass sessionID into sessionStorage, detect new browser, push sessionStorage to localStorage, emit payload to grab (e.g., sessionID: $value), grab payload.

Option 2: Broadcast API (safari, IE support)?

Good article here, but dated: https://blog.guya.net/2015/06/12/sharing-sessionstorage-between-tabs-for-secure-multi-tab-authentication/#:~:text=Right%2C%20sessionStorage%20is%20not%20shared%20across%20tabs.&text=The%20way%20I%20solved%20it,have%20the%20sessionStorage%20for%20us.&text=Click%20to%20%E2%80%9CSet%20the%20sessionStorage,see%20the%20sessionStorage%20is%20shared.

Fix CI Builds

Our Jenkins CI build @ https://builds.apache.org/job/useralejs-ci has... vanished.

Explore new Jenkins option or Explore Git Action for NPM package (seems easy)

https://docs.github.com/en/actions/guides/building-and-testing-nodejs