dataanalyticsinstudenthands/hnetdatamaps

A data processing framework for a group of atmospheric science researchers at the University of Houston

HNETDataMaps Client

This is the web frontend for a data processing framework for atmospheric sensors who can monitor and edit their data.

The documentation for end users can be found in the Wiki.

The following instructions are for developers who want to setup their own version. This is a meteor based framework.

Prerequisites

• Data Folder: The framework expects a certain data folder structure. /hnet/incoming/current should exist. This location is hard coded and need be changed if desired. There also needs to be an outgoing data directory which can be defined as an environment varaible outgoingDir.

• We are running with a local MongoDB

to install e.g on Mac OS install via homebrew:

brew install mongodb

• meteor

to install e.g. on Mac OS or Linux:

curl https://install.meteor.com/ | sh

• For pushing data out, the server is using lftp which must be installed.

Getting started

Clone and run: meteor npm install

Testing in local development environment

MONGO_URL=mongodb://localhost:27017/DataMaps meteor

For debugging with node inspector run MONGO_URL=mongodb://localhost:27017/DataMaps meteor debug and open the app in Chrome with the port listed once the app has started.

Deployment with PM2

• change into the working directory and run meteor build .. - this will generate a *.tar .gz file
• move the file to the install location and extract it (you will end up with a bundle directory
• cd bundle/programs/server/ and npm install
• generate a configuration file for PM2 (see example gist) outside of bundle
• run pm2 start [your_pm2_conf_file] --node-args="--max_old_space_size=6144"

hnetdatamaps's Issues

Flags for aggregated data points with less than 75% data is not set to "offline"- N during aggregation

From @peggylind on June 7, 2017 17:45

Copied from original issue: DataAnalyticsinStudentHands/DataMaps#71

Missing data should be flagged N by default.

From @peggylind on February 28, 2017 16:41

Copied from original issue: DataAnalyticsinStudentHands/DataMaps#64

Create 1 week, 2 week views for aggregated data

From @peggylind on February 8, 2017 4:16

• add another selector for dates with buttons for 1 week, 2 weeks

Copied from original issue: DataAnalyticsinStudentHands/DataMaps#57

Create placeholder for icons in the footer

From @peggylind on January 6, 2017 20:36

Need a place for an icon image in the footer.

Copied from original issue: DataAnalyticsinStudentHands/DataMaps#51

update site list for UH campus composite plots

Admin interface: channels reappear as empty when deleted

From @peggylind on June 8, 2017 4:34

Copied from original issue: DataAnalyticsinStudentHands/DataMaps#73

use a tail solution to only read the changes in a file instead of having to read the whole incoming file

From @peggylind on February 9, 2017 2:31

Copied from original issue: DataAnalyticsinStudentHands/DataMaps#58

Download of data for BC2 sites doesn't work in Actions

Export Data button stopped working (under Datamanagement -> Actions)

From @peggylind on June 7, 2017 17:36

Copied from original issue: DataAnalyticsinStudentHands/DataMaps#70

Add pulldown menu for notes

Eugenia will send list for pull down

Pulldown should also have other

File View is messing up lines

From @peggylind on December 19, 2016 17:40

Copied from original issue: DataAnalyticsinStudentHands/DataMaps#50

make composite plot for campus sites (MT and LT)

Reimport data should have option for adding edits or not

Currently the reimport function is mainly for files that have not been imported (and therefore no edits should exist).

It is desired to reimport and add exiting edits as well as an option.

Some public sites don't show current data

From @peggylind on June 7, 2017 18:7

Copied from original issue: DataAnalyticsinStudentHands/DataMaps#72

Edit modal doesn't show list of edit points anymore

this happened after separating the EditPoints template to be used also for the BC2 site template

Add TCEQPushing as feature for livesites

Introduce a new feature in the Admin panel which will be called something “TCEQ Pushing”. Similar to “Status” it can only have two values: Active or Inactive.

If Inactive, no automatic pushing will happen. The buttons in the datamanagement interface will not be affected only the automatic pushes.

Channel name switching causes trouble for aggregation & publication

From @peggylind on December 13, 2016 19:1

When channel names are changed to new names in the incoming files, the aggregation & publication pipelines can get stuck.

The user should receive a warning about trying and the aggregation should stop and continue with a new aggregation using the new name.

Copied from original issue: DataAnalyticsinStudentHands/DataMaps#48

Edits are allowed without note

From @peggylind on November 23, 2016 22:35

• need to enforce notes on edits

Copied from original issue: DataAnalyticsinStudentHands/DataMaps#42

add y-axes scaling function for both composite plots

Sites stop pushing

Reported via email

Something really strange is happening . In most of the sites data is being pushed to TCEQ sporadically like what it used to happen before you updated the code. For Jones Forest it started 5/20 ( I think part of this time was due to internet connection the first 2 weeks) , for West liberty 5/30, for Moody Tower 6/7 for Sugar Land 6/6 and for Clear Creek 6/1.

I am going to re push them manually but I wanted to make you aware of. I will take a couple of screen shoots before I do.

HNET website crashed when loading BC2_WL website

https://hnet.uh.edu/bc2site/99996

Can access all other sites, just not BC2_WL. Please make sure it isn't a linux thing by going to the site on a non-linux machine (though I highly doubt it is a linux thing). The recent commit to back end may be the culprit.

It only happens when you load data for BC2. Don't think the site matters, just that there is BC2 data.

shift to midpoint view for 5 min aggregate data

add 150 seconds

Channel IDs are not set in TCEQ formated data files for Launch Trailer

Channel names are matched to IDs with a case sensitive match.
Will add to the constants for a quick fix.

remove accounts sign up functionality

Request by jimmy before site goes public

add new level of color coding for aggregate data

e.g.
valid_NOconc = red
valid_NO2conc = blue

Flags in span for Moody Tower for NOx instrument are not being set to both channels

example data point: in aggregateddata
{site: '482010695', epoch: 1524735300}

NOconc gets a '1'
NO2conc gets a '3' (span)
the instrument flag is set to3

Treat RH data the same in regular data files

We had been done some treatment for faulty processing of RH data when collected in met files. The same treatment should be done when collected in daq factory files.