liflab/labpal

Easily run experiments on a computer

Easily run experiments on a computer

If you are doing research in Computer Science, it is very likely that every now and then, you need to run experiments on your computer. LabPal is a Java library that allows you to quickly setup an environment for running these experiments, collating their results and processing them in various ways: generating tables, data files, plots, etc.

Read more about LabPal in this recent article:

• S. Hallé, R. Khoury, M. Awesso. (2018). Streamlining the Inclusion of Computer Experiments In a Research Paper. IEEE Computer 51(11): 78-89. DOI: 10.1109/MC.2018.2876075

Features

• All your experimental setup (including your code, its input files and library dependencies) can be bundled into a single, runnable JAR file, making it easy for anybody to download and re-run your experiments.
• The runnable JAR acts as a web server; when launched, a user can see all the experiments and control their execution using a web browser.
• For a few more lines of code, the input/output parameters of an experiment can come with a small textual description that is displayed in the web interface, so that a user can understand the meaning of each data element. You can do the same with the lab itself, and each experiment it contains.
• Automated generation of PDF and PNG plots (using GRAL or Gnuplot) and tables (in beautified LaTeX, HTML or CSV). You can perform transformations to the tables before plotting them, and also customize the display of your plots (scales, colors, labels) very easily.
• Each running experiment can update a visual progress bar; LabPal can even estimate and show the time remaining before they complete.
• A set of partially executed experiments can be saved to disk, then reloaded and resumed at a later time (or even on a different machine).

Why use LabPal?

To run experiments on a computer, you probably already write command-line scripts for various tasks: generating your data, saving it into text files, process and display them as plots or tables to include in a paper or a presentation. But soon enough, your handful of "quick and dirty" batch files becomes a bunch of arcane, poorly documented scripts that generate and pass around various kinds of obscure temporary files. This situation brings two important problems in terms of research methodology:

• Problem 1: no one can reproduce your experiments. Too much cleaning up would be required to your setup before anybody else could understand how it works, so chances are you'll never make your scripts and data publicly available.

• Problem 2: you waste your time. Most of your batch and data files are so specific to your experiments that even yourself are unlikely to reuse them on your next project; you'll start from scratch instead. This is not a very productive use of your time.

I want to use LabPal!

• Just download the latest release and include it in your classpath.
• Or download our template project that includes a few extra features (like a boilerplate Readme and Ant build script).

If you want to know more about LabPal's features:

• Read the short tutorial
• Look at our lab examples to learn how to use various features
• Look at our slightly longer instructions
• Consult the online API documentation

About LabPal

LabPal was developed by Sylvain Hallé, Full Processor at Université du Québec à Chicoutimi, Canada. Pr. Hallé is also the head of LIF, a research lab, where LabPal is extensively used for the processing of experimental results.

labpal's Issues

Lab assistant does not remember its cumulative running time

Once its queue runs out, the total running time goes back to 0

Auto-escape replacement patterns in template pages

Instead of manually calling Matcher.quoteReplacement() when replacing placeholders with content in every template page (and potentially forget to do it), create a replacePlaceholder method that would do it automatically.

Zip lab contents and allow upload of zip file

NoSuchMethodException when running from JAR

There's something wrong with the build script. Running from Eclipse is OK, but calling the plots page when running from a compiled jar leads to:

java.lang.NoSuchMethodError: ca.uqac.lif.labpal.table.Table.getDataTable()Lca/uqac/lif/labpal/table/DataTable;
	at ca.uqac.lif.labpal.plot.gnuplot.Scatterplot.toGnuplot(Scatterplot.java:130)
	at ca.uqac.lif.labpal.plot.gnuplot.GnuPlot.toGnuplot(GnuPlot.java:90)
	at ca.uqac.lif.labpal.plot.gnuplot.GnuPlot.getImage(GnuPlot.java:107)
	at ca.uqac.lif.labpal.plot.Plot.getImage(Plot.java:321)

Loading labs does not work

Exception in thread "main" java.lang.NullPointerException
	at pattheminer.MainLab.setup(MainLab.java:124)
	at ca.uqac.lif.labpal.Laboratory.loadFromJson(Laboratory.java:651)
	at ca.uqac.lif.labpal.Laboratory.loadFromString(Laboratory.java:634)
	at ca.uqac.lif.labpal.Laboratory.loadFromZip(Laboratory.java:760)
	at ca.uqac.lif.labpal.Laboratory.loadFromFilename(Laboratory.java:791)
	at ca.uqac.lif.labpal.Laboratory.initialize(Laboratory.java:943)
	at ca.uqac.lif.labpal.Laboratory.initialize(Laboratory.java:891)
	at pattheminer.MainLab.main(MainLab.java:525)

Hyperlinks in LaTeX tables

Containing the datapoint ID of each cell

Bug in PiePlot

Try to show the third plot in BrowserLab. You get this:

java.lang.ClassCastException: ca.uqac.lif.mtnp.table.PrimitiveValue cannot be cast to java.lang.Number
	at de.erichseifert.gral.plots.PiePlot$PiePlotLegend.getEntries(SourceFile:783)
	at de.erichseifert.gral.plots.legends.AbstractLegend.add(SourceFile:275)
	at de.erichseifert.gral.plots.legends.ValueLegend.add(SourceFile:86)
	at de.erichseifert.gral.plots.AbstractPlot.add(SourceFile:581)
	at de.erichseifert.gral.plots.PiePlot.add(SourceFile:917)
	at de.erichseifert.gral.plots.AbstractPlot.add(SourceFile:564)
	at de.erichseifert.gral.plots.AbstractPlot.add(SourceFile:554)
	at de.erichseifert.gral.plots.PiePlot.(SourceFile:854)
	at ca.uqac.lif.mtnp.plot.gral.PieChart.getPlot(PieChart.java:42)
	at ca.uqac.lif.mtnp.plot.gral.GralPlot.getPlot(GralPlot.java:119)
	at ca.uqac.lif.mtnp.plot.gral.GralPlot.getImage(GralPlot.java:87)
	at ca.uqac.lif.mtnp.plot.Plot.getImage(Plot.java:311)
	at ca.uqac.lif.labpal.server.PlotImageCallback.process(PlotImageCallback.java:93)
	at ca.uqac.lif.jerrydog.Server.handle(Server.java:252)
	at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
	at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)
	at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)
	at sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)
	at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
	at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647)
	at sun.net.httpserver.ServerImpl$DefaultExecutor.execute(ServerImpl.java:158)
	at sun.net.httpserver.ServerImpl$Dispatcher.handle(ServerImpl.java:431)
	at sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:396)
	at java.lang.Thread.run(Thread.java:748)

Broken pipe on HTTP request

Should be fixed by addressing sylvainhalle/Jerrydog#2

Plot download button does not work in Lynx

In the text browser Lynx, the "Download all plots" button does not work. It says "Bad HTML! No form action defined". Individual plot buttons work, through.

Auto-merge experiments from multiple lab instances

"Start the assistant" in info div does not work

When addding experiments to an assistant, the page reloads with the message "x experiments added; start the assistant". The "start" link in the message does lead to the assistant page, but the assistant does not start.

This is caused by the fact that the assistant page looks for the "start" parameter only in a POST HTTP request. That link loads the page with the correct parameter, but in a GET request.

Cannot remove experiments from assistant

The Remove button does not seem to work

Inconsistency in FileHelper API

The methods to read a file accept a path relative to the location of the class given in parameter. The method to list the contents of a directory disregards the location of the class and expects an absolute path from the root of the project.

Stopping the assistant does not stop the running experiment

...it only stops the assistant from starting new experiments. The assistant should also kill the current thread.

Progress bar: special color for running experiments

In the progress bar of the status page, an experiment that is running is categorized in "Other". There should be a special category (and color) for the running experiments.

CLI stops if environment conditions not met

It should display the warning, but still start

Create subgroups

...i.e. groups within a group.

Check if each experiment has an empty constructor

If it doesn't, the Azrael serialization library cannot deserialize a lab and will throw an error message like "method MyExperiment() does not exist".

When one adds an experiment to a lab, the lab should check if the class of that experiment has an empty constructor. If not, a warning should be displayed in the Status page, saying something like "Loading this lab from a file won't work".

Responsive GUI

Running LabPal on a phone is a bit hard. There should be a responsive GUI, or at least (using media queries) a different template for small screens.

Beautify file upload button

https://tympanus.net/codrops/2015/09/15/styling-customizing-file-inputs-smart-way/

CommandRunner: give it an InputStream

There should be a method in CommandRunner where the contents sent to the command's standard input come from an InputStream instead of a String.

Suggestion:

• Create such a method
• Change the current method runAndGet into a delegate to the new method, simply wrapping the String into a StringInputStream

Send experiment results from a lab to another lab instance via HTTP

Button Clean in GUI does nothing

Solution is obvious ;-)

Export lab UI to static HTML bundle

Change LDI numbering scheme

Currently, the LDI is a sequential number. This means that the same experiment may be assigned a different LDI in a lab, depending on what other experiments there are and the order in which they were added. (The same applies for a plot or a table.)

For an experiment, the LDI should rather be based on its type and the values of its input parameters. Maybe some form of hash?

What should a proper LDI be for a table or a plot?

Add box for selecting a batch of experiments by number

In the experiments page

Implement multi-thread lab assistant

Fix the select-all checkbox

Currently, it assumes that the experiments page has experiments with IDs 1 to the length of the list in a single contiguous interval. If you use a filter, this is no longer the case.

Plots: don't show key when there is only one data series

The title says it all

CommandRunner should catch standard error

And there should be a method for retrieving what the command sent there

Experiment warnings are not saved

If field m_warnings in class Experimentcontains elements (of type Exception), the Azrael serializer falls into an infinite loop trying to serialize these objects. The current workaround is to declare this field transient, but this means that experiment warnings are not preserved when saving/loading.

Set a color scheme to the GUI

If you run the same experiments on multiple machines, it may become confusing to know what window corresponds to what host. One could start LabPal and, optionally, select a "color scheme" so that the GUI would display using a slightly different shade.

The --help command line option prints the program info twice

Enough said

Experiment checkbox does not uncheck

In the experiments page, checking the top checkbox selects all experiments. Checking it again does not unselect all experiments (it does nothing).

Catch OutOfMemory errors

Since experiments in a lab will likely have a high load, one should catch out of memory exceptions and gracefully deal with them (currently, LabPal simply crashes when this happens).

Refactor method isEnvironmentOK

Currently, the method is expected to return null when everything is OK, and a string when there is a problem with the environment. That is quite an odd way of implementing it.

Suggestion:

• make method output type void
• method simply returns when everything is OK
• method throws an EnvironmentException when a problem is detected. The exception's message describes the problem.

This requires:

• creating the class EnvironmentException
• changing the bit in the Laboratory class where isEnvironmentOk() is called (to take into account this refactoring)

Shadow experiments

File upload does not restore groups and descriptions

When experiments are re-uploaded from a JSON file:

1. They are not put back into their respective experiment groups
2. Their description seems to be missing (creates a null pointer when displaying the experiment's page)
3. Plots made from these experiments are empty, so very probably they are not associated back to the proper experiment tables

Status page should show currently running experiment

Kill time includes prerequisites

When an experiment has to generate prerequisites before running, its execution time includes that process. This means that an experiment may get killed by the assistant while it is still generating the prerequisites.

An experiment should have two "start" times: the start of the whole process, and the start of the run method. Timeout should apply to the second, not the first.

Checkbox over a group selects all experiments

When viewing experiments in a group, clicking on the checkbox in the list header selects all experiments in all groups; it should select all experiments in the group only.

Enable caching on the server

Sort macros by name in macros page

Not straightforward, as the MacroMap object creates multiple macros and they will not necessarily be displayed in sequence among other macros.

Cannot set port from command line

Defaults to 21212

Ignore input files that don't end by .labo

Get rid of JSON in experiment parameters

This is a remnant from the time where experiments were serialized "by hand"; imposing experiment parameters to be a JSON map facilitated its serialization. Now that LabPal uses Azrael, there is no reason to force the user to use a JSON map for parameters; it should simply be a Map<String,Object>.

Editable experiment

Add a feature in the GUI to edit the input parameters of an experiment live while the lab is running

Web console does not work in Elinks

Elinks does not know what to do with the HTTP response. Is its Content-type correct? Lynx does not have this problem.

Sign experiments results

In the GUI (or somewhere else), an author could provide its private key, and include in the saved lab an encrypted hash of the lab's results. See #4 for one possible use.

The "assistant started" page should not show the button "Start the assistant"

When visiting the URL "assistant?start" with a GET request, the resulting page shows the message "Assistant started", but the button "Start the assistant" is also present. It should rather be the Stop button.