haxall/haxall

Haxall IoT framework

Haxall Overview

Haxall is an open source software framework for the Internet of Things. It includes an extensive toolkit for working with Project Haystack data. Use it right out of the box as a flexible IoT data gateway which runs at the edge. Haxall is written in Fantom with runtime support for both the Java VM and JavaScript environments.

See https://haxall.io for more more information.

Getting Started

All the documentation is hosted on haxall.io:

• Setup: install and get Haxall running
• Build: instructions to build from source
• Learn: quick links to learn more

License

Haxall is released under the Academic Free License 3.0.

Running Docker Containers

If you want to run the default haxall docker image on a container, rather than setting it up manually onto your local environment, you can do so in one of three ways. They all require that you download Docker Desktop and have it running in the background.

If you follow the instructions, you will create a local bind-mount located somewhere on your filesystem. Any data will be persisted there, and can be changed on the local system to affect the container and vice versa.

Once the container is running, go to http://localhost:8080 to use haxall. Default suuser and supass should be

1. Build the image yourself with Docker Compose

• Make sure you have dowloaded and are running Docker Desktop.
• Download the latest release of haxall (any releases before Jun 26, 2024 will not have this functionality), or clone the repository.
• Run this command on a terminal in the root folder of haxall on your local system:

docker compose up

The command will build and run the image for haxall, creating a local bind-mount in the dbs folder.

Use Github Packages

There is a prebuilt image that rebuilds on every push to the repo, so it is the most up-to-date. You can set it up either on command line, or through the Docker Desktop app.

• On the repository website, go to packages, haxall, and copy the command to install the image from the command line.
• Run that command in a terminal from anywhere.

2. Command Line

• After doing the above, run this command:

docker run -v ./haxall:/app/haxall/dbs -p 8080:8080 --name haxall_run ghcr.io/haxall/haxall

This will create a local bind-mount in a folder called haxall, from wherever you run the command.

3. Docker Desktop

• Open the Docker Desktop app
• Go to images, the haxall image should be there.
• On the right side under actions, hit the arrow.
• Here are example inputs:

In the example inputs, a local bind-mount will be created at the folder specified by path, in this case C:\Apps\haxall.

haxall's Issues

Unknown Pod Error

After download and unzipping, when I try to verify using the hx version command returns :

sys::UnknownPodErr: hx

I've tried both executing from the haxall directory and adding \bin to PATH.
I can see in \lib\fan that the hx pod is present.
I've tried both version 3.1.3 and 3.1.4 and get the same.

Do I need to change something in the setup to look for the pod in that location? I suspect it is very simple.

Thanks in advance for the help. I am extremely new to Fantom and trying to learn as part of working with Haxall. Excited to see what the future of the project holds.

Haxall as Haystack server

I managed to install Haxall on Ubuntu 18 and I am able to start the hx daemon and the Axon shell UI.
Not sure about the scope of Haxall or what to do next :)

I am mostly interested in using it as a Haystack server.
Is this possible? How do I send requests to the API?

Haystack Authentication

Cannot build haxall

cannot build haxall on ubuntu in docker
error:
/root/haxall/src/lib/hxShell/fan/HxShellWeb.fan(63,7): Unknown method 'web::WebOutStream.initJs'

Setup issues

Fresh git clone of the main branch on debian 12.

The documentation calls for bin/fanlaunch which doesn't exist.

As a hacky solution I copied bin/fanlaunch from 3.1.9 and it seemed to work.

I then attempted to run ./hx and got the following error.

Where is this file located?

I had been playing with 3.1.9 but it does not have a /src file. Since 3.1.9 is working I copied over /src but this is janky.

Thanks for the help!

ModbusBlockGap issue with Coils

One of our devs found the below issue while working on a customer project and it appears to be in the modbus pod which we don't have access to. The screenshots are from FIN, but this was also reproduced in the Skyspark UI v3.0.29.

How to reproduce

1. Create a new proj
2. Enable modbus ext
3. Create new registerMap called “test” like:

name,addr,data,rw,scale,dis,unit,tags
ping,00001,bit,r,,Ping,,
bit01,00001,bit,r,,,,
bit02,00002,bit,r,,,,
bit03,00003,bit,r,,,,
bit_empty,00004,bit,r,,,,
bit05,00005,bit,r,,,,
bit06,00006,bit,r,,,,
bit07,00007,bit,r,,,,

4. Create new connector with (IMPORTANT: add modbusBlockGap:3 to the default tags):

conn
modbusBlockGap:3
modbusConn
modbusRegMapUri:data/modbus/test.csv
modbusSlave:1
uri:modbus-tcp://[127.0.0.1/](http://127.0.0.1/)

5. Run a modbus slave simulator, like modsim, configured like into the attached screen, so with:

bit1 = 0
bit2 = 0
bit3 = 0
bit4 = 1 //<---- Important: set this to 1
bit5 = 0
bit6 = 0
bit7 = 0

6. Create an equip with those points referencing the registers (IMPORTANT: skip bit 4):
7. View the point live values for each
   bit1 (OK: curVal should be == 0 and is == 0)
   bit2 (OK: curVal should be == 0 and is == 0)
   bit3 (OK: curVal should be == 0 and is == 0)
   bit5 (ERROR: curVal should be == 0 and is == 1, it’s reading bit4)
   bit6 (ERROR: it’s reading bit5)
   bit7 (ERROR: it’s reading bit6)

Expected behavior:
You should see all the points with curVal == 0

Error
There is a shift of the value from bit 4 that is equals to 1 to bit 5 that should be equals to 0 but now it’s 1.

NOTES:
Removing modbusBlockGap:3 there is no error

Doing the same with numbers works fine, also with modbusBlockGap:3 (see pictures related to number tests)

Unexpected haxall build error when using jdk-17.0.1

Error occurs, Build Haxall: Step 5

error: option --boot-class-path not allowed with target 17 ERR: CompileJava failed sys::Err BUILD FAILED [820ms]!

As a general question. is there a specific version of the JDK required?

sys::UnknownServiceErr: hx::HxCryptoService

Hi,

I'm attempting to install and run your project but fail to start the server due to the error "sys::UnknownServiceErr: hx::HxCryptoService"

Configuration:

 haxall git:(main) ✗ fan hx version

Haxall CLI
Copyright (c) 2009-2021, SkyFoundry LLC
Licensed under the Academic Free License version 3.0

hx.version:       3.1.1
java.version:     17
java.vm.name:     OpenJDK 64-Bit Server VM
java.vm.vendor:   Homebrew
java.vm.version:  17+0
java.home:        /usr/local/Cellar/openjdk/17/libexec/openjdk.jdk/Contents/Home
fan.version:      1.0.77
fan.platform:     macosx-x86_64
fan.env.path:     /Users/xxx/Documents/shared-git-clones/haxall, /Users/xxx/Documents/shared-git-clones/haystack-defs, /usr/local/Cellar/fantom/1.0.77/libexec

When running fan hx run ./work I get the following output:

sys::UnknownServiceErr: hx::HxCryptoService
  hxd::HxdServiceRegistry.get (HxdServiceRegistry.fan:101)
  hxd::HxdServiceRegistry.get (HxdServiceRegistry.fan)
  hxd::HxdServiceRegistry.make$ (HxdServiceRegistry.fan:63)
  hxd::HxdServiceRegistry.make (HxdServiceRegistry.fan:19)
  hxd::HxdRuntime.servicesRebuild (HxdRuntime.fan:103)
  hxd::HxdRuntimeLibs.init (HxdRuntimeLibs.fan:72)
  hxd::HxdRuntime.make$ (HxdRuntime.fan:47)
  hxd::HxdRuntime.make (HxdRuntime.fan:27)
  hxd::HxdBoot.init (HxdBoot.fan:100)
  hxd::HxdBoot.run (HxdBoot.fan:108)
  hxd::RunCli.run (HxdBoot.fan:226)
  util::AbstractMain.main (AbstractMain.fan:373)
  hx::Main.main (HxCli.fan:88)
  jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (NativeMethodAccessorImpl.java)
  jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
  jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
  java.lang.reflect.Method.invoke (Method.java:568)
  fan.sys.Method.invoke (Method.java:573)
  fan.sys.Method$MethodFunc.callList (Method.java:212)
  fan.sys.Method.callList (Method.java:138)
  fanx.tools.Fan.callMain (Fan.java:183)
  fanx.tools.Fan.executeType (Fan.java:147)
  fanx.tools.Fan.execute (Fan.java:41)
  fanx.tools.Fan.run (Fan.java:308)
  fanx.tools.Fan.main (Fan.java:346)
```
Full error attached:
[error.txt](https://github.com/haxall/haxall/files/7380861/error.txt)

Please advice!

build error Unknown type 'FileLoc'

fan -version
Fantom Launcher
Copyright (c) 2006-2021, Brian Frank and Andy Frank
Licensed under the Academic Free License version 3.0

Java Runtime:
java.version: 17
java.vm.name: Java HotSpot(TM) 64-Bit Server VM
java.vm.vendor: Oracle Corporation
java.vm.version: 17+35-LTS-2724
java.home: C:\Program Files\Java\jdk-17
fan.platform: win32-x86_64
fan.version: 1.0.78
fan.env: util::PathEnv
fan.home: D:\tools\fantom-1.0.78

Env Path:
D:\code\haxall (work)
D:\code\haystack-defs
D:\tools\fantom-1.0.78 (home)

 fan src\build.fan
compile [oauth2]
  Compile [oauth2]
    FindSourceFiles [7 files]
    WritePod [file:/D:/code/haxall/lib/fan/oauth2.pod]
compile [ftp]
  Compile [ftp]
    FindSourceFiles [3 files]
    WritePod [file:/D:/code/haxall/lib/fan/ftp.pod]
  javaNative [ftp]
    Exec [C:\Program Files\Java\jdk-17\bin\java.exe -cp D:\tools\fantom-1.0.78\lib\java\sys.jar -Dfan.home=D:\tools\fantom-1.0.78 fanx.tools.Jstub -d D:\code\haxall\src\util\ftp\temp-java ftp]
    Java Stub [ftp]
    CompileJava
    Exec [C:\Program Files\Java\jdk-17\bin\jar.exe -xf D:\code\haxall\src\util\ftp\temp-java\ftp.jar]
    Delete [file:/D:/code/haxall/src/util/ftp/temp-java/ftp.jar]
    Delete [file:/D:/code/haxall/src/util/ftp/temp-java/meta-inf/]
    Exec [C:\Program Files\Java\jdk-17\bin\jar.exe -fu D:\code\haxall\lib\fan\ftp.pod -C D:\code\haxall\src\util\ftp\temp-java .]
    Delete [file:/D:/code/haxall/src/util/ftp/temp-java/]
compile [rdf]
  Compile [rdf]
    FindSourceFiles [9 files]
    CompileJs
    WritePod [file:/D:/code/haxall/lib/fan/rdf.pod]
compile [docker]
  Compile [docker]
    FindSourceFiles [29 files]
    WritePod [file:/D:/code/haxall/lib/fan/docker.pod]
  javaNative [docker]
    Exec [C:\Program Files\Java\jdk-17\bin\java.exe -cp D:\tools\fantom-1.0.78\lib\java\sys.jar -Dfan.home=D:\tools\fantom-1.0.78 fanx.tools.Jstub -d D:\code\haxall\src\util\docker\temp-java docker]
    Java Stub [docker]
    CompileJava
    Exec [C:\Program Files\Java\jdk-17\bin\jar.exe -xf D:\code\haxall\src\util\docker\temp-java\docker.jar]
    Delete [file:/D:/code/haxall/src/util/docker/temp-java/docker.jar]
    Delete [file:/D:/code/haxall/src/util/docker/temp-java/meta-inf/]
    Exec [C:\Program Files\Java\jdk-17\bin\jar.exe -fu D:\code\haxall\lib\fan\docker.pod -C D:\code\haxall\src\util\docker\temp-java .]
    Delete [file:/D:/code/haxall/src/util/docker/temp-java/]
compile [data]
  Compile [data]
    FindSourceFiles [9 files]
D:\code\haxall\src\core\data\fan\DataLogRec.fan(25,12): Unknown type 'FileLoc'
D:\code\haxall\src\core\data\fan\DataSpec.fan(83,19): Unknown type 'FileLoc'
BUILD FAILED [3315ms]!

Main branch build fails - Unknown method 'sys::Int.clamp'

Building the main branch from the updates yesterday fail with the unknown method below. Branch 3.1.2 builds successfully.

/Users/myUser/Documents/haxall/src/lib/hxConn/fan/ConnLib.fan(215,35): Unknown method 'sys::Int.clamp'
/Users/myUser/Documents/haxall/src/lib/hxConn/fan/ConnLib.fan(216,23): Unknown method 'sys::Int.clamp'
BUILD FAILED [5816ms]!

NullErr when calling ConnDispatch.close()

haxall/src/lib/hxConn/fan/ConnDispatch.fan

Line 98 in 216df93

This close(Err? cause)

ConnDispatch.close(Err? cause) has a nullable cause, but the underlying ConnMgr.close(Obj cause) does not allow nulls.

So, err, passing null to ConnDispatch.close() results in a NullErr.

Clarification on Folio Storage and Haxall Use Case

Just wanted to get some clarification on the way the folio database stores data and the capabilities of Haxall.

“Folio persists data to disk, but operates as an in-memory based database. Records are read from disk on startup and stored in RAM for fast access. This design supports the real-time nature of sensor data. But it also imposes limits on Folio since most hardware tend to have less RAM than disk space.”

I’m attempting to pull history data and points from multiple JACEs (100+) via nHaystack. Each of these JACEs has multiple pieces of equipment with 20+ points each.

I want to leverage the compression and query speed of Folio to create a single API endpoint for exporting data for visualizations.

Is this a valid use case for Haxall given a sufficiently capable server?
Do I need to store the historical data elsewhere?

Thank you for the info.

Trouble Building Haxall

I'm in the process of putting together a Dockerfile for Haxall and I'm running into trouble building it. I'm using the latest fantom release (1.0.76), latest haystack-defs main branch and the latest haxall main branch.

Following the directions on the README, the build seems to work fine until it fails on an Unknown method 'web::WebOutStream.initJs'.

Full trace below:

root@f4ae8ab5a88f:/haxall# src/build.fan
compile [rdf]
  Compile [rdf]
    FindSourceFiles [9 files]
    CompileJs
    WritePod [file:/haxall/lib/fan/rdf.pod]
compile [haystack]
  Compile [haystack]
    FindSourceFiles [50 files]
    CompileJs
    WritePod [file:/haxall/lib/fan/haystack.pod]
compile [auth]
  Compile [auth]
    FindSourceFiles [13 files]
    CompileJs
    WritePod [file:/haxall/lib/fan/auth.pod]
compile [def]
  Compile [def]
    FindSourceFiles [18 files]
    CompileJs
    WritePod [file:/haxall/lib/fan/def.pod]
compile [defc]
  Compile [defc]
    FindSourceFiles [45 files]
    WritePod [file:/haxall/lib/fan/defc.pod]
compile [axon]
  Compile [axon]
    FindSourceFiles [39 files]
    CompileJs
    WritePod [file:/haxall/lib/fan/axon.pod]
compile [obs]
  Compile [obs]
    FindSourceFiles [7 files]
    WritePod [file:/haxall/lib/fan/obs.pod]
compile [folio]
  Compile [folio]
    FindSourceFiles [14 files]
    CompileJs
    WritePod [file:/haxall/lib/fan/folio.pod]
compile [hx]
  Compile [hx]
    FindSourceFiles [12 files]
    CompileJs
    WritePod [file:/haxall/lib/fan/hx.pod]
compile [hxStore]
  Compile [hxStore]
    FindSourceFiles [9 files]
    WritePod [file:/haxall/lib/fan/hxStore.pod]
  javaNative [hxStore]
    Exec [/docker-java-home/bin/java -cp /fantom/lib/java/sys.jar -Dfan.home=/fantom fanx.tools.Jstub -d /haxall/src/core/hxStore/temp-java hxStore]
    Java Stub [hxStore]
    CompileJava
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
    Exec [/docker-java-home/bin/jar -xf /haxall/src/core/hxStore/temp-java/hxStore.jar]
    Delete [file:/haxall/src/core/hxStore/temp-java/hxStore.jar]
    Delete [file:/haxall/src/core/hxStore/temp-java/meta-inf/]
    Exec [/docker-java-home/bin/jar -fu /haxall/lib/fan/hxStore.pod -C /haxall/src/core/hxStore/temp-java .]
    Delete [file:/haxall/src/core/hxStore/temp-java/]
compile [hxFolio]
  Compile [hxFolio]
    FindSourceFiles [19 files]
    WritePod [file:/haxall/lib/fan/hxFolio.pod]
compile [hxd]
  Compile [hxd]
    FindSourceFiles [11 files]
    WritePod [file:/haxall/lib/fan/hxd.pod]
compile [hxApi]
  Compile [hxApi]
    FindSourceFiles [2 files]
    WritePod [file:/haxall/lib/fan/hxApi.pod]
compile [hxHttp]
  Compile [hxHttp]
    FindSourceFiles [1 files]
    WritePod [file:/haxall/lib/fan/hxHttp.pod]
compile [hxUser]
  Compile [hxUser]
    FindSourceFiles [7 files]
    WritePod [file:/haxall/lib/fan/hxUser.pod]
compile [hxShell]
  Compile [hxShell]
    FindSourceFiles [12 files]
/haxall/src/lib/hxShell/fan/HxShellWeb.fan(63,7): Unknown method 'web::WebOutStream.initJs'
BUILD FAILED [7739ms]!

The Dockerfile (and the shell script that starts it) are available here so you can see what I'm doing. I tried building with the latest master branch of fantom, but then running fan results in:

 > [11/15] RUN /haystack-defs/src/build.fan:
#14 0.383 Error: Could not find or load main class fanx.tools.Fan
#14 0.383 Caused by: java.lang.ClassNotFoundException: fanx.tools.Fan

I also tried to build on my Ubuntu machine and hit the same issues, so I don't think its a question of the Docker image not having some dependency.

Let me know if I'm missing anything or if I can provide any more info

Clarifications on the Build process

I'm trying to dockerize to this repo so that it's really easy to get a new Haxall instance up and running. There are 2 options:

1. Make a Dockerfile that downloads the pre-bundled zip from the GitHub release and sets it up
2. Make a Dockerfile in this repo that performs the build process

I have done option 1 here, and it works fine. The big downside is that I must manually run the docker build/publish process on any newly released versions.

Option 2 is ideal, as it could be integrated into this repo's CI, and published to this repo's container registry automatically. I have an attempt in this commit on this branch that tried to follow the Build documentation. However, I ran into the following issues:

• Overlapping files are created by the fantom, haystack-defs, and haxall build processes (specifically, .props and .pod files). It wasn't always clear whether the Haxall release zip had combined these, and if so which ones took precedence.
• The web UI tables aren't able to render correctly. The browser console gave the following error Uncaught: Object { "$err": Error, m_msg: "FontMetrics not supported: 700 9pt Helvetica", m_cause: undefined }.

Could you provide guidance on the process used to generate the release zips?

Cant Build

Hello, im following the building instructions in the README but am having issues getting it to build properly. Im getting the following error. Any ideas?

C:\Users\birch\Documents\GitHub\haxall\src>fan build.fan
compile [rdf]
ERR: Target 'compile' failed [/C:/Users/birch/Documents/GitHub/haxall/src/util/rdf/build.fan]
sys::Err: Undefined macro key: fan.depend
  build::BuildScript.applyMacros (BuildScript.fan:146)
  build::BuildScript.applyMacros (BuildScript.fan)
  build::BuildPod.compileFan (BuildPod.fan:220)
  fan.sys.List.map (List.java:844)
  build::BuildPod.compileFan (BuildPod.fan:220)
  build::BuildPod.compile (BuildPod.fan:166)
  java.lang.reflect.Method.invoke (Unknown)
  fan.sys.Method.invoke (Method.java:573)
  fan.sys.Method$MethodFunc.callOn (Method.java:244)
  fan.sys.Method.callOn (Method.java:139)
  build::TargetMethod.run (TargetMethod.fan:77)
  build::BuildGroup.runOnChildren (BuildGroup.fan:87)
  fan.sys.List.each (List.java:604)
  build::BuildGroup.runOnChildren (BuildGroup.fan:84)
  build::GroupTarget.run (BuildGroup.fan:120)

unable to run mqtt function

Hi 👋
I was trying to run the mqtt example provided in the official documentation:
https://haxall.io/doc/lib-mqtt/doc

I have created an mqttConn using the WebUI

id:@2c02857d-b9358a39 "MQTT Connector"
dis:MQTT Connector
conn
mod:2023-05-25T20:45:49.028Z
mqttClientId:MqttTestClient
mqttConn
mqttVersion:"v3.1.1"
uri:"mqtt://broker.hivemq.com/"

but when I run the mqtt publish function, in the haxall shell
read(mqttConn).mqttPublish("testTopic", "Hello, MQTT!", {mqttQos: 2})
I get the following error:
☹axon::EvalErr: Unknown symbol 'mqttPublish' [eval:1] axon::EvalErr: Unknown symbol 'mqttPublish' [eval:1] axon::AxonContext.resolve (AxonContext.fan:197) axon::Var.eval (Var.fan:23) axon::Expr.evalToFunc (Expr.fan:62) axon::Call.evalTarget (Call.fan:42) axon::Call.eval (Call.fan:39) hxd::HxdContext.evalOrReadAll (HxdContext.fan:165) hx::HxEvalOp.onRequest (HxApiOp.fan:193) hx::HxApiOp.onService (HxApiOp.fan:41) hxApi::HxApiWeb.onService (HxApiWeb.fan:52) hxHttp::HttpRootMod.onService (HttpRootMod.fan:54) wisp::WispActor.process (WispActor.fan:80) wisp::WispActor.receive (WispActor.fan:38) concurrent::Actor._dispatch (Actor.java:274) concurrent::Actor._work (Actor.java:229) concurrent::ThreadPool$Worker.run (ThreadPool.java:273)

Does something specific needs to be done to pods to enable functions?

Did you develop tridium too?

Find nodeid without enabling cluster

Is there a way to query for the nodeid of a haxall instance without having to enable the "cluster" library? Thank you!