spectacularai/sdk

Spectacular AI SDK

sdk's Issues

Wheel for Jetson Nano?

How can I install the Spectacular AI SDK on the Jetson Nano? On PyPI there are only files available for x86.

Support auto focus capture

Hi,
First, thanks for your great work!

I wonder if there is a plan to release a version with an auto-focus mode, as for now the distance is locked during the scan.

Thanks!

Unable to use RGB camera (Oak-D) for Feature-Tracking

Hello Spectacular AI Team,
I have an Oak-D Pro (which is basically an Oak-D S2 with an IR dot projector) and I am trying to use the RGB camera for Feature-Tracking.
The monochrome results are not usable. So Feature-Tracking with color-reference is needed.

If I use:

config = spectacularAI.depthai.Configuration()
config.useFeatureTracker = True
config.useColor = True

I get this Warning:

SpectacularAI WARN: Unsupported image format RGB888i
SpectacularAI ERROR: Cannot convert image to Spectacular AI format

Trying to change the Format for example with:

rgb_camera = pipeline.create(depthai.node.ColorCamera)
rgb_camera.setInterleaved(False)
rgb_camera.setColorOrder(depthai.ColorCameraProperties.ColorOrder.BGR)

...doesn't help because it seems that you create internally a depthai.node.ColorCamera instance.

Because I get this exception:

ColorCamera(3) - Specified camera board socket already used.

Is there anything I can do to get the RGB camera working with Feature-Tracking.

Also, I'd like to ask which is the lowest usable voxel-size? Thank you for your efforts!

Greetings to Finnland

'spectacularAI.depthai.Pipeline' object has no attribute 'ext'

Hello,

I have this error loading ros2 launch launch/mapping.py from ros2-integration2

[ros2_node-1] ... spectacularAI_ros2/install/spectacularai_depthai/lib/python3.8/site-packages/spectacularai_depthai/ros2_node.py", line 130, in init
[ros2_node-1] self.vio_pipeline.ext.rae.front.colorLeft.setFps(fps)
[ros2_node-1] AttributeError: 'spectacularAI.depthai.Pipeline' object has no attribute 'ext'

    fps = 15 # 20 Would be better but is slower.
    self.vio_pipeline.ext.rae.front.colorLeft.setFps(fps)
    self.vio_pipeline.ext.rae.front.colorRight.setFps(fps)

I run with
Python 3.8.10
spectacularAI 1.21.0
depthai 2.22.0.0
numpy 1.24.4

Best regards

Issues running the SDK examples with OAK-D-SR

I'm trying to run the vio_visu.py example with the OAK-D-SR and running into some problems.

1. The OV9782 cameras on the SR support 720p and 800p resolution, but "800p" can't be used as a value for inputResolution in the VIO pipeline config.

Here's the error message:

SpectacularAI ERROR: invalid or unsupported color input resolution 800p

My workaround was to set it manually back to 800p after constructing the VIO pipeline with 1200p but before starting the DepthAI pipeline:

config = spectacularAI.depthai.Configuration()
config.useColor = False
config.useStereo = False
config.inputResolution = "1200p"
config.useColorStereoCameras = True
vio_pipeline = spectacularAI.depthai.Pipeline(pipeline, config)
vio_pipeline.colorRight.setResolution(depthai.ColorCameraProperties.SensorResolution.THE_800_P)
vio_pipeline.colorPrimary.setResolution(depthai.ColorCameraProperties.SensorResolution.THE_800_P)
vio_pipeline.colorRight.setVideoSize(1280, 800)
vio_pipeline.colorPrimary.setVideoSize(1280, 800)

2. The VIO pipeline tries to call dai::CalibrationHandler::getCameraExtrinsics and dai::CalibrationHandler::getDistortionCoefficients on CAM_A, which isn't present on the SR (this device only has CAM_B and CAM_C).

The error message:
There is no Camera data available corresponding to the the requested source cameraId

My hacky GDB workaround to force it to never call those functions with CAM_A:

gdb --args python3 vio_visu.py
(gdb) set non-stop on
(gdb) break getCameraExtrinsics
(gdb) commands
(gdb) set $rdx=1
(gdb) c
(gdb) end
(gdb) break getDistortionCoefficients
(gdb) commands
(gdb) set $rdx=1
(gdb) c
(gdb) end
(gdb) run

3. After working around the above issues, I'm running into this one. I'm not exactly sure what the cause is, but it seems like it could be related to this.

Exception in thread Thread-1 (reader_loop):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/sdk-examples/python/oak/vio_visu.py", line 151, in reader_loop
    for vio_out in vio_source:
  File "/home/ubuntu/sdk-examples/python/oak/vio_visu.py", line 43, in live_vio_reader
    vio_pipeline.startSession(device) as vio_session:
IndexError: _Map_base::at

'SpectacularAI ERROR: x:441

Hello,
I'm trying to use sai-cli process with data from Android phone generated with SpectacutalAI app, but in any case it returned
'SpectacularAI ERROR: x:441
Abandon (core dumped)'

I have activated my nerfstudio conda environment nerfstudio is working well, and i have done pip install spectacularAI[full]

I can't find anywhere anything that help me

I tried this on 3 different machines on :
Laptop i7-8850 RAM 32GB + Quadro P3200 6 GB (windows)
Desktop AMD FX6200 RAM 16GB + RTX3060 12GB (windows)
Server Xeon(R) W-2245 RAM 128GB + 2x RTX6000 24GB (linux)

What can i do ?

Invalid rotation matrix for Femto Bolt

Hello!

I'm currently testing Orbbec Femto Bolt device.

When capturing the environment using: ./bin/sai-record-orbbec - the visualization looks fine. When I try to export a pointcloud from the captured data using: sai-cli process data/2024-06-12_18-47-56 cloud.ply. I get: SpectacularAI WARN: Calibration: camera1 imuToCamera has invalid rotation matrix!. And the resulting pointcloud is smushed up together.

I wasn't able to find the correct imuToCamera calibration inside the OrbbecSDK viewer, which leads me to believe you've done custom calibration to get those params?

Here's how the calibration.json file looks inside one of the data folders:

{"cameras":[{"focalLengthX":748.878,"focalLengthY":748.348,"principalPointX":639.032,"principalPointY":357.54,"imageWidth":1280,"imageHeight":720,"distortionCoefficients":[0.0766592,-0.105846,-0.000204852,7.88337e-05,0.0438596,0,0,0],"model":"brown-conrady","imuToCamera":[[-3.0371e-05,-1,-0.00034758,0.0381531],[-0.00123998,-0.000347542,0.999999,0.0284239],[-0.999999,3.0802e-05,-0.00123996,-0.0368053],[0,0,0,1]]},{"focalLengthX":252.522,"focalLengthY":252.458,"principalPointX":255.06,"principalPointY":256.011,"imageWidth":512,"imageHeight":512,"distortionCoefficients":[33.4157,16.6585,7.26705e-05,-1.14212e-05,0.59129,33.6846,28.0125,3.78681],"model":"brown-conrady","imuToCamera":[[0.00420137,-0.993743,-0.0055639,0.0702844],[0.110339,-0.00506858,0.993881,0.0337094],[-0.993885,-0.00478962,0.110315,-0.0347502],[0,0,0,1]]}]}

spectacularAI_native detail

May I get the spectacularAI_native source code? I want to try some new vislam algorithms based on your framework. Thank u very much!!

Unable to get the color property from a spectacularAI.depthai.Pipeline

I was trying to access the color property in a spectacularAI.depthai.Pipeline instance and I get Unable to convert function return value to a Python type! error on the 1.5.0 release from PIP on python 3.9.5 x64

Documentation for Session.addAbsolutePose()

I am a bit confused at the documentation for addAbsolutePose(): https://spectacularai.github.io/docs/sdk/wrappers/oak.html#spectacularAI.depthai.Session.addAbsolutePose

It takes in a Pose object, 3x3 matrix, and a float value. What should each of these 3 arguments be?