chili-chips-ba/openeye-camsi

A truly opensource camera serial interface. No frills. No backdoors that compromise security. Outstanding signal integrity. Hi-rez video pipeline with remote connectivity. For Sony, Series7 & open FPGA makers on limited budget. Augments openXC7 CI/CD, challenging its timing-savvy. Promotes the lesser-known EU boards.

openeye-camsi's Issues

Accumulated Cocotb issues/feature requests

1. Find a reliable alternative to instantiating glbl() in RTL (@chili-chips-ba)
   

---

The same goes here -- RTL should not be burdened with sim modeling code. Instead, that code section should be placed in the behavioral model of ISERDESE2

issue#3 - Optimizations of I2C Master controller

Improvement: i2c_top/i2c_ctrl.sv Module

Description

We need to enhance the i2c_top.sv and i2c_ctrl.sv modules by addressing several critical issues and optimizing its performance. The following tasks have been identified for this improvement:

Tasks

• Remove unnecessary flip-flops that are unused or redundant in the i2c_top.sv.
• Remove unnecessary flip-flops that are unused or redundant in the i2c_ctrl.sv.
• Fix the clock 'runt' issue occurring immediately after activation.
• Conduct a detailed simulation to observe state transitions, paying special attention to clock behavior during these transitions.
• Eliminate one of the states in the i2c_ctrl.sv module, as it appears to be unnecessary and redundant.

Details

1. Remove Unnecessary Flip-Flops

There are several flip-flops in the current design that are either not used or are redundant. These need to be identified and removed to streamline the module.

2. Fix Clock 'Runt' Issue

There is a known issue where a 'runt' clock pulse appears immediately after activation. This needs to be fixed to ensure stable clock signals throughout the module's operation.

3. Detailed Simulation of State Transitions

A thorough simulation is required to closely observe the transitions between states. Particular focus should be on the behavior of the clock signal during these transitions to ensure smooth and error-free operation.

4. Eliminate Redundant State

There appears to be a redundant state in the state machine. This state should be identified and removed to simplify the state machine and improve the efficiency of the module.

HDMI

The main limiting factor in the HDMI design regarding resolutions and frame rate support is data bandwidth. We use a timing calculator to calculate an adequate pixel clock for the desired frame rate. In the timing calculator it is possible to choose between various standards:

• CVT
• CVT-RB
• CVT-RBv2
• CEA-861
• DMT
  In the beginning, we used CVT-RBv2, but after the custom configuration was utilized

After choosing one of the standards all necessary parameters, for HDMI are calculated, among them is the pixel clock.

Note

Due to our RTL implementation pixel clock calculated here needs to be multiplied by 5

Because of this and the fact that PLL in our FPGA "Artix-7 csg324-2" can produce a maximum of 628MHz, if the driver for the clock is set to be BUFG, the maximum pixel clock we can use is 125.6MHz. A pixel clock of 125.6MHz is not enough to reproduce the output image of 1080p@60Hz, but it is enough for every resolution listed below:

• 1920 x 1080 30Hz
• 1280 x 720 120Hz
• 1280 x 720 60Hz
• 1024 x 768 60Hz
• 800 x 600 60Hz
• 640 x 480 60Hz

issue#1

Fix I2C reset issue

First Issue:

The i2c_reset signal, which is generated in the top/u_clkrst_gen module, is based on the srst signal from the top/u_clkrst_gen/u_pll_top module. Since the srst signal does not occur at the very beginning, the i2c_reset signal is also delayed, as shown in the attached image. The image illustrates that as a consequence, i2c_sda and i2c_scl start, but then halt due to the reset.

The solution would be to ensure that i2c_reset is initially set to 0 and remains 0 until the rst_delay_cnt counter reaches 4 (since the I2C is enabled only after 4 seconds from the system power-up).

Second Issue:

When the reset signal in the top/u_i2c module goes to 0, data transmission on the I2C bus begins. However, at that moment, the value of the slave address, which is the first data transmitted via I2C, is set to an undefined value. Consequently, the setting of the slave_address_plus_rw (slave address plus read/write) signal is delayed by one data transmission cycle via I2C, which is not desirable. As a result, the first slave address written is 0.

To solve this second problem, the slave address should be set before the start of the I2C section. One approach is to introduce an additional input signal (which can be cam_en, as it occurs before the i2c_reset, as shown in the image).

I2C

Overview:
In our project, we are utilizing standard I2C communication to configure camera registers for various functionalities. Currently, only a partial implementation for writing values to registers one at a time (Option 5, page 19 IMX219 ) has been incorporated. However, the camera offers a broader range of I2C communication options, including single and sequential reads, as well as sequential writes. This issue aims to expand the existing I2C implementation to support all available communication modes and enhance the configurability of camera registers.

Background:
I2C (Inter-Integrated Circuit) is a widely used serial communication protocol for interfacing with peripheral devices, such as cameras, sensors, and memory modules. It facilitates the exchange of data between a master device (e.g. FPGA) and one or more slave devices (e.g. camera) over a two-wire interface.

Current Implementation:
At present, our implementation supports single writes to camera registers, enabling the configuration of individual register values. The FPGA reads the acknowledge bit from the camera, indicating successful register configuration.

Objective:
The objective of this issue is to enhance the existing I2C communication module to support all available modes for configuring camera registers. By implementing comprehensive I2C functionality, we aim to maximize the configurability and performance of the camera within the project.

Next Steps:

• Implement support for single and sequential reads, as well as sequential writes, based on the provided camera documentation.
• Verify proper data transfer and acknowledge bit handling to ensure reliable communication between the FPGA and the camera.
• Conduct thorough testing to validate the functionality of the enhanced I2C communication module across different register configurations and communication modes.

Validation and introduction of CDC Linter into project QA workflow

YosysHQ/yosys#4493 (comment)
https://github.com/BerkeleyLab/Bedrock/blob/master/build-tools/cdc_snitch.md

issue#7 - Enable HiRez Sony camera

Add support for the 4-lane "OneInch" camera (Sony IMX294)

Description:

The next project milestone is to expand the existing 2-lane CSI to 4-lane, and establish video datapath for this particular high-end camera. This milestone encompasses the following tasks:

1. Upgrade simulation environment to allow both 2 and 4-lane data paths, with 4 video formats: RAW8, RAW10, RAW12, RAW14

• CSI BFM
• Testbench and relevant sim scripts
• Time permitting, also add 1-lane option

2. Extend RTL from only RAW8 to also support RAW10, RAW12, and RAW14 image formats

• RAW2RGB block -- If it's too messy to support all 4 formats in one file, implement it in 4 separate files
• Integration

3. End-to-end simulations

• Verify old and new data path options in functional simulation through the entirety of video processing chain

4. Validate OneInch sensor and new data path on the real hardware

• Work out I2C configurations for the camera (as our sim setup does not cover that part)
• Check video pipeline stages from CSI all the way to HDMI. If needed, use Chipscope ILA for debug
• Crate a new demo, similar to the earlier for 2-lane RPi camera, but now with higher resolution Sony IMX294 camera

issue#4 - Creation of Standalone MMCM Model

Creation of Standalone MMCM Model

Description

We need to create a standalone MMCM (Mixed-Mode Clock Manager) model and make some adjustments to the current design by removing the Python parts related to the testbench for the PLL RTL blocks. The following tasks have been identified for this improvement:

Tasks

• Create a standalone MMCM model using SystemVerilog (SV). The model doesn't need to be complete but should include the minimal functionality required for our use case.
• Remove the Python parts of the testbench for the PLL RTL blocks and replace them with the actual RTL used for them.

issue#5 - cocotb spec, dev and status

Issue: Develop Testbench Environment for Testing CSI-2 Camera Model in Cocotb

Description

The goal of this issue is to develop a testbench environment for testing a CSI-2 camera model using Cocotb. The primary objective is to create a Python class that simulates the actual operation of a camera by generating and sending cam_dphy_dat and cam_dphy_clk signals. These signals are differential inputs to the top module, and the class will be responsible for setting these signals to specific values.

Tasks

1. Basic Signal Generation

• Create a basic function to generate random signals for cam_dphy_dat and cam_dphy_clk
  • This function should produce simple signal patterns to observe their behavior in the simulation.
  • Verify the correct behavior and timing of these signals in the simulation environment.
• Add an option for Incrementing Data Pattern
  • Implement a mode to generate incrementing data patterns to facilitate downstream data integrity checks.
  • This option will help follow image transformations more easily when analyzing waveforms.

2. Camera Simulation Class

• Design and implement a Python class to simulate camera behavior
  • The class should handle the generation of differential signals for cam_dphy_dat and cam_dphy_clk.
  • Implement methods to set these signals to desired values for different test scenarios.

3. Data Transmission Functions

• Implement functions to send specific bits and bytes
  • Create methods in the camera simulation class to send individual bits and bytes over the differential signals.
  • Ensure these functions respect the number of data lanes specified as a parameter.

4. Line and Frame Control Functions

• Develop functions for sending start/end of line and frame signals
  • Add methods to handle the transmission of signals indicating the start and end of lines and frames.
  • Implement mechanisms for sending line and frame data according to the CSI-2 protocol.
• Provide an option to specify Resolution and FPS
  • Implement support for resolutions and frame rates defined in the current RTL:
    • define HDMI_720p60
    • define HDMI_1080p30
    • define HDMI_1080p60

5. ECC Calculation

• Implement ECC Calculation Function
  • Develop a function to calculate the ECC for the transmitted data based on the CSI-2 specification.
  • Ensure the function can be easily integrated and called within methods such as start_frame, end_frame, and send_line.

6. Generate Errors and Corner Cases

• Introduce options for generating errors and corner cases
  • Implement options for inserting lane-to-clock skew and lane-to-lane skew.
  • Add support for clock jitter.
  • These options should be specified as the maximum deviation from the nominal, with random deviations within the specified range.

7. Verilator Adaptations

• Adapt Primitives for Verilator Compatibility
  • Identify and adapt RTL primitives that require modification to work correctly with Verilator.
  • Ensure the simulation runs smoothly without issues related to these primitives.

8. Integration and Testing

• Integrate the class and functions into the testbench environment
  • Ensure that the class can be easily instantiated and used within Cocotb tests.
  • Write initial tests to validate the functionality of the class and its methods.

9. Documentation

• Document the class and its usage
  • Provide comprehensive documentation on how to use the class and its methods.
  • Include example test cases and expected outputs.

Expected Outcome

By completing this issue, we expect to have a fully functional testbench environment capable of simulating the CSI-2 camera model's behavior. This environment will facilitate thorough testing and validation of the top module's interaction with the camera signals, ensuring robust and reliable operation.

Line buffering

Overview:
In our project focused on transmitting images from a camera through an FPGA to a screen via HDMI, we encountered memory constraints when attempting to store entire frames. To address this, we implemented a line buffering approach, where a specialized debayer module was developed to store and process image data line by line. This issue aims to document the line buffering technique employed and its integration into the project.

Background:
Traditionally, frame buffering involves storing an entire frame of image data in memory before transmitting it to the display. However, due to memory limitations, this approach was not feasible for our application. Instead, we opted for line buffering, which involves storing and processing image data one line at a time. This not only reduces memory requirements but also facilitates real-time processing and transmission of images.

Line Buffer Implementation:
The line buffering mechanism consists of a dedicated module responsible for managing image data from the camera. This module operates as follows:

• Upon receiving pixel data from the camera, it is stored in a First-In-First-Out (FIFO) buffer.
• As each line of pixel data is completed in the FIFO buffer, it is read out for transmission via HDMI.
• To accommodate the speed mismatch between writing and reading operations, a full line buffer (1280 pixels in our case) is maintained, ensuring a continuous stream of data for HDMI transmission.

Objective:
This issue aims to address the FIFO overflow/underflow problem encountered during the line buffering implementation and propose potential solutions to mitigate its impact on image quality and stability.

Next Steps:

• Investigate the root cause of FIFO overflow/underflow and its impact on image transmission.
• Experiment with different FIFO management strategies to optimize data flow and minimize buffer inconsistencies.
• Evaluate the effectiveness of proposed solutions through simulations and real-world testing.

Pinout inconsistencies and Debayer module

Overview:
While working with the TEB0707-02 FPGA Carrier Card and integrating the debayer module for raw to BGR conversion, our team encountered pinout inconsistencies across CRUVI sites and different board versions. These inconsistencies necessitate creating different builds for each board version and incorporating RTL inverters to accommodate the variations in P and N assignments for differential pins. This issue aims to address the pinout inconsistencies, understand the implications of board version differences, and adapt the debayer module accordingly.

Background:
The debayer module is responsible for converting raw image data captured by the camera into BGR (Blue-Green-Red) format (Debayer), following the primary color filter arrangement of the image sensor (page 89 IMX219). The module utilizes line buffering to process raw lines and generate BGR pixels, which are then sent to the FIFO buffer for further processing and transmission.

Pinout Inconsistencies:
The CRUVI FPGA Carrier Card exhibits pinout inconsistencies across CRUVI sites and different board versions, posing challenges for hardware integration and RTL design. Despite efforts to manage these inconsistencies through RTL inverters, the variations in P and N assignments for differential pins necessitate creating separate builds for each board version.

Manufacturer Response:
The manufacturer acknowledges the pinout inconsistencies and explains that variations in P and N assignments for differential pins are inherent to the design of the 4x5 System on Modules (SoMs) and are mirrored in the baseboard. The flexibility in P/N mapping aims to optimize routing and accommodate different board configurations.

Debayer Module Adaptation:
In light of the pinout inconsistencies and board version differences, the debayer module requires adaptation to ensure compatibility across different CRUVI FPGA Carrier Card versions. Additionally, considerations for converting BGR format to other color spaces (e.g., RGB, HSV, XYZ) may be explored to enhance flexibility and compatibility with diverse applications.

Next Steps:

• Modify the debayer module to dynamically adjust to pinout variations and ensure compatibility with different board versions.
• Investigate options for implementing color space conversion modules (e.g., BGR to RGB, BGR to HSV) to enhance the versatility of the debayer module.
• Conduct thorough testing and validation to verify the functionality and compatibility of the adapted debayer module across different board versions.

issue#6 - csi_rx_align_word

Issue: Review of csi_rx_align_word.sv module

We need to review the implementation of csi_rx_align_word.sv module.

Modified version, as proposed in the issue#5 branch, works properly in simulation. However, something isn't right for the version that's currently on the main branch.

Below are two waveforms that show the difference.

What we see on the first wave is that valid_out is lined up with word_out, while also filtering out the B8 sync bytes, which is indeed the design intent.

On the second wave however, valid_out is encompassing the B8. That causes downstream failures. Also, few other signals (wait_for_sync, packet_done, etc.) are not looking right in the second wave.

issue#2 - Testbench for I2C subsystem

Writing Testbench for I2C Part of Our Camera

We need to write a testbench for the I2C part of our camera. For this purpose, we have utilized the cocotbext-i2c library. Specifically, we used only the I2CDevice.py module from this library, which emulates the slave device. The master part, however, was implemented directly in RTL.

Challenges Encountered:

1. SDA/SCL Bidirectional Signals:

   • The primary challenge was handling the bidirectional (inout) nature of the SDA and SCL lines.
   • To resolve this, we had to define these lines as follows in our RTL code:

     inout tri1 i2c_sda,
     inout tri1 i2c_scl,

   • This specific declaration was necessary for cocotb to properly drive these signals.

2. Simulator Selection:

   • Another significant issue was choosing the appropriate simulator.
   • Verilator, which is commonly used, does not support 'Z (high impedance) and 'X (unknown) values, supporting only log0 and log1.
   • This limitation made it unsuitable for our testbench, requiring us to explore alternative simulators that can handle these signal states.