tz-physics/computational-imaging-resources

Resources for Computational Imaging & Display & Graphics

Computational Imaging & Display & Graphics Resources

Computational Imaging

Lens Design

Deep learning-enabled framework for automatic lens design starting point generation.
G Côté, JF Lalonde, S Thibault.
Optics express, 2021. [Paper]

Comparison of methods for end-to-end co-optimization of optical systems and image processing with commercial lens design software.
A Fontbonne, H Sauer, F Goudail.
Optics Express, 2022. [Paper]

DiffOptics

Differentiable compound optics and processing pipeline optimization for end-to-end camera design.
E Tseng, A Mosleh, F Mannan, K St-Arnaud, A Sharma, Y Peng, A Braun, D Nowrouzezahrai, et al.
ACM Transactions on Graphics (TOG), 2021. [Paper]

Pixelrnn: in-pixel recurrent neural networks for end-to-end-optimized perception with neural sensors.
HM So, L Bose, P Dudek, G Wetzstein.
CVPR, 2024. [[Paper](https://openaccess.thecvf.com/content/CVPR2024/papers/So_PixelRNN_In-pixel_Recurrent_Neural_Networks_for_End-to-end-optimized_Perception_with_Neural_CVPR_2024_paper.pdf]

TrustedRobustImaging

Fog simulation on real LiDAR point clouds for 3D object detection in adverse weather.
M Hahner, C Sakaridis, D Dai, et al.
ICCV, 2021. [Paper]

Lidar snowfall simulation for robust 3d object detection.
M Hahner, C Sakaridis, M Bijelic, et al.
CVPR, 2022. [Paper] [Github]

MiniAccurateImaging

Neural nano-optics for high-quality thin lens imaging.
E Tseng, S Colburn, J Whitehead, L Huang, SH Baek, A Majumdar, F Heide.
Nature communications, 2021. [Paper] [Code]

End-to-end optimization of metasurfaces for imaging with compressed sensing.
G Arya, WF Li, C Roques-Carmes, M Soljacic, SG Johnson, Z Lin.
ACS Photonics, 2024. [Paper]

Dynamic3DImaging

Centimeter-wave free-space neural time-of-flight imaging.
SH Baek, N Walsh, I Chugunov, Z Shi, F Heide.
ACM Transactions on Graphics, 2023. [Paper]

Cross-spectral Gated-RGB Stereo Depth Estimation.
S Brucker, S Walz, M Bijelic, et al.
CVPR, 2024. [Paper] [Github]

Computational Display

Holography

Neural holography with camera-in-the-loop training.
Y Peng, S Choi, N Padmanaban, G Wetzstein.
ACM Transactions on Graphics (TOG), 2020. [Paper]

Joint neural phase retrieval and compression for energy-and computation-efficient holography on the edge.
Y Wang, P Chakravarthula, Q Sun, B Chen.
ACM Transactions on Graphics, 2022. [Paper] [Github]

Holographic glasses for virtual reality.
J Kim, M Gopakumar, S Choi, Y Peng, W Lopes, G Wetzstein.
ACM SIGGRAPH 2022 Conference Proceedings, 2022. [Paper]

Computational Graphics

Neural Rendering

Resources

awesome neural rendering.
Deep image or video generation approaches that enable explicit or implicit control of scene properties such as illumination, camera parameters, pose, geometry, appearance, and semantic structure..
[Github]

Implicit Neural Scene Representation and Rendering

Neural scene representation and rendering.
SMA Eslami, DJ Rezende, F Besse, F Viola, et al.
Science, 2018. [Github]

Deferred neural rendering: Image synthesis using neural textures.
J Thies, M Zollhöfer, M Nießner.
ACM Transactions on Graphics (TOG), 2019.

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng.
ECCV 2020. [PDF] [Project]

SIREN: Implicit Neural Representations with Periodic Activation Functions.
Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, Gordon Wetzstein.
NeurIPS 2020 (Oral). [PDF][Github]

Neural Ray-Tracing: Learning Surfaces and Reflectance for Relighting and View Synthesis.
J Knodt, SH Baek, F Heide.
ArXiv, 2021.[Github]

Autoint: Automatic integration for fast neural volume rendering.
DB Lindell, JNP Martel, et al.
CVPR, 2021.[Github]

NeRF in the Wild Neural Radiance Fields for Unconstrained Photo Collections.
R Martin-Brualla, N Radwan, et al.
CVPR, 2021.

Neural scene graphs for dynamic scenes.
J Ost, F Mannan, N Thuerey, et al.
CVPR, 2021.[Github]

ACORN: Adaptive Coordinate Networks for Neural Scene Representation.
JJNP Martel, DB Lindell, CZ Lin, ER Chan, et al.
SIGGRAPH, 2021.[Github]

3D Reconstruction

KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera.
S Izadi, D Kim, O Hilliges, D Molyneaux, et al.
UIST, 2011.

Soft 3D reconstruction for view synthesis.
E Penner, L Zhang.
ACM Transactions on Graphics (TOG), 2017.

State of the Art on 3D Reconstruction with RGB‐D Cameras.
M Zollhöfer, P Stotko, A Görlitz, et al.
Computer Graphics Forum, 2018.

State of the Art on 3D Reconstruction with RGB‐D Cameras.
M Zollhöfer, P Stotko, A Görlitz, et al.
Computer Graphics Forum, 2018.

Disn: Deep implicit surface network for high-quality single-view 3d reconstruction.
Q Xu, W Wang, D Ceylan, R Mech, et al.
NIPS, 2019.

Occupancy networks: Learning 3d reconstruction in function space.
L Mescheder, M Oechsle, M Niemeyer, et al.
CVPR, 2019.

Fast Online 3D Reconstruction of Dynamic Scenes From Individual Single-Photon Detection Events.
Y Altmann, S McLaughlin, et al.
IEEE Transactions on Signal Processing, 2019.

DI-Fusion: Online Implicit 3D Reconstruction with Deep Priors.
J Huang, SS Huang, H Song, et al.
CVPR, 2021.

SP-GAN: Sphere-guided 3D shape generation and manipulation.
R Li, X Li, KH Hui, CW Fu.
ACM Transactions on Graphics (TOG), 2021.