5g4s/paper

paper's Issues

Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning

https://arxiv.org/abs/1907.04490

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers

https://arxiv.org/abs/2002.11794

RANDOM FEATURE ATTENTION

https://arxiv.org/abs/2103.02143

Linear Transformers Are Secretly Fast Weight Programmers

https://arxiv.org/abs/2102.11174

SimMIM: a Simple Framework for Masked Image Modeling

https://openaccess.thecvf.com/content/CVPR2022/papers/Xie_SimMIM_A_Simple_Framework_for_Masked_Image_Modeling_CVPR_2022_paper.pdf

Reformer: The Efficient Transformer

https://arxiv.org/abs/2001.04451

Modulated Periodic Activations for Generalizable Local Functional Representations

https://arxiv.org/abs/2104.03960

Hamiltonian Generative Networks

https://arxiv.org/abs/1909.13789

DIFFERENTIABLE PHYSICS SIMULATION

Most current simulation algorithms can only provide forward results but not backpropagated gradients of the input.
Many learning-based tasks involving simulation can be greatly improved if gradients of simulation can be provided.
By making use of the gradients from the physically-aware simulation, differentiable simulation can optimize the unknown parameters faster and more accurately than gradient-free methods.

https://openreview.net/pdf?id=p-SG2KFY2

GhostNetV2: Enhance Cheap Operation with Long-Range Attention

https://arxiv.org/abs/2211.12905

Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization

https://arxiv.org/abs/1703.06868

スタイル変換には、スローな反復的な最適化を必要とする。そこで、コンテンツの特徴の平均と分散をスタイルの平均と分散に一致させるa novel adaptive instance normalization (AdaIN)を適用し、事前に定義したスタイルに制限されることなく、最適化の速度向上を達成した。

∇Sim: DIFFERENTIABLE SIMULATION FOR SYSTEM IDENTIFICATION AND VISUOMOTOR CONTROL

https://openreview.net/pdf?id=c_E8kFWfhp0

Ground truthの動画と物理エンジンからレンダリングして生成した推論動画の差分を下げることで物理エンジンでの摩擦係数などのパラメータを推定する

Half-Inverse Gradients for Physical Deep Learning

Problem

In learning, the loss is addressed using gradient-based optimizers with data-based normalizing schemes, such as Adam. While in physics, the optimizers of choice are higher-order techniques, such as Newton's method, which inherently make use of the inversion process.
However, Holl et al. (2021) found that these approaches can not effectively handle the joint optimization of networks and physics. Gradient-descent-based optimizers suffer from vanishing or exploding gradients, preventing effective convergence, while higher-order methods do not generally scale to the high-dimensional parameter spaces required by deep learning (Goodfellow et al., 2016).

Solution

They propose a new method for joint physics and networks called half-inverse gradients.
This method is based on a half inversion of the Jacobian and combines principles of both classical network and
physics optimizers.

Effect

A property of the method is that its runtime scales linearly with the number of network parameters.
It yields significant improvements in terms of convergence speed and final loss values over existing methods.

https://arxiv.org/abs/2203.10131

The Perception-Distortion Tradeoff

https://arxiv.org/abs/1711.06077

Distortionは元画像と生成画像との相違性(画素ごとの差分など)を表し、一方Perceptual qualityはそれらを視覚のみで評価する。
DistortionとPerceptual qualityをどちらも改善することは、アルゴリズムによらず不可能であることを証明

NewtonianVAE: Proportional Control and Goal Identification from Pixels via Physical Latent Spaces

https://arxiv.org/abs/2006.01959

To prune, or not to prune: exploring the efficacy of pruning for model compression

https://arxiv.org/abs/1710.01878

DATA-DRIVEN DEEP LEARNING OF PARTIAL DIFFERENTIAL EQUATIONS IN MODAL SPACE

https://arxiv.org/abs/1910.06948

Group Normalization

https://arxiv.org/abs/1803.08494

Sparse Training via Boosting Pruning Plasticity with Neuroregeneration

https://arxiv.org/abs/2106.10404

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

https://arxiv.org/abs/2006.16236

Unsupervised Learning of Lagrangian Dynamics from Images for Prediction and Control

https://arxiv.org/abs/2007.01926

Efficient Attention: Attention with Linear Complexities

https://arxiv.org/abs/1812.01243

Vision GNN: An Image is Worth Graph of Nodes

https://arxiv.org/abs/2206.00272

HIVIT: A SIMPLER AND MORE EFFICIENT DESIGN OF HIERARCHICAL VISION TRANSFORMER

https://openreview.net/pdf?id=3F6I-0-57SC

High-Resolution Photorealistic Image Translation in Real-Time: A Laplacian Pyramid Translation Network

https://arxiv.org/abs/2105.09188

End-to-End Differentiable Physics for Learning and Control

Problem

Physical simulation environments, such as MuJoCo, are poorly suited for deep learning settings.
The environments have some speed and numerical stability issues because those are not natively differentiable. So gradients (e.g., policy gradients for control tasks) must be evaluated via finite differencing.
A recent work developed a differentiable physical simulator, which was accomplished by an automatic differentiation framework.
But it only supported balls as objects, with limited extensibility.

Solution

This paper proposes and presents a differentiable two-dimensional physics simulator that addresses the main limitations of past work.
Specifically, Their system simulates rigid body dynamics via a linear complementarity problem(LCP) which computes the equations of motion subject to contact and friction constraints. It can use general simulation methods for determining the non-differentiable parts of the dynamics(namely, the presence of absence of collisions between convex shapes), while still providing a simulation environment that is end-to-end differentiable(given the observed set of collisions).

Effect

They can embed an entire physical simulation environment as a "layer" in a deep network, enabling agents to both learn the parameters of the environments to match observed behavior and improve control performance via traditional gradient-based learning.

https://papers.nips.cc/paper/2018/hash/842424a1d0595b76ec4fa03c46e8d755-Abstract.html

Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution

https://export.arxiv.org/abs/1904.05049

Interpreting Super-Resolution Networks with Local Attribution Maps

https://openaccess.thecvf.com/content/CVPR2021/papers/Gu_Interpreting_Super-Resolution_Networks_With_Local_Attribution_Maps_CVPR_2021_paper.pdf

超解像画像生成時に影響を与えたピクセルを可視化するlocal attribution map(LAM)を提案した。
LAMにより、4つのことを明らかにした。
1)広い範囲の入力画素を利用しているほど、精度が向上する(つまり、ネットワークを深く、広くすることで精度が向上)
2)Attention networkとnon-local networkにより広い範囲から特徴抽出を行うことが可能になっている
3)超解像画像の精度に影響する入力範囲よりもネットワークの受容野のほうがはるかに大きい
4)規則的なストライプやグリッドが注目される一方、複雑な模様は利用されにくい

RE-PARAMETERIZING YOUR OPTIMIZERS RATHER THAN ARCHITECTURES

https://arxiv.org/abs/2205.15242

Implicit Neural Representations with Periodic Activation Functions

paper
https://arxiv.org/abs/2006.09661

official web page
https://vsitzmann.github.io/siren/

解説blog
https://medium.com%[email protected]/@sallyrobotics.blog/sirens-implicit-neural-representations-with-periodic-activation-functions-f425c7f710fa

activationによく使われるReLUは２回微分が０になるので、高次元の微分を含んだモデル(例えば、Eikonal equationsやPoisson equation)に使うことができない。
このような高次元のcontinuous implicit neural representationにおいてactivationに三角関数のsinを適用することを提案している。

初期値のとり方についても言及している。

Differentiable Molecular Simulations for Control and Learning

One wishes to control the Hamiltonian to achieve desired simulation outcomes and structures.
We demonstrate how this can be achieved using differentiable simulations where bulk target observables and simulation outcomes can be analytically differentiated with respect to Hamiltonians, opening up new routes for parameterizing Hamiltonians to infer macroscopic models and develop control protocols.

https://arxiv.org/abs/2003.00868

DINO: DETR WITH IMPROVED DENOISING ANCHOR BOXES FOR END-TO-END OBJECT DETECTION

https://arxiv.org/abs/2203.03605

Lagrangian Neural Networks

https://arxiv.org/abs/2003.04630

Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation

https://arxiv.org/abs/2206.02777

An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution

https://arxiv.org/abs/1807.03247