mmcdermott/running_literature_review

A running summary of papers read & literature reviewed

Running Literature Review

Here I'll maintain a running record of literature I've read, topics I've learned about, etc., to help cement my own learnings and allow others to benefit from it (eventually--this is a stretch goal not yet realized). The list below is not remotely up to date in terms of papers I have read or want to read. As of now, this github is strictly for my own benefit, and all opinions are (obviously) my own and are recorded for solely my own posterity -- they aren't commentary on the works in any real sense, as that would require both (1) likely more thought, and (2) more careful communication to ensure my message is accurately conveyed. That all said, please feel free to leave any comments, suggestions, or thoughts via github issues.

TODOs:

1. Add papers currently notated to list below.
2. Finish structuring list below.
3. Evaluate format

Paper/Topic Queue

Maintenance is at a best effort level. Until decided otherwise, this won't be sorted by topic, the relevant sub-works will be nested.

To categorize

Shortlist (5 max, unordered)

Classic Papers

• Meta-list: https://github.com/beamlab-hsph/journalclub

Topic-Structured Papers

Deep Learning Theory

• Hopfield Networks
• Gradient Starvation
• Neural Function Modules
• Deep Evidential Regression

Domain Generalization

• In search of Lost Domain Generalization
• Measuring Robustness to Natural Distribution Shifts in Image Classification
• Wilds: A Benchmark of in-the-Wild Distribution Shifts

Representation Learning

Self-supervised Representation Learning Theory

• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• Understanding Self-supervised Learning with Dual Deep Networks
• Self-supervised Learning: Generative or Contrastive

General Self Supervised Representation Learning

• Automatic Shortcut Removal for Self-Supervised Representation Learning

Latent Graph Learning

• Meta-list: https://towardsdatascience.com/manifold-learning-2-99a25eeb677d

Pre-training

Language

• CERT: Contrastive Self-supervised Learning for Language Understanding

Clinical Time-series

• Temporal Phenotyping using Deep Predictive Clustering of Disease Progression

Graphs

• Graph-BERT
• Strategies for Pre-training Graph Neural Networks
• GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training
• Graph Information Bottleneck
• Self-Supervised Graph Transformer on Large-Scale Molecular Data

Ontologies

• STEAM: Self-Supervised Taxonomy Expansion with Mini-Paths

BERT + KG

• Enhancing Clinical BERT Embedding using a Biomedical Knowledge Base
• UmlsBERT
• Infusing Disease Knowledge into BERT for Health Question Answering

Contrastive Learning

• Towards Domain-Agnostic Contrastive Learning
• A Simple Framework for Contrastive Learning of Visual Representations

(Deep) Metric Learning:

• Deep Adversarial Metric Learning
• Hardness-Aware Deep Metric Learning
• Not All Samples Are Created Equal: Deep Learning with Importance Sampling
• Sampling Matters in Deep Embedding Learning
• A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses
• Hierarchical Triplet Loss
• The Group Loss for Deep Metric Learning
• Multi-similarity Loss with General Pair Weighting for Deep Metric Learning

Transfer Learning

• A Combinatorial Perspective on Transfer Learning

Few-shot Learning

• Meta-list: https://github.com/Duan-JM/awesome-papers-fewshot/blob/master/REMAIN_SORTED_PAPER.md

Multi-task Learning

• Which Tasks Should Be Learned Together in Multi-task Learning?
• Taskonomy: Disentangling Task Transfer Learning

Knowledge Graphs

• Knowledge Graph-based Question Answering with Electronic Health Records
• Graph embedding on biomedical networks: methods, applications and evaluations

Graph Neural Networks

• Meta Lists:
  • https://github.com/thunlp/GNNPapers
  • https://twitter.com/PetarV_93/status/1306689702020382720
• Provably More Powerful Neural Networks
• GraphSAINT
• ID-GNNs

Irregular Time-series

• A Survey on Principles, Models and Methods for Learning from Irregularly Sampled Time Series

Protein Modeling

• Meta-list: https://github.com/yangkky/Machine-learning-for-proteins
• Learning Protein Sequence Embeddings Using Information from Structure

Curiousity Papers

• Quantum Neural Networks
• Ideal Theory in AI Ethics
• Deep Learning for Scientific Discovery
• Software engineering for artificial intelligence and machine learning software
• Requirements Analysis for an Open Research Knowledge Graph

Tech Stack Reading / Tutorial Queue:

• PyTorch Lightning Bolts
• PyHealth

Archived Papers List

This list is for deprioritized papers/topics that may have been read.

Imputation

• Neural ODEs
  • https://arxiv.org/abs/2002.08071
  • https://papers.nips.cc/paper/8773-latent-ordinary-differential-equations-for-irregularly-sampled-time-series.pdf
  • http://papers.nips.cc/paper/8957-gru-ode-bayes-continuous-modeling-of-sporadically-observed-time-series.pdf
  • https://arxiv.org/pdf/2010.08304.pdf
• Deep Signatures
  • https://arxiv.org/abs/2005.12359
• General
  • https://link.springer.com/content/pdf/10.1007/s41666-020-00076-2.pdf
  • https://proceedings.icml.cc/static/paper_files/icml/2020/3129-Paper.pdf

Biomedical Modeling (e.g., task-oriented, rather than representation oriented)

• https://dl.acm.org/doi/pdf/10.1145/3394486.3403110
• Generative Probabilistic biological sequence models

Efficient Network Architectures

Transformers

• https://arxiv.org/pdf/2009.06732.pdf

Deep Learning Theory

• Lottery Ticket Hypothesis
  • https://arxiv.org/abs/2007.12223
  • https://arxiv.org/abs/1911.11134
  • https://ai.googleblog.com/2020/09/improving-sparse-training-with-rigl.html
• Deep Contextual Clinical Prediction with Reverse Distillation

Training under class imbalance

• https://arxiv.org/pdf/2009.02189v1.pdf
• https://arxiv.org/pdf/1903.01182.pdf

Clustering

• https://arxiv.org/pdf/2009.07417v1.pdf