amyxlu/cpcprot

Parameter-efficient embeddings for proteins, pretrained using a contrastive loss.

cpcprot's Issues

Error with sequences with less than 11 residues

When running the "API for Embedding Protein Sequences" example from the readme, sequences shorter than 11 residues fail.

E.g. seq = "SEQWENCEILV" works, while seq = "SEQWENCEIL" fails:

    c_mean = embedder.get_c_mean(input)   # (1, 512)
  File "/home/konsti/bio_embeddings/.venv/lib/python3.8/site-packages/CPCProt/model/heads.py", line 124, in get_c_mean
    c, mask = self.get_c(data, return_mask = True)
  File "/home/konsti/bio_embeddings/.venv/lib/python3.8/site-packages/CPCProt/model/heads.py", line 105, in get_c
    c = self.cpc.get_c(z)
  File "/home/konsti/bio_embeddings/.venv/lib/python3.8/site-packages/CPCProt/model/cpcprot.py", line 102, in get_c
    c = self.autoregressor(z[:,:max_L,:]).to(self.device)
  File "/home/konsti/bio_embeddings/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/konsti/bio_embeddings/.venv/lib/python3.8/site-packages/CPCProt/model/autoregressor.py", line 27, in forward
    output, regress_hidden_state = self.gru(input_seq, regress_hidden_state)
  File "/home/konsti/bio_embeddings/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/konsti/bio_embeddings/.venv/lib/python3.8/site-packages/torch/nn/modules/rnn.py", line 726, in forward
    result = _VF.gru(input, hx, self._flat_weights, self.bias, self.num_layers,
RuntimeError: stack expects a non-empty TensorList

A sequence with 9 residues or less fails with a different error:

    z_mean = embedder.get_z_mean(input)   # (1, 512)
  File "/home/konsti/bio_embeddings/.venv/lib/python3.8/site-packages/CPCProt/model/heads.py", line 113, in get_z_mean
    z, mask = self.get_z(data, return_mask=True)
  File "/home/konsti/bio_embeddings/.venv/lib/python3.8/site-packages/CPCProt/model/heads.py", line 91, in get_z
    z = self.cpc(x, return_early='z')
  File "/home/konsti/bio_embeddings/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/konsti/bio_embeddings/.venv/lib/python3.8/site-packages/CPCProt/model/cpcprot.py", line 115, in forward
    z = self.get_z(x)
  File "/home/konsti/bio_embeddings/.venv/lib/python3.8/site-packages/CPCProt/model/cpcprot.py", line 81, in get_z
    patch_ends = torch.arange(self.cfg.patch_len, x.shape[1]+1, self.cfg.patch_len, device=self.device)
RuntimeError: upper bound and larger bound inconsistent with step sign

No file found: CPCProt/evaluate.py

Model fine tuning

Hi! model looks really interesting. I'd like to do some fine tuning on local context, like what is suggested by fastai

Can you give me advice on how to do so with your model? The fine tuning example you have provided seems only to suggest adding a classifier head to the model.

Thanks

Need to insert a "with torch.no_grad():" in heads.py

Calling the .get_c() method from the embedder instance created in the example

embedder = CPCProtEmbedding(model)

ends up throwing a CUDA out of memory error, because it is tracking the gradients

def get_c(self, data, return_mask = False):
        z, mask = self.get_z(data, return_mask=True)
        if self.parallel:
            # workaround for accessing model attributes when DataParallel
            c = self.cpc(data, return_early='c')
        else:
            c = self.cpc.get_c(z)

wrapping the forward pass in a "with torch.no_grad():" worked for me

def get_c(self, data, return_mask = False):
        z, mask = self.get_z(data, return_mask=True)
        with torch.no_grad():
                if self.parallel:
                    # workaround for accessing model attributes when DataParallel
                    c = self.cpc(data, return_early='c')
                else:
                    c = self.cpc.get_c(z)