how to clean sequence based on both sequence name and sequences?

[liamxg]

how to clean sequence based on both sequence name and sequences?

[liamxg]

csvtk uniq -Ht -f 2,5 not csvtk uniq -Ht -f 2,4 ? As I also consider the sequence?

[shenwei356]

https://bioinf.shenwei.me/seqkit/usage/#sequence-id

$ seqkit rmdup --id-regexp '\|([^\|]+)$' seq.fa 
[INFO] 1 duplicated records removed
>ab1234|china|human|2020-10-01
AAACCCTTTTCCCCCAAACCCTTTTCCCCC
>ab1235|china|human|2020-10-03
AAACCCTTTTCCCCCAAACCCTTTTCCCCC
>ab1236|china|human|2020-10-04
AAACCCTTTTCCCCCAAACCCTTTTCCCCC
>ab1238|USA|animal|2020-10-05
AAACCCTTTTCCCCCAAACCCTTTTCCCCC

[liamxg]

why the below is removed, japan only just once?

>ab1237|japan|human|2020-10-03
AAACCCTTTTCCCCCAAACCCTTTTCCCCC
`` 

[shenwei356]

$ seqkit fx2tab seq.fa  \
    | sed 's/|/\t/g' \
    | csvtk uniq -Ht -f 2,4 \
    | awk '{print $1"|"$2"|"$3"|"$4"\t"$5}' \
    | seqkit tab2fx 

>ab1234|china|human|2020-10-01
AAACCCTTTTCCCCCAAACCCTTTTCCCCC
>ab1235|china|human|2020-10-03
AAACCCTTTTCCCCCAAACCCTTTTCCCCC
>ab1236|china|human|2020-10-04
AAACCCTTTTCCCCCAAACCCTTTTCCCCC
>ab1237|japan|human|2020-10-03
AAACCCTTTTCCCCCAAACCCTTTTCCCCC
>ab1238|USA|animal|2020-10-05
AAACCCTTTTCCCCCAAACCCTTTTCCCCC

[liamxg]

Dear @shenwei356,

please help me test this one:

>ab1233|china|human|2020-10-01
AAACCCTTTTCCCCCAAACCCTTTTCCCCT
>ab1234|china|human|2020-10-01
AAACCCTTTTCCCCCAAACCCTTTTCCCCC
>ab1235|china|human|2020-10-03
AAACCCTTTTCCCCCAAACCCTTTTCCCCC
>ab1236|china|human|2020-10-04
AAACCCTTTTCCCCCAAACCCTTTTCCCCC
>ab1237|japan|human|2020-10-03
AAACCCTTTTCCCCCAAACCCTTTTCCCCC
>ab1238|USA|animal|2020-10-05
AAACCCTTTTCCCCCAAACCCTTTTCCCCC
>ab1239|china|human|2020-10-03
AAACCCTTTTCCCCCAAACCCTTTTCCCCC

Thanks again.

[shenwei356]

please help me test this one:

Why don't you test it by yourself?

As I also consider the sequence?

If yes, just change the option values as needed.

Please explore by yourself. I'm not responsible for replying to all questions, especially those that could be resolved by reading documents/usage.

[liamxg]

Hi @shenwei356,
just tested, not work for my data.

csvtk uniq -Ht -f 2,5
ab1233	china	human	2020-10-01	AAACCCTTTTCCCCCAAACCCTTTTCCCCT	
ab1234	china	human	2020-10-01	AAACCCTTTTCCCCCAAACCCTTTTCCCCC	
ab1237	japan	human	2020-10-03	AAACCCTTTTCCCCCAAACCCTTTTCCCCC	
ab1238	USA	animal	2020-10-05	AAACCCTTTTCCCCCAAACCCTTTTCCCCC	

[liamxg]

Dear @shenwei356,
hope you can help me out, thanks.

[shenwei356]

Does this work? The previous -f 2,4 is changed to -f 2-4.

$ seqkit fx2tab seq.fa  \
    | sed 's/|/\t/g' \
    | csvtk uniq -Ht -f 2-4 \
    | awk '{print $1"|"$2"|"$3"|"$4"\t"$5}' \
    | seqkit tab2fx 

[liamxg]

Dear @shenwei356,

works for me, thanks.