suchapalaver/krust

counts k-mers, written in rust

krust's Issues

Stdin input

Try moving `from_utf8` conversion to parallel stage - maybe by chaining another `map`

Explore using the bytes crate

bytes

The biggest feature it adds over Vec is shallow cloning. In other words, calling clone() on a Bytes instance does not copy the underlying data. Instead, a Bytes instance is a reference-counted handle to some underlying data. The Bytes type is roughly an Arc<Vec> but with some added capabilities.

Try using find to look for `b’N’

Docs

build kmer indexes

could be a great functionality to add to your tool in time, maybe make one that is loadable into whatever handles hash tables in Rust

speed up by changing the utf8 processing, reverse-comp, and storage

the utf8-processing of the kmers. The kmer iterator itself should really check it has valid kmers while iterating. Also, instead of storing the reverse-complement in heap-allocated strings, you can make a lazy reverse-complemented object. Alternatively, store the kmers in u64 - one of the reasons for using kmers in the first place is that they can be packed into machine integers for speed.

checkout sourmash

It would be nice to have a better error message when you call it from command-line with wrong arguments

something that explains which arguments to pass in. When the output directory exists, you just write "File exists", which is very confusing if you have an unrelated file called "output".

avoid panicking at all in your library code

If anyone wants to import your function they won't be happy with something that crashes the whole application when it fails. You can panic in the executable portion of the program though.

Try using a struct to construct kmers

change the Config struct member kmer_len to be a usize

Rather than do let kmer_len = config.kmer_len.parse::().unwrap();, I would instead change the Config struct member kmer_len to be a usize, and perform parsing while constructing Config - Config::new already returns Result.

needletail other then bio to accelerate fasta parsing

Hello Team,

It seems needle tail is much faster than bio for fasta file parsing. For larger fasta files, parsing can also be parallelized. Is this doable?

Thanks,

Jianshu

You don't need to use extern crate

use (which you are already doing) is a newer version of the same thing.

implement map reduce

I was looking for a fast kmer counter and came across this as well as your package:
https://pirl.unc.edu/blog/shaking-the-rust-off-python-redox

I guess there's also needletail which is older, on crates.io and hence more depended on, but yours uses rayon, so a better fit for the implementation at the link. Is that right?

This is also more recently maintained, just by you. 😄

Nice that you are still maintaining this starter project! I myself am starting to learn rust. It seems you have been hoping to implement hashmaps also. Let me know if you would like a contribution--please be warned, it is early days for me!

My use case is that I want to populate a large array with kmer counts from sequencing reads, likely from FASTQ files. Since that is supported by needletail I might start there.

WebAssembly

https://rustwasm.github.io/docs/book/reference/add-wasm-support-to-crate.html#adding-support-for-webassembly

Try using rustfmt to standardise your code style

speed up using hashmaps

writing a line per kmer is too inefficient and rarely needed. Much better to just return a vector of kmer hashmaps. Alternatively, make a hashmap containing n -> m pairs, where N is the number of time some kmer has been seen, and m the number of distinct kmers having been seen n times.

look at jellyfish again