lucafulchir/libraptorq

RaptorQ RFC6330 C++11 implementation

RaptorQ

Homepage

old release: v0.1.10

Note: v0.1.10 is not compatible with prefivous versions, which were not RFC compliant!

current release: v1.0.0-rc2 (use this in new projects)

RaptorQ is a Forward Error Correction algorithm designed to deliver your data efficiently and without retransmissions for lost packets.

After sending K packets of your data as-is, RaptorQ generates as many repair symbols as you need. Once the receiver has at least K symbols, be it the source symbols, repair symbols or any combination of the two, it can reconstruct the whole input it was meant to receive.

This is called a Fountain code, and RaptorQ is the latest and most efficient code in this category.

libRaptorQ implements RFC6330, which specifies the RaptorQ algorithm.
Since the RFC is really complex, there is also a simpler (and maybe slightly faster) RAW API that you can use. The RFC API is not recommended due to the RFC complexity.

libRaptorQ is a header-only library written in C++11, and uses eigen to handle matrix manipulation.
Although header-only, the library can be compiled to create shared and static libraries with both C and C++98 compatibility.

Currently it's only been tested under Linux, but should work well under *BSD and MacOSX, too.

Unfortunately Windows is not supported. support for is broken, so you can not compile it with Visual Studio 2015.

Features

• RFC6330 API (complex, not recommended)
• RAW API (simpler, recommended)
• multi language:
  • header only C++11
  • compiled, C
  • compiled, C++98
• cached precomputations (configurable)

Developers

See the CONTRIBUTING file.

The Source Code

Although things seems to work, no stable release has been released yet.

This means you can only check this out with git.

to check out the repository:

$ git clone https://github.com/LucaFulchir/libRaptorQ.git

you can also get it from our main server:

$ git clone https://www.fenrirproject.org/Luker/libRaptorQ.git

GPG source check

Once you have cloned it, it's always a good thing to check the repository gpg signatures, so you can import my key with:

new long term key: 7393 DAD2 55BE B575 1DBD A04A B11C D823 BA27 8C85
$ gpg --keyserver pgp.mit.edu --recv-key 7393DAD255BEB5751DBDA04AB11CD823BA278C85
2016 key:
$ gpg --keyserver pgp.mit.edu --recv-key F61F6137
2015 key:
$ gpg --keyserver pgp.mit.edu --recv-key D42DDF0A

please check the full fingerprint.

Now you have the source, and the key, it's enough to check the signature of the last commit:

$ git log -n 1 --show-signature

As long as you got the right key, and you find the "gpg: Goog signature", you can be sure you have the right code.

Install

Dependencies

libRaptorQ depends from Eigen3 and LZ4
In case your system does not have Eigen3, Eigenv3.2.8 is included in the source files, so you do not need it.
LZ4 is included as a git submodule, so if you do not have it, run:
git submodule init
git submodule update
To get the library, which will be statically linked and NOT INSTALLED on your system.

Building

The build system uses CMake. So enter the source directory and we'll create a directory and build everything there:

$ mkdir build
$ cd build

$ cmake -DCMAKE_BUILD_TYPE=Release ../

$ make -j 4

Optional targets are available:
$ make tests examples docs
where:

• tests: benchmarks, rfc tests.
• examples: C/C++ examples
• docs: LATEX documentation.

or simply:
$ make -j 4 everything

finally, install everything
$ make install

libRaptorQ uses deterministic (reproducible) builds, so if you compile it twice, or on two different computers (but with the same compiler), the hash of the resulting libraries will be the same.

There are combinations of compiler and LTO/Profiling that break deterministic builds, so check the cmake warnings.

You can customize the CMake build with the following variables:

PROFILING   ON/OFF: Default:ON. Activate or deactivate profiling.
                Profiling compiles everything, then runs a test to see
                which code paths are more used. Then it recompiles everything
                but optimizing for for those code paths.
                Only for gcc/clang.
LTO         ON/OFF: Default:ON. Activate or deactivate Link time Optimization
                Makes the library smaller and better optimized.
                Only for gcc/clang.
CLANG_STDLIB ON/OFF: Default:OFF. use clang's libc++
                Note: only clang can use its standard library
USE_LZ4     ON/OFF: use the lz4 compression for caching precomputations.
                Default: ON
CLI         ON/OFF Build Command Line Interface tools.
CMAKE_C_COMPILER    gcc, clang...
CMAKE_CXX_COMPILER  choose between g++ or clang++.
RQ_LINKER   gold/ld/bsd Choose your linker. Default:autodetect.
CMAKE_BUILD_TYPE    Debug,MinSizeRel,Release,RelWithDebInfo
CMAKE_INSTALL_PREFIX Default: /usr/local

Using libRaptorQ

For the C++11, header-only version, you can include:

• "RaptorQ/RaptorQ_v1_hdr.hpp"
• "RaptorQ/RFC6330_v1_hdr.hpp"

For the linked, C+98/C++11 API:

• "RaptorQ/RaptorQ_v1.hpp"
• "RaptorQ/RFC6330_v1.hpp"

For the linked, C API:

• "RaptorQ/RaptorQ.h"
• "RaptorQ/RFC6330.h"

The C++ api is completely in sync between the linked and header-only version, so you can switch between the two just by changing the included header file.

You can compile a PDF of the documentation by doing:
$ make docs

Or you can simply visit the wiki for the full up-to-date documentation.

libraptorq's Issues

Installation difficulty on mac

I am having trouble compiling this on mac. I checked out release 1.9, and using the default c and cpp compilers that come built in with the mac developer tools, I ran into this error:

clang: error: invalid linker name in argument '-fuse-ld=gold'

I thought I might try using a different compiler, so I installed the llvm toolset. By the way, I have not spent much time with c or c++ so I am not too familiar with any of this software

Now strangely the error is a new one:

[ 32%] Linking CXX static library libRaptorQ_Static_Profiling.a
Error running link command: No such file or directory

I have not deviated much from the installation instructions, except fiddling with a few environment variables. I guess this has to do with the different build environment on a mac?

bug

int CRaptorQ::rawEnCode(const QByteArray &srcData,const float loss,quint16 &blocks,QByteArray &dst){
int ret=-1;
int dsize=srcData.length();
if(dsize>0){
//clear();
std::vector<uint8_t> src(static_cast<size_t>(dsize),0);
memcpy(src.data(),srcData.constData(),static_cast<size_t>(dsize));
//计算符号个数
size_t symbols=static_cast<size_t>(dsize);
//设置块数
blocks = static_cast<uint16_t>(RaptorQ::Block_Size::Block_10);
for (auto blk : RaptorQ::blocks) {
if (static_cast<uint16_t> (blk) >= symbols) {
blocks = static_cast<uint16_t> (blk);
break;
}
}
RaptorQ::Encoder<uint8_t, uint8_t*> enc(static_castRaptorQ::Block_Size(blocks), 1);
if (enc.set_data (src.begin().base(), src.end().base()) != symbols) {
qDebug()<<"Could not give data to the encoder ";
return ret;
}
enc.compute();
int overhead=getOverhead(loss);
std::uniform_real_distribution drop (0.0, 100.0);//设定可能丢包率
int rq = overhead,deled=0,repair=0;
auto rnd=getRandom();
std::vector<std::vector<uint8_t>> encodes;
for (auto sym_it = enc.begin_source(); sym_it != enc.end_source();++sym_it) {
float dropped = drop (rnd);
if (dropped <= loss) {
++rq;
++deled;
}
//char symbol[symbolsize]={0};
std::vector<uint8_t> symbol(1,0);
auto it = symbol.begin().base();
(sym_it) (it,symbol.end().base());
encodes.emplace_back(std::move(symbol));
//dst.append(reinterpret_cast<const char>(symbol.data()),symbolsize);
}

    auto sym_it = enc.begin_repair();
    for (; rq > 0 && sym_it != enc.end_repair (enc.max_repair());++sym_it)
    {
        float dropped = drop (rnd);
        if (dropped <= loss) {
            continue;
        }
        --rq;
        ++repair;
        std::vector<uint8_t> symbol(1,0);
        auto it = symbol.begin().base();
        (*sym_it) (it,symbol.end().base());
        encodes.emplace_back(std::move(symbol));
        //dst.append(reinterpret_cast<const char*>(symbol.data()),symbolsize);
    }
    if (sym_it == enc.end_repair (enc.max_repair())) {
        qDebug() << "Maybe losing " << loss << "% is too much?\n";
        return -2;
    }
    uint count=encodes.size();
    ret=static_cast<int>(count);
    dst.resize(ret);
    for(uint i=0;i<count;i++){
        memcpy(dst.data()+i,encodes[i].data(),1);
    }
    //endata=encodes;
}
return  ret;

}

if srcData length>220 bytes,(*sym_it) (it,symbol.end().base()) this code will excute long time,why?

Debug builds manually link -lubsan in but libubsan is unavailable on macOS

CMakeLists manually link ${RQ_UBSAN} in, which is defined to be -lubsan if the compiler supports ubsan. However, the documented way to enable ubsan at link time is not to pass the support library manually, but to specify the same -fsanitize=… flags to the C++ compiler used as the linker (such as c++ or clang++). The manual -lubsan inclusion is particularly problematic on macOS, where the support library is not available as libubsan.dylib but somewhere else not included in LDPATH:

quelthalas 11:07:58 libRaptorQ (master) $ 72 find /Applications/Xcode.app/Contents/Developer -name '*ubsan*'
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/lib/darwin/libclang_rt.ubsan_tvos_dynamic.dylib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/lib/darwin/libclang_rt.ubsan_watchossim.a
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/lib/darwin/libclang_rt.ubsan_osx.a
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/lib/darwin/libclang_rt.ubsan_ios_dynamic.dylib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/lib/darwin/libclang_rt.ubsan_osx_dynamic.dylib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/lib/darwin/libclang_rt.ubsan_tvossim_dynamic.dylib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/lib/darwin/libclang_rt.ubsan_watchos.a
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/lib/darwin/libclang_rt.ubsan_tvos.a
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/lib/darwin/libclang_rt.ubsan_iossim_dynamic.dylib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/lib/darwin/libclang_rt.ubsan_iossim.a
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/lib/darwin/libclang_rt.ubsan_tvossim.a
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/lib/darwin/libclang_rt.ubsan_watchos_dynamic.dylib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/lib/darwin/libclang_rt.ubsan_watchossim_dynamic.dylib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/lib/darwin/libclang_rt.ubsan_ios.a

how can we verify that your decoding complexity is correct?

Thank you for sharing us a good implementation. As I read the paper, it said that RaptorQ using Interactivation decoding which has lower decoding complexity than Gaussian Elimination. However, I did not find any formula which estimates the complexity of Interactivation decoding. When you implement it, how you can verify that your decoding complexity is correct? I read your document in Complexity section but it does not provide any formula for complexity of Interactivation decoding. Thank you so much

Regarding performance bottlenecks on libraptorq

Hi Luca,

We’re currently integrating libraptorq to one of our internal data transfer infrastructure project. Since data is delivering through a lossy network, we’re aiming to use RaptorQ to reduce re-transmission overhead and make the whole system more scalable. The integration is rather straightforward, we forked libRaptorQ v1.0.0-prealpha to our project and integrate it through RFC interfaces. However, we hit CPU bound when we try to enlarge the block size and symbol size (will give encode parameters in details). The maximum bit rate we can reach when transmitting the encoded symbols in a streaming manner is only about 5 Mbps on c5d.2xlarge AWS instance (3.0 GHz Intel Xeon Platinum; each core at up to 3.5 GHz using Intel Turbo Boost Technology), and it occupies 100% CPU cycle doing all of the Eigen3 compute work. So obviously CPU is a big bottleneck. However, we want to have the block size and symbol size as long as possible since the network characteristic is that packet loss is a rare event (~<1%); however, the disruption may last for long time (in the degree of 10+ minutes). We’ve notice that the whole library is designed as asynchronous when performing encoding jobs and there’s also a threadpool being designed (we assumed this is for compute acceleration). We’re wondering how can we get the benefit of parallel computing in order to solve the performance bottlenecks by utilizing more CPU cores on encoding? It seems not quite clear to me on how the whole threadpool and asynchronous computing design works on the documentation and sample codes regarding to C integration. (test_c.c) We have to use C interface (RFC6330_v1) since most of our protocol implementation is in C. Also we’re not quite sure how sub-blocks encoding works in your implementations, can you also share some lights on this?

Cheers,
Andy

VC++ still not supported?

While I realize your statement about VC++ not being a C++11 compiler used to be true, I'm not sure that's the case anymore.

https://msdn.microsoft.com/en-us/library/hh567368.aspx

Although I'm having some trouble getting it to work it does seem like it's in the ballpark. Any plans to give it a whirl?

Calling clear_data() in Encoder and reusing Encoder object for another

Sorry I couldn't add a bug report to gitlab when I tried I got this odd message Signing in using your GitLab.com account without a pre-existing GitLab account is not allowed. so I'll have to add this bug report to github.

When reusing the encoder class (C++11 header version of you code) git commit ae0c2fd for another input of data after calling clear_data() that causes an assertion in eigen3 when calling compute_sync() for the second time. I've tested both on windows sys2 64bit and linux 64bit x86 pcs and the same result. On windows I get this...

on linux I get...

QRaptorQ::unencodedInput:
Bytes in: 34 symbols: 10 symbol size: 4
encoding time 0 ms
number of packets created = 40 each of size 6
encoding loading time 0 ms
"000AB99DF63C"
"000B195CD1DE"
"000C70EDE005"
"000D0E9D4C24"
"000E117BBBA2"
"000F4A9FC138"
"001089305A2C"
"0011EDD26B77"
QRaptorQ::unencodedInput:
Bytes in: 34 symbols: 10 symbol size: 4
untitled4: /usr/include/eigen3/Eigen/src/Core/Block.h:146: Eigen::Block<XprType, BlockRows, BlockCols, InnerPanel>::Block(XprType&, Eigen::Index, Eigen::Index, Eigen::Index, Eigen::Index) [with XprType = Eigen::Matrix<RaptorQ__v1::Impl::Octet, -1, -1, 1>; int BlockRows = -1; int BlockCols = -1; bool InnerPanel = false; Eigen::Index = long int]: Assertion `startRow >= 0 && blockRows >= 0 && startRow <= xpr.rows() - blockRows && startCol >= 0 && blockCols >= 0 && startCol <= xpr.cols() - blockCols' failed.
19:10:35: The program has unexpectedly finished.

The code snippet looks like this...

#include "RaptorQ/RaptorQ_v1_hdr.hpp"

using QRaptorQ_iterator = typename QByteArray::iterator;
using Encoder_type = RaptorQ::Impl::Encoder<QRaptorQ_iterator,QRaptorQ_iterator>;
using Decoder_type = RaptorQ::Impl::Decoder<QRaptorQ_iterator,QRaptorQ_iterator>;
using QRaptorQ_symbol_iterator = RaptorQ::It::Encoder::Symbol_Iterator<QRaptorQ_iterator,QRaptorQ_iterator>;

void QRaptorQ::unencodedInput(QByteArray &unencoded_input)
{
    qDebug()<<"QRaptorQ::unencodedInput:";

    if(encoder_future.valid())
    {
        qDebug()<<"dropping input data as we are waiting for encoder to finish";
        return;
    }

    //calc symbol size to fit in number of wanted blocks
    symbolSize=unencoded_input.size()/((int)blockSize);
    if((symbolSize*((int)blockSize))!=unencoded_input.size())symbolSize++;

    QElapsedTimer timer;
    timer.start();

    //re create encoder if block size is different
    if((encoder->symbols()!=(uint16_t)blockSize)||((int)encoder->symbol_size()!=symbolSize))
    {
        delete encoder;
        encoder=new Encoder_type(blockSize,symbolSize);
        qDebug()<<"creation time for encoder"<< timer.elapsed()<<"ms";
    }

    //give the input to the encoder. the encoder answers with the size of what it can use
    if((int)encoder->set_data(unencoded_input.begin(),unencoded_input.end())!=unencoded_input.size())
    {
        qDebug()<<"Could not give data to the encoder";
        return;
    }

    qDebug()<<"Bytes in:"<<unencoded_input.size()<<"symbols:"<<encoder->symbols()<<"symbol size:"<<encoder->symbol_size();

    encoder_future=std::async(std::launch::async,[this]
    {
        QElapsedTimer timer;
        timer.start();

        // RQ needs to do its magic on the input before you can ask the symbols.
        // multiple ways to do this are available.
        // The simplest is to run the computation and block everything until
        // the work has been done. Not a problem for small sizes (<200),
        // but big sizes will take **a lot** of time, consider running this with the
        // asynchronous calls
        if (!encoder->compute_sync())
        {
            // if this happens it's a bug in the library.
            // the **Decoder** can fail, but the **Encoder** can never fail.
            QMetaObject::invokeMethod(this,[this]()
            {
                qDebug()<<"encoder-RaptorQ failure! really bad!";
                if(encoder_future.valid())encoder_future.get();
            },Qt::QueuedConnection);
            return;
        }

        qDebug()<<"encoding time"<< timer.elapsed()<<"ms";
        timer.start();

        encodedOutputBytes.clear();
        QByteArray packet;
        packet.resize(symbolSize+2);

        uint32_t wanted_repairs=encoder->symbols()*4;
        uint32_t max_repair = encoder->max_repair();
        if(wanted_repairs>max_repair)wanted_repairs=max_repair;
        if(wanted_repairs>0xFFFF)wanted_repairs=0xFFFF;
        QRaptorQ_symbol_iterator encoder_symbol_iterator = encoder->begin_repair();
        for(;encoder_symbol_iterator!=encoder->end_repair(wanted_repairs);++encoder_symbol_iterator)
        {
            //get id
            uint32_t repair_id=(*encoder_symbol_iterator).id();
            packet[0]=(uint8_t)((repair_id>>8)&0xFF);
            packet[1]=(uint8_t)(repair_id&0xFF);
            //get repair symbol
            auto it = packet.begin()+2;
            auto written = (*encoder_symbol_iterator)(it, packet.end());
            if ((int)written != symbolSize)
            {
                QMetaObject::invokeMethod(this,[this,written]()
                {
                    qDebug()<<written<<"-vs-"<<symbolSize<<"Could not get the whole repair symbol!";
                    if(encoder_future.valid())encoder_future.get();
                },Qt::QueuedConnection);
                return;
            }
            encodedOutputBytes.push_back(packet);
        }

        qDebug()<<"number of packets created ="<<encodedOutputBytes.size()<<"each of size"<<packet.size();
        qDebug()<<"encoding loading time"<< timer.elapsed()<<"ms";

        //this will cause a assert then next time we
        //load and call encoder->compute_sync()
        encoder->clear_data();

        QMetaObject::invokeMethod(this,[this]()
        {
            //tell the user they can get packets now
            emit encodedOutputReady();
            if(encoder_future.valid())encoder_future.get();
        },Qt::QueuedConnection);

    });

}

Without calling clear_data() then I don't get the assertion but of cause I just get more packets of the first message not the second one as wanted. So far the only way I can get around this is by deleting the the encoder object every time I want to make a new message encoding.

I'm also having similar problems with the decoder but haven't looked closely enough into it yet. So I'll keep this bug report just for the encoder.

Cheers,
Jonti

build failed on macOS 12.0.1

8 warnings and 1 error generated.
make[2]: *** [CMakeFiles/RaptorQ.dir/src/RaptorQ/v1/wrapper/C_RFC_API.cpp.o] Error 1
1 error generated.
make[2]: *** [CMakeFiles/RaptorQ.dir/src/RaptorQ/v1/wrapper/CPP_RAW_API_void.cpp.o] Error 1
make[1]: *** [CMakeFiles/RaptorQ.dir/all] Error 2
make: *** [all] Error 2

---

In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/include/stdio.h:342:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/include/sys/_types/_ssize_t.h:31:33: error: typedef redefinition with different types ('__darwin_ssize_t' (aka 'long') vs 'std::conditional<sizeof(size_t) == 4, int32_t, int64_t>::type' (aka 'long long'))

In file included from /Users/devharsh/Documents/libRaptorQ/src/RaptorQ/v1/wrapper/C_RAW_API.h:29:
/Users/devharsh/Documents/libRaptorQ/src/RaptorQ/v1/common.hpp:100:71: error: typedef redefinition with different types ('std::conditional<sizeof(size_t) == 4, int32_t, int64_t>::type' (aka 'long long') vs '__darwin_ssize_t' (aka 'long'))

In file included from /Users/devharsh/Documents/libRaptorQ/src/RaptorQ/v1/wrapper/CPP_RAW_API_void.hpp:24:
/Users/devharsh/Documents/libRaptorQ/src/RaptorQ/v1/common.hpp:100:71: error: typedef redefinition with different types ('std::conditional<sizeof(size_t) == 4, int32_t, int64_t>::type' (aka 'long long') vs '__darwin_ssize_t' (aka 'long'))

probability of decoding success

Question 1:
I was bothered by zero return value from RaptorQ_decode().
Any tips?
or what is the possible reason that cause this problem?
The number of symbols seems enough.

Question 2:
what is the probability of decoding success for now?
The RaptorQ technical overview has mentioned that 99.9999% from reception of K+2 encoded symbols.

Thx :)

Manual redefinition of ssize_t fails on macOS

On macOS where both long and long long are 64 bits wide,

• common.hpp chooses int64_t as ssize_t;
• int64_t is defined as long long;
• ssize_t is already defined as long

So the typedef is non-idempotent, and gives a compilation error.

Getting started

Hi is there any simple program to get start with the library

Symbol_Wrap write to const error

As a heads up, it looks like the Symbol_Wrap class has the wrong const-ness for it's _raw attribute.
In the Interleaver.hpp file.

The hunt began when I saw the following compilation error:

libRaptorQ/src/RaptorQ/v1/Interleaver.hpp:126:32: error: assignment of read-only location ‘static_cast<const uint8_t&>(((RFC6330__v1::Impl::Symbol_Wrap<T>*)this)->RFC6330__v1::Impl::Symbol_Wrap<T>::_raw[i])’
             _raw[i] ^= a._raw[i];
                                ^

Since the class seems to only modify the uint8_t values inside the array and not the pointer value itself, the _raw attribute could be uint8_t *const _raw = nullptr;.

Proposed changes:

• Change line 156 to uint8_t *const _raw = nullptr;
• Change the constructor args on line 105 to Symbol_Wrap (uint8_t *raw, const uint16_t size) : _raw (raw),

There may be other issues with this class(like setting equal different sized Symbol_Wraps), but those don't hold back the compilation process.

Best,

Craig

Is there a way to reuse a decoder for different blocks in the RAW C API?

I am learning how to use this library using the RAW C API provided (I am trying to wrap it for a higher-level language). It works well, and I have found that I can reuse an encoder with many different blocks to save some time, using clear_data. However I can't seem to do the same with the decoder; this is what happens:

• create decoder
• I call add_symbol many times on it with symbols from the first block
• when I have enough symbols, I call end_of_input followed by wait_sync
• then I call decode_bytes to read the decoded block
• then I call clear_data on the decoder
• I call add_symbol many times on it with symbols from the second block
• I call end_of_input
• and here, when I call wait_sync it just returns NEED_DATA even if I have enough symbols (and can_decode returns true)

Is it possible to reuse a decoder for multiple blocks or do I have to create a new one each time (that seems wasteful).

Thanks

Compile with NDK

I am not familiar with CMake.
So, are there any tips to compile libRaptorQ with android NDK?

Thx : )

cannot compile on mac

i have compiled the lib with make and am using this command to compile the test_cpp_raw
g++ test_cpp_raw.cpp -I../src -I../external/eigen3 -std=c++11 -o test_cpp_raw -L../lib -lRaptorQ
but it keeps giving me error
Undefined symbols for architecture x86_64: "RaptorQ__v1::Impl::Operation::Block::~Block()", referenced from: RaptorQ__v1::Impl::Operation::Operation(RaptorQ__v1::Impl::Operation::_t, Eigen::Matrix<RaptorQ__v1::Impl::Octet, -1, -1, 1, -1, -1> const&) in test_cpp_raw-672800.o "RaptorQ__v1::Impl::Operation::Reorder::~Reorder()", referenced from: RaptorQ__v1::Impl::Operation::Operation(RaptorQ__v1::Impl::Operation::_t, std::__1::vector<unsigned short, std::__1::allocator<unsigned short> > const&) in test_cpp_raw-672800.o ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation)
but this command works on my ubuntu
i wonder what's wrong on my mac

what's the meaning of alpha ^^ (i-j)?

about GAMMA, libRaptorQ is different than other implementations(like OpenRQ, raptorQ, velopyraptorQ),

in function make_GAMMA() of libRaptorQ:
GAMMA (row, col) = RaptorQ__v1::Impl::oct_exp[(row - col) % RaptorQ__v1::Impl::oct_exp.size()];
RaptorQ__v1::Impl::oct_exp.size() equals 510.

in the other implementations, they looks like this:
GAMMA (row, col) = RaptorQ__v1::Impl::oct_exp[(row - col) % 255];

the RFC document writes "GAMMA[i,j] = alpha ^^ (i-j)", actually I don't quite undstand it,
is the difference meaning?

[Windows] Thread_Pool is locked in destructor while waiting for finishing of pending _exiting threads

The issue happens only on Windows (MSYS2 MinGW toolchain is used) and so far can't be reproduced with gcc on Linux or clang on MacOS.

For some reason a worker thread is not get unlocked by notify_all after a worker thread was moved to _exiting list of threads during call of resize_pool (0, RaptorQ__v1::Work_State::ABORT_COMPUTATION); inside ~Thread_Pool().

cmake build failed.

cmake failed.

[root@localhost build]# cmake ..
-- libRaptorQ version 0.2-prealpha
-- Build selected: Release
-- Link Time Optimization activated
-- Profiling activated
-- Found git in /usr/bin/git
-- Found eigen3 in /root/libRaptorQ-master/external/eigen3
-- Could NOT find LATEX (missing: LATEX_COMPILER)
-- I could not find the latex command.
-- I could not find the pdflatex command.
-- I could not find the bibtex command.
-- I could not find the makeindex command.
-- I could not find the dvips command.
-- I could not find the ps2pdf command.
-- I could not find the pdftops command.
-- I could not find the latex2html command.
CMake Warning at CMakeLists.txt:134 (message):
Not inside a git repository. Compiler seed is now constant.

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
RQ_LZ4_INCLUDE_DIR (ADVANCED)
used as include directory in directory /root/libRaptorQ-master
used as include directory in directory /root/libRaptorQ-master
used as include directory in directory /root/libRaptorQ-master
used as include directory in directory /root/libRaptorQ-master
used as include directory in directory /root/libRaptorQ-master
used as include directory in directory /root/libRaptorQ-master
used as include directory in directory /root/libRaptorQ-master
used as include directory in directory /root/libRaptorQ-master
used as include directory in directory /root/libRaptorQ-master
used as include directory in directory /root/libRaptorQ-master
used as include directory in directory /root/libRaptorQ-master
used as include directory in directory /root/libRaptorQ-master
used as include directory in directory /root/libRaptorQ-master
used as include directory in directory /root/libRaptorQ-master
used as include directory in directory /root/libRaptorQ-master
used as include directory in directory /root/libRaptorQ-master
used as include directory in directory /root/libRaptorQ-master
used as include directory in directory /root/libRaptorQ-master
used as include directory in directory /root/libRaptorQ-master
used as include directory in directory /root/libRaptorQ-master
used as include directory in directory /root/libRaptorQ-master/doc
used as include directory in directory /root/libRaptorQ-master/doc
used as include directory in directory /root/libRaptorQ-master/doc
used as include directory in directory /root/libRaptorQ-master/doc
used as include directory in directory /root/libRaptorQ-master/doc
used as include directory in directory /root/libRaptorQ-master/doc
used as include directory in directory /root/libRaptorQ-master/doc

-- Configuring incomplete, errors occurred!
See also "/root/libRaptorQ-master/build/CMakeFiles/CMakeOutput.log".
See also "/root/libRaptorQ-master/build/CMakeFiles/CMakeError.log".

here is the tail of /root/libRaptorQ-master/build/CMakeFiles/CMakeError.log

include

int main(int argc, char** argv)
{
(void)argv;

ifndef pthread_create

return ((int*)(&pthread_create))[argc];

else

(void)argc;
return 0;

endif

}

Determining if the function pthread_create exists in the pthreads failed with the following output:
Change Dir: /root/libRaptorQ-master/build/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/gmake" "cmTC_a589a/fast"
/usr/bin/gmake -f CMakeFiles/cmTC_a589a.dir/build.make CMakeFiles/cmTC_a589a.dir/build
gmake[1]: Entering directory `/root/libRaptorQ-master/build/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_a589a.dir/CheckFunctionExists.c.o
/usr/bin/cc -DCHECK_FUNCTION_EXISTS=pthread_create -o CMakeFiles/cmTC_a589a.dir/CheckFunctionExists.c.o -c /usr/local/share/cmake-3.5/Modules/CheckFunctionExists.c
Linking C executable cmTC_a589a
/usr/local/bin/cmake -E cmake_link_script CMakeFiles/cmTC_a589a.dir/link.txt --verbose=1
/usr/bin/cc -DCHECK_FUNCTION_EXISTS=pthread_create CMakeFiles/cmTC_a589a.dir/CheckFunctionExists.c.o -o cmTC_a589a -rdynamic -lpthreads
/usr/bin/ld: cannot find -lpthreads
collect2: error: ld returned 1 exit status
gmake[1]: *** [cmTC_a589a] Error 1

my environment is centos 7.2 , cmake 3.5
i am very sure that pthread library is installed,
i install development tool kit with

yum groupinstall "Development Tools"