ipfs-shipyard/dagger

🗡️ - an elegant slicer for a more civilized age

dagger's Issues

A set of READMEs for ordinary users

Aside from the extensive help menu already available, the repo needs a relatively large writeup explaining motivation / scoping / present state-of-the-art / likely exploration directions.

This is a blocker for public release, especially given the perceived simplicity of the problem domain.

Re-write repository history

The current repo was deliberately left "unstructured"

• it contains a rather large test dataset that needs to be elided
• mechanism of adding new chunking algorithms is best demonstrated by real commits adding the go-ipfs bundled algorithms ( such commits will be linked in the dev readme )

Determine source of stream corruption

After the rewrite of the STDIN ingestion mechanism, a bug in the core interface of the "chunker chain" got exposed. The manifestation is a 1-in-20 mismatch in expected CIDs of the test data.

The usual tools ( go -race etc ) and even print-debugging are not available, as ( what I think is ) the race condition is so tight that any slowdown introduced by debugging tools causes the bug to no longer manifest itself 🤦

This is the only current "known unknown".

License + README

Enable basic chunking chains with a leading "sparse detecting" chunker

A proof of concept for this functionality was demoed in February, but subsequently was removed as it was very tightly coupled with the ( solely available at the time ) fixedsize chunker.
Now that the chunking interface is stabilized ( pending final bugs ) the sparse chunker can be easily reimplemented in terms of that interface.

This is a blocker for public release, being a cheap way to demonstrate the outsize effect of generic-structure-aware chunking.

nul is a terrible name

My version of go (go1.14.5 darwin/amd64) is unhappy with a pathname with nul in it:

package github.com/ipfs-shipyard/DAGger/internal/dagger/collector/nul: github.com/ipfs-shipyard/DAGger/internal/dagger/collector/nul: invalid import path: malformed import path "github.com/ipfs-shipyard/DAGger/internal/dagger/collector/nul": disallowed path element "nul"