Comments (3)
It seems that the root cause of the issue is that when the messagequeue sends a message, the message may contain several blocks, but only one "Sent" event is emitted.
go-graphsync/messagequeue/messagequeue.go
Lines 202 to 207 in 11d30c6
When this event bubbles up to the responsemanager subscriber, it only includes the block data for the first block (not all blocks in the message)
https://github.com/ipfs/go-graphsync/blob/master/responsemanager/subscriber.go#L42
from go-graphsync.
Hey, so this exposes a flaw in the mappable subscriber system, and provides a pretext for refactoring the interface.
from go-graphsync.
@acruikshank I am recommending a fix to the way MappableSubscriber currently works, as well as some interface changes that will make things more clear:
These are my recommended new interfaces -- please don't take these set in stone and please feel free to improve naming. (taken from: https://github.com/ipfs/go-graphsync/blob/master/notifications/types.go)
// Topic is a topic that events appear on
type Topic interface{}
// Event is a publishable event
type Event interface{}
// TopicData is data added to every message broadcast on a topic
type TopicData interface{}
// Subscriber is a subscriber that can receive events
type Subscriber interface {
OnNext(Topic, Event, ...TopicData)
OnClose(Topic, ...TopicData)
}
// TopicDataSubscriber is a subscriber that can inject data to into subscriber callbacks when events happen for a given topic
type TopicDataSubscriber interface {
Subscriber
AddTopicData(topic Topic, data TopicData)
}
// Subscribable is a stream that can be subscribed to
type Subscribable interface {
Subscribe(topic Topic, sub Subscriber) bool
Unsubscribe(sub Subscriber) bool
}
// Publisher is an publisher of events that can be subscribed to
type Publisher interface {
Close(Topic)
Publish(Topic, Event)
Shutdown()
Startup()
Subscribable
}
// EventTransform if a fucntion transforms one kind of event to another
type EventTransform func(Event) Event
// Notifee is a topic data subscriber plus a set of data you want to add to any topics subscribed to
// (used to call SubscribeWithData to inject data when events for a given topic emit)
type Notifee struct {
Data TopicData
Subscriber TopicDataSubscriber
}
// SubscribeWithData subscribes to the given subscriber on the given topic, and adds the notifiees
// custom data into the list of data injected into callbacks when events occur on that topic
func SubscribeWithData(p Subscribable, topic Topic, notifee Notifee) {
notifee.Subscriber.AddTopicData(topic, notifee.Data)
p.Subscribe(topic, notifee.Subscriber)
}
--
I also recommend:
- NewMappableSubscribe -> NewTopicDataSubscriber
- Lose the event transform parameter to NewTopicDataSubscriber -- we never use this other than to pass a function that does no transformation
- change the internal map from map[Topic]Topic to map[Topic][]TopicData -- note the fact that it's currently over written is actually the root cause of the issue
--
All of this will neccesitate changes to all of the subscribers. We will need to dispatch multiple BlockSent notifications in the subscriber for the responsemanager, which will ultimately be the solution to the root issue.
from go-graphsync.
Related Issues (20)
- Discussion: Support planned pausing HOT 3
- Convert Cancel/Update on graphsync request to single RequestType HOT 2
- Discussion: push graphsync requests HOT 6
- Discussion: Reconsidering requestor pausing HOT 2
- Discussion: Add an imperative "SendExtensions" Call to GraphSync top level interface HOT 3
- Discussion: Unify imperative commands HOT 2
- v2 message schema tweaks
- Flaky: missing tracing entries in integration tests
- Improved resumption of requests HOT 5
- Recover from panics in messaging layer
- Accept UUIDs from context in new requests
- Graphsync 2.1 Metadata Only Request
- Responder sends no blocks, just metadata
- Expose RequestMetadata method on graphsync client
- Extract the reconciled loader
- Support caching layer for metadata traversals on the responder to answer future requests quickly
- Backward compatibility for graphsync 2.1 protocol
- Include block size in metadata HOT 1
- TestGraphsyncRoundTripIgnoreNBlocks very flaky failing on Go 1.18 x86 HOT 1
- Update graphsync specs
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from go-graphsync.