Explicit Keyed Streams

[Max-Meldrum]

Explicit Keyed Streams

[segeljakt]

Flink seems to have:

• DataStream<T>
• KeyedStream<T, K> extends DataStream<T>
• AllWindowedStream<T, W extends Window>
• WindowedStream<T, K, W extends Window>
• BroadcastStream<T>
• CoGroupedStreams<T1, T2>
• ConnectedStreams<IN1, IN2>
• IterativeStream<T>
• JoinedStreams<T1, T2>
• PatternStream<T>
• QueryableStateStream<K, V>

[segeljakt]

It looks nice, I think a benefit with this is that you can have multiple key extractors for the same event type. Is the type of the stream produced by iterator also a keyed stream or something else?

[Max-Meldrum]

It looks nice, I think a benefit with this is that you can have multiple key extractors for the same event type. Is the type of the stream produced by iterator it also a keyed stream?

I take it would just be a Stream. iterator() -> Stream -> key_by(..) -> KeyedStream

We just have Stream right now, I guess we have to introduce more stream types such as in Flink Stream / KeyedStream / WindowStream?

[Max-Meldrum]

One possible way of defining key_by. Use Builder notion as in other parts of the API and require an extractor + description of how the stream is keyed.

let mut app = Application::with_conf(conf)
    .iterator((0..1000000).map(|x| Event { id: x, data: 1.5 }), |conf| {
        conf.set_timestamp_extractor(|x: &Event| x.id);
    }) // Stream<Event>
    .key_by(KeyBuilder {
         extractor: Arc::new(|event| event.id),
         description: "User Name ID",
     }) // KeyedStream<Event>