tspannhw/flank-milvus

Flink - NiFi - Kafka - Milvus - Zilliz -Cloudera

FLaNK-Milvus

FLANK-AIM Flink - NiFi- Kafka - AI - Milvus

Flink - NiFi - MiNiFi - Kafka - Milvus - Zilliz -Cloudera

./runinsertmilvus.sh "Apache NiFi works with Apache Flink and Apache Kafka to provide the ultimate real-time streaming solution for all use cases"

Updated to Milvus 2.4

Example Output

{"uuid": "10607e43-5f26-4e62-a60e-2452d9fa4e06", "systemtime": "04/28/2024 14:13:40", "ts": 1714328020, "starttime": 1714328020, "endtime": 1714328020, "milvusinsertresult": "{'insert_count': 1, 'ids': [449398921271473400]}", "milvuscollectioname": "MilvusDocs", "milvusdocsource": "Medium", "maxseqlength": "512", "embeddingdim": "1024"}

Streaming AI

Use Cases

• CDC: Milvus to Kafka/Flink/NiFi: https://github.com/zilliztech/milvus-cdc

Edge AI

MiNiFi - Milvus

Integration points

Cloudera ML -> Milvus (AMP)

Milvus -> Kafka https://milvus.io/docs/message_storage_operator.md

Milvus -> Iceberg? Ozone? https://milvus.io/docs/object_storage_operator.md

Juyter notebooks

Projects of interest

https://github.com/zilliztech/GPTCache

Meetup Location

https://www.cloudera.com/about/locations.html

https://www.meetup.com/futureofdata-princeton/

https://www.meetup.com/futureofdata-newyork/

https://www.meetup.com/futureofdata-toronto/

https://www.meetup.com/futureofdata-triangle/

https://www.meetup.com/futureofdata-atlanta/

https://www.flink-forward.org/seattle-2023 (Nov 2023)

Resources

https://github.com/cloudera/CML_AMP_LLM_Chatbot_Augmented_with_Enterprise_Data/tree/main

https://github.com/zilliztech/cloud-vectordb-examples/tree/master/milvus-java/src/main/resources

https://towhee.io/

https://zilliz.com/news/Milvus-2.3-Massively-Accelerates-AI-Powered-Applications-With-NVIDIA-GPU-Support

https://zilliz.com/blog/milvus-2-3-beta-new-features-and-updates