Comments (8)
The fileUpdatedTimestamp
and fileCreatedTimestamp
things are useful if they resubmit a few files with the same names (I don't understand why we need both), so we (or the user) can distinguish them.
from localega.
We still need a discussion to settle all those bits.
What we need in the message.
I leave it for the moment and I'll work on some JSON that I concoct.
from localega.
Answer from Jordi:
Once a file is referred, it is assigned status 0 or "New".
A periodic process is getting "New" files in batches using the database as channel.
For EGA 2.0 (microservices) we are sending messages to a queue
in order all interested microservices to get notified that
a new process is starting/requested.
[...]
I'm not happy with a mechanism that rely on an specific endpoint being up
and listening for a process that *should* happen and
that is not requiring synchronicity to proceed.
However, we can use the Endpoint mechanism temporarily.
In other words, Jordi prefers an asynchronous communication.
Like, a message is dropped somewhere, and whoever is interested picks it up (eg in a message broker like RabbitMQ)
A solution for us (as a trigger-mechanism) is to contact their message broker directly, though using certificate for authentication, and pick the message if flagged with "Sweden", or something similar.
from localega.
If we connect to their Message Broker and pick an interested message, we should think of having several connections open, for fault-tolerance.
That covers the case where a connection is lost.
We could pick the message and put it in our message broker, internally (even modifying it if necessary). The different connections should not duplicate the message.
Note: I think there is a functionality in RabbitMQ to handle that. Like routing to the right place, combined with queue filtering.
Worth investigating.
from localega.
Note from Jordi: they could send us a message to our message broker!
If we don't have one, because we go for a simple multi-threaded implementation, then we could create one with a simple purpose: a mailbox for incoming requests. (RabbitMQ might be an overkill, and ZeroMQ might suffice).
If we have one, then we need to "secure" it. I prefer to let them do that. By simple laziness, and also to let them get the blame in case the broker is compromised. Nasty but nonetheless true.
In short: I prefer to pick a message from their queue, using a certificate for authentication.
from localega.
Update: Jordi suggests to hook both message brokers, one from Spain and one from Sweden in a federated manner. That way, they could route messages to the swedish queue when necessary.
Juha is concerned about separating the components, so that if one is upgraded, we won't have issues with dependencies or incompatibilities.
It's important, and a solution is in AMQP (the Message Broker protocol). We're fine if both components, upgraded or not, do talk AMQP.
Implementing code that handles message exchange between message broker is, to me, redundant. That code would sit and wait for messages to drop in the queues and forward them to other queues. It introduces an extra layer which is good for security.
We are still very much in the discussion inside the LocalEGA Slack.
from localega.
I am in favor of hooking our MQ to their MQ (including SSL communication).
That would be step 1.
If this doesn't work, we then advise and think of a backup plan with ReST API calls.
Note: I already have some code for the ReST API, there is no effort to account for, here. Just that if we go for the MQ-linked-together solution, that code is then dormant.
from localega.
Ok, after discussing with Oscar, I have bad and good news.
Bad news: CRG does not handle submissions, so there is no message containing submission data, like file and checksum paths.
Good news: We designed one.
The message, as a first try, will look like this.
It is JSON-formatted:
{
"@class" : "eu.crg.ega.microservice.dto.message.WorkFlowCommandMessage",
"header" : {
"format" : null,
"producer" : {
"host" : null,
"ip" : null,
"application" : "workflow",
"processId" : null,
"userId" : null
},
"messageId" : null,
"conversationId" : null,
"idInSequence" : null,
"timestamp" : null,
"millisecToExpiration" : null,
"replyTo" : null
},
"messageType" : "COMMAND",
"command" : {
"commandType" : "WORKFLOW",
"version" : "v1"
},
"parameters" : null,
"submissionId":"asdasd-asdasd-asdasd",
"encryptedHash":"fdfssdfsdf",
"unencryptedHash":"adasdas",
"hashAlgorithm": "md5",
"file": "/path/to/file",
"filesize":1231313,
"fileUpdatedTimestamp":"",
"fileCreatedTimestamp":""
}
It contains some extra things I don't frankly care about, but Oscar said they will matter later.
So, fair enough, we keep them.
All I care is:
Parameter | Comment |
---|---|
submissionId | Gotten from Central EGA |
encryptedHash | he wants to send as string, not a path to a file containing the hash string |
unencryptedHash | idem |
hashAlgorithm | which I made him add, in case we want to handle other/better algorithms |
file | a path relative to the inbox |
filesize, fileUpdatedTimestamp, fileCreatedTimestamp, etc... |
for logging, I guess |
I raise some concern about the file path relative to the inbox, in case we setup several inboxes (like a plain FTP one, and some other one for Bianca, for example)
The latter is part of issue #5 related to the inbox setup. Not the concern here.
from localega.
Related Issues (20)
- Testing LocalEGA deployment in Openshift multi tenant environment HOT 1
- Integrate ELK stack for Data API HOT 1
- Create aggregated Docker Compose file for Data In and Data Out HOT 3
- Update the LocalEGA database schema HOT 7
- Provide Data Ingestion and Data API Deployment-Ready LocalEGA
- Clean up comments and documentation
- Re-Encrypt AES part if session key already used HOT 2
- Sharing microservice among EGA-DATA-API and LocalEGA
- Make it possible to deploy both ingestion and Data API using Helm charts HOT 2
- Test the designed deployment solution production-like test environment HOT 4
- Add missing Data API services to docker-compose.yml HOT 1
- Log with `correlation_id` throughout the ingestion process
- Update documentation
- nbis -> ega-archive
- Renaming components HOT 1
- DB & MQ retry logic HOT 1
- Network separation HOT 1
- Update the configuration module HOT 1
- Implement support S3 storage backend for ingestion
- Update Crypt4GH to the latest version HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from localega.