dayanro/fileprocessor

Receives the files and generates asynchronous jobs to process them

A progressive Node.js framework for building efficient and scalable server-side applications.

Description

Nest framework TypeScript starter repository.

Installation

$ npm install

Running the app

# development
$ npm run start

# watch mode
$ npm run start:dev

# production mode
$ npm run start:prod

Test

# unit tests
$ npm run test

# e2e tests
$ npm run test:e2e

# test coverage
$ npm run test:cov

Support

Nest is an MIT-licensed open source project. It can grow thanks to the sponsors and support by the amazing backers. If you'd like to join them, please read more here.

Stay in touch

• Author - Kamil Myśliwiec
• Website - https://nestjs.com
• Twitter - @nestframework

License

Nest is MIT licensed.

fileprocessor's Issues

[Task]: As a user I want to receive a data file in CSV. #2

PROBLEM TO SOLVE AND DETAILS

WHO is the beneficiary/actor, WHAT it does/require and WHY? Add detailed explanations, images, diagrams, and data.

As a User I want to receive a data file in CSV format with a header and move it to a working storage space.

SOLUTION PROPOSAL

What can be done to solve the problem? How it can be solved?

• Implement the following method receive_file(filename, filelocation) where filename is the name of the file (e.g. data.csv( and filelocation the path where the file is located (e.g. s3://data-bucket/input/UUID/ ).
  It is not necessary to implement the credentials, we will assume that all access is public.

• This method fetches the file from the given filelocation, and moves it to a different work location than the previous one (e.g. s3://work-bucket/UUID/ )

CONDITIONS OF SATISFACTION

What does success look like? List of conditions that will be checked at the end of the issue to determine that is DONE.

• As a user I can receive a data file in CSV format.

[Task]: As a user I want to perform data transformation & send a notification.

PROBLEM TO SOLVE AND DETAILS

WHO is the beneficiary/actor, WHAT it does/require and WHY? Add detailed explanations, images, diagrams, and data.

As a User I want to perform data transformation and send a notification to a remote Notifications service.

SOLUTION PROPOSAL

What can be done to solve the problem? How it can be solved?

• The processing job is to modify the CSV header and write the field names in uppercase.
• That no value of the CSV fields is null. In the event that one of them is, a notification must be sent with said error (it usually happens that the files received from our clients are of the size of hundreds of GB of data. As sending so much data by HTTP is unfeasible, it can be assumed that the data processing service has direct access to said file).
• The received file will not be overwritten, but a new one will be written with the change made.
• This job must be launched asynchronously so as not to block the API for receiving files.

CONDITIONS OF SATISFACTION

What does success look like? List of conditions that will be checked at the end of the issue to determine that is DONE.

• As a user I can move the new file to the destination.
• As a user I can notify a remote service of Notifications
• As a user I can save a file with the metadata of the operation in the metadata repository.

[Feature]: As a user I want to receive a data file in CSV and perform data transformation. #1

PROBLEM TO SOLVE AND DETAILS

WHO is the beneficiary/actor, WHAT it does/require and WHY? Add detailed explanations, images, diagrams, and data.

As a User I want to receive a data file in CSV format with a header, move it to a working storage space and perform data transformation

SOLUTION PROPOSAL

What can be done to solve the problem? How it can be solved?

• Implement an API that receives the files
• Generates asynchronous jobs to process them.
• When a file is received, a notification will be sent to a remote Notifications service.
• The file processing job has to rewrite the header line fields to uppercase.
• When finished, it will move the files to a final location and notify a remote service of completion Notifications

CONDITIONS OF SATISFACTION

What does success look like? List of conditions that will be checked at the end of the issue to determine that is DONE.

• #2
• #3.