datafibers/df

Big Data Swiss Knifes

df's Issues

Need better logging system

We need a better logging system, such as log4j.
We also need to log all jobs seperately

Need better logging system

• We need a better logging system, such as log4j.
• We also need to log all jobs seperately

Batching files to HDFS

This is new feature to batch load file to HDFS.
In this case, the file need to be arrived in DF server first. DF just move/copy the file to HDFS.

DF Active Server Refactory

Need to refactor the server of DF as follows

• Use event bus
• Split code to different sevice verticle
• Redefine/Polish MetaData
• Redefine/Polish communication protocal
• Use HTTPS
• Add authentication

Need to add test case

Test cases are needed for both client and server

Configurable persist layer on HDFS

Make the data is archived into Hadoop and/or a file storage web service before it expires from Kafka.
This will be a far away feature. Put it here as placeholder.

DF Agent unblocking isssue

DF Agent is now unblocking with verx.
When 1st thread is not fininshed streaming while 2nd thread starts. There are chances to get both threads's data mixed up. As result, the Kafka will have bad data.

The resolution is to make it blocking to stream file one by one.

stream file to hdfs has duplication

resolve is as follows

1. remove handshake from while loop since for each request we only do handshake once
2. use regular expression .+

Need documentation wiki for setup

• VM setup
• Step for demo

Stream file need to watch folder changes

Stream file function requires following improvement

• While loop to watch folder changes with timeout
• Need to support file filters so that we do not stream arriving files
• Need to archive the streamed files somewhere so that we do not messed up

df-data-collector need 24hrs function

DF Demo df-data-collector can only get updated data when US stock is open. For demo purpose, we also need data available when the market is closed. We'll consider to use spoof data and also consider to add China market as another option.

Streaming to HDFS need imporvement

Current, the streamed data is saved to local file in df server first, then upload to HDFS. If the file is too big, we will not see the file in HDFS. A better way is to start writing to HDFS once the block size is reached. Later, we can merge the file together.

CSV file ingestion

This is to ingest csv file into HDFS

Metadata Logic Improvement

• Need to update the job status in terms of metadata
• For mongo, we can update
• For Kafaka, we can send another message

Need a demo for streaming

Create a demo for streaming

• real-time client
• reporting from Kafka

Need document for introduction

Need to polish the introduction to publish in the main site

Add filter and move option for stream files

We should add options to move the files which are processed to some archive folder so that we know the files are processed.
We also need to support filter files to be process, such as files with leading _

In this case, we can collaborate with stream generator to cosume files smoothly