dicl/veloxdfs

DHT-based Distributed File System for MapReduce Jobs

veloxdfs's Issues

Additional configure option for logical block features

I added some options for logical block enabled build (--enable_lblocks & --enable-boost-static), but after the build completion, the error message is like below

Any classes that I should add?
Thanks.

LocalIO Update

In localio function update should pass value variable by reference, so it will not double copy.

Add executor

Remove commented code and use the same coding style

In the previous month we rushed to finish our main parts for EclipseDFS, now that it is working it will be good to clean up our source code.

Also, in order to ease reading our code, we should follow a coding standard. I propose Google C++ style

Problems linking with ASIO serialization

@nammh and I found that in some of our servers Eclipse does not compile since it can not find the boost libraries.

For GCC you must specify in LIBRARY_PATH the location of the the libraries, normally writing in your .bashrc the next line will make the trick:

export LIBRARY_PATH=/usr/local/lib

As for Clang++ compiler, things are a bit harder since it does not read LIBRARY_PATH. In that case, write in your .bashrc:

export CONFIG_SITE="~/.clang_fix"

Then create that ~/.clang_fix file and write inside:

#!/bin/bash
export LDFLAGS+="-L/usr/local/lib"

MapReduce framework

• Job and Task classes #8
• FS support #10
• Executor class #9
• Dynamic lib loader #16
• Application API #11

Logical block configuration

How can I configure logical block features?
Just adding adding --enable-lblocks options as configuration, /eclipse_node abort as I execute the program
Is there any additional options that I should handle?

Add Task and Job classes

eclipsed script

From this commit, we can use eclipsed to launch, check status and close eclipseDFS:

To install it run:

$ gem install eclipsed

Here is the repo of the eclipsed:

https://github.com/DICL/eclipsed

feature-dfsAPI

Make dfs API

Getting more done in GitHub with ZenHub

Hola! @olzhabay has created a ZenHub account for the DICL organization. ZenHub is the leading team collaboration and project management solution built for GitHub.

---

How do I use ZenHub?

To get set up with ZenHub, all you have to do is download the browser extension and log in with your GitHub account. Once you do, you’ll get access to ZenHub’s complete feature-set immediately.

What can ZenHub do?

ZenHub adds a series of enhancements directly inside the GitHub UI:

• Real-time, customizable task boards for GitHub issues;
• Burndown charts, estimates, and velocity tracking based on GitHub Milestones;
• Personal to-do lists and task prioritization;
• “+1” button for GitHub issues and comments;
• Drag-and-drop file sharing;
• Time-saving shortcuts like a quick repo switcher.

Add ZenHub to GitHub

Still curious? See more ZenHub features or read user reviews. This issue was written by your friendly ZenHub bot, posted by request from @olzhabay.

Change dfsput/dfsget... to dfs [put|get|format|ls]

Fixing dfs bugs

Fixing dfs bugs and changing block hash key part

Iterative map workflow discussion

We already determined the way of implement the iterative map workflow. However, @kbjin mentioned that not having shuffling phase in between the map iterations might lead to an inconsistent since the key values created from the previous iteration might be scattered in different nodes. @kbjin can you explain here your question?

Eclipse Cache

• Cache using ASIO (#1)
• Cache using MPI (#2)

Master node

Depends on #13

Network Module

Basic Comm. and Topologies

Changing the behavior of eclipse when it inits

Professor said that EclipseDFS should read the db when we launch it, not ignore it.

We should implement the logic for that, I think that @kbjin is more familiarize with that, can you make a feature for that?

[PR:108]Different length formats in two sprintf for size

In update_file_metadata PRIu64 is used while in another functions it uses PRIu32

        size=%" PRIu64 ", num_block=%u where name='%s';",

Add file system support

Open source license

Which open source license should we use in EclipseDFS?

Release 1.2.0 task list

• : |compatible| Add replica support [Critical]
• : |compatible| Add Persistence of the network in case of failure [Critical]

Centralized Topology

ASIO Eclipse Cache

• Network Class (#6)

API for application

MPI Eclipse Cache

One definition rule in Job and task classes

@zwigul all the definitions should go into the cc file. Otherwise we will have linker errors. Can you make a bugfix branch fix it and pull request to master?

DFSput removes \n between the chunks.

Here I ilustrate how the error occurs, this is the diff of a file called config.h before inserting it with dfsput and after received with dfsget. I believe that the bug is in the dfsput.cc.

--- config.h    2016-03-09 15:34:48.967514994 +0900
+++ build/config.h      2016-03-09 15:31:52.664229099 +0900
@@ -29,6 +29,7 @@

 /* Define to 1 if you have the <memory.h> header file. */
 #define HAVE_MEMORY_H 1
+
 /* Define to 1 if you have the <netdb.h> header file. */
 #define HAVE_NETDB_H 1

@@ -65,6 +66,7 @@

 /* Name of package */
 #define PACKAGE "eclipse"
+
 /* Define to the address where bug reports for this package should be sent. */
 #define PACKAGE_BUGREPORT "[email protected]"

@@ -94,7 +96,8 @@
    #define below would cause a syntax error. */
 /* #undef _UINT32_T */

-/* Define for Solaris 2.5.1 so the uint8_t typedef from <sys/synch.h>,   <pthread.h>, or <semaphore.h> is not used. If the typedef were allowed, the
+/* Define for Solaris 2.5.1 so the uint8_t typedef from <sys/synch.h>,
+   <pthread.h>, or <semaphore.h> is not used. If the typedef were allowed, the
    #define below would cause a syntax error. */
 /* #undef _UINT8_T */

Release 1.1.0 task list

There is a few changes to be made for the next minor release:

• #51 : |compatible| Add doxygen to document the code
• #50 : |compatible| Change raw pointers to smart pointers (Many bugs were caused by that #46)
• #52 : |compatible| Remove commented code and adopt unified coding style
• #49 : |compatible| Decide open source license [Critical]
• #41 : |incompatible| Change the way the EclipseDFS initializes [Critical]
• #55 : |incompatible| Change dfsput/dfsget... to dfs [put|get|format|ls]
• #59 : |incompatible| dfsformat [Critical]
• #56 : |incompatible| Create dfs [show] to display blocks locations of a file.

Note that previous releases were pre-releases, version 1.1.0 aims to be a release.
Compatibles changes will be released in versions 1.0.*, incompatibles will wait until
all of them are done to relase 1.1.0

Include list function for the Directory API

We need a member function inside directory class to display all the files inside its local sqlite db.

Application API

Change usage of std::string to movable object

Constructor of std::string does not allow any moving. Instead, it always copies all data.
Except if pointer std::string is not used. However, that is used minorly in project.
This leads to too many unnecessary copying around data, which can be size of block (64mb).

Proposal, use custom class that will not be copying data, but moving it. Also, it will control number of owners and gets destructed when owners<=0

class Slice {
  char* ptr;
  uint64_t size;
  atomic<int> owners;
}

Other way is to use std::shared_ptrstd::string

Solving this will do some improvement to performance

Change raw pointers to smart pointers

Whenever it is possible it will be desirable to use smart pointers to avoid strange bugs cause by memory leaks

Create dfs [show] to display blocks locations of a file.

This featured was mentioned in #42

Getting more done in GitHub with ZenHub

Hola! @vicentebolea has created a ZenHub account for the DICL organization. ZenHub is the leading team collaboration and project management solution built for GitHub.

---

How do I use ZenHub?

To get set up with ZenHub, all you have to do is download the browser extension and log in with your GitHub account. Once you do, you’ll get access to ZenHub’s complete feature-set immediately.

What can ZenHub do?

ZenHub adds a series of enhancements directly inside the GitHub UI:

• Real-time, customizable task boards for GitHub issues;
• Burndown charts, estimates, and velocity tracking based on GitHub Milestones;
• Personal to-do lists and task prioritization;
• “+1” button for GitHub issues and comments;
• Drag-and-drop file sharing;
• Time-saving shortcuts like a quick repo switcher.

Add ZenHub to GitHub

Still curious? See more ZenHub features or read user reviews. This issue was written by your friendly ZenHub bot, posted by request from @vicentebolea.

About dfsget and retrieve operation in DFS

I am implementing the logic for dfsget inside the eclipse network. The main ideas is that the leader node request file blocks to the neighbors, and file blocks arrives in a asynchronous manner. So in order to send them to the client i have to wait to collect all the block so that i can send them in order. So should i store in disk the blocks?

Thread pool for std::future

When calling std::async, it creates new thread. Better having thread pool and enqueue to it functions and lambdas with return of std::future.

Adding doxygen

Doxygen is the only software which generates documentation from C++ code, we have to start commenting our functions, making those comment using doxygen syntax will be trivial.

Also github supports creating a website which our project documentation, class diagrams and so on.

dfsrm does not remove certain blocks

@zwigul has observed that sometimes dfsrm does not removes some file blocks in the scratch directory.

Dynamic library loader