Giter Site home page Giter Site logo

noobaa / noobaa-core Goto Github PK

View Code? Open in Web Editor NEW
254.0 254.0 75.0 68.03 MB

Object-storage Transformer server for Node.js 🦾🤖🦾 (s3 => file / s3 / azure / google / erasure-code / tiering / merge / cache / logging / replicate / custom...)

Home Page: https://www.noobaa.io

License: Apache License 2.0

Shell 1.82% JavaScript 88.84% Python 0.18% C++ 4.90% Go 0.19% C 3.35% Dockerfile 0.25% Makefile 0.29% Groovy 0.10% PLpgSQL 0.08%
data hybrid mirroring multi-cloud nodejs noobaa s3 storage tiering

noobaa-core's Introduction

~~~ WARNING ~~~

THIS REPO IS DEPRECATED AND ARCHIVED




##Readme This guide will help you to setup your mahcine to develop with NooBaa. If you read this, it is YOUR role to take this code and make it into a script for the next person after you. If you are the 3rd person reading this (based on commits) you have a team of assholes.

This was ran on yuval's guest VM running kubuntu 14:

Linux yuvaldim-VirtualBox 3.13.0-30-generic #55-Ubuntu SMP Fri Jul 4 21:40:53 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

###Assumption (should be modified when creating the script):

  • working folder = ~/workspace
  • git user name: "yuvaldim"
  • git user email: "[email protected]"
  • DB name in mongo "nb"

##Install git

sudo apt-get install git
git config --global user.name "yuvaldim"
git config --global user.email "[email protected]"
git config --list

##Install node

sudo apt-get update
sudo apt-get install nodejs
sudo apt-get install npm
#--The following is required for compatibility
#--- http://stackoverflow.com/questions/20057790/what-are-the-differences-between-node-js-and-node
sudo ln -s `which nodejs` /usr/local/bin/node

##Install heroku

sudo wget -qO- https://toolbelt.heroku.com/install-ubuntu.sh | sh
# requires login
> heroku login

[Heroku has a windows and mac clients so you should try them out here.] (https://help.github.com/articles/set-up-git)

##Additional packages

#install gulp globaly
sudo npm install gulp -g
#install the notifier for gulp:
sudo apt-get install libnotify-bin
sudo apt-get install notify-osd

#Install mongo (2.4) We are using Mongo 2.4 so all info related to this version.

#http://docs.mongodb.org/v2.4/tutorial/install-mongodb-on-ubuntu/
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | sudo tee /etc/apt/sources.list.d/mongodb.list
sudo apt-get update
sudo apt-get install mongodb-10gen=2.4.10
#the following prevernts from auto version upgrade
sudo echo "mongodb-10gen hold" | sudo dpkg --set-selections

To create the DB:

#ET

sudo mkdir -p /data/db
sudo mongod --smallfiles

#end ET
#---get into mongo:
# mongo
#---To create db nb
# use nb
#---insert some test data
# j = { name : "mongo" }
# db.testData.insert( j )
#---verify DB was created with 
# show dbs
db.addUser( { user: "admin", pwd: "admin", roles: [ "readWrite" ] } )

#Clone the project to your device

#--- create workspace folder under the user
mkdir ~/workspace
cd ~/workspace/
#clone the noobaa repository
git clone https://github.com/guymguym/noobaa.git
cd ~/workspace/noobaa
sudo npm install

#Get updated env file from Guy and update the bucket prefix

Installing Scons

download scones from http://downloads.sourceforge.net/project/scons/scons/2.3.4/scons-2.3.4.tar.gz?r=http%3A%2F%2Fwww.scons.org%2Fdownload.php&ts=1431437307&use_mirror=garr

tar -xzvf scons-2.3.4.tar.gz
cd scons-2.3.4
python setup.py install

install makensis for installation build (MAC)

Make sure you have scons installed ([Installing Scons](# Installing Scons)) download nsis from here: (for more information) follow http://blog.alejandrocelaya.com/2014/02/01/compile-nsis-scripts-in-linux/

curl -L "http://downloads.sourceforge.net/project/nsis/NSIS%203%20Pre-release/3.0b1/nsis-3.0b1-src.tar.bz2?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fnsis%2Ffiles%2FNSIS%25203%2520Pre-release%2F3.0b1%2F&ts=1423381229&use_mirror=garr" > nsis-3.0b1-src.tar.bz2
curl -L "http://downloads.sourceforge.net/project/nsis/NSIS%203%20Pre-release/3.0b1/nsis-3.0b1.zip?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fnsis%2Ffiles%2FNSIS%25203%2520Pre-release%2F3.0b1%2F&ts=1423381286&use_mirror=garr" >> nsis-3.0b1.zip
unzip nsis-3.0b1.zip
bzip2 -dk nsis-3.0b1-src.tar.bz2
tar -xvf nsis-3.0b1-src.tar

BEFORE you run the following command scons command, update SConstruct file under nsis-3.0b1-src folder with opts.Add(BoolVariable('STRIP_CP', 'Strips cross-platform executables of any unrequired data such as symbols', 'no'))

cd nsis-3.0b1-src
scons SKIPSTUBS=all SKIPPLUGINS=all SKIPUTILS=all SKIPMISC=all NSIS_CONFIG_CONST_DATA=no PREFIX=<FULL_PATH>/nsis-3.0b1 install-compiler
chmod +x <FULL_PATH>/nsis-3.0b1/bin/makensis
ln -s <FULL_PATH>/nsis-3.0b1/bin/makensis /usr/local/bin/makensis
mkdir ./nsis-3.0b1/share
cd ./nsis-3.0b1/share
ln -s <FULL_PATH>/nsis-3.0b1 nsis

If there are problems with running scons regarding zlib errors, try installing zlib-devel

download nsis plugin

http://nsis.sourceforge.net/mediawiki/images/1/18/NsProcess.zip
  1. unzip
  2. copy /Include/nsProcess.nsh to /Include
  3. copy /Plugin/nsProcess.dll to /Plugins/x86-ansi

#build under folder /noobaa/src/planet-app/prod [TODO - Add to gulp script]

makensis noobaa.nsi

#Start it:

foreman start

noobaa-core's People

Contributors

achouhan09 avatar alphaprinz avatar aspandey avatar baum avatar dannyzaken avatar dependabot[bot] avatar eroolgui avatar fitari avatar guymguym avatar igorpick avatar jackyalbo avatar jeniawhite avatar kfir-payne avatar liranmauda avatar nadavmiz avatar naveenpaul1 avatar nb-ohad avatar neon-white avatar nimrod-becker avatar nimrodgeva avatar nixpanic avatar paulxuzhang avatar quintinbecker avatar romayalon avatar sergii-karpiak avatar shirady avatar tamireran avatar tangledbytes avatar vh05 avatar yaelal avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

noobaa-core's Issues

Streaming vs downloading and playing

  1. Small files (under 10MB) should be downloaded and played.
  2. Larger files should behave as follows:
    1. If the file is not streamable/previewable download it.
    2. If there is a configured REST server, stream/preview via the REST server.
    3. In all other cases, download.

disconnecting from signalling server cause 100% cpu

simply start an agent and disconnect from the network

The result is a massive amount of messages like this and 100% cpu

INFO:socket_host_udp.cc(298)] sendto() has failed twice returning a transient error. Dropping the packet

RPC retries should only apply on transient errors and not persistent errors

currently we don't differentiate the two types of errors, so the RPC code doesn't have the information, so we should decide on a clear separation of errors for transient vs. persistent.

transient errors could be - network issue, resource temporarily unavailable (DB, REDIS, HEROKU).

with ICE with multiplexing we should send back an indication of in-progress for requests we are processing, so that we won't retry and just add more pressure.

File parts should be tabular

Currently expected 3 in a line (based on string length for alignment).
Result is not aligned. Need to align all lines (including sublines)

image

User facing events should be issued via a framwork

Currently some events are issued to the event log.
Some issues pop up a red error message.
Some don't show anything.

Suggestion:
To force all messages to be issued vie a single framework that will get an input and will be responsible to propagate to the right place/s.

Interactive installation should provide some feedback

I installed the agent manually.
After I doubleclicked it in the windows explorer I got only a single indication - the windows certificate issue.
After it's done - no success/failure indication.

Severity explanation:
Severity 3
The task that you were attempting to use behaved in a manner which is incorrect and/or unexpected, or presented misleading or confusing information.

Not all DB queries handle the deleted != NULL situations

Currently our DB objects are not really deleted, merely marked as deleted.
In several of our queries we don't verify if the objects we received are marked as deleted (or don't filter non deleted).

We should fix this flows as they will cause us to crash.

problem writing to activity log on node.create

TypeError: Cannot read property 'name' of undefined
at transformer.js:149
at arrayFilter (tv4.js:994)
at Function.filter (nb_api.js:290)
at transformer.js:149
at angular.js:13189
at l.$get.l.$eval (angular.js:14401)
at l.$get.l.$digest (angular.js:14217)
at angular.js:14440
at e (angular.js:4905)
at angular.js:5285
transformer.js:149 RELOAD SYSTEM FAILED TypeError: Cannot read property 'name' of undefined {stack: "TypeError: Cannot read property 'name' of undefine…/ajax/libs/angularjs/1.3.14/angular.min.js:44:112", message: "Cannot read property 'name' of undefined"}message: "Cannot read property 'name' of undefined"stack: "TypeError: Cannot read property 'name' of undefined↵ at http://127.0.0.1:5001/public/js/index.js:90943:64↵ at arrayFilter (http://127.0.0.1:5001/public/js/index.js:67847:13)↵ at Function.filter (http://127.0.0.1:5001/public/js/index.js:72070:14)↵ at http://127.0.0.1:5001/public/js/index.js:90928:45↵ at http://ajax.googleapis.com/ajax/libs/angularjs/1.3.14/angular.min.js:112:20↵ at l.$get.l.$eval (http://ajax.googleapis.com/ajax/libs/angularjs/1.3.14/angular.min.js:125:305)↵ at l.$get.l.$digest (http://ajax.googleapis.com/ajax/libs/angularjs/1.3.14/angular.min.js:122:398)↵ at http://ajax.googleapis.com/ajax/libs/angularjs/1.3.14/angular.min.js:125:392↵ at e (http://ajax.googleapis.com/ajax/libs/angularjs/1.3.14/angular.min.js:40:74)↵ at http://ajax.googleapis.com/ajax/libs/angularjs/1.3.14/angular.min.js:44:112"get stack: function () { [native code] }set stack: function () { [native code] }proto: Errortransformer.js:149 (anonymous function)angular.js:13189 (anonymous function)angular.js:14401 $get.l.$evalangular.js:14217 $get.l.$digestangular.js:14440 (anonymous function)angular.js:4905 eangular.js:5285 (anonymous function)

in nb_api.js upon

case 'node.create':
l.category = 'nodes';
l.text = 'Added node ' + l.node.name;
break;

activity log error on obj.uploaded

case 'obj.uploaded':
l.category = 'files';
l.text = 'Upload completed ' + l.obj.key;
break;

i.obj is undefined:

TypeError: Cannot read property 'key' of undefined
at transformer.js:149
at arrayFilter (tv4.js:965)
at Function.filter (nb_api.js:231)
at transformer.js:149
at angular.js:13189
at l.$get.l.$eval (angular.js:14401)
at l.$get.l.$digest (angular.js:14217)
at angular.js:14440
at e (angular.js:4905)
at angular.js:5285

RELOAD SYSTEM FAILED TypeError: Cannot read property 'key' of undefined {stack: (...), message: "Cannot read property 'key' of undefined"

UI Breadcrumbs issue

Breadcrumbs-
*When entering a list (nodes list for example) there should be a way back (should add breadcrumb - nodes/ nodes list and/or add a back option). same issue on other lists.
*If "system"="overview" we should write overview/ home on bread crumbs

  • The Resources page is a little confusing since the user presses "nodes" on the menu but suddenly there is another step on the breadcrumb (System / Resources /Nodes). Same thing on files.

Rename tasks and windows services

Suggestions:
Service: NooBaa_device_manager
nssm.exe: NooBaa_Agnet_wd
node.exe: NooBaa_Agnet

Severity 3
This can include documentation which was incomplete or incorrect, making it difficult to know how to use a task.

Race: delete_object_mapping & allocate_object_parts

A race can occur between delete object mappings and allocate object parts. In a case of a chunk which is about to be deleted, but equals in hash to a chunk being uploaded, we might use that chunk in the allocation but then delete it..

We can either protect it by locking the specific chunk, or by marking it as delete pending and waiting for the deletion for some interval (checking in the allocate if the chunk is not delete pending).
Choosing the second solution we should be careful to return the RPC context and not block it on the deletion pend.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.