open-io/oio-sds

High Performance Software-Defined Object Storage for Big Data and AI, that supports Amazon S3 and Openstack Swift

OpenIO Software Defined Storage

OpenIO SDS is a software solution for object storage, targeting very large-scale unstructured data volumes.

Install

Either you go from scratch (the source) or you download the packages for your Linux distribution, install, and run!

Getting Started

There is one simple script to execute:

./tools/oio-reset.sh

And if it succeeds you will have the joy to experiment your own little SDS instance. No root privileges are required!

Build

Please refer to BUILD.md for detailed information about how to compile and configure the solution.

oio-sds's Issues

meta2 mover: m1 update policy?

We need something similar to the old fashioned m1 update policy request for meta2 migrations.

Please refactor client/c/lib/test_services.c

My eyes are bleeding ...

• Please fix the indentation to match the de facto standard of the remaining of the code (feel free to ask for more information)
• Please fix the pletora of warnings at compile time.

LB WRAND: No rawx available

Using the default lb.rawx load balancing (WRAND I think), I have some "No rawx available" errors on my platform. It occurs sometimes on a quite heavy load.
Configuration : 3 rawx and replication x3
Notice that the platform is stable, and the rawx scores are upper 50 during all the load.

I work on a little script to reproduce the problem on your env.
I will post it as soon as it will be available.

NB : When I set the lb.rawx in RR, the problem disappear.

metautils test str: assertion failed

oio.utils:ERROR:/home/noone/public_git/github.com/jkasarherou/oio-sds/metautils/lib/test_str.c:111:test_ishexa: assertion failed: (oio_str_ishexa("0",1))

LB configuration?

LB conscience configuration is not documented! Provide a minimal set of configuration examples.

LB config output on oio-cluster

When you use oio-cluster, output don't show the default loadblancing configuration.

Can you put the loadblancing configuration by default in oio-cluster output ?

Like :

LB(srv) : meta2=KEEP|1|1;sqlx=KEEP|1|1
LB(meta2) : rawx = WRAND?shorten_ratio=0.600000&standard_deviation=no

Copyright @jfsmig

Running unit tests?

Hi, is there any documentation about how to run unit tests for oio-sds?

Thanks,

Proxy/meta2 : meta2 process aborted on deleting content properties

After having initialized a meta2 content with random properties and asked the proxy for deleting one of these properties, the meta2 process crashes with a SIGABRT signal. Meanwhile, the proxy surprisingly returns a 204 answer on the deletion request.

Here is the trace :

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7f0006855700 (LWP 17900)]
0x00007f00087775d7 in raise () from /lib64/libc.so.6
(gdb) bt
0  0x00007f00087775d7 in raise () from /lib64/libc.so.6
1  0x00007f0008778cc8 in abort () from /lib64/libc.so.6
2  0x00007f00087b7e07 in __libc_message () from /lib64/libc.so.6
3  0x00007f00087bf1fd in _int_free () from /lib64/libc.so.6
4  0x00007f000b23658f in g_free () from /lib64/libglib-2.0.so.0
5  0x00007f000bd75200 in g_free0 (p=0x7effe0009940)
     at /home/blextrait/public_git/github.com/oio-sds/metautils/lib/utils_tools.c:206
6  0x000000000040b43f in _input_data_clean (input_data=0x7effe00347a0)
     at /home/blextrait/public_git/github.com/oio-sds/meta2v2/meta2_filter_context.c:66
7  0x000000000040b580 in meta2_filter_ctx_clean (ctx=0x7effe0021e00)
     at /home/blextrait/public_git/github.com/oio-sds/meta2v2/meta2_filter_context.c:117
8  0x0000000000405818 in meta2_dispatch_all (reply=0x7f0006854b50, gdata=0x1ada880,
     hdata=0x60fae0 <M2V2_PROPDEL_FILTERS>)
     at /home/blextrait/public_git/github.com/oio-sds/meta2v2/meta2_gridd_dispatcher.c:82
9  0x00007f000b92b4aa in _client_call_handler ()
     at /home/blextrait/public_git/github.com/oio-sds/server/transport_gridd.c:730
10 0x00007f000b92be20 in _client_manage_l4v (client=0x7efff4001a00, gba=0x7effe0020890)
     at /home/blextrait/public_git/github.com/oio-sds/server/transport_gridd.c:810
11 0x00007f000b92ac55 in transport_gridd_notify_input (clt=0x7efff4001a00)
     at /home/blextrait/public_git/github.com/oio-sds/server/transport_gridd.c:508
12 0x00007f000b9290c8 in _notify ()
     at /home/blextrait/public_git/github.com/oio-sds/server/network_server.c:1443
13 0x00007f000b929209 in _client_manage_input ()
     at /home/blextrait/public_git/github.com/oio-sds/server/network_server.c:1467
14 0x00007f000b9293d0 in _client_manage_event (clt=0x7efff4001a00, events=1)
     at /home/blextrait/public_git/github.com/oio-sds/server/network_server.c:1537
15 0x00007f000b92800f in _work_on_client (srv=0x1a76280, clt=0x7efff4001a00)
     at /home/blextrait/public_git/github.com/oio-sds/server/network_server.c:1043
16 0x00007f000b928253 in _thread_cb_worker (td=0x1a76280)
     at /home/blextrait/public_git/github.com/oio-sds/server/network_server.c:1092
17 0x00007f000b2566b5 in g_thread_proxy () from /lib64/libglib-2.0.so.0
18 0x00007f000852ddf5 in start_thread () from /lib64/libpthread.so.0
19 0x00007f00088381ad in clone () from /lib64/libc.so.6

meta2: segfault on PROPDEL_FILTERS

#0  0x0000000000000011 in ?? ()
#1  0x00007fb40d5579d0 in ?? ()
#2  0x00007fb40d5579d0 in ?? ()
#3  0x000000000040b6bd in _input_data_clean (input_data=0x7fb3fc086c40)
at /home/noone/public_git/github.com/jkasarherou/oio-sds/meta2v2/meta2_filter_context.c:67
#4  0x000000000040b81a in meta2_filter_ctx_clean (ctx=0x7fb3fc09d330)
at /home/noone/public_git/github.com/jkasarherou/oio-sds/meta2v2/meta2_filter_context.c:121
#5  0x0000000000405a1e in meta2_dispatch_all (reply=0x7fb40d557b50, gdata=0x23d92a0, hdata=0x60f740 <M2V2_PROPDEL_FILTERS>)
at /home/noone/public_git/github.com/jkasarherou/oio-sds/meta2v2/meta2_gridd_dispatcher.c:78
#6  0x00007fb4115ad71a in _client_call_handler () at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/transport_gridd.c:729
#7  0x00007fb4115adf65 in _client_manage_l4v (client=0x7fb3f0001a00, gba=0x7fb3fc018630)
at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/transport_gridd.c:820
#8  0x00007fb4115acf2f in transport_gridd_notify_input (clt=0x7fb3f0001a00)
at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/transport_gridd.c:504
#9  0x00007fb4115ab7c6 in _notify () at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/network_server.c:1439
#10 0x00007fb4115ab8a0 in _client_manage_input () at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/network_server.c:1463
#11 0x00007fb4115aba61 in _client_manage_event (clt=0x7fb3f0001a00, events=1)
at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/network_server.c:1533
#12 0x00007fb4115aa96c in _work_on_client (srv=0x23b3100, clt=0x7fb3f0001a00)
at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/network_server.c:1039
#13 0x00007fb4115aab7d in _thread_cb_worker (td=0x23b3100) at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/network_server.c:1088
#14 0x00007fb412d3c625 in g_test_get_filename () from /usr/lib/libglib-2.0.so.0
#15 0x0000000000000000 in ?? ()

Proxy/meta1 : wrong error code on put/get/del properties

When sending a post request to put/set/del some properties on a reference which doesn't exist, the reply is a 403 forbidden code instead of 404 not found, which would be clearer.

sqliterepo: base creation is not optimal

• A template DB file is first written.
• The base is opened, an explicit transaction is then used to save the name of the base, then the base is closed.
• The opening routine continues with the base's re-opening, where some keys are also saved into a transaction.

IMO, a single transaction is enough for all these operations. This would save a lot of I/O ops (locks, index loading, syncs, etc)

meta2: segfault on RAW_SUBST

using the M2V2_RAW_SUBST

#0  0x00007f88e8cf59d2 in ?? ()
#1  0x000000000040b6bd in _input_data_clean (input_data=0x7f88cc005c30)
at /home/noone/public_git/github.com/jkasarherou/oio-sds/meta2v2/meta2_filter_context.c:67
#2  0x000000000040b81a in meta2_filter_ctx_clean (ctx=0x7f88cc005800)
at /home/noone/public_git/github.com/jkasarherou/oio-sds/meta2v2/meta2_filter_context.c:121
#3  0x0000000000405a1e in meta2_dispatch_all (reply=0x7f88e8cf5b50, gdata=0x1cd0280, hdata=0x60f8e0 <M2V2_RAW_SUBST_filters>)
at /home/noone/public_git/github.com/jkasarherou/oio-sds/meta2v2/meta2_gridd_dispatcher.c:78
#4  0x00007f88ecd4b71a in _client_call_handler () at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/transport_gridd.c:729
#5  0x00007f88ecd4bf65 in _client_manage_l4v (client=0x7f88d0001a00, gba=0x7f88cc088a70)
at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/transport_gridd.c:820
#6  0x00007f88ecd4af2f in transport_gridd_notify_input (clt=0x7f88d0001a00)
at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/transport_gridd.c:504
#7  0x00007f88ecd497c6 in _notify () at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/network_server.c:1439
#8  0x00007f88ecd498a0 in _client_manage_input () at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/network_server.c:1463
#9  0x00007f88ecd49a61 in _client_manage_event (clt=0x7f88d0001a00, events=1)
at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/network_server.c:1533
#10 0x00007f88ecd4896c in _work_on_client (srv=0x1caab00, clt=0x7f88d0001a00)
at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/network_server.c:1039
#11 0x00007f88ecd48b7d in _thread_cb_worker (td=0x1caab00) at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/network_server.c:1088
#12 0x00007f88ee4da715 in ?? () from /usr/lib/libglib-2.0.so.0
#13 0x00007f88eaaac4a4 in start_thread () from /usr/lib/libpthread.so.0
#14 0x00007f88eb01b13d in clone () from /usr/lib/libc.so.6

proxy crash

After a simple request on the proxy:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f166e2ce700 (LWP 17634)]
0x00007f166e1cc9b0 in ?? ()
(gdb) bt
#0  0x00007f166e1cc9b0 in ?? ()
#1  0x00007f1670c51e2d in ?? () from /usr/lib/libglib-2.0.so.0
#2  0x000000000041ace8 in header_provider (k0=0x7f1668001c20 "Connection", v=0x7f1668001c2c "keep-alive")
at /home/noone/public_git/github.com/jkasarherou/oio-sds/proxy/transport_http.c:624
#3  0x0000000000419216 in _manage_and_renew_header (parser=0x7f1668001a00, buf=0x102a140)
at /home/noone/public_git/github.com/jkasarherou/oio-sds/proxy/transport_http.c:151
#4  0x00000000004193e1 in http_parse (
data=0x7f1660001a00 "GET / HTTP/1.1\r\nHost: 127.0.0.1:6002\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: HTTPie/0.9.2\r\n\r\n", available=135) at /home/noone/public_git/github.com/jkasarherou/oio-sds/proxy/transport_http.c:198
#5  0x000000000041aa20 in http_notify_input (clt=0x7f1668001a50) at /home/noone/public_git/github.com/jkasarherou/oio-sds/proxy/transport_http.c:668
#6  0x00007f1671d2f7c6 in _notify () at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/network_server.c:1439
#7  0x00007f1671d2f8a0 in _client_manage_input () at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/network_server.c:1463
#8  0x00007f1671d2fa61 in _client_manage_event (clt=0x7f1668001a50, events=1)
at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/network_server.c:1533
#9  0x00007f1671d2e96c in _work_on_client (srv=0x1028cc0, clt=0x7f1668001a50)
at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/network_server.c:1039
#10 0x00007f1671d2eb7d in _thread_cb_worker (td=0x1028cc0) at /home/noone/public_git/github.com/jkasarherou/oio-sds/server/network_server.c:1088
#11 0x00007f1670c4c625 in ?? () from /usr/lib/libglib-2.0.so.0
#12 0x00007f166eb2a4a4 in start_thread () from /usr/lib/libpthread.so.0
#13 0x00007f166f09913d in clone () from /usr/lib/libc.so.6

proxy/meta2: container properties need review

We currently have something like this for meta2 containers properties:

{
"schema_version": "1.8",
"sys.account": "test",
"sys.m2.ctime": "1435593742",
"sys.m2.init": "1",
"sys.m2.usage": "-598",
"sys.m2.version": "10",
"sys.name": "AEA6D3A7C15F0E5C3E61A68DB8DD729637A9890C67DC2ED41BAFCD1BFCA1A12A.1",
"sys.ns": "NS",
"sys.type": "meta2",
"sys.user.name": "1",
"version:main.admin": "1:0",
"version:main.alias_v2": "11:0",
"version:main.chunk_v2": "11:0",
"version:main.content_header_v2": "11:0",
"version:main.content_v2": "11:0",
"version:main.properties_v2": "1:0"
}

Looks like it could use a refactoring/cleaning.

meta2 test converter: segfault

segfault while running test_meta2_type_converter

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `./test_meta2_type_converter'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f02beab35f8 in raise () from /usr/lib/libc.so.6
(gdb) bt
#0  0x00007f02beab35f8 in raise () from /usr/lib/libc.so.6
#1  0x00007f02beab4a7a in abort () from /usr/lib/libc.so.6
#2  0x00007f02c20288c5 in g_assertion_message () from /usr/lib/libglib-2.0.so.0
#3  0x00007f02c202910e in g_assertion_message_error () from /usr/lib/libglib-2.0.so.0
#4  0x000000000040254a in _repo_wraper (ns=0x402c60 "NS", fr=0x7fff367d2738)
at /home/noone/public_git/github.com/jkasarherou/oio-sds/meta2v2/test_meta2_type_converter.c:174
#5  0x00000000004027e7 in _container_wraper (cf=0x7fff367d2760)
at /home/noone/public_git/github.com/jkasarherou/oio-sds/meta2v2/test_meta2_type_converter.c:215
#6  0x0000000000402866 in test_beans_to_raw_content () at /home/noone/public_git/github.com/jkasarherou/oio-sds/meta2v2/test_meta2_type_converter.c:227
#7  0x00007f02c202833b in ?? () from /usr/lib/libglib-2.0.so.0
#8  0x00007f02c2028503 in ?? () from /usr/lib/libglib-2.0.so.0
#9  0x00007f02c2028503 in ?? () from /usr/lib/libglib-2.0.so.0
#10 0x00007f02c202870e in g_test_run_suite () from /usr/lib/libglib-2.0.so.0
#11 0x00007f02c2028731 in g_test_run () from /usr/lib/libglib-2.0.so.0
#12 0x0000000000402a0c in main (argc=1, argv=0x7fff367d2b28) at /home/noone/public_git/github.com/jkasarherou/oio-sds/meta2v2/test_meta2_type_converter.c:277

Proxy/meta2 : too many answers on get container with a max parameter

When doing a get request on containers with the 'max' parameters set to N, the response contains N+1 contents.

meta2 : Problem during base deletion

In a replicate mode, when you delete a meta2 base, sometimes a sqlite file persists on one of the involved node.

meta0-init not initializing every meta0 correctly

During a meta0-init, even if the meta0s are registered into the conscience, some of them might not be correctly initialized.
This problem arrived on AWS EC2 instances and is reproductible.

Is there a way to re-init a meta0 that wasn't initialized in the middle of other initialized meta0s ?

metautils test gridd_client: ERROR

1444861478.220 28666 8A99 log ERR G   duplicate test case path: /metautils/gridd_client/ignored_connect/fail_start

Proxy: keep a cache of services down

The proxy might cache entries from the directory.

Currently, when the proxy attempts to access services listed in the directory, it resolves those services an serially tries each service until a final answer is received.

When the host carrying a service from the list is down, we will reach a connection timeout. Even if this timeout is short, it takes a lot of time when compared to the expected latency for a valid service.

We should keep track of problematic services, to prioritize services not marked as problematic.

Proxy/meta1 : cleaning the meta1 cache between tests

The present tests on meta1 linking functions can't be perfectly run as long as there is no way to clean the meta1 cache before each test.
Indeed, those tests often reach a point where a lot of services, which were used during previous tests and couldn't be deleted from the meta1, take the place of currently used services and make unexpected results.

Is it possible to improve that situation ?

Test grid_client.h : parameters not checked while setting properties

The current functions hc_set_container_storage_policy, hc_set_container_versioning and hc_set_container_quota don't check its arguments before setting them. It is so possible to put any random string in a container's storage policy or versioning field.

That issue could be fixed while treating the #77.

Why does the SVC monitor use "select(0, NULL, NULL, &sleep_time)" ?

Hello :-)

I'm looking through the source to improve error reporting within the SVC monitor tool, because I have errors when it tries to start Apache but these errors are not logged anywhere, from what I can see. Is that correct ?

Anyway, I noticed the svc-monitor seems to use select (2), as can be seen here, for something where I'm thinking a simple call to sleep (3) would be appropriate.

What is the reason behind that ?

Thanks for your help,
Best regards,
Conrad

Test grid_client.h : Can’t change a content property with hc_set_content_property

After having used the hc_set_content_property to put a property to a content e.g. “key1=value1”, I tried to modify the value by calling again the hc_set_content_property on the content, with the parameter “key1=value2”.
When I tried to get the key1 value from that content, the calling function returned value1.
So set_content_property doesn’t change the value of an existing property.

Is it a bug or is the set_content function designed to run that way ?

Log improvement on meta2 not master response

When requesting a non master meta2, the log trace is :

2015-08-28T09:12:33.206642+02:00 SERVER OIO,NS,meta2,6: info  24060 FBFC access INF 127.0.0.1:6006 127.0.0.1:52180 M2V2_DEL 303 0.000105 0 9D46D57BC31C7E4FA9DCA7E34CA838482EA9BDEF5558B1127C1CD77979670041 3716290A3D668E148C2380E003D60F28 t=0.000059 NS/AUTH_test/container_1//object_1|9D46D57BC31C7E4FA9DCA7E34CA838482EA9BDEF5558B1127C1CD77979670041 e=(303) 127.0.0.1:6006

It could be something more comprehensive than "e=(303)" like "not master redirecting"

Test rawx-apache2 : no error raised when a content is put without all its headers

When sending a data put request to a rawx while forgetting at least one header (e.g. "chunk-id", "content_path" or "content_containerid"), the request is treated without raising any error. So data can be saved with incomplete location informations.

Could it be possible to implement a procedure to check the presence of every content header when sent via put requests ?

proxy: refactor request/response format

Actually, the proxy uses a mix of headers, query parameters, and field JSON encoded in the request body. We should stop using headers at all (we have full control on the request/response body format) and  use JSON.
Example: To set properties on a container creation:
instead of sending parameters as headers (which is confusing for parameters case sensitivity), we should simply use the JSON encoded body.

proxy: remove 2.0 routes support

It is time to remove 2.0 routes support from the proxy. Useful for #184

core / oio_sds : download improvement

The download currently implemented is really simplistic. Among others, it lacks:

• the download of just a specified range
• the download into a buffer.

Feel free to comment to add your own wishes on the download.

meta2 test backend: assertion failed

oio.m2v2:ERROR:/home/noone/public_git/github.com/jkasarherou/oio-sds/meta2v2/test_meta2_backend.c:263:_repo_wraper: assertion failed (err == NULL): XATTR set error: Operation not supported (oio.utils, 95)

oio-account-server is too verbose when an exception is raised

Here is a sample output when I stop the redis server:

Aug 31 23:59:29 localhost OIO,NS,account,1[18010]: Error handling request
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/gunicorn/workers/async.py", line 47, in handle
self.handle_request(listener_name, req, client, addr)
File "/usr/lib/python2.7/site-packages/gunicorn/workers/async.py", line 100, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/usr/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/usr/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/jfs/public_git/github.com/jfsmig/oio-sds/python/oio/account/server.py", line 40, in status
status = get_backend().status()
File "/home/jfs/public_git/github.com/jfsmig/oio-sds/python/oio/account/backend.py", line 299, in status
account_count = conn.hlen('accounts:')
File "/usr/lib/python2.7/site-packages/redis/client.py", line 1846, in hlen
return self.execute_command('HLEN', name)
File "/usr/lib/python2.7/site-packages/redis/client.py", line 570, in execute_command
connection.send_command(*args)
File "/usr/lib/python2.7/site-packages/redis/connection.py", line 556, in send_command
self.send_packed_command(self.pack_command(*args))
File "/usr/lib/python2.7/site-packages/redis/connection.py", line 532, in send_packed_command
self.connect()
File "/usr/lib/python2.7/site-packages/redis/connection.py", line 436, in connect
raise ConnectionError(self._error_message(e))
ConnectionError: Error 111 connecting to 127.0.0.1:6379. ECONNREFUSED.

This is repeated once per second and completely fills the logs.

Proxy/meta2 : NS_Proxy broken when unlinking reference with missing parameter

After having created a reference and linked it with a meta2 service type, I tried to unlink the reference with no "service-type" parameter (unauthorized action).
It resulted in aborting the NS-proxy process.

Here is the execution trace :

Core was generated by `oio-proxy -s OIO,NS,proxy -O Bind=/home/blextrait/.oio/sds/run/NS-proxy.sock 19'.
> Program terminated with signal 6, Aborted.
> # 0  0x00007f003bd665d7 in raise () from /lib64/libc.so.6
> 
> Missing separate debuginfos, use: debuginfo-install gf-complete-1-1.el7.oio.x8664 glib2-2.40.0-4.el7.x8664 glibc-2.17-78.el7.x8664 jerasure-2-1.el7.oio.x8664 json-c-0.11-4.el70.x8664 openio-asn1c-0.9.27-1.el7.oio.x8664 openio-sds-librain-0.8-1.el7.oio.x8664 sqlite-3.7.17-4.el7.x86_64
> (gdb) bt
> # 0  0x00007f003bd665d7 in raise () from /lib64/libc.so.6
> # 1  0x00007f003bd67cc8 in abort () from /lib64/libc.so.6
> # 2  0x00007f003d3618a5 in gassertionmessage () from /lib64/libglib-2.0.so.0
> # 3  0x00007f003d36193a in gassertionmessage_expr ()
> 
>    from /lib64/libglib-2.0.so.0
> # 4  0x00007f003dc3436e in meta1v2remoteunlink_service (meta1=0x7f003b2ca810,
> 
> err=0x7f003b2ca808, url=0x7f0024002db0, srvtype=0x0) at /home/blextrait/public_git/github.com/oio-sds/meta1v2/meta1_remote.c:269
> # 5  0x000000000040e806 in hook (m1=0x7f002400b438 "192.168.56.101:6007")
> 
> ```
at /home/blextrait/public_git/github.com/oio-sds/proxy/dir_actions.c:189
```
> # 6  0x000000000040e2f3 in m1action (args=0x7f003b2ca970, m1v=0x7f002400b5e0,
> 
> ```
hook=0x7f003b2ca900)
at /home/blextrait/public_git/github.com/oio-sds/proxy/dir_actions.c:86
```
> # 7  0x000000000040e432 in m1locateandaction (args=0x7f003b2ca970,
> 
> ```
hook=0x7f003b2ca900)
at /home/blextrait/public_git/github.com/oio-sds/proxy/dir_actions.c:110
```
> # 8  0x000000000040e895 in actiondirsrv_unlink ()
> 
> ```
at /home/blextrait/public_git/github.com/oio-sds/proxy/dir_actions.c:193
```
> # 9  0x000000000041026e in actionrefunlink (args=0x7f003b2ca970)
> 
> ```
at /home/blextrait/public_git/github.com/oio-sds/proxy/dir_actions.c:627
```
> # 10 0x000000000040a6e5 in handler_action (u=0x2039160, rp=0x7f003b2cab30)
> 
> ```
at /home/blextrait/public_git/github.com/oio-sds/proxy/metacd_http.c:216
```
> # 11 0x0000000000419fb6 in httpmanagerequest ()
> 
> ```
at /home/blextrait/public_git/github.com/oio-sds/proxy/transport_http.c:583

Load-Balancing: a necessary review to reduce its complexity

Currently, the load-balancing has several drawbacks:

• It plays with service_info_s, and it is one of the last parts of the code to do so. service_info_s is the structure that notoriously prevent us to use logical ID for services instead of network addresses.
• The current complexity trade-off between time and space clearly targets space. This is a bad idea since one service item is rather small.
• Under certain circumstances, the output is not guaranteed. This can happen if the election criterions target too few services in a huge set of not-targetted services with the same type.
• The implementation is rather hard to read, and harder to extend.
• The API is hard to use.
• The current implementation of the locations is to fuzzy to be really efficient with decent compute times.

I propose to reorganize this API to make it easy to use (simple API), with guaranteed output and response time (better internal structures, stronger locations):

• Build pools of services, gathering all the services pollable in the same call. In other words: compatble  types / tags / class
• In a pool, store the services in per-location-buckets: into a bucket, all the services are considered too close to be polled in the same API call.
• Poll services from a pool, with only parameters the number of services targeted and locations constraints.

Test gridclient.h : trying to link a service and an already linked reference

When using the function hc_force_service and trying to link a new service with a reference I already linked before, I got the following error message :

(code=502) Error code not managed <502> : Handling: Request error: (SQLITE_CONSTRAINT) columns cid, srvtype, seq are not unique

It indicates that my new service has been denied because it looks like an existing service already linked to the reference.
However, in my code, I have already linked the reference with three services whose url respectively ends with 6010/6011/6012. The new service url has indeed the same cid and srvtype but ends with 6048.

The same test with an unlinked reference works perfectly.

Am I missing something like a restriction in that function’s use ? Is it a bug ?

Test grid_client.h : wrong error code on container not found

A lot of grid_client functions which use to work perfectly now return "error 406 : code not managed" when called with a wrong reference argument, instead of returning "error 431 : container not found" like before.

Here are some of the concerned functions :

Has_reference
Link_service_to_reference
Unlink_reference_service
List_reference_service
Poll_service
Force_service
Configure_service

Proxy/meta2 : error code 500 on set container property with an invalid key

When trying to set a container property with an invalid key ( = not included in the authorized property key parameters ), the http error code is 500 instead of 400 even if the answer's content is correct.

Test grid_client.h : calling hc_get_reference_property with bad args

When calling the hc_get_reference_property with an unexisting property key on the reference, the function returns all the reference's properties instead of raising an error.

Proxy/meta2 : container auto-created but not initialized

When a reference is linked for the first time with a service, a corresponding container is automatically created but not initialized.
This is confusing because a lot of put/get/delete meta2 functions will consider that ghost container as existing and will bring unexpected results.

The two solutions could be either initializing those auto-created containers or restrict container creation to the put method only.

Switching the versioning settings on a running container

Hey everyone,

Thanks for providing a Vagrant box for SDS. Makes it easy to get started ! 👍

I am trying to use the versioning function of SDS. At the moment, it works if I create a container with the -O ActivateVersioning=-1 option. However, if I don't use this option and use something like oio version NS/container -1 or oio version NS/container -- -1, it doesn't seem to change the versioning settings as described in the command like help message.

Here's a full example (with verbose output) of what I mean. I would have expected the second call to oio put to create a new version of the file instead of failing:

[openio@openio ~]$ oio version -v OPENIO/conradk -- -1
1431212546.204 7518 ADB4 log DBG h.t  Configuration
1431212546.204 7518 ADB4 log DBG g.u.m Process initiation successful
1431212546.204 7518 ADB4 log DBG g.c   Creating a new GridStorage client for namespace [OPENIO]
1431212546.205 7518 ADB4 log DBG g.c.r.m Creating a METAcd resolver for OPENIO
1431212546.205 7518 ADB4 log DBG g.c.r.m METACD using sock=/var/run/metacd.sock namespace=OPENIO timeout[cnx:5000 op:10000] (you may overload the sock with environment key GS_METACD_SOCKET)
1431212546.205 7518 ADB4 log DBG g.c.r.l Explicit resolver created for OPENIO, META0 got from gridcluster
1431212546.205 7518 ADB4 log DBG g.c     Resolution attempt NAME=[conradk] ID=[51CDB7BD42D91F5C303B460EF0983E3F030D739B6121A60722E732E4755E78F1]
1431212546.205 7518 ADB4 log DBG g.c.r.l META0 cache reload wanted
1431212546.220 7518 ADB4 log DBG g.c.r.l META0 cache has been refreshed
1431212546.221 7518 ADB4 log DBG g.c.r.l Meta2 services listed
1431212546.221 7518 ADB4 log DBG g.c.r.l Got meta2=1|meta2|172.28.128.3:6005|
1431212546.221 7518 ADB4 log DBG g.c.r.l Got meta2=1|meta2|172.28.128.3:6006|
1431212546.221 7518 ADB4 log DBG g.c.r.l Got meta2=1|meta2|172.28.128.3:6007|
1431212546.221 7518 ADB4 log DBG g.c     Trying to connect to meta2 [172.28.128.3:6007]
1431212546.221 7518 ADB4 log DBG m.m     Invalid parameter (0x24d55e0 CONTENT_PATH (nil))
1431212546.221 7518 ADB4 log DBG m.m     Invalid parameter (0x24d54b0 CONTENT_PATH (nil))
Versioning applied to [OPENIO/conradk]
1431212546.234 7518 ADB4 log DBG g.c.r.l freeing 0x1b55d60
1431212546.238 7518 ADB4 log DBG h.t     Exiting
[openio@openio ~]$ oio put -v OPENIO/conradk /etc/hostname
1431212553.734 7519 BD1B log DBG h.t  Configuration
1431212553.735 7519 BD1B log DBG g.u.m Process initiation successful
1431212553.735 7519 BD1B log DBG g.c   Creating a new GridStorage client for namespace [OPENIO]
1431212553.736 7519 BD1B log DBG g.c.r.m Creating a METAcd resolver for OPENIO
1431212553.736 7519 BD1B log DBG g.c.r.m METACD using sock=/var/run/metacd.sock namespace=OPENIO timeout[cnx:5000 op:10000] (you may overload the sock with environment key GS_METACD_SOCKET)
1431212553.737 7519 BD1B log DBG g.c.r.l Explicit resolver created for OPENIO, META0 got from gridcluster
1431212553.737 7519 BD1B log DBG h.t     Local path /etc/hostname found
1431212553.737 7519 BD1B log DBG h.t     Local path /etc/hostname found and opened
1431212553.737 7519 BD1B log DBG g.c     Resolution attempt NAME=[conradk] ID=[51CDB7BD42D91F5C303B460EF0983E3F030D739B6121A60722E732E4755E78F1]
1431212553.737 7519 BD1B log DBG g.c.r.l META0 cache reload wanted
1431212553.754 7519 BD1B log DBG g.c.r.l META0 cache has been refreshed
1431212553.755 7519 BD1B log DBG g.c.r.l Meta2 services listed
1431212553.755 7519 BD1B log DBG g.c.r.l Got meta2=1|meta2|172.28.128.3:6005|
1431212553.755 7519 BD1B log DBG g.c.r.l Got meta2=1|meta2|172.28.128.3:6006|
1431212553.755 7519 BD1B log DBG g.c.r.l Got meta2=1|meta2|172.28.128.3:6007|
1431212553.755 7519 BD1B log DBG g.c     Trying to connect to meta2 [172.28.128.3:6007]
1431212553.755 7519 BD1B log DBG m.m     Invalid parameter (0x1b46630 CONTENT_PATH (nil))
1431212553.755 7519 BD1B log DBG g.c.u   Timeout cnx 60 s ; timeout op 90 s
1431212553.757 7519 BD1B log DBG g.c.u   'hostname' of size 7 split into 1 chunks (every chunk is duplicated 3 times, there are 1 unique presets and 0 spares)
1431212553.757 7519 BD1B log DBG g.c.u   Position 0:
1431212553.757 7519 BD1B log DBG g.c.u   => http://172.28.128.3:6008/B22831827BB77F4CE8E79B0BD197B1A7DD28D4BAB2808E3D3073AF6BDE7053CE (7 bytes)
1431212553.757 7519 BD1B log DBG g.c.u   => http://172.28.128.3:6009/C7783B088255A32B30F7C8AF992BF2DE3422ABE9619DAD6DCB08DB3EEB3BA618 (7 bytes)
1431212553.757 7519 BD1B log DBG g.c.u   => http://172.28.128.3:6010/F4EA290E140C3AD6073FCA745A4B0149BA554413964F6B9B91D62095EA3ECFB5 (7 bytes)
1431212553.757 7519 BD1B log DBG h.p     Starting UPLOAD
1431212553.768 7519 BD1B log DBG h.p     DONE! http://172.28.128.3:6008/B22831827BB77F4CE8E79B0BD197B1A7DD28D4BAB2808E3D3073AF6BDE7053CE => HTTP ret: 201 => CURL ret: No error
1431212553.768 7519 BD1B log DBG h.p     DONE! http://172.28.128.3:6010/F4EA290E140C3AD6073FCA745A4B0149BA554413964F6B9B91D62095EA3ECFB5 => HTTP ret: 201 => CURL ret: No error
1431212553.773 7519 BD1B log DBG h.p     DONE! http://172.28.128.3:6009/C7783B088255A32B30F7C8AF992BF2DE3422ABE9619DAD6DCB08DB3EEB3BA618 => HTTP ret: 201 => CURL ret: No error
1431212553.773 7519 BD1B log DBG g.c.u   whole upload successful
1431212553.786 7519 BD1B log DBG g.c.u   version of the content: 0
1431212553.786 7519 BD1B log DBG h.t     Uploaded a new version of content [hostname] in container [conradk]
1431212553.786 7519 BD1B log DBG g.c.r.l freeing 0x11c6d90
1431212553.790 7519 BD1B log DBG h.t     Exiting
[openio@openio ~]$ oio put -v OPENIO/conradk /etc/hostname
1431212555.099 7520 ED4B log DBG h.t  Configuration
1431212555.099 7520 ED4B log DBG g.u.m Process initiation successful
1431212555.099 7520 ED4B log DBG g.c   Creating a new GridStorage client for namespace [OPENIO]
1431212555.100 7520 ED4B log DBG g.c.r.m Creating a METAcd resolver for OPENIO
1431212555.100 7520 ED4B log DBG g.c.r.m METACD using sock=/var/run/metacd.sock namespace=OPENIO timeout[cnx:5000 op:10000] (you may overload the sock with environment key GS_METACD_SOCKET)
1431212555.100 7520 ED4B log DBG g.c.r.l Explicit resolver created for OPENIO, META0 got from gridcluster
1431212555.100 7520 ED4B log DBG h.t     Local path /etc/hostname found
1431212555.100 7520 ED4B log DBG h.t     Local path /etc/hostname found and opened
1431212555.100 7520 ED4B log DBG g.c     Resolution attempt NAME=[conradk] ID=[51CDB7BD42D91F5C303B460EF0983E3F030D739B6121A60722E732E4755E78F1]
1431212555.100 7520 ED4B log DBG g.c.r.l META0 cache reload wanted
1431212555.117 7520 ED4B log DBG g.c.r.l META0 cache has been refreshed
1431212555.119 7520 ED4B log DBG g.c.r.l Meta2 services listed
1431212555.119 7520 ED4B log DBG g.c.r.l Got meta2=1|meta2|172.28.128.3:6005|
1431212555.119 7520 ED4B log DBG g.c.r.l Got meta2=1|meta2|172.28.128.3:6006|
1431212555.119 7520 ED4B log DBG g.c.r.l Got meta2=1|meta2|172.28.128.3:6007|
1431212555.119 7520 ED4B log DBG g.c     Trying to connect to meta2 [172.28.128.3:6007]
1431212555.119 7520 ED4B log DBG m.m     Invalid parameter (0x1e49630 CONTENT_PATH (nil))
1431212555.120 7520 ED4B log DBG g.c.u   Timeout cnx 60 s ; timeout op 90 s
1431212555.120 7520 ED4B log DBG h.t     Technical error (code 421) message:
1431212555.120 7520 ED4B log DBG h.t     (code=421) Cannot perform the whole upload Request error: Alias already present

ERROR: Content [hostname] already exists in container [conradk]

(Run action with -v option for technical details.)
1431212555.121 7520 ED4B log DBG h.t     Action error
1431212555.121 7520 ED4B log DBG g.c.r.l freeing 0x14c9d90
1431212555.129 7520 ED4B log DBG h.t     Exiting

I also tried something like oio version NS/container 5 just in case the -1 was conflicting with an option named 1, but it didn't seem to work either. I tried restarting the VM, but on restart, it seems like the container still does not version files.

Am I missing something ? What is oio version meant to do ?

Best regards,
Conrad

Proxy/meta2 : segfault on content put with invalid headers

The put content function needs two mandatory headers to run : 'x-oio-content-meta-hash' and 'x-oio-content-meta-length'. If the given hash header is somehow incoherent, sending the put request entails a segfault and the proxy crashes.

Default loadbalancing configuration

Default loadbalancing configuration should be set to WRAND instead of RR.

oio-meta0-init, confusing error message

When using oio-meta0-init with option -O IgnoreDistance=off and there is not enough meta1 services available the error message is:

META0 request error (502) : Handling: Request error: Failed to assign prefix 0 to meta1

This is not very helpful, could be something like "Not enough meta1 services available to respect distance constraint"

Storage Policy fallback?

Current storage policy implementation does not have an option to specify a possible fallback.
Example:
In such storage policy configuration:

[STORAGE_POLICY]
THREECOPIES=NONE:DUPONETHREE:NONE
[DATA_SECURITY]
DUPONETHREE=DUP:distance=1|nb_copy=3

When there is only 2 rawx services available it is impossible to create a new file, but in some cases we may want to fallback on a smaller number of chunk copies.

Ubuntu server 15.04 : oio-account-server crashes when launched by gridinit

The oio-account-server process ends with segfault loop when launched by gridinit (e.g. during the oio-reset procedure) while it perfectly works when manually started.

That could have something to do with the issue #102

Not sure it could serve, but here is the trace :

Core was generated by `/usr/bin/python /usr/local/bin/oio-account-server /home/openio/.oio/sds/conf/NS'.
Program terminated with signal SIGSEGV, Segmentation fault.

0  0x0000000000492b6a in ?? ()

(gdb) bt
0  0x0000000000492b6a in ?? ()
1  0x00000000004cdcce in PyEval_EvalFrameEx ()
2  0x00000000004cd4e2 in PyEval_EvalFrameEx ()
3  0x00000000004cb6b1 in PyEval_EvalCodeEx ()
4  0x00000000004ce7d3 in PyEval_EvalFrameEx ()
5  0x00000000004cd4e2 in PyEval_EvalFrameEx ()
6  0x00000000004cd4e2 in PyEval_EvalFrameEx ()
7  0x00000000004cd4e2 in PyEval_EvalFrameEx ()
8  0x00000000004cd4e2 in PyEval_EvalFrameEx ()
9  0x00000000004cd4e2 in PyEval_EvalFrameEx ()
10 0x00000000004cd4e2 in PyEval_EvalFrameEx ()
11 0x00000000004cb6b1 in PyEval_EvalCodeEx ()
12 0x00000000004caf56 in PyEval_EvalCode ()
13 0x00000000004d134e in PyEval_EvalFrameEx ()
14 0x00000000004cb6b1 in PyEval_EvalCodeEx ()
15 0x000000000050481f in ?? ()
16 0x00000000004fc182 in PyRun_FileExFlags ()
17 0x00000000004fb247 in PyRun_SimpleFileExFlags ()
18 0x000000000049aa6e in Py_Main ()
19 0x00007f25da1b6a40 in __libc_start_main (main=0x49a500

, argc=3,
     argv=0x7fff67bbc1a8, init=, fini=,
     rtld_fini=, stack_end=0x7fff67bbc198) at libc-start.c:289
20 0x000000000049a429 in _start ()

Ubuntu server 15.04 : oio-reset lasts until one minute

The oio-reset procedure on an Ubuntu server system often lasts until one minute instead of the around ten seconds expected.
Two times on 3, the procedure starts repeating dozens of times the following instructions :

pkill --full -0 gridinit
pkill --full gridinit
sleep 2

It seems that some gridinit process resists the kill procedure.
When using a ps command to find those running during the loop, I always find these ones :

oio-svc-monitor -s OIO,NS,account,1 -p 1 -m oio-account-monitor.py -i NS|account|192.168.56.102:6005 -c oio-account-server /home/openio/.oio/sds/conf/NS-account-server.conf

/usr/bin/python /home/openio/build/bin/oio-account-monitor.py NS|account|192.168.56.102:6005

Is it possible to improve the oio-reset procedure to solve that delay problem ?

URLencoded path uncorrectly stored

URLencoded files are uncorrectly stored when using the Swift proxy. Julien suspects a problem in oioproxy.
This is an issue when using Pydio.

System metadata/Properties in meta2

System metadata/Properties interactions in meta2 should be simpler

we have for contents things like:
"system_metadata":"mime-type=octet/stream;chunk-method=chunk-size;creation-date=1435597533"
"ctime":1435597533
"hash":"BB3CA60759F3202F1AE42E3519CD06BC"
"size":7650
"deleted":false
"ver":1
"policy":"none"

The "system_metadata" thing is ugly

Configuration parsing fail in account functional test

The configuration parser fail during functional test.

The issue appears in this two files :
https://github.com/open-io/oio-sds/blob/master/python/tests/functional/account/test_backend.py#L24
https://github.com/open-io/oio-sds/blob/master/python/tests/functional/account/test_server.py#L25

Test setUp() try to load the file /root/.oio/sds/conf/test.conf with SafeConfigParser, but the file is in json format (produced by oio-bootstrap.sh).

Got this error : "MissingSectionHeaderError: File contains no section headers."

meta2: bytes count in container is not working properly

The bytes count in meta2 is not working properly when content is replaced (overwritten), we have a negative value after a while.

Test grid_client.h : update needed on hc_set_content_storage_policy

The current function hc_set_content_storage_policy uses both a container and a path_content arguments, and doesn't work any more. However it's now possible to find a content's container address from the content's structure.
That function can so be fixed and updated to work like the others, by just using the content argument.