Can`t upload program. Cocaine + elliptics

[cocaine]

Can`t upload program. Cocaine + elliptics

[noxiouz]

That's about the tools its about Elliptics. It's not properly configured or has some issues with start.
I assume, that you should check the state of Elliptics.
Whaich version of Elliptics do u use? Also check the group number in the Elliptics configuration and the storage configuration.
If you provide the whole config files somehow, I will dig

[TimBobkov]

Thanks for the quick response.
Version of Elliptics:
~$ dpkg -l | grep elli
ii elliptics 2.26.7.0 Distributed hash table storage
ii elliptics-client 2.26.7.0 Distributed hash table storage (client library)
ii libcocaine-plugin-elliptics 2.26.7.0 Distributed hash table storage (cocaine plugin)

Config file:

{
        "logger": {
                "level": "debug",
                "frontends": [
                        {
                                "formatter": {
                                        "type": "string",
                                        "pattern": "%(timestamp)s %(request_id)s/%(lwp)s/%(pid)s %(severity)s: %(message)s %(...L)s"
                                },
                                "sink": {
                                        "type": "files",
                                        "path": "/dev/stdout",
                                        "autoflush": true,
                                        "rotation": {
                                                "move": 0
                                        }
                                }
                        }
                ]
        },
        "options": {
                "join": true,
                "flags": 20,
                "remote": [
                        "autodiscovery:224.0.0.5:1025:2"
                ],
                "address": [
                        "localhost:1025:2-0", "178.62.231.179:1025:2-1"
                ],
                "wait_timeout": 60,
                "check_timeout": 60,
                "io_thread_num": 16,
                "nonblocking_io_thread_num": 16,
                "net_thread_num": 4,
                "daemon": false,
                "auth_cookie": "qwerty",
                "bg_ionice_class": 3,
                "bg_ionice_prio": 0,
                "server_net_prio": 1,
                "client_net_prio": 6,
                "cache": {
                        "size": 68719476736
                },
                "srw": "/usr/share/doc/elliptics/examples/library_config.json",
                "indexes_shard_count": 2,
                "monitor": {
                        "port":20000
                }
        },
        "backends": [
                {
                        "backend_id": 1,
                        "group": 2,
                        "type": "blob",
                        "history": "/opt/elliptics/history.2",
                        "data": "/opt/elliptics/eblob.2/data",
                        "sync": "-1",
                        "blob_flags": "158",
                        "blob_size": "2G",
                        "records_in_blob": "1000000"
                }
        ]
}

[noxiouz]

Which version of Cocaine 11/12? Also the config of Cocaine-runtime would be useful.
Could you also run this
dnet_client backend -r hostname -f:1025:2 status and tesh that elliptics backend is enabled

[TimBobkov]

ii cocaine-dbg 0.11.3.2
ii cocaine-framework-native 0.11.1.3
ii cocaine-framework-native-dbg 0.11.1.3
ii cocaine-framework-native-dev 0.11.1.3
ii cocaine-runtime 0.11.3.2
ii libcocaine-core2 0.11.3.2
ii libcocaine-dev 0.11.3.2
ii libcocaine-plugin-docker2 0.11.2.5

[TimBobkov]

Result of command:
2015-10-08 05:17:17.849151 0000000000000000/2454/2454 ERROR: 127.0.1.1:1025: there is no connected addr in received reverse lookup data, attrs: ['source': 'dnet_add_state']
2015-10-08 05:17:17.851073 0000000000000000/2454/2454 ERROR: Coudn't connect to: ['Pe4eHbKa2:1025:2']: Error(Failed to add remote 1 addrs: Connection refused: -111,), attrs: []
2015-10-08 05:17:17.852076 0000000000000000/2459/2454 ERROR: 127.0.1.1:1025: there is no connected addr in received reverse lookup data, attrs: ['source': 'dnet_add_state']
2015-10-08 05:17:17.853297 0000000000000000/2454/2454 ERROR: 0:000000000000...000000000000: 127.0.1.1:1025: trans_send: could not find network state for address, attrs: []
2015-10-08 05:17:17.853521 0000000000000000/2454/2454 ERROR: 0:000000000000...000000000000: handled reply from: 127.0.1.1:1025, cmd: BACKEND_STATUS, flags: 0x9 [need_ack|direct], trans: 0, status: -6, size: 0, client: 1, last: 1, attrs: []
{
"error": {
"message": "insufficient results count due to checker: 0 of 1 (1): No such device or address: -6",
"code": -6
}
}

[noxiouz]

You have to check, that this
"localhost:1025:2-0", "178.62.231.179:1025:2-1"
mathes to addresses in config of cocaine-runtime.
Also

dnet_client backend -r localhost:1025:2 status
dnet_client backend -r 178.62.231.179:1025:2 status

[TimBobkov]

~$ dnet_client backend -r localhost:1025:2 status

{
    "backends": [
        {
            "backend_id": 1,
            "state": 0,
            "readonly": false,
            "defrag_state": 0,
            "last_start_err": -22,
            "last_start": "2015-10-08 05:25:13.354497"
        }
    ]
}

~$ dnet_client backend -r 178.62.231.179:1025:2 status

{
    "backends": [
        {
            "backend_id": 1,
            "state": 0,
            "readonly": false,
            "defrag_state": 0,
            "last_start_err": -22,
            "last_start": "2015-10-08 05:25:13.354497"
        }
    ]
}

[noxiouz]

Are the addresses the same in the config of cocaine-runtime? Post config here

[TimBobkov]

Yes, addresses are same.
Here's a listing of cocaine.conf just in case:

{
    "version": 2,
    "paths": {
        "plugins": "/usr/lib/cocaine",
        "runtime": "/var/run/cocaine"
    },
    "services": {
        "logging": {
            "type": "logging"
        },
        "storage": {
            "type": "elliptics",
            "args": {
                "backend": "core"
            }
        },
        "node": {
            "type": "node",
            "args": {
                "runlist": "default"
            }
        }
    },
    "storages": {
        "core": {
            "type": "elliptics",
            "args": {
                "nodes": [
                        "localhost:1025:2", "178.62.231.179:1025:2"
                ],
                "groups": [2]
            }
        }
    },
    "loggers": {
        "core": {
            "type": "syslog",
            "args": {
                "identity": "cocaine",
                "verbosity": "debug"
            }
        }
    }
}

[noxiouz]

Hmm, looks strange.
Let's do some dubeb:
replace "core" in "loggers":

{
"type": "files",
                "args": {
                    "identity": "cocaine",
                    "path" : "path_to_some_file.log",
                    "verbosity": "debug"
                 }
}

Restart cocaine and look into this file. There should be some errors about connecting to Elliptics

[TimBobkov]

List of errors:

[Thu Oct 8 06:34:10 2015] [DEBUG] storage/core: reading the 'default' object, collection: 'runlists'
[Thu Oct 8 06:34:10 2015] [ERROR] storage/core: elliptics: 2:f47326781f3f...9eeef62db553: could not find network state for request
[Thu Oct 8 06:34:10 2015] [ERROR] storage/core: elliptics: 2:f47326781f3f...9eeef62db553: handled reply from: , cmd: READ, flags: 0x1 [need_ack], trans: 0, status: -6, size: 0, client: 1, last: 1
[Thu Oct 8 06:34:10 2015] [DEBUG] storage/core: elliptics: Destroying session.
[Thu Oct 8 06:34:10 2015] [DEBUG] storage/core: elliptics: Destroying session.
[Thu Oct 8 06:34:10 2015] [WARNING] service/node: unable to read the 'default' runlist - insufficient results count due to checker: 0 of 1 (1): No such device or address: -6
[Thu Oct 8 06:34:10 2015] [DEBUG] storage/core: elliptics: Destroying session.
[Thu Oct 8 06:34:10 2015] [DEBUG] storage/core: elliptics: Destroying node.

[Thu Oct 8 06:34:09 2015] [DEBUG] storage/core: listing collection: 'groups'
[Thu Oct 8 06:34:09 2015] [INFO] storage/core: elliptics: started io thread: #0, nonblocking: 0, backend: -1
[Thu Oct 8 06:34:09 2015] [DEBUG] storage/core: elliptics: INDEXES_FIND, callback: 0x12c4490, shard_count: 2, indexes_count: 2
[Thu Oct 8 06:34:09 2015] [ERROR] storage/core: elliptics: 2:000000000000...d306d759d1eb: could not find network state for request
[Thu Oct 8 06:34:09 2015] [DEBUG] storage/core: elliptics: INDEXES_FIND, callback: 0x12c4490, group: 2, next
[Thu Oct 8 06:34:09 2015] [DEBUG] storage/core: elliptics: INDEXES_FIND, callback: 0x12c4490, group: 2, id: 2:000000000000...d306d759d1eb, state: failed
[Thu Oct 8 06:34:09 2015] [DEBUG] storage/core: elliptics: Destroying session.
[Thu Oct 8 06:34:09 2015] [DEBUG] storage/core: elliptics: Destroying session.
[Thu Oct 8 06:34:09 2015] [DEBUG] storage/core: elliptics: Destroying session.
[Thu Oct 8 06:34:09 2015] [DEBUG] storage/core: elliptics: Destroying session.
[Thu Oct 8 06:34:09 2015] [DEBUG] storage/core: elliptics: Destroying node.

[noxiouz]

cocaine can not connect to Elliptics, I have not known why yet. It's out of Cocaine code. We need some Elliptics guru here: @bioothod

[TimBobkov]

I build cocaine from git repo, and elliptics setup from reverbrain repo. May be It`s a problem in some small differences in the source code?

[noxiouz]

I don't think so. I'll try to reproduce this

[TimBobkov]

Could this be because of incorrect permissions to the folder /var/elliptics?

[noxiouz]

 "backends": [
        {
            "backend_id": 1,
            "state": 0,
            "readonly": false,
            "defrag_state": 0,
            "last_start_err": -22,
            "last_start": "2015-10-08 05:25:13.354497"
        }
    ]

I asked guys from Elliptics, and I was told that "state": 0, means that backend is disabled. It can be if Elliptics cannot write to a given directory for example. Probably there's an error in logs of Elliptics.
At first glance, I thought that state 0 means no error.

[TimBobkov]

Hooray. Problem solved. It was necessary to create and provide access to folders:
/opt/elliptics/
/opt/elliptics/history.2/
/opt/elliptics/eblob.2/
/opt/elliptics/eblob.2/data/
After that, everything worked fine. Thank you for your help.