xiaomi/pegasus-python-client

Has been migrated to https://github.com/apache/incubator-pegasus/tree/master/python-client

pegasus-python-client's Issues

AttributeError when running test_integration

test_integration
  TestIntegration
    test_0_replica_scan_exception ...                                      [OK]
    test_1of3_replica_restart ...                                       [ERROR]
    test_1of5_replica_restart ...                                       [ERROR]
    test_1of5_replica_stop ...                                          [ERROR]
    test_1of5_replica_stop_and_start ...                                [ERROR]
    test_1of5_replica_stop_and_start_with_meta_stop_and_start ...       [ERROR]
    test_2of5_replica_stop ...                                          [ERROR]
    test_3of3_replica_restart ...                                       [ERROR]
    test_5of5_replica_restart ...                                       [ERROR]
    test_can_not_connect ...                                               [OK]

===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/home/hw/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/hw/.local/lib/python2.7/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "tests/test_integration.py", line 140, in test_1of3_replica_restart
    yield self.check_data()
  File "/home/hw/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/hw/.local/lib/python2.7/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "tests/test_integration.py", line 116, in check_data
    ret = yield self.loop_op()
  File "/home/hw/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
    result = g.send(result)
  File "tests/test_integration.py", line 107, in loop_op
    ret = yield self.c.get(self.TEST_HKEY + str(i), self.TEST_SKEY, 200)
  File "pypegasus/pgclient.py", line 719, in get
    return session.operate(op, timeout)
  File "pypegasus/pgclient.py", line 90, in operate
    self._reqs[seqid] = dr
exceptions.AttributeError: 'ReplicaSession' object has no attribute '_reqs'

test_integration.TestIntegration.test_1of3_replica_restart

所有ERROR均为exceptions.AttributeError: 'ReplicaSession' object has no attribute '_reqs'

打印当前模块的属性列表，写法示例：

        logger.info('%s', dir(self))
        if not hasattr(self, '_reqs'):
            logger.error('no attr _reqs, %s', dir(self))

测试发现，一开始属性列表正常，在某一次get时该实例真的没有_reqs属性了，其余属性均存在。

Fill in partition_hash in thrift_header

Thrift header has a reserved filed called 'partition_hash' , default value is 0, server doesn't use this filed currently. However, it will be used in server supporting partition split, so we should fill in correct value of it.

Adapt partition split

1. Fill correct partition_hash value in thrift_header

• Thrift header has a reserved filed called 'partition_hash' , default value is 0. It will be used in server supporting partition split to distinguish the value is for parent or child partition.

2. Add need_check_hash filed in get_scanner_request

• After partition split, parent will hold data belonging to child, and child will also have data belonging to parent. As a result, we have to check hash while executing unordered_scan to remove duplicated data. However, we don't need to check hash while hash_scan. Hash_scan will scan only one partition, it won't have duplicated data.

3. Handle split related error code: ERR_SPLITTING, ERR_PARENT_PARTITION_MISUSED

• when parent partition is registering child partition, it will reject read write requests by return ERR_SPLITTING, client won't query meta and will retry in remaining time
• when child partition is active, the request should be sent to child, parent will return ERR_PARENT_PARTITION_MISUSED, client should update all partitions config to update both parent and child config

4. Handle the condition that client query config for a splitting table

• when client query config for a splitting table, it will get child partition config, if child is not ready, its requests should be redirected to parent
• Now server support split (new_count = old_count*2) and cancel split (new_count = old_count / 2), client should update configurations in those cases.

Wrong partition_hash calculated

In order to get partition index, we need to calculate partition_hash by hash_key or blob_key(hash_key and sort_key). However, client partition_hash is not equal to the partition_hash calculated by server.

The wrong hash_value does not lead to write data to incorrect partition, but server will check equality between client partition_hash and the value calculated by itself if client partition_hash is not zero.

As a result, we should fix this bug

bug

tools.py
line 31

error_types.ERR_OBJECT_NOT_FOUND 没这个 ERR_OK 属性
这里写错了，多写了一段 ERR_OBJECT_NOT_FOUND

return error_types.ERR_OK.value
解决