primihub/primihub-platform

PrimiHub platform is a Muti-Party Computation and Muti-Party federated task security scheduling platform for MPC and FL point to point service.

Welcome to primihub-platform

primihub-platform is a Muti-Party Computation and Muti-Party federated task security scheduling platform for mpc and fl point to point service.

Features

Providing production-level service capabilities:

• Data access
• Multi-Party Resource Fusion
• Task Scheduling
• Multi-Party Federated Model Registry
• Multi-Party Cooperation Authority Management

and have a clear directory:

├─primihub-platform
    ├─primihub-service
    │   ├─application
    │   ├─biz
    │   ├─gateway
    │   └─script
    └─primihub-webconsole

Getting Started

Before we start, please refer to primihub and start node.

then we can get started from those projects.

• primihub-meta : connect primihub-service and have the right data access.
• primihub-service : provide all most of service capabilities and api.
• primihub-webconsole : you can operate specific functions and have a clear view.

License

Apache License 2.0

Contact Us

It's pleasure to offer a primihub assistant to a contract list by scanning the QR code. You can get support on technique,business and the chance to community with us util the assistant invite you to the open source community group.

And also welcome to follow our official account and slack home page(primihub slack).

primihub-platform's Issues

建议去掉上传svn文件的大小限制

功能描述
-资源上传功能，只能上传.csv文件，且不超过1MB；建议去掉这个限制。既然是隐私计算，数据量肯定不小，为什么要限制 不应该限制1M的数据大小

添加节点信息超时，刷新之后界面已经添加上了

bug描述
添加节点信息报超时的错误，刷新之后界面已经添加上了

复现步骤
添加机构A，网关地址为http://改该节点机器的IP:30811/prod-api和http://gateway0:8080都会这样

bug日志
无报错

环境（请填写完整）:

• OS: Linux
• Arch: x86_64
• 部署方式: docker
• PrimiHub版本: 1.6.6

本地编译启动的那个端口号是什么意思，是相当于后端启动了吗？

这个端口是什么意思，还有咱们提供新增组件的流程吗，还有前端代码是不是就在primihub/primihub-platform/primihub-webconsole,后端代码在primihub/primihub-platform/primihub-service

增加隐匿查询过程日志的打印

功能描述
希望增加隐匿查询过程日志的打印

动机描述
数据量较大时，隐匿查询进行很长时间，在这个过程中不知道发生了什么

其他
参考联合建模的日志量，增加隐匿查询进行过程中的日志

Suggest participants to the Hacktoberfest event

See also the details

添加合作节点时可以添加自己，显示”对方正在审核中“，且无法删除

bug描述
添加合作节点时可以添加自己，显示”对方正在审核中“，且无法删除，建议设置为不允许添加自己

环境（请填写完整）:

• OS: Linux
• Arch: x86_64
• 部署方式: docker
• PrimiHub版本: 如1.6.6

匿踪查询大数据量，报265异常

bug描述
匿踪查询100万数据，报265异常。找不到异常的堆栈信息
复现步骤

1.node2 配置mysql数据库公开资源，node1创建匿踪查询任务。任务启动一会儿后，任务运行失败：WEB界面展示错误信息：
运行失败:SERVER:task execute encountes error
CLIENT:task execute encountes error

node1结点错误日志
I20240507 14:56:21.715778 3085 worker.cc:176] RequestId: ee23c313a633469cb4cafc7583d1ac9c Worker start execute task
I20240507 14:56:21.715879 3075 node_impl.cc:498] RequestId: ee23c313a633469cb4cafc7583d1ac9c create worker thread finished
E20240507 14:56:21.802925 3086 pir_task.cc:252] RequestId: ee23c313a633469cb4cafc7583d1ac9c [{"id":7},{"name":13}]
E20240507 14:56:21.803310 3086 pir_task.cc:129] RequestId: ee23c313a633469cb4cafc7583d1ac9c server_key_columns_: 1 server_label_columns_: 2
E20240507 14:59:41.926980 3085 worker.cc:203] RequestId: ee23c313a633469cb4cafc7583d1ac9c ERROR: 265
I20240507 14:59:42.092892 3075 node_impl.cc:523] RequestId: ee23c313a633469cb4cafc7583d1ac9c , receive request for kill task from SCHEDULER
W20240507 14:59:42.092939 3075 node_impl.cc:535] worker for RequestId: ee23c313a633469cb4cafc7583d1ac9c has finished
I20240507 14:59:42.117178 3050 node_impl.cc:523] RequestId: ee23c313a633469cb4cafc7583d1ac9c , receive request for kill task from SCHEDULER
W20240507 14:59:42.117399 3050 node_impl.cc:535] worker for RequestId: ee23c313a633469cb4cafc7583d1ac9c has finished
环境

• OS: Linux
• Arch: x86_64
• 部署方式: docker,
• PrimiHub版本: 1.7.0

base.json

I downloaded the code of version 1.5.0. After compiling and running, I found that the primihub-service/script/base.json file maybe not be updated.

Failed to run webconsole

I installed the latest version nodejs and wanted to build primihub-webconsole. I encountered some error messages as below. Please help me, thanks.

npm run dev --port=9528

[email protected] dev
vue-cli-service serve

INFO Starting development server...
ERROR Error: Cannot call .tap() on a plugin that has not yet been defined. Call plugin('preload').use() first.
Error: Cannot call .tap() on a plugin that has not yet been defined. Call plugin('preload').use() first.
at module.exports.tap (/home/songxm/devel/github/1125/primihub-platform/primihub-webconsole/node_modules/webpack-chain/src/Plugin.js:26:15)
at chainWebpack (/home/songxm/devel/github/1125/primihub-platform/primihub-webconsole/vue.config.js:72:30)
at /home/songxm/devel/github/1125/primihub-platform/primihub-webconsole/node_modules/@vue/cli-service/lib/Service.js:268:40
at Array.forEach ()
at Service.resolveChainableWebpackConfig (/home/songxm/devel/github/1125/primihub-platform/primihub-webconsole/node_modules/@vue/cli-service/lib/Service.js:268:26)
at Service.resolveWebpackConfig (/home/songxm/devel/github/1125/primihub-platform/primihub-webconsole/node_modules/@vue/cli-service/lib/Service.js:272:48)
at PluginAPI.resolveWebpackConfig (/home/songxm/devel/github/1125/primihub-platform/primihub-webconsole/node_modules/@vue/cli-service/lib/PluginAPI.js:132:25)
at serve (/home/songxm/devel/github/1125/primihub-platform/primihub-webconsole/node_modules/@vue/cli-service/lib/commands/serve.js:79:31)
at Service.run (/home/songxm/devel/github/1125/primihub-platform/primihub-webconsole/node_modules/@vue/cli-service/lib/Service.js:262:12)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Invalid container image name from primihub-service

docker pull java:8-jre
Error response from daemon: manifest for java:8-jre not found: manifest unknown: manifest unknown

See also

primihub-platform/primihub-service/Dockerfile.local

Line 10 in 4eed1fb

FROM java:8-jre

I suggest that we use the same base image ibmjava:8-jre with

primihub-platform/primihub-service/Dockerfile

Line 1 in 4eed1fb

FROM ibmjava:8-jre

[Nacos Config Boot] : The preload configuration is not enabled

代码分支：develop
NACOS版本：2.3.1

application和gateway服务运行时报如下错误信息后直接结束了启动：

Connected to the target VM, address: '127.0.0.1:65376', transport: 'socket'
2024-04-09 17:13:02.757  WARN 20696 --- [           main] c.a.nacos.client.logging.NacosLogging    : Load Logback Configuration of Nacos fail, message: ch/qos/logback/core/joran/action/AbstractEventEvaluatorAction
2024-04-09 17:13:02.763  WARN 20696 --- [           main] c.a.nacos.client.logging.NacosLogging    : Load Logback Configuration of Nacos fail, message: ch/qos/logback/core/joran/action/AbstractEventEvaluatorAction
2024-04-09 17:13:02.856  INFO 20696 --- [           main] c.a.b.n.c.u.NacosConfigPropertiesUtils   : nacosConfigProperties : NacosConfigProperties{serverAddr='127.0.0.1:8848', contextPath='null', encode='null', endpoint='null', namespace='null', accessKey='null', secretKey='null', ramRoleName='null', autoRefresh=false, dataId='null', dataIds='null', group='DEFAULT_GROUP', type=null, maxRetry='null', configLongPollTimeout='null', configRetryTime='null', enableRemoteSyncConfig=false, extConfig=[], bootstrap=Bootstrap{enable=false, logEnable=false}}
2024-04-09 17:13:02.858  INFO 20696 --- [           main] NacosConfigApplicationContextInitializer : [Nacos Config Boot] : The preload configuration is not enabled
2024-04-09 17:13:03.112  WARN 20696 --- [           main] c.a.nacos.client.logging.NacosLogging    : Load Logback Configuration of Nacos fail, message: ch/qos/logback/core/joran/action/AbstractEventEvaluatorAction
2024-04-09 17:13:03.114  INFO 20696 --- [           main] c.alibaba.nacos.client.utils.ParamUtil   : [settings] [req-serv] nacos-server port:8848
2024-04-09 17:13:03.114  INFO 20696 --- [           main] c.alibaba.nacos.client.utils.ParamUtil   : [settings] [http-client] connect timeout:1000
2024-04-09 17:13:03.117  INFO 20696 --- [           main] c.alibaba.nacos.client.utils.ParamUtil   : PER_TASK_CONFIG_SIZE: 3000.0
2024-04-09 17:13:03.330  INFO 20696 --- [           main] c.a.n.client.identify.CredentialWatcher  : null No credential found
Disconnected from the target VM, address: '127.0.0.1:65376', transport: 'socket'

Process finished with exit code 1

Failed when I try to login

I got an error message which is 登录失效，请重新登录
When I stay on the login page for a long time and enter the username and password to login

匿综查询，选择多个资源后，结果不一致

匿综查询，选择多个资源后，似乎只对第一个资源的关键字进行了匹配

界面上总弹出time out的错误提示框，但不影响平台的使用

日志报错：
I/O error on GET request for "https://node1.primihub.com/collect/operate/getNodeList": Connect to node1.primihub.com:443 [node1.primihub.com/118.190.39.100] failed: connect timed out; nested exception is org.apache.http.conn.ConnectTimeoutException: Connect to node1.primihub.com:443 [node1.primihub.com/118.190.39.100] failed: connect timed out

在操作页面，我的资源，上传资源的时候，提示添加失败，无法将资源注册到数据集中

您好，我在操作页面，我的资源，上传资源的时候，提示添加失败，无法将资源注册到数据集中，这个错是哪里出了问题，我在哪里可以看详细日志。

Unable to perform inference task.

Hi, all
I used 1.5.0 for a few days. I have trained a homo-lr model and I wanted to make an inference task.
But I found I could not selected test data on the UI. I studied the service code (primihub-service) and found it get resource information from local db (not from fusion db).
I also found there was no infer codes (*fbl) for hetero-xgb in "templates" path of primihub-service. Can the platform now do the inference task for hetero-xgb?

Thanks for your job and looking forward to your reply.

我用idea打开后端代码后提示，找不到符号，没有这个方法。

WorkerGrpcClientSpringTest.java

+.setJobId(ByteString.copyFrom("1".getBytes(StandardCharsets.UTF_8)))

匿踪查询只能在第一列中查询

例如，官方提供的测试数据“PIR-PSI用户数据A”中，包括姓名和年龄两列，查询某个姓名可以查到，查询“男”或“女”无法查到

隐私求交的KKRT算法报错

页面中隐私求交的kkrt做法，求交集时候失败

原语平台部署图

近日预研原语平台，得到原语研发团队支持，受益颇多，非常感谢！

根据最近平台部署情况，特整理以下原语平台部署图，欢迎指正

原语平台部署图

匿踪查询和联邦学习不支持使用从数据库导入的资源

启动primihub-meta服务的时候报错

希望大神看看这是什么问题

隐匿查询、隐私求交完成后，任务耗时依然增加

增加隐匿查询结束任务和删除任务功能

功能描述
建议增加隐匿查询结束任务和删除任务功能

动机描述
有时候隐匿查询进行了很长时间还没有成功或者失败，占用内存，影响其它任务的进行

其他
可以参考横向/纵向任务的取消任务功能和隐私求交任务的删除任务功能

webconsole编译不成功

bug描述

sudo npm run dev 编译不成功

复现步骤

1、cd primihub-webconsole
2、sudo npm install
3、target: 'your gateway url', (使用 route -n 命令查到的）
4、sudo npm run dev

按照文档操作出现的
https://docs.primihub.com/docs/developer-docs/platform/#%E6%8E%A7%E5%88%B6%E5%8F%B0
bug日志

环境（请填写完整）:

• OS: ubuntu20,04
• Arch: arm64
• 部署方式: 本机编译
• PrimiHub版本: primihub-platform 于2023/10/23
• npm版本: 6.14.18 (版本降至6.10也会）

部署好了平台之后发现导入数据库类型源失败，提示链接url有问题，但是用navicat是可以连接上的

save DataResource Exception：null

Branch main, when I upload a new resource, the dataResource.getResourceId() is null, and line 128 dataResource.getResourceId().toString() in DataResourceService got a npe.

table[data_pr] loss and table[data_project]'s column `organ_id`, `organ_num`,`resource_organ_ids`, `auth_resource_num` , `user_id` not exist

As the title says, Init.sql and ddl.sql do not match in the following sections：

-- Records of data_pr

INSERT INTO data_pr (id, project_id, resource_id, is_authed, is_del, create_date, update_date) VALUES (1, 1, 8, 1, 0, '2022-04-27 18:39:08.000', '2022-04-27 18:39:08.000');
INSERT INTO data_pr (id, project_id, resource_id, is_authed, is_del, create_date, update_date) VALUES (2, 1, 9, 1, 0, '2022-04-27 18:39:08.000', '2022-04-27 18:39:08.000');

-- Records of data_project

INSERT INTO data_project (project_id, project_name, project_desc, organ_id, organ_num, resource_num, resource_organ_ids, auth_resource_num, user_id, is_del, create_date, update_date) VALUES (1, '短视频', '短视频', 1000, 0, 2, '', 2, 1005, 0, '2022-04-27 18:39:08.000', '2022-04-27 18:39:08.000');

添加节点的时候，公钥有误

bug描述

添加节点时，并不能顺利添加节点，查看application日志的，显示解密有误

复现步骤

1.按照文档多机部署
2.环境为内网虚拟机

bug日志

node0：
14:44:29.548 [INFO ] [http-nio-8090-exec-8] [com.primihub.biz.service.sys.SysUserService:76] ip:192.192.192.2
14:45:17.981 [INFO ] [http-nio-8090-exec-4] [com.primihub.biz.service.data.OtherBusinessesService:181] http://172.16.1.112:30080/prod-api/share/shareData/apply
14:45:17.998 [INFO ] [http-nio-8090-exec-4] [com.primihub.biz.service.data.OtherBusinessesService:183] url:【http://172.16.1.112:30080/prod-api/share/shareData/apply】 - baseResultEntity {"code":1013,"msg":"解密失败:解密内容失败:Request processing failed; nested exception is java.lang.NullPointerException"}

node1:
14:45:17.964 [INFO ] [http-nio-8090-exec-7] [com.primihub.biz.service.sys.SysOrganService:240] {"organName":"机构A","country":"CN","applyId":"73b55fd887fb-652fa07e-ce82-45c6-ade6-7d8f4d982977","organId":"278ef6cd-e8be-42f7-8627-73b55fd887fb","lon":114.0683,"publicKey":"MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCluoBsZ25v/GBhbjzZD2iqRu7sT+9cWcs2lg8MCXSzZN4UdROSn30fX/F299jeYHZczl0WaAFJSh2PIPmLRLnRTZznBhroO30P3dIJ6MUsJ4cgLflVarE/bqzEmXdp8oZr4hPAqvViueh87klvmtQhTlIsv6DQpaNGT4oW8j1KzwIDAQAB","gateway":"http://172.16.1.111:30080/prod-api","lat":22.5455}
14:50:00.000 [INFO ] [scheduling-1] [com.primihub.biz.service.schedule.ScheduleService:66] 定时处理节点业务
14:50:06.566 [INFO ] [SimpleAsyncTaskExecutor-29] [com.primihub.biz.service.sys.SysAsyncService:70] {"code":0,"msg":"成功"}
15:00:00.006 [INFO ] [scheduling-1] [com.primihub.biz.service.schedule.ScheduleService:66] 定时处理节点业务
15:00:05.869 [INFO ] [SimpleAsyncTaskExecutor-30] [com.primihub.biz.service.sys.SysAsyncService:70] {"code":0,"msg":"成功"}
15:10:00.005 [INFO ] [scheduling-1] [com.primihub.biz.service.schedule.ScheduleService:66] 定时处理节点业务
15:10:10.123 [INFO ] [SimpleAsyncTaskExecutor-31] [com.primihub.biz.service.sys.SysAsyncService:70] {"code":0,"msg":"成功"}
15:10:20.871 [INFO ] [http-nio-8090-exec-8] [com.primihub.biz.service.sys.SysOrganService:240] {"organName":"机构A","country":"CN","applyId":"73b55fd887fb-0487c39a-a625-4147-b5cd-c7e8cd4c3475","organId":"278ef6cd-e8be-42f7-8627-73b55fd887fb","lon":114.0683,"publicKey":"MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCluoBsZ25v/GBhbjzZD2iqRu7sT+9cWcs2lg8MCXSzZN4UdROSn30fX/F299jeYHZczl0WaAFJSh2PIPmLRLnRTZznBhroO30P3dIJ6MUsJ4cgLflVarE/bqzEmXdp8oZr4hPAqvViueh87klvmtQhTlIsv6DQpaNGT4oW8j1KzwIDAQAB","gateway":"http://172.16.1.111:30080/prod-api","lat":22.5455}
15:20:00.006 [INFO ] [scheduling-1] [com.primihub.biz.service.schedule.ScheduleService:66] 定时处理节点业务
15:20:06.409 [INFO ] [SimpleAsyncTaskExecutor-32] [com.primihub.biz.service.sys.SysAsyncService:70] {"code":0,"msg":"成功"}

环境（请填写完整）:

• OS: 如ubuntu 22.02
• Arch: 如x86_64,
• 部署方式:多机docker-compose
• PrimiHub版本: 如1.6.11

匿踪查询不支持模糊查询

纵向任务报错

2023-05-22 11:06:11,749 platform=FL INFO 1 taskid=1660467895098421250 primihub.utils.logger_util :1722:host_tree_construct current role: guest, current record: 6
W20230522 11:06:50.569720 13644 thread_pool.h:135] Thread pool queue has reached maximum size. Increasing to 204800000 tasks.
E20230522 11:06:57.812204 10686 node.cc:698] task execute encountes error, begin to kill task to release resource
E20230522 11:06:57.812407 10686 node.cc:704] party info: defalut:172.28.1.10:50050:0:default
I20230522 11:06:57.813480 13870 node.cc:506] receive request for kill task for: job_id: 1 task id: 1660467895098421250 from SCHEDULER
W20230522 11:06:57.813531 13870 task.h:47] task receives kill task request and stop stauts
E20230522 11:06:57.813560 13870 task_context.h:93] stop all in data queue
E20230522 11:06:57.813599 13870 task_context.h:100] stop all out data queue
E20230522 11:06:57.813624 13870 task_context.h:107] stop all complete queue
E20230522 11:06:57.813644 13870 node.cc:539] end of VMNodeImpl::KillTask
E20230522 11:06:57.814442 10686 node.cc:704] party info: defalut:172.28.1.11:50051:0:default
2023-05-22 11:06:57.820 | ERROR | primihub.executor:execute_py:79 - Exception:
self.log_level: 20
2023-05-22 11:06:57.820 | ERROR | primihub.executor:execute_py:79 - Exception:
E20230522 11:06:57.822094 10687 py_executor.cc:240] Failed to execute python: ValueError: received data encountes error

At:
(843): recv
(1615): host_tree_construct
(1743): host_tree_construct
(1743): host_tree_construct
(1743): host_tree_construct
(1743): host_tree_construct
(1743): host_tree_construct
(1944): fit
(2166): xgb_host_logic
/usr/local/lib/python3.8/dist-packages/primihub/executor.py(84): execute_py
E20230522 11:06:58.098129 10684 fl_task.cc:74] begin to terminate py_main
I20230522 11:06:58.226557 10684 fl_task.cc:85] py_main executes result code: 383