在k8S Kuscia点对点runp集群中，用API创建JOB，split的Task报错 #373
Comments
|
|
|
方便提供下另一方的报错吗？ |
|
另一个方没有报错，日志如下：
|
|
WARNING:root:Since the GPL-licensed package 2024-07-10 15:26:46,577|bob|INFO|secretflow|entry.py:start_ray:81| Succeeded to start ray head node at job-split-1-partner-0-global.bob.svc. ....
|
|
WARNING:root:Since the GPL-licensed package 2024-07-10 15:26:47,658|alice|INFO|secretflow|entry.py:start_ray:81| Succeeded to start ray head node at job-split-1-partner-0-global.alice.svc. ....
|
|
你到bob的容器内，看下psi-output-1.csv 这个文件生成了吗 |
|
Stale issue message. Please comment to remove stale tag. Otherwise this issue will be closed soon. |
Issue Type
Running
Search for existing issues similar to yours
Yes
OS Platform and Distribution
centos7
Kuscia Version
kuscia0.8
Deployment
k8s
deployment Version
1.19
App Running type
secretflow
App Running version
1.7
Configuration file used to run kuscia.
What happend and What you expected to happen.
Kuscia log output.
apiVersion: kuscia.secretflow/v1alpha1 kind: KusciaTask metadata: annotations: kuscia.secretflow/initiator: alice kuscia.secretflow/interconn-bfia-parties: "" kuscia.secretflow/interconn-kuscia-parties: bob kuscia.secretflow/interconn-self-parties: bob kuscia.secretflow/job-id: job-best-effort-linear kuscia.secretflow/party-master-domain: bob kuscia.secretflow/self-cluster-as-initiator: "false" kuscia.secretflow/task-alias: job-split-1 creationTimestamp: "2024-07-10T07:26:37Z" generation: 1 labels: kuscia.secretflow/controller: kuscia-job kuscia.secretflow/job-uid: c74e1f88-5b96-4d44-9480-626a251be466 name: job-split-1 namespace: cross-domain ownerReferences: - apiVersion: kuscia.secretflow/v1alpha1 blockOwnerDeletion: true controller: true kind: KusciaJob name: job-best-effort-linear uid: c74e1f88-5b96-4d44-9480-626a251be466 resourceVersion: "267961" uid: 7a0a7ddd-524c-4061-8f5b-d67a97ad7f3d spec: initiator: alice parties: - appImageRef: secretflow-image domainID: alice role: partner template: spec: {} - appImageRef: secretflow-image domainID: bob role: partner template: spec: {} scheduleConfig: {} taskInputConfig: '{"sf_datasource_config":{"alice":{"id":"default-data-source"},"bob":{"id":"default-data-source"}},"sf_cluster_desc":{"parties":["alice","bob"],"devices":[{"name":"spu","type":"spu","parties":["alice","bob"],"config":"{\"runtime_config\":{\"protocol\":\"REF2K\",\"field\":\"FM64\"},\"link_desc\":{\"connect_retry_times\":60,\"connect_retry_interval_ms\":1000,\"brpc_channel_protocol\":\"http\",\"brpc_channel_connection_type\":\"pooled\",\"recv_timeout_ms\":1200000,\"http_timeout_ms\":1200000}}"},{"name":"heu","type":"heu","parties":["alice","bob"],"config":"{\"mode\": \"PHEU\", \"schema\": \"paillier\", \"key_size\": 2048}"}],"ray_fed_config":{"cross_silo_comm_backend":"brpc_link"}},"sf_node_eval_param":{"domain":"data_prep","name":"train_test_split","version":"0.0.1","attr_paths":["train_size","test_size","random_state","shuffle"],"attrs":[{"f":0.75},{"f":0.25},{"i64":1234},{"b":true}]},"sf_output_uris":["train-dataset-1.csv","test-dataset-1.csv"],"sf_output_ids":["train-dataset-1","test-dataset-1"],"sf_input_ids":["psi-output-1"]}' status: allocatedPorts: - domainID: alice namedPort: job-split-1-partner-0/client-server: 31317 job-split-1-partner-0/fed: 31313 job-split-1-partner-0/global: 31314 job-split-1-partner-0/node-manager: 31315 job-split-1-partner-0/object-manager: 31316 job-split-1-partner-0/spu: 31318 role: partner - domainID: bob namedPort: job-split-1-partner-0/client-server: 21793 job-split-1-partner-0/fed: 21795 job-split-1-partner-0/global: 21796 job-split-1-partner-0/node-manager: 21797 job-split-1-partner-0/object-manager: 21792 job-split-1-partner-0/spu: 21794 role: partner completionTime: "2024-07-10T07:26:56Z" conditions: - lastTransitionTime: "2024-07-10T07:26:37Z" status: "True" type: ResourceCreated - lastTransitionTime: "2024-07-10T07:26:40Z" status: "True" type: Running - lastTransitionTime: "2024-07-10T07:26:56Z" status: "False" type: Success lastReconcileTime: "2024-07-10T07:26:56Z" message: The remaining no-failed party task counts 1 are less than the threshold 2 that meets the conditions for task success. pending party[], running party[alice-partner], successful party[], failed party[bob-partner] partyTaskStatus: - domainID: alice phase: Failed role: partner - domainID: bob phase: Failed role: partner phase: Failed podStatuses: bob/job-split-1-partner-0: createTime: "2024-07-10T07:26:37Z" namespace: bob nodeName: kuscia-autonomy-bob-69b9749d87-dpcql podName: job-split-1-partner-0 podPhase: Failed readyTime: "2024-07-10T07:26:40Z" reason: Error startTime: "2024-07-10T07:26:40Z" terminationLog: 'container[secretflow] terminated state reason "Error", message: "... Ignore 12450 characters at the beginning ...\n)\x1b[0m 2024-07-10 15:26:48.319 INFO link.py:38 [bob] -- [Anonymous_job] brpc options: {''proxy_max_restarts'': 3, ''timeout_in_ms'': 300000, ''recv_timeout_ms'': 604800000, ''connect_retry_times'': 3600, ''connect_retry_interval_ms'': 1000, ''brpc_channel_protocol'': ''http'', ''brpc_channel_connection_type'': ''pooled'', ''exit_on_sending_failure'': True}\n\x1b[36m(SenderReceiverProxyActor pid=22634)\x1b[0m I0710 15:26:48.328158 22634 external/com_github_brpc_brpc/src/brpc/server.cpp:1181] Server[yacl::link::transport::internal::ReceiverServiceImpl] is serving on port=21795.\n\x1b[36m(SenderReceiverProxyActor pid=22634)\x1b[0m W0710 15:26:48.328190 22634 external/com_github_brpc_brpc/src/brpc/server.cpp:1187] Builtin services are disabled according to ServerOptions.has_builtin_services\n\x1b[36m(SenderReceiverProxyActor pid=22634)\x1b[0m I0710 15:26:49.969624 22690 external/com_github_brpc_brpc/src/brpc/span.cpp:506] Opened ./rpc_data/rpcz/20240710.152649.22634/id.db and ./rpc_data/rpcz/20240710.152649.22634/time.db\n2024-07-10 15:26:52.351 INFO barriers.py:465 [bob] -- [Anonymous_job] Succeeded to create receiver proxy actor.\n2024-07-10 15:26:52.351 INFO barriers.py:520 [bob] -- [Anonymous_job] Try ping [''alice''] at 0 attemp, up to 3600 attemps.\n\x1b[36m(_run pid=22355)\x1b[0m WARNING:root:Since the GPL-licensed package `unidecode` is not installed, using Python''s `unicodedata` package which yields worse results.\n\x1b[33m(raylet)\x1b[0m [2024-07-10 15:26:47,745 I 22634 22634] logging.cc:230: Set ray log level from environment variable RAY_BACKEND_LOG_LEVEL to -1\n2024-07-10 15:26:54.246 ERROR component.py:1129 [bob] -- [Anonymous_job] eval on domain: \"data_prep\"\nname: \"train_test_split\"\nversion: \"0.0.1\"\nattr_paths: \"train_size\"\nattr_paths: \"test_size\"\nattr_paths: \"random_state\"\nattr_paths: \"shuffle\"\nattrs {\n f: 0.75\n}\nattrs {\n f: 0.25\n}\nattrs {\n i64: 1234\n}\nattrs {\n b: true\n}\ninputs {\n name: \"psi-output-1.csv\"\n type: \"sf.table.vertical_table\"\n system_info {\n }\n meta {\n type_url: \"type.googleapis.com/secretflow.spec.v1.VerticalTable\"\n value: \"\\n\\335\\003\\n\\003id1\\022\\003age\\022\\teducation\\022\\007default\\022\\007balance\\022\\007housing\\022\\004loan\\022\\003day\\022\\010duration\\022\\010campaign\\022\\005pdays\\022\\010previous\\022\\017job_blue-collar\\022\\020job_entrepreneur\\022\\rjob_housemaid\\022\\016job_management\\022\\013job_retired\\022\\021job_self-employed\\022\\014job_services\\022\\013job_student\\022\\016job_technician\\022\\016job_unemployed\\022\\020marital_divorced\\022\\017marital_married\\022\\016marital_single\\\"\\003str*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float\\n\\227\\003\\n\\003id2\\022\\020contact_cellular\\022\\021contact_telephone\\022\\017contact_unknown\\022\\tmonth_apr\\022\\tmonth_aug\\022\\tmonth_dec\\022\\tmonth_feb\\022\\tmonth_jan\\022\\tmonth_jul\\022\\tmonth_jun\\022\\tmonth_mar\\022\\tmonth_may\\022\\tmonth_nov\\022\\tmonth_oct\\022\\tmonth_sep\\022\\020poutcome_failure\\022\\016poutcome_other\\022\\020poutcome_success\\022\\020poutcome_unknown\\022\\001y\\\"\\003str*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\005float*\\003int\\020\\244M\"\n }\n data_refs {\n uri: \"psi-output-1.csv\"\n party: \"alice\"\n format: \"csv\"\n }\n data_refs {\n uri: \"psi-output-1.csv\"\n party: \"bob\"\n format: \"csv\"\n }\n}\noutput_uris: \"train-dataset-1.csv\"\noutput_uris: \"test-dataset-1.csv\"\n failed, error <\x1b[36mray::_run()\x1b[39m (pid=22355, ip=job-split-1-partner-0-global.bob.svc)\n File \"/usr/local/lib/python3.10/site-packages/secretflow/device/device/pyu.py\", line 156, in _run\n return fn(*args, **kwargs)\n File \"/usr/local/lib/python3.10/site-packages/secretflow/component/data_utils.py\", line 382, in <lambda>\n lambda uri=parties_path_format[p].uri: ctx.comp_storage.get_file_meta(\n File \"/usr/local/lib/python3.10/site-packages/secretflow/component/storage/storage.py\", line 49, in get_file_meta\n return impl.get_file_meta(remote_fn)\n File \"/usr/local/lib/python3.10/site-packages/secretflow/component/storage/impl/storage_impl.py\", line 208, in get_file_meta\n assert os.path.exists(full_remote_fn)\nAssertionError>\n2024-07-10 15:26:54.246 INFO api.py:342 [bob] -- [Anonymous_job] Shutdowning rayfed intendedly...\n2024-07-10 15:26:54.246 INFO api.py:356 [bob] -- [Anonymous_job] No wait for data sending.\n2024-07-10 15:26:54.247 INFO message_queue.py:72 [bob] -- [Anonymous_job] Notify message polling thread[DataSendingQueueThread] to exit.\n2024-07-10 15:26:54.248 INFO message_queue.py:72 [bob] -- [Anonymous_job] Notify message polling thread[ErrorSendingQueueThread] to exit.\n2024-07-10 15:26:54.248 INFO api.py:384 [bob] -- [Anonymous_job] Shutdowned rayfed.\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/local/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/usr/local/lib/python3.10/site-packages/secretflow/kuscia/entry.py\", line 547, in <module>\n main()\n File \"/usr/local/lib/python3.10/site-packages/click/core.py\", line 1157, in __call__\n return self.main(*args, **kwargs)\n File \"/usr/local/lib/python3.10/site-packages/click/core.py\", line 1078, in main\n rv = self.invoke(ctx)\n File \"/usr/local/lib/python3.10/site-packages/click/core.py\", line 1434, in invoke\n return ctx.invoke(self.callback, **ctx.params)\n File \"/usr/local/lib/python3.10/site-packages/click/core.py\", line 783, in invoke\n return __callback(*args, **kwargs)\n File \"/usr/local/lib/python3.10/site-packages/secretflow/kuscia/entry.py\", line 527, in main\n res = comp_eval(sf_node_eval_param, storage_config, sf_cluster_config)\n File \"/usr/local/lib/python3.10/site-packages/secretflow/component/entry.py\", line 166, in comp_eval\n res = comp.eval(\n File \"/usr/local/lib/python3.10/site-packages/secretflow/component/component.py\", line 1131, in eval\n raise e from None\n File \"/usr/local/lib/python3.10/site-packages/secretflow/component/component.py\", line 1126, in eval\n ret = self.__eval_callback(ctx=ctx, **kwargs)\n File \"/usr/local/lib/python3.10/site-packages/secretflow/component/preprocessing/data_prep/train_test_split.py\", line 103, in train_test_split_eval_fn\n input_df = load_table(\n File \"/usr/local/lib/python3.10/site-packages/secretflow/component/data_utils.py\", line 390, in load_table\n file_metas = reveal(file_metas)\n File \"/usr/local/lib/python3.10/site-packages/secretflow/device/driver.py\", line 162, in reveal\n all_object = sfd.get(all_object_refs)\n File \"/usr/local/lib/python3.10/site-packages/secretflow/distributed/primitive.py\", line 156, in get\n return fed.get(object_refs)\n File \"/usr/local/lib/python3.10/site-packages/fed/api.py\", line 621, in get\n values = ray.get(ray_refs)\n File \"/usr/local/lib/python3.10/site-packages/ray/_private/auto_init_hook.py\", line 22, in auto_init_wrapper\n return fn(*args, **kwargs)\n File \"/usr/local/lib/python3.10/site-packages/ray/_private/client_mode_hook.py\", line 103, in wrapper\n return func(*args, **kwargs)\n File \"/usr/local/lib/python3.10/site-packages/ray/_private/worker.py\", line 2624, in get\n raise value.as_instanceof_cause()\nray.exceptions.RayTaskError(AssertionError): \x1b[36mray::_run()\x1b[39m (pid=22355, ip=job-split-1-partner-0-global.bob.svc)\n File \"/usr/local/lib/python3.10/site-packages/secretflow/device/device/pyu.py\", line 156, in _run\n return fn(*args, **kwargs)\n File \"/usr/local/lib/python3.10/site-packages/secretflow/component/data_utils.py\", line 382, in <lambda>\n lambda uri=parties_path_format[p].uri: ctx.comp_storage.get_file_meta(\n File \"/usr/local/lib/python3.10/site-packages/secretflow/component/storage/storage.py\", line 49, in get_file_meta\n return impl.get_file_meta(remote_fn)\n File \"/usr/local/lib/python3.10/site-packages/secretflow/component/storage/impl/storage_impl.py\", line 208, in get_file_meta\n assert os.path.exists(full_remote_fn)\nAssertionError\n"' serviceStatuses: bob/job-split-1-partner-0-fed: createTime: "2024-07-10T07:26:37Z" namespace: bob portName: fed portNumber: 21795 readyTime: "2024-07-10T07:26:40Z" scope: Cluster serviceName: job-split-1-partner-0-fed bob/job-split-1-partner-0-global: createTime: "2024-07-10T07:26:37Z" namespace: bob portName: global portNumber: 21796 readyTime: "2024-07-10T07:26:40Z" scope: Domain serviceName: job-split-1-partner-0-global bob/job-split-1-partner-0-spu: createTime: "2024-07-10T07:26:37Z" namespace: bob portName: spu portNumber: 21794 readyTime: "2024-07-10T07:26:40Z" scope: Cluster serviceName: job-split-1-partner-0-spu startTime: "2024-07-10T07:26:37Z"The text was updated successfully, but these errors were encountered: