Is HDFS loader support parquet orc file format? #2264

JackyYangPassion · 2022-12-01T06:47:49Z

JackyYangPassion
Dec 1, 2022

Is your feature request related to a problem? Please describe.
now it's support CSV file format for vertex/edge data

Describe the solution you'd like

import graphscope
import graphscope.nx as nx
from graphscope.framework.loader import Loader

graphscope.set_option(show_log=True)

edges = Loader("hdfs:///data/graph/twitter.e", host='localhost', port='8020')
graph = graphscope.g(directed=False)

graph.add_edges(edges)

Loader support parquet orc file ?

sighingnow · 2022-12-01T07:02:08Z

sighingnow
Dec 1, 2022
Maintainer

Loader support parquet orc file ?

Yes, but not inclueded in current stable release yet. You could install our nightly build via

pip3 install -U graphscope --pre

1 reply

JackyYangPassion Dec 1, 2022
Author

Example of use？

sighingnow · 2022-12-01T07:02:21Z

sighingnow
Dec 1, 2022
Maintainer

Closing as moving to discussion.

9 replies

sighingnow Dec 12, 2022
Maintainer

Based on the implementation details, I think prefix should work as well.

Haven't tested, though. Feel free to paste the error logs if it doesn't work.

Thanks!

JackyYangPassion Dec 12, 2022
Author

graph = (
        graph.add_vertices(Loader("hdfs:///user/hive/warehouse/community.db/transaction_vertices/job_id=112/part-00000-71582b1f-59d2-4207-98d5-54109a0fcd65.c000.snappy.orc",host="ip", port=8020)
                           ,label="address"
                           ,vid_field="address"
                          )
             .add_edges(
                   Loader("hdfs:///user/hive/warehouse/community.db/transaction_edges/job_id=112/part",host="ip", port=8020)
                   ,label="transactions"
                   ,src_label="address"
                   ,dst_label="address"
                   ,src_field='from_address'
                   ,dst_field='to_address'
                 )
             
          )

load vertices use path
hdfs:///user/hive/warehouse/community.db/transaction_vertices/job_id=112/part-00000-71582b1f-59d2-4207-98d5-54109a0fcd65.c000.snappy.orc

load edges use path with file prefix
hdfs:///user/hive/warehouse/community.db/transaction_edges/job_id=112/part

when load edge data error log:

2022-12-12 15:09:57,964 [INFO][utils:1592]: Codegened graph type: vineyard::ArrowFragment<std::string,uint64_t,vineyard::ArrowVertexMap<vineyard::arrow_string_view,uint64_t>>, Graph header: vineyard/graph/fragment/arrow_fragment.h
Loading vertex labeled address and:   0%|          | 0/10 [00:00<?, ?it/s]
Loading vertex labeled address and:  20%|██        | 2/10 [00:00<00:03,  2.53it/s]
I20221212 15:09:58.766366   111 /usr/local/include/vineyard/graph/vertex_map/arrow_vertex_map.h:242] ArrowVertexMap<string, int64_t>:
	size: 0 MB
I20221212 15:10:00.838449   111 /usr/local/include/vineyard/graph/vertex_map/arrow_vertex_map.h:242] ArrowVertexMap<string, int64_t>:
	size: 0 MB
I20221212 15:10:03.848903   111 /usr/local/include/vineyard/graph/vertex_map/arrow_vertex_map.h:242] ArrowVertexMap<string, int64_t>:
	size: 39.5097 MB
Loading vertex labeled address and:  60%|██████    | 6/10 [00:05<00:04,  1.06s/it]
I20221212 15:10:06.840247   111 /usr/local/include/vineyard/graph/vertex_map/arrow_vertex_map.h:242] ArrowVertexMap<string, int64_t>:
	size: 39.5097 MB
Loading vertex labeled address and: 100%|██████████| 10/10 [00:09<00:00,  1.01it/s]
Loading vertex labeled address and: 100%|██████████| 10/10 [00:09<00:00,  1.03it/s]
I20221212 15:10:10.437207   111 /usr/local/include/vineyard/graph/vertex_map/arrow_vertex_map.h:242] ArrowVertexMap<string, int64_t>:
	size: 39.5097 MB
2022-12-12 15:10:49,070 [INFO][utils:1592]: Codegened graph type: vineyard::ArrowFragment<std::string,uint64_t,vineyard::ArrowVertexMap<vineyard::arrow_string_view,uint64_t>>, Graph header: vineyard/graph/fragment/arrow_fragment.h
Loading edge labeled transactions:   0%|          | 0/10 [00:00<?, ?it/s]
I20221212 15:10:49.136369   111 /opt/graphscope/include/graphscope/core/loader/arrow_fragment_loader.h:764] read edge table from vineyard: o0009f46078b49134
2022-12-12 07:10:49,161 [ERROR][rpc:188]: Runstep failed with code: ANALYTICAL_ENGINE_INTERNAL_ERROR, message: Error occurred during RunStep, The traceback is: Traceback (most recent call last):
  File "/home/graphscope/.local/lib/python3.8/site-packages/gscoordinator/op_executor.py", line 102, in run_step
    for response in responses:
  File "/home/graphscope/.local/lib/python3.8/site-packages/grpc/_channel.py", line 426, in __next__
    return self._next()
  File "/home/graphscope/.local/lib/python3.8/site-packages/grpc/_channel.py", line 826, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.INTERNAL
	details = "VineyardError occurred on worker 0: VineyardError occurred on worker 0: /download/v6d/modules/graph/loader/fragment_loader_utils.cc:45: SyncSchema -> Assertion failed: field_num > 0: Empty table list cannot be used for normalizing schema
vineyard::SyncSchema(std::shared_ptr<arrow::Table> const&, grape::CommSpec const&) + 0x851
vineyard::sync_gs_error<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap<nonstd::sv_lite::basic_string_view<char, std::char_traits<char> >, unsigned long> >::loadEdgeTables(std::vector<std::shared_ptr<gs::detail::Edge>, std::allocator<std::shared_ptr<gs::detail::Edge> > > const&, int, int)::{lambda(std::shared_ptr<arrow::Table> const&)#2}&, std::shared_ptr<arrow::Table> const&>(grape::CommSpec const&, std::decay&&, (std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap<nonstd::sv_lite::basic_string_view<char, std::char_traits<char> >, unsigned long> >::loadEdgeTables(std::vector<std::shared_ptr<gs::detail::Edge>, std::allocator<std::shared_ptr<gs::detail::Edge> > > const&, int, int)::{lambda(std::shared_ptr<arrow::Table> const&)#2}&>::type&&)...)::{lambda()#2}::operator()() const + 0x30
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap<nonstd::sv_lite::basic_string_view<char, std::char_traits<char> >, unsigned long> >::loadEdgeTables(std::vector<std::shared_ptr<gs::detail::Edge>, std::allocator<std::shared_ptr<gs::detail::Edge> > > const&, int, int) + 0x1AB8
vineyard::sync_gs_error<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap<nonstd::sv_lite::basic_string_view<char, std::char_traits<char> >, unsigned long> >::LoadEdgeTables()::{lambda()#2}&>(grape::CommSpec const&, std::decay&&, (std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap<nonstd::sv_lite::basic_string_view<char, std::char_traits<char> >, unsigned long> >::LoadEdgeTables()::{lambda()#2}&>::type&&)...)::{lambda()#2}::operator()() const + 0x3A
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap<nonstd::sv_lite::basic_string_view<char, std::char_traits<char> >, unsigned long> >::LoadEdgeTables() + 0x36B
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap<nonstd::sv_lite::basic_string_view<char, std::char_traits<char> >, unsigned long> >::LoadVertexEdgeTables() + 0x421
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap<nonstd::sv_lite::basic_string_view<char, std::char_traits<char> >, unsigned long> >::addVerticesAndEdges(unsigned long) + 0x4D
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap<nonstd::sv_lite::basic_string_view<char, std::char_traits<char> >, unsigned long> >::AddLabelsToGraphAsFragmentGroup(unsigned long) + 0x41
AddLabelsToGraph + 0x271
gs::GrapeInstance::addLabelsToGraph(gs::rpc::GSParams const&) + 0x6E7
gs::GrapeInstance::OnReceive(std::shared_ptr<gs::CommandDetail>) + 0x692
gs::Dispatcher::processCmd(std::shared_ptr<gs::CommandDetail>) + 0xD0
gs::Dispatcher::publisherLoop() + 0x1E6
std::_V2::generic_category() + 0x60
start_thread + 0xD9
clone + 0x43

"
	debug_error_string = "{"created":"@1670829049.158825187","description":"Error received from peer ipv4:10.244.1.105:56871","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"VineyardError occurred on worker 0: VineyardError occurred on worker 0: /download/v6d/modules/graph/loader/fragment_loader_utils.cc:45: SyncSchema -> Assertion failed: field_num > 0: Empty table list cannot be used for normalizing schema\nvineyard::SyncSchema(std::shared_ptr<arrow::Table> const&, grape::CommSpec const&) + 0x851\nvineyard::sync_gs_error<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap<nonstd::sv_lite::basic_string_view<char, std::char_traits<char> >, unsigned long> >::loadEdgeTables(std::vector<std::shared_ptr<gs::detail::Edge>, std::allocator<std::shared_ptr<gs::detail::Edge> > > const&, int, int)::{lambda(std::shared_ptr<arrow::Table> const&)#2}&, std::shared_ptr<arrow::Table> const&>(grape::CommSpec const&, std::decay&&, (std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap<nonstd::sv_lite::basic_string_view<char, std::char_traits<char> >, unsigned long> >::loadEdgeTables(std::vector<std::shared_ptr<gs::detail::Edge>, std::allocator<std::shared_ptr<gs::detail::Edge> > > const&, int, int)::{lambda(std::shared_ptr<arrow::Table> const&)#2}&>::type&&)...)::{lambda()#2}::operator()() const + 0x30\ngs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap<nonstd::sv_lite::basic_string_view<char, std::char_traits<char> >, unsigned long> >::loadEdgeTables(std::vector<std::shared_ptr<gs::detail::Edge>, std::allocator<std::shared_ptr<gs::detail::Edge> > > const&, int, int) + 0x1AB8\nvineyard::sync_gs_error<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap<nonstd::sv_lite::basic_string_view<char, std::char_traits<char> >, unsigned long> >::LoadEdgeTables()::{lambda()#2}&>(grape::CommSpec const&, std::decay&&, (std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap<nonstd::sv_lite::basic_string_view<char, std::char_traits<char> >, unsigned long> >::LoadEdgeTables()::{lambda()#2}&>::type&&)...)::{lambda()#2}::operator()() const + 0x3A\ngs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap<nonstd::sv_lite::basic_string_view<char, std::char_traits<char> >, unsigned long> >::LoadEdgeTables() + 0x36B\ngs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap<nonstd::sv_lite::basic_string_view<char, std::char_traits<char> >, unsigned long> >::LoadVertexEdgeTables() + 0x421\ngs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap<nonstd::sv_lite::basic_string_view<char, std::char_traits<char> >, unsigned long> >::addVerticesAndEdges(unsigned long) + 0x4D\ngs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap<nonstd::sv_lite::basic_string_view<char, std::char_traits<char> >, unsigned long> >::AddLabelsToGraphAsFragmentGroup(unsigned long) + 0x41\nAddLabelsToGraph + 0x271\ngs::GrapeInstance::addLabelsToGraph(gs::rpc::GSParams const&) + 0x6E7\ngs::GrapeInstance::OnReceive(std::shared_ptr<gs::CommandDetail>) + 0x692\ngs::Dispatcher::processCmd(std::shared_ptr<gs::CommandDetail>) + 0xD0\ngs::Dispatcher::publisherLoop() + 0x1E6\nstd::_V2::generic_category() + 0x60\nstart_thread + 0xD9\nclone + 0x43\n\n","grpc_status":13}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/graphscope/.local/lib/python3.8/site-packages/gscoordinator/coordinator.py", line 300, in _RunStep
    head, bodies = self._operation_executor.run_on_analytical_engine(
  File "/home/graphscope/.local/lib/python3.8/site-packages/gscoordinator/monitor.py", line 193, in runOnAnalyticalEngineWarp
    res = func(instance, dag_def, dag_bodies, loader_op_bodies)
  File "/home/graphscope/.local/lib/python3.8/site-packages/gscoordinator/op_executor.py", line 168, in run_on_analytical_engine
    response_head, response_bodies = self.run_step(dag_def, dag_bodies)
  File "/home/graphscope/.local/lib/python3.8/site-packages/gscoordinator/op_executor.py", line 116, in run_step
    raise AnalyticalEngineInternalError(msg)
graphscope.framework.errors.AnalyticalEngineInternalError: VineyardError occurred on work ... [truncated]

JackyYangPassion Dec 13, 2022
Author

@sighingnow can you help me fix this error? thank you!

JackyYangPassion Dec 13, 2022
Author

Exception ignored in: <function Launcher.__del__ at 0x7fdb4fee6790>
Traceback (most recent call last):
2022-12-13 13:45:11,751 [INFO][utils:1592]: Codegened graph type: vineyard::ArrowFragment<std::string,uint64_t,vineyard::ArrowVertexMap<vineyard::arrow_string_view,uint64_t>>, Graph header: vineyard/graph/fragment/arrow_fragment.h
  File "/root/anaconda3/lib/python3.9/site-packages/vineyard/launcher/launcher.py", line 63, in __del__
    self.join()
  File "/root/anaconda3/lib/python3.9/site-packages/vineyard/drivers/io/stream.py", line 143, in join
    raise RuntimeError(
RuntimeError: Subprocesses failed with the following error: 
Failed to launch job [/root/anaconda3/lib/python3.9/site-packages/vineyard/drivers/io/ssh.sh localhost vineyard_parse_bytes_to_dataframe /tmp/vineyard.sock.1670939083.345418 o000aef0d4238c5d9 ...], exited with 255: 

extra diagnostics are as follows: Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.9/site-packages/vineyard/drivers/io/adaptors/parse_bytes_to_dataframe.py", line 138, in parse_bytes
    table = parse_dataframe_blocks(
  File "/root/anaconda3/lib/python3.9/site-packages/vineyard/drivers/io/adaptors/parse_bytes_to_dataframe.py", line 38, in parse_dataframe_blocks
  return pa.csv.read_csv(reader, read_options, parse_options, convert_options)
  File "pyarrow/_csv.pyx", line 886, in pyarrow._csv.read_csv
  File "pyarrow/_csv.pyx", line 895, in pyarrow._csv.read_csv
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 118, in pyarrow.lib.check_status
pyarrow.lib.ArrowKeyError: Column 'from_address' in include_columns does not exist in CSV file



E20221213 13:45:11.793395 27300 /download/v6d/modules/graph/loader/arrow_fragment_loader.cc:368] Failed to read from stream o000aef0e8651cad7: Stream failed: Stream source failed
Loading edge labeled transactions:   0%|          | 0/10 [00:00<?, ?it/s]I20221213 13:45:11.791596 27070 /opt/graphscope/include/graphscope/core/loader/arrow_fragment_loader.h:764] read edge table from vineyard: o000aef0e8651cad7
E20221213 13:45:11.803097 27070 /work/analytical_engine/core/server/dispatcher.cc:141] Worker 0: VineyardError occurred on worker 0: VineyardError occurred on worker 0: /download/v6d/modules/graph/loader/fragment_loader_utils.cc:45: SyncSchema -> Assertion failed: field_num > 0: Empty table list cannot be used for normalizing schema
2022-12-13 13:45:11,816 [ERROR][rpc:188]: Runstep failed with code: ANALYTICAL_ENGINE_INTERNAL_ERROR, message: Error occurred during RunStep, The traceback is: Traceback (most recent call last):

仔细看了下日志，通过prefix file name ,好像没用通过 adaptors/read_orc.py 去读orc文件，好像是按照csv 格式读取的ORC，造成读取文件失败
After carefully looking at the log, it seems that it is useless to read the orc file through the prefix file name. It seems to read the ORC file in csv format, causing the failure to read the file.
cc @sighingnow

sighingnow Dec 14, 2022
Maintainer

版本问题？更新一下 graphscope的版本？

pip3 install -U graphscope

sighingnow · 2022-12-13T09:26:42Z

sighingnow
Dec 13, 2022
Maintainer

Internally graphscope use fsspec to list files, see https://github.com/v6d-io/v6d/blob/main/modules/io/python/drivers/io/adaptors/read_orc.py#L126

Could you please take a try to install fsspec (https://github.com/fsspec/filesystem_spec) and see if

fs = ....
fs.glob("hdfs:///user/hive/warehouse/community.db/transaction_edges/job_id=112/part")

yields any output?

Thanks!

3 replies

JackyYangPassion Dec 13, 2022
Author

it`s success get files with prefix:

import fsspec
from fsspec.core import split_protocol
fs = fsspec.filesystem("hdfs")
files = fs.glob('hdfs:///user/hive/warehouse/community.db/transaction_edges/job_id=112/part'+ '*')
print(files)

result:
['/user/hive/warehouse/community.db/transaction_edges/job_id=112/part-00000-9ff0affe-a051-4ebe-83ff-691fb725ac0e.c000.snappy.orc', '/user/hive/warehouse/community.db/transaction_edges/job_id=112/part-00001-9ff0affe-a051-4ebe-83ff-691fb725ac0e.c000.snappy.orc']

@sighingnow

sighingnow Dec 13, 2022
Maintainer

Can I access these two files somewhere?

JackyYangPassion Dec 13, 2022
Author

This is internal data, can you simulate two orc files?

sighingnow · 2022-12-14T02:42:01Z

sighingnow
Dec 14, 2022
Maintainer

你现在版本是？

1 reply

JackyYangPassion Dec 14, 2022
Author

pip3 install -U graphscope --pre
version registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:0.19.0a20221210
@sighingnow

sighingnow · 2022-12-14T07:08:50Z

sighingnow
Dec 14, 2022
Maintainer

I see. We currently have no way to infer that "hdfs:///user/hive/warehouse/community.db/transaction_edges/job_id=112/part" indicates some ORC files....

I think we need to add some extra arguments for such cases.

11 replies

JackyYangPassion Dec 21, 2022
Author

full log:

2022-12-21 08:32:42,035 [DEBUG][script:81]: command is: /root/anaconda3/lib/python3.9/site-packages/vineyard/drivers/io/ssh.sh localhost vineyard_read_bytes /tmp/vineyard.sock.1671611546.514063 "hdfs:///user/hive/warehouse/community.db/transaction_vertices/job_id=112/part" eyJob3N0IjogIjE3Mi4zMS43Ljk4IiwgInBvcnQiOiA4MDIwLCAiZmlsZXR5cGUiOiAiT1JDIn0= eyJkZWxpbWl0ZXIiOiAiLCIsICJoZWFkZXJfcm93IjogdHJ1ZSwgInNjaGVtYSI6ICJhZGRyZXNzIiwgImNvbHVtbl90eXBlcyI6ICJzdGQ6OnN0cmluZyIsICJpbmNsdWRlX2FsbF9jb2x1bW5zIjogdHJ1ZX0= 1 0
DEBUG:vineyard:command is: /root/anaconda3/lib/python3.9/site-packages/vineyard/drivers/io/ssh.sh localhost vineyard_read_bytes /tmp/vineyard.sock.1671611546.514063 "hdfs:///user/hive/warehouse/community.db/transaction_vertices/job_id=112/part" eyJob3N0IjogIjE3Mi4zMS43Ljk4IiwgInBvcnQiOiA4MDIwLCAiZmlsZXR5cGUiOiAiT1JDIn0= eyJkZWxpbWl0ZXIiOiAiLCIsICJoZWFkZXJfcm93IjogdHJ1ZSwgInNjaGVtYSI6ICJhZGRyZXNzIiwgImNvbHVtbl90eXBlcyI6ICJzdGQ6OnN0cmluZyIsICJpbmNsdWRlX2FsbF9jb2x1bW5zIjogdHJ1ZX0= 1 0
I20221221 08:32:47.543521 15109 etcd_meta_service.cc:42] etcd watch use 17634755 microseconds, event size = 1
I20221221 08:32:47.545714 15068 etcd_meta_service.cc:159] etcd lock use 2903 microseconds
I20221221 08:32:47.546849 15078 etcd_meta_service.cc:235] etcd (last) txn use 455 microseconds
I20221221 08:32:47.546869 15109 etcd_meta_service.cc:42] etcd watch use 3300 microseconds, event size = 2
I20221221 08:32:47.547998 15109 etcd_meta_service.cc:42] etcd watch use 1108 microseconds, event size = 1
DEBUG:vineyard:driver: [return] o0010e80ea801f281
2022-12-21 08:32:47,555 [DEBUG][launcher:155]: driver: [return] o0010e80ea801f281
DEBUG:vineyard:[wait] partial ids = [o0010e80ea801f281]
2022-12-21 08:32:47,555 [DEBUG][stream:171]: [wait] partial ids = [o0010e80ea801f281]
I20221221 08:32:47.746935 15082 etcd_meta_service.cc:159] etcd lock use 700 microseconds
I20221221 08:32:47.746990 15109 etcd_meta_service.cc:42] etcd watch use 198725 microseconds, event size = 1
I20221221 08:32:47.749217 15109 etcd_meta_service.cc:42] etcd watch use 2209 microseconds, event size = 2
I20221221 08:32:47.750211 15086 etcd_meta_service.cc:235] etcd (last) txn use 1436 microseconds
I20221221 08:32:47.750700 15109 etcd_meta_service.cc:42] etcd watch use 1458 microseconds, event size = 1
2022-12-21 08:32:47,962 [DEBUG][script:81]: command is: /root/anaconda3/lib/python3.9/site-packages/vineyard/drivers/io/ssh.sh localhost vineyard_parse_bytes_to_dataframe /tmp/vineyard.sock.1671611546.514063 o0010e80eb8aac695 1 0
DEBUG:vineyard:command is: /root/anaconda3/lib/python3.9/site-packages/vineyard/drivers/io/ssh.sh localhost vineyard_parse_bytes_to_dataframe /tmp/vineyard.sock.1671611546.514063 o0010e80eb8aac695 1 0
I20221221 08:32:50.219476 15109 etcd_meta_service.cc:42] etcd watch use 2468753 microseconds, event size = 1
I20221221 08:32:50.220649 15090 etcd_meta_service.cc:159] etcd lock use 8423 microseconds
I20221221 08:32:50.223790 15094 etcd_meta_service.cc:235] etcd (last) txn use 601 microseconds
I20221221 08:32:50.224139 15109 etcd_meta_service.cc:42] etcd watch use 4621 microseconds, event size = 2
I20221221 08:32:50.224869 15109 etcd_meta_service.cc:42] etcd watch use 706 microseconds, event size = 1
DEBUG:vineyard:driver: [return] o0010e81035e3b0a5
2022-12-21 08:32:50,234 [DEBUG][launcher:155]: driver: [return] o0010e81035e3b0a5
DEBUG:vineyard:[wait] partial ids = [o0010e81035e3b0a5]
2022-12-21 08:32:50,234 [DEBUG][stream:171]: [wait] partial ids = [o0010e81035e3b0a5]
2022-12-21 08:32:50,383 [DEBUG][launcher:155]: driver: [error] Traceback (most recent call last):
DEBUG:vineyard:driver: [error] Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.9/site-packages/vineyard/drivers/io/adaptors/parse_bytes_to_dataframe.py", line 138, in parse_bytes
  File "/root/anaconda3/lib/python3.9/site-packages/vineyard/drivers/io/adaptors/parse_bytes_to_dataframe.py", line 138, in parse_bytes
    table = parse_dataframe_blocks(
    table = parse_dataframe_blocks(
  File "/root/anaconda3/lib/python3.9/site-packages/vineyard/drivers/io/adaptors/parse_bytes_to_dataframe.py", line 38, in parse_dataframe_blocks
  File "/root/anaconda3/lib/python3.9/site-packages/vineyard/drivers/io/adaptors/parse_bytes_to_dataframe.py", line 38, in parse_dataframe_blocks
    return pa.csv.read_csv(reader, read_options, parse_options, convert_options)
    return pa.csv.read_csv(reader, read_options, parse_options, convert_options)
  File "pyarrow/_csv.pyx", line 886, in pyarrow._csv.read_csv
  File "pyarrow/_csv.pyx", line 886, in pyarrow._csv.read_csv
  File "pyarrow/_csv.pyx", line 895, in pyarrow._csv.read_csv
  File "pyarrow/_csv.pyx", line 895, in pyarrow._csv.read_csv
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 118, in pyarrow.lib.check_status
  File "pyarrow/error.pxi", line 118, in pyarrow.lib.check_status
pyarrow.lib.ArrowKeyError: Column 'address' in include_columns does not exist in CSV file
pyarrow.lib.ArrowKeyError: Column 'address' in include_columns does not exist in CSV file


I20221221 08:32:50.449991 15071 etcd_meta_service.cc:159] etcd lock use 1496 microseconds
I20221221 08:32:50.450039 15109 etcd_meta_service.cc:42] etcd watch use 225132 microseconds, event size = 1
I20221221 08:32:50.454768 15109 etcd_meta_service.cc:42] etcd watch use 4713 microseconds, event size = 2
I20221221 08:32:50.454820 15099 etcd_meta_service.cc:235] etcd (last) txn use 1639 microseconds
I20221221 08:32:50.456526 15109 etcd_meta_service.cc:42] etcd watch use 1723 microseconds, event size = 1
Exception ignored in: <function Launcher.__del__ at 0x7f7df5c86790>
2022-12-21 08:32:50,765 [DEBUG][op_executor:808]: new_protocol = vineyard, new_source = o0010e810476ce29f
Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.9/site-packages/vineyard/launcher/launcher.py", line 63, in __del__
    self.join()
2022-12-21 08:32:50,768 [INFO][utils:1592]: Codegened graph type: vineyard::ArrowFragment<std::string,uint64_t,vineyard::ArrowVertexMap<vineyard::arrow_string_view,uint64_t>>, Graph header: vineyard/graph/fragment/arrow_fragment.h
  File "/root/anaconda3/lib/python3.9/site-packages/vineyard/drivers/io/stream.py", line 143, in join
    raise RuntimeError(
RuntimeError: Subprocesses failed with the following error: 
Failed to launch job [/root/anaconda3/lib/python3.9/site-packages/vineyard/drivers/io/ssh.sh localhost vineyard_parse_bytes_to_dataframe /tmp/vineyard.sock.1671611546.514063 o0010e80eb8aac695 ...], exited with 255: 

extra diagnostics are as follows: Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.9/site-packages/vineyard/drivers/io/adaptors/parse_bytes_to_dataframe.py", line 138, in parse_bytes
    table = parse_dataframe_blocks(
  File "/root/anaconda3/lib/python3.9/site-packages/vineyard/drivers/io/adaptors/parse_bytes_to_dataframe.py", line 38, in parse_dataframe_blocks
    return pa.csv.read_csv(reader, read_options, parse_options, convert_options)
  File "pyarrow/_csv.pyx", line 886, in pyarrow._csv.read_csv
  File "pyarrow/_csv.pyx", line 895, in pyarrow._csv.read_csv
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 118, in pyarrow.lib.check_status
pyarrow.lib.ArrowKeyError: Column 'address' in include_columns does not exist in CSV file



Loading vertex labeled address and:   0%|          | 0/10 [00:00<?, ?it/s]I20221221 08:32:50.797770 15131 /opt/graphscope/include/graphscope/core/loader/arrow_fragment_loader.h:354] read vertex table from vineyard: o0010e810476ce29f
I20221221 08:32:50.806114 15288 /download/v6d/modules/graph/loader/arrow_fragment_loader.cc:147] loading table from vineyard: o0010e810476ce29f, part id = 0, part num = 1
I20221221 08:32:50.828019 15288 /download/v6d/modules/graph/loader/arrow_fragment_loader.cc:53] reading recordbatches from vineyard: total chunks = 1, part id = 0, part num = 1, start to read = 0, end to read = 1, split size = 1
E20221221 08:32:50.844837 15288 /download/v6d/modules/graph/loader/arrow_fragment_loader.cc:422] Failed to read from stream o0010e810476ce29f: Stream failed: Stream source failed
E20221221 08:32:50.853673 15131 /work/analytical_engine/core/server/dispatcher.cc:141] Worker 0: VineyardError occurred on worker 0: VineyardError occurred on worker 0: /download/v6d/modules/graph/loader/fragment_loader_utils.cc:45: SyncSchema -> Assertion failed: field_num > 0: Empty table list cannot be used for normalizing schema
vineyard::SyncSchema(std::shared_ptr<arrow::Table> const&, grape::CommSpec const&) + 0x841
std::result_of<std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::loadVertexTables(std::vector<std::shared_ptr<gs::detail::Vertex>, std::allocator<std::shared_ptr<gs::detail::Vertex> > > const&, int, int)::{lambda(std::shared_ptr<arrow::Table> const&)#2}&>::type (std::result_of<std::shared_ptr<arrow::Table> const&>::type)>::type vineyard::sync_gs_error<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::loadVertexTables(std::vector<std::shared_ptr<gs::detail::Vertex>, std::allocator<std::shared_ptr<gs::detail::Vertex> > > const&, int, int)::{lambda(std::shared_ptr<arrow::Table> const&)#2}&, std::shared_ptr<arrow::Table> const&>(grape::CommSpec const&, std::decay&&, (std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::loadVertexTables(std::vector<std::shared_ptr<gs::detail::Vertex>, std::allocator<std::shared_ptr<gs::detail::Vertex> > > const&, int, int)::{lambda(std::shared_ptr<arrow::Table> const&)#2}&>::type&&)...)::{lambda()#2}::operator()() const + 0x30
2022-12-21 08:32:50,862 [ERROR][rpc:188]: Runstep failed with code: ANALYTICAL_ENGINE_INTERNAL_ERROR, message: Error occurred during RunStep, The traceback is: Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.9/site-packages/gscoordinator/op_executor.py", line 103, in run_step
    for response in responses:
  File "/root/anaconda3/lib/python3.9/site-packages/grpc/_channel.py", line 426, in __next__
    return self._next()
  File "/root/anaconda3/lib/python3.9/site-packages/grpc/_channel.py", line 809, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.INTERNAL
	details = "VineyardError occurred on worker 0: VineyardError occurred on worker 0: /download/v6d/modules/graph/loader/fragment_loader_utils.cc:45: SyncSchema -> Assertion failed: field_num > 0: Empty table list cannot be used for normalizing schema
vineyard::SyncSchema(std::shared_ptr<arrow::Table> const&, grape::CommSpec const&) + 0x841
std::result_of<std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::loadVertexTables(std::vector<std::shared_ptr<gs::detail::Vertex>, std::allocator<std::shared_ptr<gs::detail::Vertex> > > const&, int, int)::{lambda(std::shared_ptr<arrow::Table> const&)#2}&>::type (std::result_of<std::shared_ptr<arrow::Table> const&>::type)>::type vineyard::sync_gs_error<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::loadVertexTables(std::vector<std::shared_ptr<gs::detail::Vertex>, std::allocator<std::shared_ptr<gs::detail::Vertex> > > const&, int, int)::{lambda(std::shared_ptr<arrow::Table> const&)#2}&, std::shared_ptr<arrow::Table> const&>(grape::CommSpec const&, std::decay&&, (std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::loadVertexTables(std::vector<std::shared_ptr<gs::detail::Vertex>, std::allocator<std::shared_ptr<gs::detail::Vertex> > > const&, int, int)::{lambda(std::shared_ptr<arrow::Table> const&)#2}&>::type&&)...)::{lambda()#2}::operator()() const + 0x30
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::loadVertexTables(std::vector<std::shared_ptr<gs::detail::Vertex>, std::allocator<std::shared_ptr<gs::detail::Vertex> > > const&, int, int) + 0x176A
std::result_of<std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexTables()::{lambda()#2}&>::type ()>::type vineyard::sync_gs_error<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexTables()::{lambda()#2}&>(grape::CommSpec const&, std::decay&&, (std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexTables()::{lambda()#2}&>::type&&)...)::{lambda()#2}::operator()() const + 0x34
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexTables() + 0x39D
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexEdgeTables() + 0x1BC
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::AddLabelsToFragment(unsigned long) + 0x3A
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::AddLabelsToFragmentAsFragmentGroup(unsigned long) + 0x27
AddLabelsToGraph + 0x36E
gs::GrapeInstance::addLabelsToGraph(gs::rpc::GSParams const&) + 0x6E7
gs::GrapeInstance::OnReceive(std::shared_ptr<gs::CommandDetail>) + 0x69A
gs::Dispatcher::processCmd(std::shared_ptr<gs::CommandDetail>) + 0xD0
gs::Dispatcher::publisherLoop() + 0x1E6
std::_V2::generic_category() + 0x60
start_thread + 0xDB
clone + 0x3F

"
	debug_error_string = "{"created":"@1671611570.858592657","description":"Error received from peer ipv4:127.0.0.1:63932","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"VineyardError occurred on worker 0: VineyardError occurred on worker 0: /download/v6d/modules/graph/loader/fragment_loader_utils.cc:45: SyncSchema -> Assertion failed: field_num > 0: Empty table list cannot be used for normalizing schema\nvineyard::SyncSchema(std::shared_ptr<arrow::Table> const&, grape::CommSpec const&) + 0x841\nstd::result_of<std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::loadVertexTables(std::vector<std::shared_ptr<gs::detail::Vertex>, std::allocator<std::shared_ptr<gs::detail::Vertex> > > const&, int, int)::{lambda(std::shared_ptr<arrow::Table> const&)#2}&>::type (std::result_of<std::shared_ptr<arrow::Table> const&>::type)>::type vineyard::sync_gs_error<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::loadVertexTables(std::vector<std::shared_ptr<gs::detail::Vertex>, std::allocator<std::shared_ptr<gs::detail::Vertex> > > const&, int, int)::{lambda(std::shared_ptr<arrow::Table> const&)#2}&, std::shared_ptr<arrow::Table> const&>(grape::CommSpec const&, std::decay&&, (std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::loadVertexTables(std::vector<std::shared_ptr<gs::detail::Vertex>, std::allocator<std::shared_ptr<gs::detail::Vertex> > > const&, int, int)::{lambda(std::shared_ptr<arrow::Table> const&)#2}&>::type&&)...)::{lambda()#2}::operator()() const + 0x30\ngs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::loadVertexTables(std::vector<std::shared_ptr<gs::detail::Vertex>, std::allocator<std::shared_ptr<gs::detail::Vertex> > > const&, int, int) + 0x176A\nstd::result_of<std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexTables()::{lambda()#2}&>::type ()>::type vineyard::sync_gs_error<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexTables()::{lambda()#2}&>(grape::CommSpec const&, std::decay&&, (std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexTables()::{lambda()#2}&>::type&&)...)::{lambda()#2}::operator()() const + 0x34\ngs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexTables() + 0x39D\ngs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexEdgeTables() + 0x1BC\ngs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::AddLabelsToFragment(unsigned long) + 0x3A\ngs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::AddLabelsToFragmentAsFragmentGroup(unsigned long) + 0x27\nAddLabelsToGraph + 0x36E\ngs::GrapeInstance::addLabelsToGraph(gs::rpc::GSParams const&) + 0x6E7\ngs::GrapeInstance::OnReceive(std::shared_ptr<gs::CommandDetail>) + 0x69A\ngs::Dispatcher::processCmd(std::shared_ptr<gs::CommandDetail>) + 0xD0\ngs::Dispatcher::publisherLoop() + 0x1E6\nstd::_V2::generic_category() + 0x60\nstart_thread + 0xDB\nclone + 0x3F\n\n","grpc_status":13}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.9/site-packages/gscoordinator/coordinator.py", line 304, in _RunStep
    head, bodies = self._operation_executor.run_on_analytical_engine(
  File "/root/anaconda3/lib/python3.9/site-packages/gscoordinator/monitor.py", line 193, in runOnAnalyticalEngineWarp
    res = func(instance, dag_def, dag_bodies, loader_op_bodies)
  File "/root/anaconda3/lib/python3.9/site-packages/gscoordinator/op_executor.py", line 169, in run_on_analytical_engine
    response_head, response_bodies = self.run_step(dag_def, dag_bodies)
  File "/root/anaconda3/lib/python3.9/site-packages/gscoordinator/op_executor.py", line 117, in run_step
    raise AnalyticalEngineInternalError(msg)
graphscope.framework.errors.AnalyticalEngineInternalError: VineyardError occurred on worker 0: VineyardError occurred on worker 0: /download/v6d/modules/graph/loader/fragment_loader_utils.cc:45: SyncSchema -> Assertion failed: field_num > 0: Empty table list cannot be used for normalizing schema
vineyard::SyncSchema(std::shared_ptr<arrow::Table> const&, grape::CommSpec const&) + 0x841
std::result_of<std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::loadVertexTables(std::vector<std::shared_ptr<gs::detail::Vertex>, std::allocator<std::shared_ptr<gs::detail::Vertex> > > const&, int, int)::{lambda(std::shared_ptr<arrow::Table> const&)#2}&>::type (std::result_of<std::shared_ptr<arrow::Table> const&>::type)>::type vineyard::sync_gs_error<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::loadVertexTables(std::vector<std::shared_ptr<gs::detail::Vertex>, std::allocator<std::shared_ptr<gs::detail::Vertex> > > const&, int, int)::{lambda(std::shared_ptr<arrow::Table> const&)#2}&, std::shared_ptr<arrow::Table> const&>(grape::CommSpec const&, std::decay&&, (std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::loadVertexTables(std::vector<std::shared_ptr<gs::detail::Vertex>, std::allocator<std::shared_ptr<gs::detail::Vertex> > > const&, int, int)::{lambda(std::shared_ptr<arrow::Table> const&)#2}&>::type&&)...)::{lambda()#2}::operator()() const + 0x30
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::loadVertexTables(std::vector<std::shared_ptr<gs::detail::Vertex>, std::allocator<std::shared_ptr<gs::detail::Vertex> > > const&, int, int) + 0x176A
std::result_of<std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexTables()::{lambda()#2}&>::type ()>::type vineyard::sync_gs_error<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexTables()::{lambda()#2}&>(grape::CommSpec const&, std::decay&&, (std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexTables()::{lambda()#2}&>::type&&)...)::{lambda()#2}::operator()() const + 0x34
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexTables() + 0x39D
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexEdgeTables() + 0x1BC
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::AddLabelsToFragment(unsigned long) + 0x3A
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::AddLabelsToFragmentAsFragmentGroup(unsigned long) + 0x27
AddLabelsToGraph + 0x36E
gs::GrapeInstance::addLabelsToGraph(gs::rpc::GSParams const&) + 0x6E7
gs::GrapeInstance::OnReceive(std::shared_ptr<gs::CommandDetail>) + 0x69A
gs::Dispatcher::processCmd(std::shared_ptr<gs::CommandDetail>) + 0xD0
gs::Dispatcher::publisherLoop() + 0x1E6
std::_V2::generic_category() + 0x60
start_thread + 0xDB
clone + 0x3F



gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::loadVertexTables(std::vector<std::shared_ptr<gs::detail::Vertex>, std::allocator<std::shared_ptr<gs::detail::Vertex> > > const&, int, int) + 0x176A
std::result_of<std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexTables()::{lambda()#2}&>::type ()>::type vineyard::sync_gs_error<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexTables()::{lambda()#2}&>(grape::CommSpec const&, std::decay&&, (std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexTables()::{lambda()#2}&>::type&&)...)::{lambda()#2}::operator()() const + 0x34
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexTables() + 0x39D
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexEdgeTables() + 0x1BC
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::AddLabelsToFragment(unsigned long) + 0x3A
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::AddLabelsToFragmentAsFragmentGroup(unsigned long) + 0x27
AddLabelsToGraph + 0x36E
gs::GrapeInstance::addLabelsToGraph(gs::rpc::GSParams const&) + 0x6E7
gs::GrapeInstance::OnReceive(std::shared_ptr<gs::CommandDetail>) + 0x69A
gs::Dispatcher::processCmd(std::shared_ptr<gs::CommandDetail>) + 0xD0
gs::Dispatcher::publisherLoop() + 0x1E6
std::_V2::generic_category() + 0x60
start_thread + 0xDB
clone + 0x3F
---------------------------------------------------------------------------
AnalyticalEngineInternalError             Traceback (most recent call last)
Input In [5], in <cell line: 2>()
      1 graph = (
----> 2         graph.add_vertices(Loader("hdfs:///user/hive/warehouse/community.db/transaction_vertices/job_id=112/part",filetype="ORC",host="ip", port=8020)
      3                            ,label="address"
      4                            ,vid_field="address"
      5                           )
      6         .add_edges(
      7                    Loader("hdfs:///user/hive/warehouse/community.db/transaction_edges/job_id=112/part",filetype="ORC",host="ip", port=8020)
      8                    ,label="transactions"
      9                    ,src_label="address"
     10                    ,dst_label="address"
     11                    ,src_field='from_address'
     12                    ,dst_field='to_address'
     13         )
     14     )

File ~/anaconda3/lib/python3.9/site-packages/graphscope/framework/graph.py:1086, in Graph.add_vertices(self, vertices, label, properties, vid_field)
   1084 if not self.loaded():
   1085     raise RuntimeError("The graph is not loaded")
-> 1086 return self._session._wrapper(
   1087     self._graph_node.add_vertices(vertices, label, properties, vid_field)
   1088 )

File ~/anaconda3/lib/python3.9/site-packages/graphscope/client/session.py:930, in Session._wrapper(self, dag_node)
    928 def _wrapper(self, dag_node):
    929     if self.eager():
--> 930         return self.run(dag_node)
    931     return dag_node

File ~/anaconda3/lib/python3.9/site-packages/graphscope/client/session.py:952, in Session.run(self, fetches)
    933 def run(self, fetches):
    934     """Run operations of `fetches`.
    935     Args:
    936         fetches: :class:`Operation`
   (...)
    950         Different values for different output types of :class:`Operation`
    951     """
--> 952     return self.run_fetches(fetches)

File ~/anaconda3/lib/python3.9/site-packages/graphscope/client/session.py:962, in Session.run_fetches(self, fetches)
    960 fetch_handler = _FetchHandler(self.dag, fetches)
    961 try:
--> 962     response = self._grpc_client.run(fetch_handler.targets)
    963 except FatalError:
    964     self.close()

File ~/anaconda3/lib/python3.9/site-packages/graphscope/client/rpc.py:98, in GRPCClient.run(self, dag_def)
     97 def run(self, dag_def):
---> 98     return self._run_step_impl(dag_def)

File ~/anaconda3/lib/python3.9/site-packages/graphscope/client/utils.py:156, in handle_grpc_error_with_retry.<locals>.with_grpc_catch(*args, **kwargs)
    154 while True:
    155     try:
--> 156         return fn(*args, **kwargs)
    157     except grpc.RpcError as exc:
    158         code = exc.code()

File ~/anaconda3/lib/python3.9/site-packages/graphscope/client/rpc.py:198, in GRPCClient._run_step_impl(self, dag_def)
    196             raise exc[0](*exc[1:])
    197         else:
--> 198             raise exc
    199 return response

AnalyticalEngineInternalError: VineyardError occurred on worker 0: VineyardError occurred on worker 0: /download/v6d/modules/graph/loader/fragment_loader_utils.cc:45: SyncSchema -> Assertion failed: field_num > 0: Empty table list cannot be used for normalizing schema
vineyard::SyncSchema(std::shared_ptr<arrow::Table> const&, grape::CommSpec const&) + 0x841
std::result_of<std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::loadVertexTables(std::vector<std::shared_ptr<gs::detail::Vertex>, std::allocator<std::shared_ptr<gs::detail::Vertex> > > const&, int, int)::{lambda(std::shared_ptr<arrow::Table> const&)#2}&>::type (std::result_of<std::shared_ptr<arrow::Table> const&>::type)>::type vineyard::sync_gs_error<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::loadVertexTables(std::vector<std::shared_ptr<gs::detail::Vertex>, std::allocator<std::shared_ptr<gs::detail::Vertex> > > const&, int, int)::{lambda(std::shared_ptr<arrow::Table> const&)#2}&, std::shared_ptr<arrow::Table> const&>(grape::CommSpec const&, std::decay&&, (std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::loadVertexTables(std::vector<std::shared_ptr<gs::detail::Vertex>, std::allocator<std::shared_ptr<gs::detail::Vertex> > > const&, int, int)::{lambda(std::shared_ptr<arrow::Table> const&)#2}&>::type&&)...)::{lambda()#2}::operator()() const + 0x30
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::loadVertexTables(std::vector<std::shared_ptr<gs::detail::Vertex>, std::allocator<std::shared_ptr<gs::detail::Vertex> > > const&, int, int) + 0x176A
std::result_of<std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexTables()::{lambda()#2}&>::type ()>::type vineyard::sync_gs_error<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexTables()::{lambda()#2}&>(grape::CommSpec const&, std::decay&&, (std::decay<gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexTables()::{lambda()#2}&>::type&&)...)::{lambda()#2}::operator()() const + 0x34
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexTables() + 0x39D
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::LoadVertexEdgeTables() + 0x1BC
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::AddLabelsToFragment(unsigned long) + 0x3A
gs::ArrowFragmentLoader<std::string, unsigned long, vineyard::ArrowVertexMap>::AddLabelsToFragmentAsFragmentGroup(unsigned long) + 0x27
AddLabelsToGraph + 0x36E
gs::GrapeInstance::addLabelsToGraph(gs::rpc::GSParams const&) + 0x6E7
gs::GrapeInstance::OnReceive(std::shared_ptr<gs::CommandDetail>) + 0x69A
gs::Dispatcher::processCmd(std::shared_ptr<gs::CommandDetail>) + 0xD0
gs::Dispatcher::publisherLoop() + 0x1E6
std::_V2::generic_category() + 0x60
start_thread + 0xDB
clone + 0x3F

I20221221 08:33:28.183012 15104 etcd_meta_service.cc:159] etcd lock use 707 microseconds
I20221221 08:33:28.183071 15109 etcd_meta_service.cc:42] etcd watch use 37726527 microseconds, event size = 1
I20221221 08:33:28.188354 15109 etcd_meta_service.cc:42] etcd watch use 5267 microseconds, event size = 1
I20221221 08:33:28.188382 15080 etcd_meta_service.cc:235] etcd (last) txn use 541 microseconds
I20221221 08:33:28.193349 15109 etcd_meta_service.cc:42] etcd watch use 4962 microseconds, event size = 1

sighingnow Dec 21, 2022
Maintainer

As the error message indicated, these files are still treated as CSV. The newly added filetype argument requires vineyard-io>=0.11.2.

I have just uploaded the wheel to pypi. Looks like you are using local sessions, so

pip3 install vineyard-io -U

and try again, it should work. Please confirm you have installed vineyard-io==0.11.2 during the above process.

JackyYangPassion Dec 21, 2022
Author

Thank you very much, it works local seesion!!

JackyYangPassion Dec 21, 2022
Author

when run on k8s

session = graphscope.session(
                             k8s_coordinator_cpu=1,
                             k8s_coordinator_mem="1Gi",
                             k8s_etcd_cpu=1.0,
                             k8s_etcd_num_pods=1,
                             k8s_etcd_mem='1Gi',
                             k8s_vineyard_cpu=0.2,
                             k8s_vineyard_mem="1Gi",
                             vineyard_shared_mem="2Gi",
                             k8s_engine_cpu=0.2,
                             k8s_namespace='gs-new-orc',
                             k8s_engine_mem="1Gi",
                             num_workers=1)

in engin pod vineyard-io = 0.11.1

how can update vineyard-io>=0.11.2. on k8s cluster?

sighingnow Dec 22, 2022
Maintainer

Our nightly CI seems failed to publish images in recently nights.

You could launch the session, then kubectl exec -it <coordinator pod> /bin/bash to enter the coordinator pod, then pip3 install -U vineyard-io, and loading your graph from client.

JackyYangPassion · 2023-09-27T09:14:34Z

JackyYangPassion
Sep 27, 2023
Author

can support load GraphAR format file now?

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Is HDFS loader support parquet orc file format? #2264

{{title}}

Replies: 6 comments 25 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Is HDFS loader support parquet orc file format? #2264

JackyYangPassion Dec 1, 2022

Replies: 6 comments · 25 replies

sighingnow Dec 1, 2022 Maintainer

JackyYangPassion Dec 1, 2022 Author

sighingnow Dec 1, 2022 Maintainer

sighingnow Dec 12, 2022 Maintainer

JackyYangPassion Dec 12, 2022 Author

JackyYangPassion Dec 13, 2022 Author

JackyYangPassion Dec 13, 2022 Author

sighingnow Dec 14, 2022 Maintainer

sighingnow Dec 13, 2022 Maintainer

JackyYangPassion Dec 13, 2022 Author

sighingnow Dec 13, 2022 Maintainer

JackyYangPassion Dec 13, 2022 Author

sighingnow Dec 14, 2022 Maintainer

JackyYangPassion Dec 14, 2022 Author

sighingnow Dec 14, 2022 Maintainer

JackyYangPassion Dec 21, 2022 Author

sighingnow Dec 21, 2022 Maintainer

JackyYangPassion Dec 21, 2022 Author

JackyYangPassion Dec 21, 2022 Author

sighingnow Dec 22, 2022 Maintainer

JackyYangPassion Sep 27, 2023 Author

JackyYangPassion
Dec 1, 2022

Replies: 6 comments 25 replies

sighingnow
Dec 1, 2022
Maintainer

JackyYangPassion Dec 1, 2022
Author

sighingnow
Dec 1, 2022
Maintainer

sighingnow Dec 12, 2022
Maintainer

JackyYangPassion Dec 12, 2022
Author

JackyYangPassion Dec 13, 2022
Author

JackyYangPassion Dec 13, 2022
Author

sighingnow Dec 14, 2022
Maintainer

sighingnow
Dec 13, 2022
Maintainer

JackyYangPassion Dec 13, 2022
Author

sighingnow Dec 13, 2022
Maintainer

JackyYangPassion Dec 13, 2022
Author

sighingnow
Dec 14, 2022
Maintainer

JackyYangPassion Dec 14, 2022
Author

sighingnow
Dec 14, 2022
Maintainer

JackyYangPassion Dec 21, 2022
Author

sighingnow Dec 21, 2022
Maintainer

JackyYangPassion Dec 21, 2022
Author

JackyYangPassion Dec 21, 2022
Author

sighingnow Dec 22, 2022
Maintainer

JackyYangPassion
Sep 27, 2023
Author