Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] 使用cassandra作为后端，持久化配置后，容器重建无法完全加载 #2711

Open
1 task done
yuntong613 opened this issue Dec 24, 2024 · 4 comments
Open
1 task done

[Bug] 使用cassandra作为后端，持久化配置后，容器重建无法完全加载 #2711

yuntong613 opened this issue Dec 24, 2024 · 4 comments
Labels
bug Something isn't working cassandra Cassandra backend

Comments

@yuntong613
Copy link

yuntong613 commented Dec 24, 2024 •
edited
Loading

Bug Type (问题类型)

server status (启动/运行异常)

Before submit

  • 我已经确认现有的 Issues 与 FAQ 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)

Environment (环境信息)

  • Server Version: 1.5.0
  • Backend: cassandra 4
  • OS: docker
  • Data Size: 100以内

Expected & Actual behavior (期望与实际表现)

2024-12-24 07:20:12 [main] [INFO] o.a.h.c.GraphManager - Graph 'hugegraph2' was successfully configured via './conf/graphs/hugegraph2.properties'
2024-12-24 07:20:12 [main] [INFO] o.a.h.c.GraphManager - Graph 'hugegraph' was successfully configured via './conf/graphs/hugegraph.properties'
2024-12-24 07:20:12 [main] [INFO] o.a.h.c.GraphManager - RpcServer is not enabled, skip starting rpc service
2024-12-24 07:20:12 [main] [INFO] o.a.h.c.GraphManager - RpcServer is not enabled, skip wait graphs ready
2024-12-24 07:20:12 [main] [INFO] o.a.h.c.GraphManager - Check backend version
2024-12-24 07:20:12 [main] [ERROR] o.a.h.b.s.BackendStoreInfo - The backend driver version '1.11' is inconsistent with the data version 'null' of backend store for graph 'hugegraph2'
2024-12-24 07:20:12 [main] [INFO] o.a.h.StandardHugeGraph - Close graph standardhugegraph[hugegraph]
2024-12-24 07:20:18 [main] [INFO] o.a.h.StandardHugeGraph - Close graph standardhugegraph[hugegraph2]
2024-12-24 07:20:24 [main] [ERROR] o.a.h.d.HugeGraphServer - HugeRestServer start error:
org.apache.hugegraph.backend.BackendException: The backend store version is inconsistent
at org.apache.hugegraph.core.GraphManager.checkBackendVersionOrExit(GraphManager.java:456) ~[hugegraph-api-1.5.0.jar:0.71.0.0]
at org.apache.hugegraph.core.GraphManager.init(GraphManager.java:124) ~[hugegraph-api-1.5.0.jar:0.71.0.0]
at org.apache.hugegraph.server.ApplicationConfig$GraphManagerFactory$1.onEvent(ApplicationConfig.java:143) ~[hugegraph-api-1.5.0.jar:0.71.0.0]
at org.glassfish.jersey.server.internal.monitoring.CompositeApplicationEventListener.onEvent(CompositeApplicationEventListener.java:49) ~[jersey-server-3.0.3.jar:?]
at org.glassfish.jersey.server.internal.monitoring.MonitoringContainerListener.onStartup(MonitoringContainerListener.java:56) ~[jersey-server-3.0.3.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.onStartup(ApplicationHandler.java:711) ~[jersey-server-3.0.3.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.start(GrizzlyHttpContainer.java:330) ~[jersey-container-grizzly2-http-3.0.3.jar:?]
at org.glassfish.grizzly.http.server.HttpHandlerChain.start(HttpHandlerChain.java:376) ~[grizzly-http-server-3.0.1.jar:3.0.1]
at org.glassfish.grizzly.http.server.HttpServer.setupHttpHandler(HttpServer.java:268) ~[grizzly-http-server-3.0.1.jar:3.0.1]
at org.glassfish.grizzly.http.server.HttpServer.start(HttpServer.java:245) ~[grizzly-http-server-3.0.1.jar:3.0.1]
at org.apache.hugegraph.server.RestServer.start(RestServer.java:70) ~[hugegraph-api-1.5.0.jar:0.71.0.0]
at org.apache.hugegraph.server.RestServer.start(RestServer.java:177) ~[hugegraph-api-1.5.0.jar:0.71.0.0]
at org.apache.hugegraph.dist.HugeRestServer.start(HugeRestServer.java:31) ~[hugegraph-dist-1.5.0.jar:1.5.0]
at org.apache.hugegraph.dist.HugeGraphServer.(HugeGraphServer.java:61) ~[hugegraph-dist-1.5.0.jar:1.5.0]
at org.apache.hugegraph.dist.HugeGraphServer.main(HugeGraphServer.java:127) ~[hugegraph-dist-1.5.0.jar:1.5.0]
2024-12-24 07:20:24 [main] [INFO] o.a.h.HugeFactory - HugeFactory shutdown
2024-12-24 07:20:24 [hugegraph-shutdown] [INFO] o.a.h.HugeFactory - HugeGraph is shutting down
t(RestServer.java:70)
at org.apache.hugegraph.server.RestServer.start(RestServer.java:177)
at org.apache.hugegraph.dist.HugeRestServer.start(HugeRestServer.java:31)
at org.apache.hugegraph.dist.HugeGraphServer.(HugeGraphServer.java:61)
at org.apache.hugegraph.dist.HugeGraphServer.main(HugeGraphServer.java:127)

Docker Compose 配置

version: "3"

services:
  server:
    image: hugegraph/hugegraph
    container_name: cas-server
    ports:
      - 8080:8080
    restart: always
    environment:
      hugegraph.gremlin.graph: org.apache.hugegraph.HugeFactory
      hugegraph.backend: cassandra
      hugegraph.serializer: cassandra
      hugegraph.cassandra.host: cas-cassandra
      hugegraph.cassandra.port: 9042
      PASSWORD: ABCD123
    volumes:
      - ./hugegraph/conf:/hugegraph-server/conf/graphs
    networks:
      - ca-network
    depends_on:
      - cassandra
    healthcheck:
      test: ["CMD", "bin/gremlin-console.sh", "--" ,"-e", "scripts/remote-connect.groovy"]
      interval: 10s
      timeout: 30s
      retries: 3

  cassandra:
    image: cassandra:4
    container_name: cas-cassandra
    ports:
      - 7000:7000
      - 9042:9042
    restart: always
    security_opt:
      - seccomp:unconfined
    volumes:
      - ./cassandra/config/cassandra.yaml:/etc/cassandra/cassandra.yaml
      - ./cassandra/data:/var/lib/cassandra
    networks:
      - ca-network
    healthcheck:
      test: ["CMD", "cqlsh", "--execute", "describe keyspaces;"]
      interval: 10s
      timeout: 30s
      retries: 5
      
  #loader:
  #  image: hugegraph/loader
  #  container_name: hugegraph_loader
  #  restart: always
  #  volumes:
  #    - /root/hugegraph/hugegraph-dataset:/loader/dataset
  #  networks:
  #    - ca-network
    
  hubble:
    image: hugegraph/hubble
    container_name: hugegraph_hubble
    ports:
      - 8088:8088
    restart: always
    networks:
      - ca-network
networks:
  ca-network:
@yuntong613 yuntong613 added the bug Something isn't working label Dec 24, 2024
@dosubot dosubot bot added the cassandra Cassandra backend label Dec 24, 2024
@dosubot DosuBot
Copy link

dosubot bot commented Dec 24, 2024

I found a similar open issue related to backend store version inconsistency when using Cassandra as the backend. The error you're encountering, where the backend driver version is inconsistent with the data version, typically occurs when the backend store is not properly initialized.

To resolve this issue, you should initialize the backend store using the init-store.sh script. Here are the steps you should follow:

  1. Navigate to the HugeGraph directory:

    cd *hugegraph-${version}

  2. Run the initialization script:

    bin/init-store.sh

This script will set up the necessary backend store for HugeGraph to connect to Cassandra. Additionally, ensure that your hugegraph.properties file is correctly configured with the appropriate Cassandra settings, such as cassandra.host and cassandra.port, to match your Cassandra setup [1][2].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@yuntong613
Copy link
Author

yuntong613 commented Dec 24, 2024 •
edited
Loading

从日志上看hugegraph.properties 和hugegraph2.properties ，已经正确加载 @dosu

@yuntong613
Copy link
Author

拉取最新镜像后，机器重启，由于cassandra启动较慢，不会一直等待cassandra容器加载完，而导致server连接后端超时

2024-12-24 08:02:25 [main] [ERROR] o.a.h.d.HugeGraphServer - HugeRestServer start error:
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [class org.apache.hugegraph.auth.HugeFactoryAuthProxy]
        at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:84) ~[gremlin-core-3.5.1.jar:3.5.1]
        at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:72) ~[gremlin-core-3.5.1.jar:3.5.1]
        at org.apache.hugegraph.auth.StandardAuthenticator.setup(StandardAuthenticator.java:138) ~[hugegraph-api-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.auth.HugeAuthenticator.loadAuthenticator(HugeAuthenticator.java:143) ~[hugegraph-api-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.core.GraphManager.<init>(GraphManager.java:100) ~[hugegraph-api-1.5.0.jar:0.71.0.0]
        at org.apache.hugegraph.server.ApplicationConfig$GraphManagerFactory$1.onEvent(ApplicationConfig.java:141) ~[hugegraph-api-1.5.0.jar:0.71.0.0]
        at org.glassfish.jersey.server.internal.monitoring.CompositeApplicationEventListener.onEvent(CompositeApplicationEventListener.java:49) ~[jersey-server-3.0.3.jar:?]
        at org.glassfish.jersey.server.internal.monitoring.MonitoringContainerListener.onStartup(MonitoringContainerListener.java:56) ~[jersey-server-3.0.3.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.onStartup(ApplicationHandler.java:711) ~[jersey-server-3.0.3.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.start(GrizzlyHttpContainer.java:330) ~[jersey-container-grizzly2-http-3.0.3.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandlerChain.start(HttpHandlerChain.java:376) ~[grizzly-http-server-3.0.1.jar:3.0.1]
        at org.glassfish.grizzly.http.server.HttpServer.setupHttpHandler(HttpServer.java:268) ~[grizzly-http-server-3.0.1.jar:3.0.1]
        at org.glassfish.grizzly.http.server.HttpServer.start(HttpServer.java:245) ~[grizzly-http-server-3.0.1.jar:3.0.1]
        at org.apache.hugegraph.server.RestServer.start(RestServer.java:70) ~[hugegraph-api-1.5.0.jar:0.71.0.0]
        at org.apache.hugegraph.server.RestServer.start(RestServer.java:177) ~[hugegraph-api-1.5.0.jar:0.71.0.0]
        at org.apache.hugegraph.dist.HugeRestServer.start(HugeRestServer.java:31) ~[hugegraph-dist-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.dist.HugeGraphServer.<init>(HugeGraphServer.java:61) ~[hugegraph-dist-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.dist.HugeGraphServer.main(HugeGraphServer.java:127) ~[hugegraph-dist-1.5.0.jar:1.5.0]
Caused by: java.lang.reflect.InvocationTargetException
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:80) ~[gremlin-core-3.5.1.jar:3.5.1]
        ... 17 more
Caused by: org.apache.hugegraph.exception.ConnectionException: Failed to connect to Cassandra
        at org.apache.hugegraph.backend.store.cassandra.CassandraStore.open(CassandraStore.java:186) ~[hugegraph-cassandra-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.backend.tx.AbstractTransaction.<init>(AbstractTransaction.java:80) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.backend.tx.IndexableTransaction.<init>(IndexableTransaction.java:28) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.backend.tx.SchemaTransaction.<init>(SchemaTransaction.java:79) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.backend.cache.CachedSchemaTransaction.<init>(CachedSchemaTransaction.java:54) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.StandardHugeGraph.openSchemaTransaction(StandardHugeGraph.java:492) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.StandardHugeGraph$TinkerPopTransaction.getOrNewTransaction(StandardHugeGraph.java:1512) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.StandardHugeGraph$TinkerPopTransaction.doOpen(StandardHugeGraph.java:1421) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.open(AbstractTransaction.java:95) ~[gremlin-core-3.5.1.jar:3.5.1]
        at org.apache.tinkerpop.gremlin.structure.Transaction$READ_WRITE_BEHAVIOR$1.accept(Transaction.java:213) ~[gremlin-core-3.5.1.jar:3.5.1]
        at org.apache.tinkerpop.gremlin.structure.Transaction$READ_WRITE_BEHAVIOR$1.accept(Transaction.java:210) ~[gremlin-core-3.5.1.jar:3.5.1]
        at org.apache.tinkerpop.gremlin.structure.util.AbstractThreadLocalTransaction.doReadWrite(AbstractThreadLocalTransaction.java:92) ~[gremlin-core-3.5.1.jar:3.5.1]
        at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.readWrite(AbstractTransaction.java:136) ~[gremlin-core-3.5.1.jar:3.5.1]
        at org.apache.hugegraph.StandardHugeGraph.graphTransaction(StandardHugeGraph.java:569) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.StandardHugeGraph.backendStoreFeatures(StandardHugeGraph.java:287) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.StandardHugeGraph.<init>(StandardHugeGraph.java:247) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.HugeFactory.open(HugeFactory.java:88) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.HugeFactory.open(HugeFactory.java:66) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.auth.HugeFactoryAuthProxy.open(HugeFactoryAuthProxy.java:87) ~[hugegraph-api-1.5.0.jar:1.5.0]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:80) ~[gremlin-core-3.5.1.jar:3.5.1]
        ... 17 more
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cas-cassandra/172.18.0.4:9042 (com.datastax.driver.core.exceptions.TransportException: [cas-cassandra/172.18.0.4:9042] Cannot connect))
        at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:268) ~[cassandra-driver-core-3.6.0.jar:?]
        at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:107) ~[cassandra-driver-core-3.6.0.jar:?]
        at com.datastax.driver.core.Cluster$Manager.negotiateProtocolVersionAndConnect(Cluster.java:1652) ~[cassandra-driver-core-3.6.0.jar:?]
        at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1571) ~[cassandra-driver-core-3.6.0.jar:?]
        at com.datastax.driver.core.Cluster.init(Cluster.java:208) ~[cassandra-driver-core-3.6.0.jar:?]
        at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:376) ~[cassandra-driver-core-3.6.0.jar:?]
        at com.datastax.driver.core.Cluster.connect(Cluster.java:332) ~[cassandra-driver-core-3.6.0.jar:?]
        at org.apache.hugegraph.backend.store.cassandra.CassandraSessionPool$Session.open(CassandraSessionPool.java:224) ~[hugegraph-cassandra-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.backend.store.cassandra.CassandraStore.open(CassandraStore.java:168) ~[hugegraph-cassandra-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.backend.tx.AbstractTransaction.<init>(AbstractTransaction.java:80) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.backend.tx.IndexableTransaction.<init>(IndexableTransaction.java:28) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.backend.tx.SchemaTransaction.<init>(SchemaTransaction.java:79) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.backend.cache.CachedSchemaTransaction.<init>(CachedSchemaTransaction.java:54) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.StandardHugeGraph.openSchemaTransaction(StandardHugeGraph.java:492) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.StandardHugeGraph$TinkerPopTransaction.getOrNewTransaction(StandardHugeGraph.java:1512) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.StandardHugeGraph$TinkerPopTransaction.doOpen(StandardHugeGraph.java:1421) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.open(AbstractTransaction.java:95) ~[gremlin-core-3.5.1.jar:3.5.1]
        at org.apache.tinkerpop.gremlin.structure.Transaction$READ_WRITE_BEHAVIOR$1.accept(Transaction.java:213) ~[gremlin-core-3.5.1.jar:3.5.1]
        at org.apache.tinkerpop.gremlin.structure.Transaction$READ_WRITE_BEHAVIOR$1.accept(Transaction.java:210) ~[gremlin-core-3.5.1.jar:3.5.1]
        at org.apache.tinkerpop.gremlin.structure.util.AbstractThreadLocalTransaction.doReadWrite(AbstractThreadLocalTransaction.java:92) ~[gremlin-core-3.5.1.jar:3.5.1]
        at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.readWrite(AbstractTransaction.java:136) ~[gremlin-core-3.5.1.jar:3.5.1]
        at org.apache.hugegraph.StandardHugeGraph.graphTransaction(StandardHugeGraph.java:569) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.StandardHugeGraph.backendStoreFeatures(StandardHugeGraph.java:287) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.StandardHugeGraph.<init>(StandardHugeGraph.java:247) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.HugeFactory.open(HugeFactory.java:88) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.HugeFactory.open(HugeFactory.java:66) ~[hugegraph-core-1.5.0.jar:1.5.0]
        at org.apache.hugegraph.auth.HugeFactoryAuthProxy.open(HugeFactoryAuthProxy.java:87) ~[hugegraph-api-1.5.0.jar:1.5.0]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:80) ~[gremlin-core-3.5.1.jar:3.5.1]
        ... 17 more
2024-12-24 08:02:25 [main] [INFO] o.a.h.HugeFactory - HugeFactory shutdown
2024-12-24 08:02:28 [hugegraph-shutdown] [INFO] o.a.h.HugeFactory - HugeGraph is shutting down

@dosubot DosuBot
Copy link

dosubot bot commented Dec 24, 2024

请确认以下几点：

  1. 数据版本一致性：确保在重建容器之前，Cassandra 中的所有数据已经被正确初始化，并且没有版本不匹配的问题。可以尝试清空 Cassandra 数据目录后重新初始化。

  2. 配置文件路径：检查 Docker Compose 中的卷映射，确保 hugegraph.properties 和 hugegraph2.properties 文件的路径正确，并且文件内容与 Cassandra 的配置一致。

  3. 初始化步骤：在启动 HugeGraph 之前，确保已经运行 init-store.sh 脚本来初始化存储。

  4. 日志检查：查看 Cassandra 的日志，确保它在 HugeGraph 启动时已经完全启动并且没有错误。

如果这些步骤都确认无误，但问题仍然存在，可能需要进一步检查 Cassandra 和 HugeGraph 的版本兼容性，或者考虑在 HugeGraph 社区中寻求更多帮助。

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working cassandra Cassandra backend
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@yuntong613