Reconciler error in Starrocks Operator (invalid: metadata.resourceVersion) | Latest Helm Chart #28626

jimenez-f · 2023-08-04T02:35:23Z

General Question

I have an issue with the Starrocks Operator when apply the Starrocks Cluster with FE and BE. I add the Operator logs:

I0803 22:11:49.113850 1 starrockscluster_controller.go:83] StarRocksClusterReconciler reconcile the update crd name starrocks namespace kube-starrocks I0803 22:11:49.117336 1 statefulset.go:89] the statefulset name starrocks-fe new hash value 2344900785 old have value 3682884808 I0803 22:11:49.117483 1 k8sutils.go:94] Updating resource service namespace kube-starrocks name starrocks-fe kind &TypeMeta{Kind:,APIVersion:,} I0803 22:11:49.167775 1 k8sutils.go:94] Updating resource service namespace kube-starrocks name starrocks-fe-search kind &TypeMeta{Kind:,APIVersion:,} E0803 22:11:49.188632 1 fe_controller.go:111] FeController Sync create or patch internal service namespace kube-starrocks name starrocks-fe-search failed, message Service "starrocks-fe-search" is invalid: metadata.resourceVersion: Invalid value: "": must be specified for an update E0803 22:11:49.188697 1 starrockscluster_controller.go:111] StarRocksClusterReconciler reconcile sub resource reconcile failed, namespace=kube-starrocks, name=starrocks, controller=feController, error=Service "starrocks-fe-search" is invalid: metadata.resourceVersion: Invalid value: "": must be specified for an update 1.6911007091887467e+09 ERROR Reconciler error {"controller": "starrockscluster", "controllerGroup": "starrocks.com", "controllerKind": "StarRocksCluster", "StarRocksCluster": {"name":"starrocks","namespace":"kube-starrocks"}, "namespace": "kube-starrocks", "name": "starrocks", "reconcileID": "572f57e6-c84a-490b-a9fb-e11786f8711b", "error": "Service \"starrocks-fe-search\" is invalid: metadata.resourceVersion: Invalid value: \"\": must be specified for an update"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:326 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234

The environment: Tanzu WmWare - Kubernetes

Could you please help me with this?

Thanks in advance.

The text was updated successfully, but these errors were encountered:

kevincai · 2023-08-04T03:50:31Z

can you provide following info?

operator version
use crd directly or use helm chart
steps to repro

jimenez-f · 2023-08-04T14:24:50Z

Sure!

Operator version: appVersion: 3.0-latest | https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/helm-charts/charts/kube-starrocks/charts/operator/Chart.yaml
Firt step I applied via Helm the Operator Chart and then with kubectl apply -f this CRD yaml https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/deploy/starrocks.com_starrocksclusters.yaml
Finally step I applied via Helm the Starrocks Chart without CN | https://github.com/StarRocks/starrocks-kubernetes-operator/tree/main/helm-charts/charts/kube-starrocks/charts

With theses steps I got the following:

NAME READY STATUS RESTARTS AGE
kube-starrocks-operator-f848d5c6b-jk4p6 1/1 Running 0 15h
starrocks-be-0 1/1 Running 2 14h
starrocks-fe-0 1/1 Running 4 15h

and via ephemeral pod I got the backends:

root@ubuntu:/# mysql -h starrocks-fe-service.kube-starrocks.svc.cluster.local -P 9030 -uroot
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.1.0

mysql> show backends;
ERROR 2013 (HY000): Lost connection to MySQL server during query
No connection. Trying to reconnect...
Connection id: 0
Current database: *** NONE ***

+-----------+---------------------------------------------------------------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+------------------+--------------------------------------------------------+-------------------+-------------+----------+-------------------+------------+------------+ | BackendId | IP | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime | LastHeartbeat | Alive | SystemDecommissioned | ClusterDecommissioned | TabletNum | DataUsedCapacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | ErrMsg | Version | Status | DataTotalCapacity | DataUsedPct | CpuCores | NumRunningQueries | MemUsedPct | CpuUsedPct | +-----------+---------------------------------------------------------------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+------------------+--------------------------------------------------------+-------------------+-------------+----------+-------------------+------------+------------+ | 10007 | starrocks-be-0.starrocks-be-search.kube-starrocks.svc.cluster.local | 9050 | 9060 | 8040 | 8060 | 2023-08-03 23:38:44 | 2023-08-04 03:08:10 | true | false | false | 30 | 0.000 | 10.724 GB | 15.609 GB | 31.29 % | 31.29 % | | 3.0.4-f8ff06dcbc | {"lastSuccessReportTabletsTime":"2023-08-04 03:08:00"} | 10.724 GB | 0.00 % | 2 | 0 | 0.07 % | 0.2 % | +-----------+---------------------------------------------------------------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+------------------+--------------------------------------------------------+-------------------+-------------+----------+-------------------+------------+------------+ 1 row in set (1.35 sec)

By the other hand, there is a way to deploy Starrocks Cluster (FE & BE) without Starrocks Operator and CRD in Kubernetes?

Thanks in advance!

kevincai · 2023-08-04T14:32:13Z

if you are using helm charts, please refer the doc shown at https://artifacthub.io/packages/helm/kube-starrocks/kube-starrocks

helm repo add starrocks-community https://starrocks.github.io/starrocks-kubernetes-operator
helm repo update
helm install <releasename> starrocks-community/kube-starrocks

This will give you a very basic cluster with no pvc for data persistency to play with.

By the other hand, there is a way to deploy Starrocks Cluster (FE & BE) without Starrocks Operator and CRD in Kubernetes?

You can try to do it your own with deployment or statefulset, but it is not recommended. Some one has a demo here if you are interested: StarRocks/demo#26

jimenez-f · 2023-08-04T14:43:00Z

Thanks a lot for your quick answer. I will try with this way.

Quick question, If I want to set a persistent volume I could set with the command install wright?

kevincai · 2023-08-04T14:52:01Z

Quick question, If I want to set a persistent volume I could set with the command install wright?

either chart values or crd configuration, refer to https://github.com/StarRocks/starrocks-kubernetes-operator#optional-configuring-storage-volume, the "optional-configuring-storage-volume" section.

jimenez-f · 2023-08-04T14:56:28Z

Thanks! I will try with this way and comment by this loop if that's works

jimenez-f · 2023-08-04T15:36:10Z

I tried with the steps at https://artifacthub.io/packages/helm/kube-starrocks/kube-starrocks but unfortunately I got the same issue with Service \"kube-starrocks-fe-search\" is invalid . This time I just applied the Helm without custom settings.

helm repo add starrocks-community https://starrocks.github.io/starrocks-kubernetes-operator
helm repo update
helm install kube-starrocks starrocks-community/kube-starrocks
NAME: kube-starrocks
LAST DEPLOYED: Fri Aug 4 12:14:13 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:

kube-starrocks has been installed, Check its status by running:
kubectl --namespace default get starrockscluster -l "cluster=kube-starrocks"

NAME FESTATUS CNSTATUS BESTATUS
kube-starrocks

I don't understand where is the problem.

Logs here:

I0804 15:10:30.353993 1 starrockscluster_controller.go:84] StarRocksClusterReconciler reconcile the update crd name kube-starrocks namespace kube-starrocks
I0804 15:10:30.355440 1 be_controller.go:219] BeController Sync wait fe service name kube-starrocks-fe-service available occur failed Endpoints "kube-starrocks-fe-service" not found
I0804 15:10:30.357597 1 statefulset.go:89] the statefulset name kube-starrocks-fe new hash value 1482528638 old have value 3244496261
I0804 15:10:30.357654 1 k8sutils.go:94] Updating resource service namespace kube-starrocks name kube-starrocks-fe kind &TypeMeta{Kind:,APIVersion:,}
E0804 15:10:30.391366 1 starrockscluster_controller.go:132] StarRocksClusterReconciler reconcile sub resource reconcile failed, namespace=kube-starrocks, name=kube-starrocks, controller=feController, error=StatefulSet.apps "kube-starrocks-fe" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden
1.6911618303914502e+09 ERROR Reconciler error {"controller": "starrockscluster", "controllerGroup": "starrocks.com", "controllerKind": "StarRocksCluster", "StarRocksCluster": {"name":"kube-starrocks","namespace":"kube-starrocks"}, "namespace": "kube-starrocks", "name": "kube-starrocks", "reconcileID": "0148ec81-e70a-497d-a9e8-7c660eed35cd", "error": "StatefulSet.apps "kube-starrocks-fe" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:326
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234

kevincai · 2023-08-04T15:53:57Z

probably, you need to uninstall all the resources including the crd, to get a clean env with no crd or operator installed, and then try again. @yandongxiao

jimenez-f · 2023-08-04T16:02:54Z

I have done just that, removed the first deployment with all its resources and including the CRD.

What's more, this last deploy went directly to the Namespace Default.

1.6911649039678488e+09 ERROR Reconciler error {"controller": "starrockscluster", "controllerGroup": "starrocks.com", "controllerKind": "StarRocksCluster", "StarRocksCluster": {"name":"kube-starrocks","namespace":"default"}, "namespace": "default", "name": "kube-starrocks", "reconcileID": "c729de63-32d1-4237-83cf-83c1173c34a2", "error": "Service "kube-starrocks-fe-search" is invalid: metadata.resourceVersion: Invalid value: "": must be specified for an update"}

yandongxiao · 2023-08-04T16:18:47Z

I have done just that, removed the first deployment with all its resources and including the CRD.

What's more, this last deploy went directly to the Namespace Default.

1.6911649039678488e+09 ERROR Reconciler error {"controller": "starrockscluster", "controllerGroup": "starrocks.com", "controllerKind": "StarRocksCluster", "StarRocksCluster": {"name":"kube-starrocks","namespace":"default"}, "namespace": "default", "name": "kube-starrocks", "reconcileID": "c729de63-32d1-4237-83cf-83c1173c34a2", "error": "Service "kube-starrocks-fe-search" is invalid: metadata.resourceVersion: Invalid value: "": must be specified for an update"}

In order to install starrocks in a specified namespace, like sr, you should execute kubectl create ns sr; helm install -n sr starrocks starrocks-community/kube-starrocks.

yandongxiao · 2023-08-04T16:24:24Z

I tried with the steps at https://artifacthub.io/packages/helm/kube-starrocks/kube-starrocks but unfortunately I got the same issue with Service \"kube-starrocks-fe-search\" is invalid . This time I just applied the Helm without custom settings.

helm repo add starrocks-community https://starrocks.github.io/starrocks-kubernetes-operator

helm repo update

helm install kube-starrocks starrocks-community/kube-starrocks
NAME: kube-starrocks
LAST DEPLOYED: Fri Aug 4 12:14:13 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
kube-starrocks has been installed, Check its status by running:
kubectl --namespace default get starrockscluster -l "cluster=kube-starrocks"
NAME FESTATUS CNSTATUS BESTATUS
kube-starrocks

I don't understand where is the problem.

Logs here:

I0804 15:10:30.353993 1 starrockscluster_controller.go:84] StarRocksClusterReconciler reconcile the update crd name kube-starrocks namespace kube-starrocks
I0804 15:10:30.355440 1 be_controller.go:219] BeController Sync wait fe service name kube-starrocks-fe-service available occur failed Endpoints "kube-starrocks-fe-service" not found
I0804 15:10:30.357597 1 statefulset.go:89] the statefulset name kube-starrocks-fe new hash value 1482528638 old have value 3244496261
I0804 15:10:30.357654 1 k8sutils.go:94] Updating resource service namespace kube-starrocks name kube-starrocks-fe kind &TypeMeta{Kind:,APIVersion:,}
E0804 15:10:30.391366 1 starrockscluster_controller.go:132] StarRocksClusterReconciler reconcile sub resource reconcile failed, namespace=kube-starrocks, name=kube-starrocks, controller=feController, error=StatefulSet.apps "kube-starrocks-fe" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden
1.6911618303914502e+09 ERROR Reconciler error {"controller": "starrockscluster", "controllerGroup": "starrocks.com", "controllerKind": "StarRocksCluster", "StarRocksCluster": {"name":"kube-starrocks","namespace":"kube-starrocks"}, "namespace": "kube-starrocks", "name": "kube-starrocks", "reconcileID": "0148ec81-e70a-497d-a9e8-7c660eed35cd", "error": "StatefulSet.apps "kube-starrocks-fe" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:326
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234

The following log means the fileds of FE statefulset is updated, which is not allowed.

ERROR Reconciler error {"controller": "starrockscluster", "controllerGroup": "starrocks.com", "controllerKind": "StarRocksCluster", "StarRocksCluster": {"name":"kube-starrocks","namespace":"kube-starrocks"}, "namespace": "kube-starrocks", "name": "kube-starrocks", "reconcileID": "0148ec81-e70a-497d-a9e8-7c660eed35cd", "error": "StatefulSet.apps "kube-starrocks-fe" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden"}

I think you forgot to execute helm uninstall starrocks, before you reinstall.

jimenez-f · 2023-08-04T16:58:01Z

Always execute this command helm uninstall starrocks to then launch a new deploy

The first time I created a namespace and then I unistalled the Helm realese of this namespace. The second time I tried with the steps at https://artifacthub.io/packages/helm/kube-starrocks/kube-starrocks in the default namespace.

I will try again with you recomendation:

[In order to install starrocks in a specified namespace, like sr, you should execute kubectl create ns sr; helm install -n sr starrocks starrocks-community/kube-starrocks.]

jimenez-f · 2023-08-04T17:56:57Z

I followed your recommendation: I've removed all deployments and launch again. I add my steps:

➜ ~ k create namespace sr

namespace/sr created

➜ ~ helm install -n sr starrocks starrocks-community/kube-starrocks

NAME:` starrocks
LAST DEPLOYED: Fri Aug 4 14:39:37 2023
NAMESPACE: sr
STATUS: deployed
REVISION: 1
NOTES:

Get the application URL by running these commands:

kube-starrocks has been installed, Check its status by running:
kubectl --namespace sr get starrockscluster -l "cluster=kube-starrocks"

Visit https://github.com/starrocks for instructions on how to create & configure.

➜ ~ kubectl --namespace sr get starrockscluster -l "cluster=kube-starrocks"

NAME FESTATUS CNSTATUS BESTATUS
kube-starrocks reconciling reconciling

➜ ~ helm list -A

NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
starrocks sr 1 2023-08-04 14:39:37.1525889 -0300 -03 deployed kube-starrocks-1.6.1 2.5.4

Every 2.0s: kubectl get pods -n sr Fri Aug 4 14:53:48 2023

NAME READY STATUS RESTARTS AGE
kube-starrocks-be-0 1/1 Running 3 10m
kube-starrocks-fe-0 1/1 Running 1 13m
kube-starrocks-operator-7fbf59f984-hzvl8 1/1 Running 0 14m

I got the same issue:

E0804 17:51:23.344014 1 starrockscluster_controller.go:132] StarRocksClusterReconciler reconcile sub resource reconcile failed, namespace=sr, name=kube-starrocks, controller=feController, error=Service "kube-starrocks-fe-search" is invalid: metadata.resourceVersion: Invalid value: "": must be specified for an update
1.6911714833441758e+09 ERROR Reconciler error {"controller": "starrockscluster", "controllerGroup": "starrocks.com", "controllerKind": "StarRocksCluster", "StarRocksCluster": {"name":"kube-starrocks","namespace":"sr"}, "namespace": "sr", "name": "kube-starrocks", "reconcileID": "6e9d1333-a328-4545-8781-ec4d1115d5bd", "error": "Service "kube-starrocks-fe-search" is invalid: metadata.resourceVersion: Invalid value: "": must be specified for an update"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:326
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234

kevincai · 2023-08-04T22:03:57Z

which k8s version are you using?

jimenez-f · 2023-08-04T23:46:24Z

Server Version: v1.21.6+vmware.1

yandongxiao · 2023-08-05T00:55:21Z

NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
starrocks sr 1 2023-08-04 14:39:37.1525889 -0300 -03 deployed kube-starrocks-1.6.1 2.5.4

Can you use the latest chart version v1.7.1 to try it?

jimenez-f · 2023-08-06T03:00:15Z

I deleted all then re-deployed and I got the same issue. I will try in minikube the next Monday.

~ helm list -A

NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
kube-starrocks sr 1 2023-08-05 23:39:45.3199244 -0300 -03 deployed kube-starrocks-1.7.1 3.0-latest

yandongxiao · 2023-08-08T13:33:49Z

I reproduce the error invalid: metadata.resourceVersion in k8s v1.21.10, and I will fix it.

jimenez-f · 2023-08-08T14:34:08Z

Many thanks! Could you please let me know about the change to pull it the new version?

yandongxiao · 2023-08-08T14:48:25Z

Ok

kevincai · 2023-08-08T16:33:59Z

I reproduce the error invalid: metadata.resourceVersion in k8s v1.21.10, and I will fix it.

is this a k8s version specific issue?

yandongxiao · 2023-08-09T02:12:19Z

I reproduce the error invalid: metadata.resourceVersion in k8s v1.21.10, and I will fix it.

is this a k8s version specific issue?

I think so, in the v1.27.3, there is no such error in operator log

yandongxiao · 2023-08-09T12:53:02Z

Many thanks! Could you please let me know about the change to pull it the new version?

StarRocks/starrocks-kubernetes-operator#214

This PR will included in starrocks-kubernetes-operator v1.8.0 which will be published in next week.
please give an eye on https://github.com/StarRocks/starrocks-kubernetes-operator

jimenez-f · 2023-08-09T14:33:38Z

Many thanks! Could you please let me know about the change to pull it the new version?

StarRocks/starrocks-kubernetes-operator#214

This PR will included in starrocks-kubernetes-operator v1.8.0 which will be published in next week. please give an eye on https://github.com/StarRocks/starrocks-kubernetes-operator

Thanks a lot!

aikirias · 2023-08-15T14:04:31Z

Hey guys there is an update on this? I'm facing a similar issue

jimenez-f · 2023-08-18T14:28:43Z

Version 1.8.0 works well!

jimenez-f added the type/question Further information is requested label Aug 4, 2023

jimenez-f closed this as completed Aug 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reconciler error in Starrocks Operator (invalid: metadata.resourceVersion) | Latest Helm Chart #28626

Reconciler error in Starrocks Operator (invalid: metadata.resourceVersion) | Latest Helm Chart #28626

jimenez-f commented Aug 4, 2023

kevincai commented Aug 4, 2023

jimenez-f commented Aug 4, 2023 •

edited

Loading

kevincai commented Aug 4, 2023

jimenez-f commented Aug 4, 2023

kevincai commented Aug 4, 2023

jimenez-f commented Aug 4, 2023

jimenez-f commented Aug 4, 2023 •

edited

Loading

kevincai commented Aug 4, 2023

jimenez-f commented Aug 4, 2023 •

edited

Loading

yandongxiao commented Aug 4, 2023 •

edited

Loading

yandongxiao commented Aug 4, 2023 •

edited

Loading

jimenez-f commented Aug 4, 2023

jimenez-f commented Aug 4, 2023 •

edited

Loading

kevincai commented Aug 4, 2023

jimenez-f commented Aug 4, 2023

yandongxiao commented Aug 5, 2023

jimenez-f commented Aug 6, 2023

yandongxiao commented Aug 8, 2023 •

edited

Loading

jimenez-f commented Aug 8, 2023

yandongxiao commented Aug 8, 2023

kevincai commented Aug 8, 2023

yandongxiao commented Aug 9, 2023

yandongxiao commented Aug 9, 2023

jimenez-f commented Aug 9, 2023

aikirias commented Aug 15, 2023

jimenez-f commented Aug 18, 2023

Reconciler error in Starrocks Operator (invalid: metadata.resourceVersion) | Latest Helm Chart #28626

Reconciler error in Starrocks Operator (invalid: metadata.resourceVersion) | Latest Helm Chart #28626

Comments

jimenez-f commented Aug 4, 2023

General Question

kevincai commented Aug 4, 2023

jimenez-f commented Aug 4, 2023 • edited Loading

kevincai commented Aug 4, 2023

jimenez-f commented Aug 4, 2023

kevincai commented Aug 4, 2023

jimenez-f commented Aug 4, 2023

jimenez-f commented Aug 4, 2023 • edited Loading

kevincai commented Aug 4, 2023

jimenez-f commented Aug 4, 2023 • edited Loading

yandongxiao commented Aug 4, 2023 • edited Loading

yandongxiao commented Aug 4, 2023 • edited Loading

jimenez-f commented Aug 4, 2023

jimenez-f commented Aug 4, 2023 • edited Loading

kevincai commented Aug 4, 2023

jimenez-f commented Aug 4, 2023

yandongxiao commented Aug 5, 2023

jimenez-f commented Aug 6, 2023

yandongxiao commented Aug 8, 2023 • edited Loading

jimenez-f commented Aug 8, 2023

yandongxiao commented Aug 8, 2023

kevincai commented Aug 8, 2023

yandongxiao commented Aug 9, 2023

yandongxiao commented Aug 9, 2023

jimenez-f commented Aug 9, 2023

aikirias commented Aug 15, 2023

jimenez-f commented Aug 18, 2023

jimenez-f commented Aug 4, 2023 •

edited

Loading

jimenez-f commented Aug 4, 2023 •

edited

Loading

jimenez-f commented Aug 4, 2023 •

edited

Loading

yandongxiao commented Aug 4, 2023 •

edited

Loading

yandongxiao commented Aug 4, 2023 •

edited

Loading

jimenez-f commented Aug 4, 2023 •

edited

Loading

yandongxiao commented Aug 8, 2023 •

edited

Loading