[WIP] framework/db: introduce a new MySQL table based distributed lock #8968

rohityadavcloud · 2024-04-24T12:04:16Z

This introduces a MySQL innodb table based distributed lock which can be used by one or more management server and its threads. This removes usage of MySQL server provided locking functions (GET_LOCK, RELEASE_LOCK) which are not replicated or not supported currently by any MySQL clustering solutions. This would be the first main step in having CloudStack to work with a MySQL clustering solution such as InnoDB cluster, Percona Xtradb cluster, MariaDB galera cluster. There may be other changes required which can be found in due course if this feature works at scale.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

How Has This Been Tested?

In a local setup against simulator & mvn, in a single management server no exception were seen. Most tests need to be done such as:

simulator based smoketests which is one mvn-run of mgmt server + one mysql db server
smoketests (test matrix) which tests largely one mgmt server + one mysql db server
multiple (say 3) mgmt servers + one mysql db server
multiple (say 3) mgmt servers + mysql db cluster with say 3 nodes

rohityadavcloud · 2024-04-24T12:05:05Z

This is largely WIP and experimental, at this point I'm only looking for automated tests which are easier done via a PR, I'm not looking for any feedback and review - thanks.

@blueorangutan package

rohityadavcloud · 2024-04-24T17:30:54Z

@blueorangutan package

blueorangutan · 2024-04-24T17:32:03Z

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-04-24T18:45:00Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9411

rohityadavcloud · 2024-04-24T18:48:22Z

@blueorangutan test matrix

blueorangutan · 2024-04-24T18:50:03Z

@rohityadavcloud a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2024-04-24T19:13:43Z

[SF] Trillian Build Failed (tid-9995)

blueorangutan · 2024-04-24T19:15:00Z

[SF] Trillian Build Failed (tid-9991)

blueorangutan · 2024-04-24T19:25:12Z

[SF] Trillian Build Failed (tid-9993)

rohityadavcloud · 2024-04-24T20:39:13Z

@blueorangutan package

blueorangutan · 2024-04-24T20:40:03Z

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-04-24T21:56:30Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9412

rohityadavcloud · 2024-04-25T03:46:43Z

@blueorangutan test matrix

blueorangutan · 2024-04-25T03:48:03Z

@rohityadavcloud a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2024-04-25T04:08:43Z

[SF] Trillian Build Failed (tid-10003)

blueorangutan · 2024-04-25T04:08:48Z

[SF] Trillian Build Failed (tid-10001)

codecov-commenter · 2024-04-25T04:59:30Z

Codecov Report

Attention: Patch coverage is 40.84507% with 42 lines in your changes are missing coverage. Please review.

Project coverage is 14.96%. Comparing base (e409c6d) to head (9830bbe).

Files	Patch %	Lines
...java/com/cloud/upgrade/DatabaseUpgradeChecker.java	0.00%	22 Missing ⚠️
...rk/db/src/main/java/com/cloud/utils/db/DbUtil.java	62.96%	7 Missing and 3 partials ⚠️
.../src/main/java/com/cloud/utils/net/MacAddress.java	54.54%	6 Missing and 4 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.19    #8968      +/-   ##
============================================
- Coverage     14.96%   14.96%   -0.01%     
+ Complexity    10995    10989       -6     
============================================
  Files          5373     5373              
  Lines        468989   468920      -69     
  Branches      61009    57594    -3415     
============================================
- Hits          70191    70164      -27     
+ Misses       391019   390993      -26     
+ Partials       7779     7763      -16

Flag	Coverage Δ
uitests	`4.31% <ø> (ø)`
unittests	`15.66% <40.84%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

rohityadavcloud · 2024-04-25T05:17:42Z

@blueorangutan test matrix

rohityadavcloud · 2024-04-25T11:48:35Z

@blueorangutan test alma8 vmware-70u3

blueorangutan · 2024-04-25T11:50:03Z

@rohityadavcloud a [SL] Trillian-Jenkins test job (alma8 mgmt + vmware-70u3) has been kicked to run smoke tests

rohityadavcloud · 2024-04-25T12:24:43Z

@blueorangutan test alma8 kvm-alma8

blueorangutan · 2024-04-25T12:26:03Z

@rohityadavcloud a [SL] Trillian-Jenkins test job (alma8 mgmt + kvm-alma8) has been kicked to run smoke tests

blueorangutan · 2024-04-25T13:37:34Z

[SF] Trillian Build Failed (tid-10028)

blueorangutan · 2024-04-25T21:20:35Z

[SF] Trillian test result (tid-10005)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 55775 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8968-t10005-vmware-67u3.zip
Smoke tests completed. 129 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_events_resource	`Error`	346.17	test_events_resource.py

blueorangutan · 2024-04-26T06:22:46Z

[SF] Trillian test result (tid-10025)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server a8
Total time taken: 61906 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8968-t10025-vmware-70u3.zip
Smoke tests completed. 127 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_events_resource	`Error`	345.93	test_events_resource.py
test_01_list_sec_storage_vm	`Failure`	0.21	test_ssvm.py
test_03_ssvm_internals	`Failure`	3.34	test_ssvm.py
test_05_stop_ssvm	`Failure`	934.96	test_ssvm.py
test_06_stop_cpvm	`Error`	988.39	test_ssvm.py
test_08_reboot_cpvm	`Error`	914.21	test_ssvm.py
test_01_redundant_vpc_site2site_vpn	`Failure`	522.63	test_vpc_vpn.py

Cleanup old mac address handling code to use JDK11 lib instead of hacks. Also really strange to see some basic string parsing code was written by hand, replaced with Long.parseValue(str, 16) to convert hex string to long. Signed-off-by: Rohit Yadav <[email protected]>

This introduces a MySQL innodb table based distributed lock which can be used by one or more management server and its threads. This removes usage of MySQL server provided locking functions (GET_LOCK, RELEASE_LOCK) which are not replicated or supported currently by any MySQL clustering solutions. This would be the first main step in having CloudStack to work with a MySQL clustering solution such as InnoDB cluster, Percona Xtradb cluster, MariaDB galera cluster. There may be other changes required which can be found in due course if this feature works at scale. Signed-off-by: Rohit Yadav <[email protected]>

rohityadavcloud · 2024-04-26T11:05:44Z

@blueorangutan package

blueorangutan · 2024-04-26T11:08:04Z

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-04-26T12:30:48Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9446

rohityadavcloud · 2024-04-26T18:37:50Z

@blueorangutan test matrix

blueorangutan · 2024-04-26T18:40:03Z

@rohityadavcloud a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2024-04-26T19:00:40Z

[SF] Trillian Build Failed (tid-10058)

blueorangutan · 2024-04-26T19:21:49Z

[SF] Trillian Build Failed (tid-10060)

rohityadavcloud · 2024-04-27T05:04:33Z

@blueorangutan test matrix

blueorangutan · 2024-04-27T05:06:03Z

@rohityadavcloud a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2024-04-27T05:26:00Z

[SF] Trillian Build Failed (tid-10061)

blueorangutan · 2024-04-27T05:26:08Z

[SF] Trillian Build Failed (tid-10063)

blueorangutan · 2024-04-27T10:27:57Z

[SF] Trillian test result (tid-10059)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 55077 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8968-t10059-vmware-67u3.zip
Smoke tests completed. 127 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_events_resource	`Error`	358.17	test_events_resource.py
test_02_balanced_drs_algorithm	`Failure`	144.54	test_cluster_drs.py
test_03_deploy_and_scale_kubernetes_cluster	`Error`	289.72	test_kubernetes_clusters.py

blueorangutan · 2024-04-27T19:38:01Z

[SF] Trillian test result (tid-10062)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 50700 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8968-t10062-vmware-67u3.zip
Smoke tests completed. 127 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_events_resource	`Error`	336.38	test_events_resource.py
test_02_balanced_drs_algorithm	`Error`	427.88	test_cluster_drs.py
test_02_trigger_shutdown	`Failure`	341.69	test_safe_shutdown.py

github-actions · 2024-04-29T08:30:11Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

rohityadavcloud · 2024-05-22T14:49:22Z

Update and note - on further R&D and investigation, MySQL server turned out to not really be the bottleneck, thanks to the creators of jprofiler, yourkit, MAT and visual vm, I've more insights on what's lacking in CloudStack. While such a PR like this may still be useful for users who want an active-active MySQL setup, the real challenge is fixing the schema and application in the first place, for which I'll submit a separate PR (or PRs). For now this is on-hold, to be treated as an experiment.

Closing this for now.

boring-cyborg bot added the component:database label Apr 24, 2024

rohityadavcloud added status:work-in-progress status:needs-investigation labels Apr 24, 2024

rohityadavcloud closed this Apr 24, 2024

rohityadavcloud reopened this Apr 24, 2024

apache deleted a comment from blueorangutan Apr 24, 2024

rohityadavcloud closed this Apr 25, 2024

rohityadavcloud reopened this Apr 25, 2024

rohityadavcloud mentioned this pull request Apr 25, 2024

codecov isn't able to upload when PR is from forks #8972

Closed

rohityadavcloud added 2 commits April 26, 2024 15:23

rohityadavcloud force-pushed the mysql-distributed-lock branch from fb82e35 to 9830bbe Compare April 26, 2024 11:00

rohityadavcloud added the status:On Hold label Apr 26, 2024

github-actions bot added the status:has-conflicts label Apr 29, 2024

rohityadavcloud changed the title ~~framework/db: introduce a new MySQL table based distributed lock~~ [WIP] framework/db: introduce a new MySQL table based distributed lock May 3, 2024

rohityadavcloud added this to the unplanned milestone May 22, 2024

rohityadavcloud closed this Jun 13, 2024

[WIP] framework/db: introduce a new MySQL table based distributed lock #8968

[WIP] framework/db: introduce a new MySQL table based distributed lock #8968

Conversation

rohityadavcloud commented Apr 24, 2024 • edited Loading

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

How Has This Been Tested?

rohityadavcloud commented Apr 24, 2024 • edited Loading

rohityadavcloud commented Apr 24, 2024

blueorangutan commented Apr 24, 2024

blueorangutan commented Apr 24, 2024

rohityadavcloud commented Apr 24, 2024

blueorangutan commented Apr 24, 2024

blueorangutan commented Apr 24, 2024

blueorangutan commented Apr 24, 2024

blueorangutan commented Apr 24, 2024

rohityadavcloud commented Apr 24, 2024

blueorangutan commented Apr 24, 2024

blueorangutan commented Apr 24, 2024

rohityadavcloud commented Apr 25, 2024

blueorangutan commented Apr 25, 2024

blueorangutan commented Apr 25, 2024

blueorangutan commented Apr 25, 2024

codecov-commenter commented Apr 25, 2024 • edited Loading

Codecov Report

rohityadavcloud commented Apr 25, 2024

rohityadavcloud commented Apr 25, 2024

blueorangutan commented Apr 25, 2024

rohityadavcloud commented Apr 25, 2024

blueorangutan commented Apr 25, 2024

blueorangutan commented Apr 25, 2024

blueorangutan commented Apr 25, 2024

blueorangutan commented Apr 26, 2024

rohityadavcloud commented Apr 26, 2024

blueorangutan commented Apr 26, 2024

blueorangutan commented Apr 26, 2024

rohityadavcloud commented Apr 26, 2024

blueorangutan commented Apr 26, 2024

blueorangutan commented Apr 26, 2024

blueorangutan commented Apr 26, 2024

rohityadavcloud commented Apr 27, 2024

blueorangutan commented Apr 27, 2024

blueorangutan commented Apr 27, 2024

blueorangutan commented Apr 27, 2024

blueorangutan commented Apr 27, 2024

blueorangutan commented Apr 27, 2024

github-actions bot commented Apr 29, 2024

rohityadavcloud commented May 22, 2024 • edited Loading

rohityadavcloud commented Apr 24, 2024 •

edited

Loading

rohityadavcloud commented Apr 24, 2024 •

edited

Loading

codecov-commenter commented Apr 25, 2024 •

edited

Loading

rohityadavcloud commented May 22, 2024 •

edited

Loading