Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement support for metadata associated with read-write transactions #1914

Merged
merged 1 commit into from
Sep 30, 2024

Conversation

josephschorr
Copy link
Member

@josephschorr josephschorr commented May 26, 2024

This will allow callers of APIs such as WriteRelationships and DeleteRelationships to assign metadata to the transaction that will be mirrored back out in the Watch API, to provide a means for correlating updates

First part of #966

Each datastore has a different means of correlating between the transaction and the associated change(s) coming out of the watch API:

  • Memdb: The metadata is stored on the changes struct
  • Postgres: A metadata column was added to the transactions table
  • MySQL: A metadata column was added to the transactions table
  • Spanner: A transaction tag is used to correlate the transaction and the metadata is stored in an auto-GC-ing table, indexed by the transaction tag
  • CRDB: A row is inserted into a metadata table as part of the transaction, which includes the metadata. When seen at the same revision as the other changes, the metadata becomes correlated with that transaction. The rows are set to auto-GC after a period of time in CRDB >= v22

@github-actions github-actions bot added area/datastore Affects the storage system area/tooling Affects the dev or user toolchain (e.g. tests, ci, build tools) labels May 26, 2024
Copy link

github-actions bot commented May 26, 2024

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

@josephschorr
Copy link
Member Author

I have read the CLA Document and I hereby sign the CLA

@josephschorr
Copy link
Member Author

recheck

Copy link
Contributor

@vroldanbet vroldanbet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few questions and suggestions, I think the approach makes sense to me. The main concerns are how does this affect write-throughput, but at first glance it shouldn't introduce any new hotspots.

A potential issue here is the overlap with the concept of tuple-level metadata, but we've certainly heard more about this use case than tuple-level metadata. I also do wonder if folks will eventually ask for a transactions API to read these back, but that's an easy thing to do if the time comes.

Finally: I understand these are just the Datastore API changes, but we'd have to make sure there are limits put in place so that folks don't abuse that new field.

internal/datastore/crdb/crdb.go Outdated Show resolved Hide resolved
internal/datastore/crdb/crdb.go Outdated Show resolved Hide resolved
internal/datastore/mysql/query_builder.go Show resolved Hide resolved
internal/datastore/postgres/revisions.go Outdated Show resolved Hide resolved
internal/datastore/spanner/spanner.go Outdated Show resolved Hide resolved
internal/datastore/spanner/spanner.go Show resolved Hide resolved
internal/datastore/common/changes.go Outdated Show resolved Hide resolved
@josephschorr
Copy link
Member Author

Updated

@josephschorr
Copy link
Member Author

Rebased

@josephschorr josephschorr force-pushed the rw-tx-metadata branch 3 times, most recently from d5e92cc to 4233a1d Compare August 26, 2024 17:40
@josephschorr josephschorr marked this pull request as ready for review August 26, 2024 17:56
@josephschorr josephschorr requested a review from a team as a code owner August 26, 2024 17:56
tstirrat15
tstirrat15 previously approved these changes Aug 26, 2024
Copy link
Contributor

@tstirrat15 tstirrat15 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, see comment

internal/datastore/postgres/revisions.go Show resolved Hide resolved
This will allow callers of APIs such as WriteRelationships and DeleteRelationships to assign
metadata to the transaction that will be mirrored back out in the Watch API, to provide a means
for correlating updates
@josephschorr josephschorr added this pull request to the merge queue Sep 30, 2024
Merged via the queue into authzed:main with commit bfd80f7 Sep 30, 2024
21 of 22 checks passed
@josephschorr josephschorr deleted the rw-tx-metadata branch September 30, 2024 21:35
@github-actions github-actions bot locked and limited conversation to collaborators Sep 30, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area/datastore Affects the storage system area/dependencies Affects dependencies area/tooling Affects the dev or user toolchain (e.g. tests, ci, build tools)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants