Migrate plugins #2378

jieguangzhou · 2024-08-08T13:37:25Z

Description

Related Issues

Checklist

Is this code covered by new or existing unit tests or integration tests?
Did you run make unit_testing and make integration-testing successfully?
Do new classes, functions, methods and parameters all have docstrings?
Were existing docstrings updated, if necessary?
Was external documentation updated, if necessary?

Additional Notes or Comments

test/integration/backends/test_metdata.py

test/unittest/backends/mongodb/test_metadata.py

jieguangzhou · 2024-08-12T09:22:24Z

.github/workflows/ci_plugins.yaml

+      - name: Optionally run the base testing
+        run: |
+          SUPERDUPER_CONFIG="plugins/${{ matrix.plugin }}/plugin_test/config.yaml"
+          if [ -f "$SUPERDUPER_CONFIG" ]; then
+            echo "Running the base testing..."
+            make unit_testing SUPERDUPER_CONFIG=$SUPERDUPER_CONFIG
+            make usecase_testing SUPERDUPER_CONFIG=$SUPERDUPER_CONFIG
+          else
+            echo "No config file found, skipping..."
+          fi


The plugin located under the plugin_test folder, which includes a config.yaml file, not only runs its own tests but also uses this configuration file to run test cases for superduper, including both unittest and usecase tests.

jieguangzhou · 2024-08-12T09:38:17Z

superduper/components/schema.py


+class FieldType(Leaf):


A unified data type identifier used to record native datatypes, replacing the original dtype in Ibis

jieguangzhou · 2024-08-12T13:14:08Z

.github/workflows/ci_plugins.yaml

+      - name: Optionally run custom CI script
+        run: |
+          if [ -f "plugins/${{ matrix.plugin }}/.ci_extend.sh" ]; then
+            echo "Running custom CI script..."
+            bash ./plugins/${{ matrix.plugin }}/.ci_extend.sh
+          else
+            echo "No custom CI script found, skipping..."
+          fi


Use the officially released package to prepare the dependency plugins.
Do this in the next release pr

Why not just extract the dependent plugins from the pyproject.toml scripts?

[project.optional-dependencies] test = [ "vcrpy>=5.1.0", # Annotation plugin dependencies will be installed in CI # :CI: plugins/mongodb ]

Added this for installing local dependencies instead of those on PyPI, which is useful when modifying multiple interdependent plugins at the same time. Using the names directly from PyPI will lead to dependencies on outdated versions for running CI.

blythed · 2024-08-12T17:29:13Z

.github/workflows/ci_plugins.yaml

+
+      - name: Install DevKit (docs, testing, etc)
+        run: |
+          make install_devkit


Maybe we should combine these 2 things into .[test]?

blythed

This is great, I especially like how quickly the CI is running now.

What shall we do with "core" plugins/ extensions, for example ext/numpy? Shouldn't we move this somehow? Also, what is the plan for moving away from the superduper.ext.* sub-packages? Shall we raise a warning which can become an error later?

jieguangzhou · 2024-08-13T08:39:57Z

What shall we do with "core" plugins/ extensions, for example ext/numpy? Shouldn't we move this somehow? Also, what is the plan for moving away from the superduper.ext.* sub-packages? Shall we raise a warning which can become an error later?

We need to retain some core plugins, which are pre-set, and can continue to use this ext until a better name is found.

I added the warning, and I think we can directly remove those plugins in the next one or two versions, at which point an import error will be thrown.

kartik4949 · 2024-08-13T09:22:56Z

superduper/backends/base/backends.py

-    'pymongo': MongoClient,
-    'ibis': BaseBackend,
+    "lance": LanceVectorSearcher,
+    "in_memory": InMemoryVectorSearcher,


have we thought about making vector index implementations as plugin?

Yes, we should do this later. The VectorSearcher backend should be the plugin, and keep the InMemoryVectorSearcher as pre-set

kartik4949 · 2024-08-13T09:26:45Z

test/unittest/base/test_document.py

@@ -169,57 +137,46 @@ def test_refer_to_applied_item(db):


 def test_column_encoding(db):
-    import PIL
-
-    img = PIL.Image.open('test/material/data/test.png')


was this intended?

Yes, we have removed the Pillow dependency and replaced it with another non-native datatype.

kartik4949 · 2024-08-13T09:27:28Z

test/utils/database/query.py


    data = {
-        "img": PIL.Image.open("test/material/data/test.png"),
+        "df": pd.DataFrame(np.random.randn(10, 10)),


why not keep both?

We have removed the Pillow dependency and replaced it with another non-native datatype.

kartik4949 · 2024-08-13T09:28:37Z

test/utils/database/query.py

@@ -185,14 +160,14 @@ def test_insert_with_auto_schema(db):
    datas_from_db = list(table_or_collection.select().execute())

    for d, d_db in zip(datas, datas_from_db):
-        assert d["img"].size == d_db["img"].size
+        assert d["df"].sum().sum() == d_db["df"].sum().sum()


np.sum(d['df'])
might reduce it

just a small pick

Not work with using np.sum(d['df']), I changed to d["df"].values.sum()

jieguangzhou · 2024-08-13T12:26:22Z

.github/workflows/release_plugins.yaml

Release the specified plugin with the following commit message
[PLUGINS] Bump Version [plugin_name_1 | plugin_name_2]

Release all plugins with the following commit message

[PLUGINS] Bump Version [all]

blythed

Well done!

jieguangzhou force-pushed the feat/migrate-plugins branch from d5a1234 to 29c4ae3 Compare August 8, 2024 13:38

jieguangzhou mentioned this pull request Aug 8, 2024

Migrate plugins #2342

Closed

jieguangzhou changed the title ~~Feat/migrate plugins~~ Migrate plugins Aug 8, 2024

jieguangzhou force-pushed the feat/migrate-plugins branch from e4999f3 to c5e9119 Compare August 8, 2024 14:04

blythed reviewed Aug 8, 2024

View reviewed changes

test/integration/backends/test_metdata.py Outdated Show resolved Hide resolved

blythed reviewed Aug 8, 2024

View reviewed changes

test/unittest/backends/mongodb/test_metadata.py Outdated Show resolved Hide resolved

jieguangzhou force-pushed the feat/migrate-plugins branch from 6eb8da9 to e742a70 Compare August 9, 2024 08:42

jieguangzhou force-pushed the feat/migrate-plugins branch 6 times, most recently from a985549 to 3316c3c Compare August 12, 2024 09:03

jieguangzhou commented Aug 12, 2024

View reviewed changes

jieguangzhou force-pushed the feat/migrate-plugins branch from d2fca4e to a905f01 Compare August 12, 2024 09:26

jieguangzhou marked this pull request as ready for review August 12, 2024 09:33

jieguangzhou self-assigned this Aug 12, 2024

jieguangzhou requested review from blythed and kartik4949 August 12, 2024 09:33

jieguangzhou force-pushed the feat/migrate-plugins branch from a905f01 to aecb5b7 Compare August 12, 2024 09:36

jieguangzhou commented Aug 12, 2024

View reviewed changes

blythed reviewed Aug 12, 2024

View reviewed changes

jieguangzhou force-pushed the feat/migrate-plugins branch 3 times, most recently from e850ae8 to 5d1ab10 Compare August 13, 2024 08:21

jieguangzhou force-pushed the feat/migrate-plugins branch 5 times, most recently from 14e6095 to de31d8f Compare August 13, 2024 08:34

kartik4949 requested changes Aug 13, 2024

View reviewed changes

jieguangzhou force-pushed the feat/migrate-plugins branch from de31d8f to 8aafbf9 Compare August 13, 2024 12:19

jieguangzhou requested review from kartik4949 and blythed August 13, 2024 12:20

jieguangzhou commented Aug 13, 2024

View reviewed changes

jieguangzhou force-pushed the feat/migrate-plugins branch 2 times, most recently from bb8c05a to 54ccb3e Compare August 13, 2024 13:15

blythed and others added 8 commits August 13, 2024 10:21

Migrate plugins

dea372d

test ci

e363bec

Fix the tests regarding default.yaml after migration.

95806ce

Fix the tests regarding ibis.yaml after migration.

807fdf9

[PLUGIN ALL] Updated CI

5499f37

Updated plugin CI workflow

a13eb67

Removed DevKit and added a deprecated warning for the plugins.

2e6e877

[PLUGINS] Bump Version [dummy]

26eccc8

blythed force-pushed the feat/migrate-plugins branch from 54ccb3e to 26eccc8 Compare August 13, 2024 14:21

blythed approved these changes Aug 13, 2024

View reviewed changes

kartik4949 approved these changes Aug 13, 2024

View reviewed changes

blythed merged commit cc5a04d into superduper-io:main Aug 13, 2024
18 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate plugins #2378

Migrate plugins #2378

jieguangzhou commented Aug 8, 2024

jieguangzhou Aug 12, 2024

jieguangzhou Aug 12, 2024 •

edited

Loading

jieguangzhou Aug 12, 2024 •

edited

Loading

blythed Aug 12, 2024

jieguangzhou Aug 13, 2024

blythed Aug 12, 2024 •

edited

Loading

jieguangzhou Aug 13, 2024 •

edited

Loading

blythed left a comment •

edited

Loading

jieguangzhou commented Aug 13, 2024

kartik4949 Aug 13, 2024

jieguangzhou Aug 13, 2024 •

edited

Loading

blythed Aug 13, 2024

kartik4949 Aug 13, 2024

jieguangzhou Aug 13, 2024

kartik4949 Aug 13, 2024

jieguangzhou Aug 13, 2024 •

edited

Loading

kartik4949 Aug 13, 2024

jieguangzhou Aug 13, 2024 •

edited

Loading

jieguangzhou Aug 13, 2024

blythed left a comment


		class FieldType(Leaf):

Migrate plugins #2378

Migrate plugins #2378

Conversation

jieguangzhou commented Aug 8, 2024

Description

Related Issues

Checklist

Additional Notes or Comments

Choose a reason for hiding this comment

jieguangzhou Aug 12, 2024 • edited Loading

Choose a reason for hiding this comment

jieguangzhou Aug 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blythed Aug 12, 2024 • edited Loading

Choose a reason for hiding this comment

jieguangzhou Aug 13, 2024 • edited Loading

Choose a reason for hiding this comment

blythed left a comment • edited Loading

Choose a reason for hiding this comment

jieguangzhou commented Aug 13, 2024

Choose a reason for hiding this comment

jieguangzhou Aug 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jieguangzhou Aug 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jieguangzhou Aug 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blythed left a comment

Choose a reason for hiding this comment

jieguangzhou Aug 12, 2024 •

edited

Loading

jieguangzhou Aug 12, 2024 •

edited

Loading

blythed Aug 12, 2024 •

edited

Loading

jieguangzhou Aug 13, 2024 •

edited

Loading

blythed left a comment •

edited

Loading

jieguangzhou Aug 13, 2024 •

edited

Loading

jieguangzhou Aug 13, 2024 •

edited

Loading

jieguangzhou Aug 13, 2024 •

edited

Loading