Pre-loading a moto dynamodb table

[jagoodhand]

I'm using DynamoDB to cache a load of reference data - essentially acting as a dictionary lookup.

The reference data is ~600MB, and in the region of 5 million items.

Currently i'm loading this data by looping through all the items in the list of DynamoDB object dictionaries in Python and then writing them into the database using the DynamoDB batch writer.

This takes ages. And isn't really practical for running my tests.

Is there a way that I can do this more efficiently? Something like generating a database object once that I just load into the moto instance, rather than having to do it the 'proper' way?

EDIT: The proper solution for this is probably implementing the import_table method using some efficient coding to ensure the load is quick. You could then upload the data to a moto3 s3 bucket, and import it from there.

[bblommers]

Hi @jagoodhand! moto >= 5.0.7.dev32 (just released) has support for the import_table method, with the caveat that it only supports the DYNAMODB_JSON-format and does not support the ZSTD-encryption.

Note that I haven't done any performance testing/investigation - it's quite a naive implementation for now. Let me know if it is helpful in any way though.

[jagoodhand]

@bblommers oh perfect, thanks I’ll take a look!

I was just taking a look at the implementation, and I noticed that the put_item method always calls the get_item method, regardless of if it is required or not:

moto/moto/dynamodb/models/table.py

Line 580 in fa52c54

current = self.get_item(hash_value, lookup_range_value)

I believe that current should only be required if override is true, or if there is a stream_shard I would imagine that moving this statement behind an if clause would improve the efficiency of the put operation a little.

[jagoodhand]

@bblommers - tried using the import_table method to load the data, but unfortunately it made no difference to the time compared to using a batch_write operation.

Moving the current = ... bit also made negligible difference.

For now I think i'll need to look at other options.

[bblommers]

One alternative is to load it this data using boto3 (using whatever method), pickle the Moto backend once, and then reuse/load the pickled data again for every test. The initial loading will take time, but (un-)pickling that is quick.

# one time setup
from moto.dynamodb.models import dynamodb_backends
result = pickle.dumps(dynamodb_backends)

# export to file for easy reuse...
dynamodb_backends.update(pickle.loads(result))

[jagoodhand]

THIS is exactly what I was looking for. Thank you!

I ended up re-creating this (kinda) using dynalite and and dynamodb-local (dealthy slow! It took 8-9 hours to load what moto loaded in 4 mins or so - still too slow for testing, but nothing like dynamodb-local)

Thank you! Would much rather stick with my moto-for-everything approach, which this enables. 🙏

EDIT: is this kind of approach documented anywhere? I couldn't find anything in the docs, but it would be really helpful if it were.

[bblommers]

Happy to help @jagoodhand!

I should have mentioned this immediately - we consider this an internal API, so the exact function signatures/models inside dynamodb_backends may change as we need to add new features/bugfixes.

That's also the reason that is isn't documented, and the reason that I didn't immediately wanted to suggest this - the recommended approach is always to use boto3 where possible, as that guarantees a stable API. Only when that's not possible/too slow, would I recommend using the internal API (with this caveat).

The dynamodb_backends-object itself and the models that store the DynamoDB items tend to be fairly stable, so I don't expect breaking changes often, but yeah - buyer beware, it may happen. 🙂