aws-cdk-lib: significantly more memory used as module than commonjs

[everett1992]

Describe the bug

~/projects/cdk-test » node --expose-gc test.mjs
HEAP 17.45mb RSS 111.25mb EXT 2.03mb
~/projects/cdk-test ∷ node --expose-gc test.cjs
HEAP 2.91mb RSS 43.34mb EXT 1.07mb

test.mjs and .cjs are the same except import 'aws-cdk-lib vs require('aws-cdk-lib).

I see similar differences when importing aws-cdk-lib/core or aws-cdk-lib/aws-lambda.
I don't see similar issues with other large libraries (like @aws-sdk/client-ec2)

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

Similar memory use between ESM and CJS

Current Behavior

importing cdk uses more memory than requiring it.

Reproduction Steps

// test.cjs
require('aws-cdk-lib')
const MB = 1024 * 1024

function mb(n) {
  return (n / MB).toFixed((2))
}
global.gc?.()
const mem = process.memoryUsage()
console.error(`HEAP ${mb(mem.heapUsed)}mb RSS ${mb(mem.rss)}mb EXT ${mb(mem.external)}mb`)

// test.mjs
import 'aws-cdk-lib'
const MB = 1024 * 1024

function mb(n) {
  return (n / MB).toFixed((2))
}
global.gc?.()
const mem = process.memoryUsage()
console.error(`HEAP ${mb(mem.heapUsed)}mb RSS ${mb(mem.rss)}mb EXT ${mb(mem.external)}mb`)

~/projects/cdk-test » mise exec node@18 -- node test.mjs
HEAP 21.22mb RSS 84.24mb EXT 1.49mb
~/projects/cdk-test ∷ mise exec node@18 -- node test.cjs
HEAP 4.12mb RSS 40.49mb EXT 0.18mb

~/projects/cdk-test ∷ mise exec node@22 -- node test.mjs
HEAP 27.54mb RSS 110.08mb EXT 2.03mb
~/projects/cdk-test ∷ mise exec node@22 -- node test.cjs
HEAP 3.37mb RSS 43.00mb EXT 1.07mb

test.mjs uses significantly more heap and rss

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

N/A

Framework Version

2.173.2

Node.js Version

18, 20, 22

OS

Arch Linux

Language

TypeScript

Language Version

N/A (JavaScript)

Other information

No response

[everett1992]

For a more realistic test I converted the TypeScript sample app to ESM and CJS here

~/github/cdk-test(±cjs) $ node --expose-gc -r ./log-memory.cjs bin/cdk-test.mjs
rss 128.20mb	heap 59.91mb

~/github/cdk-test(±cjs) $ node --expose-gc -r ./log-memory.cjs bin/cdk-test.cjs
rss 94.88mb	heap 33.03mb

~/github/cdk-test(±cjs) $ hyperfine -L EXT cjs,mjs 'node --expose-gc -r ./log-memory.cjs bin/cdk-test.{EXT}'
Benchmark 1: node --expose-gc -r ./log-memory.cjs bin/cdk-test.cjs
  Time (mean ± σ):     244.6 ms ±   5.5 ms    [User: 260.3 ms, System: 85.3 ms]
  Range (min … max):   234.3 ms … 252.3 ms    12 runs

Benchmark 2: node --expose-gc -r ./log-memory.cjs bin/cdk-test.mjs
  Time (mean ± σ):     426.4 ms ±   7.1 ms    [User: 440.7 ms, System: 149.4 ms]
  Range (min … max):   415.4 ms … 440.7 ms    10 runs

Summary
  node --expose-gc -r ./log-memory.cjs bin/cdk-test.cjs ran
    1.74 ± 0.05 times faster than node --expose-gc -r ./log-memory.cjs bin/cdk-test.mjs

[everett1992]

By adding a resolve hook we can see that mjs loads about twice as many files

~/github/cdk-test(±cjs) ∷ node --expose-gc -r ./log-import-require.cjs -r ./log-memory.cjs bin/cdk-test.mjs
loaded files 589
rss 137.09mb	heap 60.68mb

~/github/cdk-test(±cjs) ∷ node --expose-gc -r ./log-import-require.cjs -r ./log-memory.cjs bin/cdk-test.cjs
loaded files 202
rss 103.58mb	heap 33.05mb

[everett1992]

Replacing import * as cdk from 'aws-cdk-lib' with import { App } from 'aws-cdk-lib' made no difference.
Replacing 'aws-cdk-lib' with 'aws-cdk-lib/core' where possible helped, but commonjs still uses less memory

~/github/cdk-test(±cjs) ∷ node --expose-gc -r ./log-memory.cjs bin/cdk-test.cjs
rss 98.37mb	heap 32.05mb

~/github/cdk-test(±cjs) ∷ node --expose-gc -r ./log-memory.cjs bin/cdk-test.mjs
rss 117.30mb	heap 55.68mb

[everett1992]

require('aws-cdk-lib') will load only node_modules/aws-cdk-lib/index.js while import {} from 'aws-cdk-lib' loads 417 files, including all of aws-cdk-lib/aws-*.

I'm guessing this is an interaction between lazify and node's ESM CJS interop. I think node enumerates all properties on the CJS export to discover named imports, which triggers the lazy load of sub modules.

The reason I'm looking into this is that I'm seeing Jest tests running out of memory after upgrading from node 18 to 20. I'm seeing it across different packages and aws-cdk-lib is the common factor. While upgrading to node 20 is increasing the OOM errors, there's evidence that the packages are seeing high memory usage with node 18, like limiting test parallelism or increasing node's memory.

[pahud]

Thank you. We'll bring this up to the core team's visibility.