feat: add support for redirecting Wrangler to a generated config when running deploy commands #7442

petebacondarwin · 2024-12-04T16:46:13Z

This feature is designed for build tools and frameworks to provide a deploy-specific configuration, which Wrangler can use instead of user configuration when running deploy commands.
It is not expected that developers of Workers will need to use this feature directly.

Affected commands

The commands that use this feature are:

wrangler deploy
wrangler dev
wrangler versions upload
wrangler versions deploy
wrangler pages deploy
wrangler pages build
wrangler pages build-env

Config redirect file

When running these commands, Wrangler will look up the directory tree from the current working directory for a file at the path .wrangler/deploy/config.json. This file must contain only a single JSON object of the form:

{ "configPath": "../../path/to/wrangler.json" }

When this file exists Wrangler will use the configPath (relative to the config.json file) to find an alternative Wrangler configuration file to load and use as part of this command.

When this happens Wrangler will display a warning to the user to indicate that the configuration has been redirected to a different file than the user's configuration file.

Custom build tool example

A common approach that a build tool might choose to implement.

The user writes code that uses Cloudflare Workers resources, configured via a user wrangler.toml file.
```
name = "my-worker"
main = "src/index.ts"
[[kv_namespaces]]
binding = "<BINDING_NAME1>"
id = "<NAMESPACE_ID1>"
```
Note that this configuration points main at user code entry-point.
The user runs a custom build, which might read the wrangler.toml to find the entry-point:
```
> my-tool build
```
This tool generates a dist directory that contains both compiled code and a new deployment configuration file, but also a .wrangler/deploy/config.json file that redirects Wrangler to this new deployment configuration file:
```
- dist
  - index.js
	- wrangler.json
- .wrangler
  - deploy
	  - config.json
```
The dist/wrangler.json will contain:
```
{
	"name": "my-worker",
	"main": "./index.js",
	"kv_namespaces": [{ "binding": "<BINDING_NAME1>", "id": "<NAMESPACE_ID1>" }]
}
```
And the .wrangler/deploy/config.json will contain:
```
{
	"configPath": "../../dist/wrangler.json"
}
```

Tests
- TODO (before merge)
- Tests included
- Tests not necessary because:
E2E Tests CI Job required? (Use "e2e" label or ask maintainer to run separately)
- I don't know
- Required
- Not required because:
Public documentation
- TODO (before merge)
- Cloudflare docs PR(s): wrangler: document the new "redirected" configuration feature cloudflare-docs#18757
- Documentation not necessary because:

changeset-bot · 2024-12-04T16:46:18Z

🦋 Changeset detected

Latest commit: 1f9c943

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
wrangler	Minor
@cloudflare/vitest-pool-workers	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2024-12-04T19:53:25Z

A wrangler prerelease is available for testing. You can install this latest build in your project with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/12480724262/npm-package-wrangler-7442

You can reference the automatically updated head of this PR with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/prs/7442/npm-package-wrangler-7442

Or you can use npx with this latest build directly:

npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/12480724262/npm-package-wrangler-7442 dev path/to/script.js

Additional artifacts:

wget https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/12480724262/npm-package-cloudflare-workers-bindings-extension-7442 -O ./cloudflare-workers-bindings-extension.0.0.0-v356c10f90.vsix && code --install-extension ./cloudflare-workers-bindings-extension.0.0.0-v356c10f90.vsix

npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/12480724262/npm-package-create-cloudflare-7442 --no-auto-update

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/12480724262/npm-package-cloudflare-kv-asset-handler-7442

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/12480724262/npm-package-miniflare-7442

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/12480724262/npm-package-cloudflare-pages-shared-7442

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/12480724262/npm-package-cloudflare-unenv-preset-7442

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/12480724262/npm-package-cloudflare-vitest-pool-workers-7442

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/12480724262/npm-package-cloudflare-workers-editor-shared-7442

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/12480724262/npm-package-cloudflare-workers-shared-7442

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/12480724262/npm-package-cloudflare-workflows-shared-7442

Note that these links will no longer work once the GitHub Actions artifact expires.

[email protected] includes the following runtime dependencies:

Package	Constraint	Resolved
`miniflare`	workspace:*	3.20241218.0
`workerd`	1.20241218.0	1.20241218.0
`workerd --version`	1.20241218.0	2024-12-18

Please ensure constraints are pinned, and miniflare/workerd minor versions match.

See cloudflare/workers-sdk#7442

tools/e2e/runIndividualE2EFiles.ts

petebacondarwin · 2024-12-17T11:41:26Z

fixtures/import-npm/package.json

+		"check:type": "npm run _clean_install && npm run check:type --workspaces",
+		"test:ci": "npm run _clean_install && npm run test:ci --workspaces",
+		"test:watch": "npm run _clean_install && npm run test:watch --workspaces",
+		"type:tests": "npm run _clean_install && npm run type:tests --workspaces"


I found that this fixture's tests kept failing; and I think it was because we need to ensure that the node_modules directory was in fact created by the npm i command here, and not a pnpm i command that is naturally run when setting up the workers-sdk project.

Furthemore I couldn't just run _clean_install in front of each of these scripts because turbo tries to run then in parallel. So the solution was to add a turbo dependency instead, which means that this only runs once.

petebacondarwin · 2024-12-17T11:43:41Z

packages/wrangler/e2e/dev-registry.test.ts

@@ -99,27 +99,25 @@ describe("unstable_dev()", () => {
 					const childWorker = await unstable_dev(
 						"${child.replaceAll("\\", "/")}/src/index.ts",
 						{
-							configPath: "${child.replaceAll("\\", "/")}/wrangler.toml",


It turns out that:

a) the configPath property is not valid so was not being used
b) Wrangler was using the script path above to infer the location of the config path

I could have changed the configPath property to config which is more correct, but actually not having this here at all is a good check that we are resolving the config by "finding up" from the script path (rather than the current working directory).

petebacondarwin · 2024-12-17T11:44:36Z

packages/wrangler/e2e/dev-with-resources.test.ts

@@ -8,6 +8,9 @@ import WebSocket from "ws";
 import { WranglerE2ETestHelper } from "./helpers/e2e-wrangler-test";
 import { generateResourceName } from "./helpers/generate-resource-name";

+const port = await getPort();
+const inspectorPort = await getPort();


The only changes in this file are to explicitly set both the port and inspector port to avoid concurrent tests from failing due to port 9229 being blocked.

These tests don't run concurrently in CI—was this a problem you were seeing locally?

Yeah it was when I running tests locally.

petebacondarwin · 2024-12-17T11:44:51Z

packages/wrangler/e2e/pages-dev.test.ts

-		await helper.seed({
-			"_worker.js": dedent`
+const port = await getPort();
+const inspectorPort = await getPort();


The only changes in this file are to explicitly set both the port and inspector port to avoid concurrent tests from failing due to port 9229 being blocked.

petebacondarwin · 2024-12-17T11:48:25Z

packages/wrangler/src/config/validation.ts

@@ -327,7 +330,7 @@ function applyPythonConfig(
 	config: Config,
 	args: NormalizeAndValidateConfigArgs
 ) {
-	const mainModule = "script" in args ? args.script : config.main;
+	const mainModule = args.script ?? config.main;


It is possible that the args.script property exists but is set to undefined, which means that we should still defer to config.main in that case.

This creates a lot of unwanted logging in the console. The logging is still dumped to the console if the test times out, or if `WRANGLER_LOG=debug` is set as env var.

…ITag()

… running deploy commands This new feature is designed for build tools and frameworks to provide a deploy-specific configuration, which Wrangler can use instead of user configuration when running deploy commands. It is not expected that developers of Workers will need to use this feature directly. The commands that use this feature are: - `wrangler deploy` - `wrangler dev` - `wrangler versions upload` - `wrangler versions deploy` When running these commands, Wrangler will look up the directory tree from the current working directory for a file at the path `.wrangler/deploy/config.json`. This file must contain only a single JSON object of the form: ```json { "configPath": "../../path/to/wrangler.json" } ``` When this file exists Wrangler will use the `configPath` (relative to the `deploy.json` file) to find an alternative Wrangler configuration file to load and use as part of this command. When this happens Wrangler will display a warning to the user to indicate that the configuration has been redirected to a different file than the user's configuration file. A common approach that a build tool might choose to implement. - The user writes code that uses Cloudflare Workers resources, configured via a user `wrangler.toml` file. ```toml name = "my-worker" main = "src/index.ts" [[kv_namespaces]] binding = "<BINDING_NAME1>" id = "<NAMESPACE_ID1>" ``` Note that this configuration points `main` at user code entry-point. - The user runs a custom build, which might read the `wrangler.toml` to find the entry-point: ```bash > my-tool build ``` - This tool generates a `dist` directory that contains both compiled code and a new deployment configuration file, but also a `.wrangler/deploy/config.json` file that redirects Wrangler to this new deployment configuration file: ```plain - dist - index.js - wrangler.json - .wrangler - deploy - config.json ``` The `dist/wrangler.json` will contain: ```json { "name": "my-worker", "main": "./index.js", "kv_namespaces": [{ "binding": "<BINDING_NAME1>", "id": "<NAMESPACE_ID1>" }] } ``` And the `.wrangler/deploy/config.json` will contain: ```json { "configPath": "../../dist/wrangler.json" } ```

… configuration

…ectory containing: ``` CLOUDFLARE_ACCOUNT_ID=<dev-prod-account-id> CLOUDFLARE_API_TOKEN=<dev-prod-account-api-key> WRANGLER="node <path to workers-sdk/packages/wrangler/bin/wrangler.js>" WRANGLER_IMPORT="<path to /workers-sdk/packages/wrangler/wrangler-dist/cli.js>" ````

…vided

… relative pages_build_output_dir path

…` contains absolute paths.

…onfigRedirectIfAvailable"

This fixture does not need a package lockfile to work.

packages/wrangler/src/__tests__/config/findWranglerConfig.test.ts

vicb · 2024-12-24T09:50:31Z

packages/wrangler/src/config/config-helpers.ts

+		config?: string;
+		script?: string;
+	},
+	options: { useRedirectIfAvailable?: boolean }


Should this (and the default value/behavior) be documented?

Yes documentation is in cloudflare/cloudflare-docs#18757

vicb · 2024-12-24T09:51:22Z

packages/wrangler/src/config/config-helpers.ts

-): string | undefined {
-	return (
+	referencePath: string = process.cwd(),
+	{ useRedirectIfAvailable = false } = {}


document this?

Will be in cloudflare/cloudflare-docs#18757

packages/wrangler/src/config/index.ts

petebacondarwin force-pushed the pbd/add-config-redirect-feature branch from 5a91f5b to b3bf041 Compare December 4, 2024 19:50

petebacondarwin force-pushed the pbd/add-config-redirect-feature branch 4 times, most recently from e251387 to 3756ab2 Compare December 12, 2024 12:08

petebacondarwin force-pushed the pbd/add-config-redirect-feature branch 3 times, most recently from 0e8fc8a to a54d88c Compare December 16, 2024 08:22

petebacondarwin added the e2e Run e2e tests on a PR label Dec 16, 2024

petebacondarwin mentioned this pull request Dec 16, 2024

wrangler config - add extended configuration docs cloudflare/cloudflare-docs#18146

Closed

1 task

petebacondarwin force-pushed the pbd/add-config-redirect-feature branch 2 times, most recently from 48b73e0 to a06374a Compare December 16, 2024 11:27

petebacondarwin added a commit to petebacondarwin/cloudflare-docs that referenced this pull request Dec 16, 2024

wrangler: document the new "redirected" configuration feature

d850131

See cloudflare/workers-sdk#7442

petebacondarwin mentioned this pull request Dec 16, 2024

wrangler: document the new "redirected" configuration feature cloudflare/cloudflare-docs#18757

Draft

3 tasks

petebacondarwin force-pushed the pbd/add-config-redirect-feature branch 2 times, most recently from e7fd32d to f33b32a Compare December 17, 2024 09:50

petebacondarwin marked this pull request as ready for review December 17, 2024 10:43

petebacondarwin requested review from a team as code owners December 17, 2024 10:43

petebacondarwin commented Dec 17, 2024

View reviewed changes

tools/e2e/runIndividualE2EFiles.ts Outdated Show resolved Hide resolved

petebacondarwin commented Dec 17, 2024

View reviewed changes

petebacondarwin force-pushed the pbd/add-config-redirect-feature branch from f33b32a to 13d920f Compare December 17, 2024 12:04

petebacondarwin added 23 commits December 23, 2024 14:02

Do not console log wrangler output during normal fixture tests

23c8ab5

This creates a lot of unwanted logging in the console. The logging is still dumped to the console if the test times out, or if `WRANGLER_LOG=debug` is set as env var.

no need to get relative config path for calls to verifyWorkerMatchesC…

0df860b

…ITag()

add config redirect fixture with tests

7fbf0c4

Add config redirect support for Pages commands

47dc486

refactor: compute both the actual config path and the original user's…

714c758

… configuration

add userConfigPath to the Config type so it can be used where needed

3ab41cc

add fixture and test for redirected config in a pages project

9f69bda

Do not check formatting of generated fixture files

b27a03c

clean node_modules for npm-import fixture before testing

64e3ef1

fix unstable_dev tests

9704eca

run more Wrangler e2e tests on safe ports

b8e2fc7

correctly configure python workers when no command line script is pro…

9efa9cd

…vided

test: ensure that pages functions build-env command outputs correct…

395dc67

… relative pages_build_output_dir path

Improve error message when deploy/config.json is not valid

1de7afd

Complete jsdoc description

799c85f

Update jsdocs to note that the result of `resolveWranglerConfigPath()…

fbce3bd

…` contains absolute paths.

PR feedback

c8d4bcf

rename useRedirect to useRedirectIfAvailable

a247fce

Rename command definition behaviour from "useConfigRedirect" to "useC…

3053fad

…onfigRedirectIfAvailable"

Add clarifying comment

2df8134

Do not store package-lock.json for import-npm fixture

e153667

This fixture does not need a package lockfile to work.

petebacondarwin force-pushed the pbd/add-config-redirect-feature branch from 244c408 to e153667 Compare December 23, 2024 14:03

vicb reviewed Dec 24, 2024

View reviewed changes

packages/wrangler/src/__tests__/config/findWranglerConfig.test.ts Outdated Show resolved Hide resolved

vicb reviewed Dec 24, 2024

View reviewed changes

packages/wrangler/src/__tests__/config/findWranglerConfig.test.ts Show resolved Hide resolved

vicb reviewed Dec 24, 2024

View reviewed changes

packages/wrangler/src/config/index.ts Show resolved Hide resolved

Remove unnecessary option in test

1f9c943

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support for redirecting Wrangler to a generated config when running deploy commands #7442

feat: add support for redirecting Wrangler to a generated config when running deploy commands #7442

petebacondarwin commented Dec 4, 2024 •

edited

Loading

changeset-bot bot commented Dec 4, 2024 •

edited

Loading

github-actions bot commented Dec 4, 2024 •

edited

Loading

petebacondarwin Dec 17, 2024

petebacondarwin Dec 24, 2024

petebacondarwin Dec 17, 2024

petebacondarwin Dec 17, 2024

penalosa Dec 20, 2024

petebacondarwin Dec 20, 2024

petebacondarwin Dec 17, 2024

petebacondarwin Dec 17, 2024

vicb Dec 24, 2024

petebacondarwin Dec 24, 2024

vicb Dec 24, 2024

petebacondarwin Dec 24, 2024

feat: add support for redirecting Wrangler to a generated config when running deploy commands #7442

Are you sure you want to change the base?

feat: add support for redirecting Wrangler to a generated config when running deploy commands #7442

Conversation

petebacondarwin commented Dec 4, 2024 • edited Loading

Affected commands

Config redirect file

Custom build tool example

changeset-bot bot commented Dec 4, 2024 • edited Loading

🦋 Changeset detected

github-actions bot commented Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petebacondarwin commented Dec 4, 2024 •

edited

Loading

changeset-bot bot commented Dec 4, 2024 •

edited

Loading

github-actions bot commented Dec 4, 2024 •

edited

Loading