Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A ReadableStream branch was created but never consumed #648

Open
HaidongPang opened this issue Nov 6, 2024 · 3 comments
Open

A ReadableStream branch was created but never consumed #648

HaidongPang opened this issue Nov 6, 2024 · 3 comments

Comments

@HaidongPang
Copy link

I'm using ky in cloudflare worker with following code:

import ky from "ky";
export default {
    async fetch (): Promise<Response> {
        const result = await ky.get("https://www.github.com").text();
        return new Response(result);
    }
} satisfies ExportedHandler;

and when I turn off local mode, requests raise the following warning:

A ReadableStream branch was created but never consumed. Such branches can be created, for instance, by calling the tee() method on a ReadableStream, or by calling the clone() method on a Request or Response object. If a branch is created but never consumed, it can force the runtime to buffer the entire body of the stream in memory, which may cause the Worker to exceed its memory limit and be terminated. To avoid this, ensure that all branches created are consumed.

 * Unused stream created:
    at result.<computed> [as text] (file:///....../node_modules/ky/source/core/Ky.ts:92:36)
......

Locate the following code snippet:

source/core/Ky.ts

export class Ky {
	static create(input: Input, options: Options): ResponsePromise {
		const ky = new Ky(input, options);

		const function_ = async (): Promise<Response> => {
                         ......
		};

		......

		for (const [type, mimeType] of Object.entries(responseTypes) as ObjectEntries<typeof responseTypes>) {
			result[type] = async () => {
				......
				const awaitedResult = await result;
				const response = awaitedResult.clone();

				if (type === 'json') {
					if (response.status === 204) {
						return '';
					}

					const arrayBuffer = await response.clone().arrayBuffer();
					const responseSize = arrayBuffer.byteLength;
					if (responseSize === 0) {
						return '';
					}

					if (options.parseJson) {
						return options.parseJson(await response.text());
					}
				}

				return response[type]();
			};
		}

		return result;
	}

The awaitedResult is a variable of type Response. Since it is declared, it has been cloned only once, and its ReadableStream has never been consumed.
After awaitedResult has finished cloning, its ReadableStream should be closed to ensure memory safety.
Seems like:

const awaitedResult = await result;
const response = awaitedResult.clone();
await awaitedResult.body.cancel();
@sholladay
Copy link
Collaborator

PR welcome to fix this issue. If I'm not mistaken, canceling the stream could lead to lost data even though we've cloned the stream. We'd need to test it thoroughly to make sure that's not the case.

HaidongPang added a commit to HaidongPang/ky that referenced this issue Nov 7, 2024
Clean up orphan ReadableStream remaining during parsing response (sindresorhus#648)
@HaidongPang
Copy link
Author

I‘m willing to do so, and a PR have already been submitted .
Regarding your concerns, here is a simple verification.

export default {
    async fetch (): Promise<Response> {
        const originalResponse = new Response("Some Data");
        const clonedResponse = originalResponse.clone();
        await originalResponse.body?.cancel();
        console.log("Original Stream Data:", await originalResponse.body?.getReader().read());
        console.log("Cloned Stream Data:", await clonedResponse.body?.getReader().read());
        return new Response("");
    }
} satisfies ExportedHandler;

I cloned an unconsumed response and canceled the ReadableStream of the original response before consuming the cloned response.

The result is as follows:

Original Stream Data: Object {
  done: true
}
Cloned Stream Data: Object {
  value: Uint8Array(9),
  done: false
}

The original ReadableStream has already been canceled, but the cloned response has an independent ReadableStream object that can be consumed.

@HaidongPang
Copy link
Author

There is also a similar issue about orphan ReadableStream remaining.

Related to :#650

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants