-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
We shouldn't require track transferability #113
Comments
What problem is this solving? Transferring the track has other benefits like being able to apply constraints and read track stats and settings. |
It solves the problem that you don't need track transferability to implement this API, which we consider a blocker, at least for the medium term. This proposal uses a pattern that we are already using in encoded transform and should easily allow us to have interoperable implementations. |
One use case for transferring media stream track is to create a track (via VideoTrackGenerator) and send it to sinks like RTCRtpSender or MediaRecorder. |
The proposal is that createVideoTrackGenerator() is called from Window and returns a promise with a MediaStreamTrack on Window, where the RTCRtpSender or MediaRecorder are. The generator (which no longer has a track field) is created in the worker (the application gets it via an event, just like an RTCRtpScriptTransformer). This removes the need to transfer the track. You only needed to transfer the track from the worker to window because the current spec creates the track on the worker, where it is largely useless, as all the track APIs are on Window. |
Another benefit of this API surface is that it allows feature detection on main without creating a worker. |
This can be feature detected on main like this: function isMstTransferable() {
try {
const [track] = document.createElement('canvas').captureStream().getVideoTracks();
new MessageChannel().port1.postMessage(track, [track]);
return true;
} catch (e) {
if (e.name != "DataCloneError") throw e;
return false;
}
} |
@guidou why is transfer a blocker? What do you mean by medium term? Safari has already shipped this and it works. If you explain the problem, perhaps their engineers can help? |
It's a blocker for Chromium to ship it in the short term since Chromium doesn't implement track transferability and will not have it for quite some time. I don't expect Chromium to have track transferability in the short term, so I guess we won't have an interoperable API for a long time. |
This feature-detects track transferability, not mediacapture-transform. |
I doubt that's needed. As you said, "tracks are useless on workers except for this API". If MST transfer is detected, it seems reasonable to assume some purpose awaits these tracks in the worker. This works in the only current implementation: "WebKit for Safari 18 beta adds support for MediaStreamTrack processing in a dedicated worker." This seems like a property worth emulating. I've added a note to Firefox's implementation bug to do the same. Thanks for bringing attention to this! |
If there's some difficulty or problem with the spec's transfer steps as specified, please bring it to our attention so we can address it. |
Yes, but waiting on postMessage for these measurements hardly seems ideal. In the current spec, the worker transform can inspect real-time track stats counters like |
The current version of the API requires track transferability, but this shouldn't be necessary.
Currently, tracks are useless on workers except for this API, so we shouldn't add that as a requirement.
A way to keep the API worker first which has several benefits is to follow the postMessage-like approach of webrtc-encoded-transform.
Something (subject to discussion) like:
For MediaStreamTrackProcessor:
For VideoTrackGenerator:
The text was updated successfully, but these errors were encountered: