Encoding / decoding live metadata for a long continuous stream

[twilson90]

What do you want to do with Hls.js?

The problem can be divided in 2, the first does not involve hls.js but I will ask it here anyway if that's ok.

Using ffmpeg I am sequencing many different media files consisting of various aspect ratios together and outputting a long running hls stream, which is served and presented using hls.js.

I would like to encode into the stream dynamic metadata that changes from time to time. Specifically I want to encode the original media's aspect ratio so it can be read by the client and the video can be presented on the front-end video player with less black bars (e.g. if a 4/3 video is encoded on a 16/9 stream but presented in a 4/3 container, you get black bars all around the image, a simple scale can effectively remove the top/bottom black bars if we know exactly what the original image's aspect ratio is).

Considering the segments are short (2 seconds) I suppose I could just append a single aspect ratio value to each segment whenever the segment is requested, but ideally I'd like to embed it in every frame and apply the crop for the end user with frame precision.

Assuming the above is possible, how would I use hls.js to read the metadata from the HLS stream?

The client views the video stream via a web browser. I could establish an additional websocket or http connection to my media server that will provide this metadata, which is a lot simpler but a little less efficient and would incur a delay.

What have you tried so far?

Nothing yet, just doing some preliminary research.

[robwalch]

Do you need to do any of that when you have HTMLVideoElement.videoHeight and videoWidth?

https://developer.mozilla.org/en-US/docs/Web/API/HTMLVideoElement/videoHeight
https://developer.mozilla.org/en-US/docs/Web/API/HTMLVideoElement/videoWidth

Video width and height are present in video init segments without additional metadata. HLS.js exposes this for M2TS segments on BUFFER_CODECS ({ video: { metadata: { width, height } } }). This track data applies to all subsequent video segments. HLS.js does not parse this information from mp4 init segments (although it could in a future update). You don't need this data from HLS.js since you can get the videoHeight and videoWidth on the HTMLVideoElement "resize" event independently.