fix(browser) detect WebRTC APIs and mark browser unsupported if not
Some so called "security" browser extensions override WebRTC APIs so
even if the UA string suggests the browser is supported, it won't work.
Coincidentally, this should also mark Safari in Lockdown Mode as
unsupported, since the WebRTC APIs are not available in that case.
feat(codec-selection): Use the new codec selection API (#2520)
* feat(codec-selection): Use the new codec selection API
https://github.com/w3ctag/design-reviews/issues/836. This allows the client to seamlessly switch between the codecs without having to trigger a renegotiation.
This feature is behind the flag testing.enableCodecSelectionAPI in config.js
* fix(stats): Fix local resolution stats.
The video codec for the local video sources needs to identified differently now, from the codecs field in the RTCRtpSendParameters returned by the browser. We no longer munge the remote SDP to switch to a different codec.
* feat(stats): Feed encodeTime stats for all local SSRCs to the codec selection mechanism.
* fix(codec-selection) Continue to mumge SDP for selecting H.264.
* feat(codec-selection) Make screenshare codec configurable.
If no 'screenshareCodec' is set under videoQuality or p2p settings, AV1 will be selected as default.
* squash: Address review comments
* Update modules/RTC/CodecSelection.js
Co-authored-by: Saúl Ibarra Corretgé <s@saghul.net>
* fix(codec-selection) Add codec to existing stats
---------
Co-authored-by: Saúl Ibarra Corretgé <s@saghul.net>
* fix(RTC): Remove support for plan-b SDP.
Since unified plan mode becomes the default mode, so does multi-stream mode.
* squash: Remove plan-b unit tests and fix comments
* squash: address review comments
Add SVC and Simulcast support for AV1 and VP9. (#2350)
Add SVC and Simulcast support for AV1 and VP9.
The default mode is the full SVC mode for VP9. It works as expected with the current version of JVB.
AV1 in the client is only supported when the bridge negotiates the Dependency Descriptor extension headers which are needed for reading the layer information for AV1 and H.264 video frames.
* feat: Enable H.264 simulcast support.
Multi-encoding simulcast for H.264 is supported now because of the DD header ext support added on the bridge side.
* fix(video-quality): Add default bitrates for all codecs.
Expect the videoQuality settings in the new format, deprecated configs are still supported.
videoQuality: {
AV1: {
maxBitratesVideo: {
low: 100000,
standard: 300000,
high: 1000000,
ssHigh: 1500000
},
useScalabilityModeAPI: {
enabled: true,
useSimulcast: true, //(defaults to SVC),
useKSVC: true //(defaults to L3T3_KEY)
},
},
H264: {
maxBitratesVideo: {
low: 200000,
standard: 500000,
high: 1500000,
ssHigh: 2500000
}
},
VP8: {
maxBitratesVideo: {
low: 200000,
standard: 500000,
high: 1500000,
ssHigh: 2500000
}
},
VP9: {
maxBitratesVideo: {
low: 100000,
standard: 300000,
high: 1200000,
ssHigh: 2500000
},
useScalabilityModeAPI: {
enabled: true,
useSimulcast: true, //(defaults to SVC),
useKSVC: true //(defaults to L3T3_KEY)
},
}
}
* Disable encodings when requested height for screenshare is 0.
* feat: add unit tests for scalability modes.
* ref: Move all encoding configuration calculation to TPCUtils.
* Add more unit tests for VP9 K-SVC mode.
* Make codec name in settings case insensitive.
Also read the deprecated max bitrates correctly, add unit tests to test it.
* ref(TPC): isSimulcastOn -> isSpatialScalabilityOn.
It makes more sense to call it spatial scalability than simulcast now that full SVC support is available.
* fix: Negotiate AV1 DD header exts only for AV1 and H.264.
* fix: Rename setting useL3T3Mode->useKSVC and adjust the defaults.
* fix(codec-selection) Use a configurable preferred codecs list to select the codecs.
Allow asymmetric codecs to be configured on the endpoints. This means that Firefox and Safari which have bugs with VP9 encode will now encode VP8 but decode VP9 coming in from Chromium endpoints.
* fix(codec-selection) Add unit tests for the codec selection logic.
* feat(codec-selection): Introduce mobileCodecPreferenceOrder setting in config.js.
* fix(codec-selection) Avoid unnecessary renegotiations.
On participant join/leave, check if the new intersection of codecs are already configured to be the top n codecs.
* Address review comments
* fix: Strip the codecs that are not in the codec list from the video m-lines.
Also, ignore remote codecs published in presence for p2p connections.
* fix: Define default codec order for mobile and desktop.
Fix an issue where p2p between mobile and desktop was broken.
* fix: Add default codecs for both p2p and jvb on mobile devices.
Munge the initial offer sent out by RN clients since RN doesn't support RTCRtpTransceiver#setCodecPreferences.
fix(stats) Obtain resolution/fps from 'outbound-rtp' stats. (#2265)
* fix(stats) Obtain resolution/fps from 'outbound-rtp' stats.
'Track' based stats were dropped in Chrome 112, therefore send resolution and fps for the simulcast case needs to be calculated based on the 'outbound-rtp' streams that are currently active.
* squash: remove an unwanted log
* Squash: Address review comments.
Use outbound-rtp stats for both Firefox and Safari.
Firefox - Ignore active field if not present in the stats and calc fps using 'framesSent'.
There was an error in the pattern match, everything was included in it. In
addition return a boolean in case we succeed in the UA pattern match, since it's
more correct than returning an array with the match.
fix(browser-support): Add audio track to pc always on mobile Safari.
On mobile Safari, if a user joins audio and video muted, the browser doesn't decode the incoming audio. Workaround is to always add the audio track to pc and mute it if needed.
fix(codec-selection): Fix VP9 codec switching issue in Chrome unified-plan.
Munge only the m-line that corresponds to the source that the browser will be sending.
Do not select VP9 on Firefox.
Detect support for RTCRtpTransceiver#setCodecPreferences correctly.
fix(CodecSelection): Call RTCRtpTransceiver#setCodecPreferences before renegotiation.
Call RTCRtpTransceiver#setCodecPreferences with the preferrred codec order before every createOffer/createAnswer. This ensures that the codec preference is enforced even when there is no local description available yet while the preferred codec is being set immediately after media session creation.
fix(codec-selection): Fix codec selection for unified plan browsers.
Make sure the codec order is munged on all the m-lines for unified plan clients. Implement the logic for setting the preferences through RTCRtpTransceiver#setCodecPreferences.
feat(stats): Get audio levels for the top 5 speakers only.
Capture the audio levels only for the top 5 speakers as RTCRtpReceiver#getSynchronizationSources can be expensive when we have too many audio receivers in the call.
Also, capture the audio levels for track that are unmuted if RTCRtpReceiver#getSynchronizationSources is not supported.
Switch Safari to using getStats since its reporting errorneous values, i.e., 0.000001 as audio level for all remote audio tracks.
feat(browser-support): Add support for WKWebview based browsers.
Apple added getUserMedia support for WkWebview based browsers like chrome and Firefox on iOS 14.3. These browsers behave as Safari does on iOS. Therefore, extend the Safari checks to these webkit based browsers as well.
Since we use transferable streams to transfer the streams to the worker
we need to feature detect them as Chrome 86 will launch Insertable Streams
by default without shipping transferable streams yet (status still unclear,
it might still ship too)
See here for the feature detection used:
https://groups.google.com/a/chromium.org/g/blink-dev/c/1LStSgBt6AM/m/hj0odB8pCAAJ
Add a performance stat around long tasks. Chrome supports PerformanceObserver API that lets us
register for long tasks event. Any task that takes longer than 50ms is considered a long task.