fix(TPC) Use videoType from 'source-add' for remote track creation. (#2596)
* fix(TPC) Use videoType from 'source-add' for remote track creation.
If 'source-add' for a remote video source is received before presence for that source, videoType will default to 'camera' and the client wouldn't be able to create the virtual participant tile for rendering the desktop track.
* squash: Include the videoType for no SSRC-rewriting case.
* ref(SDPDiffer) Convert to ES6 class.
Make it work directly with unified plan SDP that has multiple m-lines and add more unit tests.
* ref(xmpp) translate unified-plan SDP->Jingle directly.
Without having to run it through the SDPInterop.toPlanB cycle.
* ref(SDP) Always generate the MSID for signaling it to Jicofo.
* fix(SDPDiffer) Check explicitly for ssrc changes
* fix(SDP): Fix comments and cleanup.
Remove LOCAL_TRACK_SSRC_UPDATED event as the application ignores the event and no additional action needs to be taken when that event is fired.
* ref(SDP) Add a helper function for parsing the 'a=ssrc-group' line.
* squash: Address review comments
* ref(SDP) Convert to ES6 class.
* ref(SDP) Use enum for URNs associated with XMPP extensions.
* ref(XMPP) Replace more XEP URN constants with enum.
* fix(RTC) Fixes mediaType lookup based on source-name.
When WebRTC ICE gathering policy is set to gather once instead of continually, the controlling agent goes to completed instead of connected (no more candidates to check). This doesn't happen in Chrome or other browsers, but is reproducible with node.js wrapper around WebRTC which runs with the default settings.
This is causing a bug where the initiator side of a P2P connection does not fully switch to P2P mode and keeps on sending data on both P2P and JVB connections.
feat(quality) Add a QualityController class for runtime adjustments. (#2542)
* feat(quality) Add a QualityController class for runtime adjustments.
Make run time adjustments to the client when adaptive mode is enabled.
* feat: Update lastN and receive resolution to improve quality.
* squash: Address review comments
* squash: Add more logging and address review comments.
feat(codec-selection): Use the new codec selection API (#2520)
* feat(codec-selection): Use the new codec selection API
https://github.com/w3ctag/design-reviews/issues/836. This allows the client to seamlessly switch between the codecs without having to trigger a renegotiation.
This feature is behind the flag testing.enableCodecSelectionAPI in config.js
* fix(stats): Fix local resolution stats.
The video codec for the local video sources needs to identified differently now, from the codecs field in the RTCRtpSendParameters returned by the browser. We no longer munge the remote SDP to switch to a different codec.
* feat(stats): Feed encodeTime stats for all local SSRCs to the codec selection mechanism.
* fix(codec-selection) Continue to mumge SDP for selecting H.264.
* feat(codec-selection) Make screenshare codec configurable.
If no 'screenshareCodec' is set under videoQuality or p2p settings, AV1 will be selected as default.
* squash: Address review comments
* Update modules/RTC/CodecSelection.js
Co-authored-by: Saúl Ibarra Corretgé <s@saghul.net>
* fix(codec-selection) Add codec to existing stats
---------
Co-authored-by: Saúl Ibarra Corretgé <s@saghul.net>
fix(ssrc-rewriting): Check if track exists before updating owner/sourceName.
Fixes missing audio issue when the bridge resends the full audio map even before the track for the SSRCs signaled before are created. This is possible when the RTP packets race with the thread that completed the websocket connection.
fix(ssrc-rewriting): Check for track owner/sourceName before calling TRACK_OWNER_SET.
When the bridge WS is re-established, jvb sends the full map of audio and video sources. Without the check, the library will end up firing TRACK_REMOVED and TRACK_ADDED for all the exiting tracks.
Also, process audio and video source maps only on JVB sessions.
fix(ssrc-rewriting): Fire track removed/added instead of owner changed.
When an existing SSRC for a remote track gets re-mapped from one source to another, fire a TRACK_REMOVED event followed by TRACK_ADDED event instead of TRACK_OWNER_CHANGED event. This should simplify the application logic for track handling.
* fix(RTC): Remove support for plan-b SDP.
Since unified plan mode becomes the default mode, so does multi-stream mode.
* squash: Remove plan-b unit tests and fix comments
* squash: address review comments
Add SVC and Simulcast support for AV1 and VP9. (#2350)
Add SVC and Simulcast support for AV1 and VP9.
The default mode is the full SVC mode for VP9. It works as expected with the current version of JVB.
AV1 in the client is only supported when the bridge negotiates the Dependency Descriptor extension headers which are needed for reading the layer information for AV1 and H.264 video frames.
* feat: Enable H.264 simulcast support.
Multi-encoding simulcast for H.264 is supported now because of the DD header ext support added on the bridge side.
* fix(video-quality): Add default bitrates for all codecs.
Expect the videoQuality settings in the new format, deprecated configs are still supported.
videoQuality: {
AV1: {
maxBitratesVideo: {
low: 100000,
standard: 300000,
high: 1000000,
ssHigh: 1500000
},
useScalabilityModeAPI: {
enabled: true,
useSimulcast: true, //(defaults to SVC),
useKSVC: true //(defaults to L3T3_KEY)
},
},
H264: {
maxBitratesVideo: {
low: 200000,
standard: 500000,
high: 1500000,
ssHigh: 2500000
}
},
VP8: {
maxBitratesVideo: {
low: 200000,
standard: 500000,
high: 1500000,
ssHigh: 2500000
}
},
VP9: {
maxBitratesVideo: {
low: 100000,
standard: 300000,
high: 1200000,
ssHigh: 2500000
},
useScalabilityModeAPI: {
enabled: true,
useSimulcast: true, //(defaults to SVC),
useKSVC: true //(defaults to L3T3_KEY)
},
}
}
* Disable encodings when requested height for screenshare is 0.
* feat: add unit tests for scalability modes.
* ref: Move all encoding configuration calculation to TPCUtils.
* Add more unit tests for VP9 K-SVC mode.
* Make codec name in settings case insensitive.
Also read the deprecated max bitrates correctly, add unit tests to test it.
* ref(TPC): isSimulcastOn -> isSpatialScalabilityOn.
It makes more sense to call it spatial scalability than simulcast now that full SVC support is available.
* fix: Negotiate AV1 DD header exts only for AV1 and H.264.
* fix: Rename setting useL3T3Mode->useKSVC and adjust the defaults.
fix(JingleSession) Use the video type from presence.
Instead of using the videoType from source map signaled by the bridge which can be incorrect if the bridge doesn't receive the presence before it sends out the source map. It uses the default 'camera' as video type if presence is not received from the senders. Possibly fixes a bug where a screenshare source is displayed in the wrong tile after the SSRC is re-mapped.
Additional benefits are:
- ssrc -> source name will be updated on ssrc remap message from
the bridge.
- the map will be correctly cleaned when member leave (this logic was
not working well for _sourceName map)
- Looks cleaner and simpler.
fix(codec-selection) Pick only 1 VP9 codec for p2p negotiation.
More than one VP9 payload type is produced in the offer, more experimental ones if the local endpoint has no local video source. Munge the SDP to remove all codecs besides the one with profile-id=0 so that only one VP9 codec is negotiated with p2p peer.
fix(JingleSession) Modify encoding to stop/start outgoing media on p2p.
This is much faster than changing direction on the transceiver and doesn't need a renegotiation for the suspending/resuming media. Also fixes an issue with audio-only mode in p2p where media doesn't resume when audio-only mode is disabled if it was enabled while the jvb connection was active.
ref(JingleSession) Always call sRD->cA-sLD for renegotiation.
For renegotiations triggered by the application, it is ok to execute sRD->cA-sLD cycle on the p2p initiator instead of executing cO->sLD->sRD cycle. Chrome sometimes creates a third m-line for p2p when createOffer is called which is unexpected and therefore it breaks the client's renegotiation cycle. This has been done in other places already, i.e., when remote sources are signaled, etc. so making it uniform across all renegotiations.
fix(JingleSession): Reject all m-lines associated with sources in source-remove.
Fixes an issue when a screenshare doesn't show up on remote endpoints after one of the endpoints in the call does an ICE RESTART using session-terminate while it has 2 video sources.
* fix(codec-selection) Use a configurable preferred codecs list to select the codecs.
Allow asymmetric codecs to be configured on the endpoints. This means that Firefox and Safari which have bugs with VP9 encode will now encode VP8 but decode VP9 coming in from Chromium endpoints.
* fix(codec-selection) Add unit tests for the codec selection logic.
* feat(codec-selection): Introduce mobileCodecPreferenceOrder setting in config.js.
* fix(codec-selection) Avoid unnecessary renegotiations.
On participant join/leave, check if the new intersection of codecs are already configured to be the top n codecs.
* Address review comments
* fix: Strip the codecs that are not in the codec list from the video m-lines.
Also, ignore remote codecs published in presence for p2p connections.
* fix: Define default codec order for mobile and desktop.
Fix an issue where p2p between mobile and desktop was broken.
* fix: Add default codecs for both p2p and jvb on mobile devices.
Munge the initial offer sent out by RN clients since RN doesn't support RTCRtpTransceiver#setCodecPreferences.
fix(JingleSession) Reconfigure the stream encodings after p2p->jvb switch.
Fixes blurry screenshare in some cases. There is a higher probability of the higher layers getting suspended when all the stream encodings are enabled for low fps SS. Make sure only the highest spatial layer is sent for low fps SS after p2p->jvb switch. Make sure all the stream encodings are enabled for high fps SS after p2p->jvb switch.
fix(JingleSession): Check for source updates after every reneg
Check for source updates and notify the peer of any new sources after every renegotiation cycle. This fixes issues where source changes are not detected if any other task that requires a renegotiation is scheduled between replaceTrack and a negotiationneeded event.
fix(ssrc-rewriting) Update owners when a new SSRC is assigned to existing source.
Fixes a bug where the remote video freezes if the remote user does an ICE restart using session-terminate. In this case, a new SSRC is assigned to the remote source which will have the same source-name.
fix(JingleSession) start modification queue after session is established.
Pull initial offer/answer tasks out of the modification queue and execute them right away. Only track and codec related operations that necessitate a renegotiation cycle need to be pushed to the modification queue. The queue execution is paused until the session is established. This avoids track operations being executed before the session is established. This fixes an issue seen in p2p connections where sources of the initiator are not signaled to the remote since the tracks are added while the initiator is waiting for a session-accept from the peer.
fix(JingleSession) Avoid pushing encodings configuration task to the queue.
When configureSenderVideoEncodings task is pushed to the modification queue, there might not be a setParameters call in progress and this new one will not get chanined to any promise. However, when the task finally is executed, there is a chance for another setParameters method execution to be in progress and can result in setParameters call failing. Also, these extra steps to ensure that the sender encodings are correctly encoded are no longer needed since with multi-stream mode, we do not change the source video type and the previous encodings config will still be valid.
fix(codec-selection) Apply codec preferences to initial offer/answer.
This fixes an issue where p2p clients (with different codec preferences) fail to decode video because the negotiated codecs are removed from the supported codecs list after the media session is established. The codec preferences will be applied when the first offer/answer is created.
fix(TPC) Disable media instead of changing dir for p2p->jvb switch. (#2226)
* fix(TPC) Disable media instead of changing dir for p2p->jvb switch.
Resume or suspend the media on the jvb peerconnection by changing the RTCRtpEncodingParamters.active state instead of changing the direction on the transceiver. This avoids the needs to start a O/A renegotiation cycle for these operations. The media direction will be changed only for p2p lastn=0 case since video needs to be disabled on both the sender and the peer for p2p lastn=0 case.
* Address review comments
* Disable media after adding source while media is suspended on the jvb connection. Default 'active' state for stream encodings after the source is added is 'true'.
* Wait for all the promises to be settled before returning
Jaya Allamsetty
Jaya Allamsetty
Pawel Domas
Saúl Ibarra Corretgé
Auxane Thouary
Hristo Terezov
Jorge Oliveira
bgrozev