obs-webrtc: Add Simulcast Support

[Sean-Der]

Description

This PR adds supports for Simulcast to the obs-webrtc output. Simulcast allows for multiple quality levels to be sent over one track in WebRTC.

RFC 8853 is a technical description. For a less technical explanation Wowza, Dolby and LiveKit have all written articles on it.

This PR starts with adding a Spinbox when WHIP is selected. This allows the user to choose how many layers to send in total. The acceptable range (for now) is 1-4.

The Height/Width/Bitrate is scaled from the users global choice. So depending on the users choice.

• 1 Layers (
  • 100% Height/Width/Bitrate
• 2 Layers
  • 100% Height/Width/Bitrate,
  • 50% Height/Width/Bitrate
• 3 Layers
  • 100% Height/Width/Bitrate
  • 66% Height/Width/Bitrate
  • 33% Height/Width/Bitrate
• 4 Layers
  • 100% Height/Width/Bitrate
  • 75% Height/Width/Bitrate
  • 50% Height/Width/Bitrate
  • 25% Height/Width/Bitrate

A server may wish to only accept a subset of the layers offered. Today if the user rejects any layers we throw an error and say how many layers were accepted. The user can lower the amount of layers they are sending and try again. Services in their documentation will be expected to tell users how they want video encoded and how many layers they are willing to accept.

Motivation and Context

• Low Latency - Decoding/Re-encoding adds too much latency for interactive content
• Make self hosting easier - Running ffmpeg is costly and difficult to scale
• Streamers more control - If streamers can generate transcodes they control the quality
• E2E Encryption/Prevent Modification- Streamers can distribute keys to viewers and servers can't modify video anymore.

Some questions that have been asked by reviewers already.

What about Enhanced Broadcasting? - Simulcast is part of an existing standard that already sees a billion minutes per day so it is here to stay and is already deployed massively. I would like to see Enhanced Broadcasting in the JSON blob be able to specify the protocol. When a user starts a enhanced broadcasting session it should be able to switch between RTMP/WebRTC/$x. I think both flows can/should exist because they serve different purposes.

UI/UX Considerations - A single opt-in is non-intrusive and respects the user. It would be good to get feedback from all OBS WebRTC users (Dolby, LiveKit, Cloudflare etc...) that this is a sound decision though.

How Has This Been Tested?

This has been tested with the nvenc, qsv, and x264.

I have tested this against Twitch, Broadcast Box, LiveKit and Dolby.

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[tytan652]

Hi @tytan652 I see you gave a 👎 was that an accident?

No, I used that to avoid always adding an "off-topic" to PRs saying that the PR adds code complexity to the streaming codepath which adds more work to my plans for a Service Overhaul.

Do you have any feedback/things I can improve?

No, this is why I just reacted.

[czoli1976]

About time we get Simulcast for WHIP, holy cow.

[koolscooby]

This would be great. We'd like to have Twitch Stream Together users be able to use OBS and use Simulcast to send multiple qualities to improve the participant experience. It's a much better quality encoding & transmission experience compared to in-browser content contribution.

One UI nit / question: does checking the box force simulcast, or does it just enable it if the server/SDP exchange also indicates support for simulcast? If the latter, I'd recommend changing the UI label to Enable simulcast which makes it a bit clear that it is enabled on the client, but also requires the server to support it.

[koolscooby]

@koolscooby Fixed! Changed from Use -> Enabled

If the server rejects Simulcast it will only accept the high layer.

In the future the server should be able to drive more (how many layers, suggestions around bitrate/resolution). For a V1 I think it just being .5x and .25x is a good trade-off on complexity vs value.

This seems like a completely sensible tradeoff to start getting users testing the underlying simulcast functionality in OBS & on the server side in a bigger set of network scenarios.

edit: ... without creating an enormous amount UI code or complicated UX.

[murillo128]

Anything we can do to get this merged sooner (review/test/etc..)? From Dolby Millicast side we are really interested in this feature which would allow us to finally deprecate our custom fork.

[Fenrirthviti]

We're aware this is here, and will get to it in due time. More testing/documentation is always welcome to help things.

[davidzhao]

There are a lot of LiveKit users that are excited about the ability to simulcast with OBS. Let me know if we can help in getting this across the finish line

[Fenrirthviti]

Logging for posterity that this has an open RFC for OBS with unresolved feedback that has not been accepted.

While not impossible this is merged first, it's a little strange to submit a functional PR of a design that has not been accepted by the project or finalized yet. I appreciate the enthusiasm but "move fast, fix later" is at odds with how the project operates in general.

This is not a request for discussion on this point, I am posting for visibility of process.

[nils-ohlmeier]

We as Cloudflare are in support of adding Simulcast support to OBS. Having Simulcast support in OBS will enable new use cases for certain users and services.
I think the default values chosen in this PR for 1/2 and 1/4 resolution are used by a lot of video conferencing services in their Simulcast settings and make perfect sense as the initial values for this feature.

[kwindla]

I'd like to weigh in to say that we (Daily.co) have users who need this. In general, our experience is that simulcast is critical for expanding the use cases and kinds of users who can stream successfully. Two examples: 1) users who don't have control over their upstream bandwidth. Perhaps they are on a corporate network with unpredictable congestion. 2) users who want to live stream and send the stream out to multiple different protocols/channels in real-time without re-processing.

[Sean-Der]

I have had conversations with OBS Maintainers, WHIP Services and the WHIP spec author about Simulcast and have updated the PR from that feedback. We want to get an MVP into OBS Studio so we can build towards the next/future items and would appreciate another round of review now that I've refactored the implementation to follow the MVP.

(MVP) - Users are given a Spinbox to select how many Simulcast layers to send. If the remote server doesn't accept the configuration throw an error modal. A screenshot of this is in the PR description.

(Next) - Explore options to allow servers to configure/hint layers even before the user hits 'Start Streaming', but would not dynamically change while connected. Users wouldn't need to configure anything. The behavior would feel closer to Twitch's Enhanced Broadcasting.

(Future) Implement fully dynamic simulcast. OBS could enable/disable layers on the fly depending on CPU/GPU/Network usage, similar to how browsers implement WebRTC Simulcast.

[Bleuzen]

In this current form this is only really compatible with CBR mode (but the UI doesn't make that clear).
For example I can set the encoder to VBR mode and it only changes resolution and bitrate for each layer but the max bitrate stays the same for all layers which doesn't make sense.

So for this first version I suggest to either only allow users to enable Simulcast in CBR mode or add some special handling for other rate control modes (at least VBR makes sense because in my experience some browsers don't like the bitstream padding from CBR so VBR is more stable for WebRTC streams.)

For the future it would be cool to have entirely separate settings per layer. I'm not a fan of the calculated resolutions & bitrates here and would rather choose my own and also do things like having one layer with AV1 and another layer with H264. But that is something for the future / another PR I guess.

[Sean-Der]

@Bleuzen those are all great enhancements. In the future I would like to see

• Dynamic toggling of tracks (don’t generate if no one is viewing)
• Multi codec
• User selection of resolution/bitrate

It’s overwhelmingly complicated though. I hope we can keep this PR as is and just get it merged. If people find it useful I would love to do all this :)

[Sean-Der]

Thank you so much @Warchamp7 I am so grateful for the review :) This has been the thing that I have thought 'This is what makes Broadcast Box really cool'.

[peerasan]

From OBS's logs, I noticed some values might not be correct

• bitrate of whip_simulcast_2 (480x852), whip_simulcast_1 (240x426) is 6000 Kbps?
• keyint: 250

12:32:54.485: ---------------------------------
12:32:54.485: [x264 encoder: 'advanced_video_stream'] preset: veryfast
12:32:54.485: [x264 encoder: 'advanced_video_stream'] profile: baseline
12:32:54.485: [x264 encoder: 'advanced_video_stream'] settings:
12:32:54.485: 	rate_control: CBR
12:32:54.485: 	bitrate:      2000
12:32:54.485: 	buffer size:  2000
12:32:54.485: 	crf:          23
12:32:54.485: 	fps_num:      25
12:32:54.485: 	fps_den:      1
12:32:54.485: 	width:        720
12:32:54.485: 	height:       1280
12:32:54.485: 	keyint:       50
12:32:54.485: 
12:32:54.485: [x264 encoder: 'advanced_video_stream'] custom settings: 
12:32:54.485: 	bframes = 0
12:32:54.491: NV12 texture support enabled
12:32:54.491: P010 texture support not available
12:32:54.492: ---------------------------------
12:32:54.492: [x264 encoder: 'whip_simulcast_2'] preset: veryfast
12:32:54.492: [x264 encoder: 'whip_simulcast_2'] settings:
12:32:54.492: 	rate_control: CBR
12:32:54.492: 	bitrate:      6000
12:32:54.492: 	buffer size:  6000
12:32:54.492: 	crf:          23
12:32:54.492: 	fps_num:      25
12:32:54.492: 	fps_den:      1
12:32:54.492: 	width:        480
12:32:54.492: 	height:       852
12:32:54.492: 	keyint:       250
12:32:54.492: 
12:32:54.494: NV12 texture support enabled
12:32:54.494: P010 texture support not available
12:32:54.494: ---------------------------------
12:32:54.494: [x264 encoder: 'whip_simulcast_1'] preset: veryfast
12:32:54.494: [x264 encoder: 'whip_simulcast_1'] settings:
12:32:54.494: 	rate_control: CBR
12:32:54.494: 	bitrate:      6000
12:32:54.494: 	buffer size:  6000
12:32:54.494: 	crf:          23
12:32:54.494: 	fps_num:      25
12:32:54.494: 	fps_den:      1
12:32:54.494: 	width:        240
12:32:54.494: 	height:       426
12:32:54.494: 	keyint:       250
12:32:54.494: 
12:32:54.496: ---------------------------------
```

[Sean-Der]

@peerasan When I do Simple

info: [x264 encoder: 'simple_video_stream'] preset: veryfast
info: [x264 encoder: 'simple_video_stream'] settings:
	rate_control: CBR
	bitrate:      6000
	buffer size:  6000
	crf:          23
	fps_num:      30
	fps_den:      1
	width:        1280
	height:       720
	keyint:       250

info: NV12 texture support enabled
info: P010 texture support not available
info: ---------------------------------
info: [x264 encoder: 'whip_simulcast_3'] preset: veryfast
info: [x264 encoder: 'whip_simulcast_3'] settings:
	rate_control: CBR
	bitrate:      4500
	buffer size:  4500
	crf:          23
	fps_num:      30
	fps_den:      1
	width:        960
	height:       540
	keyint:       250

info: NV12 texture support enabled
info: P010 texture support not available
info: ---------------------------------
info: [x264 encoder: 'whip_simulcast_2'] preset: veryfast
info: [x264 encoder: 'whip_simulcast_2'] settings:
	rate_control: CBR
	bitrate:      3000
	buffer size:  3000
	crf:          23
	fps_num:      30
	fps_den:      1
	width:        640
	height:       360
	keyint:       250

info: NV12 texture support enabled
info: P010 texture support not available
info: ---------------------------------
info: [x264 encoder: 'whip_simulcast_1'] preset: veryfast
info: [x264 encoder: 'whip_simulcast_1'] settings:
	rate_control: CBR
	bitrate:      1500
	buffer size:  1500
	crf:          23
	fps_num:      30
	fps_den:      1
	width:        320
	height:       180
	keyint:       250

When I do advanced

info: [x264 encoder: 'advanced_video_stream'] preset: veryfast
info: [x264 encoder: 'advanced_video_stream'] settings:
	rate_control: CBR
	bitrate:      6000
	buffer size:  6000
	crf:          23
	fps_num:      30
	fps_den:      1
	width:        1280
	height:       720
	keyint:       250

info: NV12 texture support enabled
info: P010 texture support not available
info: ---------------------------------
info: [x264 encoder: 'whip_simulcast_3'] preset: veryfast
info: [x264 encoder: 'whip_simulcast_3'] settings:
	rate_control: CBR
	bitrate:      4500
	buffer size:  4500
	crf:          23
	fps_num:      30
	fps_den:      1
	width:        960
	height:       540
	keyint:       250

info: NV12 texture support enabled
info: P010 texture support not available
info: ---------------------------------
info: [x264 encoder: 'whip_simulcast_2'] preset: veryfast
info: [x264 encoder: 'whip_simulcast_2'] settings:
	rate_control: CBR
	bitrate:      3000
	buffer size:  3000
	crf:          23
	fps_num:      30
	fps_den:      1
	width:        640
	height:       360
	keyint:       250

info: NV12 texture support enabled
info: P010 texture support not available
info: ---------------------------------
info: [x264 encoder: 'whip_simulcast_1'] preset: veryfast
info: [x264 encoder: 'whip_simulcast_1'] settings:
	rate_control: CBR
	bitrate:      1500
	buffer size:  1500
	crf:          23
	fps_num:      30
	fps_den:      1
	width:        320
	height:       180
	keyint:       250

info: ---------------------------------
info: [FFmpeg libopus encoder: 'adv_stream_audio'] bitrate: 160, channels: 2, channel_layout: stereo, track: 1

debug: [obs-webrtc] [whip_output: 'adv_stream'] WHIP Resource URL is: https://b.siobud.com/api/whip
info: [obs-webrtc] [whip_output: 'adv_stream'] PeerConnection state is now: Connecting
debug: Maximum interleaver batch size for 'adv_stream' calculated to be 11 packets
info: ==== Streaming Start ===============================================
info: [obs-webrtc] [whip_output: 'adv_stream'] PeerConnection state is now: Connected
info: [obs-webrtc] [whip_output: 'adv_stream'] Connect time: 174ms

[Sean-Der]

@peerasan Can you backup your OBS settings, and then try from a fresh config? Any other hints/ideas on reproducing.

[peerasan]

@Sean-Der It's work! Thank you.

08:55:20.654: ---------------------------------
08:55:20.654: [x264 encoder: 'advanced_video_stream'] preset: veryfast
08:55:20.655: [x264 encoder: 'advanced_video_stream'] profile: baseline
08:55:20.655: [x264 encoder: 'advanced_video_stream'] settings:
08:55:20.655: 	rate_control: CBR
08:55:20.655: 	bitrate:      2000
08:55:20.655: 	buffer size:  2000
08:55:20.655: 	crf:          23
08:55:20.655: 	fps_num:      30
08:55:20.655: 	fps_den:      1
08:55:20.655: 	width:        1280
08:55:20.655: 	height:       720
08:55:20.655: 	keyint:       60
08:55:20.655: 
08:55:20.655: [x264 encoder: 'advanced_video_stream'] custom settings: 
08:55:20.655: 	bframes = 0
08:55:20.713: NV12 texture support enabled
08:55:20.713: P010 texture support not available
08:55:20.714: ---------------------------------
08:55:20.714: [x264 encoder: 'whip_simulcast_3'] preset: veryfast
08:55:20.714: [x264 encoder: 'whip_simulcast_3'] profile: baseline
08:55:20.714: [x264 encoder: 'whip_simulcast_3'] settings:
08:55:20.714: 	rate_control: CBR
08:55:20.714: 	bitrate:      1500
08:55:20.714: 	buffer size:  1500
08:55:20.714: 	crf:          23
08:55:20.714: 	fps_num:      30
08:55:20.714: 	fps_den:      1
08:55:20.714: 	width:        960
08:55:20.714: 	height:       540
08:55:20.714: 	keyint:       60
08:55:20.714: 
08:55:20.714: [x264 encoder: 'whip_simulcast_3'] custom settings: 
08:55:20.714: 	bframes = 0
08:55:20.721: NV12 texture support enabled
08:55:20.721: P010 texture support not available
08:55:20.722: ---------------------------------
08:55:20.722: [x264 encoder: 'whip_simulcast_2'] preset: veryfast
08:55:20.722: [x264 encoder: 'whip_simulcast_2'] profile: baseline
08:55:20.722: [x264 encoder: 'whip_simulcast_2'] settings:
08:55:20.722: 	rate_control: CBR
08:55:20.722: 	bitrate:      1000
08:55:20.722: 	buffer size:  1000
08:55:20.722: 	crf:          23
08:55:20.722: 	fps_num:      30
08:55:20.722: 	fps_den:      1
08:55:20.722: 	width:        640
08:55:20.722: 	height:       360
08:55:20.722: 	keyint:       60
08:55:20.722: 
08:55:20.722: [x264 encoder: 'whip_simulcast_2'] custom settings: 
08:55:20.722: 	bframes = 0
08:55:20.727: NV12 texture support enabled
08:55:20.727: P010 texture support not available
08:55:20.728: ---------------------------------
08:55:20.728: [x264 encoder: 'whip_simulcast_1'] preset: veryfast
08:55:20.728: [x264 encoder: 'whip_simulcast_1'] profile: baseline
08:55:20.728: [x264 encoder: 'whip_simulcast_1'] settings:
08:55:20.728: 	rate_control: CBR
08:55:20.728: 	bitrate:      500
08:55:20.728: 	buffer size:  500
08:55:20.728: 	crf:          23
08:55:20.728: 	fps_num:      30
08:55:20.728: 	fps_den:      1
08:55:20.728: 	width:        320
08:55:20.728: 	height:       180
08:55:20.728: 	keyint:       60
08:55:20.728: 
08:55:20.728: [x264 encoder: 'whip_simulcast_1'] custom settings: 
08:55:20.728: 	bframes = 0
08:55:20.733: ---------------------------------

might something conflict with exist OBS's config file. my best guess is basic.ini

[Stream1]
IgnoreRecommended=false
MultitrackVideoMaximumAggregateBitrateAuto=true
MultitrackVideoMaximumVideoTracksAuto=true
WHIPSimulcastTotalLayers=4

compare with backup config

[Stream1]
IgnoreRecommended=false
MultitrackVideoMaximumAggregateBitrateAuto=true
MultitrackVideoMaximumVideoTracksAuto=true
WHIPSimulcastTotalLayers=3
MultitrackVideoMaximumAggregateBitrate=8000
MultitrackVideoMaximumVideoTracks=0
MultitrackVideoConfigOverrideEnabled=false
MultitrackVideoStreamDumpEnabled=true

[RytoEX]

Just a few items.

[Sean-Der]

Thank you so much @RytoEX I addressed all the comments. Really appreciate you taking the time to look at this.

Also rebased.

[Sean-Der]

Seems like flatpak is failing for every PR, not WHIP specific!

[Fenrirthviti]

Yeah, the Flatpak failure is a change of the available disk space on the GitHub runners, so it has a high chance of failing due to disk space. Don't worry about that for sure, we're looking in to how to fix it.

[RytoEX]

Looks okay at a glance.

[nab-os]

Looks good to me

[kmansoft]

@Sean-Der what layer names (rids) do you use?

[kvasilye]

@Sean-Der one more question - does this PR send rid attributes per RFC 8851 - in particular we (IVS) are interested in max-br.

[Sean-Der]

The RIDs are index 0 https://github.com/obsproject/obs-studio/blob/master/plugins/obs-webrtc/whip-output.cpp#L73

It doesn't support max-br yet. I could add that though! At signaling time throw an error if the configured bitrate on OBS side exceeds what server supports.