Camera and capture options¶

A capturer produces the frames or audio of a local track. Each local track has one, in LocalVideoTrack.capturer or LocalAudioTrack.capturer:

Track	Capturer	Who starts it
`LocalParticipant.cameraTrack`	`CameraCapturer`	`setCamera(enabled:)`
`LocalParticipant.microphoneTrack`	`MicrophoneCapturer`	`setMicrophone(enabled:)`
`LocalParticipant.screenShareTrack`	The screen share capturer; see Screen sharing	`setScreenShare(enabled:)`
`LocalVideoTrack.createBufferTrack(name:source:)`	`BufferCapturer`	Your App, with `startCapture()`
`LocalAudioTrack.createBufferTrack(name:)`	`AudioBufferCapturer`	Your App, with `startCapture()`

This page covers how capture starts and stops, the camera and its options, tracks you fill with your own frames or audio, processing frames before they are sent, and the capture state and events. What is sent to each participant (codec, bitrate, frame rate) is on Publish options and encoding.

Start and stop capture¶

Call startCapture() and stopCapture() on buffer capturers only. The camera, the microphone and the screen share are switched on LocalParticipant: setCamera(enabled:), setMicrophone(enabled:) and setScreenShare(enabled:). Those switches keep the track, its publications and the hardware in step; calling the capturer's own methods on these tracks bypasses that.

LocalParticipant.isCameraActive and LocalParticipant.isMicrophoneActive change as soon as you call the switch, so a button can read them directly and quick taps are safe. The camera hardware follows a short delay after the last call and only applies the final state, both when turning on and when turning off: the device's camera indicator can stay on for that delay, but no camera frames are sent once you switch off. If the camera cannot start, for example without permission, isCameraActive goes back to false. setCamera(enabled:) and set(source: .camera, enabled:) do not report this as an error, so watch for that change.

Capture and publishing are independent: publishing a track does not start its capturer, and starting a capturer does not publish anything.

The camera¶

Get the camera's capturer from the camera track:

let camera = Room.shared.localParticipant?.cameraTrack.capturer as? CameraCapturer

The camera track and its capturer exist before the camera runs, so you can read and set options before the first setCamera(enabled: true).

Camera capture options¶

CameraCaptureOptions decides what the camera produces. The current options are in CameraCapturer.options.

Field	Default	Meaning
`dimensions`	`.h720_169` (1280×720)	The resolution you ask for. See Requested and actual size.
`fps`	30	Frames per second the camera captures. Values above 30 rarely help: when the camera delivers fewer frames, the same picture is sent again.
`position`	`.front`	Which way the camera faces: `.front`, `.back`, or `.unspecified` to let the SDK pick.
`device`	nil	A specific camera to use. When set, it wins over `position`.
`autoCorrectOrientation`	true	When true, the SDK follows the device as it rotates and turns the pixels upright before sending. When false, the output keeps the orientation it had when you turned it off.
`isMirrored`	false	Mirrors the captured picture itself, so the participant you send to, recordings and compositing all see it mirrored. To mirror only your own preview, use `VideoView.mirrorMode` instead.

尚未支援

目前還不能把相機輸出固定成指定的方向。autoCorrectOrientation 設成 false 只會停在當下的方向。

CameraCaptureOptions.Dimensions has 16:9 presets (.h90_169 to .h2160_169) and 4:3 presets (.h120_43 to .h1440_43); Dimensions(width:height:) gives any other size.

The fields cannot be changed in place. To change the options, make a copy with copyWith(...) and pass it to CameraCapturer.set(options:), while the camera runs or before it starts. A new resolution or frame rate applies without stopping the camera. set(options:) is async throws -> Bool: it returns true when the running camera now uses the new options. It returns false when the camera is stopped, where the options are only recorded and apply when it starts; when a camera change could not be added and the SDK went back to the camera you had; and when a later call replaced this one before it ran. To clear a specific camera and go back to position, pass device: .some(nil) to copyWith; a running camera switches to the one for position.

Swift

import RevueSDK

/// Ask the camera for 1080p at 15 fps. The fields of CameraCaptureOptions cannot be changed in place:
/// copy the current options with the new values and pass the copy to set(options:).
/// Returns true when the running camera now uses the new options. Returns false when the camera is off
/// (they are used when it starts), when a camera change could not be added and the old camera was put back,
/// or when a later call replaced this one.
@MainActor
func useFullHDCamera() async throws -> Bool {
    guard let camera = Room.shared.localParticipant?.cameraTrack.capturer as? CameraCapturer else { return false }
    return try await camera.set(options: camera.options.copyWith(dimensions: .h1080_169, fps: 15))
}

You can also pass options to setCamera(enabled: true, options:). It applies them the same way, but does not throw: when they cannot be applied, the camera starts with the options it had. Call set(options:) when you need to know.

Specified camera and camera in use¶

Two properties name a camera, and they can differ:

CameraCaptureOptions.device is the camera you ask for. When it is set, the camera starts with that one; when the SDK cannot open it, the camera starts with the camera for position instead. When it is nil, the SDK picks the camera for position.
CameraCapturer.device is the camera in use: the one actually capturing. It is set once the camera has started and after each successful switch, and is nil when the camera is stopped. Read it to show the camera's name, for example, even when you only set position.

CameraCapturer.captureDevices() lists the cameras you can pass as device, and CameraCapturer.canSwitchPosition() tells whether the device has both a front and a back camera.

CameraCapturer.captureSession is the AVCaptureSession the SDK captures with. Read it only: do not add inputs or outputs to it, because the SDK resets its input and format whenever it switches cameras or changes the resolution.

Switch cameras¶

Three methods switch cameras:

CameraCapturer.switchCameraPosition() switches between the front and the back camera.
CameraCapturer.set(cameraPosition:) switches to the camera facing that way. .unspecified takes the first camera the SDK finds, the front one when there is one.
CameraCapturer.set(options:) switches when the new options name another device or position.

All three are async throws -> Bool. They return once the switch has finished. The Bool tells whether the camera you asked for is now capturing:

Returns	When
`true`	The camera is running and the camera you asked for is now `CameraCapturer.device`.
`false`	The camera is stopped: the switch is only recorded and applies when the camera starts.
`false`	The camera could not be added and the SDK went back to the camera you had (see below).
`false`	A later call replaced this one before it ran. Calls made in quick succession only switch to the last one; each earlier call returns `false` and leaves `position` and `options` to the later call.

The call returns once the hardware has finished, a short delay after the last switch. Do not wait for it to update a button: position and options change as soon as you call, so show the user's choice straight away and use the returned value only to report the result.

Switching by position clears options.device, so the SDK goes back to picking the camera by position. While the camera is stopped, a switch only records the new position, which applies when the camera starts.

The three methods throw CameraCapturerError when there is no camera facing that way, or when the device passed to set(options:) while the camera runs cannot be opened, for example because it was unplugged. The error is thrown before anything changes, so position and options keep their values. CameraCapturerError has one case:

Case	When
`CameraCapturerError.deviceNotAvailable`	There is no camera facing the way you asked, or it cannot be used.

Swift

import RevueSDK

/// Switch between the front and the back camera. Returns true when the other camera is now the one
/// capturing. Returns false when the camera is off (the new direction is used when it starts), when the
/// other camera could not be added and the old one was put back, when a later switch replaced this one,
/// or when the device has no camera facing the other way (position and options keep their values).
@MainActor
func flipCamera() async -> Bool {
    guard let camera = Room.shared.localParticipant?.cameraTrack.capturer as? CameraCapturer else { return false }
    do {
        return try await camera.switchCameraPosition()
    } catch CameraCapturerError.deviceNotAvailable {
        return false  // for example: tell the user this device has no back camera
    } catch {
        return false
    }
}

Switching to a camera that exists can still fail a moment later, when the SDK adds it to the capture session, for example because another App is using it. The SDK then goes back to the camera you had: position and options return to their old values on the main thread, CameraCapturer.device stays the old camera, no error is thrown, and the method returns false.

CaptureState.error is used by screen share only. A camera that fails to switch or start does not move its capturer to error.

Requested and actual size¶

The SDK has two size types, because what you ask for and what you get can differ:

CameraCaptureOptions.Dimensions is a request: the size you want the camera to produce.
CMVideoDimensions is a measurement: the size of the frames actually captured, in VideoCapturer.dimensions and the events below.

For example, you ask for 1080p with .h1080_169, but this device's front camera goes up to 720p. The camera then sends 1280×720: options.dimensions stays 1920×1080, the capturer's VideoCapturer.dimensions reads 1280×720, and the capturer reports capturer(_:didDowngradeResolution:to:) with 1920×1080 and 1280×720. The SDK picks the closest format the camera supports and never scales the picture up.

Your own frames and audio¶

To send video or audio your App produces, such as frames from a network camera, a composited picture or a music file, create a buffer track and push buffers into its capturer:

LocalVideoTrack.createBufferTrack(name:source:) gives a video track whose capturer is a BufferCapturer.
LocalAudioTrack.createBufferTrack(name:) gives an audio track whose capturer is an AudioBufferCapturer.

This is the only way to bring your own source. Do not try to subclass VideoCapturer or implement AudioCapturer yourself: VideoCapturer has no public initializer, and a track's capturer is set by the SDK when it creates the track, so a capturer of your own cannot be attached to one.

The video track's name defaults to "buffer" and its source to .unknown; the audio track's name defaults to "audio_buffer", with source .unknown. Give the video track .camera as its source when the receiver should treat it as your camera, and give each track a name the receiver can tell apart; see Tracks. Statistics for buffer tracks are switched on with all the others by Room.reportStatistics on Room.shared; createBufferTrack has no option for them.

Start the capturer yourself¶

A buffer capturer starts idle and drops every buffer you push until you call startCapture(). Publishing the track does not start it, so a track you publish without starting it sends nothing, and no error tells you so. Call startCapture() before you push the first buffer, also when you only want a local preview, and stopCapture() when you are done. A buffer capturer released while still capturing logs an error.

Push frames¶

BufferCapturer.capture(_:) takes a CVPixelBuffer in NV12 (kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange or kCVPixelFormatType_420YpCbCr8BiPlanarFullRange) or BGRA (kCVPixelFormatType_32BGRA). The SDK converts BGRA to NV12 itself; a frame in any other format is dropped with a warning in the log. The frame is sent at the size you give it: to change the resolution, change the size of the buffers you push.

BufferCapturer.capture(_:timeStampNs:rotation:) also takes a rotation, a VideoRotation: how far to turn the frame clockwise to make it upright. The SDK turns the pixels before sending, so ._90 and ._270 swap width and height (a 1280×720 frame is sent as 720×1280). Turning happens before the processor (see Process frames), which receives the upright frame.

timeStampNs is not used for now: each frame is timed when the SDK sends it.

尚未支援

傳入的 timeStampNs 目前不使用，每一幀以 SDK 送出當下的時間為準。晚到的畫面（例如來自網路攝影機，或經過耗時處理）可能和聲音對不上。

Call BufferCapturer.capture(_:) and AudioBufferCapturer.capture(_:) from the thread or queue that produces the buffers, not from the main thread. They are the exception to the main-thread rule; see Events.

Swift

import RevueSDK
import CoreVideo
import Foundation

/// Send frames that arrive from a network camera lying on its side.
/// Start the capturer before pushing frames: until then every frame is dropped,
/// and publishing the track does not start it.
final class NetworkCameraFeed {
    let track = LocalVideoTrack.createBufferTrack(name: "ipcam", source: .camera)
    private let frameQueue = DispatchQueue(label: "ipcam.frames", qos: .userInitiated)

    func start() async throws {
        try await track.capturer?.startCapture()
    }

    /// Called by the network camera for every decoded frame.
    func didDecode(_ frame: CVPixelBuffer) {
        guard let capturer = track.capturer as? BufferCapturer else { return }
        // capture is meant to be called off the main thread, from the queue that produces the frames.
        frameQueue.async {
            // The camera is mounted on its side: turn each frame 90° clockwise to make it upright.
            capturer.capture(frame, timeStampNs: 0, rotation: ._90)
        }
    }

    func stop() async throws {
        try await track.capturer?.stopCapture()
    }
}

AudioBufferCapturer.capture(_:) takes an AVAudioPCMBuffer and plays the same role for audio: it sends the buffer to every participant you published the track to, and to the track's local audio renderers.

Process frames¶

Set LocalVideoTrack.processor to a VideoProcessor to change each frame before it is used, for example to blur the background or draw a watermark. It works on every local video track: the camera, the screen share and buffer tracks.

The SDK calls process(pixelBuffer:) once per frame on the capture thread, not on the main thread. A slow processor lowers the frame rate.
Return the frame to use, a new buffer or the same one; return nil to drop the frame.
Your local preview and every participant you send to get the processed frame, and VideoCapturer.dimensions measures it.
The placeholder sent while the camera is off or muted does not go through the processor.

Swift

import RevueSDK
import CoreImage
import CoreVideo

/// Blur every camera frame before it is shown and sent. process(pixelBuffer:) runs on the
/// capture thread once per frame, so keep it fast; return nil to drop a frame.
final class BlurProcessor: VideoProcessor {
    private let context = CIContext()

    func process(pixelBuffer: CVPixelBuffer) -> CVPixelBuffer? {
        let image = CIImage(cvPixelBuffer: pixelBuffer)
        let blurred = image.clampedToExtent().applyingGaussianBlur(sigma: 8).cropped(to: image.extent)
        context.render(blurred, to: pixelBuffer)
        return pixelBuffer
    }
}

@MainActor
func blurMyCamera(_ on: Bool) {
    Room.shared.localParticipant?.cameraTrack.processor = on ? BlurProcessor() : nil
}

Capture state and events¶

VideoCapturer.captureState and AudioCapturer.captureState are a CaptureState:

State	Meaning
`idle`	Not capturing. Every capturer starts here.
`capturing`	Capturing.
`error`	Screen share could not start, for example because the user declined. Only the screen share capturer uses it.

A video capturer tells its VideoCapturerDelegates, added to VideoCapturer.delegates, about three changes:

Event	When
`capturer(_:didUpdate:)` with a `CMVideoDimensions?`	The size of the frames changed. nil when capture stops.
`capturer(_:didUpdate:)` with a `CaptureState`	The capture state changed.
`capturer(_:didDowngradeResolution:to:)`	The camera cannot produce the size you asked for and uses a smaller one.

The size event fires only when the size changes. The downgrade event fires every time the camera picks a format it has to settle for (starting, switching cameras, new options), even when the size it sends does not change, for example when you raise the request from 1080p to 4K on a camera that already sends its 1080p maximum.

Like every delegate in RevueSDK, VideoCapturerDelegate is called on the main thread, even when the change comes from a capture thread or from a buffer you pushed on a background queue, so you can update your UI directly. Each event carries the value at the moment of the change. The delegates are held weakly.

Swift

import RevueSDK
import CoreMedia

/// Show the size the camera actually sends, and tell the user when the camera cannot reach the
/// size they asked for. Both events arrive on the main thread.
final class CameraSizeObserver: VideoCapturerDelegate {
    var onSize: ((CMVideoDimensions?) -> Void)?
    var onDowngrade: ((CMVideoDimensions, CMVideoDimensions) -> Void)?

    @MainActor
    func observe() {
        Room.shared.localParticipant?.cameraTrack.capturer?.delegates.add(delegate: self)
    }

    func capturer(_ capturer: VideoCapturer, didUpdate dimensions: CMVideoDimensions?) {
        onSize?(dimensions)  // nil once the camera stops
    }

    func capturer(_ capturer: VideoCapturer,
                  didDowngradeResolution requested: CMVideoDimensions,
                  to actual: CMVideoDimensions) {
        onDowngrade?(requested, actual)
    }
}