Mediastreamer2 is a powerful and lightweight streaming engine specialized for voice/video telephony applications.
This open source library is responsible for all the receiving and sending of multimedia streams in Linphone, including voice/video capture, encoding and decoding, and rendering.
Mediastreamer2 in Linphone architecture
- Audio codecs: opus, speex, g711, g729, gsm, iLBC, AMR, AMR-WB, g722, SILK, iSAC, BV16, Codec2
- Video codecs: VP8, H264 with resolutions up to 1080P, MPEG4
- Hardware accelerated H264 codec for Mac OSX / iOS (VideoToolbox) and Android (MediaCodec)
- Send and receive RTP / RTCP streams
- Audio conferencing API
- Features an innovative jitter buffer algorithm, which quickly adapt to very jittered network conditions and improve control of audio latency
- Adaptive bit rate control algorithm: congestion control and estimation of available bandwidth, in order to optimize audio & video quality
- ICE (RFC 5245), STUN and TURN (RFC 5766) for optimized NAT traversal allowing peer to peer audio & video connections whenever it is possible
- SRTP, zRTP and SRTP-DTLS voice and video encryption
- RTP/AVPF support: RTCP control messages for video error recovery: PLI, SLI, RPSI, FIR
- Acoustic echo cancellation using echo canceler from libspeexdsp, webrtc AECm or AEC
- Capture and playback from various platform dependent sound architectures (ALSA, PulseAudio, AudioUnit, AudioQueue, WaveApi, WASAPI, Android AudioTrack/AudioRecord, Android OpenSLES)
- Supports any webcam, based on OS's camera API: quicktime, directshow, video4linux, android.camera
- Play and record from/to raw, wav, or mkv (matroska) files
- Optimized rendering of YUV pictures, using openGL, DrawDib, X11/Xv
- Dual tones generation
- Custom tone detector
- Audio parametric equalizer
- Volume control, automatic gain control
- Can use plugins: to add new codecs, new sound input/output backend...
- GNU/Linux: x86, x86-64, ARM v5 to v7, arm64 ; Debian 8/9, Centos 7
- Windows Desktop: x86 (works also on x86_64), Windows 7 and later
- Mac OS X: x86_64 ; 10.11 and later.
- GNU/Linux embedded: Linphonec or liblinphone are good candidates to provide the software stack of an hardware phone or hardware communication system.
- Apple iOS 9 to 11 (ARM v7, ARM 64)
- Google Android 4.1 to 8.1 (ARM v7-v8, x86)
- Windows 10 UWP : mobile and desktop (ARM v7)
Design and principles
Each processing entity is contained within a MSFilter object. MSFilter(s) have inputs and/or outputs that can be used to connect from and to other MSFilters.
A trivial example to understand:
- MSRtpRecv is a MSFilter that receives RTP packets from the network, unpacketize them and post them on its only output.
- MSSpeexDec is a MSFilter that takes everything on its input assuming these are speex encoded packets, and decodes them and put the result on its output.
- MSFileRec is a MSFilter that takes everything on its input and write it to wav file (assuming the input is 16bit linear pcm).
MSFilters can be connected together to become filter chain. If we assemble the three above examples, we obtain a processing chain that receives RTP packet, decode them and write the uncompressed result into a wav file.
MSRtpRecv --> MSSpeexDec --> MSFileRec
The execution of the media processing work is scheduled by a MSTicker object, a thread that wakes up every 10 ms to process data in all the MSFilter chains it manages. Several MSTicker can be used simultaneously, for example one for audio filters, one for video filters, or one on each processor of the machine where it runs.
- RTP: A Transport Protocol for Real-Time Applications, RFC 3550
- 5. RTP Data Transfer Protocol
- 6.4 Sender and Receiver Reports
- 6.5 SDES: Source Description RTCP Packet
- 6.6 BYE: Goodbye RTCP Packet
- 6.7 APP: Application-Defined RTCP Packet
- RTP Profile for Audio and Video Conference with Minimal Control, RFC 3551
- Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP/AVPF), RFC 4585
- Symmetric RTP / RTP Control Protocol (RTCP), RFC 4961
- Session Traversal Utilities for NAT (STUN), RFC 5389 (Basic procedures)
- Secure Real Time Transport Protocol (SRTP, RFC 3711)
- ZRTP, RFC 6189
- ICE, RFC 5245 & RFC 6336
- TURN, RFC 5766
- RTP Payload for Text conversation, RFC 4103
- rtcp-mux, RFC 5761
- SRTP-DTSL, RFC 5763
- RTP Payload Format for the Opus Speech and Audio Codec, RFC 7587
- RTP Payload Format for the Speex Codec, RFC 5574
- Real-Time Transport Protocol (RTP) Payload Format and File Storage Format for the Adaptive Multi-Rate (AMR) and Adaptive Multi-Rate Wideband (AMR-WB) Audio Codecs
- only RTP octet-align=1 mode, without interleaving, crc, single channel
- RTP Payload for DTMF Digits, Telephony Tones, and Telephony Signals, RFC 4733
- RTP Payload Format for Named Telephone Events
- RTP Payload Format and File Storage Format for SILK Speech and Audio Codec
- RTP Payload Format for BroadVoice Speech Codecs, RFC 4298
- XML Schema for Media Control, RFC 5168
- Sending and processing of picture fast update command in SIP INFO
- RTP Payload Format for MPEG-4 Audio/Visual Streams, RFC 3016
- 3. RTP Packetization of MPEG-4 Visual bitstream
- RTP Payload Format for ITU-T Rec. H.263 Video, RFC 4629
- RTP Payload Format for H.264 Video, RFC 3984
- RTP Payload Format for VP8 Video
Mediastreamer2 is dual licensed. It can be licensed and distributed:
Mediastreamer2 releases can be downloaded here.
You can also use git to retrieve the latest source code (recommended for developers):
git clone git://git.linphone.org/mediastreamer2.git
msilbc (iLBC plugin)
git clone git://git.linphone.org/msilbc.git
msopenh264 (H264 plugin based on openH264 codec)
git clone git://git.linphone.org/msopenh264.git
msx264 (H264 plugin based on x264 encoder)
git clone git://git.linphone.org/msx264.git
|msamr (AMR plugin)||
git clone git://git.linphone.org/msamr.git
|mssilk (SILK plugin)||
git clone git://git.linphone.org/mssilk.git
|msbcg729 (G729 plugin)||
git clone git://git.linphone.org/bcg729.git
|mswebrtc (isac codec, Acoustic Echo Canceler)||
git clone git://git.linphone.org/mswebrtc.git
Mediastreamer2 is documented using doxygen. You can browse the API documentation here.