Mediastreamer2 is a powerful and lightweight streaming engine specialized for voice/video telephony applications.
This open source library is responsible for all the receiving and sending of multimedia streams in Linphone, including voice/video capture, encoding and decoding, and rendering.
Mediastreamer2 in Linphone architecture
- Capture and playback from various platform dependent sound architectures (ALSA, AudioUnits, AudioQueue, WaveApi)
- Send and receive RTP streams
- Encode and decode the following audio formats: speex, G711, GSM, iLBC, AMR, AMR-WB, G722, SILK, G729
- Encode and decode the following video formats: VP8 (WebM), H263, H263-1998, MPEG4, theora and H264 (thanks to a plugin based on x264 or OpenH264), with resolutions from QCIF(176x144) to SVGA(800x600) provided that network bandwidth and cpu power are sufficient.
- Audio conferencing
- Supports SRTP and zRTP (encryption of voice and video)
- Supports any webcam with a V4L or V4L2 driver under linux and Directshow driver on windows
- Acoustic echo cancellation using the great echo canceller available in libspeexdsp (works not only with speex codec of course)
- Read and write from/to a wav file
- Read YUV pictures from a camera based on platform dependents APIs.
- Optimized rendering of YUV pictures
- Dual tones generation
- Custom tone detector
- Echo cancellation, based on speex library or webrtc AEC on android
- Audio conferencing
- Audio parametric equalizer
- Volume control, automatic gain control
- ICE for optimized NAT traversal (RFC5246) to allow peer to peer audio & video connections without media relay server
- Adaptive bit rate control algorithm: for automatic adaption of encoder bit rate based on received RTCP feedback.
- Efficient bandwidth management: the bandwidth limitations are signaled using SDP (b=AS...), resulting in audio and video session established with bitrates that fits the user's network capabilities.
- Low bandwidth mode: make audio calls over EDGE
- Sound backends:
- Linux: ALSA, OSS, PulseAudio
- Windows: waveapi
- MacOSX: HAL Audio Unit
- iPhone: VoiceProcessing AudioUnit with built-in echo cancellation
- Android sound system
- JSR135 on BlackBerry
- Can use plugins: to add new codecs, or new core functionalities, such as remote directory search of sip addresses for example.
- Google Android >= 2.2
- iOS >= 6
- Windows XP, Vista, 7 and 8
- Mac OS X
- Embedded Linux: ARM and Blackfin
- BlackBerry OS10
- Windows Phone 8
Design and principles
Each processing entity is contained within a MSFilter object. MSFilter(s) have inputs and/or outputs that can be used to connect from and to other MSFilters.
A trivial example to understand:
- MSRtpRecv is a MSFilter that receives RTP packets from the network, unpacketize them and post them on its only output.
- MSSpeexDec is a MSFilter that takes everything on its input assuming these are speex encoded packets, and decodes them and put the result on its output.
- MSFileRec is a MSFilter that takes everything on its input and write it to wav file (assuming the input is 16bit linear pcm).
MSFilters can be connected together to become filter chain. If we assemble the three above examples, we obtain a processing chain that receives RTP packet, decode them and write the uncompressed result into a wav file.
MSRtpRecv --> MSSpeexDec --> MSFileRec
The execution of the media processing work is scheduled by a MSTicker object, a thread that wakes up every 10 ms to process data in all the MSFilter chains it manages. Several MSTicker can be used simultaneously, for example one for audio filters, one for video filters, or one on each processor of the machine where it runs.
- RTP: A Transport Protocol for Real-Time Applications, RFC 3550
- 5. RTP Data Transfer Protocol
- 6.4 Sender and Receiver Reports
- 6.5 SDES: Source Description RTCP Packet
- 6.6 BYE: Goodbye RTCP Packet
- 6.7 APP: Application-Defined RTCP Packet
- RTP Profile for Audio and Video Conference with Minimal Control, RFC 3551
- Symmetric RTP / RTP Control Protocol (RTCP), RFC 4961
- Session Traversal Utilities for NAT (STUN), RFC 5389 (Basic procedures)
- Secure Real Time Transport Protocol (SRTP, RFC 3711)
- ZRTP, RFC 6189
- ICE, RFC 5245 & RFC 6336
- RTP Payload Format for the Speex Codec, RFC 5574
- Real-Time Transport Protocol (RTP) Payload Format and File Storage Format for the Adaptive Multi-Rate (AMR) and Adaptive Multi-Rate Wideband (AMR-WB) Audio Codecs
- only RTP octet-align=1 mode, without interleaving, crc, single channel
- RTP Payload for DTMF Digits, Telephony Tones, and Telephony Signals, RFC 4733
- RTP Payload Format for Named Telephone Events
- RTP Payload Format and File Storage Format for SILK Speech and Audio Codec
- XML Schema for Media Control, RFC 5168
- Sending and processing of picture fast update command in SIP INFO
- RTP Payload Format for MPEG-4 Audio/Visual Streams, RFC 3016
- 3. RTP Packetization of MPEG-4 Visual bitstream
- RTP Payload Format for ITU-T Rec. H.263 Video, RFC 4629
- RTP Payload Format for H.264 Video, RFC 3984
- RTP Payload Format for VP8 Video
Mediastreamer2 is dual licensed. It can be licensed and distributed:
Mediastreamer2 releases can be downloaded here.
You can also use git to retrieve the latest source code (recommended for developers):
git clone git://git.linphone.org/mediastreamer2.git
msilbc (iLBC plugin)
git clone git://git.linphone.org/msilbc.git
|msx264 (H264 plugin)||
git clone git://git.linphone.org/msx264.git
|msamr (AMR plugin)||
git clone git://git.linphone.org/msamr.git
|mssilk (SILK plugin)||
git clone git://git.linphone.org/mssilk.git
|msbcg729 (G729 plugin)||
git clone git://git.linphone.org/bcg729.git
Mediastreamer2 is documented using doxygen. You can browse the API documentation here.