Why YUV format is used in MPEG2/MPEG4/H264encoders for compression ?

  MPEG2/MPEG4/H264 is lossy compression.
RGGB will have image details/color information in all the RGB represented bytes.
if we are applying for lossy compression, there is a chance for losing bits information.
if LSB bits are affected there wont be a big change in image. If MSB bits of RGB is lost,
this will be bigger change in image and user can perceive/can observe the changes in image.

  In YUV format, image details will be available in Y data[ this will have monochrome/ black and white image]. U and V component is used to store color information.

  Eyes can't perceive color information changes than image information. But it can easily observe the changes in image information[Y data which will have black/white image]. So In lossy compression, encoders wont disturb the image information & will
disturb or compress more in color information [UV components]

     If we are using RGB, we cannot separate image information & color information.Because all the pixels will have the image information and color information.

This is the reason why we are using YUV format in lossy compression or MPEG2/MPEG4/H264 encoders.


   

MPEG4 Decoder Problem

Somewhere in the middle the MPEG4 Decoder is failed.
if we used the MPEG4 Decoder with the MPEG4 Demux , then there will be no problem.
But if we used it with the RTP Video Source Filter, then we got the problem.
Reason:----------- This is the problem with MPEG4 Decoder.if we used any DivX free decoder filter, then it is working well.if we are starting from the middle, then the RTP source Filter will not send the Keyframe first.This is the problem
Solution:
--------------
I modified the Decoder Filter code to accept it as follows:
For the First time, if we are receiving VOS header then only allow it to display . otherwise just drop the frame.By checking the VOS Start code (00 00 01 B0) But it is not working. It seems like the ffmpeg is not sending VOS header.
So I added the code to check for the VOL start code and frame type.
if (bFirstFrame)
{
if( VOS start code or ( VOL startcode and frametype == KeyFrame))
{
bFirstFrame = false;
}
else { return S_FALSE; //Drop the Frame. }
}

How can we extract the MPEG4 Frame Type from the RTP packet ?

How can we extract the MPEG4 Frame Type from the RTP packet (I,P or B):
------------------------------------------------------------------------
We are creating a software to parse MPEG4 data. Can anyone tell me how to extract the Frame Type (I,P,B) frame type from the whole RTP packet?
I mean I have RTP packet with MPEG4 payload in it, then how to extract the Frame type information from it?
Solution :
---------------
vop_coding_type is encoded in the first 2 bits after vop_start_code.
vop_start_code is 00 00 01 b6 (hex)
vop_coding_type means:
0 0 -I VOP
0 1 -P VOP
1 0 -B VOP
1 1 -S VOP

we are sending mpeg4 video thru rtp using ffmpeg.
Every I Frame( KeyFrame) is having VOS start code. 00 00 01 B0

YUV420P to YV12 format conversion

I converted the YUV420P to YV12 format using the following :


The YV12 format is essentially the same as YUV420p, but it has the U and V data reversed: the Y values are followed by the V values, with the U values last.Our output is YUV420P.

Our MPEG4 Decoder's output is YUV420P. So we need to convert it as
YV12. I used the following code to convert from YUV420P to YV12.

long ulNumberOfPixels = m_iWidth * m_iHeight;
long ulUVBufferSize = (m_iWidth * m_iHeight) /4;
unsigned char ch; for(long i = 0; i < ulUVBufferSize; i++)
{
ch = pbYUVData[ulNumberOfPixels + i] ;
pbYUVData[ulNumberOfPixels + i] = pbYUVData[ ulNumberOfPixels + ulUVBufferSize + i];
pbYUVData[ ulNumberOfPixels + ulUVBufferSize + i] = ch;
}

MPEG4 Frame Construction( Identify the Frame Size)

I have to construct the Frames from the Bytes.
I need to develop the class for this one. and I need to test it with MPEG4 Decoder SDK in MPeg4DecoderExe;
Algorithm For constructing frame from bytes :
-----------------------------------------------
1.Pass Bytes to the algorithm with size. Add these Bytes to the custom Buffer.
2.Check For the StartCode 00 00 01 B6 and count the number of Bytes and
check for the beginning of a new frame.
3.if we found a frame, copy and send it to the MPEG4FeedData() and removed the frame bytes from the custom Buffer.
4.Call the PullData() fn...

How can we idenitfy the FrameSize in m4v file?...

How can we idenitfy the FrameSize ?.

Starts From 0x00 00 01 B6 to
Next 0x00 00 00, We can calculate the Buffer Size. That is the Frame Size.


0000 01 B0 ...
00 00 01 B6...
00 00 01 xx

if XX is B6 then there is no problem...

if XX is other than B6, we have to take it as a header.

So Next I have to develop the application to frame the buffer.

MPEG4's Maximum Frame Size

H.263 specifies maximum frame size by means of "BPPmaxKb" parameter.BPPmaxKb defines the maximum frame size in terms of 1024 bits units.The standard specifies table for BPPMaxKb, which is dependentresolution. For QCIF, value of BPPmaxKb is 64 units, which translatesinto 8 Kbytes.

Does any body know whether there is limitation for maximum H.263 framesize?
For mpeg4 simple profile QCIF the spec said:

ISO/IEC 14496-2:2003(E) \ Annex D: Video buffering verifier \ D.2 VideoRate Buffer Model Definition clause 10:

The number of bits used for coding any single VOP, di, shall not exceedk * 16384 bits, where k = 4 for QCIF and Sub-QCIF, k = 16 for CIF, k =32 for 4CIF, and k = 64 for 16CIF, unless a larger value of k isspecified in the profile and level definition.
For QCIF, k=4. So maximum encoded frame size should be 8k bytes.

How to parse MPEG4 data over the RTP stream and identify the header with SDP file ?

analysing the RTP received packets from middle of the sending and test the SDP with QuickTime and VLC player.
Qt and VLC plays the RTP stream thru SDP file .So There might be two chance for it.

1.without SDP file players can't receive RTP stream data.
Reason :
If I opened the RTP stream in a VLC player, it doesnt receive RTP stream data and gives unable to open error.

2. So based on this SDP file and we have to search for the VOL_start_code. For Each frame VOP is available. Successive VOPs can have I Frame or P Frame. for computer video, VOL_Header must have an information about Width and height

So combine the VOP information and SDP file information, it creates the header for MPEG4 Encoded data. But VOP start Code is not like that... 0000 01 B6 (vop_start_code ) it is available in more than one RTP packet consecutively.These packets are having less frame size.

VOL header is having more video Info . So it is having the header info. So we have to search for the Next VOL header.
if we started from middle of the RTP stream.

How can identify the KeyFrame from a MPEG4 Encoded data ?

how can we identify the Keyframe from MPEG4 Encoded data ?...
within MPEG4 Encoded data, they will have the configuration information u know... Every vop_start_code(00 00 01 B6) indicates a new frame.
For the First Frame Set the KeyFrame flag. Count this vop_start_code to identify the key frame.
if the vop_start_code is 30, then FrameCount % number_of_vop_start_codes == 0 then it is a keyframe.

How can identify the KeyFrame from a MPEG4 Encoded data ?

how can we identify the Keyframe from MPEG4 Encoded data ?...
within MPEG4 Encoded data, they will have the configuration information u know... Every vop_start_code(00 00 01 B6) indicates a new frame.
For the First Frame Set the KeyFrame flag. Count this vop_start_code to identify the key frame.
if the vop_start_code is 30, then FrameCount % number_of_vop_start_codes == 0 then it is a keyframe.

How can we idenify the MPEG2VIDEOINFO header's dwSequenceHeader from (received from RTP) MPEG4 Encoded buffer ?

How can we idenify the MPEG2VIDEOINFO header's dwSequenceHeader from (received from RTP) MPEG4 Encoded buffer ?

- Search for Group_of_vop_start_code (00 00 01 B3 check it without space ).Bytes Before the Group_of_vop_start_code is considered as a MPEG2VIDEOINFOHEADER's dwSequenceHeader.
From the Start of the VOS header's , we can identify the width and height of the video. otherwise we can hardcode the data with the specified configuration at the receiver side.

group_of_vop_start_code - 00 00 01 B3 -Search for 00 00 01 B3 Bytes before this code is added as a Header format;

MPEG2VIDEOINFO vidInfo;
vidInfo.dwSequenceHeader[0] = ( BYTE*) pbBytesBefore000001B3;
MPEG2VIDEOINFO vidInfo; BYTE* pbHeader = (BYTE*)vidInfo.dwSequenceHeader[0]

How can we idenify the MPEG2VIDEOINFO header's dwSequenceHeader from (received from RTP) MPEG4 Encoded buffer ?

How can we idenify the MPEG2VIDEOINFO header's dwSequenceHeader from (received from RTP) MPEG4 Encoded buffer ?

- Search for Group_of_vop_start_code (00 00 01 B3 check it without space ).Bytes Before the Group_of_vop_start_code is considered as a MPEG2VIDEOINFOHEADER's dwSequenceHeader.
From the Start of the VOS header's , we can identify the width and height of the video. otherwise we can hardcode the data with the specified configuration at the receiver side.

group_of_vop_start_code - 00 00 01 B3 -Search for 00 00 01 B3 Bytes before this code is added as a Header format;

MPEG2VIDEOINFO vidInfo;
vidInfo.dwSequenceHeader[0] = ( BYTE*) pbBytesBefore000001B3;
MPEG2VIDEOINFO vidInfo; BYTE* pbHeader = (BYTE*)vidInfo.dwSequenceHeader[0]

MPEG4 .m4v file ( it is in raw format

Transport Stream :
----------------------
we opened the .mp4 file in a GraphEdit , 1.Dumped the MPEG2VideoInfo's dwSequenceHeader in to a file.
BYTE* pbData = (BYTE*) dwSequenceHeader[0];
cbSequenceHeader is having the size of the header. 2. Dumped the Encoded data in to a file.
Now we are creating a new file by merging a header and Encoded content.and Named it as .m4v , we can play this file in a media playerotherwise name it as .bits and we can decode this file using ffmpeg as follows :
ffmpeg -vcodec mpeg4 -i "EncodedWithHdr.bits" -vcodec rawvideo EncodedWithHdr.yuv
But Program stream and Transport Stream is not always same.

we can also create the .m4v by the following command :
ffmpeg -i D:\highway.avi -f m4v D:\highway.m4v

Program Stream and TransportStream Difference

There will be no difference between Program Stream and Transport stream. Except Header...

TS have VOS,VO and VOL headers. But the PS( Program Stream) is directly having VOL

headers. Sometimes we referred it as Elementary Stream.

if the Packet is having only video or audio, we call it as Elementary Stream.
if the packet contains both audio and video, we call it as Program Stream.
if the Packet contains one or more program stream we call it as Transport Stream.
we can carry more channels over the TS.

How to get the MPEG4 encoded bitstream headers within a MPEG4 Decoder Filter

1.I opened the ( highway176x144_30fps.mp4) file in a GraphEdit.

I dumped the MPEG encoded contents in to a file using the following graph:

MP4 Source Filter -> Sample Grabber Filter ->Dump Filter ( highway176x144_30fps.bits)

if I passed the dump file to the MPEG4 decoder application,It gives an error.
( Because the Encoded content also have some Header information ... It is missing )

2.I used my Temporal decoder filter to open the same Mp4 file ( highway176x144_30fps.mp4)

MP4 Source Filter -> Temporal Decoder Filter -> Video Renderer

within my temporal Decoder, I checked the MPEG2VIDEOINFO header of an Tempora Decoder filter's Input Pin.

MPEG2VIDEOINFo header is having some values like as follows:

DWORD cbSequenceHeader; // Size of the Encoded contents Header
DWORD dwSequenceHeader[1]; // Contains header info.

we can typecast it as follows to get the buffer,

BYTE* pbData =(BYTE*) mpg2VideoInfo->dwSequenceHeader;
I dumped this Encoded data to a file.( highway176x144_30fps.mpeg4hdr)


3.I created the Sample application to merge these two files.

The new File ( highway_mpeg4.bits) must have the following :

0. Create a new file ( highway_mpeg4.bits)
i. Append the contents of highway176x144_30fps.mpeg4hdr file to the highway_mpeg4.bits
ii. Append the contents of highway176x144_30fps.bits file to the highway_mpeg4.bits

4.Next I did the following.

Using FFMpeg to decode the MPEG4 contents as follows :

ffmpeg -s 176x144 -vcodec mpeg4 -i d:\media\MpegInput\highway_mpeg4.bits -vcodec rawvideo D:\media\MpegInput\highway_mpeg4.yuv

ffmpeg successully generated the Output file "D:\media\MpegInput\highway_mpeg4.yuv".


I Opened this YUV file in YUV player. it is working fine.