TI DirectShow Filters

DirectShow Filters for TI multimedia stack

Link to latest version of this article: TI_DirectShow_Filters

Overview

DirectShow is a multimedia framework produced by Microsoft for Windows applications to perform various operations with media files or streams. In particular, the DirectShow framework is used in the WinCE environment to render audio and video. Similar to other multimedia frameworks such as Gstreamer, DirectShow uses a modular architecture, in which the application hooks up a number of software components called filters. Each filter would implement one particular stage in the processing of the data, and can have one or more input and output connections, called 'pins'. Buffers exchanged between input and output pins are called media samples. Not only do media samples hold the data to be processed, but they also contain meta-data such as timestamps. In a given DirectShow-based application (e.g. CEPlayer), a filter graph is constructed using a group of filters by making pin connections, and the resulting filter graph is run to perform data i/o and intermediate processing.

For more info on DirectShow, please refer to MSDN. An overview can be found at http://msdn.microsoft.com/en-us/library/ms778824(VS.85).aspx

When rendering a file in the WinCE Media Player (also known as CEPlayer), DirectShow's filter graph manager goes through a filter selection and connection process based on the type of media being played. WinCE comes with a set of default filters, including a file source filter, an AVI demux/splitter filter and a video rendering filter (aka the video renderer). It also comes with a default set of decode filters for MPEG1, MPEG2, MP3, and WMV. However, these decode filters do the decoding on the ARM processor. On TI OMAP and Davinci platforms, there is typically a DSP processor that can be used for accelerating decode operations. So it would make sense to supply filters that would perform the decoding on the DSP instead to free up the ARM application processor, and to provide support for unsupported codecs such as H264.

The purpose of this page is to hold information related to the DirectShow filters provided by TI to leverage some of the codecs available on the DSP, so that the latter can help offload the system.

Decode System Architecture

The CEPlayer is an application provided by Windows CE out-of-box for video and audio playback. The player calls into the DirectShow framework in order to perform the video and audio processing, and in particular the decoding is handled within the decode filter, as illustrated in Fig. 1:

Fig. 1 System Overview

TI currently provides several decode filters, one for each type of codec supported (note that even though Fig.1 only shows video decode filters, audio decode filters have a similar story). Each of these filters consists of two parts, compiled into separate DLLs. The first part, written in C++, is codec-specific and implements all of the functions that are expected to be exposed by a DirectShow filter. It defines the connection process to ensure the filter only connects to appropriate filters at its input and output (or you can think of input as being "upstream" and output as being "downstream" in data streaming terminology). It also specifies the corresponding names of the engine and codec to be opened on the DSP server. The second part is a C library called TIMM that invokes DMAI and Codec Engine to perform the processing, and is common to all filters. It acts as a wrapper around Codec Engine software stack and only needs to be compiled once for a given codec server. Some of the benefits of having such a shared DLL are:

• Reducing code size: DMAI and Codec Engine APIs are linked only once into TIMM, and the code is shared across all filters.
• Minimizing the use of a hybrid build process: The TIMM library requires a hybrid build process that uses gmake, configuro (utility provided by XDCTOOLS to use TI software) and the WinCE build system. This is somewhat complex and therefore it'd be desirable to minimize the user's exposure to this build system if possible. By having the current architecture, only the TIMM library needs to be build using this hybrid build process. The filter DLLs (i.e. the part that implements all the basic functions that are needed in a DirectShow filter) have no dependency on the binaries in Codec Engine, DMAI, etc.. Hence they can be compiled using the standard build flow in WinCE. If someone were to add a filter for a codec that is already included as part of the standard DSP codec server shipped in the DVSDK, they can write it and simply compile it using the standard WinCE build flow.
• Having an API that is more geared towards the use in a DirectShow filter: DMAI and Codec Engine are designed to be flexible and exposes many features such as creation of contiguous buffers, display/capture driver control and color space conversion just to name a few. On the other hand, TIMM exposes just the bare minimum in terms of supporting the DirectShow filters supplied by TI. So it should in theory require less of a learning curve.
• Facilitating the creation of new filters to support new codecs: When creating a filter for another video codec for example, one can reuse the application code in the TIMM library, saving development time.

Access to the DSP is enabled by TI's Codec Engine framework. The decode filter uses DMAI/Codec Engine API to control codecs compiled into the DSP codec server (ie. the DSP executable). Interprocessor communication is handled by DSPLINK underneath the hood of Codec Engine.

Details on Decode Filter Architecture

When rendering video, normally there are three main threads created by DirectShow when playing a video:

• The thread that runs the source filter and pushes the data through the AVI splitter;
• The video streaming thread, which takes data from the 'video' output pin of the AVI splitter and runs the video decode filter and renderer;
• The audio streaming thread. which takes data from the 'audio' output pin of the AVI splitter and runs the audio decode filter and renderer.

In addition, there will typically be a filter graph manager thread which oversees control of the filter graph and the application's thread.

DirectShow filters are COM-based software components that have any number of inputs and outputs, called pins. In TI's implementation of the decode filters, all filters are transform filters (i.e. inherit from the class CTransformFilter), with single input and output pins. The input pin only accepts media types that are relevant to the decoder it encapsulates. More information on the structure of a transform filter can be found at http://msdn.microsoft.com/en-us/library/aa921464.aspx

Version 1.0

This version of the filters contain only video filters. Therefore, we would focus mainly on the filter graph manager thread and the video streaming thread in this discussion.

Filter graph creation (File open)

When a video clip is opened in the WinCE Media Player (ceplayer.exe), the latter calls into the filter graph manager to create a graph of filters to render the given clip. The filter graph manager goes through a filter selection and connection process that is detailed on MSDN. The main steps are shown in Fig. 2.

Fig. 2 Connection process

This filter connection process is standard in the world of DirectShow filters, except for the opening of Codec Engine and the codec instance in step 6, which is TI-specific. For generic information regarding the connection process, refer to steps 3 and 4 in MSDN's documentation on "Writing Transform Filters" (http://msdn.microsoft.com/en-us/library/ms788165%28VS.85%29.aspx). One fine point is that over the course of the filter graph building process, the filter may potentially be instantiated multiple times. Given a source, the filter graph manager tries to connect it with each one of the filters available in the system until it finds one that can connect to it, and iterates through this process til it forms a complete graph. Each time it'd attempt to instantiate the filter, query for its supported media types, and if it fails to connect it'd destroy the instance. Hence it is always better to do as little as possible in the constructor of the filter, and leave the heavier tasks such as opening up the Codec Engine till late in the connection process, once we are almost certain that the connection will be established. In TI's decode filters, we have chosen to do it during the 'DecideBufferSize' stage, during which media types have already been agreed upon and the filter is simply trying to finalize the buffer count and buffer size for the output media samples that would be passed downstream. Typically after the DecideBufferSize function call succeeds, the connection of the filter is complete, and the filter graph is finalized by the filter graph manager.

A typical graph constructed by the filter graph manager for video playback looks like the one shown in the upper part of Fig. 1. This is the graph we will focus on from now on in our discussion. The source filter is hooked up to a demux filter (e.g. AVI Splitter), which splits the data stream into its audio and video components. The video stream is sent down to the decode filter, which output is passed to the Video Renderer. Similarly, the audio stream is decoded in a filter and rendered at the Audio Renderer. At the time of writing, only video is accelerated on the DSP.

Based on the final graph, one or more streaming threads are then created by the main thread to handle video and/or audio processing.

Data Flow in video streaming thread

The data flow in the filter while the graph is in running state is shown in Fig. 3.

Fig. 3 Receiving data in video thread

1. StartStreaming() is called by DirectShow when a filter enters the Running state. This function technically runs in the filter graph manager thread, though it is synchronized to run prior to the processing in the video thread. We leverage this function to find out the stride of the DirectDraw surfaces used by the Video Renderer. Note that we need to do this because in some cases the stride does not necessarily match up with the width of an output image (e.g. when the VRFB is enabled on the OMAP3530).
2. In the video thread, the Receive function is called on each input sample. For each sample, the filter copies the input data into a physically contiguous buffer, and passes it to the DSP, which does the processing and may return one or more display frames back to the ARM as a result. This is done via the function TIMM_algExecViddec2.
3. The filter records the meta data such as timestamps from the input sample.
4. If there are display frames returned from the DSP, the filter would allocate an output sample from the Video Renderer
5. The filter then transfers the display frame data into the sample. During the transfer of the display data, color space conversion may be necessary depending on the media type requested by the video renderer. What media type is requested by the renderer depends on how it decides to render the video, as we'll explain below.
6. The filter binds the output sample to the corresponding meta data.
7. After the output sample is fully prepared, it is finally delivered downstream.
8. The output sample is released from the decode filter. If more than one display frame comes back from the codec, the filter would repeat the whole process of allocating an output sample, filling it, binding it and delivering it for each one of the frames.

To clarify the flow in terms of buffer management, Fig. 4 summarizes how a typical video buffer moves through the system:

Fig. 4 Buffer management

As shown, there are a total of 2 buffer copies performed as part of the decode filter, if we put aside the decode operation itself.

Given memory copies are expensive, one would ask why we need to do the data copies in steps 2 and 5 from Fig. 3. Why not simply pass the buffer obtained from the input or output media sample directly to the DSP? As it turns out, the copy of the input data (step 2) is a result of the fact that the filter upstream may choose to use its own 'allocator'. An allocator is used for allocating the media samples between two filters. The input pin of the decode filter can propose an allocator to the output pin of the filter upstream, possibly one that allocates physically contiguous buffers. However, the output pin of the filter upstream always has the final say as to which allocator to use. As a result, the decode filter cannot force the filter upstream to give it samples that reside in physically contiguous memory.

Copy of the output data in step 5 is needed for a different reason. In this case, the decode filter is the upstream filter and can decide which allocator to use. However, the constraint comes now from the operation of the Video Renderer. The Video Renderer normally renders video using DirectDraw, which provides the application with direct access to the video display buffers. Meanwhile, TI decoders generally outputs display data in YUV format. Since DirectDraw supports YUV data, filling out the buffer in the output media sample is as simple as making a direct copy from the decoded buffer from the DSP -- a copy that we could have possibly avoided. However, under certain conditions (e.g. when the video window is shrunk or when it is hidden by another window), the Video Renderer will revert to using GDI for rendering the video. Because GDI does not support YUV color spaces, the Video Renderer filter would 'dynamically' request a change in the media type on the output pin to RGB. As a result, a YUV to RGB conversion would need to be performed by the filter on the decoded data from the DSP. It is customary for users to observe higher CPU utilization under these conditions. Now if the DSP were to directly use the buffers allocated from the output pin, the filter would then need to do the color conversion plus an extra copy to bring the converted data back into the output sample, given the color conversion is not performed in-place. Therefore, we have chosen to do the extra copy under DirectDraw operation to avoid further penalizing the already slower use-case. Taking from the rich to give to the poor, so to speak.

End of stream

The EndOfStream() function is called by the video streaming thread when the last frame in the video clip is reached. It flushes the DSP codec, and delivers all flushed display frames downstream. In order to get ready for a possible repeat of the same clip, it needs to close the flushed codec and re-create a new instance of it. Finally it delivers the end of stream notification downstream.

Filter graph destruction (File close)

When a file is closed, the destructor of the filter is called by the filter graph manager. It performs clean-up by closing the codec instance and exits Codec Engine itself.

Version 1.10

This version of the filters contain both video decode filters and audio decode filters. In this section we will discuss both of them.

Filter graph creation (File open)

For both video and audio filters, the filter graph creation process is essentially the same as in the previous version. Refer to the [|discussion] in version 1.0. The only difference is in DecideBufferSize(). We now spawn a worker thread of a priority level that is one less than the main video thread (i.e. higher priority) that would be later used during processing to decouple display from the input. The priority is chosen so that we minimize any accumulation of data between the two threads.

Data Flow in video streaming threads

The data flow in the filter while the graph is in running state is shown in Fig. 5.

Fig. 5 Receiving data using a worker thread

1. StartStreaming() is called by DirectShow when a filter enters the Running state. This function technically runs in the filter graph manager thread, though it is synchronized to run prior to the processing in the video thread. We leverage this function to find out the stride of the DirectDraw surfaces used by the Video Renderer. Note that we need to do this because in some cases the stride does not necessarily match up with the width of an output image (e.g. when the VRFB is enabled on the OMAP3530).
2. In the video thread, the Receive function is called on each input sample. For each sample, the filter copies the input data into a physically contiguous buffer, and passes it to the DSP, which does the processing and may return one or more display frames back to the ARM as a result. The display frames are then queued up for the worker thread.
3. The filter records the meta data such as timestamps from the input sample, and queue the meta data entry up for the worker thread.
4. The worker thread waits for a meta data entry from the main video thread.
5. The worker thread gets an output sample and fill it with meta data from the received meta data entry.
6. If the frame arrived too late or if a seek operation happened, the filter may need to discard the next decoded display frame. Otherwise, it transfers the display frame data into the sample. During the transfer of the display data, color space conversion may be necessary depending on the media type requested by the video renderer. What media type is requested by the renderer depends on how it decides to render the video, as we have explained in the architecture discussion in version 1.0.
7. After the output sample is fully prepared, it is finally delivered downstream.
8. The output sample is released from the decode filter. The worker thread then goes back to wait for the next meta data entry from the main video thread.

To deal with occasional spikes in processing time, buffers have been pre-allocated between the main video thread and the worker thread for both display frames and meta data entries. The maximum number of display frames that can be queued at one time between the two threads is NUMBUFS + the maximum number of buffers a codec may hold internally at once. The latter is dependent on the video codec itself, as there are cases when the decoder may decide to keep display frames internally and to only return them for display later on in time. Meanwhile, NUMBUFS is set in <dshow_install_dir>\timm\timm_viddec2.c, and represents the number of buffers used for smoothing video processing spikes. Similarly, the maximum number of meta entries that can be queued up is set by MAXNUMMETAENTRIES in the <dshow_install_dir>\filters\<filter_name>\<filter_name>.cpp file. Under normal operation, one would set NUMBUFS + max # of buffers held by codec to be less than or equal to MAXNUMMETAENTRIES. This is because the number of meta entries queued up represents the number of display frames expected at the output, and the total number of display frames in the queue and codec can never exceed this number at any given time. Therefore, increasing NUMBUFS and/or MAXNUMMETAENTRIES will allow for better "smoothing" of spikes in processing load over multiple frames, at the expense of a higher amount of memory allocated from CMEM and the system.

Let's now look at the data flow in terms of buffer management, Fig. 6 summarizes how a typical video buffer moves through the system:

Fig. 6 Buffer management

As shown, there are a total of 2 buffer copies performed as part of the decode filter, if we put aside the decode operation itself. The rationale behind the data copies was explained in the discussion for version 1.0.

One enhancement we did with respect to the copy of the output data in step 6 of Fig. 5 is to use the SDMA to perform the data transfer, when the hardware is available on the platform used and the corresponding SDMA proxy driver is available. This helps alleviate loading on the ARM processor.

End of stream

The EndOfStream() function is called by the main video streaming thread when the last frame in the video clip is reached. It flushes the DSP codec, and sends all flushed display frames to the worker thread which then delivers them downstream, looping through steps 4 through 8 in Fig. 5. In order to get ready for a possible repeat of the same clip, it shutdowns the worker thread, closes the flushed codec and re-creates a new instance of both the codec and the worker thread. Finally it delivers the end of stream notification downstream.

Seeks

The EndFlush() function is called by the main video thread when a seek operation occurs. The way this is handled is by recording the number of meta entries queued up between the worker thread and the main video thread and ensure that the worker thread discards those entries and their corresponding display frames. This results in a fast transition to the new video segment.

Data Flow in audio streaming thread

The data flow in an audio decode filter while the graph is in running state is shown in Fig. 7.

Fig. 7 Receiving data in audio thread

1. In the audio thread, the Receive function is called on each input sample. For each sample, the filter copies the input data into a physically contiguous circular buffer. This is done via the function TIMM_queueInputDataAuddec1.
2. The filter requests an output sample from the allocator downstream.
3. TIMM_algExecAuddec1() is called. If there is sufficient data in the circular buffer for the XDM codec's process() function to be called, the call would be performed. The decoded audio buffer is copied into the buffer provided by the output sample. However, if there is insufficient data, the audio thread returns from the Receive() function and works on the next input sample.
4. The output sample is stamped with timestamps that are computed based on the number of audio samples that have been previously rendered. This 'time counter' is reset every time there is a seek operation.
5. The output sample is delivered downstream.
6. The output sample is released from the decode filter. If more than one display frame comes back from the codec, the filter would repeat the whole process of allocating an output sample, filling it, binding it and delivering it for each one of the frames.

To clarify the flow in terms of buffer management, Fig. 8 summarizes how a typical audio buffer moves through the filter:

Fig. 8 Buffer management in audio decode filters

As shown, there are a total of 2 buffer copies performed as part of the decode filter, if we put aside the decode operation itself. The copy of the input data (step 1 in Fig. 7) is needed due to the fact that the process() function of a codec may require more input data then what is available in a given input sample. So we need to queue up the data in physically contiguous memory until there is sufficient data to process. Moreover, the filter upstream may choose to use its own 'allocator'. An allocator is used for allocating the media samples between two filters. The input pin of the decode filter can propose an allocator to the output pin of the filter upstream, possibly one that allocates physically contiguous buffers. However, the output pin of the filter upstream always has the final say as to which allocator to use. As a result, the decode filter cannot force the filter upstream to give it samples that reside in physically contiguous memory. Yet physically contiguous memory is necessary for processing by the DSP.

Copy of the output data in step 3 can arguably be removed by forcing the downstream filter to use an allocator that uses contiguous memory. However, because the volume of data in audio processing is relatively low compared to video, we have chosen to simplify things by performing the data copy. This is certainly an area in the audio filters that is open to further optimizations.

End of stream

The EndOfStream() function is called by the audio streaming thread when the last set of input data is reached. It would loop through steps 2 to 6 in Fig. 7 until there is no more data in the circular buffer, regardless of the minimum amount of data the DSP codec normally requires as input. After that, it delivers the end of stream notification downstream.

Seek operations

The EndFlush() function is called by the audio thread when a seek operation occurs. The way this is handled is by discarding all data queued up in the circular buffer to ensure a quick transition to the new segment.

Filter graph destruction (File close)

When a file is closed, the destructor of the filter is called by the filter graph manager. It performs clean-up by tearing down any working thread created by the filter, flushing all data queues and closing the codec instance. Finally it exits Codec Engine itself.

Debugging tips

How to obtain debug trace to debug the data flow in the filter

The codec-specific dlls (mpeg4videodecoder.dll, h264videodecoder.dll, mpeg2videodecoder.dll) and TIMM.dll all have debug zones implemented. The debug zones are as follows:

• Info: verbose information messages. Excellent for finding out what is happening on a frame-by-frame basis. Will cause the filters to miss real-time due to verbosity.
• Stats: periodic statistics (e.g. CPU load in the TIMM dll, # of frames per sec in the codec-specific dlls). Generally not very verbose, so has minimal effect on run-time performance.
• Perf (only available in TIMM.dll): verbose performance related messages. Gives some more info on performance, but due to its frame-by-frame nature it will cause the filters to miss real-time.
• Error: error messages
• Warning: warning messages

To enable debug zones, do the following:

• Ensure you have enabled both the kernel debugger and KITL under your OS design project properties->configuration properties->build option.
• Open up the clip of your choice in CEPlayer. This will cause the DLLs to be loaded.
• Select Target->CE Debug Zones option in Visual Studio
• Find the dll for which you want debug zones enabled in the list. Enable the zones of your choice.
• Hit OK

All output is generated using RETAILMSG calls, hence will work for both debug and retail builds of the DLLs.

You may wish to reopen the clip a second time to catch any debug messages that were missed while you are enabling the debug zones the first time.

To learn more regarding debug zones, refer to MSDN documentation.

When media failed to play

It is possible that one tries to play a clip in the CEPlayer and it wouldn't play. Typically there would be an error popup message such as "No Video Renderer found". This happens when the filter graph manager failed to construct a graph that would render the clip selected. When this happens, and you are think you are certain that the media type can be played using TI's codecs, you can check out the trace from DirectShow itself to find out about the filter construction process. To do so, enable the first 4 debug zones in quartz.dll

To enable the debug zones, do the following:

• Ensure you have enabled both the kernel debugger and KITL under your OS design project properties->configuration properties->build option.
• If you are working on the retail configuration of your OS design, rebuild the debug configuration, rename the quartz.* files in the release folder of the retail configuration to quartz_retail.*, and copy the files quartz.* from the release folder of the debug configuration to the release folder of the retail configuration. Do a makeimg on the retail configuration.
• Start your OS
• Open up the CEPlayer.
• Select Target->CE Debug Zones option in Visual Studio
• Find quartz.dll in the list of DLLs. Enable the first 4 zones in the list of debug zones. Hit Apply then OK.
• Open up the clip you are having trouble with in CEPlayer.

The resulting output in Visual Studio is very verbose. In particular, you will see the filter graph manager attempt to connect pins together based on the media type requested by output pin of the filter upstream. It will try to enumerate every filter in the system to attempt to find out that would work with that media type. For example:

41242370 PID:5580006 TID:58b000a RegEnumFilterInfo pin        0 (1: e436eb83 e436eb88 0)-(0:        0        0)
41242370 PID:5580006 TID:58b000a RegEnumFilterInfo[6e560]: Considering f3e5aa25 (Audio MP3 Decoder filter)
41242370 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73647561 55
PB Debugger Loaded 'C:\WINCE600\OSDESIGNS\EVM_OMAP3530\EVM_3530\RELDIR\TI_EVM_3530_ARMV4I_RELEASE\MPEG2VIDEODECODER.DLL', no matching symbolic information found.
41242370 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73647561 50
41242370 PID:5580006 TID:58b000a RegEnumFilterInfo: Audio MP3 Decoder filter not wanted (input wog output O)
41242370 PID:5580006 TID:58b000a RegEnumFilterInfo[6e760]: Considering ecad26de (H264 Video Decoder Filter)
41242370 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 34363268
41242370 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 34363248
41242370 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 48535356
41242370 PID:5580006 TID:58b000a RegEnumFilterInfo: H264 Video Decoder Filter not wanted (input wog output O)
41242370 PID:5580006 TID:58b000a RegEnumFilterInfo[6e960]: Considering 2b8e6cff (MPEG4 Video Decoder Filter)
41242370 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 44495658
41242748 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 58564944
41242748 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 5634504d
41242780 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 3234504d
41242780 PID:5580006 TID:58b000a RegEnumFilterInfo: MPEG4 Video Decoder Filter not wanted (input wog output O)
41242780 PID:5580006 TID:58b000a RegEnumFilterInfo[6e9e0]: Considering 1bb0ea4a (MPEG2 Video Decoder Filter)
41242780 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 e436eb81
41242780 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 e436eb80
41242780 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 3267706d
41242780 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 3247504d
41242780 PID:5580006 TID:58b000a RegEnumFilterInfo: MPEG2 Video Decoder Filter not wanted (input wog output O)
41242780 PID:5580006 TID:58b000a RegEnumFilterInfo[622c0]: Considering 5bae8e20 (WMAudio Decoder DMO)
41242780 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73647561 160
41242780 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73647561 161
41242780 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73647561 162
41242780 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73647561 163
41242780 PID:5580006 TID:58b000a RegEnumFilterInfo: WMAudio Decoder DMO not wanted (input wog output O)
41242780 PID:5580006 TID:58b000a RegEnumFilterInfo[6ea20]: Considering 38be3000 (MPEG-1 Layer-3 Decoder)
41242780 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73647561 e436eb80
41242836 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73647561 e436eb81
41242836 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73647561 55
41242836 PID:5580006 TID:58b000a RegEnumFilterInfo: MPEG-1 Layer-3 Decoder not wanted (input wog output O)
41242836 PID:5580006 TID:58b000a RegEnumFilterInfo[6e660]: Considering 3ff9f0d8 (WMVideo & MPEG4 Decoder DMO)
41242839 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 3234706d
41242839 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 3234504d
41242839 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 3334706d
41242839 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 3334504d
41242839 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 7334706d
41242839 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 5334504d
41242839 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 31564d57
41242839 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 31766d77
41242839 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 32564d57
41242839 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 32766d77
41242839 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 33564d57
41242839 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 33766d77
41242839 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 50564d57
41242887 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 70766d77
41242887 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 41564d57
41242887 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 61766d77
41242887 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 32505657
41242887 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 32707677
41242887 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 31435657
41242887 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 31637677
41242887 PID:5580006 TID:58b000a RegEnumFilterInfo: WMVideo & MPEG4 Decoder DMO not wanted (input wog output O)
41242887 PID:5580006 TID:58b000a RegEnumFilterInfo[6e5e0]: Considering 70e102b0 (Video Renderer)
41242887 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 0
41242887 PID:5580006 TID:58b000a RegEnumFilterInfo: Video Renderer not wanted (input wog output O)
41242887 PID:5580006 TID:58b000a RegEnumFilterInfo[6ea60]: Considering e30629d1 (Audio Renderer)
41242887 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73647561 0
41242979 PID:5580006 TID:58b000a RegEnumFilterInfo: Audio Renderer not wanted (input wog output O)
PB Debugger Loaded 'C:\WINCE600\OSDESIGNS\EVM_OMAP3530\EVM_3530\RELDIR\TI_EVM_3530_ARMV4I_RELEASE\DECODECOMBO1.DLL', no matching symbolic information found.
41242979 PID:5580006 TID:58b000a RegEnumFilterInfo[6eca0]: Considering 48025243 (Internal Script Command Renderer)
41242979 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73636d64 0
41242979 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73747874 0
41242979 PID:5580006 TID:58b000a RegEnumFilterInfo: Internal Script Command Renderer not wanted (input wog output O)
41242979 PID:5580006 TID:58b000a RegEnumFilterInfo[6ece0]: Considering feb50740 (MPEG Video Codec)
41242989 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 e436eb80
41242989 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73646976 e436eb81
41242989 PID:5580006 TID:58b000a RegEnumFilterInfo: MPEG Video Codec not wanted (input wog output O)
41242989 PID:5580006 TID:58b000a RegEnumFilterInfo[6eee0]: Considering 4a2286e0 (MPEG Audio Codec)
41242989 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73647561 e436eb80
41242989 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73647561 e436eb81
41242989 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73647561 50
41242989 PID:5580006 TID:58b000a RegEnumFilterInfo: MPEG Audio Codec not wanted (input wog output O)
41242989 PID:5580006 TID:58b000a RegEnumFilterInfo[625c0]: Considering 521fb373 (WMSpeech Decoder DMO)
41242989 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73647561 a
41242989 PID:5580006 TID:58b000a RegEnumFilterInfo: WMSpeech Decoder DMO not wanted (input wog output O)
41242989 PID:5580006 TID:58b000a RegEnumFilterInfo[6ef20]: Considering 6a08cf80 (ACM Wrapper)
41242989 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(73647561 0
41242989 PID:5580006 TID:58b000a RegEnumFilterInfo: ACM Wrapper not wanted (input wog output O)
41242989 PID:5580006 TID:58b000a RegEnumFilterInfo[685a0]: Considering b9d1f32e (ASF embedded stuff Handler)
41242989 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(18056900 736c7275
41242989 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(18056900 74726c6d
41242989 PID:5580006 TID:58b000a RegEnumFilterInfo: ASF embedded stuff Handler not wanted (input wog output O)
41243000 PID:5580006 TID:58b000a RegEnumFilterInfo[68900]: Considering b9d1f323 (ASF URL Handler)
41243000 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(18056900 59dacfc0
41243000 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(18056900 14082850
41243000 PID:5580006 TID:58b000a RegEnumFilterInfo: ASF URL Handler not wanted (input wog output O)
41243000 PID:5580006 TID:58b000a RegEnumFilterInfo[6d060]: Considering b9d1f322 (ASF ICM Handler)
41243000 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(18056900 bc19efc0
41243000 PID:5580006 TID:58b000a RegEnumFilterInfo: ASF ICM Handler not wanted (input wog output O)
41243000 PID:5580006 TID:58b000a RegEnumFilterInfo[6ee20]: Considering b9d1f321 (ASF ACM Handler)
41243000 PID:5580006 TID:58b000a No type match yet: Req (e436eb83 e436eb88) Found(18056900 f8699e40
41243000 PID:5580006 TID:58b000a RegEnumFilterInfo: ASF ACM Handler not wanted (input wog output O)
41243000 PID:5580006 TID:58b000a RegEnumFilterInfo[6ee60]: Considering 1b544c20 (AVI Splitter)
41243000 PID:5580006 TID:58b000a Types match: Req (e436eb83 e436eb88) Found(e436eb83 e436eb88
41243028 PID:5580006 TID:58b000a RegEnumFilterInfo returning AVI Splitter
41243028 PID:5580006 TID:58b000a EnumRegFilters returning 1 filters

The above trace snippet shows the filter graph manager trying to find a filter that would accept media type "e436eb83 e436eb88" from the output of the source filter. Eventually it found the AVI Splitter filter being able to accept such a media type at its input pin. If at some point in the trace the filter graph manager is unable to locate a filter that would support a media type, then the graph building process would fail. You would need to find out what that media type is and figure out which filter is supposed to take it and why it isn't, etc. A common root cause could be that the stream is mislabeled with a data type not recognized by the corresponding filter. See the sections "How to figure out the GUID for a specific media type" and "How to register a filter for a given media type" in this case.

Eventually, once you are successful in tracking down the problem and fixing it, you should get a complete filter graph dump. Here's an example of one that successfully played an MPEG4 video clip in an AVI container:

 686282 PID:5340006 TID:57d000a Filter graph dump
 686282 PID:5340006 TID:57d000a Filter e9460 'Video Renderer' Iunknown e9450
 686282 PID:5340006 TID:57d000a     Pin e98e8 Input (Input) connected to e664c
 686282 PID:5340006 TID:57d000a Filter e62cc 'MPEG4 Video Decoder Filter' Iunknown e62c0
 686282 PID:5340006 TID:57d000a     Pin e654c XForm In (Input) connected to e0390
 686282 PID:5340006 TID:57d000a     Pin e664c XForm Out (PINDIR_OUTPUT) connected to e98e8
 686282 PID:5340006 TID:57d000a Filter e4e10 'Audio Renderer' Iunknown e4e00
 686282 PID:5340006 TID:57d000a wo: GetPin, 0
 686282 PID:5340006 TID:57d000a     Pin e5010 Audio Input pin (rendered) (Input) connected to e48ec
 686282 PID:5340006 TID:57d000a Filter 6354c 'Audio MP3 Decoder filter' Iunknown 63540
 686282 PID:5340006 TID:57d000a     Pin e47ec XForm In (Input) connected to e0630
 686282 PID:5340006 TID:57d000a     Pin e48ec XForm Out (PINDIR_OUTPUT) connected to e5010
 686282 PID:5340006 TID:57d000a MP3: NonDelegatingQueryInterface
 686282 PID:5340006 TID:57d000a Filter 693d0 'AVI Splitter' Iunknown 693c0
 686282 PID:5340006 TID:57d000a     Pin e0630 Stream 01 (PINDIR_OUTPUT) connected to e47ec
 686282 PID:5340006 TID:57d000a     Pin e0390 Stream 00 (PINDIR_OUTPUT) connected to e654c
 686282 PID:5340006 TID:57d000a     Pin 6e6b0 input pin (Input) connected to 68da0
 686282 PID:5340006 TID:57d000a MP3: NonDelegatingQueryInterface
 686282 PID:5340006 TID:57d000a Filter 68c70 '\Storage Card\MPEG4 MP3 D1 1.7Mbps 23.97fps.AVI' Iunknown 68c60
 686282 PID:5340006 TID:57d000a     Pin 68da0 Output (PINDIR_OUTPUT) connected to 6e6b0
 686282 PID:5340006 TID:57d000a End of filter graph dump

How to figure out the GUID for a specific media type

GUIDs are identifiers used to designate media types, and are useful for understanding filter graphs and registering filters. Unfortunately there is no complete list of all the GUIDs for all media types at one place. The following items may be able to help:

• The files PUBLIC\DIRECTX\OAK\FILES\directx.reg, PUBLIC\DIRECTX\SDK\INC\uuids.h, PUBLIC\DIRECTX\SDK\INC\wmcodectypes.h
• Common FourCC codes list: http://msdn.microsoft.com/en-us/library/ms867195.aspx#fourcccodes
• A Microsoft DirectShow developer (if you know of any!)

Typically, a GUID is constructed with a prefix that contains the hexadecimal equivalent to the four letters to the FourCC code corresponding to its type, with the letters spelled in reverse order. E.g. FMP4 would have a GUID prefix of 0x34504d46, i.e. "4PMF". The suffix seems to be 0000-0010-8000-00AA00389B71 most of the time for video subtypes.

If a GUID hasn't been defined anywhere in WinCE, you can define it yourself in your filter so that you can register against it, e.g.:

DEFINE_GUID(MEDIASUBTYPE_mpg2,
0x3247504D, 0x0000, 0x0010, 0x80, 0x00, 0x00, 0xAA, 0x00, 0x38, 0x9B, 0x71);

How to register a filter for a given media type

Registration is done at multiple levels:

• In the registry. Each DirectShow filter comes with a set of registry entries which needs to be copied manually into the BSP after changes. It contains entries such as this for its input and output pins:

[HKEY_CLASSES_ROOT\CLSID\{F3E5AA25-3203-4ead-8DC6-BA46AF506294}\Pins\Input\Types\{73647561-0000-0010-8000-00AA00389B71}\{00000055-0000-0010-8000-00AA00389B71}]

The above example corresponds to registration of a major type with GUID of {73647561-0000-0010-8000-00AA00389B71} and a subtype of {00000055-0000-0010-8000-00AA00389B71}. The major type's GUID can usually be figured out by looking at the registration of existing filters in RelDir\reginit.ini. It's usually either video or audio. Figuring out the subtype is a little trickier. The subtype's GUID needs to be determined using the techniques described in the previous section on "How to figure out the GUID for a specific media type".

• In the AMOVIESETUP_MEDIATYPE structure in the filter's C code. This structure lists the media types supported by the filter. For example, the structure looks as follows for MPEG2:

<syntaxhighlight lang='c'> static const AMOVIESETUP_MEDIATYPE sudInputPinTypes[] = {

   {
       &MEDIATYPE_Video,            // Major type
       &MEDIASUBTYPE_MPEG1Payload   // Minor type
   },
   {
       &MEDIATYPE_Video,            // Major type
       &MEDIASUBTYPE_MPEG1Packet    // Minor type
   },
   {
       &MEDIATYPE_Video,            // Major type
       &MEDIASUBTYPE_MPG2           // Minor type
   },
   {
       &MEDIATYPE_Video,            // Major type
       &MEDIASUBTYPE_mpg2           // Minor type
   }

}; </syntaxhighlight>

\WINCE600\PUBLIC\DIRECTX\OAK\FILES\directx.reg, \WINCE600\PUBLIC\DIRECTX\SDK\INC\wmcodectypes.h and \WINCE600\PUBLIC\DIRECTX\SDK\INC\uuids.h are good files to look into to find out about the predefined types. If you have a media which GUID is not defined anywhere in WinCE, you can define it yourself in your filter as described in the previous section on "How to figure out the GUID for a specific media type".

• Modify the code in CheckInputType(), GetMediaType() and CheckTransform() to ensure the type is accepted by your filter's input and/or output pins. These functions are used during construction of a filter graph, and if they are not modified they may result in the filter being rejected by the filter graph manager. Check out an existing filter to find out how to write these functions.

Using Super to create video clips for CEPlayer on WinCE

Super is a free tool that is available here: http://www.erightsoft.com/SupCk2.html

This tool is very handy in order to convert your existing video into a format that is supported by the media player on WinCE.

Here's a screenshot of the tool:

To use this tool, first select the appropriate output container, output video codec and output audio codec. Here's a summary of the various container formats and codecs supported by the filters out-of-box in each major release:

Media formats supported by DirectShow filters

Version	Container formats	Codecs
1.0	avi	MPEG2, MPEG4, H264, MP3
1.10	avi	MPEG2, MPEG4, H264, MP3
	mp4/3gp	MPEG4, H264, AAC

Choose "FFmpeg" as the tool used for encode. This is a very versatile tool, and supports almost all formats that possibly exists.

For Video Scale Size, simply select the radio button "NoChange". This will preserve your original clip's resolution. Or you can choose to change the size by using the other options.

Select the bitrate you'd like to use for the output. Suggest you use < 4Mbps for smoother playback.

Right click anywhere in the main window. Select "Output File Saving Management" and choose the location at which the newly encoded file will reside.

Then drop a valid multimedia file into the box near the bottom. You can use any file that is perhaps using a different container format or encoded using a different codec. Make sure the file is checked after it has been added.

Hit the "Encode (Active Files)" button. Super will then prompt you for the variants of the encoding to use (aka Rendered file optimizer). Each variant is identified with a FourCC code (see http://www.fourcc.org/). The Directshow filters currently only support the media encoded with encoders with these FourCC codes, which have been found to work with TI decoders:

FourCC codes supported by DirectShow filters

Codec	FourCC codes supported
MPEG2	mpg2, MPG2
MPEG4	FMP4, MP4V, DIVX, XVID
H264	H264, h264, VSSH

Super will then transcode the media into the new format, and you can now play the output file on your board in CEPlayer.

Creating a filter for a new codec/algorithm

The current DirectShow filters only support a small set of codecs run on the DSP. It might be desirable at some point to add support for other xDM-compliant codecs (or even non-codec algorithms, as long as it adheres to one of the xDM interfaces). In a way, the filters supplied by TI can be considered examples of how any DSP codec can be run from DirectShow. In general, supporting a new codec on the DSP consists of doing the following:

• Create a codec server that contains the codec. This can be done by using existing tools such as the RTSC codec/server/combo package wizards (RTSC_Codec_And_Server_Package_Wizards). These tools run on a Windows PC and 'package' up the codec in such a way it can be consumed in a server package. The latter contains the DSP image that runs the codec on the DSP. Note that only one server image can run at any one time on the DSP, hence if you still intend to use the existing filters in your new system, you would need to add the corresponding codecs into your new DSP server as well.
• If the codec you want to run implements the IVIDDEC2 xDM interface, you can pick an existing filter and modify it to run your codec instead. Here's an outline of the procedure to follow:
  • Pick a filter (say MPEG4).
  • Make a copy of the directory MPEG4VideoDecoder under the 'filters' directory. Rename it to match your codec's name.
  • From the newly created directory, open MPEG4VideoDecoder.hpp and MPEG4VideoDecoder.cpp in your favorite text editor.
  • Do a search and replace "MPEG4" with the name of your codec. This includes all variable names, strings, etc.
  • At the top of the .cpp file, 'ALGNAME' is defined. This corresponds to the name of the codec on the DSP server image. Make sure you modify it to match the name of your codec, as defined in the server configuration file (server.cfg) of the server package.
  • At the top of the .cpp file, 'ENGINE_NAME' is defined. Make sure this matches the engine name you have chosen in your DSP server configuration file. If you have changed the name of the engine from the default one used in the DVSDK, you would need to perform the same change in the existing filters if you want them to continue to work.
  • Modify the functions involved in the filter connection process to accept/report the media types supported by your codec for both input and output of the filter: CheckInputType, CheckTransform, GetMediaType, SetMediaType. MSDN is a good resource if you need help dealing with media types in these functions.
  • Modify DecideBufferSize to allocate the right output buffer size. This is also where you can change the xDM parameters passed to the codec if necessary.
  • Recompile the newly created directory containing your filter.
  • Copy both your filter's codec-specific dll and the new server image (.x64P file) into the Release folder.
  • Modify the platform.reg file to add registry entries for your filter and its pins (you may wish to check out the existing entries for MPEG4 to get an idea of what to do)
  • Modify the platform.bib file to add bib entries for your new DLL and server image.
  • Rebuild your OS image. Your should now be able to run your codec as part of this new image.
• For all other xDM interfaces, you will need to study TIMM in detail and create some TIMM functions to talk to the xDM interface in question. For example, if you want to add support for the IAUDDEC1 interface, you will need to create an iauddec1.c and iauddec1.h file that make DMAI and Codec Engine calls to invoke the codec. Refer to DMAI and Codec Engine documentation on how to do this. Then recompile TIMM and write the codec-specific DirectShow filter dll that calls into your TIMM API. Given the nature of your codec may be very different from an 'IVIDDEC2' video codec, your filter may also look very different from the existing ones. If you are new to DirectShow you may wish to stick to the transform filter model (i.e. use CTransformFilter as a base class) as it is simpler to use.

Summary

TI's DirectShow filters provide the WinCE DirectShow framework with access to the DSP for acceleration of multimedia processing in the WinCE Media Player. This access is enabled by TI's traditional DMAI/Codec Engine software stack. In general, the filter code can also be used as an example of how to leverage DMAI and Codec Engine in the context of DirectShow in order to run any algorithm on the DSP.

Useful Resources

• DirectShow overview: http://msdn.microsoft.com/en-us/library/ms778824(VS.85).aspx
• Transform filters overview: http://msdn.microsoft.com/en-us/library/aa921464.aspx
• How directshow filters are registered in a system: http://msdn.microsoft.com/en-us/library/ms783789(VS.85).aspx and http://msdn.microsoft.com/en-us/library/aa918244.aspx
• Microsoft DirectShow forums: http://social.msdn.microsoft.com/Forums/en-US/windowsdirectshowdevelopment/threads
• Codec Engine overview: Codec_Engine
• DMAI (Davinci Multimedia Application Interface) overview: DMAI