UserGuideResizerDriver PSP 03.00.00.04

Resizer Driver

Introduction

This section provides overview of the Resizer Hardware.

References

1. Linux Device Drivers Edition-3 [http://lwn.net/Kernel/LDD3/]
2. Video for Linux Two API Specification [[http://v4l2spec.bytesex.org/v4l2spec/v4l2.pdf]

Acronyms & Definitions

Resizer Driver: Acronyms

Acronym	Definition
V4L2	Video for Linux Two

Hardware Overview

The OMAP35x Resizer module enables up scaling and down scaling. It resizes YUV422 image and stores output image in the RAM. The following figure shows the block diagram for Resizer module.

OMAP35x Resizer HW Block Diagram

The Resizer module performs digital zoom either up sampling or down sampling on image/video data within a range of 0.25x to 4x resizing. The input source can be sent to either the preview engine/CCDC or memory, and the output is sent to memory.
The Resizer module performs horizontal resizing, then vertical resizing, independently. Between them, there is an optional edge-enhancement feature. This process is shown in the above Figure.

Features

The Resizer driver supports the following features:

Resizes input frame stored in RAM and stores output frame in RAM.

• Supports resizing from 1/4x to 4x.
• Supports independent horizontal and vertical resizing.
• Supports YUV422 packed data and Color Separate data.
• Supports driver allocated and user provided buffers.
• Supports Luminance Enhancement.
• Supports configuration of read request cycles.

Usage

Following sections provides details about the drive supported features.

Opening and Closing the Driver

The device can be opened using open call from the application with device name and mode of operation as parameters. Mode can be blocking, non-blocking and readwrite. Application can open the driver in either blocking mode or non-blocking mode. If driver is opened in blocking mode, RSZ_RESIZE ioctl will block until resizing task is over for that channel. If the driver is opened in non-blocking mode, RSZ_RESIZE ioctl returns if hardware is busy serving other channel.

Driver can be opened multiple times. Driver maintains software channels for all opened instances. If multiple resizing task is submitted at the same time, driver serializes the resizing task.
To close a specific device, application calls the close function with the file handle.

/* call to open a Resizer logical channel in blocking mode */
rszfd_blocking = open ("/dev/omap-resizer", O_RDWR);
if (rszfd_blocking == -1) {
 perror("failed to open Resizer device\n");
 return -1;
}
/* closing of channels */
close (rszfd_blocking);

Buffer Management

Resizer Driver requires buffers for storing input/output images. Buffers can be allocated by the driver itself or application can provide the buffers. These buffers need to be virtually contiguous. Physically buffers can be scattered in the memory.
The Resizer Driver supports two memory usage models.

• Memory map/Driver allocated buffer mode
• User Pointer Exchange

Memory map/Driver Allocated buffer

In Memory map buffer mode, application can request memory from the driver by calling RSZ_REQBUF ioctl. With this ioctl, application passes pointer to the structure v4l2_requestbuffers . In this structure, buffer exchange mechanism can be specified. For memory map buffer exchange mechanism, it should be V4L2_MEMORY_MMAP. Driver always allocates buffer of maximum size required to store input or output image. If output image width is less than input image width, a single buffer can be used to store input and output images. Otherwise, at least, two buffers are required to store input and output images. Application can request as many buffers as it wants. Buffer with the index 0 is always used as the input buffer and other buffer can be used as the output buffer.

The main steps that the application must perform for buffer allocation are:

• Allocating Memory
• Getting Physical Address
• Mapping Kernel Space Address to User Space

• Allocating Memory:

Application can allocate buffers using RSZ_REQBUF IOCTL. While allocating the buffers, the application have to specify the buffer type, number of buffers and memory type. Here the buffer type must be V4L2_BUF_TYPE_VIDEO_CAPTURE. Number of buffers can be greater than or equal to one. If output image is less than input image, number of buffers can be one. Drivers always uses maximum of input and output image size as the buffer size.

This ioctl takes object of v4l2_request buffer structure.

/* structure to store buffer request parameters */
struct v4l2_requestbuffers reqbuf;
reqbuf. type =V4L2_BUF_TYPE_VIDEO_CAPTURE;
reqbuf.count = 2; /* number of buffers */
reqbuf.memory = V4L2_MEMORY_MMAP; /* Type of buffer exchange
 mechanism */
if(ioctl(rszfd, RSZ_REQBUF, &reqbuf)< 0) {
        perror("RSZ_REQBUF failed\n");
        close(rszfd);
        exit(-1);
}
/* The above example will allocate 2 buffers */

• Getting Physical Address:

The RSZ_QUERYBUF IOCTL can obtain the physical address of the allocated buffer. Application has to specify the index, buffer type and buffer's memory type at time of calling this ioctl. Buffer type must be V4L2_BUF_TYPE_VIDEO_CAPTURE. Index of each type of buffer starts from 0. Buffer memory type must be V4L2_MEMORY_MMAP for driver allocated buffers. The driver fills the size and physical address, and then returns to the application so that the relevant data can be used to mmap the buffer to user space.
This ioctl takes object of the structure v4l2_buffer.

/* allocate buffer by RSZ_REQBUF */
/* structure to query the physical address of allocated buffer */
struct v4l2_buffer buffer;
buffer.index = 0; /* buffer index - 0 */
buffer.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; /* Input buffer */
if (ioctl(rszfd, RSZ_QUERYBUF, &buffer) < 0) {
        perror("RSZ_QUERYBUF ioctl failed\n");
        close(rszfd);
        exit(-1);
}
/* The buffer.m.offset will contain the physical address returned from driver */

• Mapping Kernel Space Address to User Space:

Mapping the kernel buffer to the user space is done via the Linux mmap system call. Application must pass the buffer size and buffer's physical address for getting the user mapped address.

/* allocate buffer by RSZ_REQBUF */
/* query the buffer using RSZ_REQBUF */
/* addr hold the user space address */
unsigned long addr;
addr = mmap(NULL, buffer.size, PROT_READ | PROT_WRITE,
      MAP_SHARED, rszfd, buffer.offset);
/* buffer.offset is same as returned from RSZ_QUERYBUF */

User Pointer Exchange

Application informs driver whether to use memory mapped buffer or user buffer in memory allocation operation (RSZ_REQBUFS ioctl). This ioctl is being used when request for buffer allocation is submitted to the driver. Along with ioctl, application has to specify either memory mapped or user buffer to be used. If user provided buffer is used, application has to pass memory type as V4L2_MEMORY_USERPTR. Then at the time of en-queueing buffers, application can specify pointer to the virtually contiguous buffer allocated by the application and size of the buffer.

NOTE: The size of the buffer, specified at the time of en-queueing buffer, must be page-aligned.

/* structure to store buffer request parameters */
struct v4l2_requestbuffers reqbuf;
reqbuf. type =V4L2_BUF_TYPE_VIDEO_CAPTURE;
reqbuf.count = 2; /* number of buffers */
reqbuf.memory = V4L2_MEMORY_USERPTR; /* Type of buffer exchange
 mechanism */
if(ioctl(rszfd, RSZ_REQBUF, &reqbuf)< 0) {
        perror("RSZ_REQBUF failed\n");
        close(rszfd);
        exit(-1);
}
/* This above example will allocate 2 buffer descriptors */

Parameter/Driver Configuration

Resizing

The Resizer module takes input image from RAM, resizes it horizontally and vertically using given resizing ratio and stores output image in RAM. The Resizer module can up scale or down scale image data with independent resizing factors in the horizontal and vertical directions. The resizing ratio is calculated using formula 256/N where value of N can range from 64 to 1024. The Resizer module uses the same resampling algorithm for the horizontal and vertical directions. The resizing/resampling algorithm uses a programmable polyphase sample rate converter (resampler). The polyphase filter coefficients are programmable so that any user-specified filter can be implemented.

For horizontal and vertical direction, application has to provide 32 coefficients. These coefficient values are dependent on resizing ratio. The Resizer hardware uses 4 taps and 8 phases filters for the resizing range of 1/2x to 4x and 7 taps and 4 phases filters for a resizing range of 1/4x to 1/2x for both the direction. So different set of coefficients must be provided for 4 taps and 8 phases filters and 7 taps and 4 phases filter.

As the hardware uses multi tape poly phase filters, filter requires more input pixel than following equation calculates.

Input size = output size * N / 256; /* 256 where N is from 64 to 1024 */

Input size is also dependent on the starting phase and rounding issues in the resizing algorithm of the hardware. The input width and height parameters must be programmed strictly according to these equations given in following table otherwise, incorrect hardware operation may occur.

Resizer: Input Size Calculation

	Ratio 1/2x to 4x	Ration 1/4x to 1/2x
Input width	(32 * sph + (ow - 1) * hrsz + 16) >> (8 + 7)	(64 * sph + (ow - 1) * hrsz + 32) >> (8 + 7)
Input height	(32 * spv + (oh - 1) * vrsz + 16) >> (8 + 4)	(64 * spv + (oh - 1) * vrsz + 32) >> (8 + 7)

Where

       sph = horizontal starting phase
       spv = vertical starting phase
       ow = output width
       oh = output height
       hrsz = horizontal resize value
       vrsz = vertical resize value.

Application sets the resizing ratio by providing input size and output size parameters in RSZ_S_PARAMS ioctl. This ioctl takes object of rsz_params structures. Application provides input width, pitch and height and output width, pitch and height. Driver calculates the resizing ratio for horizontal and vertical direction using below equation.

Horizontal_ratio = (input_width-N)*256/(output_width-1);
     
Where N = 7 for ratio in between 1/4x to 1/2x 4 for ratio in between 1/2x to 4x

Similar equation is used to calculate vertical resizing ratio. So this equation must be used by the application when calculating input and output size for the given resizing ratio.
Actual Resizing operation is performed when application calls RSZ_RESIZE ioctl. This ioctl programs Resizer Hardware, submits resizing task and waits for it to be completed. This ioctl will block the application if the resizer driver is opened in blocking mode. If it is opened in non-blocking mode, it will simply return with busy if the hardware is busy. Before submitting resizing task, the input and output buffers must be en-queued to the driver so the drive will come to know which buffers to be used as input and output. Also resize ioctl, as an side effect, removed buffers from the queue. So if resizing task required to be re-submitted, buffers must be en-queued again. So whenever resizing task is submitted, input and output buffers must also be enqueued first.

Chroma Algorithm

Chroma components, which are 2:1 horizontally down sampled with respect to luma, have two methods of horizontal resizing: Filtering with luma, and Bilinear interpolation. The Chroma algorithm option can be selected in the cbilin field of rsz_params structure. However, filtering with luma is only intended for down sampling, and bilinear interpolation is only intended for up sampling.
There are two possible values of cbilin member of rsz_params structures. 0 and 1. 0 indicates that the chrominance uses the same processing as the luminance and 1 indicates that the chrominance uses bilinear interpolation processing.

Input/output image format

Resizer module supports two types of image format. One is YUV422 packed data and the other is color separate data. Input image format can be selected by providing one of RSZ_INTYPE_YCBCR422_16BIT or RSZ_INTYPE_PLANAR_8BIT value to inptype member of rsz_params structure. Configured input image format is used for both input and output image.

When YUV422 interleaved (packed) image format is selected, resizer module resizes entire image and stores it in output buffer.

When color separate image format is selected, application has to resize each of the color components separately. Application must open three instances of the resizer driver and resize each color components in one instance separately. Application must provide correct input and output size of the each color components when resizing color components.

Pixel Format

Resizer module supports two pixel format YUYV and UYVY. Pixel format can be selected by providing one of RSZ_PIX_FMT_YUYV or RSZ_PIX_FMT_UYVY value to pix_fmt member of rsz_params structure. Configured pixel format is used for both input and output image. When color separate input data is used, this field is ignored.

Luma Enhancement

Edge enhancement can be applied to the horizontally resized luminance component before the output of the horizontal stage is sent to the line memories and the vertical stage. type member of rsz_yenh member of rsz_params structure can be set to disable edge enhancement, or to select a 3-tap or a 5-tap horizontal high-pass filter (HPF) for luminance enhancement. So possible values of type is 0, 1 or 2 for disabling luma enhancement, selecting 3 tap filed or selecting 5 tap filter. If edge enhancement is selected, the two left-most and two right-most pixels in each line are not outputted to the line memories and the vertical stage. When luma enhancement is enabled, maximum output width can be 1280 when 1/2x to 4x vertical resizing ratio is selected and 640 when 1/4x to 1/2x vertical resizing ratio is selected.
Luma enhancement algorithm is as follows.

HPF (Y_IN) = Y_IN convolved with {[-0.5, 1, 0.5] or [-0.25, -0.5, 1.5, -0.5, -0.25]}
Implemented as [-1, 2, -1] >> 1, [-1, -2, 6, -2, -1] >> 2)
Saturate HPF(Y) between -256 and +255
Hpgain = (|HPF(Y)| - CORE) * SLOP
Saturate hpgain between 0 and GAIN
Y_OUT = Y_IN + (HPF (Y_IN) * hpgain + 8) >> 4
Saturate Y_OUT between 0 and 255


Application have to provide core, slope and gain members of rsz_yenh member of rsz_params structure.

Read cycle for each memory access

ISP module supports configuration of number of clock cycles between two consecutive read request from resizer module. Value supported is 0 - 0x3FF.

unsigned int read_exp;
read_exp = 0xe;
ret_val = ioctl(fd, RSZ_S_EXP, &read_exp);
if (ret_val){
     printf("\nUnable to set the read cycle expand register\n");
     return ret_val;
}

NOTE: The default configuration of read cycle is 0xE.

Architecture

Following block diagram shows the basic architecture of the Resizer Driver -

Basic Architecture of Resizer Driver

Resizer Driver provides Resizer Hardware access to a channel by using Linux Character driver interface. Driver supports all the features supported by the hardware. It provides easy way of configuring the hardware. To understand this, that the hardware module driver implements, is briefly described in this section.

Software Interface

This section describes the Data Structures, Enumerations, and API Specifications used in the OMAP35x Resizer Driver.

Application Programming Interface

open

Opens the device driver for processing.

Prototype:

int fd = open(device_name, mode);

Resizer: open system call arguments

Field	Description
device_name	mode
It is /dev/omap-resizer	O_RDWR or ORed with O_NONBLOCK

Return Values: Zero on success, or negative if an error has occurred.

close

Close the device

Prototype:

close(fd);

Resizer: close system call arguments

Field	Description
fd	File descriptor returned from open call.

Return Values: Zero on success, or EINTR, if driver could not get the handle.

ioctl

Allows user application to configure both Driver and Resizer Module through this,

Prototype:

int ioctl(fd, ioctl_no, arg );

Resizer: ioctl system call arguments

Field	Description
fd	File descriptor returned from open call.
ioctl_no	Unique IOCTL number recognized by driver
arg	Argument with respect to IOCTL

Return Values: Upon successful completion, ioctl() shall return a value other than -1. Otherwise, it shall return -1 and set errno to indicate the error.

mmap

Map the kernel space buffer to user space.

Prototype:

void * mmap(void *, size_t image_size, int prot, int flags, int fd, off_t offset)

Resizer: mmap system call arguments

Field	Description
void *	Generally NULL
image_size	Buffer size that needs to be mapped
flag	PROT_SHARED
fd	File descriptor
offset	Physical address of the buffer

Return Values: Zero on success. EAGAIN, if the address is not found.

munmap

Unmap the frame buffers that were previously mapped to user space using mmap() system call.

Prototype:

void *munmap(void *start_addr, size_t length)

Resizer: mmap system call arguments

Field	Description
start_addr	Start address of buffer which is to be unmapped.
length	Length of buffer.

Return Values: Zero on success, or Negative if an error has occurred

Supported IOCTL's

RSZ_S_PARAMS

Set the resizer parameters necessary for processing.

Prototype:

int ioctl(int fd, RSZ_S_PARAMS, struct rsz_params *argp)

Resizer: mmap system call arguments

Field	Description
fd	File handle associated with fd.
cmd	RSZ_S_PARAMS ioctl command.
argp	Pointer to rsz_params structure.

Return Values: Zero on success, EINVAL, if parameters are incorrect. EINTR, if device is in use by the same channel handle.

RSZ_G_PARAMS

Get the Resizer parameters that are previously being set.

Prototype:

int ioctl(int fd, RSZ_G_PARAMS,struct rsz_params *argp)

Resizer: mmap system call arguments

Field	Description
fd	File handle associated with fd.
cmd	RSZ_G_PARAMS ioctl command.
argp	Pointer to rsz_params structure.

Return Values: Zero on success, EINVAL, if device is not configured before calling this API.

RSZ_G_STATUS

Get the channel status for the particular current Resizer channel.

Prototype:

int ioctl(int fd, RSZ_G_STATUS, struct rsz_status *argp)

Resizer: mmap system call arguments

Field	Description
fd	File handle associated with fd.
cmd	RSZ_G_STATUS ioctl command.
argp	Pointer to rsz_status structure.

Return Values: Zero on success,

RSZ_S_EXP

Configure the Read cycle required for Resizer module. This configuration is provided per channel.

Prototype:

int ioctl(int fd, RSZ_S_EXP, unsigned int *argp)

Resizer: mmap system call arguments

Field	Description
fd	File handle associated with fd.
cmd	RSZ_S_EXP ioctl command
argp	Pointer to unsigned int

Return Values: Zero on success,

RSZ_RESIZE

Starts the Resizer processing for the parameters previously set by RSZ_S_PARAMS

Prototype:

int ioctl(int fd, RSZ_RESIZE, int *argp)

Resizer: mmap system call arguments

Field	Description
fd	File handle associated with fd.
cmd	RSZ_RESIZE ioctl command.
argp	Pointer to int.

Return Values: Zero on success, EINVAL, if parameters are incorrect. EBUSY/EINTR, if device is in use by the same channel handle.

RSZ_REQBUF

Request to allocate buffers

Prototype:

int ioctl(int fd, RSZ_REQBUF, struct v4l2_requestbuffers *argp)

Resizer: mmap system call arguments

Field	Description
fd	File handle associated with fd.
cmd	RSZ_REQBUF ioctl command.
argp	Pointer to v4l2_requestbuffers structure.

Return Values: Zero on success, ENOMEM, if memory is not available. EINTR, if device is in use by the same channel handle.

RSZ_QUERYBUF

Request physical address of buffers allocated by the RSZ_REQBUF

Prototype:

int ioctl(int fd, RSZ_QUERYBUF, struct v4l2_buffer *argp)

Resizer: mmap system call arguments

Field	Description
fd	File handle associated with fd.
cmd	RSZ_QUERYBUF ioctl command
argp	Pointer to v4l2_buffer structure.

Return Values: Zero on success, EINVAL/EFAULT, if parameters are incorrect. EINTR, if device is in use by the same channel handle.

RSZ_QUEUEBUF

Queue the buffer for resize operation.

Prototype:

int ioctl(int fd, RSZ_QUEUEBUF, struct v4l2_buffer *argp)

Resizer: mmap system call arguments

Field	Description
fd	File handle associated with fd.
cmd	RSZ_QUEUEBUF ioctl command
argp	Pointer to v4l2_buffer structure.

Return Values: Zero on success, EINVAL/EFAULT, if parameters are incorrect. EINTR, if device is in use by the same channel handle.

Data Structures

Resizer Parameters Configuration Structure

struct rsz_params {
        __s32 in_hsize;
        __s32 in_vsize;
        __s32 in_pitch;
        __s32 inptyp;
        __s32 vert_starting_pixel;
        __s32 horz_starting_pixel;
        __s32 cbilin;
        __s32 pix_fmt;
        __s32 out_hsize;
        __s32 out_vsize;
        __s32 out_pitch;
        __s32 hstph;
        __s32 vstph;
        __u16 tap4filt_coeffs[32];
        __u16 tap7filt_coeffs[32];
       struct rsz_yenh yenh_params;
} ;

Resizer: Parameters Configuration Structure fields

Name	Description
in_hsize	Width of the input image in pixels.
in_vsize	Height of the input image in pixels.
in_pitch	Pitch of input image in bytes.
inptype	Input image format.
vert_starting_pixel	Vertical starting pixel.
horz_starting_pixel	Horizontal starting pixel.
cbilin	Chroma resizing algorithm.
pix_fmt	Image Pixel format for YUV422 image.
out_hsize	Width of the output image in pixels.
out_vsize	Height of the output image in pixels.
out_pitch	Pitch of the output image in bytes.
hstph	Horizontal starting phase.
vstph	Vertical starting phase.
tap4filt_coeffs	Set of coefficients for scaling ratio 0.5x - 4x.
tap7filt_coeffs	Set of coefficients for scaling ratio 0.25x - 0.5x.
yenh_params	Luma Enhancement parameters.

Request Buffer Structure

struct v4l2_requestbuffer {
 unsigned int type;
 unsigned int count;
 enum v4l2_memory memory;
 ...
}

Resizer: Parameters Configuration Structure fields

Name	Description
type	Buffer type V4L2_BUF_TYPE_VIDEO_CAPTURE.
count	Number of buffers to be allocated.
memory	Type of the buffer exchange mechanism requested.

Buffer structure

struct v4l2_buffer {
 unsigned int index;
 unsigned int type;
 enum v4l2_memory memory;
 union {
  unsigned long offset;
  unsigned long userptr
 }m;
}

Resizer: Parameters Configuration Structure fields

Name	Description
index	Index of the input/output buffer.
type	Type of the buffer is V4L2_BUF_TYPE_VIDEO_CAPTURE.
offset/userptr	Physical/virtual address of the buffer.
memory	Type of memory, V4L2_MEMORY_MMAP or V4L2_MEMORY_USERPTR.

Luma enhancement structure

struct rsz_yenh {
 __s32 type;
 __u8 gain;
 __u8 char slop;
 __u8 core;
}

Resizer: Parameters Configuration Structure fields

Name	Description
type	Luma Enhancement algorithm.
gain	Gain.
slop	Slop.
core	Core.

Status structure

struct rsz_status {
 __s32 chan_busy;
 __s32 hw_busy;
 __s32 src;
}

Resizer: Parameters Configuration Structure fields

Name	Description
chan_busy	Status of the channel.
hw_busy	Status of the hardware.
src	Input source.

Crop Size structure

struct rsz_cropsize {
 __u32 hcrop;
 __u32 vcrop;
}

Resizer: Parameters Configuration Structure fields

Name	Description
hcrop	Number of pixels cropped in horizontal direction.
vcrop	Number of pixels cropped in vertical direction.

Input/Output image format

This describes the input and output image format, which can be YUV interleaved 16 bit or planar 8 bit. This can be specified in inptype field of the rsz_params structure.

#define RSZ_INTYPE_YCBCR422_16BIT	0
#define RSZ_INTYPE_PLANAR_8BIT		1

Pixel Format

This describes pixel format for the YUV interleaved data. This can be specified in pix_fmt member of rsz_params structure.

#define RSZ_PIX_FMT_UYVY  	1 /* cb:y:cr:y */
#define RSZ_PIX_FMT_YUYV  	0 /* y:cb:y:cr */

Driver Configuration

To enable OMAP35x Resizer driver, start Linux Kernel Configuration tool.

$ make menuconfig  ARCH=arm

• Select Device Drivers from the main menu.

...
...
Kernel Features  --->
Boot options  --->
CPU Power Management  --->
Floating point emulation  --->
Userspace binary formats  --->
Power management options  --->
[*] Networking support  --->
Device Drivers  --->
...
...

• Select Multimedia support from the menu.

...
...
Sonics Silicon Backplane  --->
Multifunction device drivers  --->
[*] Voltage and Current Regulator Support  --->
<*> Multimedia support  --->
Graphics support  --->
<*> Sound card support  --->
[*] HID Devices  --->
[*] USB support  --->
...
...

• Select Video For Linux from the menu.

...

...
 *** Multimedia core support ***
<*>   Video For Linux
[*]     Enable Video For Linux API 1 (DEPRECATED)
< >   DVB for Linux
...
...

• Select Video capture adapters from the same menu. Press <ENTER> to enter the corresponding sub-menu.

...
...
[ ]   Customize analog and hybrid tuner modules to build  --->
[*]   Video capture adapters  --->
[ ]   Radio Adapters  --->
[ ]   DAB adapters
...
...

• Select OMAP ISP Resizer from the menu.

...
...
<*>   OMAP 3 Camera support
< >   OMAP ISP Previewer
<*>   OMAP ISP Resizer
<*>   SoC camera support 
...
...

Sample Application Flow

This section shows application flow diagram for resizer application.

Resizer Sample Application Flow