Fujimoto, Takayuki (Kawasaki, JP)

Plaque It! Sponsored by: Flash of Genius

12/122286

09/04/2008

05/16/2008

Images are available in PDF form when logged in. To view PDFs, Login  or  Create Account (Free!)

Click for automatic bibliography generation

FUJITSU LIMITED (Kawasaki, JP)

382/296

G06K9/32

STAAS & HALSEY LLP (SUITE 700, 1201 NEW YORK AVENUE, N.W., WASHINGTON, DC, 20005, US)

What is claimed is:

1. An image transformation apparatus, comprising: a coordinate transformation unit configured to make a transformation to transform a projected image that is projected on a screen by an image projection apparatus having an optical axis tilted by an arbitrary angle with respect to the screen, into an image substantially equivalent to an image that is projected on the screen in a state where the optical axis of the image projection apparatus is perpendicular to the screen; and a luminance value substituting unit configured to substitute luminance values of coordinates before the transformation as luminance values corresponding to coordinate values after the transformation.

2. The image transformation apparatus as claimed in claim 1, wherein said coordinate transformation unit comprises: a part configured to carry out a −γ rotation process to rotate coordinates (X, Y) to coordinates (X′, Y′) using a formula $\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\end{array}\right)=\left(\begin{array}{cc}\cos \left(-{\gamma }^{\prime }\right)& -\sin \left(-{\gamma }^{\prime }\right)\\ \sin \left(-{\gamma }^{\prime }\right)& \cos \left(-{\gamma }^{\prime }\right)\end{array}\right)\left(\begin{array}{c}X\\ Y\end{array}\right);$ a part configured to carry out a tilt transformation process to transform the coordinates (X′, Y′) into coordinates (x′, y′) using a formula $\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right)=\frac{1}{1\cos \alpha +X^{\prime }\sin \alpha }\left(\begin{array}{c}X^{\prime }\\ Y^{\prime }\cos \alpha \end{array}\right);\mathrm{and}$ a part configured to carry out a β rotation process to rotate the coordinates (x′, y′) to coordinates (x, y) using a formula $\left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{cc}\cos \beta & -\sin \beta \\ \sin \beta & \cos \beta \end{array}\right)\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right),$ where α denotes a tilt angle of the optical axis of the image projection apparatus with respect to an axis perpendicular to the screen, β denotes an angle formed by a straight line that is a projection of the optical axis on the screen and a reference axis (x-axis) of the screen, γ denotes an angle formed by a tilt direction of the image projection apparatus on the projected image and a reference axis (X-axis) on the projected image, (X, Y) denotes the coordinates on the projected image, and (x, y) denotes the coordinates on the screen, wherein the formulas are successively calculated to obtain the coordinates (x, y) before the transformation corresponding to the coordinates (X, Y) of the projected image after the transformation.

3. The image transformation apparatus as claimed in claim 2, wherein said coordinate transformation unit comprises: a part configured to carry out the −γ rotation process to rotate the coordinates (X, Y) to the coordinates (X′, Y′) using a formula $\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\end{array}\right)=\left(\begin{array}{cc}\cos \left(-\gamma \right)& -\sin \left(-\gamma \right)\\ \sin \left(-\gamma \right)& \cos \left(-\gamma \right)\end{array}\right)\left(\begin{array}{c}I-I_0\\ J_0-J\end{array}\right)\mathrm{dp};$ a part configured to carry out the tilt transformation process to transform the coordinates (X′, Y′) to the coordinates (x′, y′) using a formula $\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right)=\frac{1}{1\cos \alpha +X^{\prime }\sin \alpha }\left(\begin{array}{c}X^{\prime }\\ Y^{\prime }\cos \alpha \end{array}\right);$ a part configured to carry out the β rotation process to rotate the coordinates (x′, y′) to coordinates (x, y) using a formula $\left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{cc}\cos \beta & -\sin \beta \\ \sin \beta & \cos \beta \end{array}\right)\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right);\mathrm{and}$ a part configured to carry out an inverse transformation process to inverse-transform the coordinates (x, y) to coordinates (i′, j′) using a formula $\left(\begin{array}{c}{i}^{\prime }\\ j^{\prime }\end{array}\right)=\left(\begin{array}{c}{i}_0+x/\mathrm{dp}\\ j_0-y/\mathrm{dp}\end{array}\right),$ where dp denotes a distance between two adjacent pixels on the screen, (I′, J′) denotes coordinates of pixels on the projected image, (i, j) denotes coordinates of pixels of the projected image after the transformation, and (I₀, J₀) and (i₀, j₀) denote centers of the respective projected images before and after the transformation, wherein the formulas are successively calculated to obtain the coordinates (I′, J′) before the transformation corresponding to the coordinates (i, j) of the pixels of the projected image after the transformation.

4. The image transformation apparatus as claimed in claim 3, wherein said luminance value substituting unit comprises: a part configured to obtain i and j from a formula $\left(\begin{array}{c}i\\ j\end{array}\right)=\left(\begin{array}{c}\left[i^{\prime }\right]\\ \left[j^{\prime }\right]\end{array}\right)$ , where 4 pixel coordinates actually existing in a neighborhood of the coordinates (i′, j′) are (i, j), (i+1, j), (i, j+1) and (i+1, j+1), and [ ] denotes a Gauss symbol; and a part configured to obtain a weighted average of pixel values of the neighboring 4 pixels using a formula
p(I, J)=(I+j−j′){(I+i−i′)P(i, j)+(i′−i)P(i+1, j)]+(j′−j){(I+i−i′)P(i, j+1)+(i′−i)P(i+1, j+1)}, where P(i, j) denotes the luminance value of the original image, and p(I, J) denotes the luminance value after the transformation.

5. An image transformation apparatus, comprising: a coordinate transformation unit configured to make a transformation to transform a projected image of an object that is projected on a screen by an image projection apparatus having an optical axis tilted by an arbitrary angle with respect to the screen, with the image projection apparatus or the object tilted by an angle with respect to the screen, into an image substantially equivalent to an image that is projected on the screen in a state where the optical axis of the image projection apparatus is perpendicular to the screen; and a luminance value substituting unit configured to substitute luminance values of coordinates before the transformation as luminance values corresponding to coordinate values after the transformation.

6. The image transformation apparatus as claimed in claim 5, wherein said coordinate transformation unit comprises: a part configured to obtain coordinates (x, y) before the transformation corresponding to coordinates (X, Y) of the projected image after the transformation, based on α, β and γ, where: α denotes a tilt angle of the optical axis of the image projection apparatus with respect to an axis perpendicular to the screen, β denotes an angle formed by a straight line that is a projection of the optical axis on the screen and a reference axis (x-axis) of the screen, γ denotes an angle formed by a tilt direction of the image projection apparatus on the projected image and a reference axis (X-axis) on the projected image, (X, Y) denotes the coordinates on the projected image, and (x, y) denotes the coordinates on the screen.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Divisional application of application Ser. No. 10/917,540 filed Aug. 13, 2004, currently pending, which is a Continuation application filed under 35 U.S.C. 111(a) claiming the benefit under 35 U.S.C. 120 and 365(a) of a PCT International Application No. PCT/JP02/01314 filed Feb. 15, 2002, in the Japanese Patent Office.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to image transformation methods and apparatuses, image recognition apparatuses, robot control apparatus and image projection apparatuses, and more particularly to image transformation method and apparatus for obtaining position coordinates by transforming image information picked up (or imaged) by an image pickup device (or imaging device), an image recognition apparatus for recognizing an image based on the position coordinates obtained by such an image transformation method, a robot control apparatus which uses such an image recognition apparatus, and an image projection apparatus which uses such an image transformation method.

2. Description of the Related Art

When assembling parts, devices and the like using a robot, it is necessary to accurately control movements of each part of an arm, a hand and the like of the robot. For example, when mounting a part on a substrate, the position of the robot hand with respect to the substrate can be detected based on an image which is picked up by a camera that is provided on the robot hand. However, as the movements of the robot hand become complex, the camera cannot always pickup the image in a direction perpendicular to the substrate surface. The robot hand may be moved to a position above the substrate surface from an outer side the substrate in a direction tilted with respect to the substrate surface, for example. In such a case, it is difficult to accurately detect various shapes within the image and the position of the robot hand above the substrate surface, based on the image which is picked up from the direction tilted with respect to the substrate surface. This is because, an angle formed by an optical axis of the camera and the substrate surface is not always constant.

It is possible to provide marks on the substrate surface and use the marks as references within the picked up image, so as to improve to a certain extent the shape detection accuracy with which the various shapes within the image are detected and the position detection accuracy with which the position of the robot hand. However, in order to improve the shape detection accuracy with respect to the various shapes within the image and the position detection accuracy with respect to the robot hand, it is necessary to increase the number of marks. But depending on the number and size of the parts to be mounted on the substrate, there is a limit to the number of marks that may be provided on the substrate, and further, the marks may become hidden by the mounted parts.

If the shape detection accuracy with respect to the various shapes within the picked up image is poor, the image recognition accuracy using the picked up image deteriorates. In addition, when the shape detection accuracy with respect to the various shapes within the picked up image and the position detection accuracy with respect to the robot hand are poor, the mounting position accuracy of the parts mounted on the substrate deteriorates, and the robot hand may hit the substrate or the parts on the substrate when the robot hand is moved. For this reason, there are demands to improve the shape detection accuracy with respect to the various shapes within the picked up image and the position detection accuracy with respect to the robot hand, that is, to accurately detect the position coordinates within the picked up image.

On the other hand, in the image projection apparatuses such as a projector, an image is projected onto a screen or the like. But it is difficult to set up the image projection apparatus so that an optical axis of the image projection apparatus is perpendicular to the screen, and it may be impossible in some cases due to space restrictions. When the optical axis of the image projection apparatus is tilted with respect to the screen, the image projected onto the screen becomes distorted, and for example, an originally rectangular shape may be distorted and projected as a trapezoidal shape on the screen. The distortion of the image projected on the screen not only deteriorates the picture quality, but also makes the image awkward, unnatural and sometimes unpleasant to the viewer.

Conventionally, a system has been proposed to correct a vertical (up-and-down direction) distortion in the image projected on the screen by the so-called trapezoidal correction to correct the trapezoidal shape generated by the distortion into the rectangular shape. In addition, another system has been proposed to correct a horizontal (right-and-left direction) distortion in the image projected on the screen by the so-called trapezoidal correction to correct the trapezoidal shape generated by the distortion into the rectangular shape. The latter system is employed in a side-shot projector VPL-HS1 (product name Cineza) manufactured by Sony Corporation of Japan, for example.

However, the conventional systems which use the trapezoidal correction cannot cope with a case where the optical axis of the image projection apparatus is tilted by an arbitrary angle in both the vertical direction and in the horizontal direction with respect to the screen. That is, the conventional system cannot cope with the case where the projection is at an arbitrary angle with respect to the screen. For this reason, there is a limit to improving the picture quality on the screen.

SUMMARY OF THE INVENTION

Accordingly, it is a general object of the present invention to provide a novel and useful image transformation method and apparatus, image recognition apparatus, robot control apparatus and image projection apparatus, in which the problems described above are eliminated.

A first specific object of the present invention is to provide image transformation method and apparatus which can transform a picked up image into an image from which position coordinates within the picked up image are accurately detectable regardless of the image pickup angle, and an image recognition apparatus for recognizing an image based on the position coordinates obtained by the image transformation method, and a robot control apparatus which uses such an image recognition apparatus.

A second specific object of the present invention is to provide an image projection apparatus which can prevent distortion from being generated within a projected image or a displayed image on a display, regardless of a projection angle or an angle of view.

Still another object of the present invention is to provide an image transformation method to be implemented by a computer to carry out an image transformation process, comprising a coordinate transformation step making a transformation to transform a picked up image that is picked up by a pickup unit having an optical axis tilted by an arbitrary angle with respect to a reference plane, into an image substantially equivalent to a picked up image that is picked up by the pickup unit in a state where the optical axis of the pickup unit is perpendicular to the reference plane; and a luminance value substituting step substituting luminance values of coordinates before the transformation as luminance values corresponding to coordinate values after the transformation. According to the image transformation method of the present invention, it is possible to realize the first object described above.

The coordinate transformation step may comprise:

carrying out a −β rotation process to rotate coordinates (x, y) to coordinates (x′, y′) using a formula

$\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right)=\left(\begin{array}{cc}\cos \left(-\beta \right)& -\sin \left(-\beta \right)\\ \sin \left(-\beta \right)& \cos \left(-\beta \right)\end{array}\right)\left(\begin{array}{c}x\\ y\end{array}\right);$

carrying out a tilt transformation process to transform the coordinates (x′, y′) into coordinates (X′, Y′) using a formula

$\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\end{array}\right)=\frac{1}{1-x^{\prime }\sin \alpha }\left(\begin{array}{c}{x}^{\prime }\cos \alpha \\ y^{\prime }\end{array}\right);\mathrm{and}$

carrying out a γ rotation process to rotate the coordinates (X′, Y′) to coordinates (X, Y) using a formula

$\left(\begin{array}{c}X\\ Y\end{array}\right)=\left(\begin{array}{cc}\cos \gamma & -\sin \gamma \\ \sin \gamma & \cos \gamma \end{array}\right)\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\end{array}\right),$

where α denotes a tilt angle of the optical axis of the image pickup unit with respect to an axis perpendicular to the reference plane, β denotes an angle formed by a straight line that is a projection of the optical axis on the reference plane and a reference axis (x-axis) of the reference plane, γ denotes an angle formed by a tilt direction of the image pickup unit on the picked up image and a reference axis (X-axis) on the picked up image, (X, Y) denotes the coordinates on the picked up image, and (x, y) denotes the coordinates on the reference plane,

wherein the formulas are successively calculated to obtain the coordinates (X, Y) before the transformation corresponding to the coordinates (x, y) of the picked up image after the transformation.

Further, the coordinate transformation step may comprise:

carrying out the −β rotation process to rotate the coordinates (x, y) to the coordinates (x′, y′) using a formula

$\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right)=\left(\begin{array}{cc}\cos \left(-\beta \right)& -\sin \left(-\beta \right)\\ \sin \left(-\beta \right)& \cos \left(-\beta \right)\end{array}\right)\left(\begin{array}{c}i-i_0\\ j_0-j\end{array}\right)\mathrm{dp};$

carrying out the tilt transformation process to transform the coordinates (x′, y′) to the coordinates (X′, Y′) using a formula

$\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\end{array}\right)=\frac{1}{1-x^{\prime }\sin \alpha }\left(\begin{array}{c}{x}^{\prime }\cos \alpha \\ y^{\prime }\end{array}\right);$

carrying out the γ rotation process to rotate the coordinates (X′, Y′) to coordinates (X, Y) using a formula

$\left(\begin{array}{c}X\\ Y\end{array}\right)=\left(\begin{array}{cc}\cos \gamma & -\sin \gamma \\ \sin \gamma & \cos \gamma \end{array}\right)\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\end{array}\right);\mathrm{and}$

carrying out an inverse transformation process to inverse-transform the coordinates (X, Y) to coordinates (I′, J′) using a formula

$\left(\begin{array}{c}{I}^{\prime }\\ J^{\prime }\end{array}\right)=\left(\begin{array}{c}{I}_0+X/\mathrm{dp}\\ J_0-Y/\mathrm{dp}\end{array}\right),$

where dp denotes a distance between two adjacent pixels on the reference plane, (I′, J′) denotes coordinates of pixels on the picked up image, (i, j) denotes coordinates of pixels of the picked up image after the transformation, and (I ₀ , J ₀ ) and (i ₀ , j ₀ ) denote centers of the respective picked up images before and after the transformation,

wherein the formulas are successively calculated to obtain the coordinates (I′, J′) before the transformation corresponding to the coordinates (i, j) of the pixels of the picked up image after the transformation.

The luminance value substituting step may comprise:

obtaining I and J from a formula

$\left(\begin{array}{c}I\\ J\end{array}\right)=\left(\begin{array}{c}\left[I^{\prime }\right]\\ \left[J^{\prime }\right]\end{array}\right)$

where 4 pixel coordinates actually existing in a neighborhood of the coordinates (I′, J′) are (I, J), (I+1, J), (I, J+1) and (I+1, J+1), and [ ] denotes a Gauss symbol; and obtaining a weighted average of pixel values of the neighboring 4 pixels using a formula p(i, j)=(I+J−J′){(I+I−I′)P(I, J)+(I′−I)P(I+1, J)}+(J′−J){(I+I−I′)P(I, J+1)+(I′−I)P(I+1, J+1)},

where P(I, J) denotes the luminance value of the original image, and p(i, j) denotes the luminance value after the transformation.

A further object of the present invention is to provide an image transformation apparatus comprising a coordinate transformation unit configured to make a transformation to transform a picked up image that is picked up by a pickup unit having an optical axis tilted by an arbitrary angle with respect to a reference plane, into an image substantially equivalent to a picked up image that is picked up by the pickup unit in a state where the optical axis of the pickup unit is perpendicular to the reference plane; and a luminance value substituting unit configured to substitute luminance values of coordinates before the transformation as luminance values corresponding to coordinate values after the transformation. According to the image transformation apparatus of the present invention, it is possible to realize the first object described above.

The coordinate transformation unit may comprise:

a part configured to carry out a −β rotation process to rotate coordinates (x, y) to coordinates (x′, y′) using a formula

$\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right)=\left(\begin{array}{cc}\cos \left(-\beta \right)& -\sin \left(-\beta \right)\\ \sin \left(-\beta \right)& \cos \left(-\beta \right)\end{array}\right)\left(\begin{array}{c}x\\ y\end{array}\right);$

a part configured to carry out a tilt transformation process to transform the coordinates (x′, y′) into coordinates (X′, Y′) using a formula

$\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\end{array}\right)=\frac{1}{1-x^{\prime }\sin \alpha }\left(\begin{array}{c}{x}^{\prime }\cos \alpha \\ y^{\prime }\end{array}\right);\mathrm{and}$

a part configured to carry out a γ rotation process to rotate the coordinates (X′, Y′) to coordinates (X, Y) using a formula

$\left(\begin{array}{c}X\\ Y\end{array}\right)=\left(\begin{array}{cc}\cos \gamma & -\sin \gamma \\ \sin \gamma & \cos \gamma \end{array}\right)\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\end{array}\right),$

where α denotes a tilt angle of the optical axis of the image pickup unit with respect to an axis perpendicular to the reference plane, β denotes an angle formed by a straight line that is a projection of the optical axis on the reference plane and a reference axis (x-axis) of the reference plane, γ denotes an angle formed by a tilt direction of the image pickup unit on the picked up image and a reference axis (X-axis) on the picked up image, (X, Y) denotes the coordinates on the picked up image, and (x, y) denotes the coordinates on the reference plane,

wherein the formulas are successively calculated to obtain the coordinates (X, Y) before the transformation corresponding to the coordinates (x, y) of the picked up image after the transformation.

Further, the coordinate transformation unit may comprise:

a part configured to carry out the −β rotation process to rotate the coordinates (x, y) to the coordinates (x′, y′) using a formula

$\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right)=\left(\begin{array}{cc}\cos \left(-\beta \right)& -\sin \left(-\beta \right)\\ \sin \left(-\beta \right)& \cos \left(-\beta \right)\end{array}\right)\left(\begin{array}{c}i-i_0\\ j_0-j\end{array}\right)\mathrm{dp};$

a part configured to carry out the tilt transformation process to transform the coordinates (x′, y′) to the coordinates (X′, Y′) using a formula

$\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\end{array}\right)=\frac{1}{1-x^{\prime }\sin \alpha }\left(\begin{array}{c}{x}^{\prime }\cos \alpha \\ y^{\prime }\end{array}\right);$

a part configured to carry out the γ rotation process to rotate the coordinates (X′, Y′) to coordinates (X, Y) using a formula

$\left(\begin{array}{c}X\\ Y\end{array}\right)=\left(\begin{array}{cc}\cos \gamma & -\sin \gamma \\ \sin \gamma & \cos \gamma \end{array}\right)\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\end{array}\right);\mathrm{and}$

a part configured to carry out an inverse transformation process to inverse-transform the coordinates (X, Y) to coordinates (I′, J′) using a formula

$\left(\begin{array}{c}{I}^{\prime }\\ J^{\prime }\end{array}\right)=\left(\begin{array}{c}{I}_0+X/\mathrm{dp}\\ J_0-Y/\mathrm{dp}\end{array}\right),$

where dp denotes a distance between two adjacent pixels on the reference plane, (I′, J′) denotes coordinates of pixels on the picked up image, (i, j) denotes coordinates of pixels of the picked up image after the transformation, and (I ₀ , J ₀ ) and (i ₀ , j ₀ ) denote centers of the respective picked up images before and after the transformation,

wherein the formulas are successively calculated to obtain the coordinates (I′, J′) before the transformation corresponding to the coordinates (i, j) of the pixels of the picked up image after the transformation.

The luminance value substituting unit may comprise:

a part configured to obtain I and J from a formula

$\left(\begin{array}{c}I\\ J\end{array}\right)=\left(\begin{array}{c}\left[I^{\prime }\right]\\ \left[J^{\prime }\right]\end{array}\right)$

where 4 pixel coordinates actually existing in a neighborhood of the coordinates (I′, J′) are (I, J), (I+1, J), (I, J+1) and (I+1, J+1), and [ ] denotes a Gauss symbol; and

a part configured to obtain a weighted average of pixel values of the neighboring 4 pixels using a formula p(i, j)=(I+J−J′){(I+I−I′)P(I, J)+(I′−I)P(I+1, J))+(J′−J){(i+I−I′)P(I, J+1)+(I′−I)P(I+1, J+1)},

where P(I, J) denotes the luminance value of the original image, and p(i, j) denotes the luminance value after the transformation.

Another object of the present invention is to provide an image recognition apparatus comprising an image recognition unit configured to carry out an image recognition process based on an image after the transformation output from the image transformation apparatus described above. According to the image recognition apparatus of the present invention, it is possible to realize the first object described above.

Still another object of the present invention is to provide a robot control apparatus comprising a control unit configured to control operations of a robot based on an image recognition result output from the image recognition apparatus described above. According to the robot control apparatus of the present invention, it is possible to realize the first object described above.

A further object of the present invention is to provide an image transformation apparatus comprising a coordinate transformation unit configured to make a transformation to transform a projected image that is projected on a screen by an image projection apparatus having an optical axis tilted by an arbitrary angle with respect to the screen, into an image substantially equivalent to an image that is projected on the screen in a state where the optical axis of the image projection apparatus is perpendicular to the screen; and a luminance value substituting unit configured to substitute luminance values of coordinates before the transformation as luminance values corresponding to coordinate values after the transformation. According to the image transformation apparatus of the present invention, it is possible to realize the second object described above.

The coordinate transformation unit may comprise:

a part configured to carry out a −γ rotation process to rotate coordinates (X, Y) to coordinates (X′, Y′) using a formula

$\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\end{array}\right)=\left(\begin{array}{cc}\cos \left(-\gamma \right)& -\sin \left(-\gamma \right)\\ \sin \left(-\gamma \right)& \cos \left(-\gamma \right)\end{array}\right)\left(\begin{array}{c}I-I_0\\ J_0-J\end{array}\right)\mathrm{dp};$

a part configured to carry out a tilt transformation process to transform the coordinates (X′, Y′) into coordinates (x′, y′) using a formula

$\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right)=\frac{1}{1\cos \alpha +X^{\prime }\sin \alpha }\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\cos \alpha \end{array}\right);\mathrm{and}$

a part configured to carry out a β rotation process to rotate the coordinates (x′, y′) to coordinates (x, y) using a formula

$\left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{cc}\cos \beta & -\sin \beta \\ \sin \beta & \cos \beta \end{array}\right)\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right)$

where α denotes a tilt angle of the optical axis of the image projection apparatus with respect to an axis perpendicular to the screen, β denotes an angle formed by a straight line that is a projection of the optical axis on the screen and a reference axis (x-axis) of the screen, γ denotes an angle formed by a tilt direction of the image projection apparatus on the projected image and a reference axis (X-axis) on the projected image, (X, Y) denotes the coordinates on the projected image, and (x, y) denotes the coordinates on the screen,

wherein the formulas are successively calculated to obtain the coordinates (x, y) before the transformation corresponding to the coordinates (X, Y) of the projected image after the transformation.

The coordinate transformation unit may comprise:

a part configured to carry out the −γ rotation process to rotate the coordinates (X, Y) to the coordinates (X′, Y′) using a formula

$\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\end{array}\right)=\left(\begin{array}{cc}\cos \left(-\gamma \right)& -\sin \left(-\gamma \right)\\ \sin \left(-\gamma \right)& \cos \left(-\gamma \right)\end{array}\right)\left(\begin{array}{c}I-I_0\\ J_0-J\end{array}\right)\mathrm{dp};$

a part configured to carry out the tilt transformation process to transform the coordinates (X′, Y′) to the coordinates (x′, y′) using a formula

$\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right)=\frac{1}{1\cos \alpha +X^{\prime }\sin \alpha }\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\cos \alpha \end{array}\right);$

a part configured to carry out the β rotation process to rotate the coordinates (x′, y′) to coordinates (x, y) using a formula

$\left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{cc}\cos \beta & -\sin \beta \\ \sin \beta & \cos \beta \end{array}\right)\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right);\mathrm{and}$

a part configured to carry out an inverse transformation process to inverse-transform the coordinates (x, y) to coordinates (i′, j′) using a formula

$\left(\begin{array}{c}{i}^{\prime }\\ j^{\prime }\end{array}\right)=\left(\begin{array}{c}{i}_0+x/\mathrm{dp}\\ j_0-y/\mathrm{dp}\end{array}\right),$

where dp denotes a distance between two adjacent pixels on the screen, (I′, J′) denotes coordinates of pixels on the projected image, (i, j) denotes coordinates of pixels of the projected image after the transformation, and (I ₀ , J ₀ ) and (i ₀ , j ₀ ) denote centers of the respective projected images before and after the transformation,

wherein the formulas are successively calculated to obtain the coordinates (I′, J′) before the transformation corresponding to the coordinates (i, j) of the pixels of the projected image after the transformation.

The luminance value substituting unit may comprise:

a part configured to obtain i and j from a formula

$\left(\begin{array}{c}i\\ j\end{array}\right)=\left(\begin{array}{c}\left[i^{\prime }\right]\\ \left[j^{\prime }\right]\end{array}\right)$

where 4 pixel coordinates actually existing in a neighborhood of the coordinates (i′, j′) are (i, j), (i+1, j), (i, j+1) and (i+1, j+1), and [ ] denotes a Gauss symbol; and

a part configured to obtain a weighted average of pixel values of the neighboring 4 pixels using a formula p(I, J)=(I+j−j′){(I+i−i′)P(i, j)+(i′−i)P(i+1, j)}+(j′−j){(I+i−i′)P(i, j+1)+(i′−i)P(i+1, j+1)},

where P(i, j) denotes the luminance value of the original image, and p(I, J) denotes the luminance value after the transformation.

Another object of the present invention is to provide an image transformation method to be implemented by a computer to carry out an image transformation process, comprising, when viewing an image displayed on a display from a viewing position that is oblique to a position perpendicular to the display, carrying out an image transformation process with respect to the image to be displayed on the display depending on an angle of the viewing position with respect to the display. According to the image transformation method of the present invention, it is possible to achieve the second object described above.

Still another object of the present invention is to provide an image transformation method to be implemented by a computer to carry out an image transformation process, comprising, when viewing an image that is projected on a screen by an image projection apparatus located at a projecting position that is oblique to a position perpendicular to the screen, from a viewing position that is oblique to the position perpendicular to the screen, carrying out an image transformation process with respect to the image to be projected on the screen depending on an angle of the projecting position with respect to the screen and an angle of the viewing position with respect to the screen. According to the image transformation method of the present invention, it is possible to achieve the second object described above.

Other objects and further features of the present invention will be apparent from the following detailed description when read in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a system block diagram showing an embodiment of a robot control apparatus according to the present invention;

FIG. 2 is a diagram showing an important part of a robot;

FIG. 3 is a diagram showing a relationship of a camera and a reference plane in an initial state;

FIG. 4 is a diagram showing a state where the camera is tilted in a positive direction of an x-axis of the reference plane;

FIG. 5 is a diagram for explaining a movement of a point on the reference plane in an x-direction;

FIG. 6 is a diagram for explaining a movement of a point on the reference plane in a y-direction;

FIG. 7 is a diagram for explaining a tilt direction of the camera with respect to the reference plane;

FIG. 8 is a diagram for explaining a tilt direction of the camera on a picked up image;

FIG. 9 is a diagram for explaining coordinates of the picked up image;

FIG. 10 is a diagram showing coordinates of 4 pixels actually existing in neighborhoods of a coordinate (I′, J′);

FIG. 11 is a flow chart for explaining an operation of an image transformation part;

FIG. 12 is a flow chart for explaining a luminance value substituting process;

FIG. 13 is a diagram for explaining a case where a hand is tilted;

FIG. 14 is a diagram for explaining a case where a plurality of hands are tilted;

FIG. 15 is a diagram for explaining an image transformation process;

FIG. 16 is a diagram for explaining a distortion of an image projected on a screen by an image projection apparatus;

FIG. 17 is a diagram for explaining a reverse image transformation process applicable to the image projection apparatus;

FIG. 18 is a diagram for explaining a case where the image projection apparatus is tilted to a right side of the screen;

FIG. 19 is a diagram for explaining a case where the image projection apparatus is tilted to a top side of the screen;

FIG. 20 is a system block diagram showing a structure of an embodiment of the image projection apparatus;

FIG. 21 is a diagram for explaining images projected on the screen;

FIG. 22 is a diagram showing a case where the hand is driven with respect to an arm by 3 motors;

FIG. 23 is a diagram for explaining a relationship of vectors;

FIG. 24 is a diagram for explaining a case where the image projection apparatus is tilted upwards by an angle θx and tilted rightwards by an angle θy with respect to the screen;

FIG. 25 is a diagram for explaining a relationship of various angles shown in FIG. 24; and

FIG. 26 is a diagram for explaining a positional relationship of the image projection apparatus, the screen and human eyes.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

A description will be given of embodiments of the image transformation method and apparatus, the image recognition apparatus, the robot control apparatus and the image projection apparatus according to the present invention, by referring to the drawings.

FIG. 1 is a system block diagram showing an embodiment of the robot control apparatus according to the present invention. This embodiment of the robot control apparatus employs an embodiment of the image transformation method according to the present invention, an embodiment of the image transformation apparatus according to the present invention, and an embodiment of the image recognition apparatus according to the present invention.

The robot control apparatus shown in FIG. 1 includes a control part 1 , an image transformation part 2 , an image recognition part 3 and a measuring part 4 . A camera 13 and a motor group 20 are provided in a robot which will be described later in conjunction with FIG. 2. The control part 1 has a known hardware structure made up of a processor such as a CPU, and a memory for storing computer programs to be executed by the processor and various kinds of data including intermediate results of operations carried out by the processor. The memory may be coupled externally to the control part 1 , and may be formed by a storage unit such as a disk drive, a semiconductor memory device such as a RAM, and the like. The control part 1 controls the entire operation of the robot control apparatus. As will be described later, the image transformation part 2 transforms an image picked up by the camera 13 into an image from which the position coordinates within the picked up image are accurately detectable. The image recognition part 3 recognizes the picked up image, based on the transformed image obtained by the image transformation part 2 . The image recognition part 3 itself may employ known structure and algorithm, but is characterized in that the image recognition is carried out based on the transformed image obtained by the image transformation part 2 . The measuring part 4 measures currents, voltages and the like that are detected by the robot. The control part 1 carries out various kinds of tests with respect to an object 32 shown in FIG. 2, based on the currents, voltages and the like that are measured by the measuring part 4 . The control part 1 controls each of the motors forming the motor group 20 , based on outputs of the image transformation part 2 and the image recognition part 3 with respect to the input from the camera 13 , and controls movements of each part of the robot with respect to the object 32 that is placed on a stage 31 shown in FIG. 2.

The image transformation part 2 employs this embodiment of the image transformation method, and forms this embodiment of the image transformation apparatus. In addition, the image recognition part 3 , together with the image transformation part 2 , forms this embodiment of the image recognition apparatus.

FIG. 2 is a diagram showing an important part of the robot. The robot includes an arm 11 , a hand 12 , the camera 13 , pins 14 , and motors 21 , 22 and 23 . The arm 11 has a known structure and is movable in an arbitrary direction. The motors 21 , 22 and 23 form the motor group 20 , and move the hand 12 in an arbitrary direction with respect to the arm 11 . The motor 21 drives the hand 12 within an angular range θ 1 , the motor 22 drives the hand 12 within an angular range θ 2 , and the motor 23 drives the hand 12 within an angular range θ 3 . The camera 13 picks up an image viewed from the hand 12 and sends the picked up image to the control part 1 , so as to enable the position of the hand 12 to be detected. The camera 13 is not limited to a specific type, and may be formed by a CCD camera or the like. The pins 14 contact electrodes of the object 32 which is placed on the stage 31 , and is used to apply currents or voltages to the object 32 and to detect currents and voltages generated in response to the applied currents or voltages. For example, the object 32 is a circuit board.

In this embodiment, it is assumed for the sake of convenience that the robot tests the object 32 , but the usage of the robot is not limited to such. For example, clamps or the like may be provided on the hand 12 in place of the pins 14 , so that the clamps may mount parts on the object 32 . In other words, the robot simply needs to have a structure including an image pickup means (or imaging means) such as the camera 13 , and controlled based on an image picked up by the image pickup means.

Next, a description will be given of an image transformation process of the image transformation part 2 .

First, a state is considered where the camera 13 is located at a position separated by a distance I from an origin on a target plane (hereinafter referred to as a reference plane RP), such as a top surface of the stage 31 , in a direction perpendicular to the reference plane RP, and parallel to a normal to the reference plane RP. In addition, the position of the camera 13 is adjusted by rotating the camera 13 parallel to the reference plane RP before fixing the position, so that an x-axis on an image picked up by the camera 13 matches an x-axis on the reference plane RP. FIG. 3 is a diagram showing a relationship of the camera 13 and the reference plane RP in this initial state.

FIG. 4 is a diagram showing a state where the camera 13 is tilted by an angle (tilt angle) α in a positive direction of the x-axis of the reference plane RP, from the initial state shown in FIG. 3. In this case, consideration will be given of how an arbitrary point (x, y) on the reference plane RP moves on the picked up image. Since tilting the camera 13 with respect to the reference plane RP is equivalent to tilting the reference plane RP with respect to the camera 13 in an opposite direction, a description will be given of a case where the reference plane RP is tilted with respect to the camera 13 , for the sake of convenience.

FIG. 5 is a diagram for explaining a movement of a point on the reference plane RP in an x-direction. When the reference plane RP is tilted by the angle α, a point x on the reference plane RP appears, on the picked up image, as if the point x moved to an original point X on the reference plane RP before the reference plane RP was tilted by the angle α. Accordingly, an angle θ formed by an imaginary line connecting the camera 13 and the point X, and a z-axis in FIG. 5, satisfies the following two formulas.

tan θ= X/I

tan θ=( x cos α)/( I−x sin α)

The following formula (1) can be obtained when the relationship of X and x is obtained from these two formulas.

X =( Ix cos α)/( I−x sin α) (1)

The following inverse transformation formula (2) can be obtained by carrying out the following transformation with respect to the formula (1).

X ( I −x sin α)= Ix cos α

x ( I cos α+ X sin α)= XI

x=XI/ ( I cos α+ X sin α) (2)

FIG. 6 is a diagram for explaining a movement of a point on the reference plane RP in the y-direction. When the reference plane RP is tilted by the angle α, a point y on the reference plane RP appears, on the picked up image, as if the point y moved to an original point Y on the reference plane RP before the reference plane RP was tilted by the angle α. Accordingly, an angle β formed by an imaginary line connecting the camera 13 and the point Y, and the z-axis in FIG. 6, satisfies the following two formulas.

tan β= Y/I

tan β= y /( I−x sin α)

The following formula (3) can be obtained when the relationship of Y and y is obtained from these two formulas.

Y=Iy/ ( I−x sin α) (3)

The following inverse transformation formula (4) can be obtained by carrying out the following transformation with respect to the formula (3) and substituting the inverse transformation formula (2).

Iy=Y ( I−x sin α)

Iy=Y{I −( XI sin α)/( I cos α+ X sin α)}

y=Y{I cos α/( I cos α+ X sin α)} (4)

The formulas (1) and (2) can be described by the following formula (5), and the formulas (3) and (4) can be described by the following formula (6).

$\begin{array}{cc}\left(\begin{array}{c}X\\ Y\end{array}\right)=\frac{1}{1-x\sin \alpha }\left(\begin{array}{c}x\cos \alpha \\ y\end{array}\right)& \left(5\right)\\ \left(\begin{array}{c}x\\ y\end{array}\right)=\frac{1}{1\cos \alpha +X\sin \alpha }\left(\begin{array}{c}X\\ Y\cos \alpha \end{array}\right)& \left(6\right)\end{array}$

The transformation formulas (5) and (6) stand on the assumption that the tilt direction of the camera 13 , the x-axis of the reference plane RP and the x-axis on the picked up image match, and thus, the transformation formula (5) or (6) is transformed into a general solution for a case where the tilt angle of the camera 13 , the x-axis of the reference plane RP and the x-axis on the picked up image do not match. It is assumed that the tilt direction of the camera 13 is β with respect to the reference plane RP as shown in FIG. 7, and γ on the picked up image as shown in FIG. 8.

In orderto transform a coordinate (X, Y) on the picked up image into a coordinate (x, y) on the reference plane RP using the transformation formula (6), the tilt direction of the camera 13 must be 0 degree (that is, in the position direction of the X-axis), and for this reason, a −γ rotation process is carried out with respect to (X, Y) as may be seen from the following formula (7).

$\begin{array}{cc}\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\end{array}\right)=\left(\begin{array}{cc}\cos \left(-\gamma \right)& -\sin \left(-\gamma \right)\\ \sin \left(-\gamma \right)& \cos \left(-\gamma \right)\end{array}\right)\left(\begin{array}{c}X\\ Y\end{array}\right)& \left(7\right)\end{array}$

In a (X′, Y′) space above, it may be regarded that the camera 13 is tilted in the positive direction of the X′-axis, and the transformation formula (6) can be applied. Hence, the tilt can be transformed according to the following formula (8).

$\begin{array}{cc}\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right)=\frac{1}{1\cos \alpha +X^{\prime }\sin \alpha }\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\cos \alpha \end{array}\right)& \left(8\right)\end{array}$

A (x, y) space in the left term of the formula (8) is not the original (x, y) space but is a coordinate system rotated by the angle β. In order to return the coordinate system to the original (x, y) coordinate system, a −β rotation process with respect to the coordinate system, that is, a +β rotation process with respect to the coordinates is necessary, as may be seen from the following formula (9).

$\begin{array}{cc}\left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{cc}\cos \beta & -\sin \beta \\ \sin \beta & \cos \beta \end{array}\right)\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right)& \left(9\right)\end{array}$

Accordingly, it is possible to transform the (X, Y) coordinate system into the (x, y) coordinate system by successively applying the formulas (7) through (9) described above.

In the description given above it is assumed that the coordinates on the picked up image are continuous values and the scale is 1:1 with respect to the reference plane RP. However, in a case where the picked up image to be treated is described by digital image data, the picked up image is represented by a collection of pixels (picture elements), and the image data are formed by pixel data.

For example, suppose that one picked up image is represented by a two-dimensional array of 640×480 pixels, and dp denotes an actual distance on the reference plane RP corresponding to a distance between two mutually adjacent pixels. In this case, if coordinates of the picked up image are denoted by (I, J) as shown in FIG. 9, and considerations are given on the fact that j is defined with a direction opposite to the usual Y-axis and a center of the picked up image is (319.5, 239.5), pixel coordinates (I, J) corresponding to a point (X, Y) can be described by the following formula (10).

$\begin{array}{cc}\left(\begin{array}{c}X\\ Y\end{array}\right)=\left(\begin{array}{c}I-319.5\\ 239.5-J\end{array}\right)\mathrm{dp}& \left(10\right)\end{array}$

Similarly, pixel coordinates (i′, j′) corresponding to a point (x, y) can be described by the following formula (11), where i′ and j′ denote real numbers which are not necessarily integers.

$\begin{array}{cc}\left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{c}{i}^{\prime }-319.5\\ 239.5-j^{\prime }\end{array}\right)\mathrm{dp}& \left(11\right)\end{array}$

Accordingly, by successively carrying out the inverse transformation of the formulas (10) (7) (8) (9) (11), it is possible to transform the coordinate system (I, J) into the coordinate system (i′, j′). In order to make the conversion result the same as the original image, having pixels arranged in a lattice, i and j may be regarded as integers and a pixel value of a point (i′, j′) in a neighborhood of a point (i, j), that is, a pixel value of a point (I, J) may be employed. In order to improve the accuracy of the image having the pixels arranged in the lattice, it is conceivable to obtain a weighted average of a plurality of neighboring points, but it is difficult to perform such a calculation.

Hence, an inverse transformation to the above transformation, that is, a transformation from the (i, j) coordinate system to a (I′, J′) coordinate system is carried out, and the accuracy of the image having the pixels arranged in the lattice may be improved by regarding a weighted average of 4 points neighboring the point (I′, J′) as a pixel value of the point (i, j). In this case, the formula (11) is described by the integers i and j, and thus, the pixel coordinates (i, j) corresponding to the point (x, y) can be described by the following formula (11A).

$\begin{array}{cc}\left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{c}i-319.5\\ 239.5-j\end{array}\right)\mathrm{dp}& \left(11A\right)\end{array}$

From the inverse transformations of the formulas (11A) and (9) described above, the −β rotation process can be described by the following formula (12).

$\begin{array}{cc}\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right)=\left(\begin{array}{cc}\cos \left(-\beta \right)& -\sin \left(-\beta \right)\\ \sin \left(-\beta \right)& \cos \left(-\beta \right)\end{array}\right)\left(\begin{array}{c}i-319.5\\ 239.5-j\end{array}\right)\mathrm{dp}& \left(12\right)\end{array}$

The inverse transformation of the formula (8) can be written as the following formula (13) using the formula (5), to carry out the transformation of the tilt.

$\begin{array}{cc}\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\end{array}\right)=\frac{1}{1-x^{\prime }\sin \alpha }\left(\begin{array}{c}{x}^{\prime }\cos \alpha \\ y^{\prime }\end{array}\right)& \left(13\right)\end{array}$

From the inverse transformation of the formula (7) above, a γ rotation can be described by the following formula (14).

$\begin{array}{cc}\left(\begin{array}{c}X\\ Y\end{array}\right)=\left(\begin{array}{cc}\cos \gamma & -\sin \gamma \\ \sin \gamma & \cos \gamma \end{array}\right)\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\end{array}\right)& \left(14\right)\end{array}$

In addition, the inverse transformation of the formula (10) can be obtained from the following formula (15).

$\begin{array}{cc}\left(\begin{array}{c}{I}^{\prime }\\ J^{\prime }\end{array}\right)=\left(\begin{array}{c}319.5+X/\mathrm{dp}\\ 239.5-Y/\mathrm{dp}\end{array}\right)& \left(15\right)\end{array}$

By successively calculating the formulas (12) through (15), it is possible to obtain the coordinates (I′, J′) before the transformation and corresponding to the coordinates (i, j) of the image after the transformation, where I′ and J′ are real numbers. If the 4 pixel coordinates actually existing in the neighborhood of the coordinates (I′, J′) are (I, J), (I+1, J), (I, J+1) and (I+1, J+1) as shown in FIG. 10, I and J can be obtained from the following formula (16), where [ ] denotes a Gauss symbol.

$\begin{array}{cc}\left(\begin{array}{c}I\\ J\end{array}\right)=\left(\begin{array}{c}\left[I^{\prime }\right]\\ \left[J^{\prime }\right]\end{array}\right)& \left(16\right)\end{array}$

In this case, if the pixel value (luminance value) of the original image is denoted by P(I, J) and the pixel value (luminance value) after the transformation is denoted by p(i, j), a weighted average of the neighboring 4 pixel values can be described by the following formula (17).

p ( i, j )=( I+J−J ′){( I+I−I ′) P ( I, J )+( I′−I ) P ( I+ 1, J )}+( J′−J ){( I+I−I ′) P ( I, J+ 1)+( I−I ) P ( I+ 1, J+ 1 )} (17)

The formulas (12) through (15) are general solutions, but in cases where the tilt direction of the camera 13 is 0 degree (or 180 degrees) or 90 degrees (or 270 degrees), the formulas can be simplified. In the case where the tilt direction of the camera 13 is 0 degree (or 180 degrees), β=0, and the following formula (12A) may be used in place of the formula (12). In addition, in the case where the optical axis of the camera 13 is perpendicular to the reference plane RP and the camera 13 is simply tilted in the +x-direction or the −x-direction, γ=0 if there is no rotation of the camera 13 about the optical axis, and the formula (14) may be simplified.

$\begin{array}{cc}\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right)=\left(\begin{array}{c}i-319.5\\ 239.5-j\end{array}\right)\mathrm{dp}& \left(12A\right)\end{array}$

Moreover, in the case where the tilt direction of the camera 13 is 90 degrees (or 270 degrees), x and y in the formula (13) may be interchanged and treated similarly to the case where the tilt direction of the camera 13 is 0 degree (or 180 degrees), so as to simplify the formula (13). More particularly, the formula (12A) may be used in place of the formula (12), and the following formula (13A) may be used in place of the formula (13). Furthermore, the formula (14) may be simplified in a simple case where γ=0, similarly as described above.

$\begin{array}{cc}\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\end{array}\right)=\frac{1}{1-y^{\prime }\sin \alpha }\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\cos \alpha \end{array}\right)& \left(13A\right)\end{array}$

FIG. 11 is a flow chart for explaining the operation of the image transformation part 2 . The process shown in FIG. 11 may be carried out by an exclusive hardware made up of a processor or the like forming the image transformation part 2 or, carried out by a software which realizes the functions of the image transformation part 2 by the processor that forms the control part 1 .

In FIG. 11, a step S 1 inputs the image data of the image that is picked up by the camera 13 , and temporarily stores the image data in the memory within the control part 1 , for example. A step S 2 acquires parameters α, β, γ and I described above. The parameters (angles) α, β and γ may be obtained by a method which will be described later. In addition, by presetting a reference distance I _ref with respect to a reference position where the optical axis of the camera 13 becomes perpendicular with respect to the reference plane RP such as the stage 31 , the parameter (distance) I can be obtained based on a moving quantity from the reference distance I _ref . A step S 3 initially sets i and j to i=0 and j=0.

Steps S 4 through S 6 carry out a coordinate transformation process from the point (i, j) to the point (I′, J′). More particularly, the step S 4 uses the formula (12) described above to carry out a −β rotation process, the step S 5 uses the formula (13) to carry out a tilt transformation process, and the step S 6 uses the formula (14) to carry out a γ rotation process.

The step S 7 carries out a luminance value substituting process. FIG. 12 is a flow chart for explaining the luminance value substituting process of the step S 7 . If it is assumed for the sake of convenience that the result of the coordinate transformation process of the steps S 4 through S 6 shown in FIG. 11 is (I′, J′), a step S 71 shown in FIG. 12 decides whether or not 0≦I′<639. If the decision result in the step S 71 is YES, a step S 72 decides whether or not 0≦J′<439. If the decision result in the step S 71 or S 72 is NO, a step S 73 sets P(i, j) to P(i, j)=0, and the process advances to a step S 8 shown in FIG. 11. On the other hand, if the decision result in the step S 72 is YES, a step S 74 calculates the formulas (16) and (17) described above, and the process advances to the step S 8 .

Returning now to the description of FIG. 11, the step S 8 increments i to i=i+1, and a step S 9 decides whether or not i=640. If the decision result in the step S 9 is NO, the process returns to the step S 4 . On the other hand, if the decision result in the step S 9 is YES, a step S 10 sets i to i=0. A step S 11 increments j to j=j+1, and a step S 12 decides whether or not j=480. If the decision result in the step S 12 is NO, the process returns to the step S 4 . If the decision result in the step S 12 is YES, a step S 13 outputs the image data of the transformed image, that is, a transformed image data, and the process ends. The transformed image data is stored in the memory within the control part 1 , for example, and is used for the image recognition by the image recognition part 3 .

As described above, the image recognition part 3 itself may employ known structure and algorithm, but is characterized in that the image recognition is carried out based on the transformed image obtained by the image transformation part 2 . By carrying out a pattern matching or a template matching, for example, in the image recognition part 3 based on the transformed image data, it is possible to accurately obtain the position of the hand 12 with respect to the object 32 , that is, the position of the pins 14 in this embodiment. Accordingly, as shown in a simplified manner in FIG. 13, it becomes possible to tilt the hand 12 and move the tip end above the electrode of the object 32 , for example, so that the hand 12 will not hit an obstructing object 32 a on the object 32 . In addition, as shown in a simplified manner in FIG. 14, in a case where a plurality of hands 12 are provided, it is possible to avoid interference among the hands 12 by tilting the hands 12 .

In the image transformation process described above, the image which is picked up in a state where the camera 13 is tilted with respect to the reference plane RP is transformed into the transformed image which is as if the transformed image were picked up from a position perpendicular with respect to the reference plane RP. Accordingly, in a case where the original shape in the plane is a rectangle shown in FIG. 15( a ), the shape within the image picked up by the camera 13 is a trapezoid shown in FIG. 15( b ), but the shape within the transformed image after the image transformation process is a rectangle shown in FIG. 15( c ) which would enable an accurate image recognition process to be carried out.

Therefore, by carrying out a reverse image transformation process to the image transformation process described above, the present invention may also be applied to the image projection apparatus such as a projector. For example, if the image projection apparatus is not set up at a position such that the optical axis of the image projection apparatus is perpendicular to a screen, the rectangular image shown in FIG. 16( a ) which should originally be projected on the screen becomes a trapezoidal image shown in FIG. 16( b ). In the trapezoidal image shown in FIG. 16( b ), a distortion is generated and the considerable deterioration of the picture quality makes the image on the screen very.awkward, unnatural and sometimes unpleasant to the viewer. Accordingly by carrying out the image transformation process which is in reverse to the image transformation process described above, the original rectangular image shown in FIG. 17( a ) can be transformed into a trapezoidal image shown in FIG. 17( b ) and projected on the screen, so that a rectangular image shown in FIG. 17( c ) having no distortion and a high picture quality is actually projected on the screen and visible by the viewer

A description will now be given of the reverse image transformation process. In the case of the reverse image transformation process, the following formulas (18) through (20) are used in place of the formulas (7) through (9) described above. A −β rotation process is carried out by using the formula (18), a tilt transformation (inverse transformation) process is carried out using the formula (19), and a γ rotation process is carried out using the formula (20). It is assumed for the sake of convenience that the image projection apparatus is tilted from a state separated by a distance I in a direction perpendicular to the screen (reference plane RP).

$\begin{array}{cc}\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right)=\left(\begin{array}{cc}\cos \left(-\beta \right)& -\sin \left(-\beta \right)\\ \sin \left(-\beta \right)& \cos \left(-\beta \right)\end{array}\right)\left(\begin{array}{c}x\\ y\end{array}\right)& \left(18\right)\\ \left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\end{array}\right)=\frac{1}{1-x^{\prime }\sin \alpha }\left(\begin{array}{c}{x}^{\prime }\cos \alpha \\ y^{\prime }\end{array}\right)& \left(19\right)\\ \left(\begin{array}{c}X\\ Y\end{array}\right)=\left(\begin{array}{cc}\cos \gamma & -\sin \gamma \\ \sin \gamma & \cos \gamma \end{array}\right)\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\end{array}\right)& \left(20\right)\end{array}$

When carrying out such a reverse image transformation process, in a case where the image to be treated is described by digital image data, the image is represented by a collection of pixels (picture elements), and the image data are formed by pixel data. Hence, the calculation may be made similarly to the case of the image transformation process described above, but in an opposite direction, so as to obtain a weighted average of the pixel values of 4 pixels neighboring the corresponding pixel of the original image.

For example, a −γ rotation process may be carried out using the following formula (21), a tilt transformation process may be carried out using the following formula (22), a β rotation process may be carried out using the following formula (23), a transformation process to the pixel coordinates may be carried out using the following formula (24), an integer process may be carried out using the following formula (25), and a pixel value setting process may be carried out using the following formula (26).

$\begin{array}{cc}\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\end{array}\right)=\left(\begin{array}{cc}\cos \left(-\gamma \right)& -\sin \left(-\gamma \right)\\ \sin \left(-\gamma \right)& \cos \left(-\gamma \right)\end{array}\right)\left(\begin{array}{c}I-319.5\\ 239.5-J\end{array}\right)\mathrm{dp}& \left(21\right)\\ \left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right)=\frac{1}{1\cos \alpha +X^{\prime }\sin \alpha }\left(\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\cos \alpha \end{array}\right)& \left(22\right)\\ \left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{cc}\cos \beta & -\sin \beta \\ \sin \beta & \cos \beta \end{array}\right)\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right)& \left(23\right)\\ \left(\begin{array}{c}{i}^{\prime }\\ j^{\prime }\end{array}\right)=\left(\begin{array}{c}319.5+x/\mathrm{dp}\\ 239.5-y/\mathrm{dp}\end{array}\right)& \left(24\right)\\ \left(\begin{array}{c}i\\ j\end{array}\right)=\left(\begin{array}{c}\left[i^{\prime }\right]\\ \left[j^{\prime }\right]\end{array}\right)& \left(25\right)\\ p\left(I,J\right)=\left(1+j-j^{\prime }\right)\left\{\left(1+i-i^{\prime }\right)P\left(i,j\right)+\left(i^{\prime }-i\right)P\left(i+1,j\right)\right\}+\left(j^{\prime }-j\right)\left\{\left(1+i-i^{\prime }\right)P\left(i,j+1\right)+\left(i^{\prime }-i\right)P\left(i+1,j+1\right)\right\}& \left(26\right)\end{array}$

By successively calculating the formulas (21) through (26), it is possible to carry out the reverse image transformation process with respect to the image that is formed by 640×480 pixels.

In the case of the image projection apparatus, the calculation may be simplified if the optical axis of the image projection apparatus is tilted with respect to the screen in only one of the vertical and horizontal directions (up-and-down direction and right-and-left direction) of the screen.

For example, in a case where an image projection apparatus 31 is tilted by an angle α to the right side of a screen 32 as shown in FIG. 18, β=γ=0 may be set in the formulas (21) through (26) described above, and thus, the following formulas (21A), (22) and (24A) may be calculated in place of the formulas (21) through (24). In the case where the image projection apparatus 31