Uchiyama, Shinji (Kanagawa, JP)
Satoh, Kiyohide (Kanagawa, JP)
Eto, Daisuke (Kyoto, JP)
Nakatsuru, Tomoyasu (Kyoto, JP)
Yoshikawa, Tsuneo (Nara, JP)
Plaque It!
|
| 6243158 | Projection exposure apparatus and method | June, 2001 | Shiraishi | 355/53 |
| 6275773 | GPS vehicle collision avoidance warning and control system and method | August, 2001 | Lemelson et al. | 701/301 |
| 6281806 | Driver road hazard warning and illumination system | August, 2001 | Smith et al. | 340/901 |
| 6845296 | Object taking out apparatus | January, 2005 | Ban et al. | 700/245 |
| 7092109 | Position/orientation measurement method, and position/orientation measurement apparatus | August, 2006 | Satoh et al. | 356/620 |
| 20020103617 | Position and orientation determining method and apparatus and storage medium | August, 2002 | Uchiyama et al. | 702/150 |
| 20040090444 | Image processing device and method therefor and program codes, storing medium | May, 2004 | Satoh | 345/633 |
| JP7257228 | October, 1995 | |||
| JP7280579 | October, 1995 | |||
| JP10281794 | October, 1998 | GUIDANCE DISPLAY DEVICE FOR VEHICLE | ||
| JP1165431 | March, 1999 | |||
| JP11084307 | March, 1999 | HEAD-MOUNTED OPTICAL DEVICE | ||
| JP2000041173 | February, 2000 | DECIDING METHOD FOR VISUAL POINT POSITION ATTITUDE, CAMERA DEVICE AND VISUAL POINT POSITION SENSOR | ||
| JP2000132677 | May, 2000 | PICTURE PROCESSING METHOD | ||
| JP2000276613 | October, 2000 | DEVICE AND METHOD FOR PROCESSING INFORMATION | ||
| JP2000354230 | December, 2000 | VIDEO ON-DEMAND SYSTEM AND METHOD FOR ELIMINATING INTER- FRAME ENCODED PICTURE AFTER CUT CHANGEOVER THEREFOR | ||
| JP2001160200 | June, 2001 | FORWARD ROAD INFORMATION ACQUIRING MEANS SELECTOR | ||
| JP200219491 | January, 2002 |
Donald B. Gennery, “Visual Tracking of Known Three-Dimensional Objects” International Journal of Computer Vision, vol. 7, No. 3 pp. 243-270 (1992).
Y. Yokokohji, Y. Sugawara, T. Yoshikawa, “Accurate Image Overlay on Video See-Through HMDs Using Vision and Accelerometers” Transactions of the Virtual Reality Society of Japan, vol. 4, No. 4, pp. 589-598 (1999).
Japanese Official Action dated May 9, 2008 in Japanese Application No. 2002-268832.
1. A measurement apparatus which measures a position and orientation of an image sensing device or an object sensed by the image sensing device by detecting indices on an image of the object sensed by the image sensing device, comprising: an index selection unit adapted to select indices which are to be observed on the sensed image without being occluded, wherein the position and orientation of the image sensing device or the object sensed by the image sensing device are measured on the basis of information of the selected indices.
2. The apparatus according to claim 1, wherein said index selection unit comprises: index information holding unit adapted to hold information associated with each index; and occlusion determination unit adapted to determine a presence/absence of occlusion of each index on the basis of the information associated with each index held by said index information holding unit, and said index selection unit selects indices which are determined to be free from occlusion.
3. The apparatus according to claim 2, wherein said index information holding unit holds at least position/orientation information of the image sensing device or the object upon sensing a template used to detect each index, and said occlusion determination unit determines the presence/absence of occlusion by comparing the position/orientation information of the image sensing device or the object held by said index information holding unit, and estimated values of a position and orientation of the image sensing device or the object.
4. The apparatus according to claim 2, wherein said index information holding unit holds at least information associated with a direction of each index, and said occlusion determination unit determines the presence/absence of occlusion by comparing the information associated with a direction of each index held by said index information holding unit with an estimated value of a direction of the image sensing device.
5. The apparatus according to claim 2, wherein said index information holding unit holds at least a position of each index, and said occlusion determination unit determines the presence/absence of occlusion on the basis of the position of each index, estimated values of a position and orientation of the image sensing device, and shape information of a real space.
6. A measurement method which measures a position and orientation of an image sensing device or an object sensed by the image sensing device by detecting indices on an image of the object sensed by the image sensing device, comprising: an index selection step of selecting indices which are to be observed on the sensed image without being occluded, wherein the position and orientation of the image sensing device or the object sensed by the image sensing device are measured on the basis of information of the selected indices.
7. The method according to claim 6, wherein the index selection step comprises: an index information holding step of holding information associated with each index in a memory; and an occlusion determination step of determining a presence/absence of occlusion of each index on the basis of the information associated with each index held by the memory, and the index selection step includes a step of selecting indices which are determined to be free from occlusion.
8. The method according to claim 7, wherein the memory holds at least information associated with a direction of each index, and the occlusion determination step includes a step of determining the presence/absence of occlusion by comparing the information associated with a direction of each index held by the memory with an estimated value of a direction of the image sensing device.
FIELD OF THE INVENTION
The present invention relates to an image display technique which comprises a display device that displays an image to be presented to a passenger on a vehicle on the front windshield portion of the vehicle, an identification technique of an index used to measure the position and orientation of an image sensing device, which comprises an image sensing unit for sensing an image of a real space, and a position/orientation measurement unit for roughly measuring the position and orientation, or an object to be measured.
BACKGROUND OF THE INVENTION
Car navigation systems have prevailed since they can offer many functions at lower prices in recent years. Along with the advances of ITS (Intelligent Transport Systems), a car navigation display is expected to function as a terminal which presents various kinds of information that assist the driver in addition to navigation information.
However, with the conventional car navigation system, the driver must temporarily turn his or her eyes to the interior of the vehicle to observe the display. Hence, the driver can only recognize traffic by peripheral vision, and must temporarily shift attention away from the scene in the front of the vehicle.
As a technique that can solve this problem, an HUD (Head Up Display) is known. The HUD is a device for projecting and displaying an image on the front windshield. With this device, the driver can acquire navigation information without turning his or her eyes from the real scene.
With the conventional car navigation system, the driver must associate navigation information (normally superposed on a map) presented on the display with the real scene by himself or herself, and such information is not easily recognized intuitively.
To solve this problem, a device called “On-the-Scene HUD” has been conventionally proposed. The On-the-Scene HUD presents navigation information on the front windshield of a vehicle, which is considered as a see-through display, so that the navigation information that the driver wants is superimposed at an appropriate position on the real scene (for example, refer to J. Fukano, S. Okabayashi, & M. Sakata, “Automotive Head-Up Displays for Navigation Use”, The 14th International Technical Conference on Enhanced Safety of Vehicles, No. 94-S2-0-02, pp. 306-314, 1994 (non-patent reference 1)).
As described in non-patent reference 1, the driver can quickly recognize navigation information using the On-the-Scene HUD. However, this reference does not describe any practical registration method required to present navigation information to be superimposed at an appropriate position on the real scene.
In general, a technique for superimposing predetermined information at a predetermined position on the real scene is called a mixed reality technique. In a general arrangement of a mixed reality presentation apparatus, an observer can simultaneously observe an image displayed on a display and a real scene observed via the display by wearing a see-through type HMD (Head Mounted Display) on his or her head. At this time, in order to superimpose a predetermined image at a predetermined position on the real scene, the viewpoint position and orientation of the observer in the real scene must be measured, and an image must be generated accordingly.
The viewpoint position and orientation of the observer can be measured by various methods. A method of attaching a magnetic sensor or ultrasonic sensor to the HMD is normally used. However, since a magnetism or ultrasonic wave source must be arranged in the real scene, the movable range of the observer is limited. Also, such method cannot obtain sufficiently high measurement precision.
On the other hand, as a method that can measure the viewpoint position and orientation with high precision without any restrictions on the measurement range, a method of detecting an index in the real scene from video data sensed by a video camera attached to an HMD, and measuring the viewpoint position and orientation on the basis of the detected index has been proposed. For example, in a conventional system, since the image coordinate position of the index extracted from the image is used as an input to an Extended Kalman Filter, the viewpoint position and orientation are estimated as state variables (for example, refer to Yasuyoshi Yokokoji, Yoshihiko Sugawara, & Tsuneo Yoshikawa, “Accurate Image Overlay on HMD using Vision and Accelerometers”, Transactions of the Virtual Reality Society of Japan, Vol. 4, No. 4, pp. 589-598, 1999 (non-patent reference 2)).
However, the registration method in the aforementioned mixed reality presentation apparatus using the HMD cannot realize registration in the On-the-Scene HUD. This is because the viewpoint of the observer is fixed with respect to the display in the HMD, while the relative positional relationship between the viewpoint of the observer (driver) and display is not fixed in the HUD.
It is required for a conventional mixed reality presentation apparatus in which the positional relationship between the display and viewpoint is not fixed to measure the position and orientation of a display screen in the real scene and the position of the observer with respect to the display screen so as to attain registration (for example, refer to Japanese Patent Laid-Open No. 2000-276613 (patent reference 1)).
On the other hand, as for measurement of the position and azimuth of a vehicle, a vehicle measurement apparatus based on a GPS (Global Registration System) and inertial navigation is used in conventional car navigation systems.
A device described in patent reference 1 is premised on use of a display unit held by a hand, and assumes use of a magnetic sensor or ultrasonic sensor to measure the position and orientation of the display unit. Hence, this patent reference does not describe any method of measuring a vehicle which moves over a broad range.
Since the method described in non-patent reference 2 is that for measuring the head position of a person who can take an arbitrary position and orientation, its solution has a high degree of freedom, and a wrong estimated value is often output.
Since the method described in non-patent reference 2 executes a detection process for all indices which may be included in a view volume of the video camera, even when there is an index which is not observed on an image since it is occluded behind another object in the real scene, the detection process of that index is executed, and causes a detection error of the index, thus outputting the estimated values of a wrong position and orientation.
The vehicle measurement apparatus based on the GPS and inertial navigation has poor precision, and can hardly be used in applications such as travel direction indication at a crossing, advance indication of a lane direction, and the like, in which accurate registration with the real scene is indispensable.
The present invention has been made in consideration of the above problems, and has as its object to provide a technique for superimposing navigation information in a vehicle at an appropriate position on a real scene.
The position and orientation of an image sensing unit (to be also referred to as a camera hereinafter) such as a camera used to sense a real space must be measured in a mixed reality system which displays a real space and virtual space together. As a prior art associated with such technique, a method of correcting a measurement error of a position/orientation sensor that measures the position and orientation of a camera using a marker whose position is known and which is arranged in the real space or a feature point in a real space whose position is known (the marker and feature points will be generally referred to as an index hereinafter) is available (for example, refer to Japanese Patent Laid-Open No. 11-084307 (patent reference 2), Japanese Patent Laid-Open No. 2000-041173 (patent reference 3), and Japanese Patent Application No. 2000-354230 (patent reference 4)).
In the prior arts associated with this method, although they use different calculation principles, means, and steps, the position and orientation of a camera are obtained on the basis of information obtained from a position/orientation sensor of six degree of freedom, which is used to measure the position and orientation of the camera, information of indices whose positions are known and which are arranged in a real space, and information obtained by capturing these indices by the camera.
In these methods, as one of means for determining which of indices arranged in the real space corresponds to an index detected from an image, determination means which compares the coordinate position of an index detected from an image with that of an index on an image plane, which is obtained by projection based on the position and orientation measurement values, and determines indices which have a smaller distance as those which correspond to each other is used.
Assume that the “indices used to correct measurement errors of the position/orientation sensor” are arranged on the side surfaces of a tower-like object in various directions with respect to a real space where the tower-like object is present, as shown in FIG. 11.
Referring to FIG. 11, reference numeral 5201 denotes a camera used to sense such real space; 5202, a three-dimensional (3D) position/orientation sensor used to roughly measure the position and orientation of the camera 5201; 5203 and 5204, indices used to correct measurement errors of the 3D position/orientation sensor 5202; and 5205, a tower-like object where the indices 5203 and 5204 are arranged. Furthermore, the camera 5201 is movable around the tower-like object 5205, and may sense all side surfaces of the tower-like object 5205.
In such case, the tower-like object 5205 may be sensed from a position nearly perpendicular to a given side surface of the tower-like object 5205, as shown in FIG. 12. In FIG. 12, reference numeral 5300 denotes an image (image sensing frame) sensed by the camera 5201; and 5301, the index 5203 (FIG. 11) which appears in the image. Reference numeral 5302 denotes a coordinate position on the image sensing frame, which is obtained by projecting and calculating the 3D position of the index 5203 on the image plane of the sensed image 5300 on the basis of the position and orientation of the camera 5201 measured by the 3D position/orientation sensor 5202; and 5303, a coordinate position on the image sensing frame, which is obtained by projecting and calculating the 3D position of the index 5204 onto the image plane of the sensed image 5300.
If the position and orientation measurement values of the 3D position/orientation sensor 5202 are free from any errors, the coordinate position 5302 and the coordinate position of the image 5301 originally indicate the same coordinate position. However, in practice, since the position and orientation measurement values include an error, they do not indicate the same position. In the prior art, as means for determining which of indices which are actually arranged corresponds to an index (5301 in FIG. 12) detected from a sensed image, the distance between the coordinate value of an index detected from an image, and that of the index on a screen, which is calculated based on the position and orientation measurement values is used. For this reason, in the example shown in FIG. 12, since the distance between the coordinate position 5303 and the coordinate position of the image 5301 is smaller than that between the coordinate position 5302 and the coordinate position of the image 5301 due to the influence of sensor errors, 5301 and 5303 are determined as corresponding indices. Since 5303 is a point obtained by projecting the index arranged on the back side of this tower-like object 5205, the above result means wrong correspondence.
It is, therefore, another object of the present invention to accurately identify which of indices arranged in a real space corresponds to an index detected from an image.
SUMMARY OF THE INVENTION
In order to achieve the above object, for example, an image display apparatus of the present invention comprises the following arrangement.
That is, an image display apparatus which comprises a display device that displays an image to be presented to a passenger on a vehicle on a front windshield portion of the vehicle, comprises:
vehicle measurement unit adapted to measure a position and azimuth of the vehicle on a world coordinate system;
head measurement unit adapted to measure a position of a head of the passenger on the vehicle on a coordinate system defined in the vehicle; and
control unit adapted to control the display device to display the image at a position according to the position and azimuth of the vehicle measured by said vehicle measurement unit, and the position of the head measured by said head measurement unit, on the front windshield portion.
In order to achieve the above object, for example, a measurement apparatus of the present invention comprises the following arrangement.
That is, a measurement apparatus which is mounted on a vehicle and is used to measure a position and azimuth of the vehicle on a world coordinate system, comprisies:
external world image sensing unit adapted to sense an image of an external world of the vehicle;
position calculation unit adapted to calculate positions of indices in the image, which are included in the image of the external world sensed by said external world image sensing unit; and
position/azimuth measurement unit adapted to measure the position and azimuth of the vehicle on the world coordinate system on the basis of the positions of the indices in the image calculated by said position calculation unit.
In order to achieve the above object, for example, a measurement apparatus of the present invention comprises the following arrangement.
That is, a measurement apparatus which measures a position and orientation of an image sensing device or an object sensed by the image sensing device by detecting indices on an image of the object sensed by the image sensing device, comprises:
an index selection unit adapted to select indices which are to be observed on the sensed image without being occluded,
wherein the position and orientation of the image sensing device or the object sensed by the image sensing device are measured on the basis of information of the selected indices.
In order to achieve the above object, for example, an image display method of the present invention comprises the following arrangement.
That is, an image display method executed by an image display apparatus which comprises a display device that displays an image to be presented to a passenger on a vehicle on a front windshield portion of the vehicle, comprises:
a vehicle measurement step of measuring a position and azimuth of the vehicle on a world coordinate system;
a head measurement step of measuring a position of a head of the passenger on the vehicle on a coordinate system defined in the vehicle; and
a control step of controlling the display device to display the image at a position according to the position and azimuth of the vehicle measured in the vehicle measurement step, and the position of the head measured in the head measurement step, on the front windshield portion.
In order to achieve the above object, for example, a measurement method of the present invention comprises the following arrangement.
That is, a measurement method which is mounted on a vehicle and is used to measure a position and azimuth of the vehicle on a world coordinate system, comprises:
an external world image sensing step of sensing an image of an external world of the vehicle using an image sensing device;
a position calculation step of calculating positions of indices in the image, which are included in the image of the external world sensed in the external world image sensing step; and
a position/azimuth measurement step of measuring the position and azimuth of the vehicle on the world coordinate system on the basis of the positions of the indices in the image calculated in the position calculation step.
In order to achieve the above object, for example, a measurement method of the present invention comprises the following arrangement.
That is, a measurement method which measures a position and orientation of an image sensing device or an object sensed by the image sensing device by detecting indices on an image of the object sensed by the image sensing device, comprises:
an index selection step of selecting indices which are to be observed on the sensed image without being occluded,
wherein the position and orientation of the image sensing device or the object sensed by the image sensing device are measured on the basis of information of the selected indices.
In order to achieve the above object, for example, an information processing method of the present invention comprises the following arrangement.
That is, an information processing method which comprises a detection step of detecting a position of an index in a sensed image sensed by image sensing unit adapted to sense an image of a real space where the index is laid out, and a first calculation step of calculating a coordinate position of the index upon projecting the index in the real space onto the sensed image on the basis of the position of the index in the real space and a position and orientation of the image sensing unit when at least one of the position of the index in the real space and the position and orientation of the image sensing unit is obtained based on a measurement, and executes a process for determining correspondence between indices at coordinate positions with a smaller distance on the basis of coordinate positions of indices detected in the detection step-and the coordinate position calculated in the first calculation step on the sensed image, comprises:
a second calculation step of calculating a value using a normal vector to an index of interest, and a visual axis vector of the image sensing unit; and
a determination step of determining, on the basis of a range of the value calculated in the second calculation step, whether or not the process for calculating the coordinate position of the index of interest on the sensed image in the first calculation step is to be executed,
wherein when it is determined in the determination step that the process for calculating the coordinate position of the index of interest on the sensed image in the first calculation step is to be executed, the process for determining correspondence between indices at coordinate positions with a smaller distance is executed based on the coordinate position of the index of interest calculated in the first calculation step and the coordinate positions of the indices detected in the detection step.
In order to achieve the above object, for example, an information processing apparatus of the present invention comprises the following arrangement.
That is, an information processing apparatus which comprises image sensing unit adapted to sense an image of a real space where an index is laid out, detection unit adapted to detect a position of the index in a sensed image sensed by the image sensing unit, and first calculation unit adapted to, when at least one of a position of an index in the real space and a position and orientation of the image sensing unit is obtained based on a measurement, calculating a coordinate position of the index upon projecting the index in the real space onto the sensed image on the basis of the position of the index in the real space and the position and orientation of the image sensing unit, and executes a process for determining correspondence between indices at coordinate positions with a smaller distance on the basis of coordinate positions of indices detected by the detection unit and the coordinate position calculated by the first calculation unit, on the sensed image, comprises:
second calculation unit adapted to calculate a value using a normal vector to an index of interest, and a visual axis vector of the image sensing unit; and
determination unit adapted to determine, on the basis of a range of the value calculated by said second calculation unit, whether or not the process for calculating the coordinate position of the index of interest on the sensed image by the first calculation unit is to be executed,
wherein when said determination unit determines that the process for calculating the coordinate position of the index of interest on the sensed image by the first calculation unit is to be executed, the process for determining correspondence between indices at coordinate positions with a smaller distance is executed based on the coordinate position of the index of interest calculated by the first calculation unit and the coordinate positions of the indices detected by the detection unit.
In order to achieve the above object, for example, an identification method of the present invention comprises the following arrangement.
That is, an identification method of an index used to measure a position and orientation of an image sensing device for sensing an image of a real space, comprises:
a position/orientation measuring step of measuring roughly a position and orientation of the image sensing device;
an image sensing step of sensing an image of a real space including an index using the image sensing device;
a first image coordinate calculation step of calculating coordinate of the index, which is included in the image of the real space obtained in the image sensing step, in the obtained image; and
a second image coordinate calculation step of calculating a position of the index in a sensed image sensed by the image sensing device, whose position and orientation are obtained in the position/orientation measuring step,
wherein the index is identified on the basis of the coordinate of the index calculated in the first image coordinate calculation step, the coordinate of the index calculated in the second image coordinate calculation step, and relationship between a visual axis vector of the image sensing device obtained in the position/orientation measuring step and a normal vector of the index.
In order to achieve the above object, for example, an identification method of the present invention comprises the following arrangement.
That is, an identification method of an index used to measure a position and orientation of an object in a real space, comprises:
a position/orientation measuring step of measuring roughly a position and orientation of the object;
an image sensing step of sensing an image of the object using an image sensing device, which is fixed in position;
a first image coordinate calculation step of calculating coordinate of an index, which is included in the image of the object obtained in the image sensing step, in the obtained image; and
a second image coordinate calculation step of calculating a position of the index in a sensed image sensed by the image sensing device, on the basis of position and orientation are obtained in the position/orientation measuring step,
wherein the index is identified on the basis of the coordinate of the index calculated in the first image coordinate calculation step, the coordinate of the index calculated in the second image coordinate calculation step, and relationship between a visual axis vector of the image sensing device obtained in the position/orientation measuring step and a normal vector of the index.
In order to achieve the above object, for example, an identification method of the present invention comprises the following arrangement.
That is, an identification method of an index used to measure a position and orientation of an object in a real space, comprises:
a first position/orientation measuring step of measuring roughly a position and orientation of the object;
a second position/orientation measuring step of measuring roughly a position and orientation of an image sensing device which senses the object;
an image sensing step of sensing an image of the object using the image sensing device;
a first image coordinate calculation step of calculating coordinate of an index, which is included in the image of the object obtained in the image sensing step, in the obtained image; and
a second image coordinate calculation step of calculating a position of the index in a sensed image sensed by the image sensing device, on the basis of a position and orientation of the object with reference to the image sensing device, or of the image sensing device with reference to the object, which is obtained in the first and second position/orientation measuring step,
wherein the index is identified on the basis of the coordinate of the index calculated in the first image coordinate calculation step, the coordinate of the index calculated in the second image coordinate calculation step, and relationship between a visual axis vector of the image sensing device obtained in the second position/orientation measuring step, and a normal vector of the index obtained in the first position/orientation measuring step.
Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.
BRIEF DESCRIPTION OF THE DRAWINGS
The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.
FIG. 1 is a block diagram showing the functional arrangement of an image display apparatus according to the first embodiment of the present invention, which is mounted on a vehicle;
FIG. 2 shows an example of a virtual image 171;
FIG. 3 shows the state of a front windshield portion observed by a passenger 132;
FIG. 4 is a flow chart showing the process to be executed by vehicle measurement means;
FIG. 5 is a view showing the arrangement for detecting the rotational velocity of the right rear wheel;
FIG. 6 is a block diagram showing the basic arrangement of an external index detection unit 105;
FIG. 7 is a flow chart showing the process to be executed by head measurement means;
FIG. 8 is a flow chart showing the process to be executed by an image generation unit 150 as control means;
FIG. 9 shows an external image sensing camera 100 mounted on a hood of a vehicle 900;
FIG. 10 is a block diagram showing the functional arrangement of a system including an information processing apparatus according to the 18th embodiment of the present invention;
FIG. 11 is a schematic perspective view showing markers as indices arranged in a real space, and a camera to which a 3D position/orientation measurement device is fixed so as to explain the conventional problems;
FIG. 12 is a schematic view showing a coordinate position obtained by projecting the measurement value of a 3D position/orientation sensor onto a screen, and a coordinate position obtained by detecting an index from an image so as to explain the conventional problems;
FIG. 13 is a flow chart of an index identification process according to the 18th embodiment of the present invention;
FIG. 14 is a block diagram showing the functional arrangement of a system including an information processing apparatus according to the 19th embodiment of the present invention;
FIG. 15 is a perspective view of a system according to the 19th embodiment of the present invention;
FIG. 16 is a flow chart of an index identification process according to the 19th embodiment of the present invention;
FIG. 17 shows an example of an index as a figure which has a two-dimensional spread in the 21st embodiment of the present invention;
FIG. 18 shows the contents of a file that records index information data;
FIG. 19 is a block diagram showing the functional arrangement of a system including an information processing apparatus according to the 22nd embodiment of the present invention;
FIG. 20 is a flow chart showing the process to be executed by a computer 51050 according to the 22nd embodiment of the present invention until a composite image is output to a display unit 51011;
FIG. 21 is a view for explaining the method of calculating a normal vector to the index shown in FIG. 17; and
FIG. 22 is a block diagram showing the basic arrangement of a computer 5150.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
Preferred embodiments of the present invention will now be described in detail in accordance with the accompanying drawings.
First Embodiment
FIG. 1 is a block diagram showing the functional arrangement of an image display apparatus according to this embodiment, which is mounted on a vehicle. The image display apparatus of this embodiment comprises an external image sensing camera 100, external index detection unit 105, tachometer 110, vehicle measurement unit 120, passenger image sensing camera 130, head index detection unit 135, head measurement unit 140, image generation unit 150, projector 160, and combiner 170.
A passenger 132 of the vehicle (in this embodiment, the passenger is also a driver of the vehicle) can visually observe an object 151 of an external world via a front windshield portion 131, and can also visually observe an image indicating navigation information which is displayed on the combiner 170 attached to the front windshield portion 131 and is generated by a process to be described later of the image display apparatus. That is, the passenger 132 can visually observe the navigation information superimposed on the object 151 of the external world.
Vehicle measurement means formed by the external image sensing camera 100, external index detection unit 105, tachometer 110, and vehicle measurement unit 120 will be explained below. The external image sensing camera 100 is mounted on the hood of the vehicle to face forward with respect to the vehicle. The camera 100 senses an actual scene of the external world outside the vehicle, and outputs the sensed image to the external index detection unit 105. FIG. 9 shows the external image sensing camera 100 mounted on the hood of a vehicle 900. The external index detection unit 105 receives the sensed image output from the external image sensing camera 100, detects external indices (to be described later) set in the real scene from that image, and outputs the position (coordinates) of the detected external indices in the image to the vehicle measurement unit 120. The tachometer 110 is connected to the right and left rear wheels of the vehicle to measure the rotational velocities of the respective wheels, and outputs the measurement results to the vehicle measurement unit 120. The vehicle measurement unit 120 receives the image coordinate of the external indices output from external index detection unit 105, and the rotational velocities of the respective wheels output from the tachometer 110, measures the position and azimuth of the vehicle on the world coordinate system based on these velocities, and outputs the measurement results to the image generation unit 150.
Head measurement means formed by the passenger image sensing camera 130, head index detection unit 135, and head measurement unit 140 will be described below. The passenger image sensing camera 130 is mounted on the upper portion of the front windshield portion 131 to face the head of the passenger 132, senses an image of the head, and outputs the sensed image to the head; index detection unit 135. The head index detection unit 135 receives the sensed image output from the passenger image sensing camera 130, detects head indices (to be described later) set on the head of the passenger 132 from the image, and outputs the position (coordinates) of the detected head indices in the image to the head measurement unit 140. The head measurement unit 140 receives the image coordinate positions of the head indices output from the head index detection unit 135, measures the position of the head of the passenger 132 on a coordinate system (to be described later) defined within the passenger room on the basis of the received image coordinate position, and outputs the result to the image generation unit 150.
The image generation unit 150 as control means will be described below. The image generation unit 150 receives the position and azimuth of the vehicle from the vehicle measurement unit 120, and also the head position of the passenger 132 from the head measurement unit 140, and generates a display image so that the passenger can observe an image indicating predetermined navigation information, which is superimposed on a predetermined position (on the front surface of the objects 151 (the surface on the passenger 132 side) in the example of FIG. 1) of the real scene. The image generation unit 150 outputs the generated image to the projector 160.
A display device as the projector 160 will be described below. The projector 160 is set below the front windshield portion 131, receives the display image output from the image generation unit 150, and projects it toward the combiner 170. The combiner 170 is arranged on the front windshield portion 131. The image projected by the projector 160 is reflected on the surface of the combiner 170, and the passenger 132 can observe an image which includes the image indicating the navigation information as a virtual image 171.
FIG. 2 shows an example of the virtual image 171. In the example of FIG. 2, an arrow 200 is displayed as navigation information. The virtual image 171 in FIG. 2 is seen through except for a region of the arrow image 200. FIG. 3 shows the state of the front windshield portion to be observed by the passenger 132. As shown in FIG. 3, the passenger 132 can observe the object 151 of the external world via the front windshield portion 131, and can also observe the virtual image 171, i.e., the arrow image 200, which is superimposed on the object 151.
The vehicle measurement means, head measurement means, control means, and display device will be described in more detail hereinafter. Prior to the description, principal variables used in the following description will be explained.
Σ: a coordinate system. For example, Σe represents a coordinate system defined by e.
x: a scalar value that represents an x-coordinate. For example, e1xe2 indicates the x-coordinate of e2 on coordinate system Σe1.
y: a scalar value that represents a y-coordinate. For example, e1ye2 indicates the y-coordinate of e2 on coordinate system Σe1.
z: a scalar value that represents a z-coordinate. For example, e1ze2 indicates the z-coordinate of e2 on coordinate system Σe1.
r: a ternary vector that represents a position. For example, e1re2 indicates the position of e2 on coordinate system Σe1.
p: a binary vector that represents an image position. For example, Ipe2 indicates the position of e2 in image I.
R: a 3×3 matrix that represents an orientation. For example, e1Re2 indicates the orientation of e2 on coordinate system Σe1.
φ: a ternary vector that represents an orientation expressed by an Euler angle.
ω: a ternary vector that represents an angular velocity about each axis.
b: a real scene. Σb represents a real scene coordinate system (world coordinate system).
w: a vehicle 10. Σw represents a vehicle coordinate system.
u: the head of the passenger 132. Σu represents a head coordinate system.
c1: the external image sensing camera 100. Σc1 represents an external image sensing camera coordinate system.
c2: the passenger image sensing camera 130. Σc2 represents a passenger image sensing camera coordinate system.
d: a virtual image plane. Σd represents a virtual image plane coordinate system.
I: a sensed image. Coordinate system ΣI represents an image coordinate system (two dimensions).
g: an object on which navigation information is to be superimposed (e.g., the object 151 of the external world).
q: an image to be displayed as navigation information (e.g., the arrow 200).
ζ: a scalar value that represents an azimuth angle (an angle with respect to the y-axis in the x-y plane). bζw represents the azimuth angle of the vehicle in the real scene.
x′: a scalar value that represents an x-velocity component.
y′: a scalar value that represents a y-velocity component.
ζ′: a scalar value that represents an azimuth angular velocity.
t: a time. For example, rw(t) indicates the position of the vehicle at time t.
k: an integer that represents a sample time. For example, tk is a time in the k-th process.
Δt: an infinitesimal interval between sample times. That is, tk+1=tk+Δt.
m: an external index 101. mi indicates the i-th detected external index.
M: the total number of external indices detected on image I1.
n: a head index 133. ni indicates the i-th detected head index.
N: the total number of head indices detected on image I2.
ψ′: the rotational angular velocity (measured value) of a wheel.
v: a scalar value that represents the velocity of the vehicle.
A: a distance from the middle point of the right and left rear wheels to each of the right and left rear wheels, and this value is known.
B: the diameter of the wheel, and this value is known.
a1: a scalar value that represents the focal length of the external image sensing camera 100. This value is known.
a2: a scalar value that represents the focal length of the passenger image sensing camera 130. This value is known.
yk: an observation vector.
yk: an estimated value of the observation vector.
xk: a state vector.
xk: an estimated value of the state vector.
fk(xk): a state equation.
hk(xk): an observation equation.
Kk: a Kalman gain.
The following explanation will be given under the condition that respective coordinate systems are defined as follows. That is, coordinate system Σb of the real scene serving as a reference assumes origin bO at an appropriate position on a horizontal plane, bX- and bY-axes in orthogonal directions in the horizontal plane, and bZ-axis in a direction normal to the horizontal plane. Vehicle coordinate system Σw assumes origin wO at an appropriate position on the vehicle, and wX-axis in the left-to-right direction, wY-axis in the back-to-forth direction, and wZ-axis in the down-to-up direction upon sitting on the driver's seat.
Head coordinate system Σu assumes origin uO at the middle point of the two eyes of the passenger 132, uX-axis in a direction from the left eye to the right eye, uY-axis in a direction that connects the middle point of the two eyes and a nose, and uZ-axis in the front direction of the passenger 132. Each of external image sensing camera coordinate system Σc1 and passenger image sensing camera coordinate system Σc2 assumes cZ-axis pointing ahead of the optical axis, cX-axis in the left-to-right direction of an image, and cY-axis in the up-to-down direction of the image. Also, virtual image plane coordinate system Σd assumes origin dO on a virtual image plane projected by the projector, dX-axis from the right-to-left direction toward the virtual image plane, dY-axis in the up-to-down direction, and dZ-axis in a direction normal to a virtual image plane.
<Vehicle Measurement Means>
FIG. 4 is a flow chart showing the process to be executed by the vehicle measurement means. The process of the vehicle measurement means will be explained below with reference to FIG. 4. The vehicle measurement is done according to the scheme of an Extended Kalman Filter.
In the following description, state vector xk (six dimensions) of the Extended Kalman Filter is defined as a vector as a sequence of the x-coordinate position, y-coordinate position, and azimuth angle of the vehicle on real scene coordinate system Σb at time tk and their time derivative values.
xk=[bxw
(tk)byw
(tk)bζw(t
In step S400, the vehicle measurement unit 120 calculates estimated value xk of the state vector at time tk by substituting state vector xk−1 at time tk−1 in state equation fk(xk) of the Extended Kalman Filter. Under the assumption that the vehicle moves at an equal velocity between respective sample times, state equation fk(xk) can be defined by:
In step S410, the tachometer 110 measures the rotational velocities of the right and left rear wheels. The vehicle measurement unit 120 stores these pieces of information as rotational velocities ψ′R(tk) and ψ′L(tk) at time tk.
In step S420, the external index detection unit 105 detects image coordinates I1pmi (i=1, 2, . . . , M) of external indices mi on input image I1 input from the external image sensing camera 100. Note that the external indices mean portions, which are registered in advance in the image display apparatus as indices to be used, of characteristic portions of various objects present in the external world, and include, for example, the corners of buildings, text on signboards, and the like. Details of the external indices and their detection process will be described later. The vehicle measurement unit 120 stores detected image coordinates I1pmi(tk) of respective indices mi at time tk.
In step S430, the vehicle measurement unit 120 generates, using the information input in steps S410 and S420, observation vector yk ((2M+2) dimensions) of the Extended Kalman Filter by:
yk=[I1p
m1T(tk) . . . I1pmT(tk)ψ′R(tk) ψ′L(tk)]T (3)
In step S440, the vehicle measurement unit 120 calculates estimated value yk of the observation vector at time tk by substituting estimated value x
k of the state vector in observation equation hk(xk) of the Extended Kalman Filter defined by:
Note that vk in equation (4) is calculated from xk using:
vk=−sinbζw<
/sub>(tk)bx′w(tk)+cosbζ(tk)by′w(tk
) (5)
Also, I1pmi in equation (4) is the estimated value of the image coordinates of each index mi observed on input image I1, and is calculated by:
where c1xmi, c1ymi, and c1zmi are the position of index mi on external image sensing camera coordinate system Σc1, and are calculated from xk by:
where brmi is the position of index mi on real scene coordinate system Σb, and wrc1 and wRc1 are the position and orientation of the external image sensing camera 100 on vehicle coordinate system Σw, which are measured in advance. bRw(tk) is the orientation of the vehicle on real scene coordinate system Σb, and is calculated on the basis of bζw(tk). On the other hand, brw(tk) is the position of the vehicle on real scene coordinate system Σb, and is calculated based on bxw(tk) and byw(tk). Assume that the tilt angle (roll and pitch angles) of the vehicle with respect to the x-y plane of real scene coordinate system Σb is a constant value, and is pre-stored as a known value in this embodiment. Also, assume that z-coordinate bzw of the vehicle on real scene coordinate system Σb is a constant value, and is pre-stored as a known value.
In step S450, the vehicle measurement unit 120 updates the state vector on the basis of estimated value xk of the state vector, estimated value y
k of the observation vector, and observation vector yk using:
xk=xk+Kk[yk
yk
] (8)
where Kalman gain Kk is a coefficient matrix (6×(2M+2)) defined based on state equation fk(xk), observation equation hk(xk), and state vector xk−1. Since the calculation method of the Kalman gain is disclosed in, e.g., Kiyoshi Nishiyama: Optimal Filtering, Baihukan, 2001, a detailed description thereof will be omitted.
In step S460, the vehicle measurement means acquires position (bxw(tk), byw(tk)) and azimuth bζw(tk) of the vehicle from updated state vector xk, and outputs them to the image generation unit 150 as the control means.
The vehicle measurement means determines in step S470 if the process is to end. If the process is to continue, k is incremented (k+1), and the processes in step S400 and subsequent steps are repeated. Note that the external index detection unit 105 and vehicle measurement unit 120 can be implemented by, e.g., a single versatile computer.
Details of the tachometer 110 will be described below using FIG. 5. FIG. 5 shows the arrangement for detecting the rotational velocity of the right rear wheel, and that of the left rear wheel can be detected using the same arrangement. The rotational velocity of a right rear wheel 500R is measured by a rotary encoder 510R and wheel 520R. The wheel 520R is attached to the rotational shaft of the rotary encoder 510R, and is arranged to contact the right rear wheel 500R. The rotary encoder 510R measures rotational angular velocity ψ′R of the wheel 520R, and outputs it as information that represents the rotational velocity of the right rear wheel 500R. The measurement of a left rear wheel 500L (not shown) is realized by the same arrangement, and rotational angular velocity ψ′L of a wheel 520L (not shown) is output.
Details of the process in the external index detection unit 105 will be described below using FIG. 6. FIG. 6 is a block diagram showing the basic arrangement of the external index detection unit 105. The external index detection unit 105 comprises an estimated coordinate calculator 600, index selector 610, and image processor 620. The index selector 610 comprises an occlusion determination section 630 and index information holding section 640.
The estimated coordinate calculator 600 receives the estimated values of the position and azimuth of the vehicle calculated by the vehicle measurement unit 120, and calculates estimated values I1pmi of image coordinates of all external indices on input image I1 using equations (6) and (7). Identifiers of indices the calculated estimated values of which are included within the range of the image sensing frame are output to the occlusion determination section 630. The index information holding section 640 holds, for respective indices, the positions and azimuths of the vehicle upon sensing template images used to detect indices.
The occlusion determination section 630 determines the presence/absence of occlusion for each of indices, the estimated values, obtained from the estimated coordinate calculator 600, of which are included within the range of the image sensing frame. More specifically, when the estimated values of the position and azimuth of the vehicle calculated by the vehicle measurement unit 120 are considerably separated from those of the vehicle upon sensing a template image used to detect an index of interest, which are held by the index information holding section 640, it is determined that the index of interest is occluded. The identifier of an index, which is determined not to be occluded, is output to the image processor 620 as an object to be detected.
Note that the position and azimuth of the vehicle upon sensing a template image may be compared with the current position and azimuth of the vehicle on the basis of, e.g., the distance between two points or on the basis of the difference between the azimuth angles. Also, a common threshold value may be used to determine all indices, or different threshold values may be set for respective indices. Note that the same function can be implemented with reference to the position and orientation of the external image sensing camera 100 in place of the position and azimuth of the vehicle.
The image processor 620 detects an index selected by the index selector 610 as an index to be processed from image I1 by template matching using a template image of each index, which is registered in advance. At this time, the search range Is preferably limited using the estimated values I1pmi of the image coordinates calculated by the estimated coordinate calculator 600.
With the aforementioned processing sequence, position (bxw(tk), byw(tk)) and azimuth bζw(tk) of the vehicle on real scene coordinate system ζb are measured.
<Head Measurement Means>
FIG. 7 is a flow chart showing the process to be executed by the head measurement means. The process of the head measurement means will be explained below with reference to FIG. 7. The head measurement is done according to the scheme of an Extended Kalman Filter as in the vehicle measurement.
In the following description, state vector xk (12 dimensions) is defined as a vector as a sequence of position wru, orientation wφu, velocity wr′u, and angular velocity wωuof the passenger 132 on vehicle coordinate system Σw at time tk.
xk=[wru
T(tk)wφu
T(tk)wr
uT(tk)wω
uT(tk)]T (9)
In step S700, the head measurement unit 140 calculates estimated value xk of the state vector at time tk by substituting state vector xk−1 at time tk−1 in state equation fk(xk). Under the assumption that each position and orientation change at an equal velocity between respective sample times, state equation fk(xk) can be defined by:
where φ′ is the time derivative value of orientation φ, and can be derived as a function of orientation φ and angular velocity ω, as is generally known. Since details of this process are disclosed in, e.g., Donald B. Gennery: Visual tracking of known three-dimensional objects, International Journal of Computer Vision, Vol. 7, No. 3, pp. 243-270, 1992, a description thereof will be omitted.
In step S710, the head index detection unit 135 detects image coordinates I2pni(i=1, 2, . . . , N) of head indices ni observed on input image I2 input from the passenger image sensing camera 130 using template images of respective head indices, which are registered in advance. Note that the head-indices indicate eyes, nose, mouth, and the like, and their partial regions present on the head (face), which are registered in advance in the image display apparatus as indices to be used. The head measurement unit 140 stores these pieces of information as image coordinates I2pni of indices ni at time tk. Since the process in the head index detection unit 135 is the same as that in the external index detection unit 105, a detailed description thereof will be omitted.
In step S720, the head measurement unit 140 generates observation vector yk (2N dimensions) using the information input in step S700 by:
yk=[I2p
nT(tk)]T (11)
In step S730, the head measurement unit 140 calculates estimated value yk of the observation vector at time tk by substituting estimated value x
k of the state vector in observation equation hk(xk) defined by:
yk=hk(xk)=[
I2pnT(tk)I2
pn
T(tk) . . . I2pn
where I2pni is the estimated value of the image coordinates of each index ni observed on input image I2 and is calculated by:
where c2xni, c2yni, and c2zni are the coordinate positions of index ni on head image sensing camera coordinate system Σc2, and are calculated based on state vector xk using:
where urni is the position of head index ni on head coordinate system Σu, wrc2 and wRc2 are the position and orientation of the passenger image sensing camera 130 on vehicle coordinate system Σw, which are measured in advance. Also, wRu(tk) is a matrix that represents the orientation of the head of the passenger 132 on vehicle coordinate system Σw, and is calculated based on wφu(tk).
In step S740, the head measurement unit 140 updates the state vector on the basis of observation vector yk, estimated value xk of the state vector, estimated value y
k of the observation vector using equation (8). Since the calculation method of Kalman gain Kk is known to those who are skilled in the art, a detailed description thereof will be omitted.
In step S750, the head measurement means acquires position wru(tk) of the head from updated state vector xk, and outputs it to the image generation unit 150 as the control means.
The head measurement means determines in step S760 if the process is to end. If the process is to continue, k is incremented (k+1), and the processes in step S700 and subsequent steps are repeated. Note that the head index detection unit 135 and head measurement unit 140 can be implemented by, e.g., a single versatile computer. With the aforementioned processing sequence, position wru of the head of the passenger 132 on vehicle coordinate system Σw is measured.
<Control Means>
FIG. 8 is a flow chart showing the process to be executed by the image generation unit 150 as the control means. The process of the control means will be described below with reference to FIG. 8. In step S800, the image generation unit 150 receives position (bxw(tk), byw(tk)) and azimuth bζw(tk) of the vehicle on real scene coordinate system Σb, which are measured by the vehicle measurement means, and converts them into position vector brw and orientation matrix bRw.
In step S810, the image generation unit 150 receives position wru of the head of the passenger 132 on vehicle coordinate system Σw, which is measured by the head measurement means.
In step S820, the image generation unit 150 calculates position drg of object 151 to be superimposed on virtual image plane coordinate system Σd by:
drg=w
Rd−1{bRw<
/sub>−1(brg−brw)−wrd} (15)
where brg is the position of the object 151 to be superimposed on real scene coordinate system Σb, and wrd and wRd are the position and orientation of the virtual image plane on vehicle coordinate system Σw, which are measured in advance.
In step S830, the image generation unit 150 calculates position dru of the head of the passenger 132 on virtual image plane coordinate system Σd by:
dru=w
Rd−1(wru<
/sub>−wrd) (16)
In step S840, the image generation unit 150 calculates display position drq of an image indicating navigation information (arrow 200) on virtual image plane coordinate system Σd by:
In step S850, the image generation unit 150 generates a display image so that the image indicating the navigation information (arrow 200) is displayed at position drq on the virtual image plane, and outputs it to the projector 160 as a display device.
In step S860, the projector 160 projects the generated display image toward the combiner 170.
The image generation unit 150 as the control means determines in step S870 if the process is to end. If the process is to continue, k is incremented (k+1), and the processes in step S800 and subsequent steps are repeated. Note that image generation unit 150 can be implemented by, e.g., a single versatile computer.
With the aforementioned processing sequence, display that allows the passenger to observe the image indicating navigation information (arrow 200) superimposed on the object 151 to be superimposed can be realized.
In this embodiment, in order to obtain the position and azimuth of the vehicle, the relative positional relationship between the object of the external world as an object to be superimposed, and the vehicle is directly measured in place of data using a GPS. Hence, the position and azimuth of the vehicle can be obtained with higher precision than the GPS.
In this embodiment, since the image indicating navigation information is displayed at a position in accordance with not only the position and orientation of the vehicle but also the head position of the passenger, even when the head of the passenger pitches and rolls, the image indicating navigation information can always be presented at an appropriate position with respect to the passenger.
Second Embodiment
In the above embodiment, the vehicle measurement unit 120 defines state equation fk(xk) using equation (2). Since equation (2) is the state equation based on a model that allows the vehicle to freely pitch and roll, the estimated value of the position of the vehicle obtained using this equation is often largely different from an actual value. This embodiment introduces a nonholonomic constraint based on the assumption that the vehicle never skids, and sets, in place of equation (2), state equation fk(xk) which is improved as:
Note that the above equation merely considers a nonholonomic constraint in estimation of the state vector, and the dimensions themselves of the state vector do not degenerate. Thus, by importing appropriate system noise characteristics, quantity of skid of the vehicle can be considered. Hence, with the method of this embodiment, the position and azimuth of the vehicle can be estimated more accurately than both the case wherein the state equation is formulated on the condition of a perfect nonholonomic constraint, and the case of the above embodiment wherein the state equation is formulated regardless of any nonholonomic constraint. As a result, the measurement precision of the position and azimuth of the vehicle in the vehicle measurement unit 120 can be improved.
Third Embodiment
The tachometer 110 in the above embodiment measures the rotational velocity of the wheel using the rotary encoder. However, the arrangement of the tachometer 110 is not limited to such specific one, and any other arrangements may be adopted as long as the rotational velocity of the wheel can be measured. For example, pulse sensors attached to the two rear wheels for the purpose of, e.g., orientation control of the vehicle may be used as the tachometer 110. In this embodiment, the diameter of the rear wheel 500 can be used as the value B in equation (4) in place of the wheel 520.
Also, the same effect can be obtained by arranging the tachometer 110 by a vehicle velocity sensor and gyro sensor used in inertial navigation of a conventional car navigation system. Let μ be the vehicle velocity measured by the vehicle velocity sensor, and ξ′ be the azimuth angular velocity of the vehicle measured by the gyro sensor. Then, the tachometer 110 in this case derives rotational velocities ψ′R and ψ′L of the two rear wheels on the basis of the measured μ and ξ′ by:
and outputs the calculated velocities to the vehicle measurement unit 120.
Fourth Embodiment
The above embodiment uses the tachometer 110 to obtain the rotational velocities of the two wheels. Alternatively, the same effect as in the above embodiment can be obtained using data associated with a vehicle velocity measured by another sensor. For example, a vehicle velocity sensor based on rotary pulse measurement of a propeller shaft, and a gyro sensor, which are used in inertial navigation of a conventional car navigation system, may be equipped in place of the tachometer 110. In addition, the rotational velocities may be calculated based on the engine speed and the currently selected gear ratio of a transmission. In this case, the vehicle measurement unit 120 receives image coordinates I1pmi(tk) of an external index detected by the external index detection unit 105, vehicle velocity μ(tk) measured by the vehicle velocity sensor, and azimuth angular velocity ξ′(tk) measured by the gyro sensor, and measures the position and azimuth of the vehicle on the basis of these input data. The position and azimuth of the vehicle can be measured in the same sequence as in the above embodiment by defining observation vector yk and observation equation hk(xk) as:
yk=[I1p
mT(tk)μ(tk)ξ
(tk)]T (20)
yk=hk(xk)=[
I1pmT(tk) . . . I1Pm
where vk is the estimated value of the vehicle velocity and is calculated from state vector xk using equation (5).
Fifth Embodiment
The vehicle measurement means may be comprised of only the external image sensing camera 100, external index detection unit 105, and vehicle measurement unit 120. That is, the vehicle measurement unit 120 measures the position and azimuth of the vehicle on the basis of image coordinates I1pmi(tk) of an external index input from the external index detection unit 105. In this case, the position and azimuth of the vehicle can be measured in the same sequence as in the above embodiment by defining observation vector yk and observation equation hk(xk) as:
yk=[I1p
mT(tk)]T (22)
yk=hk(xk)=[
I1pmT(tk) . . . I1pm
Sixth Embodiment
The vehicle measurement means in the above embodiment has only one external image sensing camera 100, but may have a plurality of external image sensing cameras 100. In this case, the vehicle measurement unit 120 receives the image coordinates of external indices detected by a plurality of external index detection units 105, and defines observation vector yk and observation equation hk(xk) based on them. The position and orientation of the vehicle can be obtained in substantially the same sequence as in the above embodiment, except that parameters wRc1, wrc1, and a1 used in the derivation process of the observation equation assume values unique to the corresponding external image sensing cameras 100. In this way, since a larger number of index coordinates can be stably detected, the measurement precision of the position and azimuth of the vehicle can be improved.
In the above embodiment, the external image sensing camera 100 is mounted on the hood of the vehicle to face forward with respect to the vehicle. However, the mount position of the external image sensing camera 100 is not limited to such specific position as long as it has a position and direction that allow to observe external indices in the real scene. More specifically, the camera 100 may face backward or sideward or may be mounted on a fender portion, roof, or the like of the vehicle.
Likewise, the head measurement means can adopt an arrangement having a plurality of passenger image sensing cameras 130, and the measurement precision of the head position can be improved. Also, the mount position of the passenger image sensing camera 130 is not particularly limited as long as the head indices can be observed.
Seventh Embodiment
The head measurement means in the above embodiment measures the position of the head of the passenger 132. The head position measurement need not always be continuously made. That is, the head measurement means measures the position of the head at only a predetermined timing, and outputs the measured head position to the control means. The control means holds the latest head position input from the head measurement means, and generates an image using the held head position. Note that the predetermined timing may be the startup timing of the image display apparatus, or an arrangement that further has means for detecting an on-board timing or drive start timing of the passenger 132 may be adopted, and that detection timing may be used. Alternatively, an arrangement that further has means for inputting a passenger's instruction may be adopted, and the head measurement process may be executed at the issuance timing of the passenger's instruction.
Eighth Embodiment
In the above embodiment, the index selector 610 determines occlusion on the basis of the estimated values of the position and azimuth of the vehicle, which are calculated by the vehicle measurement unit 120, and the position and azimuth of the vehicle upon sensing a template image of an index. Alternatively, occlusion may be determined by other arrangements.
For example, the index information holding section 640 may hold information associated with the direction of each index, and occlusion of an index may be determined on the basis of the direction of the index and the orientation of the external image sensing camera 100.
For example, the index information holding section 640 may hold, for respective indices, normal vectors ηmi indicating normal directions of indices mi as information associated with the direction of each index. The occlusion determination section 630 may calculate the orientation of the external image sensing camera 100 on the basis of the orientation of the vehicle calculated until time tk−1, and may determine that an index of interest is occluded when an angle—c1Z-axis makes with normal vector ηmi is equal to or larger than a predetermined threshold value. Note that a common threshold value may be used for all indices, or different threshold values may be set for respective indices.
With this method, an index which is obviously occluded (e.g., an index arranged on the back side of a wall) can be easily determined. Note that any other kinds of information may be used as information associated with the direction of each index.
Ninth Embodiment
The index selector 610 may further have shape model holding means for holding shape models of objects in the real scene, and the index information holding section 640 may hold positions brmi of respective indices mi. The occlusion determination section 630 may calculate the position and orientation of the external image sensing camera 100 on the basis of the orientation of the vehicle calculated until time tk−1. Then, depth ordering of objects other than indices in the real scene and respective indices may be determined on the basis of the shape models of objects in the real scene held in the shape model holding means and positions brmi of respective indices mi held by the index information holding section 640 based on the calculated position and orientation of the camera 100, thus determining whether or not each index is occluded.