JP6036065B2 - Gaze position detection device and gaze position detection method - Google Patents

Description

  The present invention relates to a gaze position detection device and a gaze position detection method for detecting a position at which a vehicle occupant gazes, for example.

  In recent years, in order to support driving of a vehicle by a driver, a technique has been proposed in which a position at which the driver gazes is detected and a point corresponding to the gaze position is identified on an image obtained by photographing the periphery of the vehicle (for example, (See Patent Documents 1 to 3).

  For example, the driving support apparatus described in Patent Literature 1 detects the gaze angle of the right eye and the gaze angle of the left eye of the driver based on an image obtained by photographing the driver's face. The driving support device detects a vector connecting from the midpoint of the right eye and the left eye to the intersection of the vector representing the right eye gaze angle and the vector representing the left eye gaze angle as the gaze vector.

  Moreover, the gaze direction measuring device described in Patent Literature 2 detects the gaze direction of the subject based on output signals from an eye camera and a three-dimensional magnetic sensor attached to the subject. This gaze direction measuring device is generated by a TV camera that captures the scenery in front of the subject, assuming that the viewpoint position is at an infinite point when the subject performs an operation of looking far away through the windshield. The viewpoint position is mapped on the video.

  Furthermore, the safe driving support apparatus described in Patent Document 3 detects an object that the driver should pay attention to from two images obtained by photographing the front of the vehicle with two cameras, and triangulation Based on this principle, the relative positional relationship between the vehicle and the object is obtained. Moreover, this safe driving assistance apparatus detects a driver | operator's gaze direction by a pupil cornea reflection method, for example. The safe driving support apparatus sets a two-dimensional space between the object and the driver, and maps the object to a position where the line connecting the position of the driver's eyes and the object intersects the two-dimensional space. In addition, the driver's line-of-sight direction is mapped to the two-dimensional space. By doing so, this safe driving support device can compare the relationship between the object and the line-of-sight direction in the two-dimensional space.

JP 2009-43003 A Japanese Patent Laid-Open No. 6-189906 International Publication No. 2007/069489

  Since the camera that captures the surroundings of the vehicle and the driver are separated from each other, there is a parallax between the field of view of the camera and the field of view of the driver. Therefore, in order to accurately specify a point on the image corresponding to the position where the driver is gazing, it is preferable that the distance from the driver to the gazing position is obtained.

  In this regard, in the technique described in Patent Document 1, since the difference between the gaze angle of the left eye and the gaze angle of the right eye is small, it is detected that an error is included in the detection result of any eye gaze angle. There is a possibility that the distance to the gaze position may deviate greatly from the distance to the original gaze position.

  In the technique described in Patent Document 2, although the gaze position is set to infinity, the driver may gaze at an object in the vicinity of the vehicle. Therefore, in the technique described in Patent Document 2, there is a possibility that the viewpoint position is set at a position different from the position on the image corresponding to the gaze position of the driver.

  Furthermore, the technique described in Patent Document 3 obtains the distance from the vehicle to the object instead of obtaining the distance from the driver to the gaze position. However, since this technique uses two cameras to determine the distance from the host vehicle to the object, the configuration of the apparatus becomes complicated, resulting in an increase in the cost of the apparatus.

  In view of this, it is an object of the present specification to provide a gaze position detection device that can improve the accuracy of the gaze position of a driver on an image in which the periphery of the vehicle is captured.

  According to one embodiment, a gaze position detection device is provided. The gaze position detection device captures at least a part of the scenery around the vehicle, generates an image in which the scenery is captured, a gaze detection unit that detects the direction of the gaze of a vehicle occupant, Projection plane setting that sets a virtual plane at a position away from the vehicle by a distance from the vehicle to a position where there is a possibility that an object to be watched by the occupant is present at least at a part of the periphery. An intersection calculation unit that obtains an intersection between the virtual plane and the sight line of the occupant, and a projection unit that detects a point on the image obtained by projecting the intersection point on the image as a gaze position of the occupant. Have.

The objects and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims.
It should be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention as claimed.

  The gaze position detecting device disclosed in the present specification can improve the accuracy of the gaze position of the driver on the image in which the periphery of the vehicle is captured.

It is a schematic block diagram of the driving assistance device by which the gaze position detection device by one embodiment was mounted. It is a figure which shows an example of arrangement | positioning of the driving assistance apparatus in a vehicle. It is a functional block diagram of a control part. It is a figure which shows an example of a gaze projection surface. It is a figure which shows an example of the relationship between the intersection of a driver | operator's eyes | visual_axis and a gaze projection surface, and the gaze position on an image outside a vehicle. It is an operation | movement flowchart of the driving assistance process including a gaze position detection process.

Hereinafter, a gaze position detection device according to one embodiment will be described with reference to the drawings.
This gaze position detection device is a virtual direction of travel of the vehicle at a position away from the outside imaging unit that captures at least a part of the scenery around the vehicle by a distance where an object to be watched by the driver may exist. A plane parallel to is set, and the intersection of the plane and the driver's line of sight is obtained. The gaze position detection device projects at least a part of the scenery around the vehicle by projecting the intersection point on the image generated by the exterior imaging unit according to the distance from the exterior imaging unit to the intersection point. The user's gaze position on the captured image is specified.

  In the present embodiment, the gaze position detection device is mounted on the driving support device. However, the gaze position detection device can also be used as a device for analyzing what the driver is interested in from the gaze position of the driver. Further, the gaze position detection device may detect the gaze position of another occupant in the vehicle interior instead of detecting the gaze position of the driver.

  FIG. 1 is a schematic configuration diagram of a driving support device according to an embodiment in which a gaze position detection device is mounted. FIG. 2 is a diagram illustrating an example of the arrangement of the driving support devices in the vehicle. The driving support device 1 includes a light source 2, an in-vehicle imaging unit 3, an out-of-vehicle imaging unit 4, a storage unit 5, and a control unit 6. The driving support device 1 may further include a display (not shown) for displaying an image generated by the vehicle exterior imaging unit 4. For example, each component which the driving assistance apparatus 1 has may be accommodated in one housing. In this case, on the dashboard of the vehicle 10, the driving assistance device 1 allows the outside imaging unit 4 to capture at least a part of the scenery around the vehicle 10, and the in-vehicle imaging unit 3 is the face of the driver 11. Is arranged so that it can be photographed.

  Or the one part component of the driving assistance device 1 may be arrange | positioned separately from another component. For example, the light source 2 may be attached to emit light toward the driver 11 near an instrument panel or a ceiling in front of the vehicle interior. The in-vehicle image capturing unit 3 may also be attached to the vicinity of the ceiling in front of the instrument panel or the vehicle interior, for example, facing the driver. Further, the outside-vehicle imaging unit 4 is attached so as to be able to photograph at least a part of the scenery around the vehicle 10, for example, a scenery in front of the vehicle 10. The light source 2, the vehicle interior image capturing unit 3, and the vehicle exterior image capturing unit 4 include a storage unit 5 and a control unit 6 accommodated via a signal line, and a housing disposed on the back side of the instrument panel or on the dashboard. It may be connected.

The light source 2 illuminates the user's face, particularly the eyes and surroundings. Therefore, the light source 2 includes, for example, at least one infrared light emitting diode and a drive circuit that supplies power from a power source (not shown) to the infrared light emitting diode in accordance with a control signal from the control unit 6. And the light source 2 will emit illumination light, if the control signal which lights a light source from the control part 6 is received.
In addition, the number of the light sources 2 which the driving assistance apparatus 1 has is not restricted to one, A plurality may be sufficient.

  The in-vehicle imaging unit 3 generates an image showing the driver's face, which is used to detect the driver's line-of-sight direction. For this purpose, the in-vehicle imaging unit 3 includes an image sensor having a solid-state imaging device that is two-dimensionally arranged with sensitivity to light from the light source 2 and an imaging optical system that forms an image of a driver's face on the image sensor. Have The in-vehicle imaging unit 3 further includes a visible light cut filter between the image sensor and the imaging optical system in order to suppress detection of a reflection image by the iris and a corneal reflection image of light from a light source other than the light source 2. You may have. In addition, the in-vehicle imaging unit 3 is disposed, for example, close to the light source 2 so that an image of the light source 2 appears on the user's cornea on the image.

  Further, the number of pixels of the image sensor of the in-vehicle imaging unit 3 can be set to, for example, 1 million, 2 million, or 4 million so that the image has a resolution that can identify the Purkinje image of the light source 2. The image generated by the image sensor is a color image. Alternatively, the image generated by the image sensor may be a gray image in which the luminance of each pixel increases as the intensity of light received by the solid-state imaging device increases.

  The in-vehicle imaging unit 3 captures the face of the driver at a predetermined shooting rate (for example, 10 to 30 frames / second) according to a predetermined exposure condition notified from the control unit 6 while the gaze position detection process is being executed. By doing so, an image showing the driver's face is generated. The in-vehicle imaging unit 3 outputs the image to the control unit 6 every time an image is generated. Hereinafter, for the sake of convenience, an image generated by the in-vehicle imaging unit 3 is referred to as an in-vehicle image.

  The outside imaging unit 4 is installed so as to be able to photograph at least a part of the scenery around the vehicle 10, and periodically captures an image of the scenery at a predetermined photographing rate (for example, 10 to 30 frames / second). Generate. For this purpose, the vehicle exterior imaging unit 4 includes image sensors of solid-state imaging devices arranged in a two-dimensional array, and an imaging optical system that forms an image of a landscape in front of the vehicle on the image sensor. In the present embodiment, the vehicle exterior imaging unit 4 is installed so that the optical axis of the vehicle exterior imaging unit 4 is parallel to the traveling direction of the vehicle 10 and the vehicle exterior imaging unit 4 can capture a scene in front of the vehicle 10. The vehicle exterior imaging unit 4 may be installed so as to photograph the right side or the left side of the vehicle 10. Further, the image generated by the vehicle exterior imaging unit 4 may be a color image or a gray image. The outside imaging unit 4 passes the image to the control unit 6 every time an image is generated. Hereinafter, for the sake of convenience, an image generated by the vehicle exterior imaging unit 4 is referred to as a vehicle exterior image.

  The storage unit 5 includes, for example, a readable / writable nonvolatile or volatile semiconductor memory and a read-only nonvolatile semiconductor memory. And the memory | storage part 5 memorize | stores the program of the gaze position detection process performed on the control part 6. FIG. In addition, the storage unit 5 stores various data used for detecting the gaze position of the driver on the outside image.

  The control unit 6 includes one or a plurality of processors and their peripheral circuits, and is connected to each unit of the driving support device 1 through a signal line. And the control part 6 controls each part of the driving assistance apparatus 1 by transmitting a control signal to each part of the driving assistance apparatus 1 through a signal line.

FIG. 3 is a functional block diagram of the control unit 6. The control unit 6 includes a line-of-sight detection unit 11, a line-of-sight projection plane setting unit 12, an intersection calculation unit 13, a projection unit 14, a gaze target detection unit 15, and a caution determination unit 16. Among these, the gaze detection unit 11, the gaze projection plane setting unit 12, the intersection calculation unit 13, and the projection unit 14 are included in the gaze position detection device.
Each of these units included in the control unit 6 is a functional module realized by a computer program executed on a processor included in the control unit 6.
Each of these units included in the control unit 6 may be mounted on the driving support device 1 separately from the processor included in the control unit 6 as one integrated circuit in which circuits corresponding to the respective units are integrated.

  The line-of-sight detection unit 11 detects the line-of-sight direction of the driver based on the vehicle interior image every time the control unit 6 receives the vehicle interior image from the vehicle interior imaging unit 3. In this embodiment, the line-of-sight detection unit 11 detects the Purkinje image of the light source 2 and the driver's pupil center of gravity reflected by the driver's cornea on the vehicle interior image, and determines the driver's pupil center of gravity according to the positional relationship between the Purkinje image and the pupil center of gravity. Detect gaze direction.

Therefore, for example, the line-of-sight detection unit 11 detects an area where the driver's face is shown on the in-vehicle image.
In this embodiment, at the time of shooting, the driver's face is illuminated with infrared light from the light source 2, and the skin reflectance for the infrared light is relatively high (for example, the skin reflectance is near infrared). (Several tens of percent in the wavelength range), the brightness of the pixels in the car image where the facial skin is reflected is high. On the other hand, since the area of the hair or the area behind the user on the in-vehicle image has a low reflectance for infrared light or is far from the light source 2, the luminance of the pixel in which the area of the hair or the area behind the user is reflected is relatively low. Become.
Therefore, when the value of each pixel of the in-vehicle image is a color image represented by the RGB color system, the line-of-sight detection unit 11 converts the value of each pixel into a value represented by the YUV color system. Then, the line-of-sight detection unit 11 extracts pixels whose luminance component (Y component) value of each pixel is equal to or greater than a predetermined threshold as face area candidate pixels that may have a face. When the value of each pixel in the in-vehicle image is a gray image representing luminance, the line-of-sight detection unit 11 compares the value of each pixel with a predetermined threshold value. For example, the predetermined threshold is set to a value obtained by multiplying the maximum value of the luminance component on the image by 0.8.

In addition, the size of the area occupied by the driver's face on the in-vehicle image is estimated to some extent.
Therefore, the line-of-sight detection unit 11 performs a labeling process on the face area candidate pixels, and sets a set of face area candidate pixels adjacent to each other as a face candidate area. The line-of-sight detection unit 11 determines whether the size of the face candidate area is included in a reference range corresponding to the size of the user's face. If the size of the face candidate area is included in the reference range corresponding to the size of the user's face, the line-of-sight detection unit 11 determines that the face candidate area is a face area in which the driver's face is reflected.

  The line-of-sight detection unit 11 may detect the face area according to any of various other methods for detecting the face area shown on the image. For example, the line-of-sight detection unit 11 performs template matching between the face candidate area and a template corresponding to a general face shape, calculates the degree of coincidence between the face candidate area and the template, and the degree of coincidence is predetermined. If the value is greater than or equal to the value, the face candidate area may be determined as a face area.

Next, the line-of-sight detection unit 11 detects an area corresponding to the eye in the face area. The luminance of the pixel corresponding to the eye is significantly different from the luminance of the pixel corresponding to the periphery of the eye. Accordingly, the line-of-sight detection unit 11 detects edge pixels whose luminance changes in the vertical direction by performing a difference calculation between neighboring pixels in the vertical direction using, for example, a Sobel filter for each pixel in the face region. The line-of-sight detection unit 11 sets, for example, a region surrounded by two edge lines in which edge pixels are connected in a substantially horizontal direction by a predetermined number or more corresponding to the size of the eye.
Alternatively, the line-of-sight detection unit 11 detects an area that most closely matches the template in the face area by template matching between the template representing the image of the eye on the image and the face area, and uses the detected area as the eye area. Also good.

When the eye region is detected, the line-of-sight detection unit 11 detects a pupil region within the eye region. In the present embodiment, the line-of-sight detection unit 11 performs template matching between the template corresponding to the pupil and the eye region, and detects a region having the highest degree of matching with the template in the eye region. Then, when the highest coincidence value is higher than a predetermined coincidence threshold value, the line-of-sight detection unit 11 determines that the pupil is reflected in the detected region.
The line-of-sight detection unit 11 may detect the pupil region by using any of various other methods for detecting the pupil region on the image.

Further, the line-of-sight detection unit 11 detects the Purkinje image of the light source 2 within the eye region.
The luminance of the region where the Purkinje image of the light source 2 is reflected is higher than the luminance of the surrounding region. In addition, the shape of the region where the Purkinje image of the light source 2 is reflected substantially matches the shape of the light emitting surface of the light source 2. Therefore, the line-of-sight detection unit 11 sets two rings that have a shape that substantially matches the contour shape of the light-emitting surface of the light source 2 in the eye region and that have different sizes and centers. Then, the line-of-sight detection unit 11 has a difference value obtained by subtracting the average value of the luminance of the outer pixels from the average value of the luminance of the pixels corresponding to the inner ring, which is larger than a predetermined difference threshold value. A region where the average luminance value is higher than a predetermined luminance threshold is detected. The line-of-sight detection unit 11 sets a region surrounded by the inner ring in the detected region as a Purkinje image of the light source 2. Note that the difference threshold value can be, for example, an average value of difference values between neighboring pixels in the eye region. The predetermined luminance threshold can be set to 80% of the maximum luminance value in the eye region, for example.
Note that the line-of-sight detection unit 11 may detect the region where the Purkinje image of the light source is captured using any of various other methods for detecting the region where the Purkinje image of the light source is captured on the image. .

  The line-of-sight detection unit 11 calculates the average value of the horizontal coordinate values and the average value of the vertical coordinate values of each pixel included in the Purkinje image as the coordinates of the center of gravity of the Purkinje image.

  Since the surface of the cornea is substantially spherical, the position of the Purkinje image of the light source 2 is substantially constant regardless of the line of sight of the driver. On the other hand, the center of gravity of the pupil moves according to the line of sight of the driver. Therefore, the line-of-sight detection unit 11 can detect the driver's line-of-sight direction by obtaining the relative position of the pupil center of gravity relative to the center of gravity of the Purkinje image.

  In this embodiment, the line-of-sight detection unit 11 determines the relative position of the pupil centroid relative to the centroid of the Purkinje image of the light source, for example, the horizontal direction of the centroid of the Purkinje image from the horizontal coordinate and vertical coordinate of the pupil centroid. It is obtained by subtracting the coordinate and the vertical coordinate. Then, the line-of-sight detection unit 11 determines the line-of-sight direction of the driver by referring to a line-of-sight direction reference table that represents the relationship between the relative position of the pupil center of gravity and the user's line-of-sight direction. The line-of-sight direction reference table is created in advance, for example, by detecting the position of the center of the pupil and the center of gravity of the Purkinje image when the driver looks at the direction for each of a plurality of known directions. Is remembered.

Note that the line-of-sight detection unit 11 may detect the line-of-sight direction of the driver by a method that does not use the Purkinje image. For example, the line-of-sight detection unit 11 may detect the line-of-sight direction of the driver based on the relative position of the center of gravity of the pupil with reference to the position of a characteristic part of the driver's face on the in-vehicle image.
For example, the line-of-sight detection unit 11 obtains coordinates on an image of a characteristic part such as a nasal apex or an eye that has a small change in relative positional relationship with the eye. For this purpose, the line-of-sight detection unit 11 performs, for example, template matching while changing the relative position between the face region and a template corresponding to the general shape of these characteristic parts, The degree of coincidence of is calculated. The line-of-sight detection unit 11 determines that a part corresponding to the template is captured at the specific position when the degree of coincidence is a predetermined value or more at the specific position in the face region.

The line-of-sight detection unit 11 is obtained from a reference image in which the driver's face is photographed in a state where the driver is facing a specific direction from the coordinates of the characteristic part, and the feature stored in the storage unit 5 The position correction amount is obtained by subtracting the coordinates of the reference position of a typical part. Then, the line-of-sight detection unit 11 subtracts the position correction amount from the coordinate value of the pupil centroid to cancel the difference between the position of the pupil centroid on the in-vehicle image and the position of the pupil centroid on the reference image corresponding to the position correction amount. Thus, the corrected coordinates of the center of gravity of the pupil are obtained.
The line-of-sight detection unit 11 determines the line-of-sight direction of the driver corresponding to the corrected pupil center of gravity by referring to the pupil-gaze direction reference table stored in the storage unit 5.

  The driver's line-of-sight direction can be expressed as a unit vector in the world coordinate system. Therefore, the line-of-sight detection unit 11 notifies the intersection point calculation unit 13 of the unit vector representing the line-of-sight direction of the driver and the eye position on the in-vehicle image.

  The line-of-sight projection plane setting unit 12 sets the line-of-sight projection plane, which is a virtual plane for projecting the driver's line of sight, to at least a part of the periphery of the vehicle 10. In the present embodiment, the line-of-sight projection plane setting unit 12 moves the line-of-sight projection plane away from the vehicle 10 by a distance from the vehicle 10 to a position where there is a possibility that an object to be watched exists in parallel with the traveling direction of the vehicle 10. Set to position. In this embodiment, in order to facilitate the handling of the line-of-sight projection plane in the world coordinate system, the distance from the vehicle 10 to the line-of-sight projection plane is determined from the vehicle exterior imaging unit 4 mounted on the vehicle 10 to the line-of-sight projection plane. Expressed as a distance. However, the distance from the vehicle 10 to the line-of-sight projection plane may be represented by the distance from the center line of the vehicle 10 to the line-of-sight projection plane, which is parallel to the side surface of the vehicle 10 or the traveling direction of the vehicle 10.

  FIG. 4 is a diagram illustrating an example of a line-of-sight projection plane. As shown in FIG. 4, the line-of-sight projection plane 400 is set so as to surround the left side, the right side, and the upper side of the vehicle 10. In this example, the right gaze projection plane 400a and the left gaze projection plane 400b are set along a direction perpendicular to the road surface. On the other hand, the upper line-of-sight projection plane 400c is set parallel to the road surface. In order to avoid dangers such as a collision in advance during driving, the driver may pay attention to another vehicle traveling in the oncoming lane or a pedestrian or bicycle walking on the side road. Therefore, the line-of-sight projection plane setting unit 12 sets the distance from the vehicle exterior imaging unit 4 to the line of sight projection plane set on the right side of the vehicle 10, for example, the distance from the vehicle exterior imaging unit 4 to the opposite lane, or the vehicle exterior imaging unit 4. To the right end of the road on which the vehicle 10 is traveling. This distance may be set in advance and stored in the storage unit 5.

  Alternatively, the line-of-sight projection plane setting unit 12 estimates the distance from the vehicle exterior imaging unit 4 to the center line or the right end of the road by detecting the center line or the right end of the road drawn on the road on the vehicle exterior image. May be. When detecting the center line drawn on the road or the right end of the road on the outside image, the line-of-sight projection plane setting unit 12 executes, for example, an edge detection process in an area where the road on the outside image is shown. To detect edge pixels in the region. Then, the line-of-sight projection plane setting unit 12 performs a Hough transform on the edge pixels, for example, so that a straight line extending along the traveling direction of the vehicle 10 in a region corresponding to the right side of the vehicle 10 What is necessary is just to detect as a right end. The line-of-sight projection plane setting unit 12 may detect the center line or the right end of the road according to any of various other methods for detecting the end of the road.

  The optical axis direction of the outside imaging unit 4 and the angle of view of the outside imaging unit 4 are known in advance. Therefore, the direction toward the object reflected in an arbitrary pixel on the vehicle outside image can be known from the vehicle outside imaging unit 4. Further, the height from the road surface to the vehicle exterior imaging unit 4 is also known. Therefore, the line-of-sight projection plane setting unit 12 can also determine the distance from the vehicle exterior imaging unit 4 to the road surface reflected in the pixel by specifying the pixel where the road surface is captured in the vehicle exterior image. Therefore, the line-of-sight projection plane setting unit 12 can obtain the distance from the vehicle exterior imaging unit 4 to the center line or the right end of the road based on any pixel in which the center line or the right end of the road is shown.

ただし、(x,y,z)は、車外撮像部４の前側主点を原点とする世界座標系における座標を表す。このうち、zは、車両１０の進行方向、すなわち、車外撮像部４の光軸に平行な方向の座標を表す。またxは、路面と平行で、かつ車両１０の進行方向に直交する方向の座標を表す。そしてyは、路面鉛直方向の座標を表す。なお、xは、原点から右へ向かう方向が正であり、yは、原点から上方へ向かう方向が正であり、zは、原点から車両１０の前方へ向かう方向が正である。 If the distance from the vehicle exterior imaging unit 4 to the line-of-sight projection plane set on the right side of the vehicle 10 is D, the line-of-sight projection plane s set on the right side of the vehicle 10 is expressed by the following equation.

However, (x, y, z) represents coordinates in the world coordinate system with the front principal point of the outside-vehicle imaging unit 4 as the origin. Among these, z represents the coordinate of the advancing direction of the vehicle 10, ie, a direction parallel to the optical axis of the imaging unit 4 outside the vehicle. X represents coordinates in a direction parallel to the road surface and orthogonal to the traveling direction of the vehicle 10. And y represents the coordinate in the road surface vertical direction. Note that x is positive in the direction from the origin to the right, y is positive in the direction from the origin to the upper side, and z is positive in the direction from the origin to the front of the vehicle 10.

  Further, the line-of-sight projection plane setting unit 12 sets the distance from the vehicle exterior imaging unit 4 to the line-of-sight projection plane set on the left side of the vehicle 10, for example, the distance from the vehicle exterior imaging unit 4 to the left end of the road on which the vehicle 10 is traveling. Set to. Also in this case, the distance from the vehicle exterior imaging unit 4 to the left end of the road on which the vehicle 10 is traveling may be stored in the storage unit 5 in advance. Alternatively, the line-of-sight projection plane setting unit 12 may determine the distance by detecting the left end of the road on the outside image. Note that the line-of-sight projection plane s ′ set on the left side of the vehicle 10 is represented by the following formula (1), where D is the distance from the vehicle exterior imaging unit 4 to the line-of-sight projection plane set on the left side of the vehicle 10. Instead, it is calculated by (-D).

  Further, the line-of-sight projection plane setting unit 12 determines the distance from the vehicle exterior imaging unit 4 to the line-of-sight projection plane set on the upper side of the vehicle 10, for example, the height from the vehicle exterior imaging unit 4 to a road sign or a traffic light (for example, 4 m ).

  Note that the line-of-sight projection plane setting unit 12 sets the line-of-sight setting plane only on the right side and the left side of the vehicle 10 and does not need to set the line-of-sight projection plane above the vehicle 10. This is because an object that approaches the vehicle 10 from above the vehicle 10 generally does not exist. In addition, the line-of-sight projection plane setting unit 12 may set the line-of-sight projection plane around the vehicle 10 in a cylindrical surface centered on an axis parallel to the traveling direction of the vehicle 10.

  The line-of-sight projection plane setting unit 12 notifies the intersection calculation unit 13 of an expression representing the line-of-sight projection plane.

The intersection calculation unit 13 calculates an intersection between the line-of-sight projection plane and the driver's line of sight. The driver's line of sight is represented by v (x, y, z) as a unit vector representing the driver's line of sight, and g (x, If y, z), it is represented by a straight line (g + kv). However, k is a real number having a positive value. Therefore, the intersection calculation unit 13 can calculate the intersection between the line-of-sight projection plane and the driver's line of sight according to the following equation. In the following expression, v and g are expressed in homogeneous coordinate expression, and are v (x, y, z, 0) and g (x, y, z, 1), respectively.

The driver's eye position is determined based on, for example, the eye position detected from the in-vehicle image generated by the in-vehicle imaging unit 3. Arbitrary pixels on the in-vehicle image and the direction from the in-vehicle image capturing unit 3 to the subject shown in the pixel correspond to each other on a one-to-one basis. Therefore, by specifying a pixel corresponding to the center of gravity of the eye on the in-vehicle image, the direction from the in-vehicle imaging unit 3 toward the driver's eyes is specified. Since the optical axis direction, the angle of view, and the installation position of the in-vehicle image capturing unit 3 are known, a straight line from the in-vehicle image capturing unit 3 toward the driver's eyes in the world coordinate system can be obtained. Furthermore, the distance from the in-vehicle imaging unit 3 to the driver is also estimated in advance based on the positional relationship between the in-vehicle imaging unit 3 and the seat. Therefore, it is estimated that the driver's eyes are located at a position away from the in-vehicle image capturing unit 3 by an estimated distance from the in-vehicle image capturing unit 3 to the driver's eyes along the straight line.
Note that the position of the eyes of the driver may be measured in advance and stored in the storage unit 5.

Further, the eye position on the in-vehicle image obtained by photographing the driver's head when the driver's eye position in the real space is known is stored in the storage unit 5 in advance as the eye reference position. May be. The intersection calculation unit 13 obtains a difference between the eye position on the in-vehicle image obtained during execution of the gaze position detection process and the eye reference position, and according to the difference, the driver corresponding to the reference position The position of the driver's eye may be obtained by correcting the position of the eye.
Further, the intersection calculation unit 13 may detect the position of the driver's eye using any of various other techniques for detecting the position of the driver's eye.

  The intersection calculation unit 13 notifies the projection unit 14 of the coordinates of the intersection between the driver's line of sight and the line-of-sight projection plane.

  The projection unit 14 projects the intersection of the driver's line of sight and the line-of-sight projection plane onto the vehicle exterior image. As described above, the parameters related to the vehicle exterior imaging unit 4 such as the installation position of the vehicle exterior imaging unit 4, the optical axis direction, the angle of view, and the size of one pixel are known. Therefore, when the distortion aberration of the imaging optical system of the imaging unit 4 outside the vehicle is so small as to be negligible, the projection unit 14 uses a projection matrix F that projects an arbitrary point on the world coordinate system on the outside image based on these parameters. Can be determined. Alternatively, in order to determine the value of each element of the projection matrix F, a chart showing a known pattern is arranged at a position away from the vehicle exterior imaging unit 4 by a predetermined distance, and the chart is displayed by the vehicle exterior imaging unit 4. An image obtained by photographing may be used. In this case, a plurality of feature points on the chart are extracted from the image showing the chart. Then, each element of the projection matrix F so that the sum of squares of the distance between the coordinates and the feature points when the points on the chart corresponding to the feature points are projected onto the outside image using the projection matrix F is minimized. Is calculated. For example, the projection matrix F may be calculated and stored in the storage unit 5 when the driving support apparatus 1 is installed in the vehicle 10 or in a calibration process that is periodically performed.

なお、（３）式では、交点q及び対応する車外画像上の注視位置pを同次座標表現で表している。すなわち、（３）式で得られた注視位置pの一番上の要素を1/zした値が、車外画像上での実際の水平方向の座標uとなり、（３）式で得られた注視位置pの上から2番目の要素を1/zした値が、車外画像上での実際の垂直方向の座標wとなる。 The projection unit 14 uses the projection matrix F to calculate the gaze position p (u, w) on the outside image corresponding to the intersection point q (x, y, z) between the driver's line of sight and the line-of-sight projection plane by the following equation. To do. However, u is a coordinate in the horizontal direction on the outside image, and w is a coordinate in the vertical direction on the outside image.

In the equation (3), the intersection point q and the corresponding gaze position p on the outside image are expressed in homogeneous coordinate expression. That is, the value obtained by 1 / z the top element of the gaze position p obtained by the equation (3) becomes the actual horizontal coordinate u on the outside image, and the gaze obtained by the equation (3). The value of 1 / z of the second element from the top of the position p is the actual vertical coordinate w on the outside image.

  FIG. 5 is a diagram illustrating an example of the relationship between the intersection between the driver's line of sight and the line-of-sight projection plane and the gaze position on the vehicle exterior image. When the distance D from the vehicle exterior imaging unit 4 to the line-of-sight projection plane 502 is set so that the gaze target object 501 of the driver 500 is positioned in the vicinity of the line-of-sight projection plane 502, the line of sight 503 of the driver 500 and the line-of-sight projection plane The intersection point q with 502 is located near the gaze target object 501. Therefore, even if the driver 500 and the outside imaging unit 4 are arranged apart from each other, the angle formed by the line 511 connecting the outside imaging unit 4 and the intersection point q with the optical axis OA of the outside imaging unit 4 and the outside imaging unit 4 The difference between the line 512 connecting the gaze object 501 and the angle formed with the optical axis OA is small. Therefore, the position p on the imaging surface 504 of the imaging unit 4 outside the vehicle corresponding to the intersection point q and the position p ′ on the imaging surface 504 of the imaging unit 4 outside the vehicle corresponding to the gaze target 501 substantially coincide. Therefore, without measuring the distance from the driver to the gaze target, the control unit 6 can bring the gaze position on the vehicle outside image closer to the position on the vehicle outside image of the object actually watched by the driver.

  Note that, as in the case where the imaging optical system of the imaging unit 4 outside the vehicle is a fisheye lens, the distortion aberration of the imaging optical system may be so large that it cannot be ignored. In such a case, for example, a correction table representing a correspondence relationship between an angle from the optical axis of the outside-vehicle imaging unit 4 and a correction amount for correcting a position shift on the outside-vehicle image due to distortion is stored in the storage unit 5 in advance. Is done. Such a correction table can be created by analyzing an image obtained by photographing a chart on which a known pattern is shown in advance using the outside-vehicle imaging unit 4. Alternatively, when the distortion aberration curve of the imaging optical system of the outside-vehicle imaging unit 4 is known, the correction table may be created based on the distortion aberration curve.

  The projection unit 14 calculates an angle formed by a line from the front principal point of the vehicle exterior imaging unit 4 to the intersection and the optical axis of the vehicle exterior imaging unit 4, and obtains a correction amount corresponding to the angle with reference to the correction table. Further, the projection unit 14 obtains a straight line on the outside image from the pixel on the outside image corresponding to the optical axis toward the projection point p on the outside image of the intersection q calculated by the expression (3). Then, the projection unit 14 takes the position obtained by subtracting the distance from the pixel corresponding to the optical axis to the projection point p calculated by Equation (3) by the correction amount along the straight line as the coordinate of the gaze position p. Good.

  The projection unit 14 notifies the attention determination unit 16 of the coordinates of the gaze position p on the outside image.

  The gaze target detection unit 15 detects an object to be watched by the driver on the outside image. For example, the gaze target detection unit 15 divides the image outside the vehicle into a plurality of partial images, and extracts a feature amount representing the target object to be watched for each partial image. Then, the gaze target detection unit 15 inputs the feature amount for each partial image to the discriminator that identifies whether or not it is a target to be watched, thereby determining whether or not the target to be watched is reflected in the partial image. judge. The classifier can be, for example, an adaBoost classifier or a support vector machine. The feature amount can be, for example, a Haar-like feature amount.

  Note that the gaze target detection unit 15 may track an object to be noted over a plurality of temporally continuous images using any of various tracking methods. Then, the gaze target detection unit 15 is only in the case where the position of the object on the outside image is approaching the center of the outside image, that is, when it is estimated that the object is approaching the vehicle 10. The object may be an object to be noted.

  The gaze target detection unit 15 notifies the attention determination unit 16 of information indicating a region where the target object to be watched is shown.

  The attention determination unit 16 determines whether or not the coordinates of the gazing point on the image outside the vehicle are included in any region of the object to be gazed. The attention determination unit 16 determines that the driver has seen the object to be watched when the coordinates of the gaze point are included in a region where any of the objects to be watched is shown.

  The attention determination unit 16 determines whether or not the driver is looking at the target object when the target object is shown in the outside image generated during the predetermined period for the most recent predetermined period. Note that the predetermined period is set to about 0.1 seconds to 1 second, for example. When the driver has not looked at the target object for the predetermined period, the attention determination unit 16 causes, for example, a speaker (not shown) to emit a warning sound so as to alert the driver. Alternatively, the caution determining unit 16 surrounds the latest outside-of-vehicle image and an object to be noted detected on the outside-of-vehicle image (for example, surrounding the object) on a display (not shown) arranged in the vehicle. A specific color frame) may be displayed.

  FIG. 6 is an operation flowchart of driving support processing including gaze position detection processing controlled by the control unit 6. Each time the control unit 6 receives an image outside the vehicle, the control unit 6 executes a driving support process according to the operation flowchart.

  The line-of-sight detection unit 11 detects the line-of-sight direction of the driver based on the latest vehicle interior image received from the vehicle interior imaging unit 3 (step S101). The line-of-sight detection unit 11 notifies the intersection point calculation unit 13 of a direction vector representing the line-of-sight direction.

  The line-of-sight projection plane setting unit 12 is arranged around the vehicle 10 at a position parallel to the traveling direction of the vehicle 10 and away from the outside-of-vehicle imaging unit 4 by a distance at which an object to be noted may be located. Is set (step S102). Then, the line-of-sight projection plane setting unit 12 notifies the intersection calculation unit 13 of the line-of-sight projection plane.

  The intersection calculation unit 13 calculates the intersection between the driver's line of sight and the line-of-sight projection plane (step S103). The coordinates of this intersection are expressed in the world coordinate system. Then, the intersection calculation unit 13 notifies the projection unit 14 of the coordinates of the intersection.

  The projection unit 14 projects the intersection of the driver's line of sight and the line-of-sight projection plane onto the image outside the vehicle, and specifies the gaze position (step S104). Then, the projection unit 14 notifies the attention determination unit 16 of the coordinates of the gaze position on the outside image.

  In addition, the gaze target detection unit 15 detects a region in which an object to be watched by the driver is shown on the image outside the vehicle (step S105). The gaze target detection unit 15 notifies the attention determination unit 16 of the position and range of the region in which the target object to be watched is shown.

  The attention determination unit 16 determines whether or not the driver's gaze position on the vehicle outside image is included in any of the regions where any object to be noted is shown (step S106). When the gaze position is not included in any of the regions in which the object to be watched is reflected (No in step S106), the attention determination unit 16 determines that the driver should pay attention to the object to be watched when generating the outside image. Judge that you have not seen. Then, the attention determination unit 16 determines whether or not the period during which the driver does not look at the object to be noted continues for a predetermined period (step S107). When the period during which the driver does not look at the object to be watched continues for a predetermined period (step S107—Yes), the attention determination unit 16 determines that the object to be watched exists around the vehicle 10. Is warned (step S108). Then, the control part 6 complete | finishes a driving assistance process.

  On the other hand, in step S107, when the driver's gaze position is included in an area in which any target object to be watched is reflected (Yes in step S106), the attention determination unit 16 determines that the driver is the target to be watched. Judge that you saw an object. In this case, the control unit 6 ends the driving support process after resetting the length of the continuation period in which the target object to be noted without being warned to the driver is not reset to 0. Also, in step S108, when the continuation time during which the driver is not looking at the object to be noted has not reached the predetermined period (step S107-No), the control unit 6 performs the driving support process without warning the driver. finish.

  As described above, the driving support device in which the gaze position detection device is mounted is an intersection of the driver's line of sight and the line-of-sight projection plane set at a position where there is a possibility that an object to be noted exists. And the intersection is projected onto the image outside the vehicle. Due to the nature of the line-of-sight projection plane, the driving support device can approximate the distance from the outside imaging unit to the object that the driver is actually gazing with by the distance from the outside imaging unit to this intersection. Therefore, this driving support device can improve the accuracy of the gaze position on the outside image without measuring the distance from the outside imaging unit to the object that the driver is actually looking at. In addition, since this driving support device does not need to measure the distance from the vehicle exterior imaging unit to the object that the driver is actually watching, the configuration of the driving support device can be simplified.

  In addition, this invention is not limited to said embodiment. For example, the gaze position detection device may be mounted on a gaze object analysis device that analyzes an object that the driver gazes during driving. The gaze object analyzing apparatus can also have a configuration similar to that of the driving support apparatus shown in FIG. In this case, for example, the line-of-sight projection plane is set according to the distance from the vehicle exterior imaging unit 4 to the position where the object to be analyzed may be located. For example, when it is desired to specify an advertising billboard that the driver has watched, the line-of-sight projection plane is separated from the outside imaging unit 4 by a distance from the outside imaging unit 4 to the estimated position of the advertising signboard, and the traveling direction of the vehicle 10 What is necessary is just to set in parallel.

  In this example, the gaze object analyzing apparatus is a position detection unit (not shown) that detects the current position of the vehicle 10 based on, for example, positioning information from a GPS satellite of the Global Positioning System (GPS). ) May further be included. Furthermore, the memory | storage part of the gaze object analysis apparatus may memorize | store the map information showing the position and advertisement content of an advertisement signboard in the area where the vehicle 10 is traveling.

  In this example, the gaze target detection unit 15 of the control unit 6 reads, from the storage unit 5, for example, a template corresponding to an advertising billboard near the current position based on the current position of the vehicle 10. Then, the gaze target detection unit 15 determines that the advertising signboard corresponding to the template is shown in the region that most matches the template on the outside image by template matching between the outside image and the template.

  Further, the attention determination unit 16 of the control unit 6 determines that the driver has seen the advertisement signboard when the driver's gaze position is included in the area where it is determined that the advertisement signboard is reflected on the image outside the vehicle. When the attention determination unit 16 determines that the driver has seen the advertisement signboard in each of the images outside the vehicle acquired during the most recent predetermined period, the attention determination unit 16 reads the advertisement content of the watched advertisement signboard from the storage unit 5. . Then, the attention determination unit 16 refers to the map information, and stores related to the advertisement content of the advertising signboard that the driver watched within a predetermined range (for example, a range of 2 km × 2 km) around the current position of the vehicle 10. Is detected. For example, if the advertisement content of the advertising billboard watched by the driver is a restaurant advertisement, the attention determination unit 16 detects the restaurant. And the attention determination part 16 displays the information which shows the name and position of the detected shop with the map around the present position of the vehicle 10 on the display part (not shown) installed in the vehicle.

According to still another modification, the gaze object analyzing apparatus includes a communication unit (not shown) for wirelessly communicating with a base station of a mobile communication system installed outside the vehicle, via the base station. It may be possible to communicate with a predetermined place outside the vehicle, for example, a server installed in a data center. In this case, the gaze object analyzing apparatus transmits a signal including information indicating the current position of the vehicle 10, the position of the advertisement signboard watched by the driver, and the advertisement content to the server via the communication unit and the base station. The server refers to the map information, and detects a store related to the advertisement content of the advertising signboard that the driver watched within a predetermined range around the current position of the vehicle 10. Then, the server returns information indicating the name and position of the detected store to the gaze object analyzing device installed in the vehicle 10.
The control unit 6 of the gaze object analyzing apparatus displays information indicating the name and position of the detected store together with a map around the current position of the vehicle 10 on a display unit (not shown) installed in the vehicle.

According to still another modified example, the control unit 6 attaches a mark indicating a gaze position to each of the outside images generated during a certain period while the vehicle 10 is traveling, and the outside image having the marks attached thereto. May be stored in the storage unit 5.
As a result, the gaze object analyzing apparatus can analyze the movement of the driver's line of sight by tracking the gaze position on the image outside the vehicle for a certain period.

  In addition, the computer program that realizes the function of each unit of the control unit according to the above-described embodiments and modifications is recorded in a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. May be provided. The recording medium does not include a carrier wave.

  All examples and specific terms listed herein are intended for instructional purposes to help the reader understand the concepts contributed by the inventor to the present invention and the promotion of the technology. It should be construed that it is not limited to the construction of any example herein, such specific examples and conditions, with respect to showing the superiority and inferiority of the present invention. Although embodiments of the present invention have been described in detail, it should be understood that various changes, substitutions and modifications can be made thereto without departing from the spirit and scope of the present invention.

The following supplementary notes are further disclosed regarding the embodiment described above and its modifications.
(Appendix 1)
Detects the direction of the line of sight of the vehicle occupants,
An imaginary plane at a position away from the vehicle by a distance from the vehicle to a position where there is a possibility that an object to be watched by an occupant is present at least partially around the vehicle. Set
Finding the intersection of the virtual plane and the sight line of the occupant,
A gaze position for causing a computer to detect a point on the image obtained by projecting the intersection on an image in which at least a part of the scenery around the vehicle is captured as the gaze position of the occupant Computer program for detection.

1 Driving support device (gaze position detection device)
2 Light source 3 In-vehicle imaging unit 4 Outside imaging unit 5 Storage unit 6 Control unit 11 Gaze detection unit 12 Gaze projection plane setting unit 13 Intersection calculation unit 14 Projection unit 15 Gaze target detection unit 16 Attention determination unit

Claims (3)

An imaging unit that captures at least a part of the scenery around the vehicle and generates an image of the scenery;
A line-of-sight detection unit for detecting the direction of the line of sight of the occupant of the vehicle;
An imaginary plane at a position away from the vehicle by a distance from the vehicle to a position where there is a possibility that an object to be watched by an occupant is present at least partially around the vehicle. A projection plane setting unit for setting
An intersection calculation unit for obtaining an intersection between the virtual plane and the line of sight of the occupant;
A gaze position detection apparatus comprising: a projection unit that detects a point on the image obtained by projecting the intersection point on the image as a gaze position of the occupant. The projection plane setting unit detects a center line of a road on which the vehicle travels or an end of the road on the image, and places the virtual plane at a position of the center line of the road or the end of the road. The gaze position detection device according to claim 1, wherein the gaze position detection device is set. Get an image of at least part of the scenery around the vehicle,
Detecting the direction of the line of sight of the vehicle occupant,
An imaginary plane at a position away from the vehicle by a distance from the vehicle to a position where there is a possibility that an object to be watched by an occupant is present at least partially around the vehicle. Set
Finding the intersection of the virtual plane and the sight line of the occupant,
A gaze position detection method including detecting a point on the image obtained by projecting the intersection point on the image as a gaze position of the occupant.

Images