JPH10178564A - Panorama image generator and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a panorama image generator in which deterioration in the image quality due to deviation in the image or double exposure is less even with regard to the image including a parallax caused by a difference of the depth in the case of photographing when movement of a center of a lens of a camera is unavoidable for photographing a scene by the camera or the like. SOLUTION: A panorama image generator is provided with an image input device 101, a frame memory 102 that stores inputted divided images, an image position calculation part 106 that calculates a synthesis position of the divided images, an image parallax extract processing part 107 that extracts parallax information between images from the image including a parallax in an overlap area between adjacent divided images, an intermediate image production processing part 108 that produces plural intermediate images from the image including the parallax and an image synthesis processing part 109 that synthesizes a panorama image from the divided images. The image parallax extract processing part 107 corrects a deviation in the images in an overlap area and interpolates by the intermediate image produced by the intermediate image production processing part 108 to synthesize the panorama image.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a panoramic image creating apparatus for synthesizing a panoramic image from a plurality of still digital images.

[0002]

2. Description of the Related Art A method of connecting a plurality of digital images having overlapping areas to each other to create a panoramic image or a high-definition (high-resolution) image is disclosed, for example, in Japanese Patent Application Laid-Open No. 3-182977. A method of extracting two feature particles between adjacent images and connecting the images by using a line connecting them as a joining line, or minimizing the density difference as disclosed in JP-A-5-122606. There is a method of connecting images in such an area. Also, when joining images, instead of connecting them with a line like a joint line, a method of smoothing the color so that the density change of both images does not become discontinuous around the joint line is published by the University of Tokyo Press. Image Analysis Handbook, page 466.

[0003]

However, in photographing a digital image, when photographing a hand-held pan, when photographing a pan using a tripod, or when installing and photographing with a plurality of cameras so as to overlap each other, etc. There is. In pan shooting by hand or shooting with a plurality of cameras, the lens position of the camera moves as shown in FIG. Also, in many combinations of a camera and a tripod, the center of rotation does not coincide with the center of the lens of the camera, so that some movement of the lens occurs. These differences in the lens positions cause parallax due to the difference in the depth of the subject in the captured image. Therefore, as shown in FIG. However, the image in the overlap does not match even when a portion that is not shown on the other side occurs, or when a moving object is photographed due to a time lag in photographing.

In the conventional image joining technology, joining lines are joined by straight lines or straight lines having a width. However, when there is a difference between images as described above, there is a case where the images cannot be combined well. Many. In particular, even if some parts are connected naturally, in places where parallax occurs or where moving objects are included, misalignment on the joining line or double image occurs, and the quality of the joined panoramic image is reduced. This is the cause of the significant decrease.

[0005] The present invention solves the above-mentioned problems of the prior art. In taking a picture with a camera or the like, when the center of the camera cannot be avoided, the difference in depth is caused by the difference in depth. Provided is a panoramic image creating apparatus in which an image including parallax is less likely to deteriorate in image quality due to image shift and double image. Also, when the camera lens center is unavoidable to move, or when it is unavoidable to include a moving object, parallax caused by a difference in depth, and for images including moving objects,
Provided is a panoramic image creating apparatus in which image quality is less likely to be reduced due to image shift and double image capturing.

[0006]

According to a first aspect of the present invention, there is provided a panoramic image creating apparatus in which a plurality of image pickup means or an image pickup means is moved a plurality of times to partially overlap a picked-up image. What is claimed is: 1. A panorama image creating apparatus for creating a panoramic image by joining captured images and obtaining the obtained divided images, said imaging means; a storage means for storing a divided image input from said imaging means; Image input processing means for instructing the means for capturing the divided image and for specifying the storage address of the captured divided image for the storage means, and calculating the composite position of the divided image. Image position calculating means; disparity information extracting means for extracting disparity information between images from an image including disparity in an overlap area between adjacent divided images; An intermediate image creating unit that creates a plurality of intermediate images from an image including parallax in the overlap region; and an image combining unit that combines a panoramic image from the divided images. It is characterized in that the panoramic image is synthesized by interpolating with the intermediate image created by the intermediate image creating unit, while being corrected by the information extracting unit.

According to a second aspect of the present invention, there is provided a panoramic image creating apparatus which is obtained by moving a plurality of image pickup means or moving the image pickup means a plurality of times to overlap a part of the picked-up image. What is claimed is: 1. A panorama image creating apparatus for joining a divided image to create a panoramic image, comprising: an image pickup unit; a storage unit configured to store a divided image input from the image pickup unit; Image input processing means for instructing capture of an image, and for instructing the storage destination address of the captured divided image, image position calculating means for calculating a combined position of the divided images, Parallax information extracting means for extracting parallax information between images from an image including parallax in an overlap area between the adjacent divided images, and parallax information extracting means. Based on the obtained depth information, a joining line determining unit that sets a joining line on the contour of the subject closest to the imaging unit of the image of the overlap area, and an image combining unit that combines a panoramic image from the divided images. Wherein the displacement of the image in the overlap area is corrected by the parallax information extracting means, and the divided images are joined based on the joining lines obtained by the joining line determining means.

According to a third aspect of the present invention, there is provided the panoramic image creating apparatus according to the second aspect, wherein the joining line determining means determines whether or not the subject closest to the image capturing means for the image of the overlap area is overlaid. The present invention is characterized in that a joining line is set at a contour closest to an image end on the lap area side. According to a fourth aspect of the present invention, there is provided the panoramic image creating apparatus according to the second or third aspect, wherein the joining line determining unit is configured to determine a reference for determining a joining line in the overlap area. The method is characterized in that an optimal joint line is selected from a plurality of joint lines by combining the image selection and the joint line determination method, and the divided images are joined.

According to a fifth aspect of the present invention, there is provided a recording medium comprising: a plurality of image pickup means, or a plurality of movements of the image pickup means, and a photographed image obtained by overlapping a part of a photographed image; Is a program for creating a panoramic image by linking the image data. The image input processing means instructs the imaging means to capture the divided image, and instructs the storage means to specify the storage destination address and captures the divided image. The divided image thus stored is stored in a storage unit, an image position calculating unit calculates a combined position of the divided images, and a disparity information extracting unit includes a disparity in an overlap area between adjacent divided images. Causing the parallax information between the images to be extracted from the image, and causing the intermediate image generating means to generate a plurality of intermediate images from the image including the parallax in the overlap area,
The image synthesizing unit synthesizes a panoramic image from the divided images. At this time, the parallax information extracting unit corrects the image shift in the overlap area, and interpolates with the intermediate image created by the intermediate image creating unit. And a program for executing an operation of synthesizing a panoramic image.

According to a sixth aspect of the present invention, there is provided a recording medium comprising: a plurality of image pickup means, or a plurality of image pickup means, which are moved a plurality of times, and a part of the picked-up image is overlapped and photographed; Is a program for creating a panoramic image by linking the image data. The image input processing means instructs the imaging means to capture the divided image, and instructs the storage means to specify the storage destination address and captures the divided image. The divided image thus stored is stored in a storage unit, an image position calculating unit calculates a combined position of the divided images, and a disparity information extracting unit includes a disparity in an overlap area between adjacent divided images. Parallax information between the images is extracted from the image, and a joining line determining unit captures an image of the overlap region based on the depth information obtained by the parallax information extracting unit. A joining line is set on the contour of the subject closest to the step, and a panoramic image is synthesized from the divided images by the image synthesizing unit. At this time, the disparity information extracting unit corrects the image shift in the overlap area. In addition, a program for executing an operation of synthesizing a panoramic image by joining the divided images based on the joining line determined by the joining line determining means is stored.

[0011]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing an example of an embodiment according to claim 1 of the present invention. 101 is an image input device, 102 is a frame memory,
103 is an image output device, 104 is an image input processing unit, 10
5 is an image preprocessing unit, 106 is an image position calculation unit, 107 is an image parallax extraction processing unit, 108 is an intermediate image generation processing unit, 1
09 is an image synthesis processing unit, 110 is an image output processing unit, 11
1 is a timing control unit, and 112 is a RAM. FIG.
FIG. 3 is a diagram showing a configuration example of the RAM 112. Here, the RAM 112 is a place for obtaining information from a predetermined address and storing information generated by each process by some processes described below.

FIG. 8 is a diagram showing a procedure of processing according to an embodiment of the present invention.

Hereinafter, the contents relating to the first embodiment of the present invention will be described with reference to FIG.

When the entire system is started, the timing control unit 111 controls the image input processing unit 104, the image preprocessing unit 1
05, image position calculation unit 106, image parallax extraction processing unit 10
7, intermediate image generation processing unit 108, image synthesis processing unit 10
9. Send a reset signal to the image output processing unit 110.

The processing in each block enters a wait state until a start signal from the timing control unit 111 is received. First, the timing control unit 111 sends a start signal to the image input processing unit 104 to store an input image in the frame memory 102, and enters a wait state until receiving an end signal from the image input processing unit 104.

When the image input processing unit 104 receives a start signal from the timing control unit 111, the image input processing unit 104
Then, a process of transferring the image to the frame memory 102 is performed. If an image is to be synthesized online, the image input device 101 comprises image input means and A / D conversion means for converting an analog image signal obtained from the image input means into a digital image signal. Processing unit 1
The image signal is converted into a digital image signal by an A / D conversion unit for capturing an image and an analog image signal based on the signal from the signal 04, and is transferred to a predetermined address of the frame memory 102. In the case of off-line image synthesis, the image input device 101 corresponds to an image storage device, and the image input processing unit 1
Step 04 is a process of transferring an image signal stored in the image input device 101 to a predetermined address of the frame memory 102. When the transfer of all the images to the frame memory 102 is completed, the image input processing unit 104 sequentially writes the number of input images and the respective storage addresses to predetermined addresses in the RAM 112. After that, the image input processing unit 104
Transmits an end signal notifying the end of the image input processing to the timing control unit 111, and ends the processing.

When the timing control unit 111 receives the end signal of the transfer of the input image to the frame memory, it transmits a start signal to the image preprocessing unit 105, and
It enters a wait state until it receives an end signal from.

When receiving the start signal from the timing control unit 111, the image preprocessing unit 105 reads out and stores the number of images to be processed from the RAM 112, and sets the counter k to one. Next, the address in the frame memory 102 of the image I (k) corresponding to the counter k is read from the RAM 112, and a predetermined image in the frame memory 102 is read from the address (see FIGS. 8A and 8B). . The read image I (k) is subjected to a process of removing a known distortion from the image. For example, there is aberration correction of a lens of an imaging system. The image from which the image has been removed is written again at the same address in the frame memory 102. Thereafter, the counter k is incremented by 1, and the counter k is compared with the number of images. If the counter k is equal to or less than the number of images, preprocessing for the next image is performed. If the counter k is larger than the number of images, the image preprocessing unit 105 transmits an end signal notifying the end of the image preprocessing to the timing control unit 111 and ends the processing.

Upon receiving the end signal from the image pre-processing unit 105, the timing control unit 111
A start signal is transmitted to 06 and a wait state is entered until an end signal from the image position calculation unit 106 is received.

Upon receiving the start signal from the timing control unit 111, the image position calculation unit 106 reads out and stores the number of images to be processed from the RAM 112, and sets a counter k to one. Next, the address of the image corresponding to the counter k in the frame memory 102 is read out from the RAM 112, and from the address, the predetermined image I (k) in the frame memory 102 is read as shown in FIGS. 9A and 9B. The part to be the reference area for pattern matching as shown (rx0, ry
0)-(rx1, ry1) is read.

Further, an image I (k +
The search area (sx0, sy0)-(sx1, sy1) of 1) is similarly read. For the search area, when the positional relationship between the images is known to some extent, that is, when the image is captured so that the positional relationship is constant, or when the rough positional relationship is externally given after the image capturing. Based on the information, the search area is limited and read out. If the positional relationship is undecided, the entire area of the image is read out as the search area.

Pattern matching is performed on the basis of the reference area and the search area of each read image, and the image signal of the reference area has the highest matching value in the search area, that is, the correlation value with the reference area. Position (mx0, my0)
Ask for. The correlation value is calculated by the following equation, for example.

[0023]

(Equation 1)

Here, R _area is a reference area, and R (x, y)
Is the luminance value at the coordinates (x, y) of the reference area, and S (x,
y) is the luminance value at the coordinates (x, y) of the search area, (of
fset _x , offset _y ) is an offset from the origin (0, 0) of the search image. Matching position obtained here (mx0, m
(y0) is the position of each of the reference region and the search region cut out from the original image in the original image as shown in FIG. 9 (c). _move , y _move ).

[0025]

(Equation 2)

As described above, the shift amount (parallel movement amount) between images can be calculated.

If necessary, FIGS. 9D and 9
By providing two or more reference areas as shown in (e), a change in the scale of the image based on a change in the distance between the reference areas and the distance between the matching areas, and a relative difference in the direction formed between the reference area and the matching area. The rotation angle is also required. For example, when a vector Vr passing through the centers of a plurality of reference regions and a vector Vm passing through the centers of the matching regions are Vm, a change in scale and a rotation angle are obtained by the following equations.

[0028]

(Equation 3)

In this case, based on the scale S (k) and the rotation angle θ (k) with respect to the image I (k), the image I (k + 1) rotated by 1 / Sk times and −θ (k) is re-framed. At the address of the original image I (k + 1). Further, the obtained parallel movement amount is written in a predetermined position of the RAM 112 corresponding to the counter k. After performing the above processing, the counter k is incremented by 1, and the counter k is compared with the number of images. If the counter k is smaller than the number of images, calculation processing of the next image position is performed. If the counter k is equal to or larger than the number of images, the image position calculation unit 106
An end signal for notifying the end of the image position calculation process is transmitted to the timing control unit 111, and the process ends.

Upon receiving the end signal from the image position calculation unit 106, the timing control unit 111 transmits a start signal to the image parallax extraction unit 107, and the image parallax extraction processing unit 107
It enters a wait state until it receives an end signal from.

Upon receiving the start signal from the timing control unit 111, the image parallax extraction processing unit 107 reads out and stores the number of images to be processed from the RAM 112, and sets the counter k to 1. Next, the address in the frame memory 102 of the image corresponding to the counter k is read from the RAM 112, and the image in the frame memory 102 is read from the address. This is called an image I (k). Further, an image adjacent to the image (corresponding to the counter k + 1) is read. This image is called an image I (k + 1). After that, the parallel movement amount obtained by the image position calculation unit 106 is stored in the RAM 112.
And the area where the image I (k) and the image I (k + 1) overlap is defined as an image I (k) and an image I (k + 1).
Ask for each.

Then, the overlap portion of the image I (k) is divided, each of which is used as a reference region, and the image I (k +
A corresponding position is searched from 1) by using a pattern matching technique. At this time, since the search is performed based on a shift between images due to parallax, the search may be performed only in the horizontal direction. This shift amount is an apparent motion amount of the object due to parallax, and is caused by the depth. Therefore, the depth of the object in the reference area can be estimated from the shift direction and the shift amount (see FIG. 10). reference).

Also, when the points at a certain depth distance are matched, even within the overlap of the same width, due to parallax,
An object closer to the camera than a certain point has a larger shift amount to the left, and a farther object from the camera has a larger shift amount to the right. Here, the point at a certain distance is a distance corresponding to the parallel movement amount obtained by the image position calculation unit 106, but a correction may be made based on the deviation amount within the overlap obtained here. . For example, if matching at the deepest place in the image, the reference value is set to the minimum value of the shift amount,
In addition, if the adjustment is performed at a place where the depth is the smallest in the image, the reference value may be set to the maximum value of the shift amount, and the average of the shift amounts may be set to the reference value. RAM 11 for the determined reference value
2 is added, and each shift amount in the overlap area just obtained is corrected by taking a difference from a reference value. By doing so, the deviation amount as the reference value can be corrected to zero.

In this process, an accurate depth is not obtained, and the parallax due to the difference in depth, that is, the shift amount between pixels itself is stored in the RAM 112 for each region or for each pixel.

FIG. 6 is a diagram showing a state of matching for performing parallax extraction.

The cells in the image (A) correspond to the divided reference areas, and the circles in the area indicate the center of the area. The frame in the image (B) indicates the overlap area, and the circle corresponds to the circle indicating the center of the reference area of the image (A), and the arrow extending therefrom indicates the direction of the shift due to the matching and the shift amount. Is shown. Here, as shown in FIG. 6, if the image on the right is taken at a position to the right of the position at which the image on the left was taken as shown in FIG. It can be said that the greater the deviation amount, the shallower the depth. The image (C) shows this shift amount as the luminance value of the pixel, taking the left direction as positive. A darker portion indicates a smaller parallax, that is, a deeper depth, and a brighter portion indicates a larger parallax, that is, a shallower depth. Further, by further reducing the reference area, finer parallax extraction can be performed (FIG. 8C).
(See (d)).

Similarly, the image I (k + 1) is referred to as a reference side,
Depth extraction is performed using the image I (k) as a search side. The parallax extraction result of the image I (k + 1) is also stored in the RAM 112 in the same manner.

When the depth extraction processing of the images I (k) and I (k + 1) is completed, the counter k is incremented by 1, and the counter k is compared with the number of images. If the counter k is smaller than the number of images, a depth extraction process is performed on the images I (k) and I (k + 1). If the counter k is equal to or greater than the number of images, the image parallax extraction processing unit 107 transmits an end signal notifying the end of the image parallax extraction processing to the timing control unit 111, and ends the processing.

Upon receiving the end signal from the image parallax extraction processing unit 107, the timing control unit 111 transmits a start signal to the intermediate image generation unit 108,
It enters a wait state until it receives an end signal from.

Upon receiving the start signal from the timing control unit 111, the intermediate image generation processing unit 108 reads out and stores the number of images to be processed from the RAM 112, and sets the counter k to 1. Next, the address of the image corresponding to the counter k in the frame memory 102 is read from the RAM 112, and the image I in the frame memory 102 is read from the address.
Read (k). Further, an image I (k + 1) adjacent to the image (corresponding to the counter k + 1) is read.
Thereafter, the parallel movement amount obtained by the image position calculation unit 106 is represented by R
The overlap area of the image I (k) and the overlap area of the image I (k + 1) are obtained by reading from the AM 112.

An intermediate image is an image created by calculating an image captured at a position intermediate between two viewpoints from two images captured from different viewpoints. The position where the image I (k) was captured is _defined as an imaging position _Tk . Here, the image in the overlap is divided into n-1 in the horizontal direction in the area, and the image pickup position _Tk and the image pickup position _{Tk + 1} are divided into 1: n-1 from the left, and 2: , n-1,..., n−1: an intermediate image II _k (l) captured from an imaging position internally divided into 1: 1, l = 1, 2,.
n-1 is created by the following method.

The imaging position T _k and the imaging position T _{k + 1} are represented by 1: n−1
2 shows a method of creating an intermediate image in generating an intermediate image at a position internally divided. Image I obtained by parallax extraction processing section 107
The amount of shift between (k) and the image I (k + 1) is read from the RAM 112 for each pixel or area.

As shown in FIGS. 7 (A) and 7 (B), an arbitrary region in the overlap region R1 (rx0, ry0)-(rx1, ry1) of the image I (k) with the image I (k + 1) is obtained. Let P1 be the point, the coordinates of point P1 be (x1, y1), and the brightness value at point P1 be D1.
(P1), the shift amount at the point P1 is represented by M (P1) (FIG. 7 (A)
), An arbitrary point in the overlap area R2 (rx2, ry2)-(rx3, ry3) of the image I (k + 1) with the image I (k) is P2, and the coordinates of the point P2 are (x2, y2), point P
2 is D (P2), and the shift amount at point P2 is M (P
2) (arrow in FIG. 7B).

First, an intermediate image I based on the image I (k)
I _k (l) FIG. 7C is created. Each point P1 (x, y) in the overlap region R1, rx0 ≦ x ≦ rx1, ry0 ≦ y
≦ respect ry1, points on the intermediate image _{II k (l) (x +} M (P
1) Let the luminance value of * l / n-rx0, y-ry0) be D (P1).

An intermediate image II _k (l) using only the image I (k)
Is created, the luminance value of a plurality of pixels is written in a certain portion of the intermediate image due to the difference in the shift amount due to the difference in the parallax of each pixel, or as shown in the black pixel portion of FIG. Undefined pixels may occur. If a luminance value has already been written during generation of the intermediate image, priority is given to the luminance value of a pixel having a large shift amount M (P1). This is because the larger the displacement amount M (P1) is, the closer the object is to the camera, and it is unnatural to receive the luminance value of an object or the background behind the camera.

For the undefined pixels, the intermediate image II created from the image I (k + 1) as follows:
Interpolate it with _{k + 1} (l).

Similarly to the creation of the intermediate image II _k (l), the intermediate image II _k ′ (l) based on the image I (k + 1)
(D) is created.

Each point P2 in the overlap region R2
(X, y), with respect to rx2 ≦ x ≦ rx3, ry2 ≦ y ≦ ry3, the intermediate image II _k '(l) on the point (x + M (P2) * (n-
l) / n-rx0, y-ry0) is defined as D (P2). However, as described above, when a luminance value has already been written, priority is given to a luminance value of a pixel having a large displacement amount M (P1). Thereafter, the image I (k + 1) and II _k ′ (l) created from the shift amount for each pixel are used to generate the image I (k + 1).
II _k (l) created from (k) and the shift amount for each pixel
Are interpolated by the corresponding pixel of II _k ′ (l).

Finally, when there is a portion where no luminance value is written in the intermediate image, interpolation is performed using the surrounding written luminance value. FIG. 14 shows the state of interpolation. Each square corresponds to one pixel, and those with a circle at the center are undefined pixels.
Now, the luminance value of the undefined pixel surrounded by the bold line is changed to the already defined pixel in 8 neighborhoods surrounded by the bold broken line, as shown in FIG.
Inside, the average of the pixels with a triangular mark in the center. The above operation is repeated for all the undefined pixels. If 8
If all the neighboring pixels are undefined, the interpolation operation is performed again after the interpolation of the other undefined pixels is completed, and the process is repeated until there are no more undefined pixels.

Further, if interpolation is performed from undefined pixels having a large number of already defined pixels in the vicinity of 8, an interpolation image with higher quality can be obtained. As described above, the intermediate image II _k (l) is generated, but the area used for synthesis is (dx * (l−1),
0)-(dx * 1, (ry1-ry0)). Therefore, only that part is written into the frame memory 102 and the image I
The generation of an intermediate image of l: n−1 in the overlap region between (k) and the image I (k + 1) is completed.

After the above-described intermediate image creation processing up to l = 1,..., 9, the number of intermediate images created in the RAM 112 is written into a predetermined address, and the counter k is incremented by one.
The counter k is compared with the number of images. If the counter k is smaller than the number of images, the intermediate image processing is performed on the image corresponding to the updated counter. If the counter k is equal to the number of images, the intermediate image generation processing unit 108 transmits an end signal notifying the end of the intermediate image generation processing to the timing control unit 111, and ends the processing.

Upon receiving the end signal from the intermediate image generation processing unit 109, the timing control unit 111 transmits a start signal to the image synthesis processing unit 109,
It enters a wait state until it receives an end signal from.

Upon receiving the start signal from the timing control section 111, the image synthesis processing section 109 reads out and stores the number of images to be processed from the RAM 112, and sets the counter k to one. Next, the address of the image corresponding to the counter k in the frame memory 102 is read from the RAM 112, and the image I in the frame memory 102 is read from the address.
Read (k).

When the counter k is 1, the image I (k)
At a predetermined position in the output image area of the frame memory 102.

When the counter k is larger than 1, the deviation M (k) from the image I (k-1) is called from the RAM 112. Overlap region R _k (r with image I (k-1)
x0, ry0)-(rx1, ry1) is calculated. Further, the image I (k−1) and the image I
The number n of intermediate images for (k) is called from the RAM 112. Intermediate image _{II k (i), i =} 1, ..., if the stuck position of n was P _i, the position P _i is

[0056]

(Equation 4)

The intermediate image II _k (i), i =
1, ..., copies the n in the composite image memory position P _i. Next, after the position (Σ ^k _{i = 1} M (i) + rx1, ry1), the image I
The part after the right of the x coordinate rx1 of (k) is copied (FIG. 8).
(E) and (f)).

When the above processing is completed, the counter k is set to 1
Then, the counter k is compared with the number of images. If the counter k is equal to or smaller than the number of images, the above-described processing is continuously performed on the updated image I (k). If the counter k is larger than the number of images, the image synthesis processing unit 109 transmits an end signal notifying the end of the image synthesis processing to the timing control unit 111, and ends the processing.

When receiving the end signal from the image synthesis processing unit 109, the timing control unit 111 transmits a start signal to the image output processing unit 110 and enters a wait state until receiving the end signal from the image output processing unit 110. . Upon receiving the start signal from the timing control unit 111, the image output processing unit 110 reads an output image address in the frame memory 102 from the RAM 112. Next, the image output device 1
A start signal for starting image output is sent to the image memory device 03, and the output image signal is sequentially read from the frame memory 102 and output to the image output device 103. When the output image has been output from the image output device 103, the image output processing unit 110
An end signal for notifying the end of the image output process is transmitted to the timing control unit 111, and the process ends.

When the timing control unit 111 receives the end signal from the image output processing unit 110, the operation ends.

According to the flow of the above series of processes, the difference in parallax between subjects in the overlap between two images is reduced by creating an intermediate image. As in the past, it can be shifted or doubled as in the past,
Disappears. Naturally, a more natural panoramic image can be obtained by increasing the number of intermediate images created, that is, the number of divisions within the overlap. Further, in the embodiment according to claim 1 of the present invention, since subjects having different amounts of displacement due to parallax are combined with an overlap having the same width, objects closer to the camera are combined with a smaller width, and objects farther away are combined with a wider width. Is done. If you want the part close to the camera to be natural,
In the image parallax extraction processing unit 107, the entire shift amount may be corrected using the shift amount of an object close to the camera as a reference value. When it is desired to make a portion far from the camera natural, the image parallax extraction processing unit 107 may correct the entire shift amount using the shift amount of an object far from the camera as a reference value.

The embodiment described above is an example of the embodiment according to claim 1 of the present invention, and is not limited to the contents of the present embodiment unless the gist of the present invention is changed.

FIG. 2 is a block diagram showing one embodiment of the second embodiment of the present invention.

Reference numeral 101 denotes an image input device, 102 denotes a frame memory, 103 denotes an image output device, 104 denotes an image input processing unit, 105 denotes an image preprocessing unit, 106 denotes an image position calculation unit,
107 'is an image parallax extraction processing unit, 108' is a joint line determination processing unit, 109 'is an image synthesis processing unit, 110 is an image output processing unit, 111 is a timing control unit, and 112 is a RAM.

FIG. 11 is a diagram showing the procedure of the process according to the second embodiment of the present invention.

Hereinafter, the contents of the second embodiment of the present invention will be described with reference to FIG.

Image input processing section 104, image preprocessing section 10
5. The functions of the image position calculation unit 106 and the processing up to the image position calculation unit 106 are the same as those in the embodiment according to claim 1 of the present invention, and thus the details are omitted.

When receiving the end signal from the image position calculation unit 106, the timing control unit 111 transmits a start signal to the image parallax extraction processing unit 107 'until receiving the end signal from the image parallax extraction processing unit 107'. Enter the waiting state.

Upon receiving the start signal from the timing control section 111, the image parallax extraction processing section 107 '
And reads and stores the number of images to be processed from
Is set to 1. Next, the address of the image corresponding to the counter k in the frame memory 102 is read from the RAM 112, and the image I (k) in the frame memory 102 is read from the address. Further, an image I (k + 1) adjacent to the image (corresponding to the counter k + 1) is read. Thereafter, the parallel movement amount obtained by the image position calculation unit 106 is read from the RAM 112, and an overlap area of the image I (k) and an overlap area of the image I (k + 1) are obtained.

Then, the overlap portion of the image I (k) is divided, each of which is used as a reference area, and the image I (k +
A corresponding position is searched from 1) by using a pattern matching technique. At this time, since the search is performed based on a shift between images due to parallax, the search may be performed only in the horizontal direction. This shift amount is an apparent motion amount of the object due to parallax, and is caused by the depth. Therefore, the depth of the object in the reference area can be estimated from the shift direction and the shift amount. Also, when a point at a certain distance is matched, even within an overlap of the same width, due to parallax, an object closer to the camera than a certain point has a larger amount of shift to the left, and a point farther from the camera to the right than the certain point has a rightward shift. Shift amount becomes large.

Here, in the obtained area or pixel,
When the image with the largest depth is obtained, that is, when the left direction is defined as positive, the area having the smallest amount and the shift amount there are subtracted from each area or the distance between pixels, and the image position calculation unit 106 obtains the area. By adding this to the parallel movement amount, the shift of the image is set as the reference shift with the deepest part inside the image. Accordingly, the amount of parallel movement in the RAM 112 and the amount of displacement in the overlap area are also corrected. Similarly, parallax extraction is performed using the image I (k + 1) as a reference side and the image I (k) as a search side. The parallax extraction result of the image I (k + 1) is also stored in the RAM 112 in the same manner.

After the parallax extraction processing of the images I (k) and I (k + 1) is completed, the counter k is incremented by 1, and the counter k is compared with the number of images. If the counter k is smaller than the number of images, parallax extraction processing is performed on the images I (k) and I (k + 1). If the counter k is equal to or larger than the number of images, the image parallax extraction processing unit 107 'transmits an end signal notifying the end of the image parallax extraction processing to the timing control unit 111, and ends the processing.

When receiving the end signal from the image parallax extraction processing unit 107 ', the timing control unit 111 transmits a start signal to the joint line determination processing unit 108', and receives the end signal from the joint line determination processing unit 108 '. Enter waiting state until received.

Upon receiving the start signal from the timing control unit 111, the joining line determination processing unit 108 'reads out and stores the number of images to be processed from the RAM 112, and sets the counter k to 1. Next, the address of the image corresponding to the counter k in the frame memory 102 is read from the RAM 112, and the image I in the frame memory 102 is read from the address.
Read (k). Further, an image I (k + 1) adjacent to the image (corresponding to the counter k + 1) is read.
Then, the image I obtained by the image parallax extraction processing unit 107 '
(K) The parallax amount of the image I (k + 1) is read from the RAM 112.

Here, the image I in the second, third and fourth aspects will be described.
The method for obtaining the joint line from (k) is shown in FIGS.
13 will be described. The image I (k) (FIG. 11A) and the image I (k +
1) From the parallax image (FIG. 11 (c)) for the image I (k) obtained from (FIG. 11 (b)), a point at which the parallax of the image changes so as to be larger when viewed from the right end of the image, that is, Extract satisfying points.

[0076]

(Equation 5)

Here, E (x, y) is the coordinates (x, y) of the image.
The amount of parallax in y), Threshold, is an appropriate threshold. This corresponds to a white circle in the image in FIG.
These dot sequences show the right outline of the subject and the image I
On the left side of the corresponding position on (k + 1), the image I
Invisible parts of (k) are visible, and the images I (k) and I (k) near these points, especially on the left side,
+1) is large. Also, near the right of these points, the same thing is visible in both images,
The difference between the images I (k) and I (k + 1) is small.

In the joint line extraction method according to the second aspect, in consideration of the above, a point having the same parallax as much as possible is extracted from the obtained point sequence having a large change in parallax. Is a method of extracting the contour line of, and using it as a joining line. In addition, when there is no portion where the parallax change is large above and below as shown in FIG. 11E, a line drawn from the edge of the closest contour line to the upper end or the lower end of the image is set as the joining line.

According to a third aspect of the present invention, there is provided a method for extracting a joint line, comprising: a point sequence having a large parallax change, which is obtained in the same manner, and which is closer to the end of the image on the side of the overlap region exceeding a certain threshold Threshold in the overlap region; That is, in this case, the line connecting the rightmost point sequence is set as the joining line L (k) (see FIG. 12 (i)).

A method for extracting a joint line according to claim 4 is as follows. In the method for extracting a joint line according to the second and third aspects, a point sequence having a large parallax change from left to right is obtained based on the image I (k + 1).
It can be adapted as well. That is, the image I is
(K) By obtaining a joint line from the image I (k + 1), a plurality of joint line candidates are obtained. Threshold
By changing the value of, a plurality of joining line candidates can be obtained. Let them be L _i (k), i = 1, 2,.
3 (I), (II)).

The most optimal connection line candidate L _i (k), i = 1, 2,... Is a connection line having the following values which are the smallest.

[0082]

(Equation 6)

Here, C is a path on the joining line L _i (k), and L _i (k) (x, y) is a point (x, y) on the joining line L _i (k) in the image I (k). y) (or R
Vector of each value of GB), L _i ′ (k)
(X ', y') joint line in the image I (k + 1) L _i
It is a luminance value (or a vector of each value of RGB) on a point (x ′, y ′) corresponding to the point (x, y) in the image I (k) on (k). A joining line candidate L _i (k) that minimizes the above equation is obtained and is set as a joining line (FIG. 13).
(III)). The joining line L obtained as described above
(K) is sequentially written to a predetermined address of the RAM 112, and the processing is terminated.

When the joining line determination processing of the images I (k) and I (k + 1) is completed, the counter k is incremented by 1, and the counter k is compared with the number of images. If the counter k is smaller than the number of images, a joining line determination process is performed on the images I (k) and I (k + 1). If the counter k is equal to or greater than the number of images, the joining line determination processing unit 108 'transmits an end signal notifying the end of the joining line determination process to the timing control unit 111, and ends the process.

When the timing control unit 111 receives the end signal from the joining line determination processing unit 108 ′, it transmits a start signal to the image synthesis processing unit 109 ′, and
It enters a wait state until it receives an end signal from 9 '.

Upon receiving the start signal from the timing control section 111, the image synthesis processing section 109 'reads out and stores the number of images to be processed from the RAM 112, and sets the counter k to 1
Set to. Next, the address of the image corresponding to the counter k in the frame memory 102 is read from the RAM 112, and the image I in the frame memory 102 is read from the address.
Read (k). If the counter k is 1, the image I
(K) is written in a predetermined position of the output image area of the frame memory 102.

When the counter k is larger than 1, the parallel movement amount M (k) with the image I (k-1) and the joint line L (k-
1) is called from the RAM 112.

The position where the image I (k) is placed is

[0089]

(Equation 7)

Is obtained. Also, the joining line is corrected to a position shifted by P _k−1 .

When copying the image I (k) to the combining memory, the image is combined under the following conditions.

Pixels to the left of the joining line L _k-1 are not written. The right w dots of the joining line L _k−1 are smoothed with the image I (k) by density smoothing. -Pixels are written as they are after the right w dot of the joining line _Lk-1 .

Here, w is an appropriate constant value. This w
By performing density smoothing with dots, joining becomes more natural. In FIG. 11, a mask is applied as shown in FIG. 11 (h), a region for performing density smoothing is provided, and a combined image as shown in FIG. 11 (i) is generated by combining with FIG. 11 (g). become.

When the above processing is completed, the counter k is set to 1
Then, the counter k is compared with the number of images. If the counter k is equal to or smaller than the number of images, the above-described processing is continuously performed on the updated image I (k). If the counter k is larger than the number of images, the image synthesis processing unit 109 'transmits an end signal notifying the end of the image synthesis processing to the timing control unit 111, and ends the processing.

When the timing control section 111 receives the end signal from the image synthesis processing section 109 ', it transmits a start signal to the image output processing section 110 and waits until receiving the end signal from the image output processing section 110. enter.

Upon receiving the start signal from the timing control unit 111, the image output processing unit 110 reads the output image address in the frame memory 102 from the RAM 112. Next, a start signal for starting image output is sent to the image output device 103, and the output image signal is stored in the frame memory 1
02 and sequentially output to the image output device 103. When the output image has been output from the image output device 103, the image output processing unit 110
Then, an end signal for notifying the end of the image output processing is transmitted to the processing, and the processing is ended.

The timing control unit 111 receives the end signal from the image output processing unit 110, and ends all operations.

According to the flow of the series of processes described above, the difference in the parallax in the overlap area between the two images is larger for a subject closer to the camera, and if the composition is performed there, it causes a shift or a double. However, by providing a joining line on the contour of an object close to the camera, joining at a portion where the parallax is greatly different is reduced, and the joined image becomes more natural than before and the quality is improved.

In the embodiments according to claims 2, 3, and 4 of the present invention, even if an image includes a moving object in the image, the image appears as a parallax in parallax extraction. Can be joined. The embodiment described above is an example of the embodiment according to claims 2, 3 and 4 of the present invention, and is not limited to the content of the embodiment unless the gist of the present invention is changed.

Further, in the panoramic image creating apparatus of the present invention, a program for realizing a process of creating a series of panoramic images includes a medium such as a floppy disk, a hard disk, a magnetic tape, a CD-ROM / optical disk / magneto-optical disk, and an MD. And the following contents stored in a recording medium such as a ROM / RAM memory.

As an example of the stored program contents, a divided image obtained by photographing with a plurality of image pickup means or by moving the image pickup means a plurality of times and overlapping a part of the picked-up images, The image input processing means instructs the imaging means to perform the capture, and instructs the storage means to specify the storage destination address to store the captured divided image in the storage means. Calculating the combined position of the divided images, causing the disparity information extracting means to extract the disparity information between the images from the image including the disparity in the overlap area between the adjacent divided images, and allowing the intermediate image creating means to A plurality of intermediate images are created from an image including parallax in the overlap area, and a panoramic image is synthesized from the divided images by an image synthesizing unit. Together to correct the deviation of the image of the overlap region in the stage, it is intended to execute the operation for synthesizing a panoramic image by interpolating an intermediate image created in the intermediate image creating unit.

Another example of the stored program contents is obtained by photographing with a plurality of image pickup means or moving the image pickup means a plurality of times and overlapping a part of the picked-up images. The image input processing means instructs the imaging means to capture the divided images,
The storage unit instructs a storage destination address to instruct the storage destination address and stores the captured divided image in the storage unit, the image position calculation unit calculates the combined position of the divided image, and the disparity information extraction unit Parallax information between the images is extracted from the image including the parallax in the overlap region between the divided images, and the joining line determination unit determines the overlap region of the overlap region based on the depth information obtained by the parallax information extraction unit. A joining line is set on the contour of the subject closest to the image capturing means, and a panoramic image is synthesized from the divided images by the image synthesizing means. At this time, the image of the overlap area is shifted by the parallax information extracting means. And performing an operation of synthesizing a panoramic image by joining the divided images based on the joining line determined by the joining line determining means. It is intended to.

The above contents are only examples of the embodiments according to claims 5 and 6 of the present invention, and are not limited to the above contents unless the gist of the present invention is changed.

As described above, the program according to the present invention can be used in various information devices such as a personal computer, a portable information terminal, and a camera-integrated image processing device by using the recording medium.

[0105]

As described above, the panoramic image creating apparatus of the present invention has the following effects in each claim.

According to the first and fifth aspects of the present invention, the parallax due to the depth of the subject is calculated by calculation within the overlap between the images, and an intermediate position for interpolating the position where the image is taken based on the calculated parallax. By interpolating the images in the overlap using the images, the effect of generating a more natural panoramic image than before can be obtained even when there is parallax due to the depth of the subject.

According to the second and sixth aspects of the present invention, the parallax caused by the depth of the subject and the moving object are obtained by calculation within the overlap between the images, and the object between the two images is obtained based on the calculation. By using a line of an image that does not include a place where the depth difference is large as a joining line,
Even when there is a parallax due to the depth of the subject or when a moving object is included, the effect of generating a natural panoramic image with higher quality and less distortion than before can be obtained.

According to a third aspect of the present invention, even when a nearby subject has a complicated shape, the contour of a set of relatively close subjects can be obtained even when a joint line cannot be determined well. By providing the joining line in the above, an effect that a natural panoramic image with good quality and little distortion can be generated can be obtained.

According to the fourth aspect of the present invention, depending on the position of the subject in the overlap, it may not be successful if the reference image is always fixed in one direction in the second and third aspects of the present invention. Yes, in this case, a plurality of joining line candidates created by combining the selection of a reference image and the joining line determination method according to the second and third aspects of the present invention are selected, and a joining line is selected from them. By selecting the optimal joint line that minimizes the difference between the images on the line, an effect is obtained in which a natural panoramic image with higher quality and less distortion can be generated more stably.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating an example of an embodiment according to claim 1 of the present invention.

FIG. 2 is a block diagram illustrating an example of an embodiment according to claim 2 of the present invention.

FIG. 3 is a RAM according to the first embodiment of the present invention;
FIG. 4 is an explanatory diagram showing an example of the configuration of FIG.

FIG. 4 is an explanatory diagram showing a state of moving a camera or capturing an image having an overlap area by a plurality of cameras.

5 is an explanatory diagram showing that the images captured in FIG. 4 do not match in the overlap due to a difference in depth. FIG.

FIG. 6 is an explanatory diagram showing a state of matching for performing parallax extraction.

FIG. 7 is an explanatory diagram showing a state of creating an intermediate image.

FIG. 8 is an explanatory diagram showing a procedure of a process according to the first embodiment of the present invention.

FIG. 9 is an explanatory diagram showing a state in which an image position is obtained in claims 1 and 2 of the present invention.

FIG. 10 is an explanatory diagram showing a relationship between a matching shift amount between images and a depth from a camera.

FIG. 11 is an explanatory diagram showing a procedure of a process according to an embodiment of the present invention.

FIG. 12 is an explanatory diagram showing a state of determining a joining line according to claim 3 of the present invention.

FIG. 13 is an explanatory diagram showing how a joint line is determined according to claim 4 of the present invention.

FIG. 14 is an explanatory diagram showing a state of an interpolation method from around undefined pixels.

[Explanation of symbols]

 Reference Signs List 101 Image input device 102 Frame memory 103 Image output device 104 Image input processing unit 105 Image preprocessing unit 106 Image position calculation unit 107 Image parallax extraction processing unit 108 Intermediate image generation processing unit 109 Image synthesis processing unit 110 Image output processing unit 111 Timing Control unit 112 RAM 107 'Image parallax extraction processing unit 108' Join line determination processing unit 109 'Image synthesis processing unit

Claims (6)

[Claims] 1. A panoramic image is created by a plurality of image pickup means or by moving an image pickup means a plurality of times, taking a part of the picked-up images while overlapping them, and joining the obtained divided images together. A panoramic image creating apparatus, wherein the imaging unit; a storage unit configured to store the divided image input from the imaging unit; and an instruction to capture the divided image to the imaging unit. Image input processing means for designating a storage destination address of the captured divided image; image position calculating means for calculating a combined position of the divided images; and A disparity information extracting unit configured to extract disparity information between the images from the image including the disparity; and generating a plurality of intermediate images from the images including the disparity in the overlap region. Intermediate image creating means, and an image synthesizing means for synthesizing a panoramic image from the divided images, wherein the displacement of the image in the overlap area is corrected by the parallax information extracting means and created by the intermediate image creating means. A panoramic image creating apparatus, which synthesizes a panoramic image by interpolating with an intermediate image. 2. A panoramic image is created by a plurality of image pickup means or by moving the image pickup means a plurality of times so that a part of the picked-up images are photographed with overlapping, and the obtained divided images are joined together. A panoramic image creating apparatus, wherein the imaging unit; a storage unit configured to store the divided image input from the imaging unit; and an instruction to capture the divided image to the imaging unit. Image input processing means for designating a storage destination address of the captured divided image; image position calculating means for calculating a combined position of the divided images; and A disparity information extracting unit that extracts disparity information between the images from the image including the disparity, based on the depth information obtained by the disparity information extracting unit, A joining line determining unit that sets a joining line on the contour of the subject closest to the image capturing unit of the image of the overlap area; and an image combining unit that combines a panoramic image from the divided images. Is corrected by the parallax information extracting means, and the divided images are joined based on the joining lines obtained by the joining line determining means. 3. The panoramic image creating apparatus according to claim 2, wherein the joining line determining unit joins a contour closest to an image edge on the overlapping region side of a subject closest to the image capturing unit of the image of the overlapping region. A panoramic image creation device, wherein a line is set. 4. The panoramic image creating apparatus according to claim 2, wherein the joining line determining means selects a reference image at the time of determining a joining line and determines the joining line in the overlap area. A panoramic image creating apparatus, wherein an optimum joining line is selected from a plurality of joining lines by a combination of methods, and the divided images are joined. 5. A panorama image is created by a plurality of image pickup means or by moving the image pickup means a plurality of times to shoot a part of the picked-up images while overlapping them, and joining the obtained divided images together. The image input processing means instructs the imaging means to capture the divided image, and instructs the storage means to specify the storage destination address and stores the captured divided image in the storage means. Causing the image position calculating means to calculate the combined position of the divided images; and causing the disparity information extracting means to extract the disparity information between the images from the image including the disparity in the overlap region between the adjacent divided images. The intermediate image creating means creates a plurality of intermediate images from the image including the parallax in the overlap area, and the image synthesizing means combines the panoramic images from the divided images. At this time, a program for causing the parallax information extracting means to correct a shift of the image in the overlap area and to execute an operation of synthesizing a panoramic image by interpolating with the intermediate image created by the intermediate image creating means is stored. Recording medium. 6. A panoramic image is created by a plurality of image pickup means or by moving the image pickup means a plurality of times to shoot a part of the picked-up images while overlapping, and joining the obtained divided images together. The image input processing means instructs the imaging means to capture the divided image, and instructs the storage means to specify the storage destination address and stores the captured divided image in the storage means. Causing the image position calculating means to calculate the combined position of the divided images; and causing the disparity information extracting means to extract the disparity information between the images from the image including the disparity in the overlap region between the adjacent divided images. Based on the depth information obtained by the parallax information extracting means, the joining line determining means on the contour of the subject closest to the imaging means of the image of the overlap area. A merging line is set, and a panoramic image is synthesized from the divided images by the image synthesizing unit. At this time, the parallax information extracting unit corrects the image shift in the overlap area and obtains the image by the joining line determining unit. A recording medium storing a program for executing an operation of synthesizing a panoramic image by joining divided images based on the obtained joining line.