KR102776445B1 - Device and method for obtaining depth of space using camera - Google Patents

Abstract

A device and method for acquiring depth of a space using a camera are provided. The method for acquiring the depth of a feature region by a device using a camera module includes: acquiring a plurality of images by sequentially rotating the camera module at a preset angle and photographing the periphery of the camera multiple times; identifying a first feature region in the first image and an n-th feature region identical to the first feature region in the n-th image by comparing adjacent images between a first image and an n-th image among the plurality of images; acquiring a base line value for the first image and the n-th image; acquiring a disparity value between the first feature region and the n-th feature region; and calculating the depth of the first feature region or the n-th feature region based on the base line value and the disparity value.

Description

{DEVICE AND METHOD FOR OBTAINING DEPTH OF SPACE USING CAMERA}

The present disclosure relates to a device and method for obtaining depth of space using a camera, and more particularly, to a device and method for calculating depth values of a surrounding space using a rotating camera.

Augmented Reality is a technology that projects virtual images onto the physical environment or real world objects of the real world and displays them as a single image.

Augmented reality devices allow users to view real scenes and virtual images together through a see-through display module that is placed in front of the user's eyes while worn on the user's face or head. As research on such augmented reality devices is actively progressing, various types of wearable devices are being released or are scheduled to be released. For example, wearable glasses and head-mounted displays are wearable display devices that are currently being released or are scheduled to be released.

These augmented reality devices need to measure the depth of real objects in order to recognize user gestures or display virtual objects in natural linkage with real objects. However, existing augmented reality devices measure the depth of the surrounding space using multiple cameras or TOF (Time of Flight) cameras in order to recognize objects in the surrounding space, which increases the weight or number of cameras. In addition, this causes problems such as increased volume, weight, and battery consumption of the augmented reality device.

Accordingly, it is important to make the augmented reality device lightweight and miniaturized so that the user does not feel uncomfortable even when the augmented reality device is worn for a long time or frequently.

One embodiment of the present disclosure can provide a device and method capable of effectively calculating the depth of a specific area while reducing the weight of the device by comparing images obtained by sequentially rotating a camera module at a preset angle and taking multiple shots of the surroundings of the camera.

In addition, one embodiment of the present disclosure can provide a device and method capable of efficiently identifying the same specific region within a plurality of sequentially captured images by comparing adjacent images, and calculating depth values of the specific regions based on the locations of the identified specific regions.

In addition, one embodiment of the present disclosure can provide a device and method capable of setting a shooting angle range through a camera module according to a shooting mode.

As a technical means for achieving the above-described technical task, a first aspect of the present disclosure comprises the steps of: acquiring a plurality of images by sequentially rotating a camera module at a preset angle and shooting a periphery of the camera multiple times; identifying a first feature region in the first image and an n-th feature region identical to the first feature region in the n-th image by comparing adjacent images between a first image and an n-th image among the plurality of images; acquiring a base line value for the first image and the n-th image based on the arrangement of the camera module when shooting the first image and the arrangement of the camera module when shooting the n-th image; acquiring a disparity value between the first feature region and the n-th feature region based on the position of the first feature region in the first image and the position of the n-th feature region in the n-th image; And a step of calculating the depth of the first feature region or the nth feature region based on the baseline value and the disparity value; The device can provide a method for obtaining the depth of a feature region using a camera module.

In addition, a second aspect of the present disclosure comprises: a camera module; a shooting direction control unit for rotating the camera module at a predetermined angle; a display; a memory for storing one or more instructions; And a processor for executing the one or more instructions; wherein the processor controls the shooting direction control unit and the camera module to sequentially rotate the camera module at a preset angle and capture a plurality of images by capturing a plurality of images around the camera, and compares adjacent images between a first image and an n-th image among the plurality of images to identify a first feature region in the first image and an n-th feature region that is identical to the first feature region in the n-th image, and obtains a base line value for the first image and the n-th image based on the arrangement of the camera module when capturing the first image and the arrangement of the camera module when capturing the n-th image, and obtains a disparity value between the first feature region and the n-th feature region based on the position of the first feature region in the first image and the position of the n-th feature region in the n-th image, and calculates a depth of the first feature region or the n-th feature region based on the base line value and the disparity value. A device for acquiring depth of a feature region using a camera module can be provided.

In addition, a third aspect of the present disclosure can provide a computer-readable recording medium having recorded thereon a program for executing the method of the first aspect on a computer.

FIG. 1 is a schematic diagram of a method for a device to obtain depth of a surrounding area according to one embodiment of the present disclosure.
FIG. 2 is a block diagram of a device according to one embodiment of the present disclosure.
FIG. 3 is a drawing showing an example of a camera module rotating within a device according to one embodiment of the present disclosure.
FIG. 4A is a diagram illustrating an example of obtaining multiple images while a camera module within a device rotates according to one embodiment of the present disclosure.
FIG. 4b is a diagram illustrating an example in which identical feature regions in adjacent images are sequentially identified according to one embodiment of the present disclosure.
FIG. 4c is a diagram illustrating an example of a device according to one embodiment of the present disclosure identifying identical identification regions within images with the greatest angular difference.
FIG. 5A is a diagram illustrating an example of a device photographing its surroundings according to one embodiment of the present disclosure.
FIG. 5b is a diagram illustrating an example of a device virtually arranging a camera module according to one embodiment of the present disclosure.
FIG. 5c is a diagram illustrating an example of a device calculating a disparity value according to one embodiment of the present disclosure.
FIG. 6 is a diagram showing an example of a device calculating the depth of a feature region according to one embodiment of the present disclosure.
FIG. 7 is a flowchart of a method for calculating the depth of a feature region in an image by a device according to one embodiment of the present disclosure.
FIG. 8 is a diagram illustrating an example of a device calculating a disparity value according to one embodiment of the present disclosure.
FIG. 9 is a flowchart of a method for a device to recognize its surroundings according to a shooting mode according to one embodiment of the present disclosure.
FIG. 10A is a diagram illustrating an example of a device capturing a user's hand movements in gesture mode according to one embodiment of the present disclosure.
FIG. 10b is a diagram illustrating an example of a device (1000) recognizing a gesture from image sets according to one embodiment of the present disclosure.
FIG. 11A is a diagram illustrating an example of a device grouping multiple images acquired by photographing the surroundings of the device into multiple image sets according to one embodiment of the present disclosure.
FIG. 11b is a diagram illustrating an example of multiple image sets grouped by a device according to one embodiment of the present disclosure.
FIG. 11C is a diagram illustrating an example of a device generating and analyzing a depth map representing space around the device from depth maps generated from multiple sets of images according to one embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily practice the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present disclosure in the drawings, parts that are not related to the description are omitted, and similar parts are assigned similar drawing reference numerals throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element in between. Also, when a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

In this disclosure, 'Augmented Reality (AR)' means displaying a virtual image together within a physical environment space of the real world or displaying a real object and a virtual image together.

In addition, the term 'Augmented Reality Device' refers to a device capable of expressing 'Augmented Reality', and generally includes not only the Augmented Reality Glasses, which are glasses-shaped devices worn by the user on the face, but also a Head Mounted Display Apparatus (HMD) worn on the head, or an Augmented Reality Helmet.

Meanwhile, a 'real scene' is a scene of the real world that a user sees through an augmented reality device, and may include a real world object. In addition, a 'virtual image' is an image generated through an optical engine, and may include both static and dynamic images. This virtual image is observed together with the real scene, and may be an image that shows information about a real object in the real scene, information about the operation of the augmented reality device, a control menu, etc.

Accordingly, a typical augmented reality device comprises an optical engine for generating a virtual image composed of light generated from a light source, and a wave guide formed of a transparent material to guide the virtual image generated from the optical engine to the user's eyes and also to allow the user to view scenes of the real world. As described above, the augmented reality device must be able to observe scenes of the real world as well, and therefore, in order to guide the light generated from the optical engine to the user's eyes through the wave guide, an optical element for basically changing the path of the light, which has a straight path, is required. At this time, the light path may be changed by using reflection by a mirror, etc., or the light path may be changed by diffraction by a diffractive optical element such as a DOE (Diffractive optical element) or a HOE (Holographic optical element), but is not limited thereto.

Additionally, in the present disclosure, the gesture recognition mode may be a shooting mode for shooting a gesture made by a user's body located at a close range in a shooting angle range smaller than a predetermined threshold, and the space recognition mode may be a shooting mode for shooting a space around the device (1000) in a shooting angle range larger than a predetermined threshold.

Additionally, in the present disclosure, the disparity value for two images may be a value indicating how much a feature region in one of the two captured images is shifted in the other image.

Additionally, in the present disclosure, the baseline value may be a value representing the distance between the shooting center of the camera module when shooting one image and the shooting center of the camera module when shooting another image. The shooting center of the camera module may be, for example, the center point of the lens of the camera module.

The present disclosure will be described in detail with reference to the attached drawings below.

FIG. 1 is a schematic diagram of a method for a device to obtain depth of a surrounding area according to one embodiment of the present disclosure.

Referring to FIG. 1, the device (1000) sequentially captures the surroundings of the device (1000) by rotating the camera module (1400) at a preset angle through the rotating camera module (1400), and obtains a plurality of images in which the surroundings of the device (1000) are captured. The device (1000) obtains a plurality of images including a first image to an n-th image, and identifies identical feature regions within the plurality of images by comparing adjacent images among the plurality of images. In addition, through this, the device (1000) can obtain a disparity value between identical feature regions within the first image and the n-th image, and calculate a depth value for the feature region.

The device (1000) may be an augmented reality device capable of expressing augmented reality. The device (1000) may include, for example, augmented reality glasses in the shape of glasses worn by a user on the face, a head mounted display apparatus (HMD: Head Mounted Display Apparatus) worn on the head, an augmented reality helmet, etc.

In addition, the device (1000) may be a smart phone, a tablet PC, a PC, a smart TV, a mobile phone, a PDA (personal digital assistant), a laptop, a media player, a micro server, a GPS (global positioning system) device, an e-book reader, a digital broadcasting terminal, a navigation device, a kiosk, an MP3 player, a digital camera, a home appliance, and other mobile or non-mobile computing devices. However, the present invention is not limited thereto, and the device (1000) may include any type of device capable of processing an image acquired by controlling a camera module.

FIG. 2 is a block diagram of a device according to one embodiment of the present disclosure.

Referring to FIG. 2, a device (1000) according to one embodiment of the present disclosure may include a user input unit (1100), a microphone (1200), a display unit (1300), a camera module (1400), a shooting direction control unit (1500), a communication interface (1600), a storage unit (1700), and a processor (1800).

The user input unit (1100) refers to a means for a user to input data for controlling the device (1000). For example, the user input unit (1100) may include, but is not limited to, at least one of a key pad, a dome switch, a touch pad (contact electrostatic capacitance type, pressure resistive film type, infrared detection type, surface ultrasonic conduction type, integral tension measurement type, piezo effect type, etc.), a jog wheel, or a jog switch.

The user input unit (1100) can capture images of the surroundings of the device (1000) using a camera module (1400) to be described later, and receive user input for providing services from the device (1000) or a server (not shown) based on the captured images.

The microphone (1200) receives an external sound signal and processes it into electrical voice data. For example, the microphone (1200) can receive an sound signal from an external device or a speaker. The microphone (1200) can use various noise removal algorithms to remove noise generated in the process of receiving an external sound signal. The microphone (1200) can receive a user's voice input for controlling the device (1000).

The display unit (1300) outputs information processed in the device (1000). For example, the display unit (1300) may display a user interface for capturing the surroundings of the device (1000) and information related to a service provided based on captured images of the surroundings of the device (1000). According to one embodiment, the device (1000) may provide an AR (Augmented Reality) image. The display (1300) according to one embodiment may include a wave guide (not shown) and a display module (not shown). The wave guide (not shown) may be made of a transparent material through which a portion of the back surface is visible when the user wears the device (1000). The wave guide (not shown) may be made of a single-layer or multi-layer flat plate made of a transparent material through which light may be reflected and transmitted. The wave guide (not shown) may receive light of a virtual image projected facing the emission surface of the display module. Here, the transparent material means a material through which light can pass, and the transparency may not be 100% and may have a predetermined color. In one embodiment, since the wave guide (not shown) is formed of a transparent material, the user can view not only a virtual object of a virtual image through the display (1300), but also an external actual scene, and therefore, the wave guide (not shown) may be referred to as a see through display. The display (1300) can provide an augmented reality image by outputting a virtual object of a virtual image through the wave guide.

The camera module (1400) can capture images of the surroundings of the device (1000). When an application requiring a shooting function is executed, the camera module can obtain image frames, such as still images or moving images, through an image sensor. Images captured through the image sensor can be processed through a processor (1800) to be described later or a separate image processing unit (not shown).

The shooting direction control unit (1500) can change the shooting direction of the camera module (1400). The shooting direction control unit (1500) can include a hardware structure that can change the shooting direction of the camera module (1400) by panning the camera module (1400). Through the shooting direction control unit (1500), the camera module (1400) can be rotated clockwise or counterclockwise with respect to a predetermined axis, and the camera module (1400) can sequentially shoot around the device (1000) while rotating at a predetermined angle. The shooting direction control unit (1500) can control the shooting direction of the camera module (1400) by using a magnetic force generated by, for example, applying electricity to the electromagnet, including an electromagnet located near the camera module (1400). Alternatively, the shooting direction control unit (1500) may include, for example, a motor physically connected to the camera module (1400), and may control the shooting direction of the camera module (1400) using the motor. However, the method by which the shooting direction control unit (1500) controls the shooting direction of the camera module (1400) is not limited thereto, and the shooting direction of the camera module (1400) may be controlled by rotating the camera module (1400) through various methods.

The communication interface (1600) can transmit and receive data for receiving a service based on an image acquired by photographing the surroundings of the device (1000) with an external device (not shown) and a server (not shown).

The storage unit (1700) can store a program to be executed by the processor (1800) to be described later, and can store data input to or output from the device (1000).

The storage unit (1700) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, an SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk.

The programs stored in the storage unit (1700) can be classified into a plurality of modules according to their functions, and may include, for example, a shooting mode identification module (1710), a shooting angle determination module (1720), a shooting module (1730), an image comparison module (1740), an image arrangement module (1750), a depth calculation module (1760), a gesture recognition module (1770), and a space recognition module (1780).

The processor (1800) controls the overall operation of the device (1000). For example, the processor (1800) can control the user input unit (1100), the microphone (1200), the display unit (1300), the camera module (1400), the shooting direction control unit (1500), the communication interface (1600), and the storage unit (1700) by executing programs stored in the storage unit (1700).

The processor (1800) can identify the shooting mode of the camera module (1400) by executing the shooting mode identification module (1710) stored in the storage (1700). The shooting mode of the camera module (1400) can include, for example, a gesture recognition mode for recognizing a user's gesture and a space recognition mode for recognizing a space around the device (1000). The gesture recognition mode is a shooting mode for shooting a gesture made by a user's body located at a close range in a shooting angle range smaller than a predetermined threshold, and the space recognition mode can be a shooting mode for shooting a space around the device (1000) in a shooting angle range larger than a predetermined threshold.

The processor (1800) can identify the shooting mode of the camera module (1400) according to preset criteria. For example, when the device (1000) is powered on or the device (1000) is activated from an inactive state, the processor (1800) can identify the shooting mode of the camera module (1400) as a gesture recognition mode in order to receive a user's gesture input. In addition, the processor (1800) can recognize the user's gesture in the gesture recognition mode and change the shooting mode of the camera module (1400) according to the recognized gesture. The processor (1800) can execute an application corresponding to the gesture and change the shooting mode of the camera module (1400) according to a shooting mode required by the executed application. In addition, for example, the processor (1800) can deactivate the camera module (1400) when an application that does not require shooting through the camera module (1400) is executed.

Alternatively, for example, when the device (1000) is powered on or the device (1000) is activated from an inactive state, the device (1000) may identify the shooting mode of the camera module (1400) as the space recognition mode. In addition, the processor (1800) may recognize the surrounding space of the device (1000) by shooting the surroundings of the device (1000) in the space recognition mode.

Additionally, for example, when a preset user input for the device (1000) is received, the processor (1800) may identify the shooting mode of the camera module (1400) as a gesture mode or a spatial recognition mode.

The processor (1800) can determine the range of the shooting angle of the camera module (1400) by executing the shooting angle determination module (1720) stored in the storage unit (1700). The processor (1800) can determine the range of the shooting angle according to the identified shooting mode. In this case, the range of the shooting angle according to the shooting mode can be set in advance.

For example, when the shooting mode of the camera module (1400) is gesture mode, the camera module (1400) may be set to shoot the surroundings of the device (1000) within a shooting angle range of -15 degrees to 15 degrees. For example, when the angle when the camera module (1400) is facing the front is assumed to be 0 degrees, when the shooting angle range of the camera module (1400) is -15 degrees to 15 degrees, the camera module (1400) may rotate to the left by -15 degrees and to the right by 15 degrees based on the front.

For example, when the shooting mode of the camera module (1400) is the space recognition mode, the camera module (1400) may be set to shoot the surroundings of the device (1000) within a shooting angle range of -60 degrees to 60 degrees. For example, when the angle when the camera module (1400) is facing the front is assumed to be 0 degrees, when the shooting angle range of the camera module (1400) is -60 degrees to 60 degrees, the camera module (1400) may rotate to the left by -60 degrees and to the right by 60 degrees based on the front.

The processor (1800) can determine the shooting interval of the camera module (1400) by executing the shooting angle determination module (1720) stored in the storage (1700). The processor (1800) can determine the angle interval at which to shoot the surroundings of the device (1000) within the shooting angle range. For example, the processor (1800) can determine the shooting interval to shoot the surroundings of the device (1000) at 5-degree intervals within the shooting angle range. In addition, the shooting interval of the camera module (1400) can be set differently according to, for example, the shooting mode and the shooting environment. The shooting environment can include, but is not limited to, brightness around the device (1000), the number of subjects, and movement of subjects.

The processor (1800) can capture images of the surroundings of the device (1000) through the camera module (1400) by executing the capture module (1730) stored in the storage (1700). The processor (1800) can sequentially capture images of the surroundings of the device (1000) at a determined capture interval within a range of identified capture angles by controlling the camera module (1400) and the capture direction control unit (1500), thereby obtaining a plurality of images. The camera module (1400) can capture images of the surroundings of the device (1000) multiple times at a capture interval while rotating within the capture angle range. Some of the adjacent images among the images captured by the camera module (1400) may overlap each other.

Alternatively, the processor (1800) may control the camera module (1400) and the shooting direction control unit (1500) to capture a video while rotating the camera module (1400) within a shooting angle range. In this case, the processor (1800) may extract a plurality of frames from among the frames in the captured video according to the shooting interval. Some of the adjacent images from among the images of the extracted frames may overlap each other.

The processor (1800) can change the focal length of the camera module (1400) by executing the shooting module (1730) stored in the storage (1700). When the camera module (1400) photographs subjects located at a distance closer than a predetermined threshold, the processor (1800) may not change the focal length of the camera module (1400). When the camera module (1400) is activated, the focal length of the camera module (1400) may be set to measure subjects located at a close distance, and thus, the processor (1800) may not change the focal length of the camera module (1400) to photograph subjects at a close distance. In addition, since the focal lengths for photographing subjects at a close distance near a predetermined distance value are substantially the same, the processor (1800) may not change the focal length of the camera module (1400). In addition, when the camera module (1400) photographs subjects located at a distance greater than a predetermined threshold, the processor (1800) can control the camera module (1400) to change the focal length depending on the subject and photograph the surroundings of the device (1000) according to the changed focal length.

The processor (1800) can compare adjacent images among a plurality of images acquired through the camera module (1400) by executing the image comparison module (1740) stored in the storage unit (1700). The processor (1800) can identify identical feature regions within the adjacent images by comparing the adjacent images with each other.

For example, when five images are acquired from the camera module (1400), the processor (1800) can compare the first image with the second image, the second image with the third image, the third image with the fourth image, and the fourth image with the fifth image. In addition, the processor (1800) can identify that the first feature region in the first image is identical to the second feature region in the second image, identify that the second feature region in the second image is identical to the third feature region in the third image, identify that the third feature region in the third image is identical to the fourth feature region in the fourth image, and identify that the fourth feature region in the fourth image is identical to the fifth feature region in the fifth image. Accordingly, the processor (1800) can more effectively identify that the first feature region in the first image and the fifth feature region in the fifth image are identical. In addition, for example, the feature region in the image may be a predetermined feature point.

The processor (1800) can arrange two images selected from among a plurality of images by executing the image arrangement module (1750) stored in the storage unit (1700). The processor (1800) can select two images whose shooting angles are different by a predetermined threshold or more from among the plurality of images in order to calculate the depth of the feature region. For example, the processor (1800) can select two images whose shooting angles are the greatest difference from among the plurality of images but include the same feature region, thereby allowing the depth of the feature region to be calculated more accurately. For example, when five images are acquired from the camera module (1400), the processor (1800) can select the first image and the fifth image, which are different by a predetermined threshold or more, from among the five acquired images.

Additionally, the processor (1800) may virtually arrange the two images to obtain disparity values for the two selected images. The disparity values for the two images may be values indicating how much a feature region in one of the two captured images is shifted in the other image.

The processor (1800) can virtually arrange the camera modules (1400) so that the shooting directions of the camera modules (1400) become parallel to each other. For example, while maintaining the baseline value between the camera module (1400) when shooting the first image and the camera module (1400) when shooting the fifth image, the camera module (1400) when shooting the first image and the camera module (1400) when shooting the fifth image can be virtually arranged so that the shooting direction of the camera module (1400) when shooting the first image and the shooting direction of the camera module (1400) when shooting the fifth image become parallel to each other. The baseline value can be a value representing the distance between the shooting center of the camera module (1400) when shooting one image and the shooting center of the camera module (1400) when shooting another image. The shooting center of the camera module (1400) can be, for example, the center point of the lens of the camera module (1400). A method for virtually arranging camera modules (1400) will be described later.

The processor (1800) can arrange the images based on the positions of the virtually arranged camera modules (1400). For example, the first image and the fifth image can be virtually arranged based on the arranged positions of the camera modules (1400) when capturing the first image and the arranged positions of the camera modules when capturing the fifth image. In this case, the processor (1800) can obtain a disparity value between the first feature region in the first image and the fifth feature region in the fifth image based on the positions of the first feature region in the virtually arranged first image and the fifth feature region in the virtually arranged fifth image. For example, the disparity value can be calculated based on the distance between the same feature regions in the virtually arranged images.

The processor (1800) can calculate the depth of a feature region by executing a depth calculation module (1760) stored in the storage unit (1700). The processor (1800) can calculate the depth of the feature region by using a disparity value for the same feature regions in the two images, a baseline value of the camera module (1400), and a focal length of the camera module (1400). The processor (1800) can calculate the depth of the feature region by using the fact that a ratio between the baseline value and the focal length is the same as a ratio between the disparity value and the depth.

The processor (1800) can generate a depth map for the surrounding space of the device (1000) by calculating the depth of feature areas within the image. The depth map is an image representing three-dimensional distance information of objects or spaces existing within the image, and each pixel value within the image can represent depth information of the corresponding pixel. The depth information can be a value representing a distance from a view point to a space corresponding to a specific pixel. The depth map can distinguishably represent a part far from the view point and a part near the view point. For example, in the depth map, a part far from the user's view point to a part near the user's view point can be displayed in a gradient manner so that it gradually darkens from white to black. Accordingly, the shapes and depths of objects within the surrounding space of the device (1000) can be distinctly expressed in the depth map.

The processor (1800) can generate a plurality of depth maps for parts of the surrounding space of the device (1000) from a predetermined unit of image sets, and complete the depth map for the surrounding space of the device (1000) by connecting the generated plurality of depth maps. For example, when the camera module (1400) acquires 20 images by photographing the surroundings of the device (1000) while rotating, the processor (1800) can generate a first partial depth map from a first image set including the first to tenth images, generate a second partial depth map from a second image set including the sixth to fifteenth images, and generate a third partial depth map from a third image set including the eleventh to twentieth images. In addition, the processor (1800) can generate an entire depth map by using the first partial depth map, the second partial depth map, and the third partial depth map.

Meanwhile, the processor (1800) can identify the depth of the feature region using an artificial intelligence model. In this case, the processor (1800) can obtain information about the depth of the feature region by inputting images captured from the camera module (1400) into an artificial intelligence model trained to calculate the depth of the feature region from images. The depth information of the feature region obtained from the artificial intelligence model can be used to verify the depth information calculated by the depth calculation module (1760), but is not limited thereto. In addition, the images captured through the camera module (1400) and the depth information calculated by the depth calculation module (1760) can be used to update the artificial intelligence model for depth calculation.

In addition, the processor (1800) can obtain a depth map using an artificial intelligence model. In this case, the processor (1800) can obtain a depth map for the space around the device (1000) by inputting images captured from the camera module (1400) into an artificial intelligence model trained to generate a depth map from images. The depth map obtained from the artificial intelligence model can be used to verify the depth map generated by the depth calculation module (1760), but is not limited thereto. In addition, the images captured through the camera module (1400) and the depth map generated by the depth calculation module (1760) can be used to update an artificial intelligence model for generating a depth map.

The processor (1800) can recognize a user's gesture by executing a gesture recognition module (1770) stored in the storage unit (1700). The processor (1800) can acquire images by repeatedly rotating multiple times within a shooting angle range. For example, the processor (1800) can acquire a first set of n gesture images by rotating for the first time within the shooting angle range, a second set of n gesture images by rotating for the second time within the shooting angle range, and a third set of n gesture images by rotating for the third time within the shooting angle range. In addition, for example, the processor (1800) can identify a first shape and a first position of the user's hand by calculating a depth of the user's hand from images within the first gesture image set. For example, the processor (1800) can identify a second shape and a second position of the user's hand by calculating a depth of the user's hand from images within the second gesture image set. For example, the processor (1800) may calculate the depth of the user's hand from images in the third gesture image set to identify the third shape and the third position of the user's hand. In addition, the processor (1800) may identify a change in the shape of the user's hand based on the first shape, the second shape, and the third shape of the user's hand, and recognize a gesture according to the change in the shape of the identified hand. In addition, the processor (1800) may identify a change in the position of the hand based on the first position, the second position, and the third position of the user's hand, and recognize a gesture according to the change in the position of the identified hand. The processor (1800) may recognize a gesture by considering both the shape change and the position change of the user's hand. In the above, gestures according to the user's hand have been described, but other body parts other than the user's hand may be recognized in order to recognize gestures.

The processor (1800) can identify a gesture using an artificial intelligence model. In this case, the processor (1800) can identify a user's gesture by inputting images captured from the camera module (1400) into an artificial intelligence model trained to identify a user's gesture from images. The gesture identified from the artificial intelligence model can be used to verify a gesture recognized by the gesture recognition model, but is not limited thereto. In addition, the images captured through the camera module (1400) and the gesture recognized by the gesture recognition module (1770) can be used to update an artificial intelligence model for gesture recognition.

The processor (1800) can recognize the space around the device (1000) by executing the space recognition module (1780) stored in the storage (1700). The processor (1800) can identify what feature areas are around the device (1000) from a depth map generated for the periphery of the device (1000). For example, the processor (1800) can generate identification information about what object corresponds to the feature area around the device (1000), coordinate information indicating the relative position of the feature area with respect to the device (1000), depth information indicating the distance between the device (1000) and the feature area, etc.

In addition, the processor (1800) can identify an unrecognized space among the spaces surrounding the device (1000), and when the unrecognized space is captured, generate a depth map for the unrecognized space and add it to the overall depth map. In this case, the processor (1800) can store a history of which areas around the device (1000) the camera module (1400) captured, and activate a space recognition mode when the camera module (1400) faces the space that has not been captured based on the stored history.

The processor (1800) can recognize a space using an artificial intelligence model. In this case, the processor (1800) can recognize a surrounding space of the device (1000) by inputting images or a depth map captured from the camera module (1400) into an artificial intelligence model trained to identify a feature region in a space from images. The spatial information recognized from the artificial intelligence model can be used to verify the space recognized by the spatial recognition model, but is not limited thereto. In addition, the images or depth map captured through the camera module (1400) and the spatial information recognized by the spatial recognition module (1780) can be used to update an artificial intelligence model for spatial recognition.

FIG. 3 is a drawing showing an example of a camera module rotating within a device according to one embodiment of the present disclosure.

Referring to FIG. 3, the shooting direction control unit (1500) can rotate the camera module (1400) clockwise or counterclockwise (32) based on the rotation center (30). Accordingly, the camera module (1400) can sequentially capture images of the periphery of the device (1000) while rotating at a predetermined angle. In addition, the camera module (1400) can capture images of the periphery of the device (1000) while panning or tilting at a predetermined angle around the rotation center (30).

FIG. 4A is a diagram illustrating an example of obtaining multiple images while a camera module within a device rotates according to one embodiment of the present disclosure.

Referring to FIG. 4A, the camera module (1400) in the device (1000) can sequentially capture images of the periphery of the device (1000) while rotating clockwise or counterclockwise at predetermined angles within a predetermined capturing angle range (40). For example, the device (1000) can sequentially capture images of the periphery of the device (1000) while rotating the camera module (1400) clockwise at 5 degrees each. The capturing angle range (40) of the camera module (1400) can be set differently depending on the capturing mode of the device (1000). For example, when the capturing mode of the device (1000) is gesture recognition mode, the capturing angle range (40) can be set to a range of -15 degrees to +15 degrees with respect to the reference axis (42). In addition, for example, when the shooting mode of the device (1000) is a space recognition mode, the shooting angle range (40) may be a range of -30 degrees to +30 degrees with respect to the reference axis (42). The reference axis (42) may be determined by the shooting direction of the camera module (1400) when the camera module (1400) is facing the front of the device (1000).

The device (1000) can obtain first to nth images by photographing the surroundings of the device (1000) through the camera module (1400). In this case, adjacent images among the first to nth images may overlap each other to a certain extent.

FIG. 4b is a diagram illustrating an example in which identical feature regions in adjacent images are sequentially identified according to one embodiment of the present disclosure.

Referring to FIG. 4b, the device (1000) can identify identical feature regions in the images through a stereo matching technique. The device (1000) can identify that a first feature region in a first image is identical to a second feature region in a second image. Additionally, the device (1000) can identify that a second feature region in a second image is identical to a third feature region in a third image. In this way, the device (1000) can identify that an n-1th feature region in an n-1th image is identical to an nth feature region in an nth image by sequentially comparing two adjacent images.

FIG. 4c is a diagram illustrating an example of a device according to one embodiment of the present disclosure identifying identical identification regions within images with the greatest angular difference.

Referring to FIG. 4c, the device (1000) can identify that the first feature region in the first image is the same as the n-th feature region in the n-th image based on the identification result in FIG. 4b. Accordingly, the device (1000) can identify the same feature region in two images having the largest difference in shooting angles among the first to n-th images, and the device (1000) can more accurately calculate the depth of the feature region, which will be described later.

Since adjacent images overlap each other to a large extent and the same specific regions in the adjacent images are displayed in similar positions in the adjacent images, the device (1000) can identify the same specific regions more effectively by comparing the adjacent images. For example, since the first image overlaps the second image to a greater extent than the other images and the position of the first feature region in the first image has a small difference from the position of the second feature region in the second image, even if the device (1000) compares a smaller portion of the first image and the second image, the device (1000) can effectively and accurately identify that the first feature region and the second feature region are the same. In addition, the device (1000) can more accurately identify that the first feature region of the first image and the n-th feature region of the n-th image are the same by sequentially comparing the adjacent images and using the results of the sequential comparison.

Although FIGS. 4A to 4C describe that the device (1000) identifies identical feature regions within the images using all of the captured images, the present invention is not limited thereto. The device (1000) may select a plurality of images, which are part of the captured images, and identify identical feature regions within the selected plurality of images by comparing the selected plurality of images. For example, the device may select third to n-3th images among the first to n-th images, and identify that the third feature region within the third image and the n-3th feature region within the n-3th image are identical by comparing adjacent images among the third to n-3th images.

In FIGS. 4A to 4C, it has been described that the device (1000) identifies the same feature regions from the first image captured at the smallest shooting angle and the n-th image captured at the largest shooting angle within the shooting angle range, but is not limited thereto. The device (1000) may identify the same feature regions from two images having a shooting angle difference greater than or equal to a predetermined threshold, and may also calculate the depth of the feature regions using the identified feature regions. For example, the device (1000) may select the first image and the n-3th image among the first to n-th images, wherein the difference in shooting angles is greater than or equal to a predetermined threshold, and identify the same feature regions from the selected first image and the n-3th image.

FIGS. 5A to 5C are diagrams illustrating examples of a device arranging captured images and obtaining disparity values of feature regions according to one embodiment of the present disclosure.

FIG. 5A is a diagram illustrating an example of a device photographing its surroundings according to one embodiment of the present disclosure.

Referring to FIG. 5A, the device (1000) can sequentially capture images of the periphery of the device (1000) at predetermined capture angle intervals within a capture angle range. In addition, a baseline value (b) for the first image and the n-th image can be determined by the capture center of the camera module (1400-1) when capturing the first image, which are the two images with the greatest difference within the capture angle range, and the capture center of the camera module (1400-n) when capturing the n-th image. For example, the distance between the lens center of the camera module (1400-1) when capturing the first image and the lens center of the camera module (1400-n) when capturing the n-th image can be the baseline value (b).

FIG. 5b is a diagram illustrating an example of a device virtually arranging a camera module according to one embodiment of the present disclosure.

Referring to FIG. 5b, the device (1000) can virtually arrange the camera module (1400-1) when shooting the first image and the camera module (1400-n) when shooting the nth image so that the shooting direction (53) of the camera module (1400-1) when shooting the first image and the shooting direction (54) of the camera module (1400-n) when shooting the nth image become parallel to each other while maintaining the baseline value (b) between the camera module (1400-1) when shooting the first image and the camera module (1400-n) when shooting the nth image.

FIG. 5c is a diagram illustrating an example of a device calculating a disparity value according to one embodiment of the present disclosure.

Referring to FIG. 5c, the device (1000) may arrange the first image and the n-th image to calculate disparity values (d) of the same feature regions in the first image and the n-th image. For example, the device (1000) may arrange the first image and the n-th image along the vertical axis (55) and the vertical axis (56). In addition, for example, the vertical sides of the first image and the n-th image may not coincide with the vertical axis (55) and the vertical axis (56). In this case, the device (1000) may rectify the first image and the n-th image according to a predetermined algorithm so that the vertical sides of the first image and the n-th image coincide with the vertical axis (55) and the vertical axis (56).

Additionally, the device (1000) can calculate a disparity value (d) between the first feature region and the nth feature region based on the position of the first feature region in the arranged first image and the position of the nth feature region in the arranged nth image. For example, the disparity value (d) between the first feature region and the nth feature region can be determined by the distance between a vertical line (57) crossing the first feature region and a vertical line (58) crossing the nth feature region.

FIG. 6 is a diagram showing an example of a device calculating the depth of a feature region according to one embodiment of the present disclosure.

Referring to FIG. 6, the depth for the same feature region in the two images can be calculated using the disparity values for the same feature regions in the two images, the baseline value of the camera module (1400), and the focal length of the camera module (1400). For example, based on the fact that the ratio between the depth (z) of the feature region and the focal length (f) of the camera module (1400) is equal to the ratio of the disparity value (d) and the baseline value (b), the depth (z) of the feature region can be calculated as in Mathematical Expression 1 below.

<Mathematical formula 1>

Depth of feature region (z) = {disparity value (d) x focal length (f)}/baseline value (b)

For example, in mathematical expression 1, the depth (z) of the feature region is the depth of the first feature region and the nth feature region, which are the same feature regions within the image, the disparity value (d) is a disparity value for the first feature region and the nth feature region within the image, the baseline value (b) is a distance between the center of the lens of the camera module (1400) when photographing the first feature region and the center of the lens of the camera module (1400) when photographing the nth feature region, and the focal length (f) may be a focal length of the camera lens (1400) when photographing the first feature region and the nth feature region.

FIG. 7 is a flowchart of a method for calculating the depth of a feature region in an image by a device according to one embodiment of the present disclosure.

In operation S700, the device (1000) can sequentially capture images of the surroundings of the device (1000) while rotating the camera module (1400). The device (1000) can sequentially capture images of the surroundings of the device (1000) at a determined capturing interval within a range of identified capturing angles by controlling the camera module (1400) and the capturing direction control unit (1500), thereby obtaining a plurality of images. The camera module (1400) can capture images of the surroundings of the device (1000) multiple times at a capturing interval while rotating within a capturing angle range. Some of the adjacent images among the images captured by the camera module (1400) may overlap each other.

In S710, the device (1000) can identify identical feature regions in the images by comparing adjacent images among the plurality of captured images. The device (1000) can identify that a first feature region in a first image is identical to a second feature region in a second image. Additionally, the device (1000) can identify that a second feature region in a second image is identical to a third feature region in a third image. In this way, the device (1000) can identify that an n-1th feature region in an n-1th image is identical to an nth feature region in an nth image by sequentially comparing two adjacent images.

In operation S720, the device (1000) may select a first image and an n-th image from among a plurality of images, wherein the shooting angles are greater than or equal to a predetermined value. For example, when n images are captured, the first image captured at the smallest shooting angle and the n-th image captured at the largest shooting angle within the shooting angle range may be selected, but is not limited thereto. The device (1000) may identify identical feature regions from two images having a shooting angle difference greater than or equal to a predetermined threshold, and may calculate the depth of the feature regions using the identified feature regions. For example, the device (1000) may select a first image and an n-3th image from among the first to n-th images, wherein the shooting angle difference is greater than or equal to a predetermined threshold, and may identify identical feature regions from the selected first image and the n-3th image.

In operation S730, the device (1000) can obtain a baseline value based on the shooting center of the camera module (1400) when shooting a first image and the shooting center of the camera module (1400) when shooting a second image. The baseline value may be a value representing a distance between the shooting center of the camera module (1400) when shooting one image and the shooting center of the camera module (1400) when shooting another image. The shooting center of the camera module (1400) may be, for example, the center point of the lens of the camera module (1400).

In operation S740, the device (1000) can calculate a disparity value between a first feature region in a first image and an n-th feature region in an n-th image. The device (1000) can virtually arrange a camera module when capturing a first image and a camera module when capturing an n-th image while maintaining a baseline value, and arrange the first image and the second image according to a predetermined criterion. In addition, the device (1000) can calculate a disparity value between the first feature region and the second feature region based on a position of the first feature region in the arranged first image and a position of the n-th feature region in the arranged n-th image.

In operation S750, the device (1000) can calculate the depth of a feature region based on a disparity value, a baseline value, and a focal length of a camera module. The device (1000) can calculate the depth for the same feature region in the two images by using the disparity values for the same feature regions in the two images, the baseline value of the camera module (1400), and the focal length of the camera module (1400). The device (1000) can calculate the depth of the feature region by using the fact that a ratio between the baseline value and the focal length is the same as a ratio between the disparity value and the depth.

FIG. 8 is a diagram illustrating an example of a device calculating a disparity value according to one embodiment of the present disclosure.

In S800, the device (1000) can virtually arrange the camera module (1400-1) when shooting the first image and the camera module (1400-n) when shooting the n-th image while maintaining the baseline value. The device (1000) can virtually arrange the camera module (1400-1) when shooting the first image and the camera module (1400-n) when shooting the n-th image so that the shooting direction of the camera module (1400-1) when shooting the first image and the shooting direction of the camera module (1400-n) when shooting the n-th image become parallel to each other.

In S810, the device (1000) can arrange the first image and the n-th image. The device (1000) can arrange the first image and the n-th image along the vertical axis (55) and the vertical axis (56). When the vertical sides of the first image and the n-th image do not coincide with the vertical axis (55) and the vertical axis (56), the device (1000) can rectify the first image and the n-th image according to a predetermined algorithm so that the vertical sides of the first image and the n-th image coincide with the vertical axis (55) and the vertical axis (56).

In operation S820, the device (1000) can calculate a distance between a first feature region in the first image and an n-th feature region in the n-th image. The device (1000) can calculate a distance between the first feature region and the n-th feature region based on a position of the first feature region in the first image arranged in operation S820 and a position of the second feature region in the second image arranged in operation S820.

FIG. 9 is a flowchart of a method for a device to recognize its surroundings according to a shooting mode according to one embodiment of the present disclosure.

In operation S900, the device (1000) can identify the shooting mode. For example, when the device (1000) is powered on or the device (1000) is activated from an inactive state, the device (1000) can identify the shooting mode of the camera module (1400) as the gesture recognition mode in order to receive a user's gesture input. In addition, the device (1000) can recognize the user's gesture in the gesture recognition mode and change the shooting mode of the camera module (1400) according to the recognized gesture. The device (1000) can execute an application corresponding to the gesture and change the shooting mode of the camera module (1400) according to the shooting mode required by the executed application. In addition, for example, when an application that does not require shooting through the camera module (1400) is executed, the device (1000) can deactivate the camera module (1400).

Alternatively, for example, when the device (1000) is powered on or the device (1000) is activated from an inactive state, the device (1000) may identify the shooting mode of the camera module (1400) as the space recognition mode.

Additionally, for example, when a preset user input for the device (1000) is received, the processor (1800) may identify the shooting mode of the camera module (1400) as a gesture mode or a spatial recognition mode.

As a result of the identification in operation S900, if the shooting mode is identified as the gesture mode, the device (1000) can identify a first shooting angle range corresponding to the gesture mode in operation S905. For example, if the shooting mode of the camera module (1400) is the gesture mode, the camera module (1400) can be set to shoot the surroundings of the device (1000) in a shooting angle range of -15 degrees to 15 degrees. For example, if the angle when the camera module (1400) is facing the front is 0 degrees, when the shooting angle range of the camera module (1400) is -15 degrees to 15 degrees, the camera module (1400) can rotate to the left by -15 degrees and to the right by 15 degrees based on the front.

In operation S910, the device (1000) can capture the surroundings of the device (1000) within a first shooting angle range. The device (1000) can sequentially capture the surroundings of the device (1000) at a predetermined shooting interval within the first shooting angle range by controlling the camera module (1400) and the shooting direction control unit (1500), thereby obtaining a plurality of images. In this case, the device (1000) can determine the shooting interval of the camera module (1400).

In operation S915, the device (1000) can identify an object related to a gesture from the captured images. The device (1000) can calculate depth values of feature areas related to multiple objects in the captured images and compare the calculated depth values. The device (1000) can identify an object related to the user's gesture by comparing the calculated depth values. In addition, the device (1000) can identify a shape and a location of the identified object.

In operation S920, the device (1000) can identify a gesture based on the shape and position of the object. The device (1000) can identify a user's gesture based on changes in the shape of the object and changes in the position of the object.

As a result of the identification in operation S900, if the shooting mode is identified as the space recognition mode, the device (1000) can identify a second shooting angle range corresponding to the space recognition mode in operation S950. For example, if the shooting mode of the camera module (1400) is the space recognition mode, the camera module (1400) can be set to shoot the surroundings of the device (1000) in a shooting angle range of -60 degrees to 60 degrees. For example, if the angle when the camera module (1400) is facing the front is 0 degrees, when the shooting angle range of the camera module (1400) is -60 degrees to 60 degrees, the camera module (1400) can rotate to the left by -60 degrees and to the right by 60 degrees based on the front.

In operation S955, the device (1000) can capture the surroundings of the device (1000) within the second shooting angle range. The device (1000) can sequentially capture the surroundings of the device (1000) at a predetermined shooting interval within the second shooting angle range by controlling the camera module (1400) and the shooting direction control unit (1500), thereby obtaining a plurality of images. In this case, the device (1000) can determine the shooting interval of the camera module (1400).

In operation S960, the device (1000) can identify surrounding objects from the captured images. The device (1000) can calculate depth values of feature areas related to multiple objects in the captured images and compare the calculated depth values. By comparing the calculated depth values, the device (1000) can identify objects around the device (1000), thereby recognizing the surrounding space of the device (1000).

According to one embodiment, the device (1000) can identify an object that a user is gazing at. When the user's eyes are fixed for a predetermined period of time, the device (1000) can sequentially capture images of the surroundings in the direction in which the user is looking while rotating the camera module (1400) within a predetermined shooting angle range. In addition, the device (1000) can generate a depth map of the surroundings in the direction in which the user is looking, and identify the object that the user is gazing at. In addition, the device (1000) can display additional information about the identified object. The additional information can include, for example, an identification value of the object and a depth value of the object, but is not limited thereto. The device (1000) can also display a GUI for providing the user with additional information about the identified object.

FIG. 10A is a diagram illustrating an example of a device capturing a user's hand movements in gesture mode according to one embodiment of the present disclosure.

Referring to FIG. 10A, a user may perform a gesture of bending a finger (102), and the device (1000) may sequentially capture multiple images of the bent finger (102) in the gesture mode. For example, the device (1000) may sequentially capture images of the finger (102) while rotating the camera module (1400) multiple times at a predetermined shooting interval within a first shooting angle range corresponding to the gesture mode. Accordingly, the device (1000) may obtain multiple image sets including n images. For example, the device (1000) may obtain a first image set (104) captured in a state where the finger (102) is extended, and a second image set (105) captured in a state where the finger (102) is bent.

FIG. 10b is a diagram illustrating an example of a device (1000) recognizing a gesture from image sets according to one embodiment of the present disclosure.

Referring to FIG. 10b, the device (1000) can generate a first depth map (112) from a first image set (104) captured in a state where the finger (102) is extended, and can generate a second depth map (113) from a second image set (105) captured in a state where the finger (102) is bent.

Additionally, the device (1000) can identify a spread finger (115) from the first depth map (112) and a bent finger (116) from the second depth map (113). Thereafter, the device (1000) can identify that the user's gesture is a 'click' motion by analyzing the shape of the spread finger (115) and the shape of the bent finger (116).

FIGS. 11A to 11C are diagrams showing examples of a device analyzing a space around a device according to one embodiment of the present disclosure.

FIG. 11A is a diagram illustrating an example of a device grouping multiple images acquired by photographing the surroundings of the device into multiple image sets according to one embodiment of the present disclosure.

Referring to FIG. 11A, the device (1000) may sequentially capture images of the surroundings of the device (1000) while rotating the camera module (1400) multiple times at predetermined shooting intervals within a second shooting angle range corresponding to the space recognition mode. In addition, the device (1000) may obtain multiple image sets by grouping adjacent images among the multiple images. The multiple image sets may be composed of consecutive images. Some of the consecutive images in the image set may overlap some of the consecutive images in the adjacent image set.

FIG. 11b is a diagram illustrating an example of multiple image sets grouped by a device according to one embodiment of the present disclosure.

Referring to FIG. 11b, the device (1000) can obtain a first image set (114), a second image set (115), and a third image set (116) from a plurality of images captured around the device (1000). The first image set (114) can include image (1), image (2), image (3), and image (4), the second image set (115) can include image (3), image (4), image (5), and image (6), and the third image set (116) can include image (5), image (6), image (7), and image (8).

Additionally, since the feature region (9) disappears from the image (5) but exists in the images in the first image set (114), the device (1000) can calculate the depth value of the feature region (9) from the image set (114).

FIG. 11C is a diagram illustrating an example of a device generating and analyzing a depth map representing space around the device from depth maps generated from multiple sets of images according to one embodiment of the present disclosure.

Additionally, the device (1000) can generate a first partial depth map (117) from the first image set (114), a second partial depth map (118) from the second image set (115), and a third partial depth map (119) from the third image set (116). Additionally, the device (1000) can generate and analyze a full depth map (120) using the first partial depth map (117), the second partial depth map (118), and the third partial depth map (119).

An embodiment of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules, that are executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Additionally, computer-readable media can include computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Communication media typically includes computer-readable instructions, data structures, or other data in a modulated data signal, such as program modules.

Additionally, the computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term 'non-transitory storage medium' means a tangible device and does not contain signals (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently in the storage medium and cases where data is stored temporarily. For example, the 'non-transitory storage medium' may include a buffer in which data is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play StoreTM) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a part of the computer program product (e.g., a downloadable app) may be at least temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or an intermediary server.

Additionally, in this specification, a “part” may be a hardware component such as a processor or a circuit, and/or a software component executed by a hardware component such as a processor.

Additionally, in this specification, “comprising at least one of a, b, or c” can mean “comprising only a, including only b, including only c, including both a and b, including both b and c, including both a and b, and including all of a, b, and c.”

The function related to artificial intelligence according to the present disclosure is operated through a processor and a memory. The processor may be composed of one or more processors. At this time, one or more processors may be a general-purpose processor such as a CPU, an AP, a DSP (Digital Signal Processor), a graphics-only processor such as a GPU, a VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU. One or more processors control to process input data according to a predefined operation rule or artificial intelligence model stored in a memory. Alternatively, when one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is learned by using a plurality of learning data by a learning algorithm, thereby creating a predefined operation rules or artificial intelligence model set to perform a desired characteristic (or purpose). Such learning may be performed in the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and/or system. Examples of the learning algorithm include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

The artificial intelligence model may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and performs neural network operations through operations between the operation results of the previous layer and the plurality of weights. The plurality of weights of the plurality of neural network layers may be optimized by the learning results of the artificial intelligence model. For example, the plurality of weights may be updated so that the loss value or cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks.

An artificial intelligence model according to the present disclosure can use image data as input data of an artificial intelligence model to output output data that recognizes an image or a feature region within an image. The artificial intelligence model can be created through learning. Here, being created through learning means that a basic artificial intelligence model is learned by using a plurality of learning data by a learning algorithm, thereby creating a predefined operation rule or artificial intelligence model set to perform a desired characteristic (or purpose). The artificial intelligence model can be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and performs a neural network operation through an operation between the operation result of the previous layer and the plurality of weights.

Visual understanding is a technology that recognizes and processes objects like human vision, and includes object recognition, object tracking, image retrieval, human recognition, scene recognition, spatial understanding (3D reconstruction/localization), and image enhancement.

The above description of the present disclosure is for illustrative purposes, and those skilled in the art will understand that the present disclosure can be easily modified into other specific forms without changing the technical idea or essential features of the present disclosure. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. For example, each component described as a single component may be implemented in a distributed manner, and likewise, components described as distributed may be implemented in a combined manner.

The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

Claims (20)

In a method for a device to obtain depth of a feature region using one camera module,
A step of acquiring multiple images by sequentially rotating one camera module at a preset angle and taking multiple shots of the surroundings of said one camera module;
A step of identifying a first feature region in the first image and an n-th feature region that is identical to the first feature region in the n-th image by comparing adjacent images between a first image and an n-th image among the plurality of images;
A step of obtaining a base line value for the first image and the nth image based on the arrangement of the one camera module when capturing the first image and the arrangement of the one camera module when capturing the nth image;
A step of obtaining a disparity value between the first feature region and the nth feature region based on the location of the first feature region in the first image and the location of the nth feature region in the nth image; and
A step of calculating the depth of the first feature region or the nth feature region based on the baseline value and the disparity value;
A method comprising:
In the first paragraph,
The step of obtaining the above multiple images is:
A method for sequentially photographing the surroundings of one camera module while panning the one camera module at a predetermined angular interval within a predetermined shooting angle range.
In the first paragraph,
The step of identifying the first feature region and the nth feature region is:
A step of identifying feature regions that are identical to the first feature region among feature regions in the first image to the nth image by sequentially comparing adjacent images among the first image to the nth image in the plurality of images;
A method comprising:
In the first paragraph,
The above one camera module is sequentially rotated at a preset angle, thereby changing the shooting angle of the above one camera module.
A method according to claim 1, wherein the baseline values for the first image and the nth image are determined based on a distance value between a position of a camera lens of the one camera module when acquiring the first image and a position of a camera lens of the one camera module when acquiring the first image.
In the first paragraph,
The step of obtaining the above disparity value is:
A step of virtually arranging the one camera module when shooting the first image and the one camera module when shooting the n-th image so that the shooting direction of the one camera module when shooting the first image and the shooting direction of the one camera module when shooting the n-th image become parallel to each other;
A step of arranging the first image and the nth image based on the shooting direction of the above-deployed one camera module; and
A step of obtaining a distance value between the first feature region in the arranged first image and the nth feature region in the arranged nth image;
A method comprising:
In clause 5,
A method according to claim 1, wherein the step of placing virtually places one camera module when shooting the first image and one camera module when shooting the n-th image while maintaining the baseline values for the first image and the n-th image.
In the first paragraph,
A method according to claim 1, wherein the step of calculating the depth calculates the depth based on the baseline value, the disparity value, and the focal length of the one camera module when capturing the first image and the nth image.
In the first paragraph,
A step of determining a shooting mode of the above one camera module; and
A step of identifying a preset shooting angle range according to the above shooting mode;
Including more,
A method wherein the step of acquiring the plurality of images comprises acquiring the plurality of images while sequentially rotating the single camera module at the preset angle within the identified shooting angle range.
In Article 8,
A method according to claim 1, wherein the shooting mode includes a gesture recognition mode for identifying a user's gesture around the one camera module and a space recognition mode for recognizing a space around the one camera module.
In the first paragraph,
The step of obtaining the above multiple images is:
A step of determining a focal length of the above one camera module;
Including more,
A method wherein the surroundings of one camera module are photographed according to the determined focal length.
In a device that acquires depth of a feature area using a single camera module,
One camera module;
A shooting direction control unit that rotates the above one camera module at a predetermined angle;
display;
Memory that stores one or more instructions; and
A processor executing one or more of the above instructions;
Including,
The above processor,
By controlling the above shooting direction control unit and the one camera module, the one camera module is sequentially rotated at a preset angle and the surroundings of the one camera module are photographed multiple times to obtain multiple images.
By comparing adjacent images between a first image and an n-th image among the plurality of images, a first feature region in the first image and an n-th feature region identical to the first feature region in the n-th image are identified,
Based on the arrangement of the one camera module when shooting the first image and the arrangement of the one camera module when shooting the nth image, a base line value for the first image and the nth image is obtained,
Based on the location of the first feature region in the first image and the location of the nth feature region in the nth image, a disparity value between the first feature region and the nth feature region is obtained,
A device that calculates the depth of the first feature region or the nth feature region based on the baseline value and the disparity value.
In Article 11,
A device wherein the processor sequentially captures images of the surroundings of the one camera module by executing the one or more instructions, thereby panning the one camera module at a predetermined angular interval within a predetermined shooting angle range.
In Article 11,
The above processor executes one or more of the above instructions,
A device for identifying feature regions that are identical to the first feature region among feature regions in the first image to the nth image by sequentially comparing adjacent images among the first image to the nth image in the plurality of images.
In Article 11,
The above one camera module is sequentially rotated at a preset angle, thereby changing the shooting angle of the above one camera module.
A device, wherein the baseline values for the first image and the nth image are determined based on a distance value between a position of a camera lens of the one camera module when acquiring the first image and a position of a camera lens of the one camera module when acquiring the first image.
In Article 11,
The above processor executes one or more of the above instructions,
When shooting the first image, the one camera module and the one camera module when shooting the nth image are virtually arranged so that the shooting direction of the one camera module when shooting the first image and the shooting direction of the one camera module when shooting the nth image are parallel to each other,
Arrange the first image and the nth image based on the shooting direction of the above-deployed one camera module,
A device for obtaining a distance value between the first feature region in the arranged first image and the nth feature region in the arranged nth image.
In Article 15,
The above processor executes one or more of the above instructions,
A device for virtually positioning one camera module when capturing the first image and one camera module when capturing the n-th image while maintaining the baseline values for the first image and the n-th image.
In Article 11,
The above processor executes one or more of the above instructions,
A device that calculates the depth based on the baseline value, the disparity value, and the focal length of the one camera module when capturing the first image and the nth image.
In Article 11,
The above processor executes one or more of the above instructions,
Determine the shooting mode of the above one camera module,
Identify the preset shooting angle range according to the above shooting mode,
A device that acquires the plurality of images by sequentially rotating the one camera module at the preset angle within the identified shooting angle range.
In Article 18,
A device wherein the above shooting mode includes a gesture recognition mode for identifying a user's gesture around the one camera module and a space recognition mode for recognizing a space around the one camera module.
A computer-readable recording medium having recorded thereon a program for executing the method of claim 1 on a computer.

Images