TW201514830A - Interactive operation method of electronic apparatus - Google Patents

Abstract

An interactive operation method of an electronic apparatus is provided. An image sequence is captured by an image capturing unit. An image pre-processing is executed for an image of the image sequence. A fingertip candidate region is obtained from the image. Whether the fingertip candidate region is connected with a hand region is detected. If the fingertip candidate region is connected with the hand region, whether a click event is generated is detected. When the click event is generated, a corresponding function is executed.

Description

Interactive method of operation

The invention relates to an interactive control mechanism, and in particular to an interactive operation method based on gesture recognition.

With the development of electronic devices in recent years, electronic devices such as smart phones and tablet computers have become more and more popular. People began to pay attention to issues such as the quality and ability that electronic devices can provide to users. For example, the electronic device can provide a Human Machine Interface (HMI) to assist the user in interacting with the electronic device. The design of the human-machine interface depends on the needs and habits of the users. For example, physical controllers such as mice, keyboards, and remote controls are the most common human-machine interfaces available today.

Users use these physical controllers to operate electronic devices such as computers or televisions. With the development of the human-machine interface, it is now moving from the physical controller to the virtual controller. The virtual controller not only provides users with a new experience, but also has many benefits. First, the virtual controller provides a variety of input methods, that is, the user can switch between the physical controller and the virtual controller according to the usage requirements to select an appropriate input method. In addition, the virtual controller can also be based on the user The need to change its size and appearance, and the virtual controller does not take up physical space.

Augmented Reality (AR) is a technique for calculating the position and angle of camera images in real time and adding corresponding images. The goal of this technology is to put the virtual world on the screen and interact with it in the real world. . Most existing augmented reality systems are usually based on web cameras. However, these types of cameras can only capture two dimensional (2D) data. The use of limited two-dimensional data to complete three-dimensional (3D) spatial positioning requires not only very advanced algorithms, but also the results obtained are not accurate, resulting in the virtual controller unable to explain the user's intentions and commands.

The present invention provides an interactive operation method for detecting a user's finger, thereby transmitting an instruction through a finger to allow the user to interact with the electronic device.

The interactive operation method of the electronic device of the present invention comprises: capturing an image sequence through the image capturing unit; performing image preprocessing on the image in the image sequence; obtaining a fingertip candidate region from the image; and determining whether the fingertip candidate region is with the hand The area is connected; if the fingertip candidate area is connected to the hand area, the fingertip candidate area is used as the target fingertip area; by continuously tracking the target fingertip area, it is judged whether a click event occurs; and when the click event occurs, execution is performed The function corresponding to the electronic device.

In an embodiment of the invention, the step of determining whether the fingertip candidate region is connected to the hand region comprises: obtaining a center point of the fingertip candidate region as a reference point; obtaining the first side point from the four directions of the reference point respectively Second side point, third side point and a four-sided point, wherein the first side point, the second side point, the third side point, and the fourth side point are respectively located outside the fingertip candidate area; the first side point and the second side point are obtained from the depth information of the image, a first depth value, a second depth value, a third depth value, and a fourth depth value of the third side point and the fourth side point; and determining the first depth value, the second depth value, the third depth value, and the fourth depth Whether the value is greater than 0; determining that the fingertip candidate region is connected to the hand region in a case where only one of the first depth value, the second depth value, the third depth value, and the fourth depth value is greater than 0; and in other cases Next, it is determined that the fingertip candidate area is not connected to the hand area.

In an embodiment of the invention, the method further includes: receiving from the present The target fingertip area of the image obtains the first tracking point; determines whether the first tracking point corresponds to the position of the display position in the display unit is located; and if the display position corresponding to the first tracking point is located at the position of the function item, The second tracking point is taken out from the target fingertip area of the previously received image, and the first tracking point and the second tracking point are compared to determine whether a click event occurs.

In an embodiment of the invention, the comparing the first tracking point with the second tracking The step of determining whether a click event occurs includes: comparing a vertical axis displacement and a horizontal axis displacement between the first tracking point and the second tracking point; and comparing the first tracking point with the second according to the depth information Tracking the amount of depth change between the points; if the vertical axis displacement is less than the first preset value, the horizontal axis displacement is less than the second preset value, and the depth change is less than the third preset value, determining that a click event occurs; And if the vertical axis displacement is greater than or equal to the first preset value, the horizontal axis displacement is greater than or equal to the second preset value, and the depth variation is greater than or equal to the third preset value, wherein at least one condition is satisfied, the determination does not occur point Select the event.

In an embodiment of the invention, the step of determining whether a click event occurs The step further includes: taking the first calculation point and the second calculation point in the target fingertip area of the currently received image based on the first tracking point, wherein the first tracking point is located between the first calculation point and the second calculation point Calculating the depth difference between the first calculation point and the second calculation point according to the depth information; the displacement on the vertical axis is less than the first preset value, the displacement on the horizontal axis is less than the second preset value, and the depth variation is less than the third In the case of the preset value, if the depth difference is greater than or equal to the fourth preset value, it is determined that a click event occurs; if the depth difference is less than the fourth preset value, it is determined that the click event does not occur.

In an embodiment of the invention, the method further includes: in the display unit Displaying the augmented reality interactive interface; displaying the received image in the augmented reality interactive interface; displaying the first virtual layer in the augmented reality interactive interface when the target face region is obtained in the image, wherein the first virtual layer The function item is included; when the function item is triggered, the second virtual layer is displayed in the augmented reality interaction interface, wherein the second virtual layer includes a virtual control interface.

In an embodiment of the invention, the step of performing the pre-image processing package Include: Perform a background removal procedure. The background removal process includes: detecting a plurality of objects to be processed in the image; and filtering one or more unimportant objects from the object to be processed according to the depth information of the image, wherein the depth value of the unimportant object is greater than a preset Depth value. That is, the object to be processed whose depth value is smaller than the preset depth value is retained.

In an embodiment of the invention, after the step of executing the background removal procedure, the method further includes performing a face pose estimation procedure. Above The face pose estimation program includes: performing a face detection process on the remaining objects to be processed, and obtaining a plurality of face regions; and extracting a target face region from the obtained face region according to the depth information of the image, wherein The depth value of the target face region is the minimum value of the depth values of the face regions (ie, the closest to the image capturing unit); and one of the objects to be processed in which the target face region with the minimum depth value is the minimum value, Filter out other objects to be processed.

In an embodiment of the invention, the above-described method for performing a face pose estimation program The step further includes performing a hand detection process. The above hand detection program includes: using a skin color detection algorithm to obtain a hand region.

In an embodiment of the invention, after the step of executing the hand detection program, the method further includes performing a fingertip detection procedure to obtain a fingertip candidate area.

Based on the above, the function corresponding to the electronic device is executed by detecting the finger operated by the user in the three-dimensional space and detecting that the pointing event is triggered by the finger. Accordingly, the user can interact with the electronic device by transmitting an instruction through the finger.

The above described features and advantages of the invention will be apparent from the following description.

100‧‧‧Electronic devices

110‧‧‧Image capture unit

120‧‧‧Processing unit

130‧‧‧Display unit

140‧‧‧ storage unit

300‧‧‧Image Processing Module

310‧‧‧Background removal module

320‧‧‧Face posture estimation module

330‧‧‧Hand Detection Module

340‧‧‧ fingertip detection module

350‧‧‧ fingertip tracking module

360‧‧‧Click event identification module

400‧‧‧ images

401‧‧‧Hand area

501, 60, 61‧‧‧ candidate fingertip areas

601, 611, 621, 631, P1‧‧‧ first side points

602, 612, 622, 632, P2‧‧‧ second side points

603, 613, 623, 633, P3‧‧‧ third side points

604, 614, 624, 634, P4‧‧‧ fourth side point

701‧‧‧ first calculation point

702‧‧‧First Tracking Point

703‧‧‧ second calculation point

710‧‧‧ Target fingertip area

800‧‧‧Augmented Reality Interactive Interface

810‧‧‧ Target face area

820, 830, 860‧‧‧ functional items

840, 850‧‧‧ virtual control interface

R, R1, R2‧‧‧ reference points

θ 1 , θ 2‧‧‧ angle

Steps of the S205~S235‧‧‧ interactive operation method

1 is a block diagram of an interactive operating system in accordance with an embodiment of the present invention.

2 is a flow chart of an interactive method of operation in accordance with an embodiment of the present invention.

FIG. 3 is a schematic diagram of an image processing module according to an embodiment of the invention.

4 is a schematic illustration of an image with a hand region in accordance with an embodiment of the present invention.

Figure 5 is a schematic illustration of a hand region in accordance with an embodiment of the present invention.

6A and 6B are schematic diagrams showing the manner of determining a target fingertip region according to an embodiment of the invention.

FIG. 7 is a schematic diagram of determining a click event according to an embodiment of the invention.

8A and 8B are schematic diagrams of an operational method of incorporating an augmented reality interaction interface in accordance with the present invention.

With the development of technology, people began to pay attention to the functions of electronic devices provided to users and the quality of the devices themselves. The invention provides an interactive operation method of an electronic device, which allows a user to use his finger to transmit instructions in a three-dimensional space to control an electronic device. In order to clarify the content of the present invention, the following specific examples are given as examples in which the present invention can be implemented.

1 is a block diagram of an interactive operating system in accordance with an embodiment of the present invention. Referring to FIG. 1 , the interactive operating system 100 includes an image capturing unit 110 , a processing unit 120 , a display unit 130 , and a storage unit 140 . The processing unit 120 is coupled to the image capturing unit 110, the display unit 130, and the storage unit 140. The image capturing unit 110 captures the image of the user, and the processing unit 120 recognizes the motion of the user in the image, thereby executing the function corresponding to the electronic device 100. For details of each component, please refer to the following.

The image capturing unit 110 is configured to capture an image. For example, the image capturing unit 110 may be A depth camera or stereo camera, or any camera or camera having a charge coupled device (CCD) lens, a complementary metal oxide semiconductor transistor (CMOS) lens, or an infrared lens . The direction in which the image capturing unit 110 faces may be a direction that facilitates photographing to the user.

The processing unit 120 is configured to analyze the image captured by the image capturing unit 110. The processing unit 120 is, for example, a central processing unit (CPU), or other programmable general purpose or special purpose microprocessor (Microprocessor), digital signal processor (DSP), programmable Controllers, Application Specific Integrated Circuits (ASICs), Programmable Logic Devices (PLDs), Field-programmable Gate Arrays (FPGAs), or other similar devices or A combination of these devices.

The display unit 130 can be any type of display, such as a liquid crystal display (LCD) or a light emitting diode (LED), or a flat display, or a projection display, or a soft display. .

The storage unit 140 is, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory (Flash memory), hard Discs or other similar devices or combinations of these devices are used to record a plurality of modules executable by processing unit 120 to implement an interactive method of operation.

This embodiment is implemented by a code. For example, the storage unit 140 stores There are a plurality of code segments, and the above code segments are executed by the processing unit 120 after being installed. Accordingly, the interactive operating system 100 can accurately detect the user's finger in a complex natural environment, and pass the finger to the user to interact with the machine. The steps of controlling the interactive operating system 100 by detecting a finger are further explained below.

2 is a flow chart of an interactive method of operation in accordance with an embodiment of the present invention. Referring to FIG. 1 and FIG. 2 simultaneously, in step S205, the image sequence is captured by the image capturing unit 110. For example, the image capturing unit 110 captures an image every other sampling frequency. Next, in step S210, the processing unit 120 performs image pre-processing on the image in the image sequence. The above image pre-processing includes a background removal program, a face posture estimation program, a hand detection program, and the like. After obtaining the hand region, in step S215, the processing unit 120 obtains a fingertip candidate region from the image. This means finding the area that may be the fingertip in the hand area.

Then, in step S220, the processing unit 120 determines that the fingertip candidate area is No connection to the hand area. As far as the general human body structure is concerned, one side of the fingertip is connected to other hands. Therefore, the correct fingertip area is found here by judging whether the fingertip candidate area is connected to the hand area. If the fingertip candidate area is connected to the hand area, in step S220, the processing unit 120 uses the fingertip candidate area as the target fingertip area. If the fingertip candidate area is not connected to the hand area, it indicates that there is no fingertip in the currently received image, and the process returns to step S205, and then continues to receive the next image from the image capturing unit 110.

After obtaining the target fingertip region, in step S230, the processing unit 120 determines whether a click event occurs by continuously tracking the target fingertip region in the image sequence. That is, it is detected whether the user performs a specific gesture in the three-dimensional space to prepare to drive the functions in the electronic device 100. When a specific gesture is detected, that is, when a click event is recognized, the processing unit 120 performs a corresponding function in step S235. On the other hand, if the occurrence of the click event is not detected, the process proceeds to step S230. For example, when the display unit 130 displays a screen and the user performs a click gesture at a position corresponding to the three-dimensional space with respect to a function item in the screen, the processing unit 120 obtains the image acquisition unit 110 by analyzing the image capturing unit 110. The image knows that a click event occurs here, and the processing unit 120 can execute the function item.

An example will be given below to explain the implementation process of the above method. However, the following embodiments are merely illustrative of one of them, and are not limited thereto.

FIG. 3 is a schematic diagram of an image processing module according to an embodiment of the invention. In this embodiment, the image processing module 300 is a computer software composed of code segments stored in the storage unit 140 for execution by the processing unit 120. In other embodiments, the image processing module 300 may also be a hardware composed of various types of chips, and coupled to the processing unit 120, and driven by the processing unit 120, and does not limit the image processing mode. The implementation of group 300. The image processing module 300 includes a background removal module 310, a face pose estimation module 320, a hand detection module 330, a fingertip detection module 340, a fingertip tracking module 350, and a point. Event identification module 360 is selected.

The background removal module 310 performs a background removal process to remove the background. Keep the area most likely to exist for the user. Here, a background subtraction method can be used to obtain a region in which a human body exists. For example, a background image in which the user does not exist is created in the electronic device 100 in advance, and then the image captured by the image capturing unit 110 is subtracted from the background image, so that two images can be obtained. Areas with differences.

The face pose estimation module 320 performs a face pose estimation program, which is used to The image output by the background removal module 310 is subjected to a face detection process and a face tracking process. After obtaining the image with the background removed, the face pose estimation module 320 is used to determine whether there is a face in the image. For example, a Haar-like feature based Adaptive Boosting (AdaBoost) learning algorithm is used to identify a face in an image to obtain a face region.

After detecting the face region, the face pose estimation module 320 further uses the Continuously Adaptive Mean-Shift (Camshift) algorithm to continuously track the face position. The Camshift algorithm tracks the color of a moving object, such as a face area. Therefore, regardless of how the user moves and turns his head, the face position of the user can still be obtained through the Camshift algorithm.

The hand detection module 330 performs a hand detection process that detects a hand region in the image using a skin color detection algorithm. The action of detecting the hand area can be further divided into three parts, that is, body mask, skin color detection, and image enhancement. In order to find the face and the human body, after detecting the face area, the hand detection module 330 uses a body mask to cover the face area in the image and Human body area. Since the depth value of the body is substantially the same as the depth value of the face, the size and coverage area of the body mask can be automatically changed according to the depth value of the detected face region.

After obtaining the body area, the hand detection module 330 further performs a skin color detection algorithm. For example, first, the image is converted from the RGB color space to the YCbCr color space using Equation (1) below. In the YCbCr color space, Y represents image luminance, and Cb and Cr represent chrominance of the image, respectively.

After converting the color space, use the bottom program (2) to divide the pixels into skin color or non-skin. That is, the pixel has a Cr value between 133 and 173, and the Cb value is between 77 and 127, that is, the pixel is determined to be a skin color pixel. Pixels that do not meet the above conditions are treated as non-skinning pixels.

After performing the skin tone detection algorithm, an image enhancement algorithm can be further performed to remove the noise. For example, a closing operation or an opening operation in a morphology is used to remove noise. Then, a Gaussian Blur filter is used to remove the remaining noise and smooth the shape.

For example, Figure 4 is a hand region in accordance with an embodiment of the present invention. A schematic representation of the image. Referring to FIG. 4 , after the image 400 is processed by the background removal module 310 , the face posture estimation module 320 , and the hand detection module 330 , the hand region 401 can be obtained.

The fingertip detection module 340 performs a fingertip detection procedure to find in the hand area Get the correct fingertip area. Specifically, after the hand region is obtained, the fingertips in the hand region are detected using the defined fingertip property, and the fingertip region is filtered out. Here, the characteristics of the finger can be defined for the real finger, and the fingertip property is defined in advance. Fingertip attributes include: the relationship between the hand and the fingertip is like a tree with branches; and one side of the fingertip is connected to the hand.

Fingertip detection module 340 can utilize morphological disconnection operations (opening Operation) first erosion and then dilation. That is to say, the image is first eroded to narrow the area, and then the image is expanded to expand the area; or, the erosion is repeated until all unwanted points or lines of noise are removed, and then the original image is restored by expansion. After such a procedure, the noise points can be removed. Here, in order to skip the palm portion, a disconnection operation is performed with a 6×6 cross-shaped structuring element. After the disconnection operation is performed, a first image is obtained, and the original image is subtracted from the original image to obtain a candidate fingertip region.

Thereafter, the fingertip detection module 340 can further further have a square of 3×3. The structural elements are used to perform a break operation to make the shape of the candidate fingertip region smoother and to remove noise. For subsequent operations, a bounding box is used to represent the candidate fingertip regions. For example, Figure 5 is a schematic illustration of a hand region in accordance with an embodiment of the present invention. Please refer to FIG. 5, in the case of the hand area 401 of FIG. After the contour of the finger may be reached, the detected contour of the finger may be framed by a bounding box to obtain the candidate fingertip region 501.

Even if the above operation can obtain the correct fingertip area, in order to avoid the presence of non-fingertip areas, the depth information of the image can be further utilized to filter out the non-fingertip area. A depth map is generated for the image, and the purpose of the depth map is to determine whether the pixel belongs to the foreground. For example, in the depth map, the depth of the pixel belonging to the background is 0, and the depth of the pixel covered by the body mask established by the hand detection module 330 is 0, and the hand detection module is not The depth of the pixel covered by the body mask established by 330 is a value greater than zero. That is, in the depth map, the depth values of the pixels belonging to the background object are zeroed, and the pixels belonging to the foreground object have depth values greater than zero.

After obtaining the depth information, the fingertip detection module 340 can use the depth value to exclude the non-fingertip region. The description will be made below with reference to FIG. 5. First, the center point of the fingertip candidate area 501 is obtained as the reference point R, and then the first side point P1, the second side point P2, and the third are respectively obtained from the four directions (up, down, left, and right) of the reference point R. Side point P3 and fourth side point P4. Here, the first side point P1, the second side point P2, the third side point P3, and the fourth side point P4 are located outside the fingertip candidate area 501, respectively.

It is assumed that H and W represent the height and width of the fingertip candidate region 501, respectively. The first side point P1 and the second side point point P2 are points obtained by taking the reference point R upward and downward and taking 0.75×H, respectively. Further, the third side point P3 and the fourth side point point P4 are points obtained by taking 0.9×W from the reference point R to the left and right, respectively. The above 0.75×H and 0.9×W are merely illustrative. In other embodiments, only the first side point P1 and the second side point are provided. P2, the third side point P3, and the fourth side point P4 may be located outside the fingertip candidate area 501, respectively.

After obtaining the four side points, the first depth value of the first side point P1, the second side point P2, the third side point P3, and the fourth side point P4 are obtained from the depth information (the above depth map), respectively. Two depth values, a third depth value, and a fourth depth value. Next, it is determined whether the first depth value, the second depth value, the third depth value, and the fourth depth value are greater than zero. If only one of the first depth value, the second depth value, the third depth value, and the fourth depth value is greater than 0, it is determined that the fingertip candidate region 501 is connected to the hand region. Further, in the processed depth map, the depth value of the pixel as the background object is zeroed, and only the pixel as the foreground object has a depth value greater than zero. Since the first side point P1, the second side point P2, the third side point P3, and the fourth side point P4 have been processed, if only one of the first, second, third, and fourth side points belongs to the foreground object, that is, If only one of the first to fourth depth values is greater than 0, it is determined that the fingertip candidate region 501 is connected to the hand region. In other cases, it is determined that the fingertip candidate area 501 is not connected to the hand area.

In addition, since the user's fingers may not be erect, in other cases they may be skewed. Therefore, after obtaining four side points at the center point, the four side points may be further rotated clockwise or counterclockwise, and then the depth of the four side points after the rotation is used to determine whether the fingertip candidate area is Connected to the hand area. For example, FIG. 6A and FIG. 6B are schematic diagrams showing the manner of determining a target fingertip region according to an embodiment of the invention. Figure 6A is a clockwise rotation and Figure 6B is a counterclockwise rotation.

In FIG. 6A, the center point of the fingertip candidate region 60 is found as the reference point R1, starting from the reference point R1, going up, down, left, and right, in the fingertip candidate area. The first side point 601, the second side point 602, the third side point 603, and the fourth side point 604 are respectively obtained on the outer side of the 60. Then, after the first side point 601, the second side point 602, the third side point 603, and the fourth side point 604 are clockwise rotated by an angle θ 1 , the first side point 611 and the second side point 612 are regained. The third side point 613 and the fourth side point 614. Thereafter, if the depth value of only one of the first side point 611, the second side point 612, the third side point 613, and the fourth side point 614 is greater than 0, it is determined that the fingertip candidate area 60 is connected to the hand area. .

Similarly, in FIG. 6B, the center point of the fingertip candidate region 61 is found as the reference point R2, starting from the reference point R2 in the four directions of up, down, left, and right, respectively, on the outer side of the fingertip candidate region 61. The first side point 621, the second side point 622, the third side point 623, and the fourth side point 624. Then, after the first side point 621, the second side point 622, the third side point 623, and the fourth side point 624 are rotated counterclockwise by an angle θ 2 , the first side point 631 and the second side point 632 are regained. The third side point 633 and the fourth side point 634. Thereafter, if the depth value of only one of the first side point 631, the second side point 632, the third side point 633, and the fourth side point 634 is greater than 0, it is determined that the fingertip candidate area 61 is connected to the hand area. .

The fingertip tracking module 350 performs a fingertip tracking program to track the user's fingers. For example, first, the fingertip tracking module 350 effectively finds a plurality of good feature points in the target fingertip region using corner detection. In order to accurately analyze the movement of the user's fingertip, the centroid of the plurality of feature points is taken as a tracking point. The above angle detection is a method for capturing features and inferring image content in a computer vision system. Angle detection is used for motion detection, image registration, video tracking (video) Tracking), object recognition, and the like.

Next, the fingertip tracking module 350 first performs a dynamic tracking algorithm of a continuous image, such as an optical flow. Here, the Lucas-Kanade tracking method is used to estimate the change of optical flow, and the image pyramid concept is used to extend the Lucas-Kanade tracking method. The image pyramid concept can analyze faster movements and achieve more accurate offsets.

The click event recognition module 360 executes a click event recognition program to determine whether the user has triggered a specific function. In general, if a user wants to press a function item in the display unit 130 with a finger in a three-dimensional space, the user can roughly divide into two behaviors. In the first behavior, the user's fingers do not move in the direction of up, down, left, or right, but move forward. In the second behavior, the difference between the depth value of the top pixel of the fingertip and the depth value of the bottom pixel is greater than a threshold.

Specifically, the click event recognition module 360 obtains the first tracking point from the target fingertip area of the currently received image. Next, it is determined whether the first tracking point corresponds to whether the display position in the display unit 130 is located at the position of the function item. If the display position corresponding to the first tracking point is located at the location of the function item, the click event recognition module 360 takes out the second tracking point from the target fingertip area of the previously received image, and then compares the first tracking point with The second tracking point to determine if a click event occurred. For example, the vertical axis displacement amount or the horizontal axis displacement amount between the first tracking point and the second tracking point is compared, and the depth change amount between the first tracking point and the second tracking point is compared according to the depth information.

If the vertical axis shift between the tracking points of the two images is smaller than the first If the preset value and the horizontal axis displacement are both smaller than the second preset value, and the depth change between the tracking points of the two images is smaller than the third preset value, the click event recognition module 360 determines that the click is selected. event. On the other hand, if the vertical axis displacement is greater than or equal to the first preset value, the horizontal axis displacement is greater than or equal to the second preset value, and the depth variation is greater than or equal to the third preset value, at least one of the conditions is true, then The click event identification module 360 determines that no click event has occurred. For example, it is determined by the bottom program (3) whether or not a click event occurs.

Where (X_old, Y_old) is the coordinate of the second tracking point of the previous received image, (X_new, Y_new) is the coordinate of the first tracking point of the currently received image, and |X_old-X_new| is the horizontal axis displacement, and | Y_old-Y_new| is the vertical axis displacement. In addition, d_old is the depth value of the second tracking point of the previous received image, d_new is the depth value of the first tracking point of the currently received image, and |d_old-d_new| is the depth variation amount. The first preset value, the second preset value, and the third preset value are respectively exemplified by 10 pixels, 10 pixels, and 0.5 cm.

In addition, the step of determining whether a click event occurs may further determine whether a click event occurs according to two calculation points of the current image. For example, FIG. 7 is a schematic diagram for determining a click event according to an embodiment of the invention. FIG. 7 shows the hand region in the currently received image. After the target fingertip region 710 is obtained, the first tracking point 702 is obtained by the fingertip tracking module 350. After that, at the first tracking point 702 Two calculation points, namely, a first calculation point 701 and a second calculation point 703, are taken out respectively. Here, the first tracking point 702 is located between the first calculation point 701 and the second calculation point 703. Then, the point event identification module 360 calculates the depth difference between the first calculation point 701 and the second calculation point 703 according to the depth information. If the depth difference is greater than or equal to the fourth preset value, it is determined that a click event occurs. If the depth difference is less than the fourth preset value, it is determined that no click event has occurred. For example, it is determined by the bottom program (4) whether or not a click event occurs.

Wherein, d_down represents a depth value of the second calculation point 703 located below the first tracking point 702, and d_up represents a depth value of the first calculation point 701 located above the first tracking point 702. The fourth preset value is exemplified here as 1.2 cm.

In addition, it is worth mentioning that the above equations (3) and (4) can also be combined to determine whether a click event occurs. For example, if the vertical axis displacement between the tracking points of the two images before and after is smaller than the first preset value, the horizontal axis displacement is less than the second preset value, and the depth change amount is less than the third preset value, If the depth difference between the first calculation point 701 and the second calculation point 703 in the current image is greater than or equal to the fourth preset value, it is determined that the click event occurs. That is, the vertical axis displacement (|Y_old-Y_new|) is less than 10 pixels, the horizontal axis displacement (|X_old-X_new|) is less than 10 pixels, and the depth variation (|d_old-d_new|) is less than 0.5. In the case of centimeters, if the depth difference (d_down-d_up) is greater than or equal to 1.2 cm, the click event can be determined.

The click event identification module 360 determines the processing unit after determining the occurrence of the click event. 120 will go to perform the corresponding function.

The above embodiment can be applied not only to one user but also to a single user after performing appropriate processing when there is a majority in the image capturing range of the image capturing unit 110. Specifically, in the case that there are a majority of people in the image, in order to distinguish the uninterested objects to obtain possible human objects in the scene, the background removal module 310 can further detect multiple to-be-processed images. An object (such as a plurality of portrait areas), and according to the depth information of the image, the unimportant objects are filtered out from the objects to be processed. Here, a depth threshold can be set to filter out unimportant objects (eg, users who are too far away). For example, suppose the depth threshold is set to 150 cm, and it is assumed that the image includes three objects A, B, and C to be tested, and the corresponding depth values are 160 cm, 110 cm, and 140 cm, respectively. According to this, the background removal module 310 filters the object A to be tested whose depth value is greater than the preset depth value, and retains the objects B and C whose depth value is smaller than the preset depth value.

Then, when the face pose estimation module 320 executes the face pose estimation program, the face pose estimation module 320 performs a face detection process on the remaining objects B and C to obtain multiple faces. The area, and the target face area is taken out from these face areas according to the depth information of the image. Here, the depth value of the target face region is the minimum value among the depth values of these face regions. That is, the closer to the user of the image capturing unit 110, the smaller the depth value of the face region obtained. Then, the face pose estimation module 320 retains the object B to be processed in which the target face region whose depth value is the minimum value, and filters out other objects to be processed C. That is to say, the user who is the closest to the image capturing unit 110 corresponds to the object to be processed in the image. Here, the image capturing unit 110 is disposed on the display Near the unit 130.

In addition, it is worth mentioning that the above embodiment can be combined with the augmented reality interactive interface, and the finger can be detected by the finger to interact with the computer through the augmented reality interactive interface in the three-dimensional space.

For example, Figures 8A and 8B are schematic illustrations of methods of operation in conjunction with an augmented reality interactive interface in accordance with the present invention. The description will be made below with reference to Fig. 1. In FIG. 8A, an augmented reality interactive interface 800 is displayed in display unit 130. The image currently received by the image capturing unit 110 is displayed in the augmented reality interactive interface 800. For example, the image capturing unit 110 is photographed toward the user and disposed in the vicinity of the position where the display unit 130 is located (for example, disposed above the display unit 130). When there is a user in the image capturing range of the image capturing unit 110, Then, the user is simultaneously displayed in the live image presented by the augmented reality interactive interface 800 of FIG. 8A. Accordingly, the user can operate the augmented reality interactive interface 800 correspondingly by viewing the corresponding portrait in the augmented reality interactive interface 800. Here, regarding the determination of the user's finger recognition and the click event, reference may be made to the related description of the image processing module 300 shown in FIG. 3, which is omitted here.

When the target face region 810 is obtained in the image, as shown in FIG. 8A, the first virtual layer is displayed in the augmented reality interaction interface 800, wherein the first virtual layer includes at least one function item. In this embodiment, the first virtual layer includes two functional items 820, 830. The function item 830 is used to open the second virtual layer, and the function item 820 is used to exit the first virtual layer.

When the function item 830 is triggered, as shown in FIG. 8B, the second virtual layer is displayed in the augmented reality interaction interface 800, wherein the second virtual layer includes at least one virtual control interface. In FIG. 8B, the second virtual layer includes two virtual control interfaces 840, 850 and a function item 860. The virtual control interface 840 is, for example, a menu, the virtual control interface 850 is a virtual keyboard, and the function item 860 is used to exit the second virtual layer, or directly close the augmented reality interaction interface 800. The positions displayed by the virtual control interfaces 840 and 850 are merely illustrative and not limited thereto.

According to this, when the user operates the electronic device 100 in a three-dimensional space by viewing the portrait corresponding to the augmented reality interaction interface 800, the image processing module 300 determines whether the behavior is based on the motion trajectory of the user's finger. The conditions of the defined click event (such as equations (3), (4) above) are satisfied, thereby determining whether the user intends to operate the virtual control interface 840 or the virtual control interface 850, or click on the function item 860.

For example, the user moves his finger in a three-dimensional space and learns through the augmented reality interactive interface 800 that the finger position of the portrait in the image has corresponded to the position of the function option 860, at which time the user stops moving his finger. And perform a click gesture in 3D space. Furthermore, the image processing module 300 can determine that a click event occurs, and accordingly, the augmented reality interaction interface 800 of FIG. 8B returns to the first virtual layer as shown in FIG. 8A or directly closes the augmented reality interaction. Interface 800 (subject to the execution function set by function item 860). The user can also operate the virtual control interface 840 or the virtual control interface 850 in a similar manner as described above.

In summary, the above embodiment provides an interactive method for understanding the instructions transmitted by the user's finger in the three-dimensional space by analyzing the image obtained by the image capturing unit, and allowing the user to interact with the electronic device. According to this, the user does not need to wear or wear Any auxiliary equipment, such as a hand without marking any color markers or wearing any data glove, can use your fingers to interact with the electronic device. In addition, through the above embodiments, it is also possible to operate in a natural environment without setting the position of the somatosensory device and limiting the environment in which the user is located. Moreover, the augmented reality interactive interface can be further combined, thereby facilitating the user to interact with the electronic device.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and any one of ordinary skill in the art can make some changes and refinements without departing from the spirit and scope of the present invention. The scope of the invention is defined by the scope of the appended claims.

Steps of the S205~S235‧‧‧ interactive operation method

Claims (10)

An interactive operation method of an electronic device includes: capturing an image sequence through an image capturing unit; performing an image pre-processing on an image in the image sequence; obtaining a fingertip candidate region from the image; determining the fingertip Whether the candidate region is connected to the one-hand region; if the fingertip candidate region is connected to the hand region, the fingertip candidate region is used as a target fingertip region; and by continuously tracking the target fingertip region, determining whether a point selection occurs An event; and when the click event occurs, performing a function corresponding to the electronic device. The method of claim 1, wherein the determining whether the fingertip candidate region is connected to the hand region comprises: obtaining a center point of the fingertip candidate region as a reference point; from the reference point The first direction, the second side point, the third side point, and the fourth side point are respectively obtained in the four directions, wherein the first side point, the second side point, the third side point and the The fourth side points are respectively located outside the fingertip candidate area; the first side point, the second side point, the third side point, and the first side point are obtained from a depth information of the image. a depth value, a second depth value, a third depth value, and a fourth depth value; and determining whether the first depth value, the second depth value, the third depth value, and the fourth depth value are greater than 0; In a case where only one of the first depth value, the second depth value, the third depth value, and the fourth depth value is greater than 0, determining that the fingertip candidate region is connected to the hand region; and In this case, it is determined that the fingertip candidate area is not connected to the hand area. The method of claim 1, further comprising: obtaining a first tracking point from the target fingertip region of the image currently received; determining that the first tracking point corresponds to a display position in a display unit Whether it is located at a location of a function item; and if the display position corresponding to the first tracking point is located at the location of the function item, comprising: taking a second tracking point from the target fingertip area of the image received last time; And comparing the first tracking point with the second tracking point to determine whether the clicking event occurs. The method of claim 3, wherein the step of comparing the first tracking point with the second tracking point to determine whether the click event occurs comprises: comparing the first tracking point with the second a vertical axis displacement between the tracking points; a horizontal axis displacement between the first tracking point and the second tracking point; comparing the first tracking point with the second tracking according to a depth information a depth variation between points; If the vertical axis displacement is less than a first preset value, the horizontal axis displacement is less than a second preset value, and the depth change is less than a third preset value, determining that the click event occurs; and if The vertical axis displacement is greater than or equal to the first preset value, the horizontal axis displacement is greater than or equal to the second preset value, and the depth variation is greater than or equal to the third preset value, wherein at least one condition is met. Then it is determined that the click event does not occur. The method of claim 4, wherein the step of determining whether the click event occurs, further comprising: extracting, based on the first tracking point, a target in the target fingertip region of the currently received image a calculation point and a second calculation point, wherein the first tracking point is located between the first calculation point and the second calculation point; and calculating, according to the depth information, one of the first calculation point and the second calculation point a depth difference; if the vertical axis displacement is less than the first preset value, the horizontal axis displacement is less than the second preset value, and the depth change is less than the third preset value, if the depth is If the difference is greater than or equal to a fourth preset value, it is determined that the click event occurs; if the depth difference is less than the fourth preset value, it is determined that the click event does not occur. The method of claim 1, further comprising displaying an augmented reality interactive interface in a display unit; displaying the received image in the augmented reality interactive interface; obtaining a image in the image a first virtual layer is displayed in the augmented reality interaction interface, wherein the first virtual layer includes a function item; When the functional item is triggered, a second virtual layer is displayed in the augmented reality interaction interface, wherein the second virtual layer includes a virtual control interface. The method of claim 1, wherein the performing the image pre-processing comprises: performing a background removal process, comprising: detecting a plurality of objects to be processed in the image; and determining a depth information according to the image Extracting one or more unimportant objects from the objects to be processed, wherein a depth value of the or the unimportant objects is greater than a predetermined depth value. The method of claim 7, wherein after the step of performing the background removal procedure, the method further comprises: performing a face pose estimation procedure, comprising: performing a face detection on the remaining objects to be processed Measure the program to obtain a plurality of face regions; and according to the depth information of the image, extract a target face region from the face regions, wherein the depth value of the target face region is the depth of the face regions The minimum value of the value; and retaining one of the objects to be processed in the target face region where the depth value is a small value, and filtering out other objects to be processed. The method of claim 8, wherein after the step of performing the face pose estimation procedure, the method further comprises: performing a hand detection procedure, including: The hand region is obtained using a skin tone detection algorithm. The method of claim 9, wherein after performing the step of the hand detection procedure, the method further comprises: performing a fingertip detection procedure to obtain the fingertip candidate area.