KR102194272B1 - Enhancing touch inputs with gestures - Google Patents

Abstract

Methods and systems in accordance with one or more embodiments are provided to enhance interactive inputs. In one embodiment, the method includes simultaneously capturing touch input data on a screen of the user device and non-touch gesture input data away from the screen of the user device. The method also includes determining an input command based at least in part on a combination of simultaneously captured touch input data and non-touch gesture input data. And, the method further includes acting on the operation of the user device based on the determined input command.

Description

Enhancement of touch inputs using gestures {ENHANCING TOUCH INPUTS WITH GESTURES}

[0001] The present disclosure relates generally to interactive inputs on user device interfaces.

[0002] Currently, user devices (eg, smart phones, tablets, laptops, etc.) with interactive input capabilities such as touch screens generally have small-sized screens. Such small screens may limit interactive input capabilities. For example, a small screen may interfere with interactions such as smooth scrolling, panning, zooming, and the like. Touch interactions may require the user to repeatedly perform actions to perform a task, such as multiple pinches, selections, or scroll motions on the user device's touch screen. . Additionally, touch inputs make it difficult to see affected content by obscuring the screen of the user device.

Accordingly, there is a need in the art to improve interactive inputs on user device interfaces.

[0004] Systems and methods according to one or more embodiments of the present disclosure, by means of sensors such as ultrasonic sensors, image or video capturing capabilities (eg, camera), depth sensors, etc. Non-touch input data and touch input data that may be detected (for example, detected from gestures performed using a control object such as a position or motion of a hand, finger, etc.) are combined. It provides enhanced interactive inputs that can be used. Advantageously, the view of the user device's interface (eg, display or screen) may remain substantially unobstructed.

According to an embodiment, a method includes simultaneously capturing touch input data on a screen of a user device and non-touch gesture input data away from the user device; Determining an input command based at least in part on a combination of simultaneously captured touch input data and non-touch gesture input data; And acting on the operation of the user device based on the determined input command.

[0006] According to another embodiment, a method includes at least one of a touch on a screen of a user device, and a pose of at least a portion of a hand not touching the user device, and a motion of the portion in a plane substantially parallel to the screen. Simultaneously detecting one at the user device; Determining a command based at least in part on a combination of the detected touch and the detected pose or motion; And adjusting the operation of the user device based on the determined command. In one embodiment, the portion of the hand includes another portion of the hand that is used to touch the screen. In another embodiment, the method further includes, prior to determining the command, determining that the portion of the hand includes another portion of the hand that is used to touch the screen. In another embodiment, detecting the pose or motion further includes identifying one or more extremities of the hand. In another embodiment, the hand is different from the hand used to touch the screen. In another embodiment, the motion includes a swipe. In another embodiment, the motion includes rotation or sweep. In another embodiment, the portion of the hand is outside a volume comprising a screen and a screen vertical extension perpendicular to the screen. In another embodiment, detecting the touch further comprises detecting a plurality of touches on the screen. In another embodiment, the location of the touch on the screen identifies the object or motion to be motion adjusted. In another embodiment, the detecting of the touch further includes detecting a movement of the touch according to the screen, wherein the determining of the command includes a combination of the detected touch movement and the detected pose or motion. And determining the command based at least in part on the. In another embodiment, the detected pose or motion includes a detected pose. In another embodiment, the detected pose or motion includes the detected motion. In another embodiment, the detected pose or motion includes a detected motion performed while the portion of the hand is in the detected pose.

1A-1C are diagrams illustrating a flow for enhancing a touch using non-touch gestures, according to an embodiment of the present disclosure.
2A-2B are diagrams illustrating a flow for enhancing a touch using non-touch gestures, according to another embodiment of the present disclosure. The drawings have been changed.
3A-3C are diagrams illustrating a flow for enhancing touch using non-touch gestures, according to another embodiment of the present disclosure.
4 is a flow diagram illustrating a method of enhancing touch inputs using non-touch gestures, according to an embodiment of the present disclosure.
5 is a block diagram of a system for implementing a device according to an embodiment of the present disclosure.

Systems and methods according to one or more embodiments of the present disclosure, for example, through a simultaneous multimodal sensor scheme, off-screen (off-screen) non-touch input It is provided to enhance interactive inputs such as touch inputs with s (eg, gestures). Non-touch inputs are performed using a control object such as hand, finger, etc., which may be detected by sensors such as ultrasonic sensors, image or video capturing capabilities (e.g., camera), depth sensors, etc. It may also include gestures such as pose or motion. In that regard, sensors or technologies configured to detect non-touch inputs may be included within the user device or system and/or when non-touch data such as gestures is performed directly in front of the user device. Various surfaces of the user device, for example, so that they may be captured (on-screen) as well as when performed outside the direct line of the screen field of view of the user device (off-screen). , May be located on the top, bottom, left, right, and/or rear of the user device. In general, off-screen non-touch inputs may hereinafter also be referred to as "off-screen gestures", where "off-screen gestures" refer to when a control object such as a hand, finger, pen, etc. touches the user device. Although not, it may refer to the position or motion data of a control object in proximity to the user device. These “off-screen” non-touch gestures may not only be removed from the screen of the user device, but also the portion of the control object that is laterally offset from the device with respect to the screen or display of the device. It may also include. For example, a volume that extends away from the display or screen of the device in a direction substantially perpendicular to the plane of the display or screen can be envisioned. "Off-screen" gestures may include gestures in which at least a portion of the control object performing the gesture is outside this volume. For example, in FIGS. 1A-1C and 2A-2B, the users' fingers are “off-screen”.

[0013] Simultaneous interactive inputs such as touch inputs and off-screen non-touch inputs (eg, gestures) may generate advanced commands. In one embodiment, the user may touch the target item displayed on the screen of the user device, and at the same time make off-screen gestures (e.g., the same hand or another) to adjust the aspect of the target item. Fingers of the hand) can also be used.

[0014] Off-screen non-touch gestures may include any suitable pre-defined gesture corresponding to adjusting a particular aspect of the target item. For example, a specific hand motion or position may be used to adjust the setting, eg, a downward hand motion may correspond to reducing the brightness setting of the screen.

[0015] Touch input-sensing data and non-touch off-screen input sensing data (hereinafter referred to as "gesture input-sensing data") may be combined with these input-sensing data sets to generate one input command. So that they are captured substantially simultaneously. That is, as novel complex commands may be made possible, the input target and variable adjustment of the input target may be combined. It should be noted that, in various embodiments, there may be a minimal or negligible offset between when a gesture, such as motion, begins or ends and when a touch input is detected. In some embodiments herein, touch input-sensing data and gesture input-sensing data may be captured substantially together or simultaneously, e.g., to allow a small leeway between the timing of the touch and the gesture. May be.

[0016] In various embodiments, gesture input-sensing data may be captured by detection of the position and movement of an object. For example, the detected movement of an object such as a hand or a finger may be substantially parallel or coplanar with a surface such as a display or screen of a user device, or may be performed off-screen rather than on a screen. . Therefore, blocking of the display or screen is avoided or reduced, thus allowing the screen to be visible.

[0017] There are many examples of complex commands that may combine input target and variable adjustment. Such examples may include adjusting one or multiple settings by using one or two hands, including the following.

[0018] Camera: Selection of the target item through touch and non-touch gestures where the target item acts to adjust, for example, focus, aperture, scene mode, etc. For example, in the case of taking a picture with a camera, the user may select out-of-focus of the picture through a touch (e.g., using an object such as a finger or a pen) to adjust the focus. While making a selection, you may perform non-touch gestures that may be captured on the side of the camera. In that regard, since non-touch gestures may be performed off-screen (e.g. on the side of the camera, on the top, on the bottom, or behind), the displayed picture is not obstructed from the user's view. . This example will be illustrated in more detail below with respect to FIGS. 1A-1C according to one embodiment.

[0019] Settings: Selection of a desired setting area or icon through a touch or gesture adjusts the setting or a setting associated with the icon, for example, volume, brightness, and the like. Similar to the camera example described above, the user may perform a non-touch gesture that may be captured off-screen while selecting an icon representing a desired setting or setting such as brightness through a touch. The non-touch gesture may correspond to adjusting the brightness. In various embodiments, any command icon associated with the variable control may be touched and then variability adjusted by gestures such as sweeping or rotating a hand around. Thus, in some embodiments, for example, a second menu with a slider bar for adjusting brightness, volume, etc. may not be required. Because gestures are performed and captured off-screen, the display or screen is not obstructed from the user's view. For example, the user device may detect a touch input such as a user's thumb touching an icon representing brightness adjustment, on a screen of the user device or on a side of a bezel. At the same time, the user device may detect a gesture such as rotating the fingers of the same hand of the user (while touching the brightness icon) to adjust the brightness, e.g. rotating to the right may increase the brightness. Yes, and rotating it to the left may reduce the brightness.

Gaming: Selection of the opponent through gestures and touches such as hand position or motion for setting an action to be performed on the opponent (opponent). Similar to the camera example described above, a user playing a game through a user device may perform a non-touch gesture, which may be captured off-screen, while selecting an opponent through touch. The non-touch gesture may correspond to setting an action to be performed on the other party, and since the non-touch gesture is performed and captured off-screen, the display or screen of the user device is not obstructed from the user's view. For example, in a chess game displayed on the screen of the user device, the user device may detect a touch input, such as a user's thumb touching an icon representing an opponent's piece. At the same time, the user device may detect a gesture such as the user holding the same hand (while touching an icon representing the opponent's chess piece) and remove the chess piece according to the game rules.

[0021] Timer: Selection of a timing area or icon through a touch and a gesture to adjust the duration. Similar to the camera example described above, the user may perform a non-touch gesture that may be captured off-screen while selecting a timer area or icon representing a setting such as a timer duration through touch. The non-touch gesture may correspond to adjusting the timer duration, and since the non-touch gesture is performed and captured off-screen, the display or screen is not obstructed from the user's view. In one example, touching the clock and then rotating the hand one way or another may also be used to set the time (e.g., forward/reverse time). Moving to may be determined by the number of fingers spread out, which may make a distinction between changing the hour or changing the minute In another example, there is only one way to adjust the time. There may be a gulf, for example, every rotation of the hand may adjust the minute.

Map: Selection of a POI through touch and gestures to adjust the zoom while keeping the user's touch or a point of interest (POI) centered under the hand. In one embodiment, the user may move his or her touch to adjust where the POI is displayed on the screen. Similar to the camera example described above, the user may perform a non-touch gesture, which may be captured off-screen, while selecting a POI through touch on a map displayed on the user device. The non-touch gesture may correspond to adjusting zooming, and since the non-touch gesture is performed and captured off-screen, the display or screen is not obstructed from the user's view.

[0023] Creating different types of "clicks": to click on the selected object (eg, to initiate an action on the selected object), on the screen of the user device through touch and gesture Selection of displayed objects. For example, the user may perform a non-touch gesture that may be captured off-screen while selecting an object such as a file displayed on the screen of the user device through touch. A non-touch gesture may be equivalent to a grasped hand, which may be equivalent to a right-mouse click (RMC), or a left-mouse click (LMC) or an alternate click. It may also include an open hand that is located. In some embodiments, different hand poses may be used to represent RMC and/or LMC, or may be used to represent different types of clicks.

[0024] Embodiments of the present disclosure may be applied to many other use cases where complex commands for variable adjustment may combine touch input and non-touch gesture, including the following.

[0025] The user device touches the item with one hand (for example, the hand curled around the rear of the user device to hold the device with a thumb), and then It is also possible to detect a touch input by a user swiping off-screen using the other hand. In embodiments for a system with “live tiles” (or other items with dynamic content), the swiping motion is across content (eg, across email messages or You can also scroll up/down in the list. Conversely, these interactions may also work in other ways, e.g., the user device may detect the touching of an item that keeps the item anchored in place, but with the other hand. The detection of a swiping gesture by may cause "home screens" to switch behind the item. When the user has located an appropriate home screen to place the item on, the user releases the item by moving his thumb away from the screen in order to "drop" the item on the currently displayed home screen. release).

[0026] In various embodiments, the touch input may not be a static touch (or a single touch). Dragging items from one place to another is based on a detected gesture, e.g., a hand pose in the case of dragging (e.g., one finger spread out versus the entire hand spread out). It may have different effects based on it. Similarly, a pinch zoom may have different effects depending on the detected gesture, for example, an open hand may act as a coarse zoom, and a grasped hand may have a fine ) You can also activate the zoom.

As described above, some use cases or examples may include adjusting one or multiple settings via a gesture using one or two hands. In some embodiments, it may be a matter of whether the gesture was made with the same hand or with a different hand. Thus, in some embodiments, the user device may be configured to determine whether the same hand is touching the device and is performing a gesture, such as a motion.

Advantageously, new compound commands (eg, new compound input gestures) including both an input target and variable adjustment may be possible. Therefore, the main interface of the user device may be extended.

1A-1C, the drawings illustrate a flow for enhancing touch using non-touch gestures, according to an embodiment of the present disclosure. In particular, FIGS. 1A-1C illustrate an example for adjusting focus of a touch target on a camera using a non-touch gesture.

In FIG. 1A, a user device 102, which may be a camera or has a camera as part of a user device, includes a screen or view finder for viewing a target item. In this case, the target item appears to be out of focus. The user may touch the screen of the user device 102 using an object such as a thumb at a point indicated by the reference numeral 101 on the target item to be operated.

In Figure 1B, the user performs an off-screen gesture to act on the target item. As can be observed in FIG. 1B, the user's fingers are laterally offset from the device's display, and thus may be detected as performing an off-screen gesture. In one example, to focus the target item, the user may, while keeping his thumb at point 101 on the target item, on a plane substantially parallel to the screen of the user device 102, reference numeral 104 ), you can also rotate your hand to focus the target item. The non-touch gesture is performed off-screen so that one or more sensors of the user device 102 (eg, ultrasound technology sensors, etc.) may capture off-screen gestures. The user may perform a rotation non-touch gesture until the user thinks that the target item is properly focused. In one embodiment, the user may touch anywhere on the screen of the user device 102 to change focus, or, in other embodiments, the user may have a specific object to improve autofocus. You can also touch the image.

[0032] In one embodiment, the user may be positioned close to the edge of the camera, such as on the right edge or on the screen, so that the user may focus using non-touch gestures around the side or corner of the display. You can also touch a button, area, or icon near the corner of. In this way, the display is not obstructed from the view.

In FIG. 1C, the user may release a touch from the user device 102 (eg, disengage his finger from the point 101) to disengage the target item. have.

[0034] In various embodiments, both gesture input-sensing data and touch input-sensing data may be interrupted upon releasing a touch from the user device, such as releasing a touch from the screen or display of the user device. .

Hence, embodiments herein may provide a clear way to engage and disengage commands on a user device.

[0036] Referring now to FIGS. 2A-2B, the drawings illustrate a flow for enhancing touch using non-touch off-screen gestures, according to another embodiment of the present disclosure. In particular, FIGS. 2A-2B illustrate an example of a shortcut for refreshing or scrolling a list or page on a screen of a user device.

In FIG. 2A, the user may press and hold a point 201 corresponding to the target item on the screen of the user device 202 with a finger. In Fig. 2B, the user is a non-touch off-screen gesture of moving the hand downward, as indicated by the reference numeral 207, centering on the finger pressing the target item at, for example, point 201 You can also do In this way, the list or page on the user device may be easily refreshed or scrolled, eg, without affecting other items on the screen. A non-touch gesture is a plane that is substantially the same as the screen of the user device (i.e., a plane that is substantially parallel), regardless of the proximity or distance in the vertical plane (i.e. It should be noted that it may be performed off-screen according to ).

[0038] In various embodiments, certain non-touch gestures may correspond to certain commands. In the example of Figs. 2A-2B, it may correspond to a downward motion of a hand as indicated by reference numeral 207, and a "refresh" command such as for refreshing a page. In other embodiments, the combination of touch and non-touch gesture inputs illustrated in FIGS. 2A-2B may be used as a shortcut to shuffle media. Such commands may be embedded in some embodiments, and a lookup table may define which gestures correspond to which commands.

[0039] Referring now to FIGS. 3A-3C, the drawings illustrate a flow for enhancing a touch using non-touch gestures, according to another embodiment of the present disclosure. In particular, FIGS. 3A-3C illustrate an example for organizing items such as applications in a target destination such as a file.

In FIG. 3A, a user may press and hold with a finger a point 301 corresponding to a target destination for one or more collected items on the screen of the user device 302.

[0041] In FIG. 3B, a user may begin to select one or more items to be collected and placed at a target destination. In this regard, while holding down with a finger at point 301, the user may select one or more items, such as items 302B, using other fingers of the same hand. Alternatively, while holding down with a finger at point 301, the user may select one or more items using fingers from another hand.

[0042] In FIG. 3C, once the user has finished selecting one or more items to be collected, the user has selected one or more toward the target destination indicated by the long-pressed finger at point 301 A gesture, such as a sweeping gesture (eg, as indicated by arrow 316), may be performed to move the collected items of.

[0043] Such one or more items are organized to a target destination, ie the items are organized into an appropriate file or folder.

According to one or more embodiments of the present disclosure, various techniques may be used to enhance touch inputs using non-touch off-screen gestures. In that regard, any gesture detection techniques may be combined with touch input techniques. Such techniques include, for example, Hand touch onscreen plus ultrasonic gesture detection, hand touch onscreen plus vision-based gesture detection (e.g., front and/or rear camera, and /Or using one or more cameras mounted on the side or edge of the device), ultrasonic pen touch on-screen plus hand vision detection (ultrasound, vision-based, depth sensor, camera-based, IR Etc.), hand touch on-screen plus ultrasonic pen gesture detection, electromyography (EMG), and/or any other suitable techniques. In this manner, gestures may be detected in front of the device, behind the device, and/or on one or more sides of the device. In some embodiments, ends may be identified and/or tracked to determine a pose or motion. For example, poses or motions of various fingers may be identified, and also specific gesture models may be used. Touch screen technology, such as capacitive touch screens and/or projective capacitance technology, may be used to detect touch inputs from a user. It should be noted that in various embodiments, touching on the screen may not be required. For example, touch inputs may be detected from the user touching the edge or even just touching the edge of the device. For example, the user may rotate his hand, e.g. to control the volume, which is detected using, for example, touch sensors located on the side, top or bottom of the device or using electric field sensors. It could be.

[0045] Referring now to FIG. 4, a flow chart illustrates a method of enhancing touch inputs using non-touch gestures, according to an embodiment of the present disclosure. It should be noted that the method illustrated in FIG. 4 may be implemented by the system illustrated in FIG. 5 according to an embodiment.

[0046] At block 402, a system such as system 1500 illustrated in FIG. 5 may simultaneously capture touch input data on the screen of the user device and non-touch gesture input data away from the screen of the user device. have. Touch input data and non-touch gesture input data may be captured, for example, through the input component 1516 illustrated in FIG. 5. In that regard, as described above with respect to various embodiments including the embodiments illustrated in FIGS. 1-3, the user, at the same time, touches the target item on the screen or display of the user device, and It is also possible to use non-touch gestures (using the same hand or another hand) to adjust the aspect. For example, a user may select a target item on the camera, such as an image to be focused, by touching the screen of the camera or an image on a viewer. At the same time, while touching the image on the screen of the camera, the user may perform an off-screen non-touch gesture, such as a hand motion or pose, to adjust the focus of the image. In that regard, the off-screen non-touch gesture may be performed, for example, by hand motion or pose in a plane substantially parallel to the screen of the user device. Further, the off-screen non-touch gesture may be captured by one or more sensors located on the user device, eg, on the sides, top, bottom, or rear of the user device. The sensor(s) may include various gesture detection techniques such as ultrasound techniques, IR, and the like.

At block 404, the system may determine an input command based at least in part on a combination of simultaneously captured touch input data and non-touch gesture input data. In block 406, an operation of the user device may be triggered based on the determined input command. Such determination of the input command and the effect of the input command on the operation of the user device may be performed, for example, by the processing component 1504 illustrated in FIG. 5. In one embodiment, the input command may act on the selected target item without acting on any other-non-selected items.

In various embodiments, the input command may be determined to be completed when the touch input data is disengaged. For example, in the camera focus embodiment illustrated in FIGS. 1A-1C, once focusing is achieved, the user may release a touch of the target item to disengage from the target item.

In accordance with one or more embodiments of the present disclosure, on-screen touch detection may enable off-screen object (eg, finger) detection. Off-screen object detection may then be used for operation of the control. For example, rotating the detected finger off-screen may be used to control the settings or actions of the application. On-screen touch detection or touch input may be used with an on-off switch, but non-touch gestures may be motion control. That is, touch inputs may be anchor points for non-touch off-screen gestures.

[0050] According to one or more embodiments of the present disclosure, the terms “gesture input-sensing data”, “non-touch gestures” or “gestures” are generally used in addition to touch-screen interactions. It should be noted that off-screen non-touch gestures or interactions captured by user sensors of the device may include, for example, non-touch hand motions or poses. Therefore, in embodiments herein, non-touch gestures may include detecting or recognizing a user's interactive input, such as a hand pose. In various embodiments, “gesture input-sensing data” refers to at least one or more points or areas of the device (e.g., sides, areas, in addition to touch sensors), for interpreting user actions as control input. It may include the use of data from sensors located at the top, bottom, rear), or from other interactions performed on or directly on the touch screen. That is, gesture input data may involve the use of sensors or techniques to capture the user's off-screen gestures without touching the user device.

[0051] In general, user devices are, for example, ultrasonic gesture detection sensors, cameras (eg, visible light cameras, range imaging cameras such as time-of-flight cameras) , Gesture recognition, which may include image or video capturing components, such as structured optical cameras, stereo cameras, etc.), hand vision detection (ultrasound, vision-based, IR, etc.) sensors, ultrasonic pen gesture detection sensors, etc. It may include or may be part of any device including gesture input data sensors. That is, devices may have ultrasound, IR, vision-based gesture capabilities, etc. to capture the user's off-screen gestures (ie, non-touch gestures such as hand poses) without touching the device. .

In various embodiments, the touch input may act on the operation of other sensors. For example, a touch input may turn on other sensors, or may increase the duty cycle of such other devices, for example. In addition, touch input may cause adjustments to gestures other sensors are configured to detect. For example, instead of being sensitive to the engagement pose to initiate gesture control, the touch input warns the device to search for gestures such as swipes, sweeps, poses, etc. other than the engagement pose. You may. In some embodiments, this may be made effective by loading one or more gesture models, increasing the size or range of a gesture vocabulary, and/or in any of a number of other ways.

[0053] With respect to capturing off-screen non-touch gestures according to one or more embodiments, movement of the user beyond the surface of the user device (eg, beyond the screen) or The orientation of the object (eg, one or more fingers, hands, pen, etc.) may be substantially parallel to the surface (eg, screen or display) of the user device. In various examples, motion of an object parallel to the surface of the device may include a sweeping motion beyond the screen or display boundaries, thumb rotation, etc. on the same plane as the display or screen of the device, and in the z-direction. Although motion may be considered in some embodiments, it may or may not be the only considered motion, in some embodiments. In that regard, the z-direction movement may be, for example, motion in a direction towards or away from the screen or display, for example to increase or decrease the proximity to the display. That is, the motion of the object may be performed outside the perimeter of the screen and aligned with the screen. In some embodiments, gestures may be detected when parallel to the screen (eg, when using a tablet or a larger form factor).

Referring now to FIG. 5, a block diagram of a system for implementing a device according to an embodiment of the present disclosure is illustrated.

It will be appreciated that the methods and systems described herein may be implemented by or included in a wide variety of electronic systems and devices. For example, system 1500 may be used to implement any type of device, including wireless or wired devices such as mobile devices, smart phones, personal digital assistants (PDAs), tablets, laptops, personal computers, TVs, etc. May be. Other example electronic systems, such as a music player, video player, communication device, network server, etc., may also be configured in accordance with the present disclosure. In addition, other electronic systems may be, for example, a wearable device mounted on the user's wrist (which may detect gestures or other control inputs and then pass them to another device for processing). It may also include.

System 1500 may be suitable for implementing embodiments of the present disclosure including user devices 102, 202, and 302 illustrated in FIGS. 1, 2, and 3, respectively. System 1500, such as part of a device, such as a smart phone, tablet, personal computer, and/or network server, includes a bus 1502 or other communication mechanism for communicating information, and includes a bus 1502 or other communication. The mechanisms include a processing component 1504 (e.g., a processor, micro-controller, digital signal processor (DSP), etc.), a system memory component 1506 (e.g., RAM), a static storage component (e.g. 1508) (e.g., ROM), network interface component 1512, display component 1514 (or alternatively, an interface to an external display), input component 1516 (e.g., keypad or keyboard, touch Screen, an interactive input component such as gesture recognition, etc.), and a cursor control component 1518 (eg, a mouse pad), and subsystems and components including one or more of them. In various embodiments, input component 1516 may implement one or more gesture detection techniques, which may be combined with touch input techniques. As mentioned above, such techniques include, for example, hand touch on-screen plus ultrasonic gesture detection, hand touch on-screen plus vision-based gesture detection (e.g., front and/or rear camera, and/or side of the device. Or using one or more cameras mounted on the edge), ultrasonic pen touch on-screen plus hand vision detection (ultrasound, vision-based, depth sensor, camera-based, IR, etc.), hand touch on-screen plus ultrasonic pen Gesture detection, electromyography (EMG), projected capacitive, electric field sensors, and/or any other suitable techniques may be included.

[0057] In accordance with embodiments of the present disclosure, system 1500 is by processing component 1504 that executes one or more sequences of one or more instructions included in system memory component 1506. Perform specific actions. These instructions may be read from other computer-readable media, such as static storage component 1508 to system memory component 1506. These may contain instructions to enhance interactive inputs via a simultaneous multimode sensor scheme. In other embodiments, a hard-wired circuit may be used instead of or in combination with software instructions for implementation of one or more embodiments of the present disclosure.

Logic may be encoded in a computer-readable medium, and computer-readable medium may refer to any medium that participates in providing instructions to processing component 1504 for execution. Such media may take many forms including, but not limited to, non-volatile media, volatile media, and transmission media. In various implementations, volatile media include dynamic memory, such as system memory component 1506, and transmission media include coaxial cables, copper wire, and optical fibers, including wires including bus 1502. In one embodiment, the transmission media may take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications. Some common types of computer readable media are, for example, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, carrier, or any other medium (from any other medium, computer readable Adapted to). Computer-readable media may be non-transitory.

In various embodiments of the present disclosure, execution of instruction sequences for carrying out the present disclosure may be performed by system 1500. In various other embodiments, a plurality of systems 1500 coupled by a communication link 1520 (e.g., WiFi, or various other wired or wireless networks) are coordinated with each other to carry out the present disclosure. Command sequences may be performed. System 1500 receives, via communication link 1520 and network interface component 1512, inputs, messages, data, information and instructions including one or more programs (i.e., application code). And may be extended. Received program code as received and/or stored in disk drive component 1510 or some other non-volatile storage component for execution may be executed by processing component 1504.

[0060] As those skilled in the art will recognize by now, and depending on the particular application at hand, without departing from the spirit and scope of the present disclosure, many modifications, substitutions and modifications may be made to the materials, devices of the present disclosure. , Configurations, and methods of use of devices. In this respect, the scope of the present disclosure should not be limited to the scope of the specific embodiments illustrated and described herein, as the specific embodiments are only through some examples of them, rather than the specific embodiments. As they should fully correspond to the scope of the appended claims and their functional equivalents.

Claims (31)

As a method for detecting interactive inputs,
Capturing touch input data on a screen of a user device, wherein the touch input data represents a touch performed on the screen of the user device;
Determining an input command based on the touch input data;
Affecting an operation of the user device based on the touch input data, the operation affecting the display of content on the screen of the user device;
Capturing non-touch gesture input data-the gesture input data represents an off-screen gesture performed in an area offset from the screen of the user device, And the gesture input data is related to the determined input command; And
And further acting on the operation of the user device based on the determined input command and the gesture input data. The method of claim 1,
The touch input data identifies a target item on the screen of the user device, and
Wherein the input command comprises adjusting an aspect of the target item based on the gesture input data. The method of claim 2,
Wherein the input command includes an avriable adjustment of the target item, and the variable adjustment is determined from the gesture input data. The method of claim 1,
The step of capturing the touch input data,
Receiving a touch on a desired target item to be operated on a screen of the user device from a user; And
And determining that the target item has been disengaged by detecting a release of a touch on the screen of the user device. The method of claim 1,
The method for detecting interactive inputs, wherein capturing the non-touch gesture input data comprises detecting a position and movement of an object. The method of claim 5,
The motion of the object further comprises motion in a plane substantially the same as the screen of the user device. The method of claim 1,
Capturing the non-touch gesture input data includes one or more sensors adapted to detect an object beyond the surface of the user device via ultrasound techniques, image or video capturing techniques, or IR techniques. A method for detecting interactive inputs, comprising using. The method of claim 1,
Wherein the determined input command comprises one of a plurality of different types of inputs indicating selection of a target item on the screen of the user device. The method of claim 8,
The different types of inputs are generated by a right-mouse click (RMC) generated by a first pose of a hand touching the screen, or a second pose of a hand touching the screen. A method for detecting interactive inputs, further comprising a generated left-mouse click (LMC) or alternate click, wherein the first pose is different from the second pose. The method of claim 1,
Wherein capturing the non-touch gesture input data comprises capturing a hand pose or hand motion. As a system,
A display configured to display one or more images;
One or more sensors configured to detect touch input data in the display;
One or more sensors configured to detect non-touch gesture input data representing an off-screen gesture performed in an area offset from the display; And
Including one or more processors,
The one or more processors are:
Capturing the touch input data, wherein the touch input data represents a touch performed on the display;
Determine an input command based on the touch input data;
Act on an operation of the system based on the touch input data, the operation affecting the content on the display;
Capturing the non-touch gesture input data, the gesture input data being associated with the determined input command; And
To further act on the operation of the system based on the determined input command and the gesture input data
Consisting of a system. The method of claim 11,
The touch input data identifies a target item on the display, and
The input command includes adjusting an aspect of the target item based on the gesture input data. The method of claim 12,
Wherein the input command includes a variable adjustment of the target item, and the variable adjustment is determined from the gesture input data. The method of claim 11,
The processor is further configured to determine that the target item has been disengaged by receiving a touch from a user on the display on a desired target item, and detecting a release of the touch on the display. The method of claim 11,
The one or more sensors configured to detect the non-touch gesture input data are further configured to capture the position and movement of an object. The method of claim 15,
The motion of the object further comprises motion in substantially the same plane as the display. The method of claim 16,
The one or more sensors configured to detect the non-touch gesture input data comprise ultrasonic sensors, image or video capturing sensors, or IR sensors adapted to capture the non-touch gesture input data. The method of claim 11,
Wherein the determined input command comprises one of a plurality of different types of inputs indicating selection of a target item on the display. The method of claim 18,
The different types of inputs may be a right mouse click (RMC) generated by a first pose of a hand touching the display, or a left mouse click (LMC) generated by a second pose of a hand touching the display, or Further comprising a replacement click, wherein the first pose is different from the second pose. The method of claim 11,
The one or more sensors configured to detect the non-touch gesture input data are configured to capture a hand pose or hand motion. An apparatus for detecting interactive inputs, comprising:
Means for capturing touch input data on a screen of a user device, the touch input data representing a touch performed on the screen of the user device;
Means for determining an input command based on the touch input data;
Means for acting on an operation of the user device based on the touch input data, the operation affecting the display of content on the screen of the user device;
Means for capturing non-touch gesture input data, wherein the gesture input data represents an off-screen gesture performed in an area offset from the screen of the user device, and the gesture input data is associated with the determined input command -; And
And means for further acting on the operation of the user device based on the determined input command and the gesture input data. The method of claim 21,
The touch input data identifies a target item on the screen of the user device, and
Wherein the input command comprises adjusting an aspect of the target item based on the gesture input data. The method of claim 22,
Wherein the input command includes a variable adjustment of the target item, and the variable adjustment is determined from the gesture input data. The method of claim 21,
The means for capturing the touch input data comprises means for receiving, from a user on a screen of the user device, a touch on a desired target item to be acted upon; And
The apparatus further comprises means for determining that the target item has been disengaged by detecting the release of a touch on the screen of the user device. The method of claim 21,
The device for detecting interactive inputs, wherein the means for capturing non-touch gesture input data comprises means for detecting a position and movement of an object. The method of claim 25,
The apparatus for detecting interactive inputs, wherein the movement of the object further comprises movement in a plane substantially the same as the screen of the user device. The method of claim 21,
The means for capturing the non-touch gesture input data comprises one or more sensors adapted to detect an object beyond the surface of the device via ultrasonic techniques, image or video capturing techniques, or IR techniques. , Apparatus for detecting interactive inputs. The method of claim 21,
Wherein the determined input command comprises one of a plurality of different types of inputs indicating selection of a target item on the screen of the user device. The method of claim 28,
The different types of inputs may be a right mouse click (RMC) generated by a first pose of a hand touching the screen, or a left mouse click (LMC) generated by a second pose of a hand touching the screen, or An apparatus for detecting interactive inputs, further comprising an alternate click, wherein the first pose is different from the second pose. The method of claim 21,
The apparatus for detecting interactive inputs, wherein the means for capturing non-touch gesture input data comprises means for capturing a hand pose or hand motion. A non-transitory computer-readable storage medium storing computer-readable instructions, comprising:
The computer-readable instructions, when executed by a processor, cause the processor to:
Capturing touch input data on the screen of the user device, wherein the touch input data represents a touch performed on the screen of the user device;
Determine an input command based on the touch input data;
Act on an operation of the user device based on the touch input data, the operation affecting the display of content on the screen of the user device;
Capturing non-touch gesture input data, wherein the gesture input data represents an off-screen gesture performed in an area offset from the screen of the user device, and the gesture input data is associated with the determined input command; And
A non-transitory computer-readable storage medium for further acting on the operation of the user device based on the determined input command and the gesture input data.

Images