TW202324041A - User interactions with remote devices - Google Patents

Abstract

Systems, methods, and non-transitory media are provided for presenting information associated with at least one input option. An example method can include receiving data identifying one or more input options associated with a first device in a scene; determining, including using at least one memory, information relevant to at least one of the scene, the first device, and a user associated with a second device; and based on the one or more input options and the information, output user guidance data corresponding to an input option for which relevant context information has been determined.

Description

Interact with the user of the remote device

This case is generally about interactions with remote devices. For example, aspects of this case include filtering and/or suggesting virtual content for user interaction with a remote device.

Extended reality technology can be used to present virtual content to users, and/or can combine real and virtual environments from the real world to provide users with an extended reality experience. The term extended reality can encompass virtual reality, augmented reality, mixed reality, etc. Each of these forms of extended reality allows users to experience or interact with immersive virtual environments or content. For example, an extended reality experience may allow a user to interact with a real or physical environment that is augmented or augmented with virtual content.

Extended reality technology can be implemented to enhance user experience in various environments such as entertainment, healthcare, retail, education, social media, and more.

Systems, devices, methods and computers for revealing user interaction data for determining remote device interactions (e.g., interactions between an extended reality (XR) device and one or more remote devices, such as Internet of Things network devices) Readable media. According to at least one example, a method for presenting information associated with at least one input option is provided. The method includes the steps of: receiving data identifying one or more input options associated with a device in a scene; determining (including using at least one memory) at least one of the scene, the device, and the user associated with the electronic device an associated message; and based on the one or more input options and information, outputting user guidance data corresponding to the input option for which the associated context information has been determined.

In another example, an apparatus for presenting information associated with at least one input option is provided that includes at least one memory and at least one processor coupled to the at least one memory (e.g., in circuitry implemented in). The at least one processor is configured and may: receive data identifying one or more input options associated with a device in a scene; determine (including using at least one memory) information associated with a scene, a device, and an electronic device; at least one of the user-related information; and based on the one or more input options and information, outputting user guidance data corresponding to the input options for which relevant contextual information has been determined.

In another example, a non-transitory computer readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: receive recognition and scene information about one or more input options associated with a device in an electronic device; determining (including using at least one memory) information related to at least one of a scene, a device, and a user associated with an electronic device; and based on one or more input options and information, and output user guidance data corresponding to the input options for which relevant contextual information has been determined.

In another example, an apparatus for presenting information associated with at least one input option is provided. The apparatus includes means for: receiving data identifying one or more input options associated with a device in a scene; determining (including using at least one memory) a usage associated with the scene, the device, and the electronic device information related to at least one of them; and based on the one or more input options and information, outputting user guidance data corresponding to the input options for which relevant contextual information has been determined.

In some aspects, the above-described methods, non-transitory computer-readable media, and apparatus may include: predicting a user interaction with the device based on the information; and based on the one or more input options and the predicted user interaction, presenting a corresponding Enter user guidance information for the option.

In some examples, user guidance data may include user input elements associated with input options, virtual overlays on physical objects associated with input options, and/or prompts indicating how to provide input associated with input options at least one of the

In some examples, the device may include a connected device with network communication capabilities, and the above-described methods, non-transitory computer-readable media, and apparatus may include: determining, based on the information and one or more input options, an Gestures for user interaction; and presenting user guidance information. In some examples, the predicted user interaction may include predicted user input to the device. In some cases, user guidance data may include indications of gestures that, when detected, cause actual user input at the device.

In some aspects, presenting the user guidance material can include rendering a virtual overlay at a display associated with the electronic device, the virtual overlay configured to appear to be located on a surface of the device. In some examples, the virtual overlay can include user interface elements associated with input options. In some cases, the user interface element may include at least one of a virtual user input object associated with the input option and a visual indication of a physical control object on the device configured to receive input corresponding to the input option.

In some examples, the information includes at least one of the user's eye gaze and the user's gesture, and the methods, non-transitory computer-readable media, and apparatus described above may include: at least one of the gestures to predict user interaction with the device; after presenting the user guidance material, detecting actual user input associated with the input options, the actual user input representing the predicted user interaction; and sending the device Commands corresponding to actual user input associated with input options are transmitted.

In some examples, outputting user guidance data corresponding to the input options may include displaying the user guidance data. In some examples, outputting user guidance data corresponding to the input options may include outputting audio data representing the user guidance data.

In some examples, outputting user guidance data corresponding to the input options may include displaying the user guidance data; and outputting audio data associated with the displayed user guidance data.

In some aspects, the methods, non-transitory computer-readable media, and apparatus described above can include receiving, from a device, data identifying one or more input options associated with the device. In some aspects, the methods, non-transitory computer-readable media, and apparatus described above can include receiving from a server data identifying one or more input options associated with a device.

In some cases, the device has no external user interface for receiving one or more user inputs.

In some aspects, the methods, non-transitory computer-readable media, and apparatus described above can include refraining from presenting additional user guidance material associated with the device based on the information.

In some aspects, the above-described methods, non-transitory computer-readable media, and apparatus may include: after presenting the user-guidance material, obtaining user input associated with input options; and transmitting to the device a corresponding instructions. In some cases, the instructions may be configured to control one or more operations of the device.

In some aspects, the device includes a camera, a mobile device (e.g., a mobile phone or "smartphone" or other mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality ( AR) devices or Mixed Reality (MR) devices), PCs, laptops, server computers or any other device. In some aspects, the device includes one or more cameras for capturing one or more images. In some aspects, the device also includes a display for displaying one or more images, notifications, and/or other displayable data. In some aspects, the apparatus described above may include one or more sensors.

This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to the entire specification of this patent, any or all drawings, and each claim in due part.

The foregoing, as well as other features and embodiments, will become more apparent with reference to the following specification, claims and drawings.

Certain aspects and examples of the present case are provided below. It will be apparent to those skilled in the art that some of the described aspects and embodiments can be applied independently and some of them can be applied in combination. In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of the embodiments of the present case. It may be evident, however, that various embodiments may be practiced without these specific details. The figures and description are not intended to be limiting.

The ensuing description provides exemplary embodiments only, and is not intended to limit the scope, applicability, or configuration of the present case. Rather, the ensuing description of the example embodiments will provide those skilled in the art with an enabling description for implementing the example embodiments. It should be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the invention as set forth in the appended claims.

Users often interact with different devices that provide certain functionality of interest to the user. For example, users can interact with smart devices (e.g., Internet of Things (IoT) or other connected devices), mobile devices, control devices (e.g., remote controls for TVs, appliances, speakers, etc.), system consoles, appliances, etc. . In various illustrative examples, the user can interact with the Internet-connected TV to manage viewing content or change the power setting of the Internet-connected TV, interact with the Internet-connected light bulb to control the light emitted by the Internet-connected light bulb or operation of a network-connected light bulb, interacting with a network-connected router to configure network-connected router operations and settings, interacting with a network-connected thermostat to control the temperature or configure settings of a network-connected thermostat, and so on.

In some cases, a device may include a hardware user interface or may present (for example, by displaying the user interface using a display or other technology) a graphical user interface that a user may use to interact with the device. However, in some cases, it may be difficult for a user to interact with the device via the user interface of the device. In one example, the user interface may be inaccessible to the user, which prevents the user from interacting with (or makes it difficult to interact with) the user interface associated with the device. As another example, the content displayed by the user interface may be in a language that the user does not understand, may be too small for the user to recognize, or may be inconvenient for the user to use the user interface. In some instances, input options associated with a device (e.g., supported inputs, supported input methods, etc.) may not be readily understood or understood by the user (e.g., gestures or voice commands as opposed to user interface input, etc.) , which can make it difficult for the user to interact with the device. In other instances, the user interface may not have accessibility settings (or sufficient accessibility settings) to account for users with visual, speech, and/or hearing disabilities. In some cases, the device may not have outward/external controls visible or otherwise accessible to the user, and/or may not include a display presenting a user interface that the user may use to interact with the device. Furthermore, in many cases, a scene may include multiple remote devices such as, for example, connected light bulbs, televisions, connected plugs, connected speakers, and the like. In such a situation where there are multiple devices in the scene, it may be difficult for the user device to interact with and/or manage the interaction with a particular remote device among the multiple devices in the scene.

For example, a room (eg, kitchen, bedroom, office, living room, etc.) may have multiple devices with connectivity and/or interaction capabilities. In order to initiate and/or manage communication and/or interaction with one of the multiple devices, it may be difficult for the user device to identify a specific device from among multiple devices, manage and/or simplify user interaction with and/or related to that specific device data, manage content related to that particular device, etc. In some cases, a user device may obtain interaction data (eg, input options/capabilities, input, output, GUI, user interaction aids, etc.) from multiple devices in the scene. If the user device does not have sufficient knowledge and/or understanding of the scene and/or the current context, and/or does not receive clear instructions from the user, the user device may be overloaded with data (e.g., interaction data, device data, etc.) . In some instances, a user device may have difficulty managing content and/or interactions with one or more of the devices in the scene. However, the systems and techniques described herein may allow user devices to constrain, restrict, filter, declutter, etc., virtual content associated with remote devices in a scene based on contextual information. In some cases, using the systems and techniques described herein, a user device can refine the content that is processed and/or presented for communication/interaction with a remote device in a scene, and/or in a manner that is most adaptive/appropriate to the context (e.g. large, small, overlay, world locked, head or device locked, etc.) to render the content. In some examples, the user device can use contextual information to understand how to interact with a particular remote device in a scene, what interaction data and/or virtual content is relevant to that particular remote device, how to manage interactions with any remote device in a scene device interaction, etc.

As further described herein, in many cases, the systems and techniques described herein may allow devices to unify and/or simplify user interactions with remote devices in a scene. In the above cases, the device can better manage, unify, simplify and/or facilitate communication and/or interaction with remote devices in the scene. In some cases, a device may facilitate and/or support user interaction with one or more remote devices even under more challenging scenarios and/or conditions. To illustrate, it may be difficult for a user and/or a device associated with the user to interact with the device. For example, a user may experience a Struggling to interact with the TV remote in poor lighting conditions. As another example, a user may struggle to interact with a networked router or connected device (e.g., networked thermostat, light bulb, speaker, camera, appliance, switch, etc.) that has no external control, especially if the user cannot Access the user interface for interacting with network routers or IoT devices. As yet another example, if the console (or certain controls of the console) are out of reach of the user, or the user does not know which control to use for the desired operation/interaction, it may be difficult for the user to interact with the console ( such as vehicle or elevator consoles).

As described in greater detail herein, described herein are systems, apparatus, methods (also referred to as processes, and computer-readable media (collectively referred to herein as "Systems and Technologies")). In some instances, electronic devices can unify, simplify, and/or facilitate interaction with other devices, such as, for example, connected devices (e.g., network-connected devices), mobile devices, devices lacking outward/external controls, devices lacking displays and/or user interface devices, devices with certain characteristics that present one or more challenges to users wishing to interact with the device (e.g., interfaces in different languages, not recognized/understood by users, with limited devices with accessibility options, etc.), devices with controls/interfaces that are not accessible to the user, and/or any other device. In some examples, electronic devices configured to facilitate interaction with other devices may include smartphones, smart wearable devices (e.g., smart watches, smart earbuds, etc.), extended reality (XR) systems or devices (e.g., smart glasses , head-mounted displays (HMDs), etc.), etc. Although examples are described here using an XR system as an example of an electronic device that may implement the techniques described herein, other electronic devices (eg, mobile devices, smart wearable devices, etc.) may be used to perform the techniques.

In general, an XR system or device may provide virtual content to a user and/or may combine a real world or physical environment with a virtual environment (consisting of virtual content) to provide an XR experience to a user. The real-world environment may include real-world objects (also referred to as physical objects), such as books, people, vehicles, buildings, tables, chairs, and/or other real-world or physical objects. An XR system or device may facilitate interaction with different types of XR environments (eg, a user may use an XR system or device to interact with an XR environment). XR systems may include virtual reality (VR) systems that facilitate interaction with VR environments, augmented reality (AR) systems that facilitate interaction with AR environments, mixed reality (MR) systems that facilitate interaction with MR environments, and/or other XR systems. As used herein, the terms XR system and XR device are used interchangeably. Examples of XR systems or devices include HMDs, smart glasses (eg, internet-connected glasses that can communicate using a communication network), and the like.

AR is a technology that provides virtual or computer-generated content (referred to as AR content) over a user's view of a physical, real-world scene or environment. AR content may include virtual content such as video, images, graphical content, location data (eg, global positioning system (GPS) data or other location data), sound, any combination thereof, and/or other enhanced content. AR systems or devices are intended to augment (or augment) rather than replace a person's current perception of reality. For example, a user can see a real stationary or moving physical object through the AR device display (for example, the lens of AR glasses), but the user's visual perception of the physical object can be enhanced or enhanced in the following ways: through the object's Virtual images (e.g., a real-world car replaced by a DeLorean's virtual image), via AR content added to a physical object (e.g., virtual wings added to a live animal), via AR content displayed relative to a physical object (e.g., informative virtual content displayed near a sign on a building, a virtual coffee mug virtually anchored to (e.g., placed on) a table in the real world in one or more images, etc.), and/or Via displaying other types of AR content. Various types of AR systems can be used for gaming, entertainment, and/or other applications.

In some cases, two types of AR systems that may be used to provide AR content include video see-through (also known as video-through) displays and optical see-through displays. Video see-through and optical see-through displays can be used to enhance a user's visual perception of the real world or physical objects. In a video see-through system, a live video of a real world scene is displayed (eg, including one or more objects augmented or augmented on the live video). A video see-through system may be implemented using a mobile device (eg, video on a mobile phone display), an HMD, or other suitable device that can display video and computer-generated objects on top of the video.

Optical see-through systems with AR features can display AR content directly on a view of a real-world scene (eg, without displaying video content of a real-world scene). For example, a user may view physical objects in a real world scene via a display (e.g., glasses or lenses), and the AR system may display AR content on the display (e.g., projected or otherwise displayed) to provide the user with insight into a or enhanced visual perception of multiple real-world objects. An example of an optical see-through AR system or device is AR glasses, an HMD, another AR headset, or other similar device, which may include a lens or glasses in front of each eye (or a single lens or glasses on both eyes ) to allow the user to directly see the real world scene with the physical object, and also allow the enhanced image of the object or additional AR content to be projected on the display to enhance the user's visual perception of the real world scene.

VR provides a fully immersive experience in a three-dimensional computer-generated VR environment or a video that illustrates a virtual version of a real-world environment. VR environments can be interacted with in ways that appear to be real or physical. When the user experiencing the VR environment moves in the real world, the image rendered in the virtual environment also changes, giving the user a feeling that the user is moving in the VR environment. For example, the user may turn left or right, look up or down, and/or move forward or backward, thereby changing the user's viewpoint of the VR environment. The VR content presented to the user can change accordingly, making the user's experience as flawless as in the real world. In some cases, VR content can include VR video, which can be captured and rendered at very high quality, potentially providing a truly immersive virtual reality experience. Virtual reality applications may include gaming, training, education, sports video, online shopping, and more. VR content may be rendered and displayed using a VR system or device, such as a VR HMD or other VR headset, which completely covers the user's eyes during the VR experience.

MR technology can combine various aspects of VR and AR to provide users with an immersive experience. For example, in an MR environment, the real world and computer-generated objects can interact (eg, real people can interact with virtual people as if the virtual people were real people).

In some cases, the XR system can track parts of the user (eg, the user's hand and/or fingertips) to allow the user to interact with items of virtual content. An XR system such as smart glasses or an HMD may implement a camera and/or one or more sensors to track the location of the XR system and other objects within the physical environment in which the XR system is located. The XR system can use this tracking information to provide a real XR experience for the user of the XR system. For example, an XR system may allow a user to experience or interact with an immersive virtual environment or content. To provide a realistic XR experience, some XR systems or devices can combine virtual content with the real world. In some cases, an XR system or device can match the relative pose and movement of objects and devices. For example, an XR system may use tracking information to calculate relative poses of devices, objects, and/or maps of the real-world environment in order to match relative positions and movements of devices, objects, and/or other parts of the real-world environment. Using gestures and movements of one or more devices, objects, and/or other parts of the real-world environment, the XR system can anchor content to the real-world environment in a manner that appears realistic to a user of the XR system. Relative pose information can be used to match virtual content to the user's perceived motion and spatio-temporal state of devices, objects, and other parts of the real-world environment.

In some examples, the XR system can be used to provide user guidance material for interacting with one or more other devices (such as one or more connected devices, remote controls, consoles, mobile devices, etc.). According to the systems and techniques described herein, XR systems can be utilized to enable more intuitive and natural content and/or interaction with other devices. In some examples, the XR system can detect other devices in the scene, and facilitate and/or manage interactions with other devices. In some cases, the XR system may have previously established connections (eg, paired, etc.) with other devices, which may allow the XR system to detect other devices in the scene. In other examples, the XR system can maintain a map of the scene including other devices located in the scene, which the XR system can use to detect other devices when the XR system is in the scene. In some examples, the XR system may use one or more sensors, such as image sensors, audio sensors, radar sensors, LIDAR sensors, etc., which the XR system may use to Sense other devices in the scene. In some cases, the XR system may use contextual information associated with the scene to determine that other devices are in the scene. For example, the XR system may include contextual information about the scene and/or other devices that indicate to the XR device that the other devices are present in the scene. When the XR system is in the scene, the XR system can determine that other devices are in the scene based on contextual information. The XR system can obtain information from other devices detected in the scene to facilitate and manage interactions with other devices.

For example, the XR system may obtain input data from other devices detected in the scene indicating or identifying one or more input options for the other devices and/or context associated with the scene, the other devices, and/or the XR system Information. The XR system can process input data and/or contextual information and present a user interface, virtual content, and/or input options to a user using (e.g., wearing) the XR system for interacting with the device (e.g., controlling, storing access content, access status information, access output, etc.). In some cases, the XR system can also be used to control other devices in the scene based on input data and/or contextual information, as further described herein.

In some examples, the XR system may obtain or receive input indicative of one or more input options from other devices, from a server (e.g., a cloud-based server related to the operation of other devices), and/or from another source material. Input profiles may indicate some or all of the input options available for interacting with other devices, such as type of input (e.g., gesture-based, voice-based, touch-based, etc.), functionality based on a particular input (e.g., temperature, etc.), and other information. In some instances, if the device does not transmit any input data to the XR system (e.g., after the XR system sends a request to the device), the XR system may determine or infer that the device does not have a connection with the XR system (e.g., via a wireless network). ability to communicate. In such instances, the XR system can present instructions or other information (eg, highlighting a particular button on the device) to help the user decide how to interact with the device.

XR systems can use contextual information to determine content, input options, user interface and/or modalities for output to the user for interaction with other devices. In some examples, the XR system can be implemented locally and/or from one or more remote sources (e.g., server, cloud, Internet, other device, one or more sensors, etc.) (e.g., using One or more sensors of the XR system, such as one or more cameras, one or more inertial measurement units (IMUs), etc., obtain or receive contextual information. Contextual information may relate to other devices with which a user may interact with the XR System, the scene or environment in which the XR System and/or other devices are located, the user of the XR System attempting to interact with the other device, and/or any given Other contexts at the point in time. For example, contextual information may include expected user interactions with other devices, one or more actions of the user within a scene, characteristic or personal information associated with the user, historical information associated with the user and other devices ( For example, the user's past use of the device, etc.), the user interface capabilities of other devices (for example, whether it has outward/external controls that are visible or otherwise accessible to the user), the information associated with other devices information (eg, how far other devices are from the XR system), information associated with the scene (eg, lighting, noise, etc.), and/or other information.

The contextual information may provide the XR system with a contextual awareness of the situation/context associated with the user, the XR system, other device(s), the scene, etc. Using contextual information and data identifying input options associated with other devices, the XR system can output (e.g., present, provide, generate, etc.) user interaction data corresponding to one or more input options that cause the user to or to interact with other devices. In some cases, the XR system may present visual content/material corresponding to one or more input options. For example, user interaction data may include one or more user interface elements associated with an input option, prompts indicating how to provide input associated with an input option, and/or other data. In some cases, the XR system may alternatively or additionally output non-visual material corresponding to one or more input options. For example, an XR system may output tactile and/or audio information, such as audio prompts or instructions corresponding to one or more input options.

Contextual information may enable an XR system to output (e.g., visual virtual content, audio content, haptic feedback, etc.) content that is contextually appropriate given the situation/context associated with the user, XR system, other device, scene , and/or otherwise facilitate interaction with other devices. In some examples, the XR system can use contextual information to simplify user interaction and/or associated data/content. For example, in some cases other devices may have multiple input functions/capabilities. The XR system may use contextual information to filter input options associated with devices relevant to the XR system and/or the current context. The XR system can output filter/reduce number of input options to simplify user interaction, user interaction data/content, etc.

As mentioned above, the XR system can use the input data and contextual information to render a user interface and/or input options for the user to interact with other devices from the XR system. In some cases, the XR system can communicate with other devices to provide input/commands to the other devices based on user interaction with the user interface rendered by the XR system. For example, an XR system may display a user interface such that it appears to the user to be an overlay on or part of another device (e.g., controls, faces, displays, panels, etc.) to facilitate use with other devices interaction. The XR system can detect and interpret user interactions with the user interface to generate control commands and/or user interaction commands. For example, the XR system can detect user gestures and/or input via the device (e.g., via the XR system and/or the controller) and interpret/translate such user gestures and/or input into interactions with the user interface interactive. The XR system can then generate commands/instructions for controlling other devices based on the interpreted/translated user gestures and/or inputs. The XR system can use interaction with the overlay user interface to control other devices based on user input and/or accessing data/output from other devices. In some cases, in order to facilitate and/or improve user interaction with other devices, the XR system may locate and/or map other devices so that the XR system can accurately render the user interface relative to the other device or a portion of the other device .

In some examples, the XR system can present a user interface superimposed on the scene in a world-locked or screen-locked manner, so as to provide the user with user interaction guidance information and/or input options through the user interface. In some cases, the overlay may include one or more GUI elements with guidance information instructing the user how to interact with the one or more GUI elements.

Further details regarding the creation of virtual private spaces are provided herein with reference to the various figures. FIG. 1 is a diagram illustrating an example extended reality (XR) system 100 in accordance with some aspects of the present disclosure. The XR system 100 can run (or execute) XR applications and implement XR operations. In some instances, XR system 100 may perform tracking and positioning, mapping of the real world (eg, scene), and positioning and rendering of virtual content on display 109 as part of the XR experience. For example, the XR system 100 can generate a map of a scene in the real world (e.g., a three-dimensional (3D) map), track the pose (e.g., location and position) of the XR system 100 relative to the scene (e.g., relative to the 3D map of the scene), positioning and/or anchoring the virtual content at specific location(s) on the scene map, and rendering the virtual content on the display 109 such that the virtual content appears to be located on the scene map at which the virtual content is positioned and/or anchored The location in the scene corresponding to the specific location of the . The display 109 may include glass, a screen, one or more lenses, a projector, and/or other display mechanisms that allow the user to see the real world environment and also allow XR virtual content to be displayed thereon.

In this illustrative example, XR system 100 includes one or more image sensors 102, accelerometer 104, gyroscope 106, storage device 107, computing element 110, XR engine 120, input options engine 122, context management engine 123 , an image processing engine 124 and a rendering engine 126 . It should be noted that the elements 102-126 shown in FIG. 1 are non-limiting examples provided for purposes of illustration and explanation, and that other examples may include more, fewer, or different elements than those shown in FIG. 1 . For example, in some cases, XR system 100 may include one or more other sensors (e.g., one or more inertial measurement units (IMUs) in addition to accelerometer 104 and gyroscope 106, radar, light detection and ranging (LIDAR) sensors, audio sensors, etc.), one or more display devices, one or more other processing engines, one or more other hardware components, and/or one or more other software and/or hardware components. An exemplary architecture and exemplary hardware elements that may be implemented by the XR system 100 are further described below with reference to FIG. 9 .

Furthermore, for simplicity and explanation purposes, one or more image sensors 102 will be referred to herein as image sensors 102 (eg, in the singular). However, those of ordinary skill will recognize that XR system 100 may include a single image sensor or multiple image sensors. Furthermore, reference to any element of XR system 100 in singular or plural (eg, 102 - 126 ) should not be construed as limiting the number of such elements implemented by XR system 100 to one or more than one. For example, reference to accelerometer 104 in the singular should not be construed as limiting the number of accelerometers implemented by XR system 100 to one. Those of ordinary skill will recognize that for any of the elements 102-126 shown in FIG. 1, the XR system 100 may include only one such element or more than one such element.

The XR system 100 includes or is in communication (wired or wirelessly) with an input device 108 . Input device 108 may include any suitable input device, such as a touch screen, pen or other pointing device, keyboard, mouse, buttons or keys, a microphone for receiving voice commands, a gesture input device for receiving gesture commands, Any combination thereof and/or other input devices. In some cases, image sensor 102 may capture images that may be processed to interpret gesture commands.

XR system 100 may be part of or be implemented by a single computing device or multiple computing devices. In some examples, XR system 100 may be a device such as an extended reality head-mounted display (HMD) device, extended reality glasses (eg, augmented reality or AR glasses), a camera system (eg, digital camera, IP camera, video camera, security cameras, etc.), telephone systems (e.g., smartphones, cellular phones, conferencing systems, etc.), desktops, laptops or notebooks, tablets, set-top boxes, smart TVs, display devices, game consoles , a video streaming device, an IoT (Internet of Things) device and/or any other suitable electronic device(s) that is part of an electronic device (or devices).

In some implementations, one or more of image sensor 102, accelerometer 104, gyroscope 106, storage device 107, computing element 110, XR engine 120, input options engine 122, context management engine 123, image processing Engine 124 and rendering engine 126 may be part of the same computing device. For example, in some cases, one or more of image sensor 102, accelerometer 104, gyroscope 106, storage device 107, computing element 110, XR engine 120, input options engine 122, context management engine 123, image Processing engine 124 and rendering engine 126 may be integrated into an HMD, extended reality glasses, smartphone, laptop, tablet, gaming system, and/or any other computing device. However, in some implementations, one or more of image sensor 102, accelerometer 104, gyroscope 106, storage device 107, computing element 110, XR engine 120, input options engine 122, context management engine 123, graph Like processing engine 124 and rendering engine 126 may be part of two or more separate computing devices. For example, in some cases some of elements 102-126 may be part of or implemented by one computing device, while remaining elements may be part of or implemented by one or more other computing devices. device implementation.

Storage device 107 may be any storage device(s) for storing data. Additionally, storage device 107 may store data from any of the components of XR system 100 . For example, storage device 107 may store data from image sensor 102 (eg, image or video data), data from accelerometer 104 (eg, measurements), data from gyroscope 106 (eg, measurements ), data from computing element 110 (e.g., processing parameters, preferences, virtual content, rendered content, scene maps, tracking and positioning data, object detection data, privacy data, XR application data, face recognition data, occlusion data, etc. ), data from the XR engine 120, data from the input options engine 122, data from the context management engine 123, data from the image processing engine 124, and/or data from the rendering engine 126 (e.g., output frames) . In some examples, storage device 107 may include a buffer for storing frames for processing by computing element 110 .

One or more computing elements 110 may include a central processing unit (CPU) 112 , a graphics processing unit (GPU) 114 , a digital signal processor (DSP) 116 and/or an image signal processor (ISP) 118 . Computing element 110 may perform various operations such as image enhancement, computer vision, graphics rendering, extended reality (e.g., tracking, localization, pose estimation, mapping, content anchoring, content rendering, etc.), image/video processing, sensing processor processing, recognition (eg, text recognition, face recognition, object recognition, feature recognition, tracking or pattern recognition, scene recognition, occlusion detection, etc.), machine learning, filtering, and any of the various operations described herein. In this example, computing element 110 implements XR engine 120 , input options engine 122 , context management engine 123 , image processing engine 124 , and rendering engine 126 . In other examples, the computing element 110 may also implement one or more other processing engines.

The image sensor 102 may include any image and/or video sensor or capture device. In some examples, image sensor 102 may be part of a multi-camera assembly, such as a two-camera assembly, three-camera assembly, four-camera assembly, or other number of cameras. In some examples, image sensor 102 may include one or more visible light cameras (eg, configured to capture monochrome or color images, such as red-green-blue or RGB images), one or more infrared ( IR) camera and/or near infrared (NIR) camera, one or more depth sensors and/or other types of image sensor(s) or camera(s).

Image sensor 102 may capture image and/or video content (e.g., raw image and/or video data), which may then be generated by computing element 110, XR engine 120, input options engine 122, context management engine 123 , image processing engine 124 and/or rendering engine 126, as described herein. For example, image sensor 102 may capture image data, and may generate frames based on the image data, and/or may provide image data or frames to computing element 110, XR engine 120, input option engine 122 , the context management engine 123, the image processing engine 124 and/or the rendering engine 126 for processing. A frame may comprise a video frame or a still image of a video sequence. A frame may include an array of primitives representing a scene. For example, a frame could be a red-green-blue (RGB) frame with red, green, and blue components per picture element; chroma-blue), chroma-red, chroma-blue (YCbCr) frames; or any other suitable type of color or monochrome picture.

In some cases, image sensor 102 (and/or other cameras of XR system 100 ) may be configured to also capture depth information. For example, in some implementations, image sensor 102 (and/or other cameras) may include an RGB depth (RGB-D) camera. In some examples, XR system 100 may include one or more depth sensors (not shown) that are separate from image sensor 102 (and/or other cameras) and that may capture depth information. For example, such a depth sensor can obtain depth information independently of the image sensor 102 . In some examples, a depth sensor may be physically mounted at the same general location as image sensor 102 , but may operate at a different frequency or frame rate than image sensor 102 . In some examples, the depth sensor can take the form of a light source that can project a structured or textured light pattern onto one or more objects in the scene, the light pattern can include one or more narrow-band Light. Depth information can then be obtained by exploiting the geometric distortion of the projected pattern caused by the surface shape of the object. In one example, depth information may be obtained from a stereo sensor such as a combination of an infrared structured light projector and an infrared camera registered to the camera (eg, an RGB camera).

XR system 100 also includes one or more sensors other than image sensor 102 . The one or more sensors may include one or more accelerometers (eg, accelerometer 104 ), one or more gyroscopes (eg, gyroscope 106 ), and/or other sensors. One or more sensors may provide velocity, orientation, and/or other position-related information to computing element 110 . For example, accelerometer 104 may detect acceleration of XR system 100 and may generate an acceleration measurement based on the detected acceleration. In some cases, accelerometer 104 may provide one or more translation vectors (eg, up/down, left/right, forward/backward), which may be used to determine the position or attitude of XR system 100 . The gyroscope 106 can detect and measure the orientation and angular velocity of the XR system 100 . For example, gyroscope 106 may be used to measure pitch, roll, and yaw of XR system 100 . In some cases, gyroscope 106 may provide one or more rotation vectors (eg, pitch, yaw, roll). In some examples, image sensor 102 and/or XR engine 120 may use data obtained from accelerometer 104 (eg, one or more translation vectors) and/or gyroscope 106 (eg, one or more rotation vectors). to calculate the attitude of the XR system 100 . As mentioned above, in other examples, the XR system 100 may also include other sensors, such as inertial measurement units (IMUs), magnetometers, gaze and/or eye-tracking sensors (eg, eye-tracking cameras), machine vision sensors, smart scene sensors, voice recognition sensors, impact sensors, vibration sensors, position sensors, tilt sensors, etc.

In some cases, the one or more sensors may include at least one IMU. An IMU is an electronic device that measures specific force, angular velocity, and/or orientation of the XR system 100 using a combination of one or more accelerometers, one or more gyroscopes, and/or one or more magnetometers. In some examples, one or more sensors may output measurement information and/or Depth information obtained using one or more depth sensors of the XR system 100 .

The output of one or more sensors (e.g., accelerometer 104, gyroscope 106, one or more other types of IMUs, and/or other sensors) may be used by extended reality engine 120 to determine the pose of XR system 100 (also referred to as head pose) and/or the pose of the image sensor 102 (or other cameras of the XR system 100). In some cases, the pose of XR system 100 and the pose of image sensor 102 (or other camera) may be the same. The pose of the image sensor 102 refers to the position and orientation of the image sensor 102 relative to a frame of reference (eg, relative to the object 202 ). In some implementations, the camera pose can be determined for 6 degrees of freedom (6DOF), which refers to three translational components (e.g., which can be defined by X (horizontal), Y (vertical) and Z (depth) coordinates) and three angular components (for example, roll, pitch, and yaw relative to the same frame of reference).

In some cases, a device tracker (not shown) may use measurements from one or more sensors and image data from image sensor 102 to track the pose of XR system 100 (e.g., 6DOF pose ). For example, a device tracker can fuse visual data from imagery (e.g., using a visual tracking solution) with inertial data from measurements to determine how the XR system 100 behaves relative to the real world (e.g., a scene) and Map location and movement. As described below, in some examples, the device tracker may generate a three-dimensional (3D) map of a scene (eg, the real world) and/or generate updates to the 3D map of the scene while tracking the pose of the XR system 100 . 3D map updates may include, for example and without limitation, new or updated features and/or features or landmark points associated with the scene and/or the 3D map of the scene, identifying or updating the position of the XR system 100 in the scene and/or the 3D map of the scene Location updates for locations within the . A 3D map can provide a digital representation of a scene in the real/real world. In some examples, a 3D map can anchor location-based objects and/or content to real-world coordinates and/or objects. The XR system 100 may use a mapped scene (eg, a scene in the real world represented by and/or associated with a 3D map) to merge real and virtual worlds and/or to merge virtual content or objects with a real environment.

In some aspects, the pose of image sensor 102 and/or XR system 100 as a whole may be used by computing element 110 based on images captured by image sensor 102 (and/or other cameras of XR system 100 ). Vision tracking solutions to determine and/or track. For example, in some examples computing element 110 may perform tracking using computer vision-based tracking, model-based tracking, and/or simultaneous localization and mapping (SLAM) techniques. For example, computing element 110 may perform SLAM or may communicate (wired or wirelessly) with a SLAM engine (not shown). SLAM refers to a class in which a map of an environment (e.g., a map of the environment modeled by the XR system 100) is created while simultaneously tracking the pose of the camera (e.g., the image sensor 102) and/or the XR system 100 relative to the map technology. This map may be referred to as a SLAM map, and may be three-dimensional (3D). SLAM techniques can be performed using color or grayscale image data captured by image sensor 102 (and/or other cameras of XR system 100) and can be used to generate image sensor 102 and/or XR system 100 Estimation of 6DOF pose measurements of system 100 . Such SLAM techniques configured to perform 6DOF tracking may be referred to as 6DOF SLAM. In some cases, the output of one or more sensors (e.g., accelerometer 104, gyroscope 106, one or more IMUs, and/or other sensors) may be used to estimate, correct, and/or otherwise Adjust the estimated pose.

In some cases, 6DOF SLAM (eg, 6DOF tracking) may associate features observed from certain input images from image sensor 102 (and/or other cameras) with a SLAM map. For example, 6DOF SLAM can use feature point associations from an input image to determine the pose (position and orientation) of the image sensor 102 and/or the XR system 100 of the input image. It is also possible to perform 6DOF mapping to update the SLAM map. In some cases, a SLAM map maintained using 6DOF SLAM can contain 3D feature points triangulated from two or more images. For example, keyframes may be selected from an input image or video stream to represent an observed scene. For each keyframe, the corresponding 6DOF camera pose associated with the image can be determined. The pose of the image sensor 102 and/or the XR system 100 can be determined by projecting features from the 3D SLAM map into an image or video frame and updating the camera pose according to the verified 2D-3D correspondence.

In an illustrative example, computing element 110 may extract feature points from each input image or each keyframe. As used herein, a feature point (also called a registration point) is a unique or identifiable part of an image, such as a part of a hand, the edge of a table, and so on. Features extracted from the captured image may represent different feature points along a three-dimensional space (eg, coordinates on X, Y, and Z axes), and each feature point may have an associated feature location. The feature points in the keyframe match (same or correspond to) or do not match the feature points of the previously captured input image or keyframe. Feature detection can be used to detect feature points. Feature detection may include image processing operations for examining one or more primitives of an image to determine whether a feature is present at a particular primitive. Feature detection can be used to process the whole captured image or some parts of the image. For each image or keyframe, once a feature is detected, local image patches around the feature can be extracted. Features can be extracted using any suitable technique, such as Scale Invariant Feature Transform (SIFT) (which locates features and produces their descriptions), Speeded Up Robust Features (SURF), Gradient Location-Orientation Histogram (GLOH), Normalized Interaction Correlation (NCC) or other suitable techniques.

In some cases, the XR system 100 may also track the user's hands and/or fingers to allow the user to interact with virtual content in the virtual environment (e.g., virtual content displayed in a virtual private space) and/or control the virtual environment virtual content in . For example, the XR system 100 may track gestures and/or movements of a user's hand and/or fingertips to recognize or translate user interactions with the virtual environment. User interaction may include, for example and without limitation, moving an item of virtual content, resizing an item of virtual content and/or the location of a virtual private space, selecting an input interface element in a virtual user interface (e.g., a virtual display, virtual keyboard and/or other virtual interface), providing input via a virtual user interface, etc.

2 is a diagram illustrating exemplary landmarks of the hand 200 that may be used to track the position of the hand 200 and the interaction of the hand 200 with a virtual environment, such as virtual content displayed within a virtual private space described herein. The landmarks illustrated in FIG. 2 correspond to different parts of the hand 200, including landmarks 235 on the palm of the hand 200, landmarks on the thumb 230 of the hand 200, landmarks on the index finger 232 of the hand 200, A marker point on the middle finger 234 , a marker point on the ring finger 236 of the hand 200 , and a marker point on the little finger 238 of the hand 200 . The palm of hand 200 can move in three translational directions (e.g., measured in X, Y, and Z directions relative to a plane such as the image plane) and in three rotational directions (e.g., in yaw relative to a plane , pitch and roll measurements), and thus provide six degrees of freedom (6DOF) that can be used for registration and/or tracking. The 6DOF movement of the palm is shown as a square in FIG. 2 , as indicated by legend 240 .

Different joints of the fingers of hand 200 allow for different degrees of motion, as shown in legend 240 . As shown by the rhombus in Figure 2 (such as rhombus 233), the base of each finger (corresponding to the metacarpophalangeal (MCP) joint between the proximal phalanx and the metacarpal) has a Two degrees of freedom (2DOF). Each upper joint of each finger (corresponding to the interphalangeal joints between the distal, middle and proximal phalanxes) has a freedom corresponding to flexion and extension, as indicated by the circles in FIG. degrees (1DOF). As a result, the hand 200 provides 26 degrees of freedom (26 DOF) from which to track the hand 200 and its interaction with the virtual content rendered by the XR system 100 .

XR system 100 may use one or more of the landmark points on hand 200 to track hand 200 (eg, track the pose and/or movement of hand 200 ) and track interactions with the virtual environment rendered by XR system 100 . As before, as a result of detecting one or more landmark points on hand 200 , a pose of the relative physical position of the landmark (and hand and fingers) with respect to XR system 100 may be established. For example, a landmark (eg, landmark 235 ) on the palm of hand 200 may be detected in the image, and the location of the landmark relative to image sensor 102 of XR system 100 may be determined. A point (e.g., a center point, such as a centroid or other center point) of a virtual content item rendered by XR system 100 may be translated into a position on a display (e.g., display 109 of FIG. 1 ) of XR system 100 relative to hand 200 The position of the place (or rendering on the monitor) is determined by the marker point on the palm.

As described below, XR system 100 may also register virtual content and/or hand 200 to points in the real world (as detected in one or more images) and/or other parts of the user. For example, in some embodiments, in addition to determining the physical pose of hand 200 relative to XR system 100 (or XR system 100 ) and/or the virtual content item, XR system 100 may also determine the location of other landmarks, such as walls A unique point on a surface (called a feature point), one or more corners of an object, a feature on a floor, a point on a human face, a point on a nearby device, and so on. In some cases, XR system 100 may place virtual content within specific locations relative to detected feature points in the environment, which may correspond to, for example, objects and/or people detected in the environment.

In some examples, the pose of XR system 100 (and/or the user's head) may be determined using, for example, image data from image sensor 102 and/or measurements from one or more sensors. , the one or more sensors such as accelerometer 104, gyroscope 106, and/or one or more other sensors (eg, one or more magnetometers, one or more inertial measurement units (IMUs), etc. ). Head pose can be used to determine the position of virtual content, hands 200 and/or objects and/or people in the environment.

The operations of XR engine 120 , input options engine 122 , context management engine 123 , image processing engine 124 , and rendering engine 126 (and any image processing engines) may be implemented by any of computing elements 110 . In one illustrative example, operations of rendering engine 126 may be implemented by GPU 114, and operations of XR engine 120, input options engine 122, context management engine 123, and image processing engine 124 may be implemented by CPU 112, DSP 116, and/or ISP 118 implementation. In some cases, computing element 110 may include other electronic circuitry or hardware, computer software, firmware, or any combination thereof, to perform any of the various operations described herein.

In some examples, XR engine 120 may perform XR operations based on input from image sensor 102, accelerometer 104, gyroscope 106, and/or one or more sensors on XR system 100, such as one or more IMU, radar, etc.) to generate XR experience. In some examples, XR engine 120 may perform tracking, localization, pose estimation, mapping, content anchoring operations, and/or any other XR operations/functions. The XR experience may include using the XR system 100 to present XR content (eg, virtual reality content, augmented reality content, mixed reality content, etc.) to a user during a virtual communication session. In some examples, XR content and experiences can be provided by the XR system 100 via XR applications (e.g., executed or implemented by the XR engine 120) that provide experiences such as, for example, an XR gaming experience, an XR classroom experience, an XR shopping experience , XR entertainment experiences, XR activities (eg, operations, troubleshooting activities, etc.), etc. During an XR experience, a user may use the XR system 100 to view and/or interact with virtual content. In some cases, a user may be able to view and/or interact with virtual content while also being able to view and/or interact with the physical environment surrounding the user, thereby allowing the user to interact with and interact with the physical environment and Immersive experience between virtual content mixed or integrated with physical environment.

XR engine 120, input options engine 122, and context management engine 123 may perform various operations to determine (and manage) how, where, and/or when certain virtual content is rendered relative to one or more other devices. For example, XR engine 120, input options engine 122, and context management engine 123 can facilitate interaction with other devices, such as, for example, connected devices (e.g., network-connected cameras, speakers, light bulbs, hubs, locks, plugs, thermostats, monitors, alarm systems, televisions (TVs), gadgets, appliances, etc.), mobile devices, devices lacking outward/external controls, devices lacking a display and/or user interface, and applications that wish to interact with such devices devices with certain characteristics that present one or more challenges (e.g., devices with controls/interface in a language the user does not understand, controls/interface that the user does not recognize/understand, limited accessibility options, use not easily identifiable/understandable input options, etc.), devices with controls/interfaces that are not accessible to the user, and/or any other device. For example, input options engine 122 may obtain or receive input data indicating or identifying one or more input options for another device. Input options engine 122 may send input data to XR engine 120 . The context management engine 123 may obtain or receive information about other devices (with which the user can interact with the XR system 100 ), the scene or environment where the XR system and/or other devices are located, or the user of the XR system trying to interact with other devices, and/or other contextually relevant information (eg, contextual information, etc.). The context management engine 123 can send context information to the XR engine 120 .

Using input data from one or more input options of a pointing device and/or usage context information, XR engine 120 may cause rendering engine 126 to present relevant information to a user. For example, XR engine 120 may cause rendering engine 126 to output guidance material corresponding to input options for which relevant contextual information has been determined. Instructional material may inform the user what input options are available for other devices, how to provide such input, and so on. In some examples, guidance material may filter out information that may not be relevant given the current context (eg, in view of information related to the XR system 100 , the scene, other equipment, and/or the user associated with the XR system 100 ). Input options (and/or associated information) that are too relevant or unavailable. In some examples, using input data and/or usage context information indicating one or more input options for the device, XR engine 120 may cause rendering engine 126 to render for interacting with the device (e.g., control, access content, access status information, access output, etc.) user interface and/or input options.

For example, based on contextual information and input data identifying input options associated with other devices, XR engine 120 may cause rendering engine 126 to present user interactions corresponding to one or more input options that enable the user to interact with other devices material. For example, user interaction data may include one or more user interface elements associated with an input option (e.g., selectable controls, etc.), prompts indicating how to provide input associated with an input option (e.g., highlighting, arrows, text, etc.) and/or other material. Context information may enable XR engine 120 to render (e.g., virtual content, audio content, user interface content, etc.) contextual given the situation/context associated with the user, XR system 100, other device, scene Appropriate content, and/or content that otherwise facilitates interaction with other devices. In some cases, XR engine 120 may cause rendering engine 126 to render no content, or reduce or filter the amount of virtual content to be displayed based on context (eg, display a subset of user interface options).

In some cases, XR engine 120 may utilize its XR capabilities to facilitate interaction with other devices. For example, the XR system 100 may have AR capabilities, such as the ability to display virtual content on the display of the XR system 100, while also allowing the user to view the real world environment via the display. XR engine 120 may utilize AR capabilities to render a user interface for a user to interact with other devices directly or by providing input to XR engine 120 (eg, using input device 108 ). For example, rendering engine 126 may render the user interface on a display such that the user interface appears to the user as an overlay on other devices or parts of other devices (e.g., controls, surfaces, displays, panels, etc.) to Facilitate interaction with users of other devices. XR engine 120 (or other elements of XR system 100 ) may use interaction with an overlay user interface to control other devices based on user input and/or accessing data/output from other devices. In some cases, the XR system 100 may locate and/or map other devices in order to facilitate and/or improve user interaction with the other devices. For example, XR system 100 may locate other devices and use the positioning information to render a user interface overlaid on the other device or a portion of the other device.

The XR system 100 may register or anchor the virtual content item to (eg, relative to) the detected feature points in the scene. For example, input options engine 122, context management engine 123, and/or image processing engine 124 may coordinate with XR engine 120 and/or rendering engine 126 to anchor the virtual content of the user interface to the surface on which the virtual content will be displayed. Feature points.

In some examples, XR system 100 may communicate with one or more other devices to provide input/commands to the other devices based on user interaction with the user interface rendered by rendering engine 126 . For example, XR engine 120 may cause XR system 100 to send commands to the device based on user input (using a transmitter or transceiver) received via a rendered user interface and/or input options. The command may cause the device to perform one or more functions based on user input.

In other examples, XR engine 120 may utilize one or more modes of interaction (eg, visual, voice/audio, gesture-based, motion-based, etc.) to facilitate user interaction with other devices. For example, XR engine 120 may use hand tracking and/or gesture recognition capabilities to allow a user to interact (eg, control, access, etc.) with other devices using gestures and other interactions. As another example, XR engine 120 may use voice recognition to allow a user to interact with other devices using voice commands.

As previously mentioned, input data may indicate some or all of the input options available for interacting with other devices. For example, input options may include the types of input supported by the device (e.g., gesture-based, voice-based, touch-based, etc.), the functions the device can perform based on a particular input (e.g., swiping to the right causes the thermostat to increase the temperature, etc.), and other Information.

In some examples, input options engine 122 may obtain or receive one or more information indicating the other device from the other device, from a server (eg, a cloud-based server related to the operation of the other device), and/or from another source. Input data for each input option. For example, the input options engine 122 may send (or cause a transmitter or transceiver to send) a request for input options for a device. In one example, the XR system 100 may detect or sense the device, such as based on previous network pairing with the device, based on periodic beacon signals transmitted (e.g., broadcast) by the device indicating its presence, based on The device is detected in one or more images provided by image sensor 102, and so on. In response to detecting or sensing a device, the XR system 100 may request input data from the device. In response to the request, the device may respond with input data indicating any input options associated with the device. In another example, the input options engine 122 may request input options for a given device from a server associated with the device (eg, a Google ^™ server associated with a Google Home ^™ device). The server may respond with input data indicating input options associated with the device.

In some cases, the input option engine 122 may determine one or more input options of other devices by processing one or more images of other devices captured by the image sensor 102 . For example, using an elevator console as an illustrative example of a user interface for a device (an elevator), input options engine 122 may receive an image of an elevator console from image sensor 102 . using machine learning (for example, using one or more neural network-based object detectors or classifiers), computer vision (for example, using computer vision-based object detectors or classifiers), or other image analysis techniques, The input options engine 122 may determine that the console includes fifteen numerical values corresponding to building floors, a door open button, a door close button, a panic button, and/or other physical or virtual buttons.

In some instances, if the device does not transmit any input data to the XR system (e.g., after the XR system sends a request to the device), the XR system may determine or infer that the device does not have a connection with the XR system (e.g., via a wireless network). ability to communicate. In such instances, the XR system can present instructions or other information (eg, highlighting a particular button on the device) to help the user decide how to interact with the device. For example, in the elevator example above, the elevator may not have the capability to communicate (eg, via a communication network) with the XR system. In such an example, the XR system may present virtual content that assists the user in interacting with the elevator, rather than sending one or more commands to control the elevator based on received user input.

The context management engine 123 can send context information to the XR engine 120 . The contextual information provides the XR engine 120 with a contextual awareness of the situation/context associated with the user, the XR system 100 , other device(s), the scene, and the like. The XR engine 120 may use the contextual information to manage, modulate and/or determine the content, input options, user interface and/or modalities presented to the user to interact with other devices. For example, as previously described, the XR engine 120 may use contextual information to render content (e.g., virtual content, audio content, user interface content, etc.) that is associated with the user, the XR system 100, other devices, the scene /Context is context-sensitive and/or otherwise facilitates interaction with other devices.

Contextual information may relate to other devices with which a user may interact using the XR system 100 , the scene or environment in which the XR system 100 and/or other devices are located, users of the XR system 100 attempting to interact with other devices, and/or Other contextually relevant at any given point in time. In some instances, contextual information may include the user's expected interactions with other devices, such as expectations to interact with the device, expectations to interact with specific input options (e.g., specific UI control elements) of the device's user interface , and/or other expected user interactions. The context management engine 123 may estimate expected user interaction, such as based on eye gaze, specific gestures being performed, user holding other devices, user walking toward the device, and/or other information. For example, the context management engine 123 may determine that the user is gazing at a thermostat, and based on the determined gaze, determine that the user intends to interact with the thermostat. In another example, the context information may include one or more actions of the user in the scene. For example, one or more actions may include the user walking towards a device, walking towards a door in a scene, sitting in a chair where the user normally watches television, and the like. Other examples of contextual information include characteristics associated with the user (e.g., visual quality, spoken language(s), etc.), historical information associated with the user and other devices (e.g., the user's past use of the device , the user's level of experience with the device or similar devices, etc.), the user interface capabilities of other devices (e.g., whether it has outward/external controls that are visible or otherwise accessible to the user), interactions with other devices Associated information (e.g., how far other devices are from the XR system 100), information associated with the scene (e.g., lighting, noise levels such as ambient sounds, objects or other obstructions between the XR system 100 and the device , whether there are any other users in the scene, etc.), and/or other information.

In some examples, the context management engine 123 can determine, obtain or receive context information locally. For example, context management engine 123 may obtain senses from one or more sensors of XR system 100 (eg, image sensor 102, accelerometer 104, gyroscope 106, and/or other sensors of XR system 100). Meter information. The context management engine 123 may process sensor information to determine contextual information, such as one or more intentional interactions of the user with other devices, one or more actions of the user in the scene, user interface capabilities of other devices, Information associated with other devices (e.g., the distance between the device and the XR system 100 and the user), information associated with the scene (e.g., lighting, noise, objects or obstacles between the XR system 100 and the device, etc.) , and/or other contextual information. In one illustrative example, context management engine 123 may receive an image from image sensor 102 indicating that the user is looking at an oven, and may also receive an image from accelerometer 104 and/or gyroscope 106 indicating that the user is walking toward an oven. Sensor data for the oven. Based on the image and sensor data, the context management engine 123 can determine that the user intends to interact with the oven.

In some examples, the context management engine 123 may obtain or receive context information from one or more remote sources (eg, server, cloud, Internet, other devices, one or more sensors, etc.). For example, the context management engine 123 may access a user profile stored on a web-based or cloud-based system associated with the user, the profile indicating characteristics associated with the user (e.g., the user's visual quality, language spoken by the user, etc.) and/or historical information associated with the user and other devices (e.g., the user's level of experience with one or more devices, how long the user has owned one or more devices, etc. ). In some cases, the user profile may be stored locally on the XR system 100 .

As previously described, the context information provides context awareness to the XR engine 120 so that the XR system 100 can present content that is contextually relevant to the particular situation in which the XR system 100 is being used. For example, XR engine 120 may use contextual information including characteristics of the user, other devices, etc., when deciding what AR content to present to the user to guide and/or facilitate user interaction with other devices and/or how to present the AR content . In one example, the XR engine 120 can take into account the language the user understands/speaks to ensure that the presented AR content is in the language the user understands/speaks. As another example, XR engine 120 may tailor user interface guidance to expectations regarding the user's knowledge of the device. In one illustrative example, if the user is estimated to be new to using the device (e.g., based on the amount of time the user has owned the device, the number of times the user has used the device, and/or based on other factors), XR engine 120 may Outputting additional support to the user (eg, by presenting instructions as to what inputs can be used to control the device). In another example, if the user is estimated to be an expert in using the device (e.g., the user is estimated to have at least a threshold amount of experience/familiarity), the XR engine 120 may not present additional support to the user, or may Reduce/minimize such support. As another example, if the user is wearing one or more items that may affect interactions with other devices, the XR engine 120 may adjust which user interface, controls, and/or guidance is output to the user based on what the user is wearing . In one illustrative example, if the user is wearing gloves that may impede the user's ability to select/touch controls, if the user is wearing glasses (or no glasses) or sunglasses that may impede visibility, if the user has For medical devices that restrict the user's movement and ability to interact with certain controls, the XR engine 120 can adjust the user interface, controls and/or guidance it provides.

As another example, XR engine 120 may use contextual information (eg, environmental factors) associated with a scene when deciding how to present AR content and/or what AR content to present to a user. For example, XR engine 120 may take into account lighting conditions, such as low or bright light conditions (e.g., it may suggest audio cues and/or visual cues), ambient sounds (e.g., it may suggest visual cues instead of audio cues), The presence of other people (eg, which may suggest a need for discreetness or privacy, such as not providing any audio output, not presenting input options involving hand or gesture movements, etc.), etc.

In some examples, the XR engine 120 may determine which of the plurality of devices the user intends to interact with. In some cases, XR engine 120 may cause rendering engine 126 to present a user interface to allow the user to interact with the particular device with which the user wants to interact. In some cases, XR engine 120 may output instructions (eg, as visual or audio content) on how to interact with and/or control a particular device. For example, in a kitchen with a smart home assistant playing music, a connected refrigerator, and a connected stove, a user may be interested in changing the temperature of the stove. Based on the contextual information received from the context management engine 123, the XR engine 120 may determine that the user intends to interact with the connected stove rather than the smart home assistant or the connected refrigerator. For example, based on contextual information, the XR engine 120 may determine the user's eye gaze, posture, and/or movement, and determine that the user intends to interact with a connected stove rather than a smart home assistant or a connected refrigerator. In some examples, XR engine 120 may determine a number of possible gestures that a user may perform to interact with the connected stove. For example, the XR engine 120 may determine that the user may use a knob turning gesture to change the temperature of the stove. The XR engine 120 may provide an output informing the user (eg, via visual cues, audio cues, etc.) that a knob turning gesture may be used to change the temperature of the furnace. In some cases, XR engine 120 may detect a knob turning gesture from a user and communicate the associated input to the connected stove. In some cases, the connected stove can directly detect the knob turning gesture.

The image processing engine 124 may perform one or more image processing operations related to the virtual user interface content being presented. For example, image processing engine 124 may perform image processing operations based on data from image sensor 102 . In some cases, image processing engine 124 may perform image processing operations such as, for example, filtering, demosaicing, scaling, color correction, color conversion, segmentation, denoising filtering, spatial filtering, artifact correction, and the like. Rendering engine 126 may obtain image data generated and/or processed by computing element 110, image sensor 102, XR engine 120, input options engine 122, context management engine 123, and/or image processing engine 124, and may Renders video and/or image frames for presentation on a display device.

Although XR system 100 is shown as including certain elements, one of ordinary skill will appreciate that XR system 100 may include more or fewer elements than shown in FIG. 1 . For example, in some cases, XR system 100 may also include one or more memory devices (e.g., RAM, ROM, cache, and/or the like), one or more network interfaces (e.g., wired and and/or wireless communication interfaces and the like), one or more display devices, and/or other hardware or processing devices not shown in FIG. 1 . An illustrative example of a computing system and hardware elements that may be implemented with XR system 100 is described below with reference to FIG. 9 .

FIG. 3 is a diagram illustrating an example of an extended reality system 300 worn by a user 301 . In some cases, extended reality system 300 is similar to XR system 100 of FIG. 1 and may perform similar operations. Extended reality system 300 may include any suitable type of XR device or system, such as AR or MR glasses, AR, VR or MR HMD, or other XR devices. For purposes of illustration, AR may be used to describe some of the examples described below. However, the aspects described below can be applied to other types of XR, such as VR and MR. The extended reality system 300 shown in FIG. 3 may include an optical see-through AR device, which allows the user 301 to watch the real world while wearing the extended reality system 300 .

For example, the user 301 can view the object 303 in the real world environment on the plane 304 at a certain distance from the user 301 . As shown in FIG. 3 , the extended reality system 300 has an image sensor 302 and a display 309 . As previously mentioned, display 309 may include glass, screens, lenses, and/or other display mechanisms that allow user 301 to see the real world environment and also allow AR content to be displayed thereon. AR content (eg, images, videos, graphics, virtual or AR objects, or other AR content) may be projected or otherwise displayed on display 309 . In one example, AR content may include an enhanced version of object 303 . In another example, the AR content may include additional AR content related to object 303 and/or related to one or more other objects in the real world environment. Although one image sensor 302 and one display 309 are illustrated in FIG. 3 , in some implementations, the augmented reality system 300 may include multiple cameras and/or multiple displays (e.g., a display for the right eye). and display for the left eye).

As described above with reference to FIG. 1 , XR engine 122 may utilize input data and context data indicating one or more input options for the device to cause rendering engine 126 to render for interaction with the device (e.g., control, access content, access state information, access to output, etc.) user interface and/or input options. In one illustrative example, XR system 100 may assist a user in interacting with a console (such as an elevator console, a console of a vehicle, etc.). 4A , 4B and 4C are diagrams illustrating an example of a user interacting with a console 410 of an elevator using the XR system 400 . For example, when a user wearing XR system 400 enters an elevator, XR system 400 may determine the user's eye gaze (e.g., via gaze scanning, based on eye gaze camera, etc.) and/or one or more gestures, such as using A gesture performed by the hand 405 of the user. Based on the eye gaze and/or gesture(s), the XR system 400 may determine that the user cannot find or has difficulty finding the floor number to press on the elevator console 410 . For example, the XR system 400 may determine that the user's eye gaze is moving back and forth (as if the user were searching for the correct value).

Based on the contextual information obtained by the XR system 400, the XR system 400 may determine that the user is searching for control buttons associated with a particular floor. For example, contextual information may include registration information at a hotel (e.g., obtained by XR system 400 from a server associated with the hotel), voice commands provided by the user to XR system 400, user preferences/inputs (e.g., A user may enter a floor number into the XR system 100), recognition audio detected from the user (e.g., the XR system 400 recognizes the word "floor 16" from the user, such as using always-on audio), detects (eg, detected from one or more images captured by the image sensor 102 , etc.).

The XR system 400 can present prompts on the elevator console associated with the control buttons for a particular floor to guide the user to the correct button. For example, the XR system 400 may determine that the user is searching for the control button corresponding to floor 16 (associated with the button having the value "16" in FIGS. 4A and 4B ). As shown in FIG. 4B , XR system 400 may display virtual content 412 highlighting the control button corresponding to floor 16 . The virtual content 412 appears to be overlaid on the actual control buttons corresponding to the floors 16 to help the user easily identify the control button the user is looking for. As shown in FIG. 4C , the XR system 400 can display virtual content 412 highlighting the control button corresponding to floor 16, and can also present text 414 ("the button for floor 16 is here") and an arrow icon providing further information , to help users identify the correct control button. While FIGS. 4B and 4C illustrate highlights and text as examples of visual cues, the cues may additionally or alternatively include other types of visual cues, audio cues that indicate to the user the identification or location of the correct button (e.g., based on hand tracking to guide the user's hand) and/or other types of cues. This is especially useful for users with disabilities or who have difficulty remembering/learning the correct button.

In some cases, after multiple trips in the elevator, the XR system 400 can learn that the user can find the correct button quickly and/or without assistance. In response, XR system 400 may decide to stop providing virtual content to help identify control buttons. In other cases, XR system 400 may continue to provide such assistance to the user (eg, when contextual information specific to the user indicates that the user has a disability). In some cases, XR system 400 may similarly assist a user in interacting with other consoles, devices, industrial machinery, and the like.

In another illustrative example, XR system 100 may facilitate user interaction with a remote control that may be used to control the device. 5A and 5B are diagrams illustrating an example of a user using the XR system 500 to interact with a remote control 510 that can control a television 511 . XR system 500 may determine that remote control 510 may be difficult for a user to use based on various factors. For example, XR system 500 may determine (e.g., based on one or more images of the room obtained using image sensor 102, based on an ambient light sensor of XR system 500, etc.) that television 511 and the room in which the user is located have Poor lighting conditions. Additionally, based on contextual information obtained by XR system 500 indicative of user characteristics, XR system 500 may determine that the user has poor near vision. XR system 500 may additionally or alternatively determine that labels or buttons on remote control 510 are difficult to read, are in a language that the user does not understand, and/or otherwise make remote control 510 difficult to interact with the user. Based on this contextual information, the XR system 500 can determine that the controls/labels of the remote control 510 may be difficult for the user to see. In some cases, the XR system 500 may additionally or alternatively detect that the user is having difficulty interacting with the remote. For example, the XR system 500 can use eye tracking to determine that the user is squinting and/or scanning the remote for the correct button.

The XR system 500 may obtain further contextual information indicating that the user would like to switch to a particular channel (channel 34) broadcasting events of interest to the user. For example, the XR system 500 may determine that the user has scheduled a sports event on the user's digital calendar. In another example, the XR system 500 may determine that the user always watches a particular channel at a particular time of night. Using the contextual knowledge that the remote control 510 may be difficult to interact with and that the user wants to switch to a particular channel (for example, because the user frequently watches events in a particular channel at the current time), the XR system 500 can display a virtual profile 512, highlighting The correct "3" and "4" buttons on the remote control 510 to help the user identify the button and select the button to switch to a specific channel 34 . In some examples, the XR system 500 may sequentially highlight button "3," followed by button "4," so that the user knows to press button "3" before pressing button "4."

In some cases, XR system 500 may use audio to confirm options (eg, events and/or associated buttons/channels) determined by XR system 500 in addition to or instead of highlighting or emphasizing values as described above. For example, the XR system 500 may provide an audio prompt asking the user to confirm that the user wishes to switch to a particular channel broadcasting the event. XR system 500 may receive confirmation from the user (eg, based on user input provided via an input device such as input device 108 ) and proceed with the assistance. In one example, in response to receiving the confirmation, XR system 500 can highlight the correct buttons (eg, "3" and "4" buttons) on remote control 510 that correspond to the channel (eg, channel 34). In another example, in response to receiving the confirmation, XR system 500 can automatically send a command to television 511 and/or remote control 510 that causes television 511 to change to a channel (eg, channel 34).

In another illustrative example, XR system 100 may assist a user in interacting with a thermostat. 6A and 6B are diagrams illustrating an example of a user interacting with a thermostat 610 using the XR system 600 . For example, thermostat 610 may be configured to interpret one or more gestures using gesture recognition, and may perform one or more functions based on the detected gestures. However, the user may not be aware of the correct gesture commands that can be used to cause the thermostat 610 to perform certain functions. Similar to what was described above, the XR system 600 uses the acquired contextual information to determine that the user is having difficulty interacting with the thermostat 610 . For example, XR system 600 may use eye tracking to determine that the user is looking at thermostat 610, and/or may process one or more images to determine that the user is performing a gesture (e.g., using hand 605) but the thermostat is not. Any functionality based on gestures. XR system 600 may use any other contextual information to determine that it is difficult for the user to interact with thermostat 610 .

In some cases, the user may issue a voice command or other input that identifies the user's desired interaction with thermostat 610 . For example, the user can recite "set the temperature to 68 degrees," and the XR system 600 can recognize the voice command. XR system 600 may determine which gesture commands thermostat 610 is configured to interpret, such as from thermostat 610 , from a servo associated with thermostat 610 (eg, a Nest ^™ servo associated with a Nest ^™ thermostat).

As shown in FIG. 6B , in response to recognizing a voice command, receiving another input indicating a desired setting for the user, and/or determining that the user is having difficulty interacting with the thermostat 610, the XR system 600 may present a set of gestures to the user. commands (including gesture command 612 , gesture command 614 , and gesture command 616 ), the set of gesture commands can be applied (eg, with or without voice commands) to cause thermostat 610 to perform a desired function. In some cases, as shown in FIG. 6B , gesture commands may have corresponding numerical values indicating the order in which gesture commands 612 , 614 , and 616 should be executed in order to cause thermostat 610 to perform the desired function. For example, a user may execute gesture command 612 to put thermostat 610 into a temperature adjustment mode. The user may then execute gesture command 614 to cause thermostat 610 to increase the temperature. For example, the thermostat 610 may increase the temperature by one degree Fahrenheit each time the user performs the “thumbs up” gesture command 614 . The user may then execute gesture command 616 to exit thermostat 610 from thermostat mode.

In another illustrative example, XR system 100 may assist a user in interacting with a digital photo frame. 7A and 7B are diagrams illustrating an example of a user 701 using an XR system 700 that can decide whether to provide user interface input options for interacting with a digital photo frame 710 . For example, digital picture frame 710 may be configured to display metadata related to content displayed by digital picture frame 710 (eg, descriptions of displayed art, backgrounds, characters, etc.). For example, as shown in FIG. 7B , digital photo frame 710 may display metadata 712 next to a photo of two people dancing with the title "This is Nancy and Bob on their wedding day."

User 701 may not be interested in the metadata if user 701 is not gazing at digital picture frame 710 , if the user walks past digital picture frame 710 in a hurry, and/or otherwise may not be interested in metadata 712 . XR system 700 may obtain contextual information, such as detected eye gaze indicating that the user is not looking at digital photo frame 710, sensed motion indicating that the user is walking, user 701's calendar (e.g., indicating that the user is looking at different locations have upcoming appointments), user 701 preferences, user history data, user communications, and/or other contextual information. Based on contextual information obtained by XR system 700 , XR system 700 may determine that the user is walking past digital picture frame 710 , that the user is not looking at digital picture frame 710 , and/or that the user may not otherwise be interested in metadata 712 . XR system 700 may then send a command to digital picture frame 710 to prevent digital picture frame 710 from presenting metadata 712 (which could potentially distract the user).

8A is a diagram illustrating an example of a user 802 using an XR system 800 to interact with one or more devices when there are multiple devices in the scene. In this example, the scene includes a digital picture frame 810 , a remote control 812 and a digital thermostat 814 . User 802 can use XR system 800 to interact with any device in the scene, including digital photo frame 810 , remote control 812 and/or digital thermostat 814 . XR system 800 can receive input options and/or related data from digital picture frame 810, remote control 812, and digital thermostat 814, and can present information related to the input options corresponding to digital picture frame 810, remote control 812, and/or digital thermostat 814. linked user guides.

In some cases, when a scene includes multiple remote devices as shown in FIG. It presents guidance material. In some examples, XR system 800 may receive input options from digital picture frame 810 , remote control 812 , and digital thermostat 814 . Input options may include information about the types of inputs available/accepted at digital picture frame 810 , remote control 812 , and digital thermostat 814 . However, when receiving such data from multiple devices in a scene, XR system 800 may be overloaded with data from multiple devices. Data overload can make it difficult for the XR system 800 to present relevant information to the user 802, present information to the user without significant confusion, manage data and/or interact with the device, and the like.

For example, referring to FIG. 8B , XR system 800 may display profile 820 related to digital photo frame 810 , profile 822 related to remote control 812 , and profile 824 related to digital thermostat 814 . For XR system 800 and/or user 802, data 820, 822, and 824 may become overwhelming. For example, as shown in FIG. 8B, data 820, 822, and 824 may become cluttered when rendered by XR system 800, and the information rendered by XR system 800 may become overloaded and difficult to parse, manage, understand, etc. In some examples, XR system 800 may filter data from digital picture frame 810, remote control 812, and digital thermostat 814 to simplify and/or unify presentation by XR system 800 for digital picture frame 810, remote control 812, and digital thermostat 814. data of. In some cases, the XR system 800 may limit the presented materials to those corresponding to a particular device of interest.

To illustrate, referring to FIG. 8C , the XR system 800 can predict that the user wishes to interact with the remote device 812 or will interact with the remote device 812 . XR system 800 may use this information to filter out data 820 associated with digital photo frame 810 and data 824 associated with digital thermostat 814 in order to simplify the data presented by XR system 800 . The XR system 800 can present data 822 associated with the remote device 812 that is predicted to be relevant to the user 802 and/or the current context. In some examples, data 822 may include indications of input options that a user may use to interact with remote control 812 and/or input options that inform the user how to interact with remote control 812 . In some cases, material 822 may include user guidance material to facilitate user interaction with remote device 812 . In some examples, the presentation shown in FIG. 8C may clean up and/or simplify the data presented by XR system 800 .

In some cases, XR system 800 may use contextual information to predict which of digital picture frame 810, remote control 812, and digital thermostat 814 that user 802 will interact with are relevant to user 802, etc., and/or provide information to the user. Which data from digital picture frame 810 , remote control 812 and/or digital thermostat 814 are presented. In some cases, XR system 800 can present simplified information related to digital photo frame 810, remote control 812, and digital thermostat 814, which can be used by XR system 800 and user 802 to filter out less relevant information and The amount of information presented to the user 802 is reduced to the information most relevant to the user 802 .

For example, referring to FIG. 8D , XR system 800 may present a menu 840 of input options. Menu 840 may indicate various devices available for user interaction and provide user 802 with the ability to select a particular device of interest to the user. If the user selects a specific device, the XR system 800 can present information corresponding to the specific device, such as input options. For example, if user 802 selects digital photo frame 810 from menu 840 , XR system 800 may present data related to digital photo frame 810 and exclude other data related to remote control 812 and/or thermostat 814 .

In another illustrative example, XR system 100 may assist a user in using vehicle controls. For example, a user driving a vehicle may pull over to the side of the road. The XR system 100 may detect that the user has pulled over (eg, based on motion information, voice data from the user, data from the vehicle, etc.). Based on the determination that the user has pulled the vehicle over to the side of the road, the XR system 100 may determine that the user should activate the warning lights on the vehicle. Users can scan the vehicle's dashboard for the warning light button. The XR system 100 may use eye tracking, image analysis, and/or other techniques to determine that the user cannot find or have difficulty finding the warning light button. The XR system 100 can use this contextual information to determine that the user needs help finding the warning light button. The XR system 100 can locate/recognize the warning light button, and overlay AR content around the warning light button to guide/assist the user in locating the warning light button.

Although some illustrative examples of XR system 100 (and other XR systems) using input data and context information are described above to determine the input options presented to the user, XR system 100 can perform any task based on input data and context information. Other functions to assist the user of the XR system 100 to interact with one or more other devices.

9 is a flow diagram illustrating an example of a process 900 for presenting information associated with at least one input option using one or more techniques described herein. At block 902, process 900 may include receiving data identifying one or more input options associated with a device in a scene. For example, an XR system (eg, XR system 100 ) may receive data identifying what input options are available at one or more remote devices in a scene.

At block 904 , process 900 may include determining (including using at least one memory) information related to at least one of a scene, a device, and a user associated with the electronic device (eg, XR system 100 ). In some instances, this information can include contextual information. Contextual information may provide, for example, information about a scene, user, device, and/or electronic device.

At block 906, process 900 may include outputting, based on the one or more input options and information, user guidance material corresponding to the input options for which relevant contextual information has been determined. In some examples, user guidance data may include user input elements associated with input options, virtual overlays on physical objects associated with input options, and/or prompts indicating how to provide input associated with input options at least one of the

In some aspects, process 900 can include: predicting a user interaction with the device based on the information; and presenting user guidance material corresponding to the input option based on the one or more input options and the predicted user interaction.

In some examples, the device may include a connected device with network communication capabilities, and process 900 may include: determining a gesture representing a predicted user interaction based on the information and one or more input options; and presenting user guidance material. In some examples, the predicted user interaction may include predicted user input to the device. In some cases, user guidance data may include indications of gestures that, when detected, cause actual user input at the device.

In some aspects, presenting the user guidance material can include rendering a virtual overlay at a display associated with the electronic device, the virtual overlay configured to appear to be located on a surface of the device. In some examples, the virtual overlay can include user interface elements associated with input options. In some cases, the user interface element may include at least one of a virtual user input object associated with the input option and a visual indication of a physical control object on the device configured to receive input corresponding to the input option.

In some examples, the information includes at least one of the user's eye gaze and the user's gesture, and process 900 may include: predicting and using the device based on the at least one of the user's eye gaze and the user's gesture after presenting the user guidance material, detecting actual user input associated with the input options, the actual user input representing the predicted user interaction; and transmitting to the device corresponding to the actual usage associated with the input options command entered by the user.

In some examples, outputting user guidance data corresponding to the input options may include displaying the user guidance data. In some examples, outputting user guidance data corresponding to the input options may include outputting audio data representing the user guidance data.

In some examples, outputting user guidance data corresponding to the input options may include displaying the user guidance data; and outputting audio data associated with the displayed user guidance data.

In some aspects, process 900 can include receiving, from a device, data identifying one or more input options associated with the device. In some aspects, process 900 can include receiving from a server data identifying one or more input options associated with a device.

In some cases, the device has no external user interface for receiving one or more user inputs. In some aspects, process 900 can include refraining from presenting additional user guidance material associated with the device based on the information.

In some aspects, process 900 can include: after presenting the user guidance material, obtaining user input associated with the input options; and transmitting to the device instructions corresponding to the user input. In some cases, the instructions may be configured to control one or more operations of the device.

In some examples, the processes described herein (eg, process 900 and/or other processes described herein) can be performed by a computing device or apparatus. In one example, process 900 may be performed by XR system 100 of FIG. 1 . In another example, process 900 may be performed by a computing device having computing system 1000 shown in FIG. 10 . For example, a computing device having the computing architecture shown in FIG. 10 may include elements of the XR system 100 of FIG. 1 and may implement the operations of FIG. 9 .

The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, Internet-connected watches or smart watches, or other wearable devices), server computers, autonomous vehicles or computing devices for automated vehicles, robotic devices, televisions, and/or resources capable of performing the processes described herein, including process 800 any other computing device. In some cases, a computing device or apparatus may include various elements, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers , one or more cameras, one or more sensors, and/or other element(s) configured to perform steps of the processes described herein. In some examples, a computing device may include a display, a network interface configured to transmit and/or receive data, any combination thereof, and/or other element(s). A network interface may be configured to transmit and/or receive Internet Protocol (IP) based data or other types of data.

Elements of a computing device may be implemented in circuitry. For example, an element may include and/or be implemented using an electronic circuit or other electronic hardware that may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), ), digital signal processor (DSP), central processing unit (CPU) and/or other suitable electronic circuits), and/or may include and/or be implemented using computer software, firmware, or any combination thereof, to execute the Various operations described.

Process 900 is shown as a logic flow diagram whose operations represent a series of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, operations represent computer-executable instructions stored on one or more computer-readable storage media, which when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and/or in parallel to implement the processes.

Additionally, process 900 and/or other processes described herein can be performed under the control of one or more computer systems configured with executable instructions, and can be implemented via hardware or a combination thereof as collectively on one or more processors. Executed code (for example, executable instructions, one or more computer programs, or one or more application programs). As before, code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. A computer-readable or machine-readable storage medium may be non-transitory.

10 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. In particular, FIG. 10 illustrates an example of a computing system 1000, which may be, for example, any computing device, a remote computing system, a camera, or any element thereof that constitutes an internal computing system, wherein the elements of the system communicate with each other using a connection 1005. . Connection 1005 may be a physical connection such as using a bus bar in a chipset architecture or directly to processor 1010 . The connection 1005 can also be a virtual connection, a network connection or a logical connection.

In some embodiments, computing system 1000 is a distributed system, where the functionality described herein may be distributed across a data center, multiple data centers, a peer network, or the like. In some embodiments, one or more of the described system elements represents a plurality of such elements, each element performing some or all of the functions described for that element. In some embodiments, an element may be a physical device or a virtual device.

Exemplary system 1000 includes at least one processing unit (CPU or processor) 1010 and connections 1005 that will include system memory 1015 such as read only memory (ROM) 1020 and random access memory (RAM) 1025 Various system elements are coupled to processor 1010 . Computing system 1000 may include cache memory 1012 of high speed memory coupled directly to processor 1010 , proximate to processor 1010 , or integrated as part of processor 1010 .

Processor 1010 may include any general-purpose processor and hardware services or software services, such as services 1032, 1034, and 1036 stored in storage device 1030, configured to control processor 1010 and where software instructions are incorporated into the actual processor A dedicated processor in the design. Processor 1010 may essentially be a completely self-contained computing system including multiple cores or processors, a bus, memory controller, cache memory, and the like. Multicore processors can be symmetric or asymmetric.

To enable user interaction, computing system 1000 includes input devices 1045, which may represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, voice wait. Computing system 1000 may also include an output device 1035, which may be one or more of a variety of output mechanisms. In some examples, a multimodal system may enable a user to provide multiple types of input/output to communicate with computing system 1000 . Computing system 1000 can include communication interface 1040, which can generally control and manage user input and system output. Communication interfaces may use wired and/or wireless transceivers to perform or facilitate the reception and/or transmission of wired or wireless communications, including their use of audio jacks/plugs, microphone jacks/plugs, Universal Serial Bus (USB) ports/ Plug, Apple® Lightning® Port/Plug, Ethernet Port/Plug, Fiber Optic Port/Plug, Proprietary Wired Port/Plug, Bluetooth® Wireless Signal Transmission, Bluetooth® Low Energy (BLE) Wireless Signal Transmission, IBEACON® wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short-range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) Signal transmission, visible light communication (VLC), worldwide interoperability for microwave access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission , 3G/4G/5G/LTE cellular data network wireless signal transmission, self-organizing network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless along the electromagnetic spectrum signal transmission, or some combination thereof. Communication interface 1040 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for communicating based on receiving one or more signals from one or more satellites associated with one or more GNSS systems A location for the computing system 1000 is determined. GNSS systems include, but are not limited to, the US-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the Chinese-based Beidou Navigation Satellite System (BDS), and the European-based Galileo GNSS. There is no restriction on operation on any particular hardware configuration, so the basic features herein can easily be replaced with improved hardware or firmware configurations as they are developed.

Storage device 1030 may be a non-volatile and/or non-transitory and/or computer-readable memory device, and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a magnetic Cassettes, flash memory cards, solid state memory devices, digital versatile disks, cassette tapes, floppy disks, flex disks, hard disks, tapes, magnetic strips/stripes, any other magnetic storage media, flash memory memory, memristor memory, any other solid-state memory, compact disc read-only memory (CD-ROM) discs, rewritable compact discs (CD) discs, digital video discs (DVD) discs, Blu-ray discs, holographic Optical discs, another optical media, Secure Digital (SD) cards, Micro Secure Digital (microSD) cards, Memory Stick® cards, Smart Card chips, Europay MasterCard and Visa (EMV) chips, Subscriber Identification Module (SIM) cards, Mini/micro/nano/pico SIM card, another integrated circuit (IC) chip/card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read only memory (ROM) , programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory EPROM (FLASHEPROM), fast Access memory (L1/L2/L3/L4/L5/L#), resistance random access memory (RRAM/ReRAM), phase change memory (PCM), spin transfer torque RAM (STT-RAM), Another memory chip or cartridge, and/or combinations thereof.

Storage device 1030 may include software services, servers, services, etc. that, when code defining such software is executed by processor 1010, cause the system to perform functions. In some embodiments, hardware services to perform specific functions may include software elements stored on computer readable media in combination with necessary hardware elements such as processor 1010, connections 1005, output devices 1035, etc. , to execute the function. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices and various other media capable of storing, containing or carrying instruction(s) and/or data. Computer-readable media may include non-transitory media that can store data and do not include carrier waves and/or transitory electronic signals that travel wirelessly or via wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, memory or storage devices. A computer-readable medium may have stored thereon code and/or machine-executable instructions, which may represent a program, function, subroutine, program, routine, subroutine, module, package, instruction, data A class or any combination of structures or program statements. A code segment can be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents. Information, parameters, parameters, data, etc. may be passed, forwarded, or transmitted by any suitable means (including memory sharing, message passing, token passing, network transmission, etc.).

In some embodiments, computer-readable storage devices, media, and memory may include wired or wireless signals including bit streams and the like. However, when mentioned, non-transitory computer readable storage media expressly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

In the above description specific details are provided to provide a thorough understanding of the embodiments and examples provided herein. However, one of ordinary skill will understand that the embodiments may be practiced without these specific details. For clarity of explanation, the technology may in some cases be presented as comprising individual functional blocks including methods embodied in software or a combination of hardware and software to include devices, device elements, A function block of a step or routine. Other elements may be used in addition to those shown in the figures and/or described herein. For example, circuits, systems, networks, processes and other elements may be shown in block diagram form as elements in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, procedures, algorithms, structures and techniques may be illustrated without unnecessary detail in order not to obscure the embodiments.

Various embodiments may be described above as processes or methods illustrated as flowcharts, process schematics, material flow diagrams, block diagrams, or block diagrams. Although a flowchart can describe operations as a sequential process, many operations can be performed in parallel or concurrently. Additionally, the order of operations can be rearranged. A process terminates when its operations are complete, but there may be other steps not included in the diagram. A procedure may correspond to a method, function, procedure, subroutine, subroutine, or the like. When a procedure corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

Procedures and methods according to the examples above may be implemented using computer-executable instructions stored on or otherwise obtained from a computer-readable medium. Such instructions may include, for example, instructions and materials which cause or otherwise configure a general purpose computer, special purpose computer or processing device to perform a certain function or group of functions. Part of the computer resources used can be accessed via the Internet. Computer-executable instructions may be, for example, binary files, intermediate format instructions (such as assembly language, firmware, source code, etc.). Examples of computer-readable media that can be used to store instructions, information used, and/or information created during methods according to the described examples include magnetic or optical disks, flash memory, non-volatile memory provided USB devices, network storage devices, etc.

An apparatus for implementing processes and methods in accordance with these disclosures may comprise hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the code or code segments (eg, a computer program product) that perform the necessary tasks may be stored on a computer-readable or machine-readable medium. Processor(s) can perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mount devices, stand-alone devices, and the like. The functions described herein may also be embodied in peripheral devices or add-on cards. By way of further example, such functionality may also be implemented on circuit boards between different dies, or on different processes performed in a single device.

Instructions, media for carrying such instructions, computing resources for executing such instructions, and other structures for supporting such computing resources are exemplary means for providing the functionality described herein.

In the foregoing description, aspects of the invention have been described with reference to particular embodiments thereof, but those skilled in the art will recognize that the invention is not limited thereto. Therefore, while illustrative embodiments of the present invention have been described in detail herein, it is to be understood that the inventive concepts may be embodied and used in various ways and the appended claims are intended to be construed to cover such variations except as provided by prior art beyond the limits. The various features and aspects of the applications described above can be used alone or in combination. Furthermore, the embodiments may be used in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the description. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive. For purposes of illustration, the methods are described in a particular order. It should be understood that, in alternative embodiments, the methods may be performed in an order different than that described.

Those of ordinary skill will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, and Do not depart from the scope of this manual.

Where an element is described as being "configured" to perform certain operations, the operations may be performed, for example, by designing an electronic circuit or other hardware, by programming an electronic circuit (such as a microprocessor, or other suitable electronic circuitry) to perform the operations, or any combination thereof to achieve such a configuration.

The phrase "coupled to" refers to any element that is directly or indirectly physically connected to another element, and/or communicates directly or indirectly with another element (for example, via a wired or wireless connection and/or other suitable communication interface to any element of another element).

"At least one of" and/or "one or more" of a set enumerated in claim language or otherwise indicates that a member of the set or members of the set (in any combination) satisfy the claim . For example, "at least one of A and B" or "at least one of A or B" listed in the claim language refers to A, B, or A and B. In another example, "at least one of A, B, and C" or "at least one of A, B, or C" listed in the claim language refers to A, B, C, or A and B, or A and C, or B and C, or A and B and C. "At least one of" the language set and/or "one or more" of the set does not limit the set to the projects listed in the set. For example, "at least one of A and B" or "at least one of A or B" listed in the claim language may mean A, B, or A and B, and may additionally include those not listed in the set of A and B project.

The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative elements, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. This technology can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device mobile phones, or integrated circuit devices with multiple uses, including applications in wireless communication device mobile phones and other devices. Any features described as modules or elements may be implemented together in an integrated logic device or separately as separate but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a computer-readable data storage medium comprising program code, including instructions, which, when executed, perform the methods, algorithms and/or operations described above one or more of . A computer readable data storage medium may form a part of a computer program product, which may include packaging materials. Computer readable media may include memory or data storage media such as random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read only memory (ROM), non-volatile RAM Access memory (NVRAM), electronically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the techniques may be implemented at least in part by a computer-readable communication medium that carries or transmits program code in the form of instructions or data structures that can be accessed, read by the computer, fetch and/or execute, such as a propagated signal or wave.

The code is executable by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable Logic array (FPGA) or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this application. A general-purpose processor can be a microprocessor, but in the alternative, the processor can be any well-known processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term "processor," as used herein may refer to any of the foregoing structure, any combination of the foregoing, or any other structure or device suitable for implementation of the techniques described herein.

Illustrative aspects of the case include:

Aspect 1: An apparatus for outputting information associated with at least one input option, comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor being configured to: receiving data identifying one or more input options associated with a device in a scene; determining (including using at least one memory) information related to at least one of a scene, a device, and a user associated with a device; and based on One or more input options and information, outputting user guidance data corresponding to the input options for which relevant contextual information has been determined.

Aspect 2: The device of Aspect 1, wherein in order to receive data identifying one or more input options, the at least one processor is configured to: perform object recognition in the scene to identify one or more Enter options.

Aspect 3: The device of Aspect 2, wherein the object is at least one of within a threshold proximity to the user and within a field of view (FOV) of the user.

Aspect 4: The device of any of Aspects 1 to 3, wherein the at least one processor is configured to: detect one or more additional devices in the scene; a confidence value for the device or the one or more additional device interactions; and predicting a user interaction with the device in response to the determined confidence value exceeding a threshold.

Aspect 5: The apparatus of Aspect 4, wherein the at least one processor is configured to: filter content associated with one or more additional devices in response to a predicted user interaction with the device; and output device-related content .

Aspect 6: The apparatus of any of Aspects 4 to 5, wherein at least one processor is configured to identify which of the device and the one or more additional devices is predicted to be interacted with by the user.

Aspect 7: The device of any of Aspects 4 to 6, wherein the at least one processor is configured to simplify presentation of user interface content to avoid at least one of content overload, clutter, and content clutter.

Aspect 8: The apparatus of any of Aspects 1 to 7, wherein the device includes a connected device with network communication capabilities, and the at least one processor is configured to: based on the information and one or more input options , determining a gesture representing a predicted user interaction comprising predicted user input to the device; and presenting user guidance data, wherein the user guidance data includes an indication of the gesture that was detected when Actual user input is elicited at the device when detected.

Aspect 9: The device of any one of Aspects 1 to 8, wherein the user guidance data includes user input elements associated with input options, virtual overlays on physical objects associated with input options, and instructions on how to At least one of the prompts for input associated with the input option is provided.

Aspect 10: The apparatus of Aspect 7, wherein the at least one processor is configured to: based on the information, predict a user interaction with the device; and based on the one or more input options and the predicted user interaction, present a corresponding User guidance information for input options.

Aspect 11: The apparatus of any of Aspects 1 to 10, wherein to present the user guidance material, the at least one processor is configured to render at a display associated with the apparatus configured to appear to be located on the device's A virtual overlay on the surface, the virtual overlay including a user interface element associated with the input option, wherein the user interface element includes a virtual user input object associated with the input option and configured to receive an input corresponding to the input option At least one of the visual indications of the physical control object on the input device.

Aspect 12: The device of any of Aspects 1 to 11, wherein the information includes at least one of a user's eye gaze and a user's gesture, the at least one processor configured to: Predicting user interaction with the device by looking at at least one of gaze and gesture of the user; after presenting the user guidance material, detecting actual user input associated with the input options, the actual user input representing the predicted user interacting; and transmitting commands to the device corresponding to actual user input associated with the input options.

Aspect 13: The device of any of Aspects 1 to 12, wherein to output the user guidance data corresponding to the input options, the at least one processor is configured to: display the user guidance data.

Aspect 14: The device of any of Aspects 1 to 13, wherein to output the user guidance data corresponding to the input options, the at least one processor is configured to: output audio data representing the user guidance data.

Aspect 15: The device of any of Aspects 1 to 14, wherein to output the user guidance data corresponding to the input options, the at least one processor is configured to: display the user guidance data; and output the user guidance data corresponding to the displayed Audio data associated with user guidance data.

Aspect 16: The apparatus of any of Aspects 1 to 15, wherein the at least one processor is configured to: receive from the device data identifying one or more input options associated with the device.

Aspect 17: The apparatus of any of Aspects 1 to 16, wherein the at least one processor is configured to: receive from the server data identifying one or more input options associated with the device.

Aspect 18: The device of any of Aspects 1 to 17, wherein the device has no external user interface for receiving one or more user inputs.

Aspect 19: The apparatus of any of Aspects 1 to 18, wherein at least one processor is configured to, based on the information, refrain from presenting additional user guidance material associated with the device.

Aspect 20: The device of any one of Aspects 1 to 19, wherein the device is an extended reality device.

Aspect 21: The device according to any one of Aspects 1 to 20, which also includes a display.

Aspect 22: The device of Aspect 21, wherein the display is configured to display at least user guidance material.

Aspect 23: The apparatus of any of Aspects 1 to 22, wherein the at least one processor is configured to: obtain user input associated with an input option after presenting the user-guidance material; and transmit to the device a corresponding Instructions entered by the user are configured to control one or more operations of the device.

Aspect 24: The apparatus of any of Aspects 1 to 23, wherein the information associated with at least one of the scene, the device, and the user includes predicted user interactions with the device, one or more of the users in the scene at least one of an action, characteristics associated with the user, historical information associated with the user and the device, user interface capabilities of the device, information associated with the device, and information associated with the scene.

Aspect 25: The apparatus of any of Aspects 1 to 24, wherein the at least one processor is configured to: detect one or more additional devices in the scene; or a confidence value for the likelihood of the interaction of the plurality of additional devices; and predicting user interaction in response to the determined confidence value exceeding a threshold.

Aspect 26: The device of any one of Aspects 1 to 25, wherein the at least one processor is also configured to: receive an acknowledgment of the user interaction data from the user; respond to the acknowledgment from the user, communicate with the device interactive.

Aspect 27: The device of Aspect 26, wherein the confirmation is an audio confirmation.

Aspect 28: The device of any of Aspects 26-27, wherein the confirmation is user input received at the device.

Aspect 29: A method for outputting information associated with at least one input option, comprising the steps of: receiving data identifying one or more input options associated with a device in a scene; determining (including using at least one memory entity) information related to at least one of a scene, a device, and a user associated with an electronic device; and based on one or more input options and information, outputting a user corresponding to an input option for which relevant context information has been determined guidance material.

Aspect 30: The method of Aspect 29, wherein receiving data identifying one or more input options includes: performing object recognition in the scene to identify one or more input options for manipulating the object.

Aspect 31: The method of Aspect 30, wherein the object is at least one of within a threshold proximity to the user and within a field of view (FOV) of the user.

Aspect 32: The method of any one of aspects 29 to 31, further comprising the steps of: detecting one or more additional devices in the scene; Confidence values for interactions with one or more additional devices; and predicting user interactions with the devices in response to the determined confidence values exceeding a threshold.

Aspect 33: The method of any of Aspects 29 to 32, further comprising the steps of: responsive to the predicted user interaction with the device, filtering content associated with one or more additional devices; and outputting device-related Content.

Aspect 34: The method of any of Aspects 29 to 33, further comprising the step of identifying which of the device and the one or more additional devices is predicted to be interacted with by the user.

Aspect 35: The method of Aspect 34, further comprising the step of simplifying presentation of user interface content to avoid at least one of content overload, clutter and content confusion.

Aspect 36: The method of any of Aspects 29 to 35, wherein the user guidance data includes user input elements associated with the input options, virtual overlays on physical objects associated with the input options, and instructions on how to At least one of the prompts for input associated with the input option is provided.

Aspect 37: The method of any one of Aspects 29 to 36, further comprising the steps of: predicting a user interaction with the device based on the information; and presenting a corresponding User guidance information for input options.

Aspect 38: The method of Aspect 37, wherein the device includes a connected device having network communication capabilities, and the method further comprises the step of: based on the information and the one or more input options, determining a user interaction representing a prediction gestures, the predicted user interaction comprising predicted user input to the device; and presenting user guidance data, wherein the user guidance data comprises indications of gestures which, when detected, trigger actual User input.

Aspect 39. The method of any of Aspects 29 to 38, wherein presenting the user guidance material comprises: rendering at a display associated with the electronic device a virtual overlay configured to appear to be located on a surface of the device, The virtual overlay includes a user interface element associated with the input option, wherein the user interface element includes a virtual user input object associated with the input option and a physical control on the device configured to receive input corresponding to the input option At least one of the visual indications of the object.

Aspect 40: The method of any one of aspects 29 to 39, wherein the information includes at least one of the user's eye gaze and the user's gesture, the method further comprising the step of: based on the user's eye gaze and at least one of the user's gestures to predict user interaction with the device; after presenting the user guidance material, detecting actual user input associated with the input options, the actual user input representing the predicted user interaction; and transmitting to the device a command corresponding to the actual user input associated with the input option.

Aspect 41: The method of any of Aspects 29 to 40, wherein outputting the user guidance data corresponding to the input options comprises: displaying the user guidance data.

Aspect 42: The method of any of Aspects 29 to 41, wherein outputting user guidance data corresponding to the input options includes outputting audio data representing the user guidance data.

Aspect 43: The method of any of Aspects 29 to 42, wherein outputting user guidance data corresponding to the input options comprises: displaying the user guidance data; and outputting audio associated with the displayed user guidance data material.

Aspect 44: The method of any of Aspects 29 to 43, further comprising the step of receiving from the device data identifying one or more input options associated with the device.

Aspect 45: The method of any one of Aspects 29 to 44, further comprising the step of: receiving from a server data identifying one or more input options associated with the device.

Aspect 46: The method of any of Aspects 29-45, wherein the device has no external user interface for receiving one or more user inputs.

Aspect 47: The method of any one of Aspects 29 to 46, further comprising the step of refraining from presenting additional user guidance material associated with the device based on the information.

Aspect 48: The method of any one of Aspects 29 to 47, further comprising the steps of: after presenting the user guidance material, obtaining user input associated with the input option; and transmitting to the device a corresponding instructions configured to control one or more operations of a device.

Aspect 49: The method of any of Aspects 29 to 48, wherein the information related to at least one of the context, the device, and the user includes predicted user interactions with the device, one or more of the users in the context at least one of an action, characteristics associated with the user, historical information associated with the user and the device, user interface capabilities of the device, information associated with the device, and information associated with the scene.

Aspect 50: The method of any one of Aspects 29 to 49, further comprising the steps of: detecting one or more additional devices in the scene; a confidence value for the likelihood of device interaction; and predicting a user interaction in response to the determined confidence value exceeding a threshold.

Aspect 51: The method according to any one of aspects 29 to 50, further comprising the steps of: receiving confirmation from the user of the user interaction data; and interacting with the device in response to the confirmation from the user.

Aspect 52: The method of Aspect 51, wherein the confirmation is an audio confirmation.

Aspect 53: The method of any of Aspects 51 to 52, wherein the confirmation is user input received at the device.

Aspect 54: A non-transitory computer-readable medium having stored thereon instructions which, when executed by the one or more processors, cause the one or more processors to perform any of aspects 29-53. One method.

Aspect 55: An apparatus comprising means for performing the method according to any one of aspects 29-53.

0~24: button 100: XR system 102: Image sensor 104: Accelerometer 106: Gyroscope 107: storage device 108: Input device 109: Display 110: Computing element 112: CPU 114: Graphics Processing Unit (GPU) 116:DSP 118:ISP 120: XR engine 122:Enter option engine 123:Context management engine 124: Image processing engine 126: Rendering engine 200: hands 230: thumb 231: round 232: index finger 233: Rhombus 234: middle finger 235:mark point 236: ring finger 238: little finger 240:Legend 300: Extended Reality Systems 301: user 302: image sensor 303: object 304: plane 309: display 400:XR system 405: hand 410: console 412: virtual content 414: text 500:XR system 510: remote control 511: TV 512: virtual data 600:XR system 605: hand 610: thermostat 612: Gesture command 614: Gesture command 616: Gesture command 700:XR system 701: user 710:Digital photo frame 712:Metadata 800:XR system 802: user 810:Digital photo frame 812: remote control 814:Digital thermostat 820: data 822: data 824: data 840: function table 900: process 902: block 904: block 906: block 1000: computing system 1005: connect 1010: Processor 1012: cache memory 1015: System memory 1020: Read-only memory (ROM) 1025: random access memory (RAM) 1030: storage equipment 1032: Service 1034: Service 1035: output device 1036: Service 1040: communication interface 1045: input device

Illustrative embodiments of the present case are described in detail below with reference to the following drawings:

FIG. 1 is a block diagram illustrating an exemplary extended reality (XR) system according to some examples of the present disclosure;

2 is a diagram illustrating exemplary landmark points of a hand that may be used to track the position of the hand and the interaction of the hand with a virtual environment, according to some examples of the present disclosure;

3 is a diagram illustrating an example of an XR system worn by a user, according to some examples of the present disclosure;

4A, 4B, and 4C are diagrams illustrating examples of a user interacting with a console of an elevator using an XR system, according to some examples of the present disclosure;

5A and 5B are diagrams illustrating examples of a user using an XR system to interact with a remote control that can control a television, according to some examples of the present disclosure;

6A and 6B are diagrams illustrating examples of user interaction with a thermostat using an XR system, according to some examples of the present disclosure;

7A and 7B are diagrams illustrating examples of a user using an XR system that can determine whether to provide user interface input options for interacting with a picture frame, according to some examples of the present disclosure;

8A-8D are diagrams illustrating examples of a user interacting with one or more devices using an augmented reality system when multiple devices are present in a scene, according to some examples of the present disclosure;

9 is a flowchart illustrating an example of a process for presenting information associated with at least one input option, according to some examples of the present disclosure; and

Figure 10 illustrates an exemplary computing system according to some examples of the present disclosure.

Domestic deposit information (please note in order of depositor, date, and number) none Overseas storage information (please note in order of storage country, institution, date, and number) none

800:XR system

802: user

810:Digital photo frame

812: remote control

814:Digital thermostat

Claims (31)

An apparatus for outputting information associated with at least one input option, comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor configured to: receiving data identifying one or more input options associated with a device in a scene; including determining, using at least one memory, information associated with at least one of the scene, the device, and a user associated with the device; and Based on the one or more input options and the information, outputting user guidance data corresponding to an input option for which relevant contextual information has been determined. The device according to claim 1, wherein the user guidance data includes a user input element associated with the input option, a virtual overlay on a physical object associated with the input option and instructions on how to provide information related to the input Option at least one of a prompt associated with an input. The device according to claim 1, wherein the at least one processor is configured to: predicting a user interaction with the device based on the information; and Based on the one or more input options and the predicted user interaction, the user guidance material corresponding to the input options is presented. The apparatus according to claim 1, wherein the device comprises a connection device having network communication capabilities, and the at least one processor is configured to: determining a gesture representing a predicted user interaction comprising a predicted user input to the device based on the information and the one or more input options; and The user guidance data is presented, wherein the user guidance data includes an indication of the gesture that, when detected, causes an actual user input at the device. The device according to claim 1, wherein for presenting the user guidance material, the at least one processor is configured to: Rendering at a display associated with the device a virtual overlay configured to appear to be located on a surface of the device, the virtual overlay including a user interface element associated with the input option, wherein the use The user interface element includes at least one of a virtual user input object associated with the input option and a visual indication of a physical control object on the device configured to receive the input corresponding to the input option. The device according to claim 1, wherein the information includes at least one of an eye gaze of the user and a gesture of the user, the at least one processor being configured to: predicting a user interaction with the device based on at least one of the eye gaze of the user and the gesture of the user; after presenting the user guidance material, detecting an actual user input associated with the input option, the actual user input representing the predicted user interaction; and A command corresponding to the actual user input associated with the input option is transmitted to the device. The device according to claim 1, wherein in order to output the user guidance data corresponding to the input option, the at least one processor is configured to: Display the user guide information. The device according to claim 1, wherein in order to output the user guidance data corresponding to the input option, the at least one processor is configured to: Audio data representing the user guidance data is output. The device according to claim 1, wherein in order to output the user guidance data corresponding to the input option, the at least one processor is configured to: display the user guidance material; and Audio data associated with the displayed user guidance data is output. The device according to claim 1, wherein the at least one processor is configured to: Data identifying the one or more input options associated with the device is received from the device. The device according to claim 1, wherein the at least one processor is configured to: The data identifying the one or more input options associated with the device is received from a server. The device according to claim 1, wherein the device does not have an external user interface for receiving one or more user inputs. The device according to claim 1, wherein the at least one processor is configured to: Based on the information, rendering of additional user guidance material associated with the device is suppressed. The device according to claim 1, wherein the device is an extended reality device. The device according to claim 1 also includes a display. The device according to claim 15, wherein the display is configured to display at least the user guidance information. The device according to claim 1, wherein the at least one processor is configured to: obtaining a user input associated with the input option after presenting the user guidance material; and A command corresponding to the user input is transmitted to the device, the command being configured to control one or more operations of the device. A method for outputting information associated with at least one input option, comprising the steps of: receiving data identifying one or more input options associated with a device in a scene; including determining, using at least one memory, information associated with at least one of the scene, the device, and a user associated with an electronic device; and Based on the one or more input options and the information, outputting user guidance data corresponding to an input option for which relevant contextual information has been determined. The method according to claim 18, wherein the user guidance data includes a user input element associated with the input option, a virtual overlay on a physical object associated with the input option and instructions on how to provide information related to the input Option at least one of a prompt associated with an input. The method according to claim 18 also includes the following steps: predicting a user interaction with the device based on the information; and Based on the one or more input options and the predicted user interaction, the user guidance material corresponding to the input options is presented. The method according to claim 20, wherein the device includes a connection device with network communication capabilities, and the method also includes the following steps: determining a gesture representing a predicted user interaction comprising a predicted user input to the device based on the information and the one or more input options; and The user guidance data is presented, wherein the user guidance data includes an indication of the gesture that, when detected, causes an actual user input at the device. The method according to claim 18, wherein the step of presenting the user guidance information comprises the following steps: rendering at a display associated with the electronic device a virtual overlay configured to appear to be located on a surface of the device, the virtual overlay including a user interface element associated with the input option, wherein the The user interface element includes at least one of a virtual user input object associated with the input option and a visual indication of a physical control object on the device configured to receive the input corresponding to the input option. The method according to claim 18, wherein the information includes at least one of an eye gaze of the user and a gesture of the user, the method also comprising the steps of: predicting a user interaction with the device based on at least one of the eye gaze of the user and the gesture of the user; after presenting the user guidance material, detecting an actual user input associated with the input option, the actual user input representing the predicted user interaction; and A command corresponding to the actual user input associated with the input option is transmitted to the device. The method according to claim 18, wherein the step of outputting the user guidance data corresponding to the input option comprises the following steps: Display the user guide information. The method according to claim 18, wherein the step of outputting the user guidance data corresponding to the input option comprises the following steps: Audio data representing the user guidance data is output. The method according to claim 18, wherein the step of outputting the user guidance data corresponding to the input option comprises the following steps: display the user guidance material; and Audio data associated with the displayed user guidance data is output. The method according to claim 18 also includes the following steps: Data identifying the one or more input options associated with the device is received from the device. The method according to claim 18 also includes the following steps: The data identifying the one or more input options associated with the device is received from a server. The method according to claim 18, wherein the device has no external user interface for receiving one or more user inputs. The method according to claim 18 also includes the following steps: Based on the information, rendering of additional user guidance material associated with the device is suppressed. The method according to claim 18 also includes the following steps: obtaining a user input associated with the input option after presenting the user guidance material; and A command corresponding to the user input is transmitted to the device, the command being configured to control one or more operations of the device.

Images