KR102213725B1 - Tracking head movement when wearing mobile device - Google Patents

Abstract

A method for tracking the head position of an end user of an HMD relative to a head mounted display device (HMD) is described. In some embodiments, the HMD determines an initial head tracking vector associated with the end user's initial head position relative to the HMD, and calculates one or more head tracking vectors corresponding to the end user's one or more subsequent head positions relative to the HMD. It is also possible to determine and track the end user's head movement over time based on the initial head tracking vector and one or more head tracking vectors, and adjust the position of the virtual object displayed to the end user based on the head movement. have. In some embodiments, the resolution and/or number of virtual objects created and displayed to the end user may be modified based on the degree of head movement of the end user relative to the HMD.

Description

Tracking head movement when wearing a mobile device {TRACKING HEAD MOVEMENT WHEN WEARING MOBILE DEVICE}

In augmented reality (AR), perception of a real-world environment (or data representing a real-world environment) is augmented by using computer-generated virtual data. It relates to providing a modified augmented real-world environment. For example, data representing the real-world environment may be captured in the real-world environment using a sensory input device such as a camera or microphone, and augmented using computer-generated virtual data including virtual images and virtual sounds. May be. The virtual data may also include information relating to the real world environment, such as textual descriptions relating to real world objects in the real world environment. Objects in the AR environment may include real objects (ie, objects that exist in a specific real-world environment) and virtual objects (ie, objects that do not exist in a specific real-world environment).

In order to realistically integrate virtual objects into the AR environment, AR systems typically perform several tasks including mapping and localization. Mapping relates to the process of creating a map of the real world environment. Localization relates to the process of locating a specific point of view or pose on a map of the real world environment. In some cases, in order to determine a specific view associated with the mobile device that needs to be augmented as the mobile device moves within the real world environment, the AR system localizes the pose of the mobile device moving within the real world environment in real time. You may.

A technique for tracking the end user's head position of an HMD relative to a head-mounted display device (HMD) over time is described. In some embodiments, the HMD determines an end user's initial head position relative to the HMD, determines one or more subsequent head positions of the end user relative to the HMD, and over time based on the one or more subsequent head positions. It is also possible to track the movement of the end user's head and adjust the location of the virtual object displayed to the end user based on the movement of the end user's head. In some embodiments, the resolution and/or number of virtual objects created and displayed to the end user may be modified based on the degree of head movement of the end user relative to the HMD.

This summary is provided to introduce in a simplified form a selection of concepts that are further described in the detailed description below. This summary is not intended to identify major or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

1 is a block diagram of an embodiment of a networked computing environment in which the disclosed techniques may be practiced.
2A depicts an embodiment of a mobile device in communication with a second mobile device.
2B depicts an embodiment of a portion of an HMD.
Figure 2c is an implementation of a portion of an HMD in which a gaze vector extending to a point of gaze is used to align the far inter-pupillary distance (IPD). Describe the form.
2D depicts an embodiment of a portion of an HMD in which a gaze vector extending to a gaze point is used to align near pupillary distance (IPD).
3A depicts one embodiment of a portion of an HMD with a movable display optical system including a gaze detection element.
3B depicts an alternative embodiment of a portion of an HMD with a movable display optical system including a gaze detection element.
4A depicts an embodiment of an HMD worn by an end user.
4B depicts a pair of images taken using an inward facing camera integrated with an HMD.
4C depicts the identification of an end user's eye and corresponding feature point.
4D depicts a 3D model of an end user's eye positioned relative to the HMD.
4E depicts a second pair of images taken using an introverted camera integrated with an HMD.
4F depicts an end user's eye and identification of corresponding feature points.
4G depicts an updated 3D model of an end user's eye positioned relative to the HMD.
4H depicts one embodiment of a representation of a change in an end user's eye position over time.
5 is a flowchart illustrating one embodiment of a method for tracking an end user's head position of the HMD relative to the HMD and for adjusting the position of a virtual object based on the head position relative to the HMD.
6 is a flow chart illustrating an alternative embodiment of a method for tracking the head position of an end user of an HMD relative to the HMD.
7 is a block diagram of one embodiment of a mobile device.

A technique for tracking the head position of an end user of an HMD with respect to a head mounted display device (HMD) over time is described. In some embodiments, the HMD determines an initial head tracking vector associated with the end user's initial head position relative to the HMD, and one corresponding to one or more subsequent head positions of the end user relative to the HMD. Determine more than one head tracking vector, track the end user's head movement over time based on the initial head tracking vector and one or more head tracking vectors, and tell the end user based on the end user's head movement. You can also adjust the position of the displayed virtual object. In this case, the end user's head position (or eye position) is (e.g., due to the end user running on a bumpy road when the end user is wearing the HMD, or by accidentally touching the HMD). Thus) the HMD may adjust the position of the virtual object displayed to the end-user so that when moving relative to the initial placement for the HMD, the virtual object appears to be properly fixed or otherwise positioned within the augmented reality environment. . In some embodiments, the resolution and/or number of virtual objects created and displayed to the end user may be modified based on the degree of head movement of the end user relative to the HMD. In one embodiment, both the resolution and the number of virtual objects displayed to the end user are determined if the degree of head movement exceeds a threshold (e.g., the end user's head position is more than 4 centimeters relative to the HMD from the initial head position. If you move a lot), it may decrease. The HMD may also provide a notification (eg, display notification or audio notification) to the end user when the degree of head movement exceeds a threshold.

In some cases, the HMD may capture an image of the end user's eyes and/or the end user's facial area surrounding the eyes while the end user is viewing the augmented reality environment using the HMD. The image may be captured using one or more introspective cameras (eg, eye tracking cameras) integrated with the HMD. Since the end user's eyes are fixed with respect to their head position, a fixed feature point associated with the end user's eye may be used to determine the head position relative to the end user. The HMD is characterized by the inner canthus (i.e. the inner corner or portion of the eye closest to the nasal bridge where the upper and lower eyelids meet) and the outer canthus (i.e., the upper and lower eyelids). Image processing techniques may be applied to the captured image to detect and track eye features, such as the outer corner or part of the eye furthest from the nose, where the eyelids meet. Other eye features such as the pupil and iris may also be identified and tracked to determine the eye position relative to the HMD. When a feature point related to one or both eyes of the end user's eyes is identified, position and orientation information related to the 3D model of the end user's eye may be updated based on the feature point, or the end user's A head position vector (represented as a 2D or 3D vector within the coordinate space for the HMD) may be determined based on the 3D model of the eye. To determine the position and/or orientation of the end user's head (or eyes) relative to the HMD, the HMD may also detect and track fixed facial features, such as the end user's nose.

In one embodiment, the absolute head position vector representing the end user's absolute head position vector with respect to the environment in which the HMD exists is determined by determining a pose vector related to the pose of the HMD within the environment. It may be determined by adding a vector offset based on the end user's head position vector for the next HMD. The absolute head position vector may be transmitted by the HMD to a server (eg, associated with a retail store) or may be used by an application running on the HMD (eg, a gaming application). In some embodiments, the HMD determines a pose of six degrees of freedom (6DOF) corresponding to the image of the environment captured from an outward facing camera, that image relative to the 3D map of the environment. To do, you can also use it. The 6DOF pose may include information related to the position and orientation of the HMD in the environment. The 6DOF pose may be used to localize the HMD and to create a virtual object, such that the virtual object appears to exist at an appropriate location in the environment. More information on determining the 6DOF pose can be found in US patent application Ser. No. 13/152,220, "Distributed Asynchronous Localization and Mapping for Augmented Reality." More information on performing pose estimation and/or localization for mobile devices can be found in US Patent Application Ser. No. 13/017,474, "Mobile Camera Localization Using Depth Maps."

1 is a block diagram of one embodiment of a networked computing environment 100 in which the disclosed techniques may be practiced. Networked computing environment 100 includes a plurality of computing devices interconnected through one or more networks 180. One or more networks 180 allow certain computing devices to connect and communicate with other computing devices. The computing devices depicted include a mobile device 11, a mobile device 12, a mobile device 19, and a server 15. In some embodiments, the plurality of computing devices may include other computing devices not shown. In some embodiments, the plurality of computing devices may include more or less than the number of computing devices shown in FIG. 1. One or more networks 180 include secure networks such as enterprise private networks, non-secure networks such as wireless open networks, local area networks (LANs), wide area networks (WANs), and the Internet. You may. Each network of one or more networks 180 may include a hub, a bridge, a router, a switch, and a wired transmission medium such as a wired network or a direct-wired connection.

Server 15, which may include a supplemental information server or application server, relates to the client downloading information (e.g., text, audio, image, and video files) from the server or to specific information stored on the server. You can also allow you to perform search queries that are In general, a "server" may include a hardware device acting as a host in a client-server relationship, or a software process that shares resources with one or more clients or performs tasks for one or more clients. Communication between computing devices in a client-server relationship may be initiated by a client sending a request to a server requesting access to a particular resource or a particular task to be performed. Subsequently, the server may perform the requested action or send a response back to the client.

One embodiment of the server 15 includes a network interface 155, a processor 156, a memory 157, and a translator 158, all of which communicate with each other. Network interface 155 allows server 15 to connect to one or more networks 180. The network interface 155 may include a wireless network interface, a modem, and/or a wired network interface. Processor 156 allows server 15 to execute computer readable instructions stored in memory 157 to perform the processes discussed herein. The converter 158 performs mapping logic for converting the first file of the first file format to a corresponding second file of the second file format (ie, the second file may be a converted version of the first file). It can also be included. The converter 158 may be configured using a file mapping command that provides a command for mapping a file (or a part thereof) of the first file format to a corresponding file of the second file format.

One embodiment of mobile device 19 includes network interface 145, processor 146, memory 147, camera 148, sensor 149, and display 150, all of which communicate with each other. do. Network interface 145 allows mobile device 19 to connect to one or more networks 180. The network interface 144 may include a wireless network interface, a modem, and/or a wired network interface. Processor 146 allows mobile device 19 to execute computer readable instructions stored in memory 147 to perform the processes discussed herein. The camera 148 may capture a color image and/or a depth image of the environment. The mobile device 19 may include an outward camera for capturing an image of the environment and an inward camera for capturing an image of an end user of the mobile device. Sensor 149 may generate motion and/or direction information related to mobile device 19. In some cases, sensor 149 may include an inertial measurement unit (IMU). Display 150 may display digital images and/or video. Display 150 may also include a see-through display.

In some embodiments, various components of mobile device 19 including network interface 145, processor 146, memory 147, camera 148 and sensor 149 may be integrated on a single chip substrate. have. In one example, the network interface 145, processor 146, memory 147, camera 148, and sensor 149 may be integrated as a system on a chip (SOC). In other embodiments, the network interface 145, the processor 146, the memory 147, the camera 148, and the sensor 149 may be integrated into a single package.

In some embodiments, mobile device 19 provides a natural user interface (NUI) by utilizing gesture recognition software running on camera 148, sensor 149, and processor 146. You may. Through the natural user interface, a person's body parts and movements may be detected, interpreted, and used to control various aspects of a computing application. In one example, a computing device that utilizes a natural user interface may infer the intention of a person interacting with the computing device (e.g., that the end user performed a specific gesture to control the computing device). .

Networked computing environment 100 may provide a cloud computing environment for one or more computing devices. Cloud computing refers to Internet-based computing, in which shared resources, software, and/or information are provided on-demand to one or more computing devices via the Internet (or other global network). The term "cloud" is used as a metaphor for the Internet, based on a cloud (cloud) picture used in computer networking diagrams to describe the Internet as an abstract concept of the underlying infrastructure it represents.

In one example, the mobile device 19 includes an HMD that provides an augmented reality environment or a mixed reality environment to an end user of a head mounted display device (HMD). The HMD may include video see-through and/or optical see-through systems. The optical see-through HMD being worn by the end user may allow real, direct viewing of the real-world environment (eg, through a transparent lens), while at the same time allowing the virtual object's view into the end user's field of view. By projecting an image, the real world environment recognized by the end user may be augmented using a virtual object.

Using the HMD, the end-user may wear the HMD to move around a real-world environment (eg, a living room) or see a view of the real world in which images of virtual objects are superimposed. Virtual objects may appear to maintain a coherent spatial relationship with the real-world environment (i.e., when the end-user turns or moves their head within the real-world environment, the virtual object is perceived by the end-user. The image displayed to the end user will change so that it appears to exist within the real-world environment as it is). Virtual objects may also appear fixed to the end user's perspective (e.g., the top right of the end user's point of view, regardless of how the end user turns or moves their heads within the real world environment). Virtual menu always visible in the corner). In one embodiment, environment mapping of the real world environment may be performed by the server (ie, on the server side) while camera localization may be performed on the mobile device 19 (ie, on the client side). Virtual objects may include textual descriptions associated with real world objects.

In some embodiments, a mobile device, e.g., mobile device 19, may communicate with a server in the cloud, e.g., server 15, and tells the server location information associated with the mobile device (e.g., mobile device via GPS coordinates). The location of the device) and/or image information (eg, information about an object detected within the field of view of the mobile device) may be provided. In response, the server may transmit one or more virtual objects to the mobile device based on location information and/or image information provided to the server. In one embodiment, the mobile device 19 may specify a specific file format for receiving one or more virtual objects and the server 15 may specify one or more virtual objects embodied within a file of the specific file format. ).

In some embodiments, the HMD, such as mobile device 19, may provide an augmented reality environment including virtual objects to end users of the HMD. Periodically (e.g., every 0.5 ms), the HMD determines the position of the end user's head relative to the HMD, and the virtual object appears to be stationary within the augmented reality environment even if the position of the end user's head relative to the HMD is changed. Thus, the position of the image associated with the virtual object displayed to the end user may be adjusted. The ability to adjust the position of the image relative to the virtual object based on the position of the end user's head relative to the HMD is (for example, the end user is running on a bumpy road when the end user is wearing the HMD. The HMD provides a consistent augmented reality experience to the end user, even if the end user's eye position may move relative to the HMD's see-through display over time (due to or due to accidentally touching the HMD). Allow that.

2A depicts an embodiment of a mobile device 19 in communication with a second mobile device 5. Mobile device 19 may include a see-through HMD. As depicted, the mobile device 19 communicates with the mobile device 5 via a wired connection 6. However, the mobile device 19 may also communicate with the mobile device 5 via a wireless connection. Mobile device 5 provides virtual object information and other data that may be used to offload computationally intensive processing operations (e.g., rendering of virtual objects) and to provide an augmented reality environment on mobile device 19 It may be used by the mobile device 19 to store it. The mobile device 5 may also provide motion and/or direction information related to the mobile device 5 to the mobile device 19. In one example, the motion information may include a velocity and acceleration associated with the mobile device 5 and the direction information may include an Euler angle, wherein the Euler angle is centered on a specific coordinate system or frame of reference. Provides rotation information. In some cases, the mobile device 5 may comprise a motion and direction sensor, such as an inertial measurement unit (IMU), to obtain motion and/or rotation information associated with the mobile device 5.

2B depicts an embodiment of a portion of an HMD such as mobile device 19 in FIG. 1. Only the right side of the HMD 200 is depicted. The HMD 200 includes a right temple 202, a nose bridge 204, an eyeglass 216, and a frame 214. The right temple 202 includes a capture device 213 (eg, a front camera and/or microphone) in communication with the processing unit 236. The capture device 213 may include one or more cameras for recording digital images and/or video and may transmit visual recordings to the processing unit 236. One or more cameras may capture color information, IR information, and/or depth information. The capture device 213 may also include one or more microphones for recording sound and may transmit the audio recording to the processing unit 236.

The right temple 202 also includes a biometric sensor 220, an eye tracking system 221, an earphone 230, a motion and direction sensor 238, a GPS receiver 232, a power supply 239, and a wireless interface. 237, all of which communicate with the processing unit 236. The biometric sensor 220 may include one or more electrodes for determining a pulse or heart rate associated with the end user of the HMD 200 and a temperature sensor for determining a body temperature associated with the end user of the HMD 200 have. In one embodiment, the biometric sensor 220 includes a sensor for measuring a pulse that presses the temple of the end user. The motion and direction sensor 238 may include a three-axis magnetometer, a three-axis gyro, and/or a three-axis accelerometer. In one embodiment, the motion and direction sensor 238 may comprise an inertial measurement unit (IMU). The GPS receiver may determine a GPS location associated with the HMD 200. The processing unit 236 may include one or more processors and a memory for storing computer readable instructions to be executed on the one or more processors. The memory may also store other types of data to be executed on one or more processors.

In one embodiment, the system 221 for eye tracking may include one or more inward cameras. In other embodiments, the system 221 for eye tracking may include an eye tracking illumination source and an associated image sensor for eye tracking. In one embodiment, the illumination source for eye tracking is one or more infrared (IR) emitters, such as an infrared light emitting diode (LED) or laser, emitting a center of a predetermined IR wavelength or a range of wavelengths. For example, VCSEL) may be included. In some embodiments, the eye tracking sensor may include an IR position sensitive detector (PSD) or an IR camera to track the glint position. For more information on eye tracking systems, see US Pat. No. 7,401,920 entitled "Head Mounted Eye Tracking and Display System" issued July 22, 2008, and filed September 26, 2011. The invention may be found in US patent application Ser. No. 13/245,700 entitled “Integrated Eye Tracking and Display System”.

In one embodiment, eyeglasses 216 may include a see-through display, whereby an image generated by processing unit 236 may be projected and/or displayed on the see-through display. The capture device 213 may be calibrated such that the field of view captured by the capture device 213 corresponds to the field of view seen by the end user of the HMD 200. The earphone 230 may be used to output sound related to the projected image of the virtual object. In some embodiments, HMD 200 may include two or more front cameras (e.g., one for each temple) to obtain depth from stereoscopic information related to the field of view captured by the front camera. have. Two or more front cameras may also include 3D, IR, and/or RGB cameras. Depth information may also be obtained from a single camera utilizing depth from motion technology. For example, two images related to two spatially different points at different points in time may be obtained from a single camera. Then, given the location information for two different points spatially, a parallax calculation may be performed.

In some embodiments, the HMD 200 uses a gaze detection element and a three-dimensional coordinate system with respect to one or more components of the human eye, such as a corneal center, a center of eye rotation, or a pupil center. It is also possible to perform gaze detection for each eye of. Gaze detection may identify where the end user is focusing within the field of view. Examples of the gaze detection element may include a glint generating illuminator and a sensor for capturing data indicative of the reflected glint. In some cases, the center of the cornea may be determined based on two glints using planar geometry. The center of the cornea connects the center of the pupil and the center of rotation of the eye, which may be treated as a fixed location to determine the optical axis of the end user at a given gaze or viewing angle.

2C depicts an embodiment of a portion of the HMD 2 in which a gaze vector extending to the gaze point is used to align the distant pupillary distance (IPD). HMD 2 is an example of a mobile device such as mobile device 19 in FIG. 1. As depicted, the gaze vectors 180l and 180r intersect at a gaze point remote from the end user (i.e., the gaze vectors 180l and 180r do not intersect when the end user is looking at a distant object). A model of the eye for the eyes 160 and 160r is illustrated for each eye based on the Gullstrand method eye model. Each eye includes a cornea 168 that is modeled as a sphere with a center of rotation 166 and a sphere with a center 164. The cornea 168 rotates with the eyeball, and the center of rotation 166 of the eyeball may be treated as a fixed point. The cornea 168 covers the iris 170 with the pupil 162 at its center. The glints 174 and 176 are on the surface 172 of each cornea.

As depicted in FIG. 2C, the sensor detection regions 139 (ie, 139l and 139r, respectively) are aligned with the optical axis of each display optical system 14 in the spectacle frame 115. In one example, the sensor associated with the detection area is image data representing glints 174l and 176l generated by illuminators 153a and 153b on the left side of the frame 115, respectively, and an illuminator on the right side of the frame 115. It may also include one or more cameras capable of capturing data representing glints 174r and 176r generated by 153c and 153d, respectively. Through the display optical systems 14l and 14r of the eyeglass frame 115, the end user's field of view includes both real objects 190, 192 and virtual objects 182 and 184.

The axis 178 formed from the center of rotation 166 to the pupil 162 through the corneal center 164 forms the optical axis of the eye. The gaze vector 180 may also be referred to as a line of sight or visual axis extending from the fovea through the center of the pupil 162. In some embodiments, the optical axis is determined and a small correction is determined through user calibration to obtain a visual axis selected as the gaze vector. For each end user, the virtual object may be displayed by the display device at a number of predetermined positions in different horizontal and vertical positions. The optical axis may be calculated for each eye at each location during display of the object, or a ray may be modeled as extending from that location to the user's eye. The gaze offset angle with horizontal and vertical components may be determined based on how much the optical axis must be shifted to align with the modeled ray. From the difference position, an average offset angle with a horizontal or vertical component can be selected as a small correction to be applied to each calculated optical axis. In some embodiments, for gaze offset angle correction, only the horizontal component is used.

As depicted in FIG. 2C, the gaze vectors 180l and 180r are not completely parallel as the vectors become closer to each other as the vectors extend from the eyeball to the field of view of the gaze point. In each display optical system 14, the gaze vector 180 appears to intersect the optical axis centering the sensor detection area 139. In this configuration, the optical axis is aligned with the pupillary distance (IPD). When the end user is looking straight ahead, the measured IPD is also referred to as the remote IPD.

2D depicts an embodiment of a portion of the HMD 2 in which a gaze vector extending to the gaze point is used to align the near pupillary distance (IPD). HMD 2 is an example of a mobile device such as mobile device 19 in FIG. 1. As depicted, the cornea 168l of the left eye is rotated to the right or towards the end user's nose, and the cornea 168r of the right eye is rotated to the left or towards the end user's nose. . Both pupils are staring at the real object 194 within a certain distance of the end user. The gaze vectors 180l and 180r from each eye enter the Panum's fusional region 195 where the real object 194 is located. The fusion area of panium is the area of single vision in a binocular viewing system, such as a system of human vision. The intersection of the gaze vectors 180l and 180r indicates that the end user is looking at the real object 194. At this distance, since the eyeball is rotated inward, the distance between the pupils of the eyeball decreases to the nearest IPD. The near IPD is typically about 4 mm smaller than the far IPD. The near IPD distance criterion (eg, a gaze point less than 4 feet from the end user) may be used to convert or adjust the alignment of the IPD of the display optical system 14 to that of the near IPD. In the case of a near IPD, each display optical system 14 may be moved towards the end user's nose, and thus the optical axis, and detection area 139, as indicated by the detection areas 139ln and 139rn. Moves a few millimeters toward the nose.

For more information on determining the IPD for the end user of the HMD and adjusting the display optical system accordingly, see a US patent application filed September 30, 2011 entitled "Personal Audio/Visual System". Can be found in 13/250,878.

3A depicts an embodiment of a portion of an HMD 2 with a movable display optical system including a gaze detection element. What appears as a lens for each eye represents the display optical system 14 (ie, 14l and 14r) for each eye. The display optical system includes optical elements (e.g., mirrors, filters) and a see-through lens for seamlessly fusing virtual content with a real, direct real-world view seen through the lens of the HMD. The display optical system 14 has an optical axis that is generally at the center of a see-through lens where light is generally collimated to provide a distortion-free view. For example, when an eye care professional fits ordinary glasses to the user's face, the glasses generally have a clear or distortion-free view with each pupil aligned with the center or optical axis of each lens. The glasses are fitted to the end user's nose at a position where the generally collimated light for the eye will reach the end user's eye.

As depicted in Fig. 3A, the detection areas 139r, 139l of at least one sensor are each of their respective display optical systems 14r, 14l such that the centers of the detection areas 139r, 139l capture light along the optical axis. ) Is aligned with the optical axis of. When the display optical system 14 is aligned with the end user's pupil, each detection area 139 of each sensor 134 is aligned with the end user's pupil. The reflected light of the detection area 139 is transmitted through one or more optical elements to the actual image sensor 134 of the camera, which, in the depicted embodiment, is inside the frame 115 and is dotted with a dotted line. It is illustrated. In some embodiments, sensor 134 may include an image sensor or RGB camera for capturing an image of an end user's eye or facial features surrounding the eye. Other introverted image sensors may also be integrated with frame 115 to capture fixed facial features such as the nose bridge associated with the end user.

In one embodiment, at least one sensor 134 may be a visible light camera (eg, an RGB or color camera). In one example, the optical element or element for light detection includes a partially transmissive and partially reflective visible light reflecting mirror. The visible light camera provides image data of the pupil of the eye of the end user, while the IR photodetector 152 captures glints, which are reflections in the IR portion of the spectrum. If a visible light camera is used, the reflection of the virtual image may appear in the eye data captured by the camera. If desired, image filtering techniques may be used to remove virtual image reflections. IR cameras may not be able to detect virtual image reflections from the eye.

In another embodiment, at least one sensor 134 (ie, 134l and 134r) is an IR camera or position sensitive detector (PSD) to which IR radiation may be directed. The IR radiation reflected from the eye may come from incident radiation from the illuminator 153, another IR illuminator (not shown), or may come from ambient IR radiation reflected from the eye. In some cases, the sensor 134 may be a combination of RGB and IR cameras, and the light directing element may include an element for reflecting or converting visible light and an element for reflecting or converting IR radiation. . In some cases, sensor 134 may be embedded within the lens of system 14. Additionally, in order to reduce any confusion to the user by making the camera well matched to the user's field of view, an image filtering technique may be applied.

As depicted in FIG. 3A, paired with a photodetector 152 is provided by a barrier 154 to prevent interference between the incident light generated by the illuminator 153 and the reflected light received by the photodetector 152. There are four separate sets of illuminators 153. In order to avoid unnecessary confusion in the drawings, reference numerals are shown for representative pairs. Each illuminator may be an infrared (IR) illuminator that generates a narrow beam of light near a predetermined wavelength. Each of the photodetectors may be selected to capture light near a predetermined wavelength. Infrared rays may also include near infrared rays. Because there may be a wavelength drift of the illuminator or photodetector, or a small range around a certain wavelength may be tolerated, the illuminator and photodetector may have a tolerance range with respect to the wavelength for generation and detection. In some embodiments where the sensor is an IR camera or an IR position sensing detector (PSD), the photodetector may include an additional data capture device to monitor the operation of the illuminator, e.g., wavelength drift, beam width change, etc. It can also be used. The photodetector may also provide glint data to the visible light camera as sensor 134.

As depicted in Fig. 3A, each display optical system 14 and its own device of the gaze detection element facing each eye (e.g., camera 134 and its detection area 139, illuminator 153 ), and the photodetector 152 are located on the movable inner frame portions 117l and 117r. In this example, the display adjustment mechanism includes one or more motors 203 having a shaft 205 attached to the inner frame portion 117, the inner frame portion 117 being driven by the motor 203. It slides from left to right within the frame 115 under the guidance and power of the shaft 205 or vice versa. In some embodiments, one motor 203 may drive both inner frames.

3B depicts an alternative embodiment of a portion of an HMD 2 with a movable display optical system including a gaze detection element. As depicted, each display optical system 14 is enclosed in a separate frame portion 115l, 115r. Each of the frame portions may be moved individually by a motor 203. In some embodiments, at least one sensor 134 may include an image sensor or RGB camera for capturing an image of an end user's eye or facial features surrounding the eye. Other introverted image sensors may be integrated with the frame 115 to capture fixed facial features such as the nose bridge associated with the end user.

4A depicts one embodiment of an HMD 402 worn by an end user. The HMD 402 may include a mobile device, such as mobile device 19 in FIG. 1, and one or more introverted cameras for capturing the eye of an end user or facial features surrounding the eye. 4B depicts a pair of images taken using an inward facing camera integrated with HMD 402. Image 406 includes information corresponding to the eye 404 of the end user. A pair of images are depicted, but the head position information associated with the end user is a single image (e.g., image 406) or more than two images (e.g., capturing portions of the end user's eyes). May be determined using a set of four images). In some cases, the head position information may be determined by averaging each of the pairs of images 406-407 and corresponding head position determinations. The pair of images 406-407 may correspond to the end user's initial head position.

4C depicts the identification of a feature point including an end user's eye 404 and corresponding feature points 412, 414, 415, and 416. The feature point 412 may correspond to the outer canthal angle of the right eye of the end user, the feature point 414 may correspond to the inner canthal angle of the right eye of the end user, and the feature point 415 may correspond to the pupil center of the right eye of the end user. May be. The outer canthal and the inner canthal may be identified over time, even if the end-user squints (or blinks) their eyes. The feature point 416 may correspond to an inflection point (or maximum curvature point) of the upper eyelid of the end user. By applying image processing techniques to the image 406, other feature points may also be identified, including the inflection point (or minimum curvature point) of the lower eyelid of the end user, the iris of the end user, and the nose of the end user.

4D depicts a 3D model of an end user's eye positioned relative to the HMD 402. Each of the end user's eyes may be modeled as a sphere 418-419 or a spheroid positioned within the three-dimensional coordinate space for the HMD. As depicted, the first sphere 418 may correspond to the right eye of the end user and the second sphere 419 may correspond to the left eye of the end user. The positions of the spheres 418-419 within the three-dimensional coordinate space may be determined based on the feature points 412, 414, 415, and 416. Also, since the end-user's eye position is fixed relative to their head, the IPD between the two spheres (418-419) may be determined during the initialization procedure (the first time the end user wears the HMD) and within the three-dimensional coordinate space. It may also be used to position the spheres 418-419 at.

4E depicts a second pair of images 408-409 captured using an introverted camera integrated with HMD 402. Image 408 includes information corresponding to the eye 404 of the end user. While a pair of images is depicted, head position information associated with the end user may be determined using a single image (eg, image 408) or two or more images. In some cases, the head position information may be determined by averaging each of the pairs of images 408-409 and corresponding head position determinations. The pair of images 408-409 may correspond to a subsequent head position of the end user being captured at some time following the capture of the images 406-407. In one example, the pair of images 408-409 is captured 1 ms after the pair of images 406-407 is captured.

4F depicts the identification of a feature point including an end user's eye 404 and corresponding feature points 422, 424, 425, and 426. The feature point 422 may correspond to the outer canthal angle of the right eye of the end user, the feature point 424 may correspond to the inner canal angle of the right eye of the end user, and the feature point 425 may correspond to the pupil center of the right eye of the end user. May be. The outer canthal and inner canthal may be identified, even though end users periodically squint their eyes over time. The feature point 426 may correspond to an inflection point (or maximum curvature point) of the upper eyelid of the end user. By applying image processing techniques to the image 408, other feature points may also be identified, including the inflection point (or minimum curvature point) of the lower eyelid of the end user and the nose bridge of the end user.

4G depicts an updated 3D model of an end user's eye positioned relative to the HMD 402. Each of the end user's eyes may be modeled as a spheroid or sphere 428-429 positioned within a three-dimensional coordinate space for the HMD. As depicted, the first sphere 428 may correspond to the right eye of the end user and the second sphere 429 may correspond to the left eye of the end user. The positions of the spheres 428-429 in the three-dimensional coordinate space may be determined based on the feature points 422, 424, 425, and 426. In one embodiment, the position of the eye within the updated 3D model is the feature points 422, 424, 425, and 426 that match the feature points 412, 414, 415, and 416 associated with the end user's initial head position. ) May be determined using a rotation and translation algorithm to find the best fitting 3D rotation and translation of.

4H depicts one embodiment of a representation of a change in an end user's eye position over time. As depicted, a first position and direction 432 corresponding to sphere 418 may move to a second position and direction 434 of sphere 428. The change in the center of the sphere 418 representing motion in the right eye of the end user may be described using a vector 436. Likewise, the first position and direction 442 corresponding to the sphere 419 may move to the second position and direction 444 of the sphere 429. The change in the center of the sphere 419 representing the movement in the left eye of the end user may be described using a vector 446.

In one embodiment, the head position vector is orthogonal to a line that intersects the midpoint of the line connecting the center of the sphere 428-429 and connecting the inner canal of the end user's right eye and the inner canal of the end user's left eye. It can also contain vectors. The head position vector may include a 3D vector within a 3D coordinate space for the HMD.

In some cases, the HMD may capture multiple images of the end user's eyes and/or the end user's facial area surrounding the eyes while the end user is viewing the augmented reality environment using the HMD. Since the end user's eyes are fixed with respect to their head position, a fixed feature point associated with the end user's eye may be used to determine the head position relative to the end user. The HMD may apply image processing techniques to the captured image to detect and track feature points corresponding to the eyes of the end user. Once a feature point associated with one or both eyes of the end user's eye is identified, the position and orientation relative to the 3D model of the end user's eye (e.g., including a pair of spheroids) are determined based on the feature point. It may be updated, and the head position vector (for example, represented by a 3D vector) may be determined based on the 3D model.

5 is a flowchart illustrating one embodiment of a method for tracking an end user's head position of the HMD relative to the HMD and for adjusting the position of a virtual object based on the head position relative to the HMD. In one embodiment, the process of FIG. 5 may be performed by a mobile device, such as mobile device 19 in FIG. 1.

In step 502, the positioning of the HMD worn by the end user is detected. The positioning of the HMD may be detected when the HMD is placed on or near the end user's head. In step 504, guidance for placing the HMD to the end user is provided to the end user. Guidance provided to the end user may include an indication of an alignment marker for the end user to allow the end user to properly align or fit the HMD to their head.

In step 506, an initial head position of the end user with respect to the HMD is determined at a first point in time. The initial head position may be determined in response to the end-user accurately positioning the HMD on their head. In one embodiment, the initial head position is achieved by capturing an image of the end user's eye or facial features surrounding the eye, using one or more introspective cameras, by applying image processing techniques to the captured image. It may be determined by identifying a feature point associated with the user's eye, and then using the identified feature point to determine a position and orientation associated with the 3D model of the end user's eye.

In step 508, a second head position of the end user is determined with respect to the HMD at a second time point subsequent to the first time point. The second head position may be different from the initial head position. In some cases, subsequent head position may be determined periodically (eg, every 1-2 ms). In one embodiment, the second head position includes capturing additional images of the eyes of the end user or facial features surrounding the eyes, using one or more introspective cameras, and applying image processing techniques to the captured images. By identifying a feature point associated with the eye of the end user, and then using the identified feature point to determine an updated position and orientation associated with the 3D model of the eye of the end user. In one example, the position of the end user's eye in the updated 3D model finds the 3D rotation and translation that best match the feature point associated with the second feature point that matches the feature point associated with the end user's initial head position. May also be determined using a rotation and translation algorithm.

In step 510, one or more updated positions for the one or more virtual objects are determined based on the second head position. In some cases, the end user's head position (or eye position) is due to (e.g., due to the end user running on a bumpy road when the end user is wearing the HMD or by accidentally touching the HMD). Due to), the HMD may adjust the position of the virtual object displayed to the end user so that, upon moving relative to the initial head position, the virtual object appears to be properly fixed or otherwise positioned within the augmented reality environment.

In step 512, the degree of head movement for the HMD is determined based on the initial head position and the second head position. The degree of head movement may correspond to a change in the position and/or direction of the end user's head from the initial head position to the second head position. In some embodiments, the degree of head movement may correspond to a maximum vector difference among the plurality of head tracking vectors. For example, if the end user's head position is determined and tracked every millisecond, the degree of head movement may correspond to the maximum change in head position over a period of one second (i.e., the degree of head movement is one second. It may correspond to the maximum vector difference among thousands of head tracking vectors associated with each of the head positions over a period of time).

In step 514, it is detected that the degree of head movement for the HMD exceeds a threshold. In one example, the degree of head movement relative to the HMD may be exceeded when the degree of head movement exceeds 5 cm. In step 516, the resolution of the one or more virtual objects may be adjusted based on whether the degree of head movement for the HMD exceeds a threshold. In some embodiments, the resolution and/or number of virtual objects created and displayed to the end user may be modified based on the degree of head movement of the end user relative to the HMD. In one example, both the resolution and the number of virtual objects displayed to the end user are determined by the extent of head movement exceeding a threshold (e.g., the end user's head position is more than 4 centimeters relative to the HMD from the initial head position). It can be reduced (for example, it can be halved). The HMD also provides a notification (e.g., display notification or audio notification) to the end user when the degree of head movement exceeds a threshold and provides guidance to the end user to rearrange or reposition the HMD to the end user's head. You may.

In step 518, one or more virtual objects are rendered. In step 520, one or more virtual objects are displayed using the HMD. In one embodiment, the one or more images corresponding to the one or more virtual objects are rendered and displayed on the HMD such that the one or more virtual objects are recognized as being present within the augmented reality environment displayed to the end user of the HMD.

6 is a flowchart illustrating an alternative embodiment of a method for tracking an end user's head position of an HMD relative to the HMD. In one embodiment, the process of FIG. 6 may be performed by a mobile device, such as mobile device 19 in FIG. 1.

In step 602, images of a first set of end users are captured at a first time point using the HMD. The first set of images may be captured using one or more inward cameras integrated with the HMD. In step 604, the end user's initial head position relative to the HMD is determined based on the first set of images. The initial head position may be determined in response to the end-user accurately positioning the HMD on their head. In one embodiment, the initial head position is achieved by capturing an image of the end user's eye or facial features surrounding the eye, using one or more introspective cameras, by applying image processing techniques to the captured image. It may be determined by identifying a feature point associated with the user's eye, and then using the identified feature point to determine a position and orientation associated with the 3D model of the end user's eye.

In step 606, based on the initial head position, a first image associated with the virtual object is rendered and displayed. The first image may be displayed at a first location on the HMD. In step 608, a second set of images of the end user is captured at a second viewpoint following the first viewpoint. In step 610, the updated head position of the end user may be determined based on the second set of images. The updated head position may be different from the initial head position. In some cases, subsequent head position may be determined periodically (eg, every 1-2 ms). In one embodiment, the updated head position includes capturing additional images of the end user's eyes or facial features surrounding the eyes, using one or more introspective cameras, and applying image processing techniques to the captured images. By identifying a feature point associated with the eye of the end user, and then using the identified feature point to determine an updated position and orientation associated with the 3D model of the eye of the end user. In one example, the position of the end user's eyes within the updated 3D model is the 3D rotation and translation that best match the feature points associated with the updated head position that match the feature points associated with the end user's initial head position. It may also be determined using rotation and translation algorithms to find.

In step 612, based on the updated head position, a second image associated with the virtual object is rendered and displayed. The second image may be displayed at a second location on the HMD different from the first location. In one example, the end user's head position (or eye position) is (e.g., due to the end user running on a bumpy road when the end user is wearing the HMD or by accidentally touching the HMD). Due to), the HMD may adjust the position of the virtual object displayed to the end-user so that when moving relative to the initial position relative to the HMD, the virtual object appears to be properly fixed or otherwise positioned within the augmented reality environment. have.

In step 614, the degree of head movement for the HMD is determined based on the initial head position and the updated head position. The degree of head movement may correspond to a change in the position and/or direction of the end user's head from the initial head position to the updated head position. In some embodiments, the degree of head movement may correspond to a maximum vector difference among the plurality of head tracking vectors. For example, if the end user's head position is determined and tracked every 0.2 milliseconds, the degree of head movement may correspond to a maximum change in head position over a period of two seconds.

In step 616, it is detected that the degree of head movement for the HMD exceeds a threshold. In one example, the degree of head movement relative to the HMD may be exceeded when the degree of head movement exceeds 2 cm. In step 618, a warning is displayed to the end user if the degree of head movement for the HMD exceeds the threshold. In one embodiment, the alert may be displayed to the user as visual text that appears within an augmented reality environment.

One embodiment of the disclosed technology includes one or more processors in communication with a see-through display. The one or more processors determine an initial head position of the end user of the electronic device with respect to the electronic device at a first time point and determine a second head position of the end user with respect to the electronic device at a second time point subsequent to the first time point. The second head position may be different from the initial head position. The one or more processors render a second image associated with the virtual object based on the second head position. The see-through display displays the second image so that the virtual object is recognized as being present in the augmented reality environment displayed to the end user of the electronic device.

One embodiment of the disclosed technology includes determining an end user's initial head position relative to the HMD at a first point in time and displaying a first image associated with the virtual object based on the end user's initial head position relative to the HMD. Include. The first image is displayed at a first location on the HMD. The method further includes determining a second head position of the end user relative to the HMD at a second time point subsequent to the first time point. The second head position may be different from the initial head position. The method further includes rendering and displaying a second image associated with the virtual object based on the end user's second head position relative to the HMD. The second image may be displayed at a second location on the HMD different from the first location.

One embodiment of the disclosed technique includes capturing images of a first set of end users at a first point in time. The first set of images captures at least a portion of the end user's eye. The method includes determining an end user's initial head position relative to the HMD based on a first set of images, rendering and displaying a first image associated with the virtual object based on the end user's initial head position relative to the HMD. And capturing a second set of images of the end user at a second time point subsequent to the first time point, and determining a second head position of the end user with respect to the HMD based on the second set of images. Include. The second head position may be different from the initial head position. The method further includes rendering and displaying a second image associated with the virtual object based on the end user's second head position relative to the HMD.

7 is a block diagram of one embodiment of a mobile device 8300, such as mobile device 19 in FIG. 1. Mobile devices may include laptop computers, pocket computers, mobile phones, HMDs, personal digital assistants, and handheld media devices integrated with wireless receiver/transmitter technology.

Mobile device 8300 includes one or more processes 8312 and memory 8310. The memory 8310 includes an application 8330 and non-volatile storage 8340. The memory 8310 can be any of a variety of memory storage media types, including non-volatile and volatile memory. The mobile device operating system handles different operations of the mobile device 8300 and may include a user interface for operations such as making and receiving phone calls, text messaging, voicemail checking, and the like. Application 8330 can be any assortment of programs, such as camera applications for photos and/or videos, address book, calendar applications, media players, internet browsers, games, alarm applications, and other applications. Non-volatile storage component 8340 in memory 8310 may include data such as music, photos, contact data, scheduling data, and other files.

One or more processes 8312 communicate with the see-through display 8390. The see-through display 8309 may display one or more virtual objects related to the real-world environment. One or more processors 8312 may also include an RF transmitter/receiver 8306, an infrared transmitter/receiver 8308, and a global positioning service (GPS) receiver 8366 that are eventually coupled to the antenna 8302. And, and a motion/direction sensor 8314, which may include an accelerometer and/or magnetometer. The RF transmitter/receiver 8308 may cope with wireless communication through various wireless technology standards such as Bluetooth® or IEEE 802.11 standards. The accelerometer is designed to enable applications such as intelligent user interface applications that allow the user to enter commands through gestures, and orientation applications that can automatically change the display from portrait to landscape mode when the mobile device is rotated. It is integrated into the device. The accelerometer can be provided, for example, by a micro-electromechanical system (MEMS), which is a microscopic mechanical device (in the size of a micrometer) built on a semiconductor chip. As well as the direction of acceleration, orientation, vibration and impact can be detected. One or more processes 8312 may also include a ringer/vibrator 8316, user interface keypad/screen 8318, speaker 8320, microphone 8322, camera 8324, light sensor 8326 and temperature. It communicates with the sensor 8328. The user interface keypad/screen may include a touch sensitive screen display.

One or more processes 8312 control the transmission and reception of radio signals. During transmit mode, one or more processes 8312 provide a voice signal, or other data signal from microphone 8322 to RF transmitter/receiver 8306. Transmitter/receiver 8306 transmits a signal via antenna 8302. Ringer/vibrator 8316 is used to signal incoming calls, text messages, calendar reminders, alarm clock reminders, or other notifications to the user. During receive mode, RF transmitter/receiver 8306 receives a voice signal or data signal from a remote station via antenna 8302. The received voice signal is provided to the speaker 8320 while the other received data signal is properly processed.

Additionally, a physical connector 8386 may be used to connect the mobile device 8300 to an external power source, such as an AC adapter or a powered docking station. The physical connector 8386 can be used as a data connection to an external computing device. The data connection allows operations such as synchronizing mobile device data with computing data on other devices.

The disclosed technology is operable with a variety of other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and/or configurations that may be suitable for use with the present technology include personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor based systems, set-top boxes. , Programmable consumer electronics, networked PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

The disclosed technology may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. In general, software and program modules as described herein include routines, programs, objects, components, data structures, and other types of structures that perform particular tasks or implement particular abstract data types. Hardware or a combination of hardware and software may be substituted for software modules such as those described herein.

The disclosed techniques may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote storage media, including memory storage devices.

For the purposes of this document, each process associated with the disclosed technology may be performed continuously and by one or more computing devices. Each step in the process may be performed by the same or different computing device as used in the other steps, and each step need not necessarily be performed by a single computing device.

For the purposes of this document, references to "one embodiment", "one embodiment", "several embodiments", or "other embodiments" in this specification may be used to describe different embodiments, and , Do not necessarily refer to the same embodiment.

For the purposes of this document, the connection may be a direct connection or an indirect connection (eg, through another part).

For the purposes of this document, the term "set" of objects refers to a "set" of one or more of the objects.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. . Rather, the specific features and acts described above are described as exemplary forms of implementing the claims.

Claims (20)

In an electronic device for displaying an augmented reality environment,
One or more processors; And
A see-through display in communication with the one or more processors,
The one or more processors determine an initial head position of an end user of the electronic device with respect to the electronic device at a first point in time, and the one or more processors determine a virtual head position based on the initial head position. Rendering a first image associated with an object (virtual object), wherein the at least one processor determines a second head position of the end user with respect to the electronic device at a second view subsequent to the first view, The second head position is different from the initial head position, and the one or more processors determine the head movement relative to the electronic device based on the initial head position and the second head position. Detecting that the degree has exceeded a threshold, the one or more processors rendering a second image associated with the virtual object based on the second head position;
The see-through display displays the first image using a first resolution for the virtual object, and the see-through display responds to detecting that the degree of head movement with respect to the electronic device exceeds a threshold. And displaying the second image using a second resolution for the virtual object that is smaller than the first resolution. The method of claim 1,
Wherein the electronic device comprises an HMD. The electronic device of claim 2, wherein the one or more processors reduce the number of virtual objects displayed on the HMD in response to detecting that the degree of head movement for the HMD has exceeded a threshold. The method of claim 3,
The second resolution is half of the first resolution;
The electronic device, wherein the number of virtual objects displayed on the HMD is halved in response to detecting that the degree of head movement with respect to the HMD has exceeded a threshold. The method of claim 3,
Wherein the see-through display displays a warning to the end user if the degree of head movement exceeds the threshold. The method of claim 3,
The one or more processors detect that the degree of head movement for the HMD exceeds a threshold value by determining a maximum vector difference between a plurality of head tracking vectors, and the plurality of head tracking vectors Wherein the first head tracking vector is based on a difference between the second head position and the initial head position. The method of claim 2,
Wherein the one or more processors determine an absolute head position vector representing the end user's head position relative to the environment in which the HMD is present. In the method for tracking the head position of the end user of the HMD relative to the HMD,
Determining an initial head position of the end user with respect to the HMD at a first time point;
Based on the initial head position of the end user with respect to the HMD, rendering and displaying a first image associated with a virtual object, the first image on the HMD using a first resolution for the virtual object. Displayed-;
Determining a second head position of the end user with respect to the HMD at a second time point subsequent to the first time point, wherein the second head position is different from the initial head position;
Detecting that a degree of head movement with respect to the HMD exceeds a threshold based on the initial head position and the second head position; And
Rendering and displaying a second image associated with the virtual object, based on the position of the second head of the end user with respect to the HMD-the second image has a degree of head movement with respect to the HMD exceeding a threshold. Displayed on the HMD using a second resolution for the virtual object that is less than the first resolution in response to detecting that the
Including a method for tracking head position. The method of claim 8,
Reducing the number of virtual objects displayed on the HMD in response to detecting that the degree of head movement for the HMD exceeds a threshold value
A method for tracking the head position further comprising. The method of claim 9,
The second resolution is half of the first resolution;
Wherein the number of virtual objects displayed on the HMD is halved in response to detecting that the degree of head movement for the HMD has exceeded a threshold value. The method of claim 9,
Displaying a warning to the end user when the degree of head movement exceeds the threshold.
A method for tracking the head position further comprising. The method of claim 9,
The step of detecting that the degree of head movement for the HMD exceeds a threshold value includes determining a maximum vector difference between a plurality of head tracking vectors, and a first head tracking vector among the plurality of head tracking vectors Is based on a difference between the second head position and the initial head position. The method of claim 8,
The step of determining the initial head position includes capturing one or more images of the end-user while the end-user is wearing the HMD, wherein the one or more images cover at least a portion of the eye of the end-user. Capturing and determining the initial head position comprises: identifying one or more feature points associated with the eye of the end user by applying image processing techniques to the one or more images, and identifying the one or more feature points. And determining a position and orientation associated with the 3D model of the end user's eye based on that. The method of claim 13,
Wherein capturing the one or more images is performed by an inward facing camera associated with the HMD. The method of claim 8,
And determining an absolute head position vector representing the end-user's head position relative to the environment in which the HMD is present. One or more computer readable memory devices comprising processor readable code for programming one or more processors to perform a method of tracking the head position of an end user of the HMD relative to an HMD, the method comprising:
Capturing a first set of images of the end user at a first point in time, the first set of images capturing at least a portion of the eye of the end user;
Determining an initial head position of the end user with respect to the HMD based on the first set of images;
Based on the initial head position of the end user with respect to the HMD, rendering and displaying a first image associated with a virtual object, the first image on the HMD using a first resolution for the virtual object. Displayed-;
Capturing a second set of images of the end user at a second time point subsequent to the first time point;
Determining a second head position of the end user with respect to the HMD, wherein the second head position is different from the initial head position based on the second set of images;
Detecting that a degree of head movement with respect to the HMD exceeds a threshold based on the initial head position and the second head position; And
Rendering and displaying a second image associated with the virtual object, based on the position of the second head of the end user with respect to the HMD-the second image has a degree of head movement with respect to the HMD exceeding a threshold. Displayed on the HMD using a second resolution for the virtual object that is less than the first resolution in response to detecting that the
One or more computer readable memory devices comprising a. The method of claim 16,
The above method,
Reducing the number of virtual objects displayed on the HMD in response to detecting that the degree of head movement for the HMD exceeds a threshold value
One or more computer-readable memory devices, further comprising. The method of claim 17,
The second resolution is half of the first resolution;
The one or more computer-readable memory devices, wherein the number of virtual objects displayed on the HMD is halved in response to detecting that the degree of head movement for the HMD has exceeded a threshold. The method of claim 17,
The above method,
Displaying a warning to the end user when the degree of head movement exceeds the threshold.
One or more computer-readable memory devices, further comprising. The method of claim 16,
The above method,
Determining an absolute head position vector representing the head position of the end user with respect to the environment in which the HMD exists; And
Transmitting the absolute head position vector to a server
One or more computer-readable memory devices, further comprising.

Images