JP2013500544A - Improved audio / video method and system - Google Patents

Abstract

Audio and / or video data is structured and persistent with auxiliary sensor data (eg related to acceleration, orientation or tilt) by using a single data object such as a modified MPEG file or data stream Associated with In this manner, various presentation devices can modify audio content or video content using the co-transmitted sensor data. For example, by associating accelerometer data with video data, some users can view a vibration-stabilized version of the video, and others can view video with such motion artifacts intact. become able to. Similarly, by co-transmitting camera parameters such as focal plane distance along with audio / video content, the volume can be reduced when the camera captures audio / video from a distant subject.
[Selection] Figure 2

Description

Related application data

  In the United States, this application is a legitimate application of US Provisional Patent Application No. 61/228336, filed July 24, 2009, and claims the priority benefit of the provisional application.

Introduction

  For mobile phones and other devices with video / image sensors (eg, cameras), it would be desirable to have a coding format that includes additional data streams in addition to a single existing configuration of video / audio (eg, MPEG). Some additional data streams that can be transmitted in this way are 2D / 3D accelerometer / tilt, 2D / 3D compass (magnetometer), lens zoom, aperture, focal length, depth of field, etc. Is included.

  By conveying these data along with the image information, the image can be conveniently processed according to this auxiliary information. In addition, if the auxiliary information is permanently associated with the image information, another device may use the auxiliary information to process the video (and / or audio) in a different way and at a different time. You can also.

  Use of such auxiliary information is possible by the device that captured the image and / or by other devices / systems / applications (eg, video player applications or social media websites such as YouTube). Such processing can be done at the time of image capture (eg, in manual or automated live production), or when an entertainment work is produced from video content, or during post-production (eg, converting to a different format, or different. For consumption in the environment) or when the content is finally presented to the viewer.

  To illustrate, one embodiment of the present technology is a method in which both audio data and camera data (eg, focal plane distance) are recovered from a single data object (eg, an MPEG file or stream). The audio data is processed according to the focal plane data to obtain modified audio data for presentation to the user. This process controls the volume so that the sound captured when the focal plane coincides with a distant subject is attenuated compared to when the focal plane coincides with a nearby subject (ie, Sound may be louder in a near subject). This effect can be enabled or disabled by the user depending on the selection.

  Another embodiment is a method in which both video data and sensor data (eg, accelerometer data) are recovered from a single data object. The video data is processed according to the accelerometer data, again with or without compensation for camera movement (eg, vibration) based on the selection data specified by the user.

  More generally, some aspects of the present technology involve a single data object that includes both (1) audio and / or video information, and (2) camera and / or sensor data. Particular embodiments relate to a method and apparatus for creating such a single object from different data sources and / or recovering individual data from such a single object. In other methods and configurations, sensor and / or camera data is used to modify audio and / or video information.

  The above and other features and advantages of the present technology will become more readily apparent from the following detailed description, which is provided with reference to the accompanying drawings.

1 is a diagram showing a configuration of a conventional technique in which audio and video data are transmitted between systems using MPEG encoding. It is a figure showing the composition by one mode of this art. Fig. 4 shows how different receiving units can use a common data stream coming from a mobile phone. It is a block diagram which shows the aspect of a mobile telephone.

Detailed description

  FIG. 1 shows the configuration of the prior art. A transmitting device such as a video camera includes a camera system (including a two-dimensional image sensor) that captures an image and a microphone that captures sound. The captured data is described (that is, encoded) in accordance with a standard such as MPEG. The resulting video and audio data stream is integrated in a well known manner (eg, MPEG data stream) that is compatible with all other processing and recording device methods. The receiving unit decodes the transmitted video and supplies it to the display screen. Similarly, the receiving unit decodes the transmitted voice and supplies it to the speaker.

  Newer devices include numerous sensors in addition to prior art image sensors / microphones. For example, even an inexpensive smartphone includes components such as an accelerometer (sensing hand gesture, tilt, etc.) and a magnetometer (eg sensing compass direction). In addition, optical capture parameters such as lens zoom and aperture size can be acquired as data and later used when processing image (and / or audio) data.

  FIG. 2 illustrates an exemplary implementation using aspects of the present technology. The sending unit (eg mobile phone) has 3D acceleration information (usually 3 MEMS accelerometers arranged orthogonally), 3D position coordinates (eg by GPS or otherwise), timestamp Data is provided including data, three-dimensional orientation information (usually obtained from a magnetometer or Hall effect compass), and a tilt sensor (which may comprise built-in accelerometer data or a gyroscope device). The camera data can include information about focus, zoom, aperture size, depth of field, exposure time, ISO settings, lens focal length, depth of focus, and the like. Such parameters allow a recalculation of how the image relates to its spatial domain. Other system specific timing information may be included, such as the relative delay between the transducer and the camera / video frame respectively. Ancillary data may include, for example, the frame identification number of the associated video frame, or other synchronization method associated with the MPEG I-frame.

  This just-detailed information is shared with the audio and / or video information captured at the sending unit as a single data stream, eg MPEG, or stored in a single file such as an MPEG data file. Communicated. (Such single constructs are collectively referred to as a single data object and have the advantage of permanently associating these different types of data together.)

  FIG. 2 also shows a receiving unit. This receiving unit takes some or all of the data just detailed and uses that data in creating the output presented to the user.

  Consider some examples of how such output can be associated with co-transmitted auxiliary data. One is vibration compensation (motion stabilization). Many cameras sense camera vibration (eg, by one or more accelerometers) and process the image data to remove this effect before it is output from the camera. In one configuration using the present technology, vibration is sensed as before. However, instead of removing this effect in the camera, the associated vibration data is transmitted along with the captured image data. Devices and applications that receive this data can choose to apply anti-vibration algorithms (of the kind used in prior art cameras), or images can be processed in their raw, It can also be provided as is. Whether to apply ancillary data to remove vibrating artifacts can be automatic (for example, certain video players can always show video as vibration-stabilized) or at the user's choice ( It can be determined individually, for example, as displayed by operation of the user interface controller or by referring to stored selection data.

  Another example is voice processing. Most audio capture devices include some form of automatic gain control (AGC) that attempts to maintain a relatively constant audio level. The sound sensed by the microphone is amplified when it is weak and attenuated when it is strong. This is generally desirable, for example for listener comfort. According to aspects of the present technique, AGC is applied at the time of acquisition as before. However, at the time of presentation, a certain application can control the volume according to the distance of the subject from the camera. That is, the audio output can be controlled according to auxiliary data indicating the position of the focal plane of the camera. If the camera is focused on a subject close to the camera (eg, an interview with a person a few feet away), the presentation system can output a first value of audio level. In contrast, when the camera is focused on a distant subject, the presentation device reduces the sound level to a lower second value, giving the audience an acoustic effect that enhances the distant visual effect. Can do. This is referred to herein as “dimensional” speech. (Again, whether such processing is used can be selectively controlled automatically or manually.)

  Similarly, it is possible to add an acoustic effect or to adjust the acoustic effect based on camera movement as indicated by co-transmitted accelerometer or magnetometer data. When camera vibration or jitter is sensed, a low frequency continuous sound can be superimposed on the existing audio data. If the data indicates that the camera is panning up or down, a whistle sound with a higher or lower frequency (respectively) can be added. And so on.

  FIG. 3 shows the diversity of this configuration. The single supply data is the output from the sending unit. This feed data is distributed to the various receiving units “live” or via intermediate storage and / or production. Different receiving units present various data, for example, some with motion-stabilized displays, some without, some with dimensional audio, some with conventional (AGC) audio. To present. (The receiving unit can also be a mobile phone. One of the receiving units can be a calling unit. A block diagram of an exemplary mobile phone is shown in FIG. 4).

  The receiving unit of FIG. 2 is shown to include only a display and speakers as output devices, but of course, more or fewer devices can be used. In one configuration, for example, one or more haptic-based output devices are included so that motion information sensed at the sending unit can be presented as movement (vibration) at the receiving unit. Techniques and output devices related to haptics are described in Immersion Corp., such as Patent No. 7,425,675, No. 7,561,142, and Patent Application Publication No. 2009/0096632. Can be seen from the patent.

  For encoding different data streams, MPEG-2 / H. 262 and H.H. Basic techniques of known protocols such as H.264 / MPG-4 can be used. These protocols can be modified to add other extensions and are similar to (voice) MPG multi-channel (ISO 14496-3). Another approach is to use one or more available channels among the six LCPM audio channels of MPEG-2 (these are reversibly encoded and do at a lower sample and bit rate) be able to).

  To explain, each data such as a spatial transducer, transducer axis, camera settings, optical parameters, etc. is encoded into a digital stream and inserted into one of the audio channels of the MPEG-2 stream at the time of the encoding. (With current coding, the number of channels and bit rate can be selected, and therefore one or more channels can be enabled.) Original audio from camera and microphone (s) A speech coding method is selected in which a sufficient number of additional channels is implemented, more than the number required for. Data such as a transducer includes a data stream of these additional voice channel (s). This keeps the converter data synchronized with the audio / video stream, which is desirable for subsequent processing of the audio / video data. (In another configuration, synchronization is not maintained in the protocol, but is later established by referring to a synchronization signal with digital watermarks in one or more of the auxiliary data streams along with audio and / or video data. (See, for example, patents 6,836,295 and 6,785,401.)

  Since spatial data and accelerometer data do not require very high bit rates (as discussed further below), these data streams are coupled in series in fewer audio channels (even a single audio channel). can do. As a rough example, given 32 bits per sample on each of the 3 axes (12 in total) of the 4 transducers, adding 32 camera settings, each of 32 bits, resulting in a total of 176 per image frame. Become a byte. If the image frame rate is 60 frames / second, the auxiliary data rate will be 10,560 bytes / second, which is well inside even for the lowest audio channel (84.1 44.1 kHz). Since some auxiliary data does not have to be transmitted with every frame, the channel utilization can be further reduced.

  Since the data rate is low, the auxiliary data can be encoded at a lower bit rate than conventional speech. The camera's spatial coordinates may have the lowest rate. Ancillary data describing the optical capture system (lens, optics, exposure, etc.) can be sent periodically, but only needs to be sent when it changes (usually associated with the frame where the change first appears). Is). In one particular implementation, such data is sent in alignment with MPEG video I-frames for image post-processing in the edited stream. Zoom, aperture, azimuth, depth of field, focal length, azimuth, 3D GPS, time, etc. are required only once per frame. Acceleration information is typically collected at a faster rate to preserve camera space motion (or integrated to obtain velocity and associated spatial position). Acceleration information can be collected more frequently than once per frame, but may only be transmitted at frame intervals, or may be transmitted more or less frequently. (In other words, the accelerometer data and position data need not be limited to the video frame rate.)

  Given the relatively low data rates, data compression is not necessary (but of course can be used). For slow-changing data, some implementations typically send differential updates and send full coordinates for resynchronization less frequently (similar to I-frames for B and P frames in MPEG) be able to. Various data types (eg, differential or all) can be displayed by tags (eg, XML format) in the associated data packet or field.

  The acceleration data can be integrated locally at the transmitting unit (primary and secondary integration). These parameters are included in the auxiliary data stream because they provide higher accuracy than is necessary to locate each frame. Similarly, by tracking and combining acceleration, orientation, compass, and position, spatial positioning can be calculated more accurately with high accuracy in the camera. The integration of the acceleration data effectively reduces the data bandwidth.

  In some embodiments, ancillary data is gathered together with the sync field / tag and identification field / tag into the stream to allow video analysis and modification. The accuracy of the various data can also be specified (tagged) in the stream.

  In another configuration, each sample group of spatial information is delimited or packetized with a time stamp corresponding to the data.

  In one particular configuration, the auxiliary data is encoded, multiplexed between existing voice channels, for example 5.1 voice implementations, and / or mixed into existing voice channels. For monaural audio such as from mobile phones and many other cameras, the extra channel capacity is essentially freely available. In another implementation, the auxiliary data is encoded on one or more carriers near or at the lowest or highest part of the human audible range, for example, less than 300 Hz or greater than 15 kHz. (MP3 or other encoding can be adapted to preserve such data bands.) These data bands are appropriate in the audio presented for purchase by a person at the time of playback. Can be filtered or removed.

  In yet another configuration, the auxiliary data is transmitted in steganography as a fine change in audio (or video) over part or all of the audio (video) range. Such digital watermarking techniques are described in detail in, for example, Patent Nos. 6,061,793 and 6,590,996.

  As will be apparent to those skilled in the art, a decoder configuration complementary to the encoding configuration is provided, for example, in the receiving unit (s) to extract auxiliary data.

  Yet another configuration employs a new standard encoding format in which the transducer data and camera data are encoded in a dedicated channel, which is synchronized with the video frame encoding (again, again). , But not necessarily limited to frame rate).

  Data field tags and / or packet header fields in the stream can be made extensible so that additional data types can be included in the future. In one particular configuration, the packet header is kept short to show some standard groups / packets of transducer data, whether the group is a relative coordinate group or an absolute coordinate group, and / or Easily identifies whether camera information is included. Several additional header bit combinations are optionally left for packets that use expandable content. In that case, each data element or group of data in the (expandable) packet is scoped sequentially or hierarchically (like XML). Scalability can be useful for all data groups, but desirably does not form the main part of bandwidth requirements.

  The encoded stream (or corresponding file structure) can be tagged near the start to enable players with knowledge of the protocol to use the correct parsing options, and the audio (encoded into the audio channel) And so on, so that the auxiliary data can be ignored so that the stream is not reproduced.

  For playback on legacy devices (eg legacy video / audio players), the auxiliary data can be removed in advance, or if the player knows the encoding protocol, the player can choose to ignore the data can do.

Some examples showing the use of co-communication assistance data are provided above. The number of such applications is unlimited. Some other examples are briefly described below.
Combined with spatial information from mapping (eg, similar to extended reality system Layar.eu, but operates with recorded video instead of live video).
Build a 3D image (even camera movements and vibrations are used in combination with an optical path setting to give stereoscopic information and then interpolate the image).
Track camera space position for mapping.
User tagging of image content for location searches or social applications.
Allows post production or live playback tagging of subjects in the field of view (eg, by providing real-time coordinates of friends from a mobile phone while a previously recorded vacation video clip is visiting the same location) , Where you can see where your friends are in real time).
• Enable video search for possible views of incidents (eg crimes in courtroom applications-rather than manually reviewing and identifying what each frame's view is, someone with a high-speed search Have you accidentally captured the image of Incident X in the background?)

  The combined audio / video data and auxiliary data streams can be used or recorded in live broadcasts. In the live situation, auxiliary data that is done by the viewer (at home personal computer / television) or by an intermediate service (broadcast studio or cloud processing) so that the originating unit captures the event (eg a sports event) (For example, audio / video processing together with auxiliary data).

  A more complex spatial view can be constructed by combining a plurality of camera views. In that case, the viewer can observe the event with a post-processing effect that allows viewing from various angles. With attached spatial tag data or using object recognition, the observer / viewer can track individual performers, athletes, cars, horses, with spatial data included in the stream Multiple image sources are selected / interpolated / mixed to follow the marked object / person. Additional markers and tags can be inserted into the scene at a particular spatial location.

  Position (GPS) / acceleration monitoring tags can be made for a variety of sporting events including cars, horses, football (eg, worn on the athlete's uniform). Auxiliary data repeaters (position, acceleration, etc.) can also be incorporated into a game ball, pack, or other mobile or stationary equipment, such auxiliary data can be captured in the audio / video of the competition (eg taken from the sideline) Are transmitted together. This gives a known position relative to the camera to the viewed subject.

Other supplementary explanation

  The reader is considered familiar with the various protocols and data transmission standards / specifications relevant to the above discussion. Various standards documents detailing such specifications (eg, ISO / IEC 14496-3, ISO / IEC 13818-1, ISO / IEC 14496, ITU H.262, ITU H.222, RFC 3640, etc.) are by reference. Incorporated herein.

  Although the principles of the effects of the present invention have been described and illustrated with reference to illustrative examples, it will be appreciated that the technology is not so limited.

  For example, while specific reference is made to exemplary protocols and standards (eg, MPEG-2), of course others can be adapted for the purposes detailed. These include MPEG-4, MPEG-7, MPEG-21 and the like.

  We have referred to an embodiment in which the audio is AGC processed at the time of capture and then the audio level is controllably changed according to the auxiliary data. In another configuration, speech is not AGC processed at the time of capture. Instead, the speech is encoded in its originally captured form. The sound level or other effects can still be controlled according to the auxiliary data.

  In the detailed embodiment, the auxiliary data is used in processing the co-transmitted audio and / or video data, but this is not essential. In other implementations, the auxiliary data may be used by the receiving unit for purposes unrelated to audio / video (eg, identifying the capture location on the displayed map, etc.).

  It should be understood that instead of encoding the sensor data (or camera data) itself, other information based on such data can be encoded. For example, data representing the instantaneous position information can be processed by a mobile phone to obtain motion vector data. This motion vector data (and other such post-processing data) can be encoded in a single data object along with associated audio and / or video. Motion is related to position by differentiation, but myriad other types of processing, such as integration, filtering, etc., can also be applied. Similarly, various types of auxiliary data can be combined or otherwise processed together (eg, one estimate of motion can be generated with a derivative of position data, and a second motion can be calculated with integration of accelerometer data. Can be generated, and then these two estimates can be averaged). In some configurations, the original sensor data is stored, and tags of this original data are included in the encoded data stream as needed for later reanalysis.

  Similarly, it should be understood that the type of auxiliary data detailed is illustrative and not limiting. For example, in another configuration, the athlete's smartphone can be equipped with an oxygen sensor, or a heart rate monitor, and this information can be stored in a data structure that also contains relevant audio or video information. (In some embodiments, such a sensor can be connected to a smartphone via Bluetooth or other short-range connection technology.) With such a configuration, a bicycle race video is displayed with each competitor's varying heart rate. Numbers and blood oxygen levels can be presented annotated with diagrams.

  In the detailed configuration, it is particularly considered that auxiliary data (for example, sensor data) can effectively contribute to the processing of image (video) data, but other processing configurations are also effective. For example, image data can be used to help process sensor data, or to synchronize between an image and a set of sensor data. The video stream can include a data synchronization tag inserted in the data channel, to which much larger data or alternative data can be synchronized. (Data synchronization tags may need to be different from video frames / time stamps, as it may not be possible to calculate the sensor measurement event identifier from the frame number or time. In video post-processing, the original frame information may be lost.) Similarly, the captured auxiliary data is conveniently included within a single data object (eg, in an audio channel or metadata channel). If is too large, the identification number associated with this data can instead be inserted into a single data object.

  For example, consider capturing a still picture and converter data at some point during the capture of a moving image. An image tag can be inserted into the video stream auxiliary channel and associated with the photo. (Photos can have a tag to associate with the original video frame / time.) Photo capture time can be recorded with a resolution finer than the frame rate.

  Also, consider collecting a high-speed sample sequence of position data (for example, position, orientation, and tilt) at a frame rate greater than the video frame rate when the smartphone is capturing video. The resulting auxiliary data can be too large to be included in the data object along with the video. (Or the location data may be captured in another process.) An identifier for this location data is given to the tag, and this tag (possibly at the beginning and end of this data capture) in the video stream. Inserted. This data can be processed for a variety of purposes, such as, for example, synthesizing a three-dimensional model of an imaged object, quantifying user movements associated with grasping / using a smartphone. .

  Although the detailed configuration specifically considered a single image sensor or a single audio sensor, the present technology is also well suited for use with multiple sensors.

  In one such configuration, multiple sensors capture audio or video and / or auxiliary data. The captured information can be provided to a cloud resource or other processing system where these multiple streams can be analyzed for presentation, combined, or selected between them. For example, the relative positions of multiple cameras can be calculated from associated position data (eg, encoded with the video). If one camera is within the field of view of another camera, its presence or position is indicated in the user interface (eg, by highlighting a rectangle in the viewing frame), thereby distinguishing it from the viewer Can be used. If the video reporter captures an event or interview, the spatial location information can indicate where better audio or video in a different video stream than is being viewed may be available.

  Multiple viewpoints of an event can be analyzed and combined to present a 3D model of the event. With the captured position (motion) data, it is possible to compensate for camera motion and to transform / remapping for each image based on camera orientation, focal length, field of view, etc.

  If multiple audio sensors are placed in one space and each also collects auxiliary information (eg, each instantaneous position), a rich data set can be obtained that can produce various effects. Consider that each performer turns a movie with such a microphone into a movie. The resulting audio / auxiliary data stream can be presented separately to different consumers. A consumer would like to hear the voice as if they were at the center of the performance. (For example, the average of microphone positions can be calculated, and the stereo field directivity modeled based on the known model of human binaural hearing, where appropriate, at each position, the sound field contribution by each microphone. Along with that, another consumer would follow a particular actress and would like to hear the voice as that actress heard. Again, such a system can determine the net sound field at the location of the actress as the actress moves around. (Actress's voice is always the main, others are heard depending on the distance from the actress.) When presenting the environment includes many speakers, eg front right, front left, back center (eg multi-channel Surround sound), the data can be processed taking into account 360 degree directivity as well as volume when presented to the consumer. The consumer can choose essentially any virtual listening point in the original performance with static or motion.

  In yet another embodiment, the consumer's location can be sensed (eg, using a possessed smartphone) and the audio source can be processor (eg, the consumer's location) depending on the consumer's location and movement. Smartphone). For example, in a cocktail party audio presentation (recorded using the structure of the previous paragraph), a consumer can move through physical space and listen to conversations of particular interest. This can be done without using multi-channel surround sound (eg, using a simple earphone plugged into the consumer's smartphone), or a multi-channel speaker configuration can be used.

  The configuration just shown has particular applicability in computer-based games where the player can interact with the system or other real / virtual players with respect to time and place.

  Assignee's Patent No. 7,197,160 details how position information (eg, latitude / longitude) can be encoded in images and audio with steganography.

  Although reference has been made to mobile phones, it should be understood that the present technology finds utility in all types of devices, both telephones and other portable and fixed types.

  (Particularly contemplated mobile phones include Apple iPhone 4 and mobile phones that comply with Google's Android specification, such as HTC Evo 4G and Motorola Droid X. Details of iPhone including the touch interface can be found in Apple's patent application. (It is presented in Publication No. 2008/0174570)

  The basic design of mobile phones and other devices that can serve as a sending unit and a receiving unit are well known to those skilled in the art. Generally speaking, each one or more processors, one or more memories (eg RAM), storage devices (eg disk or flash memory), user interfaces (eg keyboard, TFT liquid crystal display or organic) For communicating with LED display screens, touch or other hand gesture sensors, and software instructions implementing a graphic user interface), interconnects (eg, buses) between these elements, and other devices Interface (wireless such as GSM, CDMA, W-CDMA, CDMA2000, TDMA, EV-DO, HSDPA, WiFi, WiMax, or Bluetooth, and / or wired such as Ethernet local area network, TI Internet connection, etc. Theft, including the can). The calling device typically includes a camera and / or microphone along with one or more other components / systems for obtaining the auxiliary data described above.

  Software instructions for performing the detailed functions can be easily written by those skilled in the art from the description provided herein, such as C, C ++, Visual Basic, Java, Python, TcI, Written in Perl, Scheme, Ruby, etc. Mobile phones and other devices according to the present technology can include software modules for performing various functions and steps.

  Each device typically includes operating system software that provides an interface to hardware resources and general-purpose functions, as well as application software that can be selectively invoked to perform a specific task desired by the user. Known audio and video codecs, browser software, communication software, and media processing software can be adapted for many applications detailed herein. Software and hardware configuration data / instructions are typically stored as instructions in one or more data structures, which are entities such as magnetic or optical disks, memory cards, ROMs, etc. that can be accessed throughout the network. Transmitted by some medium. Some embodiments may be implemented as an embedded system, ie a special purpose computer system, in which the operating system software and application software are indistinguishable to the user (eg, a basic mobile phone As is common in the case of). The functions detailed herein may be implemented in the form of operating system software, application software, and / or may be implemented as embedded system software.

  Different functions can be implemented on different devices. The description that an operation is performed by a particular device (eg, the originating mobile phone encodes lens data into the audio channel of the MPEG stream) is exemplary rather than limiting. Implementation of operations by another device (e.g., a subsequent device receives or knows lens data separate from the MPEG stream) is also explicitly contemplated.

  (Similarly, the description that data is stored on a particular device is also exemplary, and data can be stored anywhere, such as localized devices, remote devices, in the cloud, distributed, etc. it can.)

  The operations need not be limitedly performed by clearly specified hardware. Rather, in some operations, other services (e.g., cloud computing) can be externally referenced, and these services perform the service with yet another, generally anonymous system. Such distributed systems can be large (eg, with computing resources around the world) or local (eg, a portable device can locate nearby devices via Bluetooth communications). If identified, one or more of these nearby devices are involved in tasks such as providing local terrain data.)

  Although this disclosure details specific ordering of steps and specific combinations of elements in illustrative embodiments, other contemplated methods can reorder the steps (in some cases It should be understood that some contemplated combinations may omit some elements, add others, etc., with some omitted and others added).

  Although disclosed as a complete system, sub-combinations of the detailed configurations are also contemplated separately.

  It should be understood that detailed processing of content signals (eg, image signals, audio signals, transducer signals, etc.) includes converting these signals into various physical shapes. Images and video (the form of electromagnetic waves that travel through physical space and depict physical objects) can be captured from physical objects using a camera or other capture device and generated by a computing device You can also. Similarly, sound pressure waves traveling through a physical medium can be captured using an audio transducer (eg, a microphone) and converted to an electronic signal (digital or analog form). These signals are typically processed in electronic and digital form to perform the components or processes described above, but also other physical forms including electronic, light, magnetic and electromagnetic forms. It can also be captured, processed, converted and stored in the form. The content signal is transformed in processing in various ways and for various purposes, thereby producing various data structure representations of the signal and related information. The data structure signals in the memory are then converted for operations during search, alignment, reading, writing and searching. The signal is also converted for capture, transfer, storage, and output by a display or audio converter (eg, a speaker).

  The features and arrangements detailed herein are detailed in co-pending application Ser. No. 12 / 484,115 filed Jun. 12, 2009 (published as US Patent Application Publication No. 2010/0048242). It can be used in combination with the features and configurations described. For example, the object recognition architecture and techniques detailed in application Ser. No. 12 / 484,115 can be used in the practice of this technique. It is understood that the methods, elements and concepts disclosed in this application are intended to be combined with the methods, elements and concepts detailed in the 12 / 484,115 application (and vice versa) I want to be. The implementation of all such combinations will be readily apparent to those skilled in the art from the teachings presented.

  In some embodiments, the calling device captures biometric information (eg, fingerprint, heart rate, retinal pattern, respiratory rate, etc.) from one or more people. This data can again be used by the receiving unit, for example, in determining how to present (whether to present) audio / video data.

  The techniques detailed are particularly useful for user generated content (UGC) sites such as YouTube. The ancillary data detailed herein can be received and stored at the UGC site and later provided to the user for use by other users in the manner described. Alternatively, at the UGC site, the auxiliary data can be used to process the audio / video data and then only the processed audio / video data is provided to the user. (That is, the “receiving unit” shown in FIG. 2 may actually be an intermediate processing unit that provides audio and / or video to yet another unit.)

  Reference was made repeatedly to GPS data. This data should be understood as a concise representation of any location related information. This information may not be obtained from the global positioning system coordinates of the satellite. For example, another technique that is suitable for generating location data relies on radio signals that are normally exchanged between devices (eg, WiFi, cell phones, etc.). Considering some communication devices, the signal itself and the imperfect digital clock signal that controls the signal form a reference system from which both high precision time and position can be extracted. Such a technique is described in detail in WO 08/073347. The person skilled in the art is based on several other location estimation techniques, based on time-of-arrival techniques, and based on the location of radio and television broadcast towers (proposed by Rosum), and based on WiFi nodes. Stuff (suggested by Skyhook Wireless and used in iPhone) and others will be familiar.

  The location information typically includes latitude data and longitude data, but may alternatively include more data, less data, or other data. For example, the position information can include orientation information such as a compass direction obtained from a magnetometer, or tilt information obtained from a gyroscope sensor or other sensor. The location information can also include altitude information, such as that obtained from a digital altimeter system.

  Digimarc has various other patent documents related to this subject. See, for example, Application No. 12 / 835,527, filed July 13, 2010, International Publication No. 20120022185, and Patent No. 6,947,571.

  It is impossible to clearly classify the myriad transformations and combinations of the techniques described herein. Applicants must be able to combine, replace and exchange the concepts herein, both within and between these concepts, as well as combine, replace and exchange with concepts known from the prior art cited. Recognize and intend. Furthermore, it should be understood that the techniques described in detail can be combined with current techniques and other techniques that will emerge from this for advantageous effects.

  In order to provide a broad disclosure without unduly lengthening this specification, Applicants incorporate by reference the literature and patent disclosures referred to above. (Such references are incorporated in their entirety, even if cited above in connection with specific ones of their teachings.) These references are incorporated into the configurations detailed herein. Disclosed are technologies and techniques that can be incorporated, as well as those that can incorporate the techniques and techniques detailed herein.

Claims (33)

Receiving video information and converting the video information for representation within a video portion of a single data object;
Receiving audio information and converting the audio information for representation within the audio portion of a single data object;
Receiving sensor information including at least one parameter relating to acceleration, orientation or tilt, and transforming the sensor information for representation in the single data object;
Transmitting the single data object to a data receiver or storing the single data object on a computer readable medium, wherein the sensor information is structured by the single data object with the audio and video information. And thus adapted for use by a processor in modifying said audio or video information.   The method of claim 1, comprising modifying the audio or video information using a processor according to at least a portion of the sensor information.   The method of claim 1, comprising converting the sensor information for representation within the video portion of the single data object.   The method of claim 1, wherein the single data object comprises an MPEG data stream or an MPEG data file.   The method of claim 1, comprising converting the sensor information to a frequency range near or at the bottom of the range that a person can hear.   The method according to claim 1, comprising expressing the sensor information as a signal hidden by steganography in the audio information or video information.   The method of claim 1, wherein the sensor data includes acceleration data.   The method of claim 1, wherein the sensor data includes orientation data.   The method of claim 1, wherein the sensor data includes tilt data.   Receiving camera data and transforming the single data object for representation in the audio portion, wherein the camera data includes focus, zoom, aperture size, depth of field, exposure time, ISO settings The method of claim 1, further comprising the step of including at least one parameter related to a value and / or depth of focus. Receiving a single data object;
Recovering audio data from the audio portion of the single data object;
Recovering sensor data from the audio portion of the single data object, the sensor data including at least one parameter related to acceleration, orientation, or tilt;
Including
The method wherein at least one of the recovering steps is performed by a hardware processor.   The method of claim 11, wherein the step of recovering sensor data includes the step of subjecting the audio data to a steganographic decoding process to extract the sensor data from the audio data.   The method of claim 11, comprising modifying the recovered voice data according to at least a portion of the sensor data.   The method of claim 11, further comprising recovering video data from a video portion of the single data object and modifying the recovered video according to at least a portion of the sensor data. Receiving a single data object;
Recovering audio data from the audio portion of the single data object;
Recovering camera data from the audio portion of the single data object, wherein the camera data includes focus, zoom, aperture size, depth of field, exposure time, ISO setpoint, and / or depth of focus. Including at least one parameter related to
Including
The method wherein at least one of the recovering steps is performed by a hardware processor.   The method of claim 15, comprising modifying the recovered voice data according to at least a portion of the sensor data.   16. The method of claim 15, further comprising recovering video data from a video portion of the single data object and modifying the recovered video according to at least a portion of the sensor data. Receiving a single data object;
Recovering both video data and sensor data from the single data object, the sensor data including at least one parameter related to acceleration, orientation, or tilt;
Processing the video data according to at least a portion of the sensor data to obtain modified video data for presentation to a user;
Including
A method wherein at least one of the steps is performed by a hardware processor. Obtaining user selection data;
Processing the video data using at least a portion of the sensor data according to the user selection data;
Presenting the processed video to a user;
The method of claim 18 comprising:   The method of claim 19, comprising compensating for vibration of the image by using the sensor data.   20. The method of claim 19, comprising obtaining the user selection data from the user via a user interface. A computer readable storage medium containing non-transitory software instructions, wherein the instructions are on a processor programmed with the instructions,
Recovering both video data and sensor data from a received single data object, wherein the sensor data includes at least one parameter related to acceleration, orientation or tilt;
Processing the video data according to at least a portion of the sensor data to produce modified video data for presentation to a user;
A computer-readable storage medium that executes Receiving a single data object;
Recovering both video data and camera data from the single data object, wherein the camera data includes focus, zoom, aperture size, depth of field, exposure time, ISO setting, and / or lens focus. Including at least one parameter related to the distance;
Processing the video data according to at least a portion of the camera data to obtain modified video data for presentation to a user;
Including
A method wherein at least one of the steps is performed by a hardware processor. Receiving a single data object;
Recovering both audio data and sensor data from the single data object, wherein the sensor data includes at least one parameter related to acceleration, orientation or tilt;
Processing the audio data according to at least a portion of the sensor data to obtain modified audio data for presentation to a user;
Including
A method wherein at least one of the steps is performed by a hardware processor. Receiving a single data object;
Recovering both audio data and camera data from the single data object, wherein the camera data is a focus, zoom, aperture size, depth of field, exposure time, ISO setting, and / or lens focus. Including at least one parameter related to the distance;
Processing the audio data according to at least a portion of the camera data to obtain modified audio data for presentation to a user;
Including
A method wherein at least one of the steps is performed by a hardware processor.   26. The method of claim 25, wherein the step of processing includes a sub-step of modifying audio amplitude according to camera focus data to obtain dimensional audio. Collecting sensor information from sensors carried by a movable object in a sporting event;
Collecting video information from the sporting event using a camera located far away from the movable object;
Creating a single data object that includes both data corresponding to the collected sensor information and data corresponding to the collected video information;
Storing the single data object in a computer readable storage medium or transmitting the single data object to a data receiver;
Including a method.   28. The method of claim 27, comprising collecting the sensor information from a sensor carried by a pack, ball, horse, competitor, or car.   28. The method of claim 27, wherein the sensor data includes at least one of acceleration data, orientation data, position data, and / or tilt data.   A mobile phone comprising a processor and at least a first sensor and a second sensor, wherein the processor detects information sensed by the first sensor and information sensed by the second sensor. A mobile phone configured to create a single data object including, wherein the first sensor comprises an image sensor or an audio sensor and the second sensor comprises an acceleration sensor, an orientation sensor, or a tilt sensor.   The mobile phone of claim 30, wherein the other sensor comprises an acceleration sensor.   The mobile phone of claim 30, wherein the other sensor comprises an orientation sensor.   The mobile phone of claim 30, wherein the other sensor comprises a tilt sensor.

Images