CN111784615A - Method and device for multimedia information processing - Google Patents

Abstract

The invention provides a method for processing video information executed by electronic equipment, which comprises the following steps: obtaining, by a first multimedia capturing device, first video information corresponding to a first focal region; obtaining, by a second multimedia capture device, second video information corresponding to a second focal region; and displaying the first video information and the second video information on a display of the electronic device.

Description

Method and device for multimedia information processing

This application is a divisional application of an invention patent application with an application date of March 25, 2016, an application number of 201610179848.4, and an invention titled "Method and Device for Multimedia Information Processing".

technical field

The present invention relates to the field of multimedia information processing, and in particular, the present invention relates to a method for processing multimedia information and a device for processing multimedia information.

Background technique

With the improvement of living standards, terminal equipment with photographing devices has become more and more popular. Obtaining high-quality images and videos has also become an important factor in enhancing the competitiveness of terminal equipment. Most of the existing shooting enhancements focus on improving the image quality, and the improvement of the video quality is less involved. Compared with image enhancement, video enhancement is relatively difficult to achieve due to the impact of spatial and temporal consistency and limited processing time.

In the existing terminal devices with shooting functions, such as mobile phones, the quality of the collected videos and images often cannot meet the needs of users. Although the quality of images captured by some mid-to-high-end terminal devices has been greatly improved, there is still much room for improvement in video quality, especially for videos captured in low-light environments. Video processing is much more difficult than image processing. There are two main reasons: 1) The processing time of images and video frames is different. For example, in a video with a frequency of 30fps, the processing time of each video frame needs to be controlled within 1/30 of a second, while The image processing time can be longer, so the image enhancement method can flexibly and automatically adjust parameters such as exposure and white balance according to the scene, and even collect multiple images to synthesize one image, which makes the brightness and color of the captured image more accurate than that of the video; 2) Video needs to maintain temporal and spatial consistency, while images have no temporal and spatial constraints. In order to ensure temporal and spatial consistency, the video acquisition parameters between adjacent frames of the video, including white balance, exposure and focus, need to have smooth transitions. Therefore, if the scene has prominent changes , For example, when the lighting changes from indoor to outdoor, the terminal has a lag in adjusting the video acquisition parameters, and the image acquisition parameters are determined according to the brightness and color of the current scene, which also causes the captured image to be more bright and color than the video. precise.

In the prior art, enhancement techniques for images or videos mainly focus on using corresponding algorithms to enhance images or videos, that is, enhancing images or videos based on their own information. Video distortion, lack of clarity and other issues.

SUMMARY OF THE INVENTION

Aiming at the limitation problem of carrying out enhanced processing to multimedia information in the prior art, the present invention proposes a method for processing multimedia information, including:

Acquiring the first type of multimedia information and the second type of multimedia information respectively collected by the two multimedia collection devices;

Corresponding processing is performed on the second type of multimedia information according to the first type of multimedia information.

Wherein, the multimedia information includes at least one of image information, video information, and audio information.

Preferably, the first type of multimedia information is image information, and the second type of multimedia information is video information; or the first type of multimedia information is video information, and the second type of multimedia information is image information.

Preferably, the second type of multimedia information is processed correspondingly according to the first type of multimedia information, specifically including:

Determine the indicators that need to be enhanced corresponding to the collected second type of multimedia information;

According to the collected multimedia information of the first type, enhance processing is performed on the determined index corresponding to the collected multimedia information of the second type.

The indicators include at least one of the following:

Resolution, color, brightness, noise and blur.

Preferably, the index that needs to be enhanced corresponding to the collected second type of multimedia information is determined by at least one of the following:

Determine the matching indicator that needs to be enhanced according to the detected enhanced triggering operation;

Determine the indicators that need to be enhanced according to the preset settings;

The index that needs to be enhanced is adaptively determined according to the adaptive parameter matching method.

Preferably, the adaptive parameter matching mode is determined by one or more pieces of information in the relevant state of the equipment, the enhanced open historical record data, the collection environment, the collection parameters and the relevant content of the multimedia information collected in real time by the multimedia collection equipment;

The device-related status includes at least one of the following: device power status, device storage status, and device motion status when collecting multimedia information;

The relevant content of the multimedia information collected in real time by the multimedia collection device includes at least one of the following: scene brightness, semantic content, and definition of salient objects.

Optionally, the method further includes:

If it is determined that there are at least two indicators that need to be enhanced, then determine the enhancement sequence of the indicators that need to be enhanced;

According to the collected first type of multimedia information, enhance processing is performed on the determined index corresponding to the collected second type of multimedia information, specifically including:

According to the collected multimedia information of the first type, and in accordance with the determined enhancement sequence, the indexes that need to be enhanced corresponding to the collected multimedia information of the second type are sequentially enhanced.

Preferably, the enhancement order of the indicators that need to be enhanced is determined by at least one of the following:

Enhancement sequence setting trigger operation; pre-set; adaptive enhancement sequence setting method.

Preferably, the self-adaptive enhancement sequence setting method is configured by one of the relevant status of the device, the enhancement setting history record information, the collection environment, the collection parameters, the relevant content of the multimedia information collected in real time by the multimedia collection device, and the influence relationship between the various indicators or multiple pieces of information to determine;

The relevant content of the multimedia information collected in real time by the multimedia collection device includes at least one item of scene brightness and semantic content.

Optionally, the method also includes: setting a main collection device and an auxiliary collection device in the two multimedia collection devices;

When the video information is processed according to the image information, the video information is acquired by the main acquisition device, and the image information is acquired by the auxiliary acquisition device;

When the image information is processed according to the video information, the image information is collected through the main collection device, and the video information is collected through the auxiliary collection device.

Preferably, the main collection device and the auxiliary collection device in the two multimedia collection devices are set in at least one of the following ways:

Set the main and auxiliary acquisition devices according to the detected setting trigger operation;

Set the main and auxiliary acquisition equipment according to the preset settings;

The primary and secondary acquisition devices are adaptively set according to the adaptive device setting method.

Preferably, the adaptive device setting mode is determined by one or more pieces of information in the relevant state of the device, the historical record data of the device setting, and the relevant content of the multimedia information collected in real time by the multimedia collection device;

Device-related status includes device power status and/or storage status;

The relevant content of the multimedia information collected by the multimedia collection device in real time includes at least one of the following: screen ratio distribution, position information of the target object in the screen, and screen quality information.

Optionally, the method further includes:

Set the acquisition parameters and enhancement strategy parameters of multimedia information;

Wherein, acquiring the first type of multimedia information and the second type of multimedia information collected by the two multimedia collection devices respectively includes:

Acquiring the first type of multimedia information and the second type of multimedia information respectively collected by the two multimedia collection devices based on the collection parameters;

Wherein, the second type of multimedia information is processed correspondingly according to the first type of multimedia information, specifically including:

According to the enhancement strategy parameter, corresponding enhancement processing is performed on the second type of multimedia information according to the first type of multimedia information;

The acquisition parameters specifically include at least one of white balance, exposure time, sensitivity, high dynamic range, resolution, focus area, and video frame acquisition frequency.

Preferably, the collection parameters and enhancement strategy parameters of multimedia information are set by any of the following methods:

Set acquisition parameters and enhanced strategy parameters according to the detected parameter setting operations;

Set acquisition parameters and enhanced strategy parameters according to pre-parameter settings;

The acquisition parameters and the enhancement strategy parameters are adaptively set according to the adaptive parameter setting method.

Preferably, the adaptive parameter setting method is determined by at least one of equipment-related status, parameter history record data, collection environment, and relevant content of multimedia information collected in real time by the multimedia collection equipment;

The device-related status includes at least one of the following: device power status, device storage status, and device motion status when collecting multimedia information;

The relevant content of the multimedia information collected by the multimedia collection device in real time includes at least one of the following: scene brightness, semantic content, sharpness, resolution, and exposure time of salient objects.

Preferably, the first type of multimedia information is image information, and when the second type of multimedia information is video information, obtain the first type of multimedia information and the second type of multimedia information collected by two multimedia collection devices respectively, specifically including:

Obtain the video information collected by a multimedia collection device, and the key frame image information corresponding to the video information collected simultaneously by another multimedia collection device according to the key frame collection frequency;

Wherein, according to the first type of multimedia information collected, corresponding processing is carried out to the collected second type of multimedia information, specifically including:

According to the collected key frame image information, the indexes that need to be enhanced corresponding to the collected video information are enhanced.

Optionally, the method further includes:

Set the key frame acquisition frequency;

The method of setting the key frame collection frequency includes at least one of the following:

Set the key frame collection frequency according to the preset frequency setting;

Set the key frame collection frequency adaptively according to the adaptive frequency setting method.

Preferably, the adaptive frequency setting mode is determined by one or more information in the relevant content of the multimedia information collected in real time by the equipment related state, the collection frequency historical record data, the collection environment, the collection parameters and the multimedia information collected in real time by the multimedia collection equipment;

The device-related state includes at least one of the following: device power state, device storage state, and device motion state when collecting multimedia information;

The relevant content of the multimedia information collected in real time by the multimedia collection device includes at least one item of scene brightness and semantic content.

Preferably, according to the collected key frame image information, the corresponding indexes of the collected video information that need to be enhanced are enhanced, specifically including:

According to the collected key frame image information, the collected video information is divided into several video clips; the key frame image information on both sides of the video clip is used to enhance the corresponding video clip corresponding indicators that need to be enhanced.

Preferably, when the index to be enhanced includes at least one of resolution, color, and brightness, the enhancement processing method includes an enhancement method based on multi-view reconstruction, and/or an enhancement method based on building an enhanced model based on machine learning.

Preferably, the enhancement method based on multi-view reconstruction specifically includes:

The matching relationship between the video pixels of the collected video information and the image pixels of the key frame image information is established, and the matched video pixels are replaced by the image pixels.

Preferably, the method of constructing an enhanced model based on machine learning specifically includes:

Extract video pixels at the position of the key frame image of the collected video information;

A mapping enhancement model of video pixels and image pixels of key frame image information is established by machine learning;

At the location of the non-key frame image of the captured video information, the video pixels are transformed by the mapping enhancement model.

Preferably, when the index to be enhanced includes noise, the enhancement processing method includes a method based on dictionary reconstruction, and/or a method based on deep learning.

Preferably, when the index to be enhanced includes blur, the enhancement processing method includes a method based on blur kernel estimation, and/or a method based on deep learning.

Optionally, also include:

When a fuzzy frame to be processed is detected in the collected video information, it is determined that the fuzzy index corresponding to the fuzzy frame to be processed is enhanced;

Wherein, the fuzzy frame to be processed is detected by at least one of the following information:

The motion state of the device when collecting video frames; the focusing information when collecting video frames; the classification result of classifying the collected video information through the classifier.

Preferably, the first type of multimedia information is image information, and when the second type of multimedia information is video information, corresponding processing is carried out to the second type of multimedia information according to the first type of multimedia information, specifically including:

The collected video information is stored and processed according to the collected image information, wherein the stored content includes at least one of the following situations:

Video information obtained by enhancing the collected video information according to the collected image information;

The collected video information and image information;

The collected video information and the enhancement model when the video information is enhanced;

The video information and the collected image information after enhancing the collected video information according to the collected image information.

Optionally, also include:

In response to the received playback trigger operation, the video information is played based on the playback mode that matches the stored content; wherein, the playback mode includes at least one of the following:

When the enhanced video information is stored, the enhanced video information is directly played;

When the collected video information and image information are stored, the collected video information is enhanced and played according to the collected image information;

When storing the collected video information and the enhanced model, the collected video information is enhanced and played through the enhanced model;

When the enhanced video information and the collected image information are stored, the enhanced video information and the collected image information are associated and played.

Preferably, the first type of multimedia information is video information, and when the second type of multimedia information is image information, obtain the first type of multimedia information and the second type of multimedia information collected by two multimedia collection devices respectively, specifically including:

Obtain the image information collected by a multimedia collection device, and the video clips corresponding to the image information collected by another multimedia collection device according to the set video frame collection frequency;

Wherein, according to the first type of multimedia information collected, corresponding processing is carried out to the collected second type of multimedia information, specifically including:

The index that needs to be enhanced corresponding to the collected image information is enhanced according to the collected video clips.

Preferably, when it is detected that the multimedia collection device that collects image information enters the preview state, or when it is detected that the multimedia collection device that collects image information starts to collect image information, another multimedia collection device collects and images according to the set video frame collection frequency. Video clips corresponding to the information;

When it is detected that the number of video frames in the captured video clip reaches the corresponding upper limit value, another multimedia capture device stops capturing video information.

Preferably, according to the collected video clips, the indexes that need to be enhanced corresponding to the collected image information are enhanced, specifically including:

Determine video keyframes in the captured video clips;

Based on the method of blur kernel estimation, the collected image information is enhanced according to the video key frame.

Preferably, the video key frame is determined by an adaptive key frame determination method;

Wherein, the adaptive key frame determination method is determined by one or more pieces of information among picture blur degree, content similarity, and video frame quality.

Optionally, also include:

Sharpness analysis of the collected image information;

If the image information belongs to a blurred image, the index that needs to be enhanced corresponding to the collected image information is enhanced according to the collected video clip; wherein, the index that needs to be enhanced is included in the blur.

Preferably, the first type of multimedia information is video information, and when the second type of multimedia information is image information, the second type of multimedia information is processed according to the first type of multimedia information, specifically including:

The collected image information is stored and processed according to the collected video information, wherein the stored content includes at least one of the following situations:

Image information obtained by enhancing the collected image information according to the collected video information;

The collected video information and image information;

The collected image information and the video key frames used for enhancing the image information in the collected video information;

The collected image information and the enhancement model when the image information is enhanced;

According to the collected video information, the collected image information is enhanced and processed, and the collected video information is obtained.

Optionally, also include:

In response to the received display trigger operation, the image information is displayed based on a display mode that matches the stored content; wherein, the display mode includes at least one of the following:

When the enhanced image information is stored, the enhanced image information is directly displayed;

When the collected video information and image information are stored, the collected image information is enhanced and displayed according to the collected video information;

When storing the collected image information and the video key frame for enhanced processing, the enhancement model is determined according to the video key frame, and the collected image information is enhanced and displayed through the enhanced mode;

When the collected image information and enhanced model are stored, the collected image information is enhanced and displayed through the enhanced model;

When the enhanced image information and the collected video information are stored, the enhanced image information and the collected video information are associated and displayed.

Preferably, the first type of multimedia information and the second type of multimedia information are video information focused on different focus areas; wherein the focus areas include global areas and/or local areas.

Wherein, the second type of multimedia information is processed correspondingly according to the first type of multimedia information, specifically including:

According to the collected video information focused on one focal area, perform joint playback processing on the collected video information focused on another focal area.

Wherein, the focus area is determined by at least one of the following ways:

When it is detected that a local area is selected by the user, it is determined that the selected local area is the focus area, and the other focus area is the global area;

When it is detected that the user has selected two local areas, it is determined that the selected two local areas are focus areas.

Preferably, the user-selected local area is detected through the user-selected focus object.

Preferably, the global area and/or the local area can be jointly played in a split-screen layout.

Preferably, corresponding processing is performed on the second type of multimedia information according to the first type of multimedia information, specifically including:

According to the collected video information focusing on one focus area, the collected video information focusing on another focus area is stored and processed, wherein the storage content includes at least one of the following situations:

The collected two video information focused on different focus areas;

According to the collected video information focusing on one focal area, the synthesized video information after synthesizing the collected video information focusing on another focal area;

The video content of interest in the determined two video information focused on different focus areas;

The collected video information focused on the global area and the location information of the local area in the video information of the global area.

Preferably, the method also includes: in response to the received play trigger operation, the video information is played based on a play mode that matches the stored content; wherein, the play mode includes at least one of the following:

When storing the collected two video information focusing on different focus areas, play the two video information separately or jointly;

When the composite video information is stored, the composite video is played;

When storing the video content of interest in the two video information determined to focus on different focus areas, play the video content of interest;

When storing the video information of the global area and the location information of the local area in the video information of the global area, the video information of the local area is determined by the location information, and the video information of the global area and the video information of the local area are played separately or jointly play.

Preferably, the second type of multimedia information is video information, and the first type of multimedia information is audio information corresponding to the video information.

Wherein, the second type of multimedia information is processed correspondingly according to the first type of multimedia information, specifically including:

Determine the target object from the collected video information;

Highlight processing is performed on the video information and/or audio information corresponding to the target object.

Wherein, the target object is determined from the collected video information in at least one of the following ways:

Determine the target object according to the specified operation of the detected target object;

The target object is determined according to the number and location information of multiple objects in the collected video information.

Preferably, highlighting processing is performed on the audio information corresponding to the target object, which specifically includes:

The collected video information is detected to determine the number of objects in the video information, the position and orientation information of each object;

Determine the audio information corresponding to each object according to the position and orientation information of each object;

Determine the audio information corresponding to the target object, and perform highlight processing.

Preferably, the second type of multimedia information is processed correspondingly according to the first type of multimedia information, specifically including:

According to the collected audio information, the collected video information is stored and processed, wherein the stored content includes at least one of the following situations:

The collected video information and audio information;

Video information and audio information corresponding to the target object.

Preferably, the method also includes: in response to the received play trigger operation, the video information and the audio information are played based on a play mode that matches the stored content; wherein, the play mode includes at least one of the following:

When storing the collected video information and audio information, the collected video information and audio information are associated and played;

When storing the collected video information and audio information, the target object in the collected video information is played in association with the corresponding audio information;

When storing the collected video information and audio information, each object in the collected video information is played in association with the corresponding audio information;

When the video information and audio information corresponding to the target object are stored, the video information and audio information corresponding to the target object are associated and played.

The present invention also provides a device for multimedia enhancement processing, including:

a multimedia information acquisition module, configured to acquire the first type of multimedia information and the second type of multimedia information collected by the two multimedia acquisition devices respectively;

The processing module is configured to perform corresponding processing on the second type of multimedia information according to the first type of multimedia information.

In the embodiment of the present invention, the first type of multimedia information and the second type of multimedia information respectively collected by the two multimedia collection devices are acquired; the second type of multimedia information is processed according to the first type of multimedia information, that is, the The association between the information and the second type of multimedia information implements corresponding processing of the second type of multimedia information based on the first type of multimedia information. However, in the prior art, each type of multimedia information (such as pictures and videos) is generally processed separately through enhancement algorithms and its own information, without considering the association relationship between the two types of multimedia information acquired at the same time, and using the association relationship Performs multimedia information enhancement, so problems such as image or video picture distortion and low definition may occur. In the present invention, two types of multimedia information are acquired at the same time, and another type of multimedia information is enhanced according to one type of multimedia information. Since the respective characteristics and associations of the two types of multimedia information are fully considered in the enhancement process, it is possible to overcome the problem of using the enhancement algorithm only. The limitation of enhancing processing for each type of multimedia information separately from its own information greatly improves the quality of the enhanced multimedia information and ensures the authenticity and clarity of the multimedia information.

Additional aspects and advantages of the present invention will be set forth in part in the following description, which will become apparent from the following description, or may be learned by practice of the present invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

1 shows a schematic flowchart of a method for processing multimedia information according to an embodiment of the present invention;

Fig. 2 is a kind of schematic diagram of the conversion curve brightness adjustment to video brightness and color enhancement method in the prior art;

Fig. 3 is the schematic diagram that utilizes the blur kernel estimation method of blurred image to denoise video frame in the prior art;

4 shows a schematic diagram of the execution steps of the video joint enhancement mode according to a specific embodiment of the present invention;

Fig. 5 shows the schematic diagram of collecting video in the handheld intelligent terminal of the specific embodiment of the present invention;

Fig. 6 shows the schematic diagram of the joint video enhancement mode in the handheld intelligent terminal of the specific embodiment of the present invention;

7 shows a schematic diagram of a joint video enhancement mode in a monitoring terminal according to a specific embodiment of the present invention;

8 shows a first schematic diagram of an image deblurring enhancement mode according to a specific embodiment of the present invention;

9 shows a second schematic diagram of an image deblurring enhancement mode according to a specific embodiment of the present invention;

Fig. 10 shows the schematic diagram of the left and right split screen video layout mode of the multi-focus area joint play mode of the specific embodiment of the present invention;

Fig. 11 shows the schematic diagram of the upper and lower split screen video layout mode of the multi-focus area joint play mode of the specific embodiment of the present invention;

Fig. 12 shows the schematic diagram of the video layout mode of the large and small screens of the multi-focus area joint play mode of the specific embodiment of the present invention;

Fig. 13 shows the schematic diagram of the global area video layout mode of the multi-focus area joint play mode of the specific embodiment of the present invention;

14 shows a schematic diagram of switching between large and small screens in the large and small screen video layout mode of the multi-focus area joint playback mode of a specific embodiment of the present invention;

Fig. 15 shows the schematic diagram of recording and playing of large and small screens in the large and small screen video layout mode of the multi-focus area joint playback mode of the specific embodiment of the present invention;

Fig. 16 shows the schematic diagram of audio and video highlighting in the target object highlighting play mode of the specific embodiment of the present invention;

FIG. 17 shows a schematic structural diagram of an apparatus for processing multimedia information according to an embodiment of the present invention.

Detailed ways

The following describes in detail the embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, but not to be construed as a limitation of the present invention.

Terms such as "module" and "system" used in this application are intended to include computer-related entities such as, but not limited to, hardware, firmware, a combination of software and hardware, software, or software in execution. For example, a module can be, but is not limited to, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. For example, both an application running on a computing device and the computing device can be modules. One or more modules may be localized within a process and/or thread of execution, and a module may be localized on a single computer and/or distributed between two or more computers.

Fig. 1 shows a schematic flowchart of a method for processing multimedia information according to an embodiment of the present invention.

Step S110: Acquire the first type of multimedia information and the second type of multimedia information collected by the two multimedia collection devices respectively; Step S120: Perform corresponding processing on the second type of multimedia information according to the first type of multimedia information.

It should be noted that the two multimedia collection devices may be placed on the same terminal device, and the above-mentioned terminal device may be a terminal such as a mobile phone, a Pad, and a monitoring device. Two multimedia collection devices simultaneously collect multimedia information, one of which collects the first type of multimedia information, and the other collection device collects the second type of multimedia information.

It can be seen from the above that the first type of multimedia information and the second type of multimedia information are interrelated; while in the prior art, generally, multimedia information (such as pictures and videos) is separately processed only through enhancement algorithms and self-information, and The correlation relationship between the two types of multimedia information acquired at the same time is not considered, and the multimedia information enhancement is performed by using the correlation relationship, so problems such as image or video picture distortion and low definition may occur. In the present invention, by acquiring two types of multimedia information at the same time, and enhancing the second type of multimedia information according to the first type of multimedia information, since the respective characteristics and associations of the two types of multimedia information are fully considered in the enhancement process, it is possible to It overcomes the limitation that each type of multimedia information is separately enhanced through enhancement algorithms and its own information, greatly improves the quality of the enhanced multimedia information, and ensures the authenticity and clarity of the multimedia information.

Specifically, the multimedia information includes at least one of image information, video information, and audio information.

In order to improve the camera configuration, the binocular camera has become a highlight of major manufacturers' R&D and promotion. Compared with the monocular camera, the binocular camera has some natural advantages: it has two sets of acquisition parameters, the two cameras can be set to different shooting modes, and the three-dimensional depth information can be obtained to improve the accuracy of segmentation, recognition, tracking and positioning. Most of the existing terminal devices with binocular cameras use depth information to provide more operation modes for image capture, such as combining images captured by the left and right cameras to obtain a high-resolution image; using depth information to segment the target area; panorama. Focus after taking a deep photo, etc. The inventor of the present invention finds that the prior art does not make full use of the parameter collection of two cameras and double sets to enhance the image and video quality, and how to make full use of the advantages of the binocular camera to improve the existing video and image quality also needs to be solved The problem.

It should be noted that, in the present invention, the multimedia collection device may be a binocular camera on a terminal device, or other implementation manners may be adopted, which is not limited herein. In addition, the two cameras can be placed side by side to simulate the function of human eyes, and the two cameras can also be set in other ways, which are not limited here.

Wherein, when the first type of multimedia information is image information, the second type of multimedia information may be video information. Specifically, one camera of the terminal device collects video information, the other camera simultaneously collects corresponding image information, and uses the collected images to enhance the video. At this time, the enhancement processing mode of the terminal device can be called image enhancement video mode.

Or when the first type of multimedia information is video information, the second type of multimedia information may be image information. Specifically, one camera of the terminal device collects image information, the other camera simultaneously collects corresponding video information, and uses the collected video to enhance the image. At this time, the enhancement processing mode of the terminal device can be called a video enhanced image mode.

Since the image or video collected by the camera corresponds to many indicators, such as brightness, resolution, etc., when using image to enhance video or using video to enhance image, one or more indicators of the video or image can be enhanced. The embodiment of the invention proposes to perform corresponding processing on the second type of multimedia information according to the first type of multimedia information, which specifically includes:

Determine the index that needs to be enhanced corresponding to the collected multimedia information of the second type; and perform enhancement processing on the determined index corresponding to the collected multimedia information of the second type according to the collected multimedia information of the first type.

Wherein, the above indicators include at least one of the following: resolution, color, brightness, noise and blur.

It should be noted that, in the detailed description of the following embodiments, the corresponding processing method for the second type of multimedia information will be performed according to the first type of multimedia information, specifically the image enhancement video mode in which the image information processes the video information, the video information Video enhanced image mode for processing image information, multi-focus area joint playback mode for video information processing, and target object highlight playback mode for audio information processing video information. Among them, the first embodiment to the eighth embodiment are specific embodiments of the image-enhanced video mode; the ninth embodiment is a specific embodiment of the video-enhanced image mode; the tenth embodiment is a specific embodiment of the multi-focus area joint play mode; the eleventh embodiment A specific example of highlighting the play mode for the target object.

In Embodiments 1 to 8 of the present invention, for the collected video and images, the images are used to enhance the video, and the corresponding determined indicators include the following five types: resolution, brightness, color, noise and blur: shooting high High-resolution images, enhance the video resolution to obtain high-resolution videos; shoot high-quality images, adjust the brightness of the video to improve the brightness of videos captured in low-light environments; shoot high-quality images, color the video. Adjust to improve the color contrast and RGB color distribution of the video shot in a non-ideal shooting environment; shoot a low-noise image, denoise the video, and obtain a low-noise and high-quality video; shoot a clear image, use the image to de-noise the video frame Blur, get sharper video. In the ninth embodiment of the present invention, the video is used to enhance the image, and the corresponding determined indicators include blur: when shooting a long-exposure high-brightness image, if the image is blurred, the short-exposure video frame is used to enhance the blurred image. Enhance the image to get high brightness and good definition.

In the image enhancement video process, first set the main/auxiliary camera, indicators to be enhanced, acquisition parameters, and enhancement strategy parameters, start two cameras to shoot, the main camera shoots video, and the auxiliary camera shoots images, and through the enhancement strategy parameters, according to The captured image performs enhancement processing on indicators that need to be enhanced in the captured video, and the terminal device may store the collected data as required, and subsequently play or display the stored data.

In the video enhancement image process, first set the main/auxiliary camera, indicators to be enhanced, acquisition parameters, and enhancement strategy parameters, start two cameras to shoot, the main camera captures images, and the auxiliary camera captures video, and through the enhancement strategy parameters, according to The captured video performs enhancement processing on the indicators that need to be enhanced in the captured image.

It should be noted that, in the detailed description of the following embodiments, according to the collected first type of multimedia information, in the step of performing enhancement processing on the determined indicators corresponding to the collected second type of multimedia information, the different determined indicators may be Enhancement processing is performed on the index of the index, and the index of the enhancement processing is different, and the corresponding enhancement mode is also different. Among them, one of the above indicators can be enhanced, and the enhancement mode includes but is not limited to: resolution enhancement mode, color enhancement mode, brightness enhancement mode Enhancement Mode, Denoising Enhancement Mode, and Deblur Enhancement Mode. Enhancement processing may also be performed on at least two of the above-mentioned indicators, which may be referred to as a joint enhancement mode.

Embodiment 1: The enhancement processing mode is the resolution enhancement mode in the image enhancement video mode

In terms of video resolution enhancement, the maximum resolution of video in existing terminal equipment is generally smaller than the maximum resolution of images; for example, the maximum resolution of images in a class of mobile terminals is 5312*2988, while the maximum resolution of videos is 3840 *2160. Due to the limitation of the CPU and memory of the terminal device, in order to allow users to see their own content in real time, the video resolution can only be reduced compared to the image resolution, otherwise the video cannot be processed in real time. One way to increase the video resolution is to increase the resolution of each video frame by interpolation, and the details of the video frame obtained by such methods will be blurred. Another way to improve video resolution is to use high-resolution images to enhance low-resolution videos. The main idea is to train a mapping model with a large number of high-resolution images and corresponding low-resolution videos, and use the mapping model to enhance the video. Resolution, that is, extracting pixel blocks of images and videos to establish a training database, and learning a mapping model based on the training database, so as to use the mapping model to transfer high-frequency detail information to low-resolution videos to obtain high-resolution videos. Such methods need to collect training data in advance. If the amount of training data is small, the generalization energy will be weak, resulting in poor enhancement effect. If the amount of training data is large, the storage space will be large. In terms of video resolution enhancement, only relying on the information of each frame in the video to enlarge, only changes the image size, can not provide users with richer detailed information, and can not achieve the purpose of improving the resolution. In addition, the high-resolution video occupies a larger memory space, and how to improve the video resolution without causing too much memory consumption is also a problem that the prior art has not considered.

In the embodiment of the present invention, the multimedia information includes image information and video information. The multimedia acquisition device can be a binocular camera.

First, enable the resolution enhancement mode of the image enhancement video, then enable the two cameras of the binocular camera to collect image information and video information respectively, and then set the camera's acquisition parameters and key frames adaptively, and at the same time according to the camera's acquisition parameters and key frames The video information is enhanced in resolution, and finally the enhanced result is compressed, transmitted and played.

Step 1, enable the resolution enhancement mode of the image enhancement video.

The index that needs to be enhanced corresponding to the collected second type of multimedia information is determined by at least one of the following:

Determine the matching indicator that needs to be enhanced according to the detected enhanced triggering operation;

Determine the indicators that need to be enhanced according to the preset settings;

The index that needs to be enhanced is adaptively determined according to the adaptive parameter matching method.

Wherein, the adaptive parameter matching method is determined by one or more pieces of information in the relevant state of the device, the enhanced open history record data, the collection environment, the collection parameters, and the relevant content of the multimedia information collected in real time by the multimedia collection equipment;

The device-related state includes at least one of the following: device power state, device storage state, and device motion state when collecting multimedia information; the relevant content of multimedia information collected by the multimedia collecting device in real time includes at least one of the following: scene brightness, semantic content , the sharpness of salient objects. Preferably, if the determined indicators that need to be enhanced are at least two, then determine the enhancement order of the indicators that need to be enhanced; Corresponding indicators that need to be enhanced are sequentially enhanced.

Specifically, the enhancement order of the indicators that need to be enhanced is determined by at least one of the following:

Enhancement sequence setting trigger operation; pre-set; adaptive enhancement sequence setting mode.

Among them, the adaptive enhancement sequence setting method uses one or more of the relevant status of the device, the enhancement setting history record information, the collection environment, the collection parameters, the relevant content of the multimedia information collected in real time by the multimedia collection equipment, and the influence relationship between the various indicators. item information to determine;

The relevant content of the multimedia information collected in real time by the multimedia collection device includes at least one item of scene brightness and semantic content.

In Embodiment 1 of the present invention, specifically, the user can turn on the resolution enhancement mode of the image-enhanced video by enhancing the triggering operation such as voice, keys, gestures, biometrics, external controllers, etc., and the terminal device can also use preset ( Such as the system default setting), or according to the adaptive parameter matching mode to open the mode adaptive mode matching mode, such as according to the relevant status of the device, the enhanced mode to open the historical record data, the collection environment, the collection parameters and the correlation of the multimedia information collected in real time by the camera The resolution enhancement mode of image-enhanced video is enabled by content adaptation, and it can also be enabled by any combination of user-triggered operation settings and system default settings; Mode Enable History Data Adaptive Enable Resolution Enhancement Mode in Image Enhancement Video Mode.

Regarding the voice activation, the terminal device presets a certain voice as the activation command, such as "start enhanced video resolution", and the terminal device receives the voice control command "start enhanced video resolution" issued by the user, and then performs voice recognition on the voice control command. , confirm that the resolution enhancement mode in the image enhancement video mode is turned on at this time.

Regarding key-on, the terminal device pre-sets a certain key as the turn-on command. The key can be a hardware key. For example, the user presses the volume key for a long time to increase the video resolution. After the terminal device receives the user's long-press volume key event, it confirms this When you need to turn on the resolution enhancement mode in the image enhancement video mode. The button can also be a virtual button, such as a virtual control button on the screen. The terminal device displays the button of this mode on the interactive interface. After receiving the event of the user clicking the virtual button, it is confirmed that the mode needs to be enabled at this time. When it is turned on by pressing the button, it can also be combined with the pressure, speed, time, frequency and other feature information when the user triggers the different representations corresponding to different meanings.

Regarding gesture activation, the terminal device pre-sets a certain gesture as the activation command. The gesture includes screen gestures, such as double-clicking the screen/or long-pressing the screen. Indicates corresponding different meanings; such as light press, for example, the pressure is less than the first predetermined value, heavy press, for example, the pressure is greater than or equal to the first predetermined value, and long press, for example, the pressing duration exceeds the second predetermined value, fast Double-click any one of them to open. Gestures also include space gestures, such as shaking/flipping/tilting the terminal, different directions, angles, speeds, and strengths when shaking/flipping/tilting can represent different meanings, such as shaking up and down, shaking left and right, drawing a circle in an empty space, etc. Indicates that this mode is turned on. The above gestures can be a single gesture or any combination of any gestures, such as long pressing the screen and shaking the terminal device.

Regarding the biometric feature, biometric features include but are not limited to handwriting features and fingerprint features. For example, if the fingerprint detected by the terminal device is consistent with the pre-registered user fingerprint, it is confirmed that the resolution enhancement mode in the image enhancement video mode needs to be turned on at this time. .

Regarding the system default setting of ON, the terminal device sets the resolution enhancement mode in the image enhancement video mode to ON or OFF by default without user interaction.

Regarding the adaptive opening according to the device-related state of the terminal device, the device-related state includes power, storage (such as memory), motion state, etc., and the first and second predetermined power can be set, wherein the first predetermined power, for example 20%, is less than the first predetermined power. 2. A predetermined power level, such as 80%, when the power level of the terminal device is less than the first predetermined power level, the video resolution enhancement mode is turned off, and when the power level is greater than the second predetermined power level, the video resolution enhancement mode is turned on, or only one set to enable Power, when the terminal power is greater than the power on, the video resolution enhancement mode is turned on by default, otherwise the video resolution enhancement mode is turned off.

Regarding the adaptive startup based on the enhanced startup history data, count the number of recent times, such as 10 times, the number of times the video resolution enhancement mode is enabled during the collection process. rate enhancement mode, otherwise turn off the resolution enhancement mode. Or it can be determined whether the current shooting is enabled according to the settings of the previous shooting.

Regarding the collection environment, it is collected by sensors, for example, the environment brightness and other information are collected by the brightness sensor; it can be adaptively turned on according to the environmental brightness, for example, when the average brightness of the environment is lower than the set threshold, the mode is turned on, otherwise the mode is turned off.

Regarding the acquisition parameters, the acquisition parameters specifically include at least one of white balance, exposure time, sensitivity, high dynamic range, resolution, focus area, and video frame acquisition frequency. It can be turned on adaptively according to the acquisition parameters. For example, when the exposure time of the video is too long (higher than the set threshold), this mode is turned on, otherwise it is turned off.

Regarding the adaptive activation according to the relevant content collected in real time, specifically, the relevant content collected in real time includes scene brightness, semantic content, clarity of salient objects, and the like. It can be turned on adaptively according to the brightness of the scene. For example, when the average brightness of the scene is lower than the set threshold, the mode will be turned on, otherwise, the mode will be turned off. It can be turned on adaptively according to the semantic content of the scene. For example, when it is detected that there are target objects in the scene, such as vehicles, people, etc., the mode will be turned on, otherwise, the mode will be turned off. The signal-to-noise ratio of the salient area of the scene, such as the license plate area, can be detected. If the signal-to-noise ratio of the salient area is lower than a given threshold, the terminal device automatically turns on the resolution enhancement mode.

Step 2: Use one camera of the binocular camera to capture images and the other camera to capture video.

This step includes setting main and auxiliary cameras, setting camera parameters and enhancement strategy parameters, and setting key frame selection.

Set the main collection device and the auxiliary collection device in the two multimedia collection devices;

When the video information is processed according to the image information, the video information is acquired by the main acquisition device, and the image information is acquired by the auxiliary acquisition device;

When the image information is processed according to the video information, the image information is collected through the main collection device, and the video information is collected through the auxiliary collection device.

The main collection device and the auxiliary collection device in the two multimedia collection devices are set in at least one of the following ways:

Set the main and auxiliary acquisition devices according to the detected setting trigger operation;

Set the main and auxiliary acquisition equipment according to the preset settings;

The primary and secondary acquisition devices are adaptively set according to the adaptive device setting method.

Wherein, the adaptive device setting mode is determined by one or more pieces of information in the relevant state of the device, the historical record data of the device setting, and the relevant content of the multimedia information collected in real time by the multimedia collection device;

Wherein, the relevant state of the device includes the power state and/or storage state of the device; the relevant content of the multimedia information collected in real time by the multimedia collection device includes at least one of the following: screen ratio distribution, position information of the target object in the screen, and screen quality information.

Step 2.1, set the primary and secondary cameras.

To make the main camera collect video information and the auxiliary camera to collect image information, the terminal device can use one of the following three methods to set the main camera and the auxiliary camera: one is the terminal device preset (such as the default setting); the second is the terminal device receives The user performs settings through a setting trigger operation sent by at least one method such as buttons, gestures, and external controllers; third, the terminal device sets which one to set according to the relevant state of the device, the historical record data of device settings, and the relevant content collected in real time. The camera is the main camera, which camera is the secondary camera.

Regarding the system default settings, the terminal device defaults to one of the cameras as the main camera and the other as the auxiliary camera. For example, the camera on one side of the scene is the main camera, and the camera on the other side is the auxiliary camera.

Regarding the button settings, the terminal device is preset to control the main and auxiliary cameras through the buttons. The buttons can be hardware buttons, for example, the volume "+" button activates the main camera on one side, and the volume "-" button activates the other side camera as the main camera. The keys can also be virtual keys, such as virtual controls on the screen, menus, etc.

Regarding gesture settings, the terminal device presets a certain gesture to switch between the main camera and the auxiliary camera. For example, swiping a circle clockwise means setting one side as the main camera, and swiping a circle counterclockwise means setting the other side as the auxiliary camera.

Regarding external controllers, external controllers include but are not limited to: stylus, remote control, smart glasses, smart headsets and other devices that can access end devices through one of the following technologies: wifi, NFC, Bluetooth and For data network, the device is equipped with a control area such as buttons or a touch screen to control the startup of the main and auxiliary cameras. For example, the up key on the remote control means that one side of the camera is the main camera, and the down key means that the other side is the main camera.

Regarding the adaptive setting based on the historical record data of the device settings, count the most recent times, such as 9 times, the setting method of the main and auxiliary cameras in the collection. The camera on that side is automatically set as the main camera at startup. Or determine the main and auxiliary cameras in this shooting according to the settings in the previous shooting.

Regarding the terminal device adaptively setting the camera according to the relevant content collected in real time, the terminal device selects the main camera according to the content collected by the two cameras. Scoring parameters include, but are not limited to: screen ratio distribution, position information of the target object in the screen, screen quality information, etc. The terminal device can adaptively adjust the main camera in real time according to the shooting picture during the shooting process, and splices the video clips shot by different main cameras according to the shooting time to obtain a complete and uninterrupted video sequence.

If the resolution enhancement mode of the image-enhanced video is enabled during the video shooting process, the camera that currently captures video information can be set as the main camera, and the other camera is the auxiliary camera to capture image information; it can also be adaptive based on the above real-time acquisition content. Set up the camera.

Step 2.2, set the acquisition parameters and enhancement strategy parameters of the camera.

Specifically, setting the acquisition parameters and enhancement strategy parameters of multimedia information;

Acquiring the first type of multimedia information and the second type of multimedia information respectively collected by the two multimedia collection devices based on the collection parameters; and performing corresponding enhancement processing on the second type of multimedia information according to the first type of multimedia information according to the enhancement strategy parameter;

The acquisition parameters specifically include at least one of white balance, exposure time, sensitivity, high dynamic range, resolution, focus area, and video frame acquisition frequency.

Wherein, the collection parameters and enhancement strategy parameters of multimedia information are set in any of the following ways:

Set acquisition parameters and enhanced strategy parameters according to the detected parameter setting operations;

Set acquisition parameters and enhanced strategy parameters according to pre-parameter settings;

The acquisition parameters and the enhancement strategy parameters are adaptively set according to the adaptive parameter setting method.

Wherein, the adaptive parameter setting mode is determined by at least one of the relevant state of the equipment, the parameter history record data, the collection environment and the relevant content of the multimedia information collected in real time by the multimedia collection equipment;

The device-related state includes at least one of the following: device power state, device storage state, and device motion state when collecting multimedia information; the relevant content of multimedia information collected by the multimedia collecting device in real time includes at least one of the following: scene brightness, semantic content , the sharpness, resolution and exposure time of salient objects.

Specifically, the acquisition parameters of the binocular camera device and the internal enhancement strategy parameters in the enhancement process are set. The acquisition parameters of the camera are the camera parameters that need to be set in the acquisition process, and the enhancement strategy parameters are the internal parameters in the selected video enhancement method. The setting methods include but are not limited to the following four: The first is the pre-parameter setting (such as the system default fixed value setting), that is, the parameter is defined as a fixed value, for example, in the brightness enhancement method based on machine learning, the pixel block size is set. It is a fixed value, such as 5 pixels; the second is that the terminal device receives the parameter setting operation sent by the user through at least one method such as voice, button or external controller to set parameters; Parameter historical record data, collection environment or related content collected in real time, etc. adaptively set acquisition parameters and enhanced strategy parameters; the fourth is adaptive setting combined with user adjustment settings to set parameters. Adjust some parameter values by pressing keys.

Regarding the voice settings under user interaction, for example, the terminal device pre-sets the voice-on command “capture high dynamic range images”. If the terminal device receives this command, it will perform voice recognition on the voice control command and determine to enable the capture of high dynamic range images. Or, the terminal device pre-sets the voice turn-on command "image white balance is fluorescent light", then the image white balance is set to fluorescent light. Alternatively, the terminal device pre-sets a voice on command "increase the exposure value of the image" to increase the exposure value of the image.

Regarding the key settings under user interaction, the keys can be hardware keys, for example, the "+" key means to increase the exposure value, the "-" key means to reduce the exposure value, and "HOME" means to capture high dynamic range images. The button can also be a virtual button, such as a slide bar, button, menu, etc. on the screen. The virtual button is arranged on the interactive interface. After the terminal device detects the event that the user clicks the virtual button, it confirms to change the setting parameters. It can also be combined with the pressure, speed, time, frequency and other characteristic information of the user to express different meanings. For example, pressing lightly means reducing the exposure value, and pressing it again means increasing the exposure value.

Regarding the external controller settings under user interaction, the external controllers include, but are not limited to, a stylus, a remote control, a smart watch, smart glasses, a smart head-mounted device, a smart clothes, or a remote device, etc. These controllers use at least the following technologies One to access the terminal device: Wifi, infrared, Bluetooth, network, the controller is equipped with a control area such as buttons or touch screen to control the terminal device, such as a stylus to make air gestures, set up to increase the exposure value, down to increase the exposure value Reduce the exposure value. The terminal device recognizes these operations and starts to adjust the parameters. The remote control sets buttons to adjust white balance, exposure, and video capture frequency. The controller detects that the user has clicked the buttons and sends them to the terminal device to adjust the parameters.

Regarding the adaptive settings based on scene brightness and semantic content, the white balance can be adjusted according to the type of the scene. For example, if the terminal device recognizes that the scene is daytime, adjust the white balance to daylight, and adjust the white balance to tungsten when the scene is yellowish at night. lamp. Focus can be adjusted based on objects of interest in the scene, such as locating a salient area or a body area and focusing on that area. The exposure can be adjusted according to the scene light. For example, if the average brightness of the scene is detected to be lower than a given threshold, the terminal device will increase the exposure, otherwise it will decrease the exposure. The number of images taken in the high dynamic range can be adaptively adjusted according to the brightness variance in the scene. For example, if the brightness variance is higher than the first given threshold, the number of images taken is increased, and if the brightness variance is lower than the second given threshold, the number of images taken is reduced. The size of the image pixel block in brightness enhancement can be set according to the size of the captured image, for example, the pixel block is defined as the image size multiplied by a certain scale factor.

Regarding the adaptive setting according to the relevant state of the device, the relevant state of the device includes power, storage (such as memory), etc., and the exposure value can be controlled according to the power to reduce the calculation amount. When the power is less than the first predetermined power, such as 50%, then reduce the exposure value , when the power level is less than the second predetermined power level, such as 5%, the high dynamic range setting is not performed, wherein the first predetermined power level is greater than the second predetermined power level. It is also possible to reduce the amount of calculation according to the parameters in the power control deblurring. If it is less than the first predetermined power, such as 50%, the size of the blur kernel is reduced, and if it is less than the second predetermined power, such as 5%, the size of the blur kernel is fixed to the minimum value. The amount of calculation can be reduced by power control based on the parameters in the color enhancement of machine learning. If it is less than the first predetermined power, such as 50%, the number of words in the word bank will be reduced. If it is less than the second predetermined power, such as 5%, the word will be fixed. The number is the minimum value, and the parameters in the brightness enhancement can be determined by the power to reduce the calculation amount. If it is less than the first predetermined power, such as 50%, the number of pixels to be sampled based on the machine learning method is reduced, which is less than the second predetermined power, such as 20 %, then replace the model mapping method in the machine learning method with the Gaussian mixture model method. The video frame capture frequency can be adjusted according to the memory. If the remaining memory is larger than the first predetermined space, such as 1G, it will be automatically adjusted to the specified high capture frequency, such as 3640*1920; otherwise, if the remaining memory is smaller than the second predetermined space, such as 300M, Then adjust to the specified low acquisition frequency, such as 1920*1080.

Regarding the adaptive setting according to the parameter historical record data, for example, adjusting the exposure amount according to the exposure value set by the user and the historical record data, the setting methods include but are not limited to the following: calculating the exposure value and the average image brightness in the parameter historical record data, A mapping table is regressed by the least squares method, and the exposure value is adjusted according to the mapping table. For example, adjust the high dynamic range setting according to the user's preferences. The setting methods include but are not limited to the following, count the most recent N times, such as 10 times, the number of times the high dynamic range is set in the brightness enhancement, if the number of times > N/2, it is set first High dynamic range. Or set the parameter value of the current shooting to the parameter value of the previous shooting.

Step 2.3, adaptively set the key frame collection frequency.

When the first type of multimedia information is image information and the second type of multimedia information is video information, the video information collected by one multimedia collection device and the corresponding video information collected simultaneously by another multimedia collection device according to the key frame collection frequency are obtained. Key frame image information; according to the collected key frame image information, the index that needs to be enhanced corresponding to the collected video information is enhanced.

Preferably, it also includes: setting the key frame collection frequency; wherein, the method for setting the key frame collection frequency includes at least one of the following:

Set the key frame collection frequency according to the preset frequency setting;

Set the key frame collection frequency adaptively according to the adaptive frequency setting method.

Wherein, the adaptive frequency setting mode is determined by one or more pieces of information in the relevant state of the equipment, the historical record data of the collection frequency, the collection environment, the collection parameters, and the relevant content of the multimedia information collected in real time by the multimedia collection equipment;

The device-related state includes at least one of the following: device power state, device storage state, and device motion state when collecting multimedia information; the relevant content of the multimedia information collected by the multimedia collecting device in real time includes at least one of scene brightness and semantic content .

Regarding the preset frequency setting, you can select key frames according to the fixed frequency of the system. For example, the capture frequency is fixed to a given frequency, such as 1 time/second. If the video capture frequency is 30 frames/second, it means that an image is captured every 30 frames.

Regarding the adaptive setting based on the relevant content collected in real time, the key frame collection frequency can be adaptively selected according to the environmental brightness transformation, and the average brightness value of the environment can be detected in real time. If the difference between the average brightness values of adjacent frames is greater than the first threshold, such as 50, then Initiate keyframe selection. Keyframes can be adaptively selected according to the semantic content, and the overall feature description factors of the environment can be extracted, including but not limited to the following: color histogram, gradient histogram, texture histogram, and features trained by neural networks, and calculate the characteristics of adjacent frames. The difference between the feature description factors. If the difference between the description factors is greater than the second threshold, key frame selection is started. The key frame can be selected in combination with the ambient brightness or content and a given frequency. The scene environment is also collected at a fixed frequency, but the collection frequency is adaptively adjusted according to the scene changes. If the scene environment brightness or content changes frequently, increase the key frame collection frequency. On the contrary, if the scene environment brightness or content remains basically unchanged, reduce the key frame collection frequency.

Regarding the selection of the key frame collection frequency according to the device-related state, the device-related state includes power, storage (eg, memory), device motion state, and the like. The key frame collection frequency can be set according to the power. For example, when the power is less than the first predetermined power, such as 50%, the key frame collection frequency is reduced, and when the power is greater than the second predetermined power, such as 80%, the key frame collection frequency is increased. , wherein the first predetermined power level is less than the second predetermined power level. The key frame frequency can be set according to the memory. For example, when the memory is less than the first predetermined value, such as 500M, the key frame collection frequency is reduced, and when the memory is greater than the second predetermined value, such as 700M, the key frame collection frequency is increased. The acquisition frequency can also be adjusted according to the motion state of the terminal device, and the motion state of the terminal device can be judged according to the internal sensor of the terminal device. If the motion amplitude is greater than a certain threshold, the key frame acquisition frequency is increased to ensure that enough high-quality key frame images are obtained.

Step 3, enhancing the resolution of the video information.

According to the collected key frame image information, the collected video information is divided into several video clips; the key frame image information on both sides of the video clip is used to enhance the corresponding video clip corresponding indicators that need to be enhanced.

Specifically, when the index that needs to be enhanced includes at least one of resolution, color, and brightness, the enhancement processing method includes an enhancement method based on multi-view reconstruction, and/or an enhancement method based on building an enhanced model based on machine learning.

Wherein, the enhancement method based on multi-view reconstruction specifically includes: establishing a matching relationship between the video pixels of the collected video information and the image pixels of the key frame image information, and replacing the matched video pixels by the image pixels.

Wherein, the method of building an enhanced model based on machine learning specifically includes: extracting video pixels at the location of the key frame image of the collected video information; building a mapping enhancement model between the video pixels and the image pixels of the key frame image information by means of machine learning ; Convert the video pixels through the mapping enhancement model at the location of the non-key frame image of the captured video information.

Step 4, storage of the enhanced video.

When the first type of multimedia information is image information, and the second type of multimedia information is video information, correspondingly processing the second type of multimedia information according to the first type of multimedia information, specifically including: storing the collected video information according to the collected image information processing, wherein the stored content includes at least one of the following situations:

Video information obtained by enhancing the collected video information according to the collected image information;

The collected video information and image information;

The collected video information and the enhancement model when the video information is enhanced;

The video information and the collected image information after enhancing the collected video information according to the collected image information.

In this embodiment, the terminal device generates four types of data: collected original video information, key frame image information, a resolution-enhanced mapping enhancement model, and enhanced video information.

The first is to directly store the enhanced video information after step 3, without saving the key frame image information, that is, to complete the enhancement processing before storage, and during storage, the video frame displays the picture in the original video frame, and the buffer is displayed above. The icon indicates that enhancement processing is in progress. After buffering is completed, the storage ends.

The second is to save the original video information and the learned enhanced model first, without saving the key frame image information. The enhancement process of step 3 is performed when the user opens the video. Since each video segment has an enhancement model, all enhancement models are stored separately, and a mapping table between enhancement models and video frame numbers is established.

The third method is to store the original video information and the captured key frame image information first. This method does not need to process the original video information before storing. After shooting, the storage can be completed immediately. The terminal device automatically arranges the processing time according to the busyness of the processor, learns the enhancement model to perform the enhancement processing in step 3 on the original video information, and deletes the key frame image information after the enhancement processing is completed. This storage method is also a high-definition video compression storage method proposed by the present invention. In order to save video storage space, low-resolution video information and high-resolution image information can be stored instead of directly storing high-resolution video information. The low-resolution video information and the high-definition image information can be obtained by simultaneously collecting the video information and the image information by the two cameras in the present invention, or the high-resolution video information can be directly extracted from the key frame to obtain the high-resolution image information, and then the original high-resolution image information can be obtained. The high-resolution video information is compressed to obtain low-resolution video information. Based on low-resolution video information and associated high-resolution image information, resolution enhancement methods can be used to obtain high-definition resolution video information.

The fourth method is to store the enhanced video information and save the key frame image information after the enhancement processing in step 3. The enhanced video information can be obtained by any one of the first three storage methods, and the key frame image information can be obtained by It can be saved in the video sequence at the same time as the video information, and can also be saved in the photo list to establish the association between the video information and the image information. Saving key frame image information can provide users with some high-definition image information for users to perform other operations.

The storage method is set by at least one of the following:

Determine the storage method according to the detected setting operation;

Determine the storage method according to the preset settings (such as system default settings);

The storage mode is adaptively determined according to the adaptive storage setting mode.

Specifically, the adaptive storage setting mode is determined by at least one of a device-related state and storage setting history data.

The setting operation is realized through the user's voice, keys, keys, gestures, and manipulation of external controllers. Device-related status includes device information such as storage space and power.

That is, for how to set the first storage mode, the present invention provides three setting methods, and the terminal device can select the first storage mode according to one of the following three methods. The first one is the default setting of the terminal device; the second one is that the terminal device accepts the user to configure or change the storage method by means of voice, keys or external controllers, and a combination of these methods; the third one is that the terminal device uses storage space, power Or set the storage method of historical data adaptively.

Regarding the system default settings, the terminal device sets one of the four storage modes as the default value, and uses this default storage mode to store videos and images until the terminal device receives an instruction to change the storage mode.

Regarding the voice settings under user interaction, for example, the terminal device presets the voice command "store enhanced video". If the command is received, it will perform voice recognition on the voice control command, and determine that the storage method is set to store the enhanced video information.

Regarding the key settings under user interaction, the keys can be hardware keys, such as the volume up and down keys to select four storage modes, and the home key to determine the currently selected storage mode as the final storage mode. The keys can also be virtual keys, such as buttons on the screen, menus, virtual keyboards on the interactive interface, etc. After the terminal device detects the event that the user clicks the virtual key, it confirms the selected storage method.

Regarding gesture settings under user interaction, the terminal device pre-sets a certain gesture to select the enhancement mode. Gestures include screen gestures, such as swiping the screen from left to right or right to left to change storage modes. Gestures also include space gestures, including shaking/tilting the terminal. Different directions when shaking/tilting represent different meanings, such as shaking up and down to change the storage method, and tilting left and right to change the storage method. The above gestures can be a single gesture or Any combination of any gestures, such as sliding the right hand horizontally to select the enhancement mode, and shaking up and down at the same time indicates that the currently selected storage mode is the final storage mode.

Regarding user settings via external controllers, external controllers include, but are not limited to, styluses, remote controls, smart watches, smart glasses, smart helmets, smart clothes, or remote devices, etc., which control via Wifi and/or infrared and/or Bluetooth and/or the network communicate with the interactive interface. For example, some buttons on the remote control represent setting different storage methods. It is detected that the user has clicked the button and sent to the interactive control system to set the storage method.

Regarding the adaptive setting according to the storage space, different storage methods can be selected according to the storage space. If the remaining storage space is less than a certain threshold, for example, lower than 50% of the storage space of the terminal device, the third compression storage method is set; Conversely, if the remaining storage space is higher than a certain threshold, for example, higher than 50% of the storage space of the terminal device, the storage method is not affected by the storage space.

Regarding the adaptive setting according to the power, the storage method can be controlled according to the power. When the power is less than a certain threshold, for example, lower than 50%, the storage method with low power consumption is selected, that is, the third storage method, that is, the original storage method is directly stored. The second storage method of video information and key frame image information is the original video and the learning model, and the video is not enhanced; when the power is less than the second predetermined power, for example, less than 15%, the first one with the least power consumption is selected. Three storage methods, namely original video information and key frame image information; if the battery level is higher than a certain threshold, for example, higher than 50%, the storage method is not affected by the battery level.

Regarding the adaptive setting according to the storage setting history data, for example, according to the storage method previously set by the user, the user's preference is analyzed, and the user's preferred storage method is set.

Step 5, video playback.

In response to the received playback trigger operation, the video information is played based on the playback mode that matches the stored content; wherein, the playback mode includes at least one of the following:

When the enhanced video information is stored, the enhanced video information is directly played;

When the collected video information and image information are stored, the collected video information is enhanced and played according to the collected image information;

When storing the collected video information and the enhanced model, the collected video information is enhanced and played through the enhanced model;

When the enhanced video information and the collected image information are stored, the enhanced video information and the collected image information are associated and played.

For example, the terminal device plays the stored video according to the detected play triggering operation.

Solution 1: The storage mode corresponding to the playback mode is to store the enhanced video information. The terminal device detects the user's operation of opening the video, and directly opens the enhanced video information. When the operation of the user clicking to play is detected, for example, the operation of the user clicking the play button is detected, the video can be played immediately, so that the user can directly view the enhanced effect.

Scheme 2: The storage mode corresponding to this playback mode is to store the original video information and the enhanced model. The terminal device detects the user's opening operation and opens the combination of the original video information and the enhanced model. After detecting the operation of the user clicking to play, for example, detecting the operation of the user clicking the play button, the terminal device starts to enhance the original video information based on the enhancement model. The operation is completed within the allowed buffer time, and then the enhanced video information is played.

Solution 3: The storage mode corresponding to this playback mode is to store the original video information and key frame image information. The terminal device detects the user's opening operation, and opens the combination of the original video information and the captured key frame image information. The operation of the user clicking to play is detected, for example, the operation of the user clicking the play button is detected. If the terminal device has completed the enhancement processing, the video can be directly played to view the enhanced video information. If the terminal device only does part of the work in the background and does not complete the enhancement step, it will need time buffering to perform video enhancement after receiving the user's click play operation. After the enhancement is completed, the enhanced video information will be played, and the user can see the enhanced video information. video information.

Scheme 4: The storage mode corresponding to this playback mode is to store the enhanced video information and key frame image information. The terminal device detects the user's opening operation, and opens the combination of the enhanced video information and key frame image information. The enhanced video information playback mode not only includes the first three playback modes, but also can get more information based on the key frame image information. playback method. After establishing the relationship between the video information and the image sequence, you can link to the video information through the image sequence, or link to the image sequence through the video information, and complete this kind of image and video by long pressing the key frame image sequence or the video play button. Jump, you can also complete the jump through voice, gesture and other settings.

Solution 5: The terminal device detects the user's opening operation, and opens the combination of the enhanced video and key frames. The position of each key frame in the video is recorded during storage, and the user can click an image in the key frame sequence to link to the corresponding video position and start playing the video from that position. It is possible but not limited to long-press the video, a sequence of key frames will appear, and click an image in the sequence to start the video. When viewing a video, the user is presented with a combination of images, the user can click on the image sequence to view it, and then click on the image to play the video.

Scheme 6: The terminal device detects the user's opening operation, for example, detects the user's click-to-play operation, displays the buffer mark, and performs resolution conversion. The conversion may include but not limited to the following methods: First, convert the entire video clip into The high-resolution video information is played from the beginning; the second is to start playing a part of the buffer, and the video conversion is performed while playing. Due to the processing capacity of the mobile phone, there may be interruptions during playback to complete the buffering; the third is to click on the key frame image information. For a certain one, only convert the video clip after the starting point from the video position corresponding to the image information. You can choose to play low-resolution video information for viewing. If you are interested in the video, you can play high-resolution video through buttons or other operations. After selecting high-resolution playback, you can play it according to the first three playback methods. You can also share the compressed video to reduce the energy consumption of the mobile phone, and you can choose a variety of playback options when the other party is playing.

Embodiment 2: The enhancement processing mode is the color enhancement mode in the image enhancement video mode

In terms of video brightness and color enhancement, at present, videos shot by terminal devices in low-light situations are generally dark in brightness and poor in quality. In the prior art, video is mostly processed by some prior models, for example, a non-linear mapping table is set, as shown in FIG. 2 , the histogram equalization mapping table of each frame of image is calculated according to the brightness information, and according to the pre- The mapping table and the calculated mapping table are weighted to obtain a conversion curve, and the brightness of the pixels in the video frame is adjusted according to the curve to enhance the video frame. Another way is to perform gamma correction on the video to obtain the transmission parameters of the video, and use the improved image degradation restoration model to process the original video and the transmission parameters obtained by the preprocessing to obtain the final enhancement result. The above methods are enhanced based on the video's own information, and the enhancement strength should be adjusted by manually setting parameters. In terms of video brightness and color adjustment, using some preset mapping curves to adjust each frame of image in the video can achieve the change of brightness and color, but the basis of the change is also limited by the preset parameters, and whether the change trend is suitable for Various scenarios are also up for consideration. Under different lighting conditions, the adjusted parameters need to be adjusted adaptively, and the preset mapping curve may have unrealistic changes, such as the video shot at night being adjusted to be too bright or the color distortion of some image areas.

The basic process of this embodiment is as follows: firstly, enable the color enhancement mode in the image enhancement video mode, start two cameras to collect image information and video information respectively; secondly, set the main and auxiliary cameras, and set the acquisition parameters and enhancement strategy parameters of the cameras, And set the selection of key frame image information, at the same time enhance the color of the video according to the enhancement strategy parameters and the key frame image information, and finally compress, transmit and play the collected results.

Step 1: Turn on Color Enhancement Mode in Image Enhancement Video Mode.

In this embodiment, step 1 adopts a similar opening method to that in the first embodiment, the difference lies in the description content of the command to turn on the function, for example, the command in the voice turn on is "start enhanced video color", and the command to turn on the key is long press The Home key, the button to enhance the video color in the virtual key, the hand-held open command is to shake the terminal, etc., which will not be repeated here.

Step 2: Use one camera of the binocular camera to capture image information and the other camera to capture video information.

This step mainly includes the setting of acquisition parameters and enhancement strategy parameters, the settings of main and auxiliary cameras, the parameter settings of acquired images and videos, and the selection of key frame image information.

Step 2.1, set the primary and secondary cameras.

In this embodiment, the method for setting the main and auxiliary cameras can be similar to the method for setting the main and auxiliary cameras in step 2.1 of the first embodiment, and details are not repeated here.

Step 2.2, set the acquisition parameters and enhancement strategy parameters of the camera.

In this embodiment, the manner of setting the acquisition parameters of the camera and the enhancement strategy parameters may be similar to the manner of setting the acquisition parameters and the enhancement strategy parameters in step 2.2 of the first embodiment, which will not be repeated here.

Step 2.3, adaptively set the key frame collection frequency.

In this embodiment, the key frame selection method can be similar to the key frame selection method in step 2.3 of the first embodiment, which is not repeated here.

Step 3, performing color enhancement processing on the video information.

The video is divided into segments based on keyframes, and the video segment is enhanced with keyframe images on both sides of the segment. Color enhancement methods include but are not limited to the following two methods: a method based on multi-view reconstruction and a method based on machine learning.

Regarding color enhancement based on multi-view reconstruction, a multi-view reconstruction method is used to establish the matching relationship between video pixels and image pixels, and image pixels are used to replace the generated video pixels.

With regard to machine learning based color enhancement, a mapping model of video pixels and image pixels is built in a machine learning manner at key frames, and video pixels are transformed with this mapping model at non-key frames.

Step 4, the storage of the video.

In this embodiment, the terminal device generates four types of data: collected original video information, key frame image information, color enhancement model, and enhanced video information. For different data types, storage methods include but are not limited to the following four.

The four storage modes in this embodiment are the same as the four storage modes introduced in step 4 in the first embodiment, and are not repeated here.

The method for setting the storage mode adopts the setting method similar to step 4 in the first embodiment, the difference is that the enhanced mode does not change the video resolution, among the four storage modes, the first storage enhanced video information occupies the smallest space, When the storage space is less than a certain threshold, for example, less than 50% of the storage space of the terminal device, the first storage mode is set. The other setting methods are the same and will not be repeated here.

Step 5, video playback.

After the data collected by the binocular camera is compressed, stored and transmitted, the high-quality video information is decompressed and played back in the playback stage. For different storage modes, the playback modes include, but are not limited to, one of the first five described in step 5 in the first embodiment. The five playback modes are the same as the first five in step 5 in the first embodiment. Repeat.

Embodiment 3: The enhancement processing mode is the brightness enhancement mode in the image enhancement video mode

The basic process of this embodiment is: firstly enable the brightness enhancement mode in the image enhancement video mode, start two cameras to collect image information and video information respectively, secondly set the camera's acquisition parameters and enhancement strategy parameters, and set the key frame image information At the same time, the brightness of the video is enhanced according to the enhancement strategy parameters and key frame image information, and finally the collected results are compressed, transmitted and played.

Step 1: Turn on Brightness Boost Mode in Image Enhancement Video Mode.

In this embodiment, the opening method of step 1 adopts the similar opening method in the first embodiment, and the difference lies in the description content of the instruction to enable the function. Long press the End key, the button for enhancing the video brightness in the virtual key, the hand-held open command is the shaking terminal, etc., which will not be repeated here.

Step 2: Use one camera of the binocular camera to capture image information, and the other camera to capture video information;

This step mainly includes the setting of acquisition parameters and enhancement strategy parameters, the settings of main and auxiliary cameras, the parameter settings of acquired images and videos, and the selection of key frame image information.

Step 2.1, set the primary and secondary cameras.

In this embodiment, the method for setting the main and auxiliary cameras can be similar to the method for setting the main and auxiliary cameras in step 2.1 of the first embodiment, which will not be repeated here.

Step 2.2, set camera parameters and enhancement strategy parameters.

In this embodiment, the manner of setting the acquisition parameters of the camera and the enhancement strategy parameters may be similar to the manner of setting the acquisition parameters and the enhancement strategy parameters in step 2.2 of the first embodiment, which will not be repeated here.

Step 2.3, select keyframes.

In this embodiment, the key frame selection method can be similar to the key frame selection method in step 2.3 of the first embodiment, which is not repeated here.

Step 3, enhance the brightness of the video.

Luminance enhancement is performed in the L channel of the Lab color model, or the V channel of the HSV color model. The image or video is first converted to color space, and then the L or V channel components are extracted for independent enhancement. The video is divided into video clips with the key frame as the boundary, and the video clip is enhanced with the key frame images on both sides of the video clip. The enhancement methods include but are not limited to the following two methods: a method based on multi-view reconstruction and a method based on machine learning.

Regarding the brightness enhancement based on multi-view reconstruction, the multi-view reconstruction method is used to establish the matching relationship between video pixels and image pixels, and the image pixel brightness is used to replace the generated video pixel brightness.

Regarding the brightness enhancement based on machine learning, a mapping model of video pixel brightness and image pixel brightness is established by machine learning at key frames, and the video brightness is converted at non-key frames with this mapping model.

Step 4, the storage of the video.

In this embodiment, the terminal device generates four types of data: collected original video information, key frame image information, brightness enhancement model and enhanced video information. For different data types, storage methods include but are not limited to the following four.

The four storage modes in this embodiment are the same as the four storage modes introduced in step 4 in the second embodiment, and will not be repeated here.

Step 5, video playback.

In this embodiment, the way of decompressing and playing the video can be the same as the way of playing the video in step 5 of the second embodiment, which will not be repeated here.

Embodiment 4. The enhancement processing mode is the denoising enhancement mode in the image enhancement video mode

In terms of video denoising and deblurring, the video noise is more than the image noise due to the short exposure time of each frame of the video. Most of the existing technologies use an image denoising method to perform noise reduction processing on video frames, and finally achieve the purpose of video denoising. Assuming that the noise type is Gaussian noise, some filtering methods are used to remove the noise, and there are also dictionary learning methods to remove the noise. The following factors may cause video blur: 1) When shooting video with a mobile phone, the hand shake will cause the picture to be blurred; 2) Noise reduction processing makes some areas blurred; 3) Defocus blur caused by inaccurate focusing, etc. . Most of the existing technologies estimate the blur kernel of the blurred image, and use the obtained blur kernel to deconvolve the blurred image to obtain a clear image; as shown in FIG. 3 .

In the same shooting environment, compared with image information, video frame video image noise is more obvious. In this embodiment, low-quality video clips are denoised by high-quality key frame image information with similar content to improve video quality. The specific implementation steps are as follows:

Step 1: Turn on the denoising enhancement mode in the image enhancement video mode.

In this embodiment, step 1 adopts a similar opening method as in the first embodiment, the difference lies in the difference in some instruction descriptions and threshold settings, and the specific differences are described as follows.

Different command descriptions are used when the user turns on the video denoising mode. For example, the command in the voice activation is "start video denoising", the key activation command is long press the Home button, and the video denoising button in the virtual key is hand-held to open. The command is to shake the terminal, etc., which will not be repeated here.

Different threshold settings are used in the terminal device to adaptively turn on the mode according to the device-related state and mode turn-on history data, for example, the power threshold, the number of times threshold, and so on. The other description methods are the same, and are not repeated here.

Three methods are given in the adaptive mode matching mode of the terminal device to adaptively enable the video denoising enhancement mode. One method is that the terminal device uses the existing method to detect the captured environment. If the shooting environment is detected as a low-light environment , such as night scene shooting, turn on the denoising enhancement mode; the second method is that the terminal device detects the camera shooting parameters, and if the sensitivity is higher than a certain threshold, the denoising enhancement mode is turned on; the third method can be the first two The combination of methods, the denoising enhancement mode is turned on when the two conditions are satisfied at the same time, that is, the light intensity in the shooting environment is lower than a certain threshold, and the sensitivity is higher than a certain threshold, the denoising enhancement mode is turned on.

The terminal device automatically turns on the video denoising mode according to the real-time captured content to calculate the signal-to-noise ratio of the captured frame. If the signal-to-noise ratio is lower than a certain threshold, the video denoising mode is turned on.

Step 2: Use one camera of the binocular camera to capture images and the other camera to capture video.

This step mainly includes the setting of acquisition parameters and enhancement strategy parameters, the settings of the main and auxiliary cameras, the parameter settings of the acquired images and videos, and the selection of key frame image information.

Step 2.1, set the primary and secondary cameras.

In this embodiment, the method for setting the main and auxiliary cameras may be similar to the method for setting the main and auxiliary cameras in step 2.1 of Embodiment 1, which will not be repeated here.

Step 2.2, set the acquisition parameters and enhancement strategy parameters of the camera.

In this embodiment, the setting of the camera's acquisition parameters and the enhancement strategy parameters may be similar to the method of setting the camera's acquisition parameters and the enhancement strategy parameters in step 2.2 of the first embodiment, which will not be repeated here.

In addition to the aforementioned parameter setting methods, this embodiment proposes a new parameter setting method for the denoising mode. In this embodiment, one of the binocular cameras is used to collect image information and the other to collect video information, and the video information is denoised. The resolution, exposure time, and sensitivity of image acquisition are mainly set here. For the consideration of energy saving and algorithm design, the resolution of the image information collected by the auxiliary camera should be consistent with the resolution setting of the video information. If the minimum resolution of the image information is higher than that of the current video information, the minimum resolution of the image information is used for collection. If the power allows, the exposure time can be adjusted according to the movement state of the terminal device. If the internal sensor of the terminal device detects that the terminal device is in a stable shooting state, increase the exposure time, for example, take the maximum value in the exposure time range, increase the brightness of image information and video information, and reduce noise interference; if the sensor detects that the terminal device is shaking Or other motion trends, appropriately reduce the exposure time to avoid blurring of image information and affect the denoising effect of video information.

Step 2.3, select key frame image information.

In this embodiment, the key frame selection method may adopt the key frame selection method similar to that in step 2.3 of the first embodiment, which will not be repeated here.

For the denoising mode, this implementation provides a new way of selecting key frame image information. If the light intensity is lower, the collection frequency of key frames will be increased. On the contrary, the higher the light intensity will be, the collection frequency of key frames will be reduced. The terminal equipment's own sensor is used to detect the motion state of the terminal equipment. If the motion amplitude is greater than a certain threshold, the collection frequency will be increased. In order to ensure that enough high-quality key frame images are obtained. If the terminal device moves when the key frame is collected, another key frame image information is collected at the end of the movement as an alternative to the previous frame key frame image information. Select the key frame image information to denoise the video clip to ensure the denoising effect.

Step 3: Use a sequence of keyframes to remove noise from the video.

When the index to be enhanced includes noise, the enhancement processing method includes a method based on dictionary reconstruction, and/or a method based on deep learning.

When the fuzzy frame to be processed is detected in the original video information, it is determined that the fuzzy frame to be processed is deblurred and enhanced;

Wherein, the fuzzy frame to be processed is detected by at least one of the following information:

The motion state of the multimedia acquisition device; the frames that failed to focus in the original video information; the classification result of classifying the original video information by the classifier.

Specifically, each high-quality key frame corresponds to a video subsequence. Suppose video key frame 1 contains 60 frames in its corresponding video subsequence, and the video is denoised by using adjacent one or more key frame images. For example, use key frame 1 to denoise these 60 frames of images, and the denoising methods include but are not limited to one of the following methods: one is a method based on dictionary reconstruction, such as NLM (non local means) algorithm; Deep learning methods, such as convolutional neural network CNN (convolutional network).

Compared with the traditional denoising method, this embodiment uses a reference image, that is, a key frame image, to denoise the noisy video. For this characteristic, the existing method is improved. The above two methods are described in detail below.

1) Method based on dictionary reconstruction, i.e. improved NLM

If the resolution of the input key frame image is inconsistent with the video frame image, first unify the scale of the key frame image and the video, zoom the video or the key frame image, or zoom the video and the key frame image at the same time. Scale down so that the keyframe image and the video have the same dimensions. Using the method of stereo matching to calculate the disparity between the key frame image and each frame in the video clip, and align the pixels in the video frame with the pixels of the key frame image, that is, the positional relationship of the consistent pixels between the images. If the input key frame image and the video frame image have the same resolution, the stereo matching method is directly used to calculate the disparity for image alignment. After obtaining the positional relationship between the pixels of the key frame image and the video frame image, similar blocks are found between the images. For each video frame, take a pixel point in the frame image as the center to take a pixel block p of size a*a, where a can be preset, or can be adaptively adjusted according to the image size or other factors. Based on the obtained positional relationship between the video frame and the consistent pixel of the key frame image, find the position of the pixel in the video frame in the key frame image, take the position as the center, and take the neighborhood of size b*b, where b can be pre- It can also be adaptively adjusted according to other factors such as image size. Taking each pixel in the neighborhood block as the center, take a pixel block q of size a*a, a total of b*b pixel blocks can be obtained, and the distance between the pixel block p and the b*b pixel blocks can be calculated. The formula can be used but is not limited to the square sum of Euclidean distances, that is, the square sum of the pixel value differences at the same position between blocks is, if a is 8, the pixel block distance is the 64 pixels in the low-quality pixel block and the key frame pixel block. The sum of the squares of the 64 pixel value differences. Two pixel blocks are considered similar if the distance is less than a certain threshold. All similar blocks found that are smaller than this threshold are marked as the set Ω. Then the weight of similar blocks needs to be calculated, and the weight w of each block is calculated according to the distance of these similar blocks. The larger the distance, the smaller the weight, and vice versa. The weights can be calculated using, but not limited to, a Gaussian function. Finally, use these similar pixel blocks to reconstruct high-noise pixel blocks, that is, weighted average of similar blocks. Here, a certain weight can also be assigned to the original pixel block, which occupies a certain proportion in the reconstruction to ensure the reconstructed image. The consistency of the pixel block with the original pixel block. Each pixel in the video frame is re-estimated by the block-based reconstruction method to achieve denoising of each video frame, and each frame in the video clip is denoised to denoise the entire video clip. .

Due to the parallax between the key frame image and the video frame, the pixels of some video frames cannot find corresponding pixels in the key frame, and there is no way to find similar pixel blocks in high-quality images for reconstruction. For these pixels, the original NLM method can be used to find similar pixel blocks in the video frame where they are located and reconstruct them for denoising. At the same time, in order to save processing time, the relationship between the video frames can be considered. If the content of the video clips does not change much between the frames, the pixels in multiple video frames can be reconstructed at the same time, that is, the same pixels in different frames can be used. The similar pixel blocks are reconstructed to achieve the effect of denoising multiple pixels at the same time.

2) Methods based on deep learning

If the input key frame image and the video frame image have different resolutions, take the same steps as in the first method to unify the key frame image and video scale. Using the method of stereo matching, the disparity between the key frame image and each frame in the video clip is calculated, and the pixels in the video frame and the pixels in the key frame image are aligned. According to the alignment results, the key frame image and the video frame are cropped to obtain a pair of image pairs with the same size and the same content, one is a noise-free image cropped from the key frame image, and the other is cropped from the video frame image. In the obtained noise image, the content of the pixels in the same position of the two images is the same, but one is from a high-quality key frame image, called noise-free pixels, and the other is from a low-quality video frame image, called noise pixels point. Based on multiple pairs of images, many pairs of pixel blocks can be obtained by taking corresponding pixel blocks. Based on these pixel blocks, using existing deep learning-based methods, a convolutional network with denoising function is trained to denoise the pixel blocks in the video frame. The model can collect a large number of samples offline to train to obtain an initial model pre-installed in the terminal device, and then use the obtained image pairs to modify the model to denoise the video frame; it can also train a model online to denoise the video frame. frame for denoising.

Step 4: Storage of the video.

In this embodiment, the terminal device generates four types of data: collected original video information, key frame image information, denoising enhancement model and enhanced video information. For different data types, storage methods include but are not limited to the following four.

The four storage modes in this embodiment are the same as the four storage modes introduced in step 4 in the second embodiment, and will not be repeated here.

The method for setting the storage mode adopts the same setting method as step 4 in the second embodiment, and will not be repeated here.

Step 5, video playback.

After the data captured by the binocular camera is compressed, stored and transmitted, the high-quality video is decompressed and browsed during the playback stage. For different storage modes, its playback mode includes but is not limited to one of the five kinds of steps 5 introduced in the second embodiment, and the five kinds of playback modes are the same as described in the second step 5 of the implementation, and will not be repeated here.

Embodiment 5. The enhancement processing mode is the deblurring enhancement mode in the image enhancement video mode

When shooting video, a variety of factors can lead to blurred images, mainly including: 1) hand shake during shooting with handheld terminal devices, 2) blurred images caused by moving objects, and 3) blurred target areas caused by failure to focus. In this embodiment, one camera of the binocular cameras is used to collect images, the other camera is used to collect videos, and the images are used to deblur the videos to obtain a high-quality video. Specific steps are as follows:

Step 1: The terminal device determines that the binocular camera needs to be used to achieve image-assisted video deblurring, and enables the deblurring enhancement mode in the image enhancement video mode.

In the present embodiment, step 1 adopts the opening mode similar to that in the first embodiment, the difference lies in the difference between some instruction descriptions and threshold settings, and the specific differences are described as follows:

Different command descriptions are used when the user turns on the enhanced deblurring mode. For example, the command in voice activation is "start video deblurring", the command in button activation is long press the Home button, and the button for video deblurring in virtual keys is hand-held to activate. The command is to shake the terminal, etc., which will not be repeated here.

Different threshold settings are used in the terminal device to adaptively turn on the mode according to the device-related state and mode turn-on history data, for example, the power threshold, the number of times threshold, and so on. The other description methods are the same, and are not repeated here.

Three methods are given in the terminal device adaptively enabling the video deblurring enhancement mode according to the acquisition environment (ie the shooting environment) and parameters. One method is that the terminal device uses the existing method to detect the movement trend of the camera terminal. The motion state, such as the shaking of the shooting terminal caused by the shaking of the hand of the handheld terminal, enables the video deblurring enhancement mode; the second method is that the terminal device detects the shooting parameters, if the video capture exposure time is higher than a certain threshold, for example, Above 300ms, the video deblurring enhancement mode is turned on; the third method can be a combination of the first two methods, and the video deblurring enhancement mode is turned on if the two conditions are satisfied at the same time, that is, the shooting terminal is in motion, and the exposure time is higher than a certain A threshold value enables video deblurring enhancement mode.

The terminal device automatically turns on the video deblurring enhancement mode according to the relevant content collected in real time, and calculates the blurriness index of the shooting frame. If it is lower than a certain threshold, the video deblurring enhancement mode is turned on.

Step 2: Use one camera of the binocular camera to capture images and the other camera to capture video.

This step mainly includes the setting of acquisition parameters and enhancement strategy parameters, the settings of main and auxiliary cameras, the parameter settings of acquired images and videos, and the selection of key frame image information.

Step 2.1, set the primary and secondary cameras.

In this embodiment, the method for setting the main and auxiliary cameras can be similar to the method for setting the main and auxiliary cameras in step 2.1 of the first embodiment, which will not be repeated here.

Step 2.2, set the acquisition parameters and enhancement strategy parameters of the camera.

In this embodiment, the setting of the camera's acquisition parameters and the enhancement strategy parameters may be similar to the method of setting the camera's acquisition parameters and the enhancement strategy parameters in step 2.2 of the first embodiment, which will not be repeated here.

In addition to the aforementioned parameter setting methods, this embodiment proposes a new setting method for the deblurring mode. In this embodiment, one camera in the binocular camera is used to collect images, and one camera is used to collect video, and the video is deblurred. Here, the resolution, exposure time, and sensitivity of image capture are mainly set. For the consideration of energy saving and algorithm design, the resolution of the image captured by the auxiliary camera should be consistent with the resolution setting of the video. If the minimum resolution is higher than that of the current video, the minimum resolution of the image is used for acquisition. If the brightness of the photographing environment is normal or higher than a certain threshold, shorten the exposure time of the video and image, increase the sensitivity appropriately, and reduce the probability of picture blurring; Reduce exposure time to avoid blurry images and affect video deblurring.

Step 2.3, select key frame image information.

In this embodiment, the key frame image information selection method can adopt the similar key frame image information selection method in step 2.3 of the first embodiment, and will not be repeated here.

For the deblurring mode, this implementation provides a new key frame selection method. If the exposure time becomes longer and the probability of video blurring becomes higher, the key frame collection frequency will increase; otherwise, if the exposure time becomes shorter, the key frame collection frequency will be reduced. When the terminal device moves, another key frame is collected at the end of the motion as an alternative to the previous key frame. When the previous key frame is blurred, the alternative key frame can be used to deblur the video clip to ensure that the deblurring effect.

Step 3: Deblur the video using a sequence of keyframes.

When the index to be enhanced includes blur, the enhancement processing method includes a method based on blur kernel estimation, and/or a method based on deep learning.

When a fuzzy frame to be processed is detected in the collected video information, it is determined that the fuzzy index corresponding to the fuzzy frame to be processed is enhanced;

Wherein, the fuzzy frame to be processed is detected by at least one of the following information:

The motion state of the device when collecting video frames; the focusing information when collecting video frames; the classification result of classifying the collected video information through the classifier.

Specifically, it is necessary to first judge which video frames appear blurry, which can be adopted but not limited to one of the following three schemes:

First, monitor the motion state of the terminal device itself according to the built-in sensor of the terminal device. If the motion amplitude is greater than a certain threshold, it is determined that the video frame captured in this time period is the blurred frame to be processed; Second, the frame that fails to focus is detected. If the focus area is detected, the focus moves in the middle of the shooting, that is, the focus is not on the target area, then the target area is considered to be the blurred area to be processed; third, through the method of machine learning, a large number of blurred images and clear images are used to train to obtain a classification The obtained classifier is used to classify the video frame, and the video frame classified into the blurred image is the blurred frame to be processed.

If blurred frames are detected, these blurred frames are deblurred. First, find clear key frame images that are similar to the content of these video frames, and the deblurring methods include but are not limited to the following two. One is a method based on fuzzy kernel estimation, and the other is a method based on deep learning.

Compared with the traditional deblurring method, the present embodiment uses a reference image, that is, key frame image information, to deblur the blurred video information. For this characteristic, the existing method is improved. The above two methods are described in detail below.

1) Based on fuzzy kernel estimation

If the resolution of the input key frame image is inconsistent with the video frame image, the same steps are taken as in the denoising method to unify the scale of the key frame image and the video. Using the method of stereo matching, the disparity between the key frame image and each frame in the video clip is calculated, and the pixels in the video frame and the pixels in the key frame image are aligned. According to the alignment results, crop the key frame image and the blurred video frame, and get a pair of images with the same size and the same content, one clean image cropped from the key frame image, and the other from the video frame image. Cropped blurred image. One large-scale image pair can be cropped, and multiple pairs of small-scale image pairs can be cropped. The least squares optimization method or other optimization methods are used to estimate the blur kernel for each group of images. If there are multiple pairs of images, multiple blur kernels can be estimated. If a blur kernel is obtained, use the blur kernel to deblur the frame image; if multiple blur kernels are obtained, an average blur kernel can be obtained by weighted averaging of the multiple blur kernels, and the frame image is deblurred by the blur kernel.

Considering the continuity of video frames, if several consecutive frames of images need to be deblurred, this embodiment provides two processing methods. One is to perform the above deblurring operation on each image separately; the other is to select several frames of discontinuous images to estimate the blur kernel, that is, there are several frames of blurred images between every two frames of images, if the two most recent frames are selected. If the estimated blur kernels are similar, it can be considered that the images between the two frames are also caused by similar blur kernels, and the same blur kernel can be used for deblurring. If the similarity of the blur kernels of the two frames is not high, it is necessary to select one or several frames between the frames to recalculate the blur kernel for deblurring. In order to reduce the calculation time and the number of times of calculating the kernel as much as possible, the binary method can be used, but it is not limited to this method. Select key frames to calculate the blur kernel. If the blur kernels at both ends of the binary interval are similar, you can simultaneously divide the The video frames are deblurred with a unified blur kernel. If the blur kernels at both ends of the two-part interval are not similar, the interval will continue to be divided into two, and so on until all video frames are deblurred.

2) Methods based on deep learning

The preprocessing method takes the same steps as in the first method to unify the scale of the keyframe image with the video. The pixels in the video frame and the pixels of the key frame image are aligned by the method of stereo matching. According to the alignment results, the key frame image and the video frame are cropped to obtain a pair of image pairs with the same size and the same content, one clean image cropped from the key frame image, and the other cropped from the video frame image. noise image. Based on multiple pairs of images, or by extracting corresponding pixel blocks from multiple pairs of images, many pairs of pixel blocks can be obtained. Based on these pixel block pairs or image pairs, using existing deep learning-based methods, a convolutional network with deblurring function is trained to deblur the video frame. The model can collect a large number of samples offline to train to obtain an initial model pre-installed in the terminal device, and then use the obtained image pair to modify the model to deblur the video frame; it can also train a model online to deblur the video frame. Frames are deblurred.

Step 4: Storage of the video.

In this embodiment, the terminal device generates four types of data: collected original video information, key frame image information, deblurred enhancement model, and enhanced video information. For different data types, storage methods include but are not limited to the following four.

The four storage modes in this embodiment are the same as the four storage modes introduced in step 4 in the second embodiment, and will not be repeated here.

The method for setting the storage mode adopts the same setting method as step 4 in the second embodiment, and will not be repeated here.

Step 5, video playback.

After the data captured by the binocular camera is compressed, stored and transmitted, the high-quality video is decompressed and browsed during the playback stage. For different storage modes, its playback mode includes but is not limited to one of the five kinds of steps 5 introduced in the second embodiment, and the five kinds of playback modes are the same as described in the second step 5 of the implementation, and will not be repeated here.

Embodiment 6: Multi-mode combined image enhancement video mode (joint enhancement mode)

The method further includes: determining a joint enhancement mode of at least two enhancement modes among the resolution enhancement mode, the color enhancement mode, the luminance enhancement mode, the denoising enhancement mode and the deblurring enhancement mode, and determining the joint enhancement mode corresponding to the at least two enhancement modes The mode enhancement order of .

The manner of determining the joint enhancement mode and the corresponding mode enhancement order by at least one of the following includes at least one of the following:

System default setting; adaptive mode setting method; mode setting trigger operation.

Wherein, the adaptive mode setting mode is determined by one or more pieces of information in the relevant state of the device, the mode setting history record information, the relevant content collected in real time by the multimedia collection device, and the influence relationship between each enhancement mode;

The relevant content collected in real time by the multimedia collection device includes at least one of scene brightness and semantic content.

The mode setting trigger operation is realized through voice, keystrokes, gestures, external controllers, etc. under the interaction with the user.

As mentioned earlier, image enhancement video modes include resolution, color, brightness, denoising, and deblur enhancement. In this embodiment, images are used to jointly enhance the five video modes. The basic process is as follows: firstly, enable multi-mode joint video enhancement; secondly, set the main and auxiliary cameras to collect image information and video information respectively, and set the camera's acquisition parameters, Enhance the strategy parameters, select key frame image information and select the mode to be enhanced, at the same time enhance the video according to the enhancement strategy parameters, key frame image information and enhancement mode, and finally compress, transmit and play the collected results.

Step 1: Turn on multi-mode combined video enhancement

In this embodiment, the opening method is similar to that in step 1 of the first embodiment, and the difference lies in the description content of the instruction to enable the function. In order to press the F1 key, the virtual key is a multi-mode joint enhanced button, and the hand-held open command is to draw a circle on the screen, etc., which will not be repeated here.

Step 2: Use one of the binocular cameras to capture images and the other to capture video.

This step includes setting main and auxiliary cameras, setting acquisition parameters and enhancement strategy parameters of the cameras, selecting key frame image information, and selecting a mode to be enhanced and a mode enhancement sequence.

Step 2.1, set the primary and secondary cameras.

In this embodiment, the method for setting the main and auxiliary cameras may adopt the similar setting methods for the main and auxiliary cameras in step 2.1 of the first embodiment, which will not be repeated here.

Step 2.2, set the acquisition parameters and enhancement strategy parameters of the camera.

In this embodiment, the setting methods of the acquisition parameters of the camera and the enhancement strategy parameters may adopt the similar setting methods of the acquisition parameters and the enhancement strategy parameters in step 2.2 of the first embodiment, which will not be repeated here.

Step 2.3, select key frame image information.

In this embodiment, the key frame image information selection method may adopt a similar key frame selection method in step 2.3 of the first embodiment, which will not be repeated here.

Step 2.4, select the mode of joint enhancement and the mode enhancement order.

Regarding the default system settings, the default settings of the terminal device enhance some modes and the enhancement order of these modes, for example, only color enhancement and brightness enhancement are turned on at startup, and color enhancement is performed first, and then brightness enhancement is performed.

Regarding the voice settings under user interaction, for example, the user presets the voice command "multi-mode joint enhancement, enable video color enhancement", and the terminal device receives the command, performs voice recognition on the voice control command, and determines to enable the color enhancement function. The voice command "multi-mode joint enhancement, turn off video color enhancement" is issued again, and the terminal device determines to turn off video color enhancement upon receiving the command, wherein the mode enhancement sequence is consistent with the issuing sequence of the voice commands.

Regarding the key setting under user interaction, the key can be a hardware key, for example, F1 means enable/disable color enhancement, F2 means enable/disable brightness enhancement, F3 means enable/disable resolution enhancement, F4 means enable/disable deblurring, F5 means Turn on/off denoising. The keys can also be virtual keys, such as buttons on the screen, menus, virtual keyboards on the interactive interface, etc. After the system detects the event of the user clicking the virtual key, it confirms to enable/disable an enhanced mode. The user can also press to express different meanings in combination with the pressure, speed, time, frequency and other characteristic information of pressing . For example, pressing lightly means turning off a certain enhanced mode, and pressing again means turning on a certain enhanced mode. Among them, the mode enhancement sequence is consistent with the click sequence of user buttons.

Regarding the gesture settings under user interaction, the system pre-sets a certain hold to enable/disable an enhanced mode. Gestures include screen gestures, such as swiping the screen from left to right to turn on/off color enhancement, and swiping from right to left to turn on/off brightness enhancement. Gestures also include space gestures, including shaking/flipping/tilting the terminal, different directions, angles, speeds, and strengths when shaking/flipping/tilting can represent different meanings, such as shaking up and down, shaking left and right, drawing a circle in an empty space, etc. The gesture can be a single gesture, such as swiping with the left hand horizontally to turn on/off color enhancement, or it can be any combination of any gestures, such as swiping with the right hand horizontally and drawing a circle in an empty space to turn on/off the resolution enhancement. Among them, the video mode enhancement sequence is consistent with the control sequence of user gestures.

Regarding settings via external controllers, including but not limited to styluses, remote controls, smart watches, smart glasses, smart head-mounted devices, smart clothes, or remote devices, etc., these controllers via Wifi and/or infrared And/or Bluetooth and/or the network communicates with the terminal device, for example, some keys on the remote control represent different enhanced modes, and the terminal device detects that the user has clicked the key and turns on/off its enhanced mode. Among them, the video mode enhancement sequence is consistent with the command issuing sequence of the external controller.

Regarding the joint enhancement of the adaptive mode according to the real-time collected content, the real-time collected content includes scene brightness, moving objects, semantic content, etc. Some enhancement modes can be turned on/off according to the brightness of the scene. For example, if the scene is detected to be dark, the brightness, color, resolution enhancement and denoising will be activated. When the light becomes brighter, the brightness, color, resolution enhancement and denoising will be turned off. noise. Some enhancement modes can be turned on/off according to moving objects. For example, if a moving object in the scene is detected with/without blur, the deblurring mode will be automatically enabled/disabled. If the scale of a moving object in the scene is detected to be less than a certain threshold, for example, the length of the moving object is less than the image length. 1/10, the resolution enhancement mode is automatically activated. Some enhancement modes can be turned on/off according to the semantic content, such as detecting the video scene switching from indoor to outdoor, turn on color enhancement to adapt to white balance changes, such as detecting the presence/absence of vehicles, human bodies and text in the video scene, turn on /Turn off resolution enhancement mode.

Regarding the setting of the enhancement order of the adaptive mode based on the relevant content collected in real time, if the terminal device adaptively selects multiple modes for enhancement, the enhancement mode needs to be prioritized. The principle of sorting is which mode is better in the shooting environment Enhancement is required, that is, the enhancement has the greatest impact on the improvement of video quality. For example, when shooting a video under the condition of insufficient light at night, regardless of the motion factor, the brightness enhancement mode has the highest priority, the denoising mode has the next priority, followed by color enhancement, then deblurring, and finally resolution enhancement ; Another example is in a moving shooting environment, the light intensity is normal, the deblurring priority is the highest, and other enhancement modes can be sorted according to other conditions; for example, in a more complex shooting environment, there are not only insufficient lighting problems, but also terminals. According to the movement of the device, the user can sort the enhanced modes, or sort the modes that the user pays attention to according to the user's historical data, and the mode that the user pays the most attention to is ranked higher.

Regarding adaptive settings based on device-related states, device states include power, memory, and the like. Mode setting and enhancement sorting can be performed according to the power consumption of the five enhancement modes. Suppose the sorting result is resolution > deblur > denoise > color enhancement > brightness enhancement. If the power is less than the first threshold, such as 50%, it will not be performed. Resolution enhancement, if the battery is less than a second threshold, such as 40%, no deblurring, if the battery is less than a third threshold, such as 30%, no denoising, if the battery is less than a fourth threshold, such as 20%, then No color enhancement is performed, and no brightness enhancement is performed if the battery level is less than a fifth threshold, eg, 10%. The mode setting and enhancement sorting can be performed according to the adaptive settings of the memory, for example, according to the buffer space of the five enhancement modes. It is assumed that the sorting result is resolution > deblur > denoising > color enhancement > brightness enhancement. If the memory is smaller than the first Threshold, such as 500M, no resolution enhancement, if the memory is less than the second threshold, such as 400M, no deblurring, if the memory is less than the third threshold, such as 300M, no denoising, if the memory is less than the fourth threshold , for example, 200M, no color enhancement is performed, and if the memory is less than the fifth threshold, for example, 100M, no brightness enhancement is performed.

Regarding the adaptive setting based on the mode setting history information, the number of times all enhancement modes are selected by the user is recorded, and the priority is sorted according to the number of times. For example, the sorting result is Resolution > Deblur > Denoise > Color Enhancement > Brightness Enhancement, When the next startup, priority will be given to resolution enhancement, followed by denoising, deblurring, color enhancement and brightness enhancement. Or the mode that needs to be enhanced for this shooting is determined according to the enhanced mode in the previous shooting.

In addition, there are influence relationships among the five enhancement modes, as shown in Table 1. In Table 1, "X" represents the incoherence between modes, and "O" represents that the enhancement of mode A affects the effect of mode B. Considering the correlation between modes, if a certain mode is enhanced, the related mode can be selectively not enhanced or enhanced. For example, if the resolution is enhanced, the video frame is relatively clear and can not be de-blurred. If the brightness in the night scene is bright After enhancement, the noise will become obvious accordingly, and denoising is required.

Table 1. Influence relationship between enhancement modes

Step 3, multi-mode joint enhancement is performed on the video information.

After determining the enhanced mode in step 2.4, this step enhances the enhanced modes one by one, and the order of mode enhancement is considered here, that is, the enhancement order is determined for the mode selected in step 2.4. Then, the selected enhancement modes can be processed one by one according to the enhancement methods in the previous embodiments 1 to 5 in the enhancement order.

Step 4, the storage of the video.

In this embodiment, if the enhancement mode includes resolution enhancement, the video storage mode can adopt the same video storage mode in step 4 in the first embodiment, which will not be repeated here. The setting method of the storage mode is also the same as the setting method in step 5 in the first embodiment, and will not be repeated here. The video storage mode can adopt the same video storage mode in step 4 in the second embodiment, which will not be repeated here. The setting method of the storage mode is also the same as the setting method in step 5 in the second embodiment, and will not be repeated here.

Step 5, video playback.

In this embodiment, the playback mode corresponds to the storage mode. If the enhancement mode includes resolution enhancement, the video playback mode adopts the same video playback mode as in Step 5 of the first embodiment, which will not be repeated here. If the enhancement mode does not include resolution enhancement, the video playback mode adopts the same video playback mode in step 5 of the second embodiment, and details are not repeated here.

The video joint enhancement mode is described in detail below with a specific embodiment, and the mode includes an opening step, a mode enhancement step, a parameter setting step, a storage step, and a playback step. The 14 sub-steps in Figure 4 cover these five steps.

Step 1): Start the camera of the terminal device, enter the photographing interface, and through voice control, the user says "enable video enhancement", the terminal device receives "enable video enhancement", and the terminal device enters the video enhancement mode.

Step 2): After the video enhancement mode is turned on, the red font "Video Enhancement Mode" appears on the camera interface, and the "Video Enhancement" button appears below. The user clicks this button to enter the enhanced mode selection interface.

Step 3): Enter the mode selection interface, the word "Video Enhancement" will turn into a yellow font, and six options will appear, corresponding to "Brightness", "Color", "Denoise", "Deblur", "Resolution", " automatic". The first five options correspond to different enhancement modes. Users can select one or any of them to enhance the modes, or select the last option “Auto” to allow the terminal device to adaptively select the mode that needs to be enhanced according to other factors such as the photographing environment. model. After checking, click the "Enhanced Mode" button again to enter the camera setting interface.

Step 4): The capture screen of the two cameras is displayed on the camera interface. The user can view the camera scene by switching cameras. The user can freely set which camera is the main camera to capture the original video, and the other camera is the auxiliary camera to capture key frame photos.

Step 5): After selecting the camera, click the setting button to set the acquisition parameters of the camera, and switch the camera to complete the setting of the acquisition parameters of the two cameras. Acquisition parameters include but are not limited to exposure time, light sensitivity. After completing the acquisition parameter settings, click the "Record" button to enter the next step.

Step 6): Enter the recording interface, start recording, the main camera shoots video, and the auxiliary camera starts to shoot key frame photos. In the video recording interface, after completing the shooting, the user presses the "Stop" button to end the shooting, and different storage and playback modes can be performed through different interaction methods. Click the "Stop" button to jump to step 7 in the figure to directly store the enhanced video; long press the "Stop" button to jump to step 10 to store the original video information and keyframe image information.

Step 7): the video is enhanced, the original video information is displayed in the image box in the lower right corner of the interface, and a buffering sign is displayed to remind the user of the progress of video enhancement, the enhancement is completed, the buffering icon disappears, and the enhanced video information is displayed in the image box. After the storage is completed, it will enter the normal shooting interface.

Step 8): The enhancement of the captured video is completed and stored in the terminal device. Before the next capture, the image frame in the lower right corner displays the video captured in the most recent time. You can click the image frame to view the enhanced video information.

Step 9): Enter the video playback interface, click the play button to play the video.

Step 10): directly store the original video information and key frame image information, and display the original video information in the image box in the lower right corner. The background will selectively process the video enhancement according to the processor usage, and if the processor is idle, the video will be enhanced. Click the bottom right image box to view the video. After receiving the click instruction, the terminal device firstly judges whether the video enhancement is completed, if the enhancement step is completed in the background, then jumps to step 11), if the background enhancement step is not completed, then jumps to step 12).

Step 11): The video enhancement has been completed, and the enhanced video playback interface is displayed. Click the play button to play the video.

Step 12): the video enhancement is not completed, the terminal device continues to enhance the video, the background image can display the original video information, and a buffer mark appears to indicate the enhancement progress. After the enhancement is completed, the buffer icon will disappear automatically, and jump to step 13).

Step 13): display the enhanced video playback interface, click the play button to enter step 14).

Step 14): Play the video.

Embodiment 7: Multi-mode combined image-enhanced video mode in a handheld terminal

The execution flow of the joint video enhancement mode in the handheld smart terminal will be described in detail below with a specific embodiment.

Step 1): The user picks up the handheld smart terminal, issues a voice command "enable video enhancement", the handheld smart terminal starts two cameras, and the camera A on one side captures images by default, and the camera B captures video on the other side, as shown in Figure 5 .

Step 2): As shown in Figure 6(a), by previewing the current shooting scene, the handheld smart terminal detects that the scene is daytime, sets the white balance to sunlight, detects that the scene brightness is high, and adaptively sets the image exposure to low, The default setting of the handheld smart terminal is that the video capture frequency is 30fps, the size is 640*480, the image size is 1920*1080, and the key frame capture frequency is set to 1 time per minute.

Step 3): According to the user's operation under the main screen of video enhancement, the handheld smart terminal responds to the following events, including: opening the setting function, and then selecting the white balance setting on the touch screen, as shown in Figure 6(b), using the sliding bar Scrolling adjusted the white balance, adjusted the exposure, changed the video capture frequency to 25fps, and the key frame capture frequency to 2 times/min.

Step 4): According to the operation of the user clicking the Capture icon on the interactive panel, as shown in Figure 6(c), the intelligent terminal starts video capture, and the default mode combination at the beginning is brightness enhancement and color enhancement.

Step 5): When there is a fast-moving child in the scene, and the video frame is blurred, the intelligent terminal adaptively starts the deblurring mode, and increases the key frame collection frequency to 4 times/min.

Step 6): as the child moves out of the screen and the blurred video frames disappear, the intelligent terminal adaptively turns off the deblurring mode, and restores the key frame collection frequency to 2 times/min.

Step 7): At this time, since the outdoor sunlight is getting brighter and brighter, the handheld smart terminal detects that the brightness is increased, thereby adaptively reducing the exposure amount.

Step 8): When the user enters the room, the light also becomes dark, and the detection brightness of the handheld smart terminal is weakened, thereby adaptively increasing the exposure.

Step 9): The user issues a voice command "start high dynamic range image", and the handheld smart terminal switches the ordinary brightness acquisition to the high dynamic range acquisition.

Step 10): Although the light becomes weak again and a lot of noise appears in the video, the handheld smart terminal detects the noise enhancement and adaptively activates the de-noising mode.

Step 11): At this time, when the battery power is lower than 30%, the handheld smart terminal adaptively turns off the high dynamic range shooting. When the battery power is lower than 10%, the handheld smart terminal turns off the color enhancement mode. When the battery power is lower than 5%, the system All enhanced modes will be turned off.

Step 12): the user sends out a voice command "close video enhancement", and the hand-held intelligent terminal identifies that the voice control command has closed the video enhancement, and compresses and stores the enhanced video;

Step 13): After that, every time the user opens the enhanced video, the handheld smart terminal recognizes and plays and watches the video.

Embodiment 8: Multi-mode combined image-enhanced video mode in monitoring terminal

In an outdoor environment, the current monitoring equipment often has shortcomings such as low acquisition resolution, poor acquisition brightness, sensitivity to light, and large noise in cloudy and rainy days. This embodiment provides a solution for replacing a monocular surveillance camera with a binocular surveillance camera, and performs video quality enhancement processing in the background to improve resolution, color, brightness, and denoising and blurring.

A dual-camera monitoring intelligent terminal device is built on the highway, as shown in Figure 7(a), to monitor the passing vehicles. One of the cameras captures video, and the video size is fixed at 480*320, and the other camera captures images. , the image size is fixed at 1920*1080, and two channels of data are transmitted to the monitoring backend through the network. The monitoring backend has a processor, a display screen and a set of control panels, as shown in Figure 6(b), in which the backend processor processes the two channels of data in real time, and displays the current monitoring video and captured key frames on the display screen. And the virtual control panel, the hardware control panel includes devices such as mouse and keyboard, and sets parameters, enhanced mode combination, and key frame selection. The video quality enhancement method in the monitoring situation is described below with an embodiment.

First, the operator starts "video enhancement" by pressing the key F1, and performs the following steps according to the situation:

Step 1), the default key frame acquisition mode is system adaptive adjustment, if the operator does not set, then jump to step 2), otherwise, the operator sets the key frame acquisition mode through the virtual control panel to be N frames/sec;

Step 2), the default mode combination is resolution enhancement and deblurring enhancement, if the operator does not set it, then jump to step 3), otherwise, the operator selects and combines five enhancement modes through the virtual control panel.

In step 3), the original captured video, the enhanced video and a group of recently captured key frames are displayed on the display screen in real time. There are three display effects: the original video as shown in Figure 7(c), the color enhancement and denoising enhancement as shown in Figure 7(d), and the brightness enhancement as shown in Figure 7(e). The operator can choose any display mode through the menu.

Step 4), when the monitoring terminal finds that there is a speeding vehicle in the scene, it can adaptively increase the key frame acquisition rate to obtain more high-definition images, adjust the image exposure to increase or decrease the brightness of the scene, set the focus area as the vehicle license plate, and select the white color. Balance to adjust for chromatic aberration.

Step 5), when the shooting scene is night, cloudy and rainy, the monitoring terminal adaptively starts denoising and brightness enhancement modes according to time and brightness.

Step 6), every 6 hours, the monitoring terminal compresses and stores the video collected in the first 6 hours, the key frame, the mode enhancement combination mode, and the setting parameters into the database.

Step 7), in order to call the data stored before viewing, the user queries a certain piece of data through the database, and the intelligent terminal displays the video and key frames before and after the enhancement on the display screen.

Embodiment 9. The enhancement processing mode is the deblurring enhancement mode in the video enhancement image mode

When the first type of multimedia information is video information and the second type of multimedia information is image information, the image information collected by one multimedia collection device and the corresponding image information collected by another multimedia collection device according to the set video frame collection frequency are obtained. video clips;

Wherein, according to the first type of multimedia information collected, corresponding processing is carried out to the collected second type of multimedia information, specifically including:

The index that needs to be enhanced corresponding to the collected image information is enhanced according to the collected video clips.

Preferably, it also includes: when it is detected that the multimedia collection device that collects image information enters a preview state, or when it is detected that the multimedia collection device that collects image information starts to collect image information, another multimedia collection device collects video frames according to the set frequency of collection. The video clip corresponding to the image information;

When it is detected that the number of video frames in the captured video clip reaches the corresponding upper limit value, another multimedia capture device stops capturing video information.

Wherein, according to the collected video clips, the indexes that need to be enhanced corresponding to the collected image information are enhanced, which specifically includes:

Determine video keyframes in the captured video clips;

Based on the method of blur kernel estimation, the collected image information is enhanced according to the video key frame.

Preferably, the video key frame is determined by an adaptive key frame determination method;

Wherein, the adaptive key frame determination method is determined by one or more pieces of information among picture blur degree, content similarity, and video frame quality.

Preferably, the method further includes: performing definition analysis on the collected image information; if the image information belongs to a blurred image, performing enhancement processing on the index that needs to be enhanced corresponding to the collected image information according to the collected video clips; Enhanced metrics are included in blur.

When the first type of multimedia information is video information, and the second type of multimedia information is image information, the collected image information is stored and processed according to the collected video information, wherein the storage content includes at least one of the following situations:

Image information obtained by enhancing the collected image information according to the collected video information;

The collected video information and image information;

The collected image information and the video key frames used for enhancing the image information in the collected video information;

The collected image information and the enhancement model when the image information is enhanced;

According to the collected video information, the collected image information is enhanced and processed, and the collected video information is obtained.

Optionally, in response to the received display trigger operation, the image information is displayed based on a display mode that matches the stored content; wherein, the display mode includes at least one of the following:

When the enhanced image information is stored, the enhanced image information is directly displayed;

When the collected video information and image information are stored, the collected image information is enhanced and displayed according to the collected video information;

When storing the collected image information and the video key frame for enhanced processing, the enhancement model is determined according to the video key frame, and the collected image information is enhanced and displayed through the enhanced mode;

When the collected image information and enhanced model are stored, the collected image information is enhanced and displayed through the enhanced model;

When the enhanced image information and the collected video information are stored, the enhanced image information and the collected video information are associated and displayed.

In this embodiment, one camera in the binocular camera collects images, and the other camera collects videos to obtain high-quality images. In order to take a high-brightness and clear image, users often take long-exposure photos. In the absence of a tripod, hand-held smart terminals are prone to shake, and the captured image will appear blurry. Start another camera at the same time as taking a picture to shoot a short video. The exposure time of each frame of the video is short, and the edge and texture information are well preserved. Compared with the image, the video is dynamic. to enhance. The motion blur kernel can be estimated from the video frames and photos, and then the image is deblurred to obtain a high-brightness and clear image. The main steps are as follows.

Step 1: Turn on the deblur enhancement mode in the video enhancement image mode.

In this embodiment, step 1 adopts a similar opening method as in the first embodiment, the difference lies in the difference in some instruction descriptions and threshold settings, and the specific differences are described as follows.

Different command descriptions are used when the user turns on the image deblurring enhancement mode. For example, the command in voice activation is "start image deblurring", the command in button activation is long-pressing the Home button, and the button for image deblurring in the virtual key is held by hand. The opening command is to shake the terminal, etc., which will not be repeated here.

Different threshold settings are used in the terminal device to adaptively turn on the mode according to the device-related state and mode turn-on history data, for example, the power threshold, the number of times threshold, and so on. The other description methods are the same, and are not repeated here.

Three methods are given in the terminal device adaptively enabling the image deblurring mode according to the shooting environment and acquisition parameters. One method is that the intelligent terminal uses the existing method to detect the movement trend of the terminal device. If the shooting terminal device is in a moving state, such as If the handheld terminal device shakes due to the hand shake of the handheld terminal device, turn on the image deblurring enhancement mode; the second method is that the terminal device detects the shooting parameters. If the exposure time of the image acquisition is higher than a certain threshold, such as high In 300ms, the image deblurring enhancement mode is turned on; the third method can be a combination of the first two methods, and the image deblurring enhancement mode is turned on when the two conditions are satisfied at the same time, that is, the shooting terminal is in motion, and the exposure time is higher than a certain Threshold turns on image deblurring enhancement mode.

The terminal device automatically turns on the image deblurring mode according to the relevant content collected in real time to calculate the blurriness index of the captured image, and if it is lower than a certain threshold, the image deblurring mode is turned on for subsequent image capture.

Step 2: Use one camera of the binocular camera to capture images and the other camera to capture video.

This step mainly includes the setting of acquisition parameters and enhancement strategy parameters, the settings of the main and auxiliary cameras, the parameter settings of the acquired images and videos, and the selection of video key frames.

Step 2.1, set the primary and secondary cameras.

In this embodiment, the method for setting the main and auxiliary cameras can be similar to the method for setting the main and auxiliary cameras in step 2.1 of the first embodiment, which will not be repeated here. In this implementation, the main camera is responsible for capturing images, and the secondary camera is responsible for capturing video.

Step 2.2, set the camera's acquisition parameters and enhancement strategy parameters

The parameters in this embodiment are the parameters that need to be set for the binocular camera to collect images and video clips. In addition to the acquisition parameters of the camera mentioned in column 1, the frequency of video frame collection is increased, that is, the number of video frames collected per second, and the image Parameters in deblurring algorithms, etc.

To set the acquisition parameters and enhancement strategy parameters of the camera, a similar setting method can be used in the first embodiment, except that the settings for the video frame acquisition frequency and the adaptive parameter adjustment for image deblurring are added. The newly added settings are described below. method is introduced.

Regarding the system default settings of the terminal device, the video frame capture frequency is set to a certain default value, and the video frame capture frequency is set to the default value before receiving the instruction to modify the video frame capture frequency. Other methods are the same, and will not be repeated here.

Regarding the user interaction setting, for the video frame collection frequency, the user can set the frame number of the video clip through voice, slide bar, button, text input, etc. The collection parameters are limited by the value range of the terminal device itself, and the manual setting by the user also needs to be selected within a certain range, otherwise, a warning from the terminal device will be received, and the specific content will not be repeated here.

Regarding the adaptive setting according to the environment, for example, if the terminal device detects that the shooting terminal is in a motion state, it increases the frequency of video frame capture collected by the secondary camera.

Regarding the adaptive setting according to the power, for example, the video frame collection frequency is controlled by the power. If the power is less than a certain threshold, for example, if it is less than 50%, the number of frames of video frames collected per second will be reduced. If it is less than 5%, the fixed number of frames will be the minimum. value. About adaptive settings based on storage space. According to the storage space, the video frame collection frequency can be adjusted. If the remaining storage space is larger than a certain threshold, for example, larger than 50% of the total storage space of the terminal device or larger than 500M, it will automatically adjust to a high video frame collection frequency, such as 30 per second. frame, otherwise, adjust to a low video frame capture frequency, such as 25 frames per second. According to the storage space, you can adjust the frequency of video frame collection. If the remaining storage space is larger than a certain threshold, for example, larger than 30% of the total storage space of the terminal device or larger than 300M, increase the video frame collection frequency collected by the secondary camera, and vice versa. Reduce the frequency of video frame capture for secondary camera capture.

Regarding the adaptive setting based on historical data, for example, setting the video frame collection frequency according to user preferences, the following method can be used but not limited to: Count the video frame collection frequency of the video set in the last N times of image deblurring, and the terminal device recommends setting the video frame frequency. The frame acquisition frequency is the average of the previously set acquisition frequencies.

Regarding the adaptive setting of acquisition parameters according to the acquired related content, in this embodiment, one of the binocular cameras is used to acquire an image and the other to acquire a video to deblur the image. Here, the resolution, exposure time, and sensitivity of video and image capture are mainly set. For the consideration of energy saving and algorithm design, the resolution of the video captured by the auxiliary camera should be consistent with the resolution of the image. If If the highest resolution of the video is lower than that of the current image, the highest resolution of the video is used for acquisition. If the brightness of the photographing environment is normal or higher than a certain threshold, shorten the image and video exposure time, appropriately increase the sensitivity, and reduce the probability of picture blurring; if the sensor detects that the terminal device has jitter or other motion trends, reduce the exposure appropriately. time, to avoid blurring of images and videos, which will affect the final effect of the image.

Step 2.3, acquisition of video clips

This step describes when the auxiliary camera starts to shoot video, which can be used but not limited to one of the following two methods: one is to shoot before the image is captured, that is, when browsing the interface; the other is to start at the same time when the photo starts to shoot. Video shooting.

When shooting in the browsing interface, in order to save the video storage space and ensure the strong correlation between the video image and the photo content, only some frames of the video captured in the browsing interface are reserved. It is also possible to adjust the frame number of this part of the video adaptively according to the total number of video frames, for example, set it to 10% of the total number of frames. If the total number of video frames is large, the number of video frames that can be saved in this part will be more, and vice versa. Set a buffer sequence to store this part of the video frame, and set the maximum number of frames stored in the sequence to the frame number of the video clip set by the terminal device. When the sequence is full and a new video frame comes, remove the frame in the sequence. The earliest captured video frame, if the video frame in the video sequence is stored according to the shooting time, the first frame is removed, and space is reserved to store new video frames, and the video sequence is updated according to this method, and only the latest video frame is kept. video clips. Continue to shoot video at the beginning of shooting, and stop video shooting according to the following two judgment conditions. One is to stop video shooting when the sum of the number of frames shot and the number of previous frames reaches the upper limit of the total number of video frames; If the number of frames does not reach the upper limit of the number of frames, stop shooting, and store the total number of video frames obtained.

When shooting starts, start video shooting at the same time, and stop video shooting when the number of video frames reaches the video frame number set by the terminal device. video clips.

Step 3: Use the video to remove the blur from the image;

After obtaining the images and video clips, the terminal device analyzes the clarity of the image, and can classify the images according to the parameters obtained by the terminal sensor during the photo shooting, or whether there is a moving target in the video clip, or existing classifiers in the literature. . If the picture is a clear image, do not perform deblurring, otherwise continue with the following deblurring.

To save processing time, you can select several frames in the video clip to deblur the image, i.e. determine video key frames. The number of video key frames can be adaptively adjusted according to the degree of image blur, and can also be set to a fixed value or a fixed ratio, for example, set to one-fifth of the total number of frames. The selection of video key frames can be based on the similarity of content to select the most similar frames; the quality of the video frames can also be used to select the first few frames of quality; the combined indicators can also be used to select video frames, such as video quality. Sort and then keyframe the number of frames based on the degree of blur.

After getting the video key frame, first unify the video key frame and the image scale, zoom the video key frame or the image, or zoom the video key frame and the image at the same time, so that the image and the video frame have the same size. size of. Use the method of stereo matching to calculate the disparity between the key frame and the image, align the video frame and the image, find the corresponding area between the video frame and the image, and create a blurred image by combining each frame of video, that is, the non-blurred image, with the corresponding area of the blurred image. To the correspondence between clear images, based on each pair of corresponding regions, multiple blur kernels can be solved using the existing blur kernel estimation methods in the literature. A final fuzzy kernel is obtained by weighted average of all the obtained fuzzy kernels, where the weight of each kernel can be distributed evenly; it can also be distributed according to the similarity of the content, the weight of high similarity is high, the weight of small similarity is high Small; the weight can also be allocated according to the quality of the video frame, the higher the quality, the higher the weight. Finally, the captured image is deblurred using the above blurring check. Deep learning methods can also be used to learn a deblurring model using these image pairs to deblur the image.

In addition to calculating the blur kernel by using the content relationship between the image and the video frame, the current algorithm can also be used to estimate the motion trajectory of the moving target or the terminal device itself based on continuous video frames to correct the blur kernel and improve the accuracy of the blur kernel. achieve better deblurring effect.

Step 4: Storage of images;

In this embodiment, the terminal device generates four types of data: original image information, video clips, deblurred models and deblurred high-quality images. Its storage method includes but is not limited to one of the following five.

The first is to store the enhanced image directly without saving the video clip, that is, to complete the enhancement process before storage. During storage, the original image is displayed in the photo frame, and a buffer icon is displayed on it, indicating that the enhancement process is in progress. , complete the storage;

The second is to store the original image and the learned deblurring model;

The third is to store the original image and the selected video key frames, that is, the video frames used to learn the deblurring model. This method reduces the processing time of the image before storage. It only needs to find out the video key frames and calculate the deblurring model. After the storage is completed, the terminal device automatically arranges the processing time according to the busyness of the terminal processor, and after calculating the deblurring model or directly deblurring the image, the stored video key frames can be deleted;

The fourth is to directly store the original image and the saved video clip, so the enhancement steps are performed by the terminal device after storage;

The fifth is to save the video key frames while storing the enhanced image. The enhanced image can be obtained by any of the first four storage methods. The saved key frame sequence can be played to see the effect of a dynamic image. .

Regarding how to set the storage mode, the present invention provides three setting modes, and the terminal device can select the storage mode according to one of the following three. The first is the default setting of the terminal device; the second is that the terminal device accepts the user to change the storage method by means of voice, keys or external controllers, and a combination of these methods; the third is that the terminal device uses storage space, power or history Data adaptive settings storage method.

Regarding the system default settings, the terminal device sets one of the five storage modes as the default value, and uses this default storage mode to store videos and images before the terminal device receives an instruction to change the storage mode.

Regarding the voice settings under user interaction, for example, the user presets the voice command "store enhanced image". If the command is received, the voice control command will be recognized by voice, and the storage mode of the setting is determined as the first storage mode, that is, storage Enhanced image. Other user interaction methods are the same as those used in step 4 in Implementation 1, and will not be repeated here.

Regarding the adaptive setting according to the storage space, different storage methods can be selected according to the storage space. If the remaining storage space is less than a certain threshold, for example, less than 10% of the storage space of the terminal device, the first storage method is set; if If the remaining storage space is less than a certain threshold, for example, it is lower than 40% of the storage space of the terminal device, it can be set to one of the first three and the fifth type; if the remaining storage space is higher than a certain threshold, such as higher than the terminal device storage space 50% of the storage space, the storage method is not affected by the storage space.

Regarding the adaptive setting according to the power, the storage mode can be controlled according to the power. When the power is less than the first predetermined power, for example, less than 50%, the storage mode with low power consumption is selected, that is, the second or third storage mode , that is, directly store the original image and video key frames or learn the model, without performing enhancement processing on the image; when the power is less than the second predetermined power, the second predetermined power is less than the first predetermined power, for example, when it is lower than 15%, select power consumption The fourth storage method with the smallest amount is the original image and video clip; if the battery level is higher than a certain threshold, such as higher than 50%, the storage method is not affected by the battery level.

Regarding the adaptive setting of the historical data according to the storage mode, for example, according to the storage mode previously set by the user, the user's preference is analyzed, and the storage mode is set as the user's preference.

Step 5, the display of the image

The terminal device displays the stored image according to the detected display operation from the user.

When displaying the stored image, there are different display modes corresponding to different storage modes, and the present invention provides five display modes. The terminal device can choose but not limited to one of the following five display modes.

The first: for the first storage method, the terminal device detects the user's opening operation and directly displays the deblurred image. The operation of the user clicking to view is detected, for example, the operation of the user clicking the view button is detected, and the image can be displayed immediately, so that the user can directly view the effect after deblurring.

The second type: For the second storage method, the terminal device detects the user's opening operation and opens the combination of the original image and the deblurring model. After detecting the operation of the user clicking to view, for example, detecting the operation of the user clicking the view button, the terminal device starts to deblur the image based on the deblurring model. Complete the operation within the allowed buffer time, and then display the enhanced image.

The third type: For the third and fourth storage methods, the terminal device detects the user's opening operation, for example, detects the user's operation of clicking the view button. If the terminal device has completed the deblurring process, the image can be displayed directly Check out the enhanced effect. If the terminal device only does part of the work in the background and does not complete the deblurring step, it will take time to buffer the image after receiving the user's click to view operation. After the processing is completed, the image will be displayed and the user can see the deblurred image image.

Fourth: For the fifth storage method, the terminal device detects the user's opening operation, and opens the combination of the deblurred image and video key frames. The display method of the deblurred image is the same as the first display method, the difference is that there is more dynamic image display. The key frame sequence playback can present the effect of a dynamic image. You can view the dynamic image associated with the image by long-pressing the photo, but it is not limited to this control method. Voice, gesture and other settings can be used to display the dynamic image.

The image deblurring enhancement mode is described in detail below with a specific implementation column.

1) The steps of opening the interaction are shown in Figure 8:

Step 1): Turn on the camera of the terminal device, enter the camera interface, and turn on the video enhanced image mode by long pressing the screen.

Step 2): After the video-enhanced image mode is turned on, the camera interface displays the capture images of the two cameras, and the user can freely set which camera is the main camera to take pictures by switching cameras to view the camera scene, and the other camera is the auxiliary camera to take videos.

Step 3): After the camera is selected, the acquisition parameters can be set for the two cameras respectively. In order to shoot high-brightness images, the exposure time can be increased, the sensitivity can be decreased, and the picture quality can be improved. After completing the parameter setting, go to the next step.

Step 4): End the interactive parameter setting, enter the photo-taking interface, and the screen displays the screen within the field of view of the main camera.

2) Mode enhancement, storage and playback steps are as follows:

Step 1): In the photographing interface, when the user sees an interesting picture, press the photographing button to take a photograph, and different storage and playback modes can be performed through different operation modes. Click to jump to step 2 in Fig. 9), directly store the photo after enhancement; Long press and jump to step 5), store original image and video clip;

Step 2): The image is enhanced, the original image is displayed in the upper left image box, and a buffer mark is displayed to remind the user of the progress of image enhancement, the enhancement is completed, the buffer icon disappears, and the enhanced image is displayed in the image box.

Step 3): the enhancement of the captured photo is completed and stored in the terminal device. Before the next photo is captured, the upper left image frame displays the photo captured in the most recent time, and the enhanced image can be viewed by clicking on the image frame.

Step 4): After clicking the image box, the enhanced image is displayed.

Step 5): Directly store the original image and video clip, and display the original image in the upper left image box. The background will selectively process the image enhancement according to the processor usage, and if the processor has the processing capacity, the image will be enhanced. Click on the upper left image box to view the image. After receiving the click instruction, the terminal device firstly judges whether the image enhancement is completed. If the enhancement step is completed in the background, it will jump to step 6). If the enhancement step has not been completed in the background, it will jump to step 7).

Step 6): Image enhancement has been completed, and the enhanced image is displayed.

Step 7): the image enhancement is not completed, the terminal device continues to enhance the image, the background image can display the original image, and a buffer mark appears to indicate the enhancement progress. After the enhancement is completed, the buffer icon disappears automatically, jump to step 8).

Step 8): Display the enhanced image.

Embodiment 10. Multi-focus area joint play mode

The purpose of this embodiment is to help the user to shoot a video with multiple focal points. The video with multiple focus can be the global area and a local area of interest to the user, or it can be two local areas of interest. For example, when shooting a dance performance of family members or friends, it is hoped that both the overall scene and the close-up of one's own family or friends, or multiple friends can be captured at the same time. The current video shooting method not only Users are required to perform manual zoom-in and zoom-out operations frequently, which can easily cause blur or jitter in the captured video, and at the same time, only the whole world or an enlarged local area can be seen, and the global image cannot be captured at the same time. and clear images of a certain part, or unable to capture multiple parts of interest at the same time. In the present invention, two cameras are used to set different focus areas, wherein one camera focuses on the global area, the other camera focuses on a local area of interest, or one camera focuses on a local area of interest, and the other camera focuses on another local area of interest , and then two cameras shoot at the same time to obtain a multi-focus video with both global and local regions of interest, or two clear local region videos.

It should be noted that, in the detailed description of the following embodiments, the corresponding processing method for the second type of multimedia information will be performed according to the first type of multimedia information. A multi-focus area co-play mode that performs co-play processing by focusing on video information in another focus area.

In this embodiment, the first type of multimedia information and the second type of multimedia information are video information focused on different focus areas; wherein the focus areas include a global area and/or a local area.

Specifically, according to the collected video information focused on one focus area, joint playback processing is performed on the collected video information focused on another focus area.

Wherein, the focus area is determined by at least one of the following ways:

When it is detected that a local area is selected by the user, it is determined that the selected local area is the focus area, and the other focus area is the global area;

When it is detected that the user has selected two local areas, it is determined that the selected two local areas are focus areas.

Preferably, the user-selected local area is detected through the user-selected focus object.

Wherein, the global area and/or the local area may be played jointly through a split-screen layout.

(1) Turn on the multi-focus area joint play mode

There are two ways, one is that the user actively starts the multi-focus video, and the other is that the terminal device prompts the user whether to start the multi-focus video shooting according to the content of the captured video. Similar to the previous description, the user can activate the multi-focus video capture mode through voice, keystrokes, gestures, biometrics, external controls, etc., and any combination of these interactions.

Regarding the use of voice activation, for example, the user can preset the voice activation command: "Enable multi-focus video shooting", if the terminal device receives the user's voice control command "Enable multi-focus video shooting", it will perform content recognition for the voice-activated command. Therefore, it is determined that multi-focus video shooting needs to be enabled at this time.

Regarding key-on, the key can be a hardware key, such as the volume key or the Home key. The user turns on the multi-focus shooting mode by long pressing the volume key or the Home key. After the terminal receives the above-mentioned long-pressing operation event from the user, it confirms that it needs to switch to up to Focus video shooting mode. The buttons can also be virtual buttons, such as virtual control buttons on the screen, menus, etc. The terminal can display the multi-focus shooting virtual buttons on the preview interface of video shooting. Focus video capture interface. When the button is turned on, it can also indicate different meanings in combination with the pressure, speed, time, frequency and other characteristic information pressed by the user. For example, pressing lightly means changing the target person of the focus point, pressing again means modifying the magnification of the person in focus, or long Press the indication to turn on the shooting mode of multiple focus points, target people, etc.

Regarding opening through gestures, gestures include screen gestures, such as double-clicking the screen/long-pressing the screen, etc. When opening through screen gestures, different meanings can be expressed in combination with the pressure, speed, time, and frequency of user gestures, such as a tap to change the focus target People, press again to modify the magnification of the focus person, and for example, long press to open the shooting mode of multiple focus points and target people, and so on. Gestures also include space gestures, such as shaking the terminal/flipping the terminal/tilting the terminal, different directions, angles, speeds and strengths of shaking/flipping/tilting can represent different meanings, such as shaking up and down to change the focus target person, shaking left and right It means to change the parameters during shooting, and for example, tilting to the left means switching the display mode, and tilting to the right means switching the storage mode. The above gestures can be a single gesture or any combination of any gestures. If you press and hold the screen and shake the terminal, it means that multi-focus video shooting is enabled and you can change the focus point in real time to shoot the target person.

Regarding opening by biometric features, including but not limited to handwriting features, fingerprint features, and voiceprint features. For example, when the terminal is in the video shooting preview interface, if the fingerprint or voiceprint detected by the fingerprint detector or the voiceprint detector is consistent with the pre-registered user, it needs to switch to the prompt to turn on the multi-focus video shooting mode.

Regarding opening through an external controller, the external controller may be a stylus, a microphone and other devices associated with the terminal device. For example, the terminal device detects that the stylus is taken out and quickly inserted back into the terminal, or the preset button of the stylus is reset. Press, or the user uses a stylus to make a preset air gesture, confirm that it is necessary to switch to the multi-focus video shooting mode at this time; the external controller can be a smart watch, smart glasses, etc., and other devices can be mobile phones or other accessories or accessories Or an independent device, the above-mentioned wearable device can access the terminal device through wifi and/or NFC and/or Bluetooth and/or data network, and the above-mentioned wearable device or other devices can interact through at least one of buttons, gestures, and biometrics. Confirm that the user wants to switch to the multi-focus video shooting mode, and notify the above terminal device.

(2) Determine the multi-focal area

When the multi-focus video shooting mode is enabled on the terminal device, the user can manually specify multiple areas on the shooting preview interface. If the user only specifies one area, the multi-focus area at this time is the entire image captured on the shooting preview interface and the specified area. of this area.

If the user specifies more than two areas, the multi-focus area video will be alternately aimed at a specified area when shooting, so as to obtain a video composed of multiple areas of interest specified by the user.

In addition to the above manual designation by the user, the terminal device can automatically determine the multi-focus area according to the object (such as a person) in the current shooting scene. For example, to detect the number of people in the scene, the area with the largest number of people is regarded as a region of interest, and the entire scene is regarded as a global area.

The focus object can also be selected by the user. Confirm the area where the focus object selected by the user is located as the focus area. When the user turns on the multi-focus video shooting mode, there are many ways to select the focus point person. For example, when entering the preview mode of multi-focus video shooting, face detection in the global area is automatically performed, and the user determines the focus point area for shooting by clicking or dragging the detected face area. When entering the shooting mode, the face tracking and recognition technology can be used to track the focused person of interest to the user in real time, use one of the binocular cameras to shoot the global area video, and the other camera to shoot the tracked focus of the user's interest. People video. When the user wants to change the target person of interest, he can double-click the screen to start the face detection in the global area video. The user can select a certain person in the detected face, or manually specify the interested person. area. At the same time, the bounding box of the region of interest can be modified dynamically. For example, the face area is enlarged to the face and shoulder area, or the upper body area of the character, or the entire body area of the character.

(3) User Interaction Mode

Different interaction methods are provided according to the horizontal or vertical screen state of the handheld terminal device when the user shoots multi-focus video. When it is detected by the gravity sensor that the terminal device is in the horizontal screen state to shoot video, the video layout method shown in Figure 10 is used for the left and right screen display. The left side is "global area video", the right side is "local focus target person video", or the left side is "local focus target person video", and the right side is "global area video". At the same time, depending on the number of people in focus, multiple target people can be displayed in the "Partially Focused Interested People Video" area on the left or right. If it is detected that the current terminal device is in the vertical screen state to shoot video, use the video layout method shown in Figure 11. The upper part is "global area video", the lower part is "local focus target person video", or the upper part is "local focus target person video", and the lower part is "global area video". At the same time, according to the number of people in focus, multiple target people can be displayed in the upper or lower "Partially Focused Interested People Video" area. When the user wants to pay more attention to the global area, and wants to store the target person he is interested in for subsequent playback, he or she can select the large and small screen playback mode as shown in Figure 12. The global video occupies almost the entire screen, and the partially focused video of the target person of interest is located in a small position on the screen, which may be the lower right corner, the lower left corner, the upper right corner or the upper left corner. Alternatively, the placement position can be specified by the user.

(4) Multi-focus video storage

Preferably, the second type of multimedia information is processed correspondingly according to the first type of multimedia information, specifically including:

According to the collected video information focusing on one focus area, the collected video information focusing on another focus area is stored and processed, wherein the storage content includes at least one of the following situations:

1) The collected two video information focused on different focus areas;

2) according to the video information that is focused on a focal area collected, the synthetic video information after the synthetic processing is carried out to the video information that is focused on another focal area collected;

3) the video content of interest in the two video messages determined to focus on different focus areas;

4) The collected video information focused on the global area and the location information of the local area in the video information of the global area.

Based on the multi-focus shooting mode, the following four storage modes are given in this implementation column:

Method 1: Store the multi-focus videos captured by the two cameras separately to obtain two video files. If one of the two cameras is in global focus and the other is in local focus, one of the two stored videos corresponds to the global focus video and the other corresponds to the local focus video; if both cameras are partially focused, the two videos correspond to the two videos respectively. Partial focus video obtained by each camera.

Method 2: This storage method provides a video synthesis storage method that can be seen and available. The storage content is the same as the picture displayed on the display screen of the terminal device, and each frame of the video in the video simultaneously presents the shooting pictures of the two cameras at the same time. For example, for the three screen presentation modes shown in Figure 10-12, for the large and small screen mode shown in Figure 12, each frame in the stored video is a screen of a large and small screen, and the content of the screen is displayed on the screen at the corresponding moment. The large screen and the small screen in the picture correspond to the content captured by the two cameras respectively.

Mode 3: This storage mode provides a method of combining and storing videos driven by user interests. This mode is aimed at the presentation mode of the screen divided into the main screen and the sub-screen. As shown in Figure 3, the large screen is the main screen and the main screen The content displayed on the screen represents the field of view that the user is currently interested in, and the final stored video is the content displayed on the main screen. Each frame in the video directly presents the area of interest to the user at that moment.

Method 4: This storage method is a shooting method for the global focus and local focus of the shooting. The global shooting video can be stored during storage, plus the location information of the four points on the bounding box of the local area obtained by real-time tracking. Identified local ROI objects. As shown in Figure 13, in the global area video, the global area video is saved, and the positions of the rectangular boxes tracked in real time in the global area, such as the four corners of the yellow area in Figure 13, are saved. And use this rectangular frame size as the standard to save the content of the local focus area captured by another camera.

Regarding how to set the storage mode, the present invention provides three setting methods, and the terminal device can select the storage mode according to one of the following three methods. The first is the system default setting of the terminal device; the second is that the terminal device accepts the user’s way of triggering operations through voice, keys or external controllers, and the combination of these methods to change the storage mode; the third is that the terminal device changes the storage mode through the device Relevant information, such as storage space, or adaptively set the storage method for historical data.

Regarding the system default settings, the terminal device sets one of the four storage modes as the default value, and uses the default storage mode to store the video before the terminal device receives an instruction to change the storage mode.

Regarding the setting method of user interaction, the setting method similar to step 4 in the first embodiment is adopted, and the difference lies in the description content of the instruction. For example, the instruction in the voice setting is "store video separately". The command performs voice recognition, and determines that the setting storage method is the first storage method. The settings for other user interactions are the same, and will not be repeated here.

Regarding the adaptive setting according to the storage space, different storage methods can be selected according to the storage space. If the remaining storage space is less than a certain threshold, for example, less than 50% of the storage space of the terminal device, it is set to one of the last three storage methods. One; on the contrary, if the remaining storage space is higher than a certain threshold, for example, higher than 50% of the storage space of the terminal device, the storage mode is not affected by the storage space.

Regarding the adaptive setting based on the historical record data, for example, according to the storage method previously set by the user, the user's preference is analyzed, and the storage method is set as the user's preference.

(5) Multi-focus video playback mode

Preferably, in response to the received play trigger operation, the video information is played based on a play mode that matches the stored content; wherein, the play mode includes at least one of the following:

1) when storing the collected two video messages focusing on different focus areas, the two video messages are played separately or jointly;

2) when the composite video information is stored, the composite video is played;

3) when storing determined focusing on the video content of interest in the two video information of different focus areas, play the video content of interest;

4) when storing the video information of the global area and the location information of the local area in the video information of the global area, determine the video information of the local area by the location information, and play the video information of the global area and the video information of the local area separately or syndicated play.

Corresponding to the above-mentioned storage mode, the present invention provides four playback modes, and the terminal equipment can select but not be limited to one of the following four kinds:

The first type: The terminal device detects the user's opening operation for two independent videos stored separately. When the operation of the user clicking to play is detected, for example, the operation of the user clicking the play button is detected, the videos captured by the two cameras can be played respectively. The two saved videos are stored in the memory of the terminal device in association with a certain time. When the user chooses to play the captured video, the two independent videos can be displayed and played in full screen respectively, or the terminal device can adaptively associate the two video images to play at the same time. Play the two video images in three ways: up and down, left and right, and large and small.

The second type: the terminal device detects the user's opening operation, and targets the stored composite video. The operation that the user clicks to play is detected, for example, the operation that the user clicks the play button is detected, and the synthesized video is played. Users can see multi-focus video captured by both cameras.

The third type: the terminal device detects the user's opening operation, and targets the stored merged video. The operation of the user clicking to play is detected, for example, the operation of the user clicking the play button is detected, and the video combined by the video clips of the dual cameras is played. The user can see a multi-focus video of the frame of interest presented in the main screen while the video was being recorded.

The fourth type: the terminal device detects the user's opening operation, which is for the combination of the stored global area video and the rectangular coordinates of the area of interest. Detecting the user's click to play operation, for example, detecting the user's click of the play button, the user can play the global area video and the local area video of the size corresponding to the four coordinate points alone; or you can associate the two videos to play. There are two ways of associated playback: one is to play two video images at the same time by referring to the three forms of "up and down screen", "left and right screen" and "large and small screen". The large screen displays the global area video or the partial focus area, and the small screen displays the partial focus area of interest or the global area video. Users can switch the display content of these two screens by clicking the large image or the small image. . In addition, the position of the small screen on the large screen may also be designated by the user, as shown in FIG. 14 . When the user does not specify the position of the small screen, the terminal device can automatically place the small screen at any position in the four corners of the screen. When the user wants to change the position of the small screen, the terminal device determines how to place it by detecting the user's gesture or operation.

The specific implementation scheme of the multi-focus area joint play mode is described in detail below with an embodiment.

Step 1. Open interactive steps:

1). Turn on the camera of the terminal device, enter the video shooting preview interface, and turn on the multi-focus video mode by double-clicking the screen.

2). After the multi-focus video mode is turned on, as shown in Figure 15, the preview interface displays the images captured by the two cameras, of which the global area image occupies the full screen. In the global area image, all faces in the scene are displayed area. The user clicks on the face region of interest and interactively pulls the detection frame to frame the entire region of interest. At this time, one part of the screen is the global area screen, and the other part is the local area of interest screen. As mentioned above, the layout of these two parts of the screen can be left and right screens, upper and lower screens, or large and small screens. Here, the large and small screens are used as an example.

3). After selecting the local area that the user is interested in, the two cameras focus on the global area and the designated local area respectively, and then start to shoot multi-focus video.

Step 2. The multi-focus video shooting, storage and playback steps are as follows:

Step 2.1 Multifocal Video Shooting

1). In the multi-focus video shooting interface, when the user sees the global area screen and the local area of interest screen, the user presses the shooting button to shoot video, and different storage and playback modes can be performed through different operation methods. Click the shooting button in the interface to jump to 2), which will directly record the screen shot of the current terminal device screen; long press the bounding box of interest in the global area to jump to 5), store the global area video and real-time tracking to get The four point positions on the bounding box are stored, and the four point positions are determined to store the video of the size of the local area of interest captured by another camera in focus. If the global area video and local area video are touched at the same time, go to 7) to store the global area video and local area of interest separately.

2). Directly record the picture captured on the screen of the current terminal device. The global area picture occupies the entire screen of the terminal device, and the local area picture is located on a small window. The position of the widget can be changed by the user moving in real time.

Step 2.2 Multifocal Video Storage

3). The current screen displays the multi-focus video composed of the global area and the local area, which is stored in the terminal device. Before shooting the next multi-focus video, the image frame in the upper left corner displays the multi-focus video captured in the most recent time. You can click the image frame to view the latest multi-focus video.

Step 2.3 Multi-focus video playback

4). After clicking the image box in the upper left corner, the latest multi-focus video shot will be displayed on the terminal device, and the playback mode at this time is the same as the content seen when shooting.

5). Store the global area video and the four point positions on the bounding box obtained by real-time tracking, and store the video of the size of the local area of interest shot by another camera that is determined by the four point positions. The user can play the global region video and the local region video corresponding to the size of the four coordinate points independently; or can play the two videos in association. The two video images are played simultaneously in the “large and small screen” association mode. The large screen displays the global area video or the local focus area, and the small screen displays the partial focus area of interest or the global area video. The display content of the two screens can be switched by means of a picture or a small picture. In addition, the position of the small screen on the large screen may also be designated by the user. When the user does not specify the position of the small screen, the terminal device can automatically place the small screen in any position of the four corners of the screen. After the video is stored, the most recent multi-focus video shot will be displayed in the frame on the upper left corner of the screen. If it is playing, go to 6).

6). Long press the content in the upper left corner of the screen to play the most recent multi-focus video.

7). The global area video and the local interest area video are stored separately, and the terminal device can play two parts of the video respectively when playing. The terminal device can also adaptively associate two video images to play at the same time. You can refer to the three methods of "up and down screen", "left and right screen" and "large and small screen" introduced in "Interaction Mode" to play the two video images. play.

Embodiment 11. The enhancement processing mode is the target object highlighting playback mode

Target objects include people and objects of interest. In the following embodiments, a person of interest is taken as an example to illustrate the embodiment.

In the video shooting of daily life and work, the people of interest are often not prominent due to the large number of people photographed. Not only is it difficult to locate the location of the speaker of interest on the image, but the voices of the people are often confused. The current video shooting methods do not highlight people of interest, but the use of binocular lenses and multi-microphones can locate the depth of the people in the scene and the orientation of the sound, which provides the necessary conditions for highlighting the people of interest when shooting. In the present invention, through the cooperation of binocular cameras and two or more microphones, the characters in the image are associated with the voice of each character when shooting the video, so as to realize only the actions and actions of a certain character of interest in the video are played. The purpose of the sound is to get the effect of highlighting the target person of interest. In this way, the action and sound of a certain character can be highlighted in the multi-person scene video obtained by shooting.

It should be noted that, in the detailed description of the following embodiments, the method of correspondingly processing the second type of multimedia information according to the first type of multimedia information, specifically, according to the collected audio information, the collected video information is processed audio. The target object highlighting playback mode of video highlighting processing.

When the second type of multimedia information is video information and the first type of multimedia information is audio information corresponding to the video information, audio and video highlight processing is performed on the collected video information according to the collected audio information.

Specifically, the target object is determined from the collected video information; highlighting processing is performed on the video information and/or audio information corresponding to the target object.

Wherein, the target object is determined from the collected video information in at least one of the following ways:

Determine the target object according to the specified operation of the detected target object;

The target object is determined according to the number and location information of multiple objects in the collected video information.

Preferably, highlighting processing is performed on the audio information corresponding to the target object, which specifically includes:

The collected video information is detected to determine the number of objects in the video information, the position and orientation information of each object;

Determine the audio information corresponding to each object according to the position and orientation information of each object;

Determine the audio information corresponding to the target object, and perform highlight processing.

Preferably, the video segment where the target object is located is determined in the collected video information, and the audio segment corresponding to the target object is determined in the collected audio information according to the corresponding relationship. The present invention proposes that, in the multi-person scene video obtained by shooting, the actions and voices of one or several persons of interest are highlighted. Through the cooperation of the binocular camera and two or more microphones, the characters appearing in the video image are associated with their respective speaking voices, so as to play or highlight only one or several characters of interest in the video. The purpose of the action and sound is to get the effect of highlighting the person of interest. The specific plans are as follows:

First, the terminal detects the face area in the video image frame for the video obtained by shooting, and according to the number of detected faces, the total number of target persons in the current scene can be obtained; secondly, according to a detected face area, the The orientation information between the person and the shooting camera; then, combined with the binocular camera, the depth information of the person from the camera is obtained by the method of stereo matching, that is, the position and orientation information of each person in the scene relative to the camera coordinate system can be obtained; further , using two or more microphones on the mobile phone to obtain the position and orientation information of each speaker in the scene relative to the microphone coordinate system; finally, by pre-calibrating the transformation relationship between the camera coordinate system and the microphone coordinate system, you can Get the correspondence between each character and audio on the image.

After obtaining the corresponding relationship between each character and audio on the image, when the user clicks to play one or several interesting characters in the image, other areas in the video image will be blurred or the area of interest will be enlarged. Thereby highlighting the area of people of interest.

The specific implementation and display method of this embodiment consists of four parts: enabling the video shooting mode of highlighting the interesting person, determining the interesting person, storing the image and voice of the interested person, and playing the image and voice of the interested person. composition.

(1) Turn on the video shooting mode of people of interest highlighting

This mode includes two modes, one is that the user actively turns on the video shooting mode of highlighting interesting people, and the other is that the terminal device automatically prompts the user whether to turn on the video shooting mode of highlighting interesting people according to the content of the captured video.

1) The method of user's active opening is mainly through voice, gesture interaction, etc. For example, when the user says "record a person of interest", the terminal device will turn on the video shooting mode of highlighting the person of interest. At this time, the face detection function will be activated, and all the characters in the current shooting scene will be displayed on the video shooting preview interface. The user can click to select A person of interest is recorded, and the recorded person of interest can be changed; or all people detected in the current video are recorded in real time, so that a specific person of interest can be selected for subsequent playback; or when only recording Switch between modes in which one, several or all characters are recorded.

2) The terminal device automatically detects the video content, mainly based on video understanding technology. For example, by analyzing the content in the video and judging that the main scene of the current video is a multi-person conference, speech, etc., the terminal device automatically prompts the user whether it needs to be turned on. Interested people highlight the video shooting mode. After turning on the video shooting mode of highlighting people of interest, it can be determined through the user's gesture or voice interaction whether to record a single person of interest, or to record all the people in the entire scene, or switch between the two methods.

(2) Determine the way of people of interest

The user shoots a video highlighting the person of interest, and the way of specifying the person of interest can be divided into the user's initiative through voice, gesture or interaction with an external device, or the terminal device automatically determines the person of interest.

1) The way that the user actively determines the person of interest includes voice, gesture, and external device interaction.

Voice interaction: The user starts recording through voice interaction. When recording, it can also be determined by voice whether a single character or multiple characters is being recorded, and can switch between the two methods. For example, when the user says "record a single person", the terminal device only records the image and sound of a certain person of interest specified by the user; when the user says "all record", the terminal device will record all the detected characters in the scene People images and voices are recorded.

Gesture interaction: The user can click on a detected character to specify the target person of interest for recording; double-click another character to change the target person of interest; click the screen to specify the recording of the entire scene. All people of interest; you can specify to record multiple people of interest by clicking multiple target people in succession.

External device interaction: The above operations through gesture interaction can be implemented through external device interaction. For example, through devices such as stylus and earphones associated with the terminal device, it is possible to designate a certain target person of interest, or designate multiple target people, or designate all characters in the entire scene as target people.

2) The terminal device automatically determines the person of interest according to the currently photographed scene. When the terminal device turns on the interesting person shooting mode, it detects the person appearing in the image on the image preview interface, and determines the person that the user is interested in according to the number of people appearing and the location of the person. For example, take each character appearing in the scene as a person of interest, and store all the characters in the scene and their corresponding voices; or take a person who is located near the center of the picture as a person of interest, and mark them prominently with It prompts the photographer that the current central character is the person of interest. If the user wants to change the person of interest determined by the current terminal device, he can change the person of interest by double-clicking on the person he is interested in. The image and voice of the person of interest are corresponding to the binocular camera and multiple microphones of the terminal device.

(3) Image and video storage methods for people of interest

Preferably, according to the collected audio information, the collected video information is stored and processed, wherein the stored content includes at least one of the following situations:

1) Collected video information and audio information;

2) Video information and audio information corresponding to the target object.

There are two main storage methods:

The first one: store the content captured by the camera and the microphone in real time, and record the interesting people specified by the user at different times, so as to adapt to various ways during playback.

The second type: Only the images and sounds of the people of interest recorded at the time of shooting are stored. That is, only the image and sound of a person specified by the user, multiple persons, or all persons in the scene are recorded at the time of shooting.

The above two methods are mainly aimed at recording the scene in the camera capture area of the current terminal device. If it is on the other side of the current terminal device, that is, the sound emitted from the back of the camera capture area, it will be stored in another file. At this time, the user can choose whether to remove the sound from the back of the camera shooting area during playback. Through the microphone on the terminal device, it can be detected whether the direction of the sound comes from the front of the camera shooting area or the back of the camera shooting area. If the current sound comes from the back of the camera's shooting area, the sound may not be recorded by the photographer, for example, the sound may be "starting recording", or the content of the current conversation when answering the phone. Therefore, this part of the voice content can be stored separately.

Regarding how to set the storage mode, the present invention provides three setting methods, and the terminal device can select the storage mode according to one of the following three methods. The first is the default setting of the terminal device; the second is that the terminal device accepts the user to change the storage mode through voice, keys or external controllers, and a combination of these methods; Adapt to how the settings are stored.

Regarding the default setting, the terminal device sets one of the two storage modes as the default value, and the terminal device uses this default storage mode to store the video before the terminal device receives an instruction to change the storage mode.

Regarding the setting method of user interaction, the setting method similar to step 4 in the first embodiment is adopted, and the difference lies in the description content of the instruction. For example, the instruction in the voice setting is "interesting video storage". The voice control command is used for voice recognition, and the setting storage mode is determined as the first storage mode. The settings for other user interactions are the same, and will not be repeated here.

Regarding the adaptive setting according to the storage space, different storage methods can be selected according to the storage space. If the remaining storage space is less than a certain threshold, for example, lower than 50% of the storage space of the terminal device, the second storage method is set; otherwise , if the remaining storage space is higher than a certain threshold, for example, higher than 50% of the storage space of the terminal device, the storage method is not affected by the storage space.

Regarding the adaptive setting based on historical data, for example, the user's preference is analyzed according to the storage mode previously set by the user, and the user's preferred storage mode is set.

(4) Image and voice playback method for people of interest

Preferably, in response to the received playback trigger operation, the video information and the audio information are played based on the playback mode that matches the stored content; wherein, the playback mode includes at least one of the following:

1) when storing the collected video information and audio information, the collected video information and audio information are associated and played;

2) when storing the collected video information and audio information, the target object in the collected video information is played in association with the corresponding audio information;

3) when storing the collected video information and audio information, each object in the collected video information is played in association with the corresponding audio information;

4) When storing the video information and audio information corresponding to the target object, play the video information and audio information corresponding to the target object in association.

Corresponding to the above-mentioned storage mode, there are two corresponding playback modes, and the terminal device can choose but not be limited to one of the following two:

The first type: the terminal device detects the user's opening operation, and is for the first type of stored complete video. An operation that the user clicks to play is detected, for example, an operation that the user clicks a play button is detected, and the video is played. The first storage method records the images and sounds of all the characters in the scene, and also records the interesting characters specified by the user in each time period. Then during playback, you can: 1) Play according to the video content specified by the user when recording: For example, if you are interested in character 1 in the first 30 seconds, only the image and sound of the target character 1 will be played, and other characters and background images will be blurred and/or or still; or enlarge the image area of the target person, and other areas are blurred and/or still. If you are interested in person 2 for the next 60 seconds, only the image and sound of the target person 2 will be played. The selection of the person of interest here is determined during recording, and the terminal device records which person or person is interested in which time period the user is interested in; 2) Play all the recorded image and sound content without processing; 3 ) Since the images and sounds of all characters in the scene are recorded, during playback, the user can change the order of the interesting characters to be played. For example, if he is interested in character 2 in the first 30 seconds, only the sound and image of the target character 2 will be played. Others Both people and background images are blurred and/or still. If you are interested in person 1 in the next 60 seconds, only the image and sound of the target person 1 will be played.

The second type: The terminal device detects the user's opening operation, and targets the video of the person of interest stored in the second type. The operation of the user clicking to play is detected, for example, the operation of the user clicking the play button is detected, and the video is played in the order of the people of interest selected during recording, that is, in the same way as the area of interest specified during shooting.

The above two playback methods are mainly aimed at the image and sound content of the front area captured by the camera. For the sound content from the back area captured by the camera mentioned in "Storage Mode", the user can use some specific voice or gesture commands. Tell the end device if playback is required. For example, you can click the background area on the screen to open and play the sound content from the back area captured by the camera, and the playback time is associated with other playback videos according to the time sequence recorded in the file. It is also possible to turn off the playback of the sound content from the back area captured by the camera by double-tapping the background area on the screen.

The specific solution of the embodiment of the interesting person highlighting video is described in detail below with an embodiment.

Step 1. Open interactive steps:

1). Turn on the camera of the terminal device, enter the video shooting preview interface, and enable the video shooting mode of people of interest highlighting by long-pressing the screen.

2). After the interesting person highlighting video mode is turned on, the preview interface displays the image captured by the left camera, which occupies the entire screen. At this point, the face detection function is activated, and all the characters in the current shooting scene are displayed on the video shooting preview interface, and all the characters detected in the current video are recorded in real time, and the specific interesting characters are selected to be played for subsequent playback. .

3). When a person in the scene is detected, the camera on one side will be activated, and the depth and direction information of the person detected in the scene will be calculated with the camera on the other side, and then the video of the person of interest will be highlighted.

Step 2. The way of shooting, storing and playing the video of people of interest highlighting is as follows:

Step 2.1 Interested people highlight video shooting

1). In the video shooting interface of people of interest highlighting, when the user sees that the detected face area in the screen is green, it indicates that the angle and position information of the characters in the scene have been estimated at this time, and the user presses the shooting button to start the video Shooting, different storage and playback modes can be performed through different operation methods. Click the shooting button in the interface to jump to 2), which will directly record the screen shot by the current terminal device; long-press a target person in the screen to jump to 5), and store the highlighted video of the person of interest and the corresponding time Click on the corresponding person of interest. At the same time, real-time detection of the sound emitted from the back of the current terminal device camera shooting area, if a sound is detected from the back, skip to 7), and store the video in the camera collection area and the audio collected on the back of the camera respectively.

2). Directly record the image captured on the current screen, and the image captured by the camera on one side occupies the entire screen of the terminal device. Real-time display of the character area in the current scene. If a person of interest is not specified during the shooting process, the person of interest can be selected in the playback stage, and jump to 5) to play the picture and audio of the specified person of interest.

Step 2.2 Interested people highlight video storage

3). What is displayed on the current screen is the highlighted video of the person of interest, which is stored in the terminal device. Before shooting the next video highlighting people of interest, the image frame in the upper left corner displays the highlight video of people of interest shot recently. You can click on the image frame to view the highlight video of people of interest recently shot.

Step 2.3 Interested people highlight video playback

4). After clicking the image box in the upper left corner, the most recent shooting video of people of interest will be displayed on the terminal device, and the playback method at this time is the same as the content seen when shooting. If you click on a character in the currently playing video, jump to step 5 to play.

5). If one or several people of interest are not specified in the shooting stage, the user can click the area of the person of interest during playback, and only the image and audio corresponding to the character will be played, and all other areas will be displayed. Still and/or blurry. If a certain period of interesting people is specified during the shooting stage, the user-specified duration and the order of the interesting people are recorded, and the playback will be played according to the order and duration of the interesting people specified at the time of shooting.

6). Press and hold the content in the upper left image box on the screen to play the most recent shooting video highlighting the person of interest.

7). The highlighted video of the person of interest and the audio from the area on the back of the camera are stored separately, and the terminal device can play two parts of content respectively when playing. As shown in Figure 16(a)-(c), if you are not interested in the audio from the back area of the camera, you can delete the audio content directly; if the user wants to keep the audio from the back area of the camera, you can follow the Time series, associate the corresponding audio and video to play together.

The present invention also provides an apparatus for multimedia enhancement processing. As shown in FIG. 17 , the apparatus includes: a multimedia information acquisition module 1701 and a processing module 1702 .

The multimedia information acquisition module 1701 acquires the first type of multimedia information and the second type of multimedia information respectively collected by the two multimedia collection devices; the processing module 1702 performs corresponding processing on the second type of multimedia information according to the first type of multimedia information.

In the solution of the present invention, for the specific function realization of each module in the provided multimedia information processing device, reference may be made to the specific steps of the multimedia information processing method provided in FIG. 1 , which will not be described in detail here.

As will be appreciated by those skilled in the art, the present invention includes apparatus for performing one or more of the operations described in this application. These devices may be specially designed and manufactured for the required purposes, or they may include those known in general purpose computers. These devices have computer programs stored in them that are selectively activated or reconfigured. Such a computer program may be stored in a device (eg, computer) readable medium including, but not limited to, any type of medium suitable for storing electronic instructions and coupled to a bus, respectively Types of disks (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), ROM (Read-Only Memory, read-only memory), RAM (Random Access Memory, random access memory), EPROM (Erasable Programmable Read-Only Memory, available Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory, Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, a readable medium includes any medium that stores or transmits information in a form that can be read by a device (e.g., a computer).

Those skilled in the art will understand that computer program instructions can be used to implement each block of these structural diagrams and/or block diagrams and/or flow diagrams, and combinations of blocks in these structural diagrams and/or block diagrams and/or flow diagrams . Those skilled in the art can understand that these computer program instructions can be provided to a general-purpose computer, a professional computer or a processor of other programmable data processing methods to implement, so that the present invention can be executed by a processor of a computer or other programmable data processing method. The block or blocks specified in the block or blocks of the block diagrams and/or block diagrams and/or flow diagrams of the invention are disclosed.

Those skilled in the art can understand that the various operations, methods, steps, measures, and solutions discussed in the present invention may be alternated, modified, combined, or deleted. Further, other steps, measures, and solutions in the various operations, methods, and processes that have been discussed in the present invention may also be alternated, modified, rearranged, decomposed, combined, or deleted. Further, steps, measures and solutions in the prior art with various operations, methods, and processes disclosed in the present invention may also be alternated, modified, rearranged, decomposed, combined or deleted.

The above are only the preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, without departing from the principles of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be It is regarded as the protection scope of the present invention.

Claims (34)

1. A method for video information processing performed by an electronic device, comprising: obtaining first video information corresponding to the first focus area by the first multimedia collection device; obtaining, by the second multimedia collection device, second video information corresponding to the second focus area; and The first video information and the second video information are displayed on a display of the electronic device. 2. The method of claim 1, wherein the second focus area is located within the first focus area, and Wherein, the second focus area is smaller than the first focus area. 3. The method of claim 2, wherein the first focus area is a global area; the second focus area is a local area. 4. The method of claim 1, wherein displaying the first video information and the second video information on a display of the electronic device comprises: The first video information and the second video information are displayed in a split screen layout on a display of the electronic device. 5. The method of claim 1, further comprising: acquiring third video information corresponding to a third focus area within the first focus area; and At least one of the first video information and the second video information and the third video information are displayed on a display of the electronic device. 6. The method according to claim 5, wherein displaying at least one of the first video information and the second video information and the third video information on a display of the electronic device comprises: At least one of the first video information and the second video information and the third video information are displayed in a split screen layout on a display of the electronic device. 7. The method of claim 1, further comprising receiving a first instruction to enter a multi-video mode prior to obtaining the first video information and the second video information. 8. The method of claim 1, further comprising receiving a second instruction to set the second focus area. 9. The method of claim 1, further comprising storing the fourth video information by synthesizing the first video information and the second video information. 10. The method of claim 9, further comprising: receiving a fourth instruction to play fourth video information; and The fourth video information is played in response to the received fourth instruction. 11. The method of claim 10, further comprising displaying a user interface for receiving an instruction to play the fourth video information. 12. The method of claim 9, further comprising receiving a third instruction to store fourth video information. 13. The method of claim 9, wherein when the orientation of the electronic device is a horizontal orientation, the fourth video information is stored in a split screen layout including a left screen and a right screen. 14. The method of claim 9, wherein when the orientation of the electronic device is a vertical orientation, the fourth video information is stored in a split screen layout including an upper screen and a lower screen. 15. The method of claim 4, further comprising detecting the orientation of the electronic device by a sensor. 16. The method of claim 15, wherein when the detected orientation of the electronic device is a horizontal orientation, the split screen layout includes a left screen and a right screen, and Wherein, when the detected direction of the electronic device is a vertical direction, the split-screen layout includes an upper screen and a lower screen. 17. The method of claim 1, wherein the second focus area is determined based on a center of the first focus area. 18. An electronic device comprising: at least two multimedia collection devices, including a first multimedia collection device and a second multimedia collection device; monitor; A multimedia information acquisition module, configured as: Obtain first video information corresponding to the first focus area through the first multimedia collection device; Obtain second video information corresponding to the second focus area by the second multimedia capture device; and The processing module is configured to control the display of the first video information and the second video information on the display. 19. The electronic device of claim 18, wherein the second focus area is located within the first focus area, and Wherein, the second focus area is smaller than the first focus area. 20. The electronic device of claim 19, wherein the first focus area is a global area; the second focus area is a local area. 21. The electronic device of claim 18, wherein the processing module is further configured to control the display of the first video information and the second video information in a split screen layout on the display. 22. The electronic device of claim 18, wherein the multimedia information acquisition module is further configured to acquire third video information corresponding to a third focus area within the first focus area, and Wherein, the processing module is further configured to control to display at least one of the first video information and the second video information and the third video information on the display. 23. The electronic device of claim 22, wherein the processing module is further configured to control to display at least one of the first video information and the second video information and the third video information in a split screen layout on the display. 24. The electronic device of claim 18, wherein the processing module is further configured to receive a first instruction to enter a multi-video mode before obtaining the first video information and the second video information. 25. The electronic device of claim 18, wherein the processing module is further configured to receive a second instruction to set a second focus area. 26. The electronic device of claim 18, wherein the processing module is further configured to store fourth video information by synthesizing the first video information and the second video information. 27. The electronic device of claim 26, wherein the processing module is further configured to: receiving a fourth instruction to play fourth video information; and Controlling to play the fourth video information on the display in response to the received fourth instruction. 28. The electronic device of claim 27, wherein the processing module is further configured to control displaying on the display a user interface for receiving an instruction to play the fourth video information. 29. The electronic device of claim 26, wherein the processing module is further configured to receive a third instruction to store fourth video information. 30. The electronic device of claim 26, wherein, when the orientation of the electronic device is a horizontal direction, the processing module is further configured to store the fourth video information in a split screen layout including a left screen and a right screen. 31. The electronic device of claim 26, wherein when the orientation of the electronic device is a vertical direction, the processing module is further configured to store the fourth video information in a split screen layout including an upper screen and a lower screen. 32. The electronic device of claim 21, further comprising a sensor configured to detect the orientation of the electronic device. 33. The electronic device of claim 32, wherein when the detected orientation of the electronic device is a horizontal orientation, the split screen layout includes a left screen and a right screen, and Wherein, when the detected direction of the electronic device is a vertical direction, the split-screen layout includes an upper screen and a lower screen. 34. The electronic device of claim 18, wherein the multimedia information acquisition module is further configured to determine the second focus area based on the center of the first focus area.

Images