CN112926461B - Neural network training, driving control method and device - Google Patents

Abstract

The disclosure provides a neural network training and driving control method, a device, an electronic device and a storage medium, wherein the method comprises the following steps: obtaining a training sample; determining a sub-network structure in a target neural network matched with each pixel point in a two-dimensional labeling frame based on the scale of the two-dimensional labeling frame of the object to be detected in the training sample; the method comprises the steps that different sub-network structures are used for extracting characteristics of pixel points in two-dimensional labeling frames with different scales in a training sample; determining three-dimensional labeling data of an object to be detected, which corresponds to at least one pixel point in a two-dimensional labeling frame in a training sample; and training each sub-network structure in the target neural network based on at least one pixel point which corresponds to each sub-network structure and has three-dimensional labeling data, so as to obtain the trained target neural network.

Description

Neural network training, driving control method and device

Technical Field

The present disclosure relates to the field of deep learning technology, and more specifically, to a neural network training, driving control method, device, electronic device and storage medium.

Background technique

Object detection is one of the core tasks of computer vision and is widely used in fields such as autonomous driving and mobile robots. The main task of object detection is to locate the objects of interest from the image and determine the category and bounding box of each object. Since it is difficult to determine the depth information of the object in a three-dimensional scene, the application effect of object detection in a three-dimensional scene is not good. For example, in the field of autonomous driving, in order to achieve smooth and safe driving of autonomous vehicles on the road, the autonomous vehicle must determine the accurate three-dimensional information of each object around it.

Therefore, it is increasingly important to propose a method that can more accurately determine the three-dimensional detection results of the target.

Summary of the invention

In view of this, the present disclosure at least provides a neural network training, driving control method, device, electronic device and storage medium.

In a first aspect, the present disclosure provides a neural network training method, comprising:

Get training samples;

Based on the scale of the two-dimensional annotation box of the object to be detected in the training sample, determine the sub-network structure in the target neural network that matches each pixel point in the two-dimensional annotation box; wherein different sub-network structures are used to extract features of the pixel points in the two-dimensional annotation box of different scales in the training sample;

Determine three-dimensional annotation data of the object to be detected corresponding to at least one pixel point within the two-dimensional annotation box in the training sample;

Based on at least one pixel point having the three-dimensional annotation data corresponding to each of the sub-network structures, each of the sub-network structures in the target neural network is trained to obtain a trained target neural network.

In the above method, by determining the sub-network structure in the target neural network that matches each pixel point in the two-dimensional annotation box, the corresponding sub-network structure is trained using the pixel points in the two-dimensional annotation box of different scales, so that the different sub-network structures after training can perform feature extraction and three-dimensional target detection on target objects of different scales in the image to be detected. Since multiple different sub-network structures can perform relatively intensive multi-level feature extraction and prediction, after using the training samples to train each sub-network structure of the target neural network, a target neural network with better performance for three-dimensional target detection can be obtained.

In a possible implementation, determining the subnetwork structure matching each pixel point in the two-dimensional annotation box based on the scale of the two-dimensional annotation box of the object to be detected in the training sample includes:

For each pixel point in the two-dimensional annotation box, based on the scale of the two-dimensional annotation box, determine a first longest distance between the pixel point and each edge of the two-dimensional annotation box;

Based on the longest first distance and a preset distance range corresponding to each sub-network structure, a sub-network structure matching the pixel point is determined.

By adopting the above method, for each pixel point in the two-dimensional standard frame, based on the scale of the two-dimensional annotation frame, the longest first distance between the pixel point and each edge of the two-dimensional annotation frame can be determined, and the matching sub-network structure can be determined for the pixel point using the longest first distance. Pixel points with different longest first distances are matched with different sub-network structures to achieve pixel-level matching, so that after the sub-network structure is trained using multiple pixel points corresponding to the sub-network structure, the trained different sub-network structures can perform feature extraction and three-dimensional target detection on target objects of different scales in the image to be detected.

In a possible implementation manner, determining the three-dimensional annotation data of the object to be detected corresponding to at least one pixel point in the two-dimensional annotation box in the training sample includes:

Determining target pixel points in the two-dimensional annotation box in the training sample that belong to foreground points based on the sub-network structures that are respectively matched with the pixel points in the two-dimensional annotation box;

Determine the three-dimensional annotation data of the object to be detected corresponding to the target pixel point.

In a possible implementation, determining the target pixel points belonging to the foreground point in the two-dimensional annotation box in the training sample based on the sub-network structure respectively matched with the pixel points in the two-dimensional annotation box includes:

For each pixel point in the two-dimensional annotation frame, determining a second distance between the pixel point and a center point of the object to be detected in the two-dimensional annotation frame;

Determining a distance threshold corresponding to the pixel point based on a stride of the subnetwork structure matched with the pixel point and a preset radius parameter;

When the second distance is smaller than the distance threshold, the pixel point is determined to be the target pixel point.

Considering that when the distance between a pixel point and the center point of the object to be detected is close, the probability that the pixel point belongs to a foreground point is high, and the feature information of the pixel point is relatively reliable. Therefore, here, the distance threshold corresponding to the pixel point can be determined based on the stride of the subnetwork structure matching the pixel point and the preset radius parameter; according to the second distance between the pixel point and the center point of the object to be detected in the two-dimensional annotation frame and the determined distance threshold, it is determined whether the pixel point belongs to the foreground point, thereby more accurately determining the foreground point in the two-dimensional inspection frame.

In a possible implementation manner, the three-dimensional annotation data includes at least one of the following data:

The offset used to characterize the deviation between the pixel point and the corresponding center point of the object to be detected, the depth of the center point of the object to be detected corresponding to the pixel point, the size of the three-dimensional detection frame, the orientation of the three-dimensional detection frame, the orientation category, the speed of the object to be detected, the centrality used to characterize the degree of proximity between the pixel point and the corresponding center point of the object to be detected, the target category of the object to be detected, and the attribute category characterizing the state of the object to be detected.

Here, the types of data included in the three-dimensional annotation data are relatively rich and diverse.

In a possible implementation, before determining the sub-network structure in the target neural network matched to each pixel point in the two-dimensional annotation box based on the scale of the two-dimensional annotation box of the object to be detected in the training sample, the method further includes:

When there is a same pixel point in the training sample that is located in multiple two-dimensional annotation boxes, determining a third distance between the pixel point and a center point of the object to be detected in each of the two-dimensional annotation boxes;

The two-dimensional annotation box corresponding to the smallest third distance is used as the two-dimensional annotation box corresponding to the pixel point.

By adopting the above method, when the sub-network structure is trained using training samples, one pixel point can only correspond to one three-dimensional annotation data. Therefore, when the same pixel point is located in multiple two-dimensional annotation boxes, the two-dimensional annotation box corresponding to the same pixel point can be determined more accurately based on the third distance between the same pixel point and the center point of the object to be detected in each two-dimensional annotation box, and the three-dimensional annotation data of the same pixel point can be further determined more accurately.

In a possible implementation, each sub-network structure in the target neural network corresponds to a regression index trained using the sample data, and the regression index is used to perform a multiple adjustment on the predicted regression data in the three-dimensional predicted data output by the detection head network connected to the sub-network structure;

Among them, the detection head network includes a classification network and a regression network. The detection head network is used to determine the three-dimensional prediction data corresponding to the sub-network structure based on the feature map output by each sub-network structure. The three-dimensional prediction data includes the predicted category data output by the classification network and the predicted regression data output by the regression network.

Taking into account that the detection head network is shared by multiple sub-network structures, that is, multiple sub-network structures are connected to the same detection head network, and since different sub-network structures are used to perform feature extraction and three-dimensional target prediction on objects to be detected of different scales, a regression indicator can be trained for each sub-network structure here, and the regression indicator is used to multiple-adjust the predicted regression data in the three-dimensional prediction data output by the detection head network connected to the sub-network structure, so that the adjusted predicted regression data matches the sub-network structure.

In a possible implementation, after obtaining the trained target neural network, the method further includes:

Acquire the image to be detected;

The trained target neural network is used to detect the image to be detected, and a three-dimensional detection result of at least one target object included in the image to be detected is determined.

By adopting the above method, a trained target neural network with good performance can be used to detect the image to be detected, and a relatively accurate three-dimensional detection result of at least one target object can be obtained.

In a possible implementation manner, detecting the image to be detected by using the target neural network to determine a three-dimensional detection result of at least one target object included in the image to be detected includes:

Detecting the image to be detected using the target neural network to generate three-dimensional detection data corresponding to a plurality of pixel points in the image to be detected;

Determine, based on the three-dimensional detection data respectively corresponding to the plurality of pixel points, a plurality of candidate three-dimensional detection frame information included in the image to be detected;

Based on the information of the multiple candidate three-dimensional detection frames, determining projection frame information of the multiple candidate three-dimensional detection frames in the bird's-eye view corresponding to the image to be detected;

Based on the plurality of projection frame information, a three-dimensional detection result of at least one target object included in the image to be detected is determined.

By adopting the above method, by determining the projection frame information of multiple candidate three-dimensional detection frames in the bird's-eye view corresponding to the image to be detected, based on the multiple projection frame information, the three-dimensional detection result of at least one target object included in the image to be detected is more accurately determined.

In a possible implementation manner, determining a three-dimensional detection result of at least one target object included in the image to be detected based on the plurality of projection frame information includes:

Based on the confidence and center degree corresponding to the target category indicated by each candidate 3D detection frame information, determine the target confidence degree of the projection frame corresponding to the candidate 3D detection frame; wherein the center degree is used to characterize the degree of proximity between the pixel point corresponding to the candidate 3D detection frame and the center point of the corresponding object to be detected;

Based on the target confidences respectively corresponding to the projection frames, a three-dimensional detection result of at least one target object included in the image to be detected is determined.

Taking into account that the center degree is used to characterize the degree of proximity between the pixel point corresponding to the candidate three-dimensional detection frame and the corresponding center point of the object to be detected, the larger the center degree, the closer the pixel point is to the center of the object to be detected, the higher the reliability of the feature information of the pixel point, and the higher the reliability of the generated three-dimensional detection frame information corresponding to the pixel point; therefore, based on the confidence and center degree corresponding to the target category indicated by each candidate three-dimensional detection frame information, the target confidence of the projection frame corresponding to the candidate three-dimensional detection frame can be determined; and then the target confidence can be used to more accurately determine the three-dimensional detection result of at least one target object included in the image to be detected.

In a second aspect, the present disclosure provides a driving control method, comprising:

Acquire road images collected by the driving device during driving;

Using a target neural network trained by the neural network training method according to any one of the first aspects, detecting the road image to obtain target three-dimensional position data of a target object included in the road image;

The travel device is controlled based on target three-dimensional position data of the target object included in the road image.

The effects of the following devices, electronic devices, etc. are described in the description of the above method and will not be repeated here.

In a third aspect, the present disclosure provides a neural network training device, comprising:

A first acquisition module, used to acquire training samples;

A first determination module is used to determine the sub-network structure in the target neural network that matches each pixel point in the two-dimensional annotation box based on the scale of the two-dimensional annotation box of the object to be detected in the training sample; wherein different sub-network structures are used to extract features of pixel points in the two-dimensional annotation box of different scales in the training sample;

A second determination module is used to determine the three-dimensional annotation data of the object to be detected corresponding to at least one pixel point in the two-dimensional annotation box in the training sample;

A training module is used to train each of the sub-network structures in the target neural network based on at least one pixel point with the three-dimensional annotation data corresponding to each of the sub-network structures, so as to obtain a trained target neural network.

In a fourth aspect, the present disclosure provides a driving control device, comprising:

A second acquisition module is used to acquire road images collected by the driving device during driving;

A detection module, configured to detect the road image using a target neural network trained by the neural network training method according to any one of the first aspects, and obtain target three-dimensional position data of a target object included in the road image;

A control module is used to control the driving device based on the target three-dimensional position data of the target object included in the road image.

In a fifth aspect, the present disclosure provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the neural network training method described in the first aspect or any one of the embodiments are executed; or the steps of the driving control method described in the second aspect are executed.

In a sixth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program executes the steps of the neural network training method as described in the first aspect or any one of the embodiments above; or executes the steps of the driving control method as described in the second aspect above.

In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

FIG1 is a schematic diagram showing a flow chart of a neural network training method provided by an embodiment of the present disclosure;

FIG2 shows a schematic diagram of a two-dimensional annotation box of a training sample in a neural network training method provided by an embodiment of the present disclosure;

FIG3 shows a schematic diagram of a training sample including multiple two-dimensional annotation boxes in a neural network training method provided by an embodiment of the present disclosure;

FIG4a is a schematic diagram showing the orientation categories of an object to be detected in a neural network training method provided by an embodiment of the present disclosure;

FIG4b is a schematic diagram showing the orientation categories of the objects to be detected in a neural network training method provided by an embodiment of the present disclosure;

FIG5 shows a schematic diagram of the architecture of a target neural network in a neural network training method provided by an embodiment of the present disclosure;

FIG6 shows a schematic flow chart of a driving control method provided by an embodiment of the present disclosure;

FIG7 shows a schematic diagram of the architecture of a neural network training device provided by an embodiment of the present disclosure;

FIG8 shows a schematic diagram of the architecture of a driving control device provided by an embodiment of the present disclosure;

FIG9 shows a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure;

FIG. 10 shows a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

Object detection is one of the core tasks of computer vision and is widely used in fields such as autonomous driving and mobile robots. The main task of object detection is to locate the objects of interest from the image and determine the category and bounding box of each object. Since it is difficult to determine the depth information of the object in a three-dimensional scene, the application effect of object detection in a three-dimensional scene is not good. For example, in the field of autonomous driving, in order to achieve smooth and safe driving of autonomous vehicles on the road, the autonomous vehicle must determine the accurate three-dimensional information of each object around it.

In order to solve the above problems, the embodiments of the present disclosure provide a neural network training, driving control method, device, electronic device and storage medium.

The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not require further definition and explanation in the subsequent drawings.

To facilitate understanding of the embodiments of the present disclosure, a neural network training method and a driving control method disclosed in the embodiments of the present disclosure are first introduced in detail. The execution subject of the neural network training method and the driving control method provided in the embodiments of the present disclosure is generally a computer device with a certain computing capability, and the computer device includes, for example: a terminal device or a server or other processing device, and the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the neural network training method and the driving control method can be implemented by a processor calling a computer-readable instruction stored in a memory.

Referring to FIG. 1 , there is shown a flowchart of a neural network training method provided by an embodiment of the present disclosure, including S101-S104, wherein:

S101, obtaining training samples.

S102, based on the scale of the two-dimensional annotation box of the object to be detected in the training sample, determine the sub-network structure in the target neural network that matches each pixel point in the two-dimensional annotation box; wherein different sub-network structures are used to extract features of pixel points in two-dimensional annotation boxes of different scales in the training sample.

S103, determining three-dimensional annotation data of the object to be detected corresponding to at least one pixel point in the two-dimensional annotation box in the training sample.

S104, based on at least one pixel point with three-dimensional annotation data corresponding to each sub-network structure, train each sub-network structure in the target neural network to obtain a trained target neural network.

In the above method, by determining the sub-network structure in the target neural network that matches each pixel point in the two-dimensional annotation box, the corresponding sub-network structure is trained using the pixel points in the two-dimensional annotation box of different scales, so that the different sub-network structures after training can be used for feature extraction and three-dimensional target detection of target objects of different scales in the image to be detected. Since multiple different sub-network structures can perform relatively intensive multi-level feature extraction and prediction, after using the training samples to train each sub-network structure of the target neural network, a target neural network with better performance for three-dimensional target detection is obtained.

S101 to S104 are described in detail below.

For S101:

Acquire training samples, which include sample images corresponding to a plurality of objects to be detected. Exemplary. When the method is applied in the field of autonomous driving, the images to be detected may include motor vehicles, non-motor vehicles, pedestrians, animals, etc.

Each object to be detected included in the training sample corresponds to a two-dimensional annotation box, which can be manually annotated or automatically annotated using a trained two-dimensional detection neural network. Each two-dimensional annotation box corresponds to a size, for example, the size of the two-dimensional annotation box can be 56×512, 512×218, 1024×218, 1024×1024, etc.

For S102:

The target neural network includes a pyramid network structure, which includes a plurality of different sub-network structures. The different sub-network structures are used to extract features of pixel points in two-dimensional annotation boxes of different scales in training samples.

Exemplarily, the subnetwork structure that matches each pixel in the two-dimensional annotation box can be determined based on the maximum length of the scale indication of the two-dimensional annotation box in the training sample. For example, a scale range can be set for each subnetwork structure, and the scale range corresponding to subnetwork structure one is (218-512], and the scale range corresponding to subnetwork structure two is (512-1024], if the scale of two-dimensional annotation box one is 512×218, the maximum length of the scale indication of the two-dimensional annotation box is 512, so each pixel in two-dimensional annotation box one matches subnetwork structure one.

In another optional implementation, based on the scale of the two-dimensional annotation box of the object to be detected in the training sample, determining the subnetwork structure matching each pixel point in the two-dimensional annotation box includes:

S1021, for each pixel point in the two-dimensional annotation box, based on the scale of the two-dimensional annotation box, determining a first longest distance between the pixel point and each edge of the two-dimensional annotation box;

S1022: Determine a sub-network structure that matches the pixel point based on the longest first distance and a preset distance range corresponding to each sub-network structure.

By adopting the above method, for each pixel point in the two-dimensional standard frame, based on the scale of the two-dimensional annotation frame, the longest first distance between the pixel point and each edge of the two-dimensional annotation frame can be determined, and the matching sub-network structure can be determined for the pixel point using the longest first distance. Pixel points with different longest first distances are matched with different sub-network structures to achieve pixel-level matching, so that after the sub-network structure is trained using multiple pixel points corresponding to the sub-network structure, the trained different sub-network structures can perform feature extraction and three-dimensional target detection on target objects of different scales in the image to be detected.

In S1021, for each pixel point in the two-dimensional annotation frame, based on the scale of the two-dimensional annotation frame and the position information of the pixel point, the first distance between the pixel point and each edge of the two-dimensional annotation frame can be determined, that is, the first distances between the pixel point and the four edges can be obtained. Referring to the schematic diagram of a two-dimensional annotation frame shown in FIG2 , the first distances between the pixel point 21 and the four edges in the figure are t, b, l, and r, respectively. It can be seen that l has the largest value, so the first distance l is the longest first distance.

In S1022, a corresponding distance range can be set for each sub-network structure. For example, the distance range corresponding to sub-network structure one is (0-256], the distance range corresponding to sub-network structure two is (256-512], and the distance range corresponding to sub-network structure three is (512-1024]. When the longest first distance corresponding to pixel point A is 500, it is determined that pixel point A matches sub-network structure two; when the longest first distance corresponding to pixel point B is 218, it is determined that pixel point B matches sub-network structure one.

Through S1021 and S1022, the sub-network structure matching each pixel point in the two-dimensional annotation box can be obtained, that is, multiple pixel points corresponding to each sub-network structure are obtained.

In an optional implementation, before determining the sub-network structure in the target neural network that matches each pixel point in the two-dimensional annotation box based on the scale of the two-dimensional annotation box of the object to be detected in the training sample, the method further includes:

Step 1: when the same pixel point in the training sample is located in multiple two-dimensional annotation frames, determine the third distance between the pixel point and the center point of the object to be detected in each two-dimensional annotation frame;

Step 2: Use the two-dimensional annotation box corresponding to the smallest third distance as the two-dimensional annotation box corresponding to the pixel point.

By adopting the above method, when the sub-network structure is trained using the training samples, one pixel point can only correspond to one three-dimensional annotation data. Therefore, when the same pixel point is located in multiple two-dimensional annotation boxes, the two-dimensional annotation box corresponding to the same pixel point can be determined more accurately based on the third distance between the same pixel point and the center point of the object to be detected in each two-dimensional annotation box, and the three-dimensional annotation data of the same pixel point can be further determined more accurately.

There are multiple objects to be detected in the training sample, each object to be detected corresponds to a two-dimensional annotation box, and there is overlap between the two-dimensional annotation boxes. The pixel points in the overlapping area are located in multiple two-dimensional annotation boxes, so it is necessary to determine a corresponding two-dimensional annotation box for the pixel point from multiple two-dimensional annotation boxes.

In a specific implementation, the third distance between the pixel point and the center point of the object to be detected in each two-dimensional annotation frame can be determined, that is, the third distance between the pixel point and the center point of the detected object can be determined. For example, the third distance between the pixel point and the center point of the detected object can be determined by a Euclidean distance determination method.

The center point of the object to be detected can be a projection point determined after the object to be detected is projected from the center of the real scene to the training sample. Exemplarily, the coordinates of the center point of the object to be detected can be determined using the intrinsic parameter matrix of the acquisition device corresponding to the training sample.

As shown in Figure 3, the figure includes a two-dimensional annotation box 31 corresponding to the object to be detected, a center point 311 of the object to be detected located in the two-dimensional annotation box 31, and a two-dimensional annotation box 32 corresponding to the object to be detected, and a center point 321 of the object to be detected located in the two-dimensional annotation box 32. For the pixel point 33 located in the overlapping area of the two-dimensional annotation box 31 and the two-dimensional annotation box 32, the third distance between the pixel point 33 and the center point 311 of the object to be detected can be calculated, and the third distance between the pixel point 33 and the center point 321 of the object to be detected can be calculated. It can be seen that the third distance between the pixel point 33 and the center point 311 of the object to be detected is small, so the pixel point 33 belongs to the two-dimensional annotation box 31.

For S103:

The three-dimensional annotation data of the object to be detected corresponding to at least one pixel point in the two-dimensional annotation box in the training sample can be determined. The three-dimensional annotation data includes at least one of the following data: offset, depth of the center point of the object to be detected corresponding to the pixel point, size of the three-dimensional detection box, orientation of the three-dimensional detection box, orientation category, speed of the object to be detected, center degree, target category of the object to be detected, and attribute category. Here, the data types included in the three-dimensional annotation data are relatively rich and diverse.

Here, the object to be detected corresponding to the pixel point refers to the object to be detected to which the pixel point belongs. For example, if the first pixel point is located on the first object to be detected, then the object to be detected corresponding to the first pixel point is the first object to be detected.

The offset can be used to characterize the deviation between the pixel point and the corresponding center point of the object to be detected, and the offset includes a horizontal axis offset Δx and a vertical axis offset Δy. The horizontal axis offset is used to characterize the deviation between the horizontal axis coordinate value of the pixel point and the horizontal axis coordinate value indicated by the center point of the object to be detected, and the vertical axis offset is used to characterize the deviation between the vertical axis coordinate value of the pixel point and the vertical axis coordinate value indicated by the center point of the object to be detected. The depth in the three-dimensional annotation data is the depth of the center point of the object to be detected corresponding to the pixel point. For example, the depth indicated in the three-dimensional annotation data of the pixel point 33 in FIG3 is the depth of the center point 311 of the object to be detected in the real scene.

The size of the 3D detection frame is the size information of the 3D detection frame of the object to be detected corresponding to the pixel point. For example, the size of the 3D detection frame indicated by the 3D annotation data corresponding to the pixel point 33 in FIG3 is the size of the 3D detection frame corresponding to the object to be detected.

The orientation of the three-dimensional detection frame is an angle between 0 and π (180°), and the orientation category may include a forward category and a reverse category, or the orientation category may include a first category and a second category. In specific implementation, the orientation of the three-dimensional detection frame and the orientation category can be used to more accurately characterize the direction of the object to be detected.

Referring to a schematic diagram of the orientation category of an object to be detected shown in FIG4a, two orientation categories of the object to be detected are shown in the figure, namely, the first category (or positive category) shown on the left side of FIG4a and the second category (reverse category) shown on the right side of FIG4a. After obtaining the training sample, the orientation category of the object to be detected in the training sample and the orientation (i.e., angle) under the orientation category can be determined.

For example, for the object to be detected on the left side of FIG4b , the orientation of the object to be detected can be determined to be θ ₁ and the orientation category can be determined to be the first category (or the positive category); for the object to be detected on the right side of FIG4b , the orientation of the object to be detected can be determined to be θ ₂ and the orientation category can be determined to be the second category (or the negative category).

The speed of the object to be detected may include a lateral speed v _x and a longitudinal speed v _y , that is, the lateral speed is used to characterize the traveling speed of the object to be detected in the horizontal axis direction, and the longitudinal speed is used to characterize the traveling speed of the object to be detected in the longitudinal axis direction.

The centrality is used to characterize the closeness between a pixel point and the corresponding central point of the object to be detected. The centrality can be determined according to the following formula (1):

Wherein, Δx is the horizontal axis offset, Δy is the vertical axis offset, and α is a parameter set to adjust the intensity attenuation from the center point to be detected to the periphery of the object to be detected.

The target categories of the objects to be detected include the categories of motor vehicles, non-motor vehicles, pedestrians, animals, etc. The attribute categories of the objects to be detected are used to characterize the state of the objects to be detected. For example, the attribute categories of the objects to be detected may include: moving, pausing (characterizing that the objects to be detected are stationary for a short time), stopping (characterizing that the objects to be detected are stationary for a long time), riding a bicycle, walking, pedestrians standing, pedestrians lying flat, pedestrians sitting, etc.

In S103, determining the three-dimensional annotation data of the object to be detected corresponding to at least one pixel point in the two-dimensional annotation box in the training sample includes:

S1031, determining target pixel points belonging to foreground points in the two-dimensional annotation box in the training sample based on the sub-network structures that are matched with the pixel points in the two-dimensional annotation box respectively;

S1032, determining the three-dimensional annotation data of the object to be detected corresponding to the target pixel point.

Here, we can first determine the target pixel points in the two-dimensional annotation box of the training sample that belong to the foreground point based on the sub-network structure that matches the pixel points in the two-dimensional annotation box respectively; and determine the other pixel points in the two-dimensional annotation box except the target pixel points as background points. Then, we determine the three-dimensional annotation data of the target pixel points that belong to the foreground point, and other pixel points that belong to the background point can be regarded as having no corresponding three-dimensional annotation data.

In a possible implementation, in S1031, based on the sub-network structures respectively matching the pixels in the two-dimensional annotation box, determining the target pixel points in the two-dimensional annotation box in the training sample that belong to the foreground point includes:

Step 1: for each pixel point in the two-dimensional annotation frame, determine a second distance between the pixel point and the center point of the object to be detected in the two-dimensional annotation frame;

Step 2: Determine the distance threshold corresponding to the pixel point based on the stride of the subnetwork structure matching the pixel point and the preset radius parameter;

Step three, when the second distance is less than the distance threshold, determine the pixel point as the target pixel point.

Considering that when the distance between a pixel point and the center point of the object to be detected is close, the probability that the pixel point belongs to the foreground point is high, and the feature information of the pixel point is relatively reliable. Therefore, here, the distance threshold corresponding to the pixel point can be determined based on the stride of the subnetwork structure matching the pixel point and the preset radius parameter; according to the second distance between the pixel point and the center point of the object to be detected in the two-dimensional annotation frame and the determined distance threshold, it is determined whether the pixel point belongs to the foreground point, and the foreground point in the two-dimensional inspection frame is determined more accurately.

The stride of the sub-network structure is preset, and different sub-network structures correspond to different strides. The radius parameter is a preset parameter used to determine the foreground point in the two-dimensional annotation box. For example, the radius parameter can be the length corresponding to 1.5 pixels, or it can be 0.5cm, etc. The radius parameter is for each sub-network structure, that is, each sub-network structure corresponds to a radius parameter. Since different sub-network structures correspond to different strides, the distance threshold corresponding to each pixel point in the two-dimensional annotation box in the training sample can be determined based on the stride and radius parameters.

Since the foreground point is a pixel point that is closer to the center point of the object to be detected, and the background point is a pixel point that is farther from the center point of the object to be detected. Therefore, after determining the second distance of each pixel point in the two-dimensional annotation box in the training sample, it is determined whether the second distance of the pixel point is less than the determined distance threshold. If so, it is determined that the pixel point belongs to the foreground point; if not, it is determined that the pixel point belongs to the background point. That is, the target pixel point belonging to the foreground point and the other pixel points belonging to the background point in the two-dimensional annotation box are determined. Then, the three-dimensional annotation data corresponding to the target pixel point belonging to the foreground point can be determined.

For S104:

For each sub-network structure, the sub-network structure is trained based on at least one pixel point containing three-dimensional annotation data corresponding to the sub-network structure. Alternatively, the sub-network structure can also be trained based on at least one pixel point containing three-dimensional annotation data corresponding to the sub-network structure and at least one pixel point belonging to a background point corresponding to the sub-network structure. By performing multiple rounds of training on each sub-network structure, a trained target neural network is obtained.

When training the target neural network, the loss value corresponding to each type of data in the three-dimensional annotation data can be determined based on the three-dimensional annotation data and the obtained three-dimensional prediction data, and the parameters of the target neural network can be adjusted using the weighted sum of the loss values corresponding to various data in the three-dimensional annotation data until the trained target neural network meets preset requirements, for example, until the accuracy of the trained target neural network is equal to the set accuracy threshold, or until the total loss value of the trained target neural network is less than the set loss threshold.

Exemplarily, for the target category of the object to be detected in the three-dimensional prediction data (or three-dimensional annotation data), the focal loss function can be used to determine the first loss corresponding to the target category. For the attribute category of the object to be detected, the softmax classification loss function can be used to determine the second loss corresponding to the attribute category; and determine the third loss corresponding to the orientation category. For the offset of the object to be detected, the depth of the center point of the object to be detected corresponding to the pixel point, the size of the three-dimensional detection box, the orientation of the three-dimensional detection box, and the speed of the object to be detected, the smooth L1 loss function can be used to determine the fourth loss. For the centrality, the binary cross entropy (BCE) loss function can be used to determine the fifth loss.

In an optional implementation, each sub-network structure in the target neural network corresponds to a regression index obtained by training with sample data, and the regression index is used to adjust the predicted regression data in the three-dimensional predicted data output by the detection head network connected to the sub-network structure by multiples;

Among them, the detection head network includes a classification network and a regression network. The detection head network is used to determine the three-dimensional prediction data corresponding to the sub-network structure based on the feature map output by each sub-network structure. The three-dimensional prediction data includes the prediction category data output by the classification network and the prediction regression data output by the regression network.

Taking into account that the detection head network is shared by multiple sub-network structures, that is, multiple sub-network structures are connected to the same detection head network, and since different sub-network structures are used to perform feature extraction and three-dimensional target prediction on objects to be detected of different scales, a regression indicator can be trained for each sub-network structure here, and the regression indicator is used to multiple-adjust the predicted regression data in the three-dimensional prediction data output by the detection head network connected to the sub-network structure, so that the adjusted predicted regression data matches the sub-network structure.

Referring to the schematic diagram of the architecture of a target neural network shown in FIG5 , the structure of the target neural network may include a backbone network, a pyramid network, and a detection head network, wherein the pyramid network includes a plurality of different sub-network structures, and the sizes of input features and/or output features corresponding to different sub-network structures are different. Each sub-network structure corresponds to a detection head network, that is, the detection head networks connected to the various sub-network structures share network parameters.

Exemplarily, the sample image is any image in the training sample, or it can also be an image to be detected. After the sample image is input into the target neural network, at least one convolution layer can be used to extract features of the sample image to obtain a feature map corresponding to the sample image, and then the obtained feature map is input into the backbone network for prediction to obtain three-dimensional prediction data.

In specific implementation, in order to reduce the consumption of video memory resources of the device deploying the target neural network, the parameters of the convolution layer for feature extraction of the sample image can be set to a smaller feature value. In order to balance the efficiency and accuracy of the target neural network, ResNet101 and deformable convolution can be used in the backbone network of the target neural network. For example, one or more convolution layers in the backbone network of ResNet101 can be set to deformable convolution.

The detection head network includes a classification network, which is used to output predicted category data, and the predicted category data includes one or more of target category, attribute category, and orientation category. The detection head network also includes a regression network, and the regression network outputs predicted regression data, and the predicted regression data includes one or more of offset, depth, size of the three-dimensional detection box, orientation of the three-dimensional detection box, speed of the object to be detected, and center degree.

Since different sub-network structures are used to process objects to be detected of different scales, that is, the sizes of the predicted regression data output by different sub-network structures are different, each sub-network structure can correspond to a trainable regression index, and the trained regression index can be used to adjust the predicted regression data in the three-dimensional predicted data output by the detection head network connected to the sub-network structure. For example, if the trained regression index corresponding to sub-network structure one is _X1 and the trained regression index corresponding to sub-network structure two is _X2 , then the predicted regression data output by the detection head network connected to sub-network structure one can be multiplied by the regression index _X1 to obtain the predicted regression data corresponding to sub-network structure one; the predicted regression data output by the detection head network connected to sub-network structure two can be multiplied by the regression index _X2 to obtain the predicted regression data corresponding to sub-network structure two.

In specific implementation, when the target neural network is trained using the training samples, the regression index can be trained, and when the trained target neural network is obtained, the trained regression index corresponding to each sub-network structure is obtained.

In an optional implementation, after obtaining the trained target neural network, the method further includes:

S105, obtaining an image to be detected;

S106, using the trained target neural network to detect the image to be detected, and determining a three-dimensional detection result of at least one target object included in the image to be detected.

By adopting the above method, a trained target neural network with good performance can be used to detect the image to be detected, and a relatively accurate three-dimensional detection result of at least one target object can be obtained.

The image to be detected can be any frame image. The acquired image to be detected is input into the trained target neural network. The trained target neural network is used to detect the image to be detected, and the three-dimensional detection result of at least one target object included in the image to be detected is determined. The three-dimensional detection result may include at least one of the following detection results: the size (length, width and height) of the three-dimensional detection frame corresponding to the target object, the coordinate information of the center point of the three-dimensional detection frame (horizontal axis coordinate, vertical axis coordinate, vertical axis coordinate), the target category, attribute category, orientation, orientation category, speed, centrality, and confidence of the target object. The target object can be any object in the image to be detected.

In S106, the target neural network is used to detect the image to be detected, and a three-dimensional detection result of at least one target object included in the image to be detected is determined, including:

S1061, using the target neural network to detect the image to be detected, and generating three-dimensional detection data corresponding to multiple pixel points in the image to be detected;

S1062, determining multiple candidate three-dimensional detection box information included in the image to be detected based on the three-dimensional detection data corresponding to the multiple pixel points respectively;

S1063, based on the information of the multiple candidate 3D detection frames, determining the projection frame information of the multiple candidate 3D detection frames in the bird's-eye view corresponding to the image to be detected;

S1064: Determine a three-dimensional detection result of at least one target object included in the image to be detected based on the multiple projection frame information.

By adopting the above method, by determining the projection frame information of multiple candidate three-dimensional detection frames in the bird's-eye view corresponding to the image to be detected, based on the multiple projection frame information, the three-dimensional detection result of at least one target object included in the image to be detected is more accurately determined.

In S1061, during the specific implementation, the target neural network can be used to detect the image to be detected, and the three-dimensional detection data corresponding to each pixel in the detection image can be generated. Then, the three-dimensional detection data corresponding to multiple pixels can be selected according to the confidence indicated by the three-dimensional detection data corresponding to each pixel. For example, a confidence threshold can be set, and the three-dimensional detection data corresponding to multiple pixels whose corresponding confidence is greater than the confidence threshold can be selected from the three-dimensional detection data corresponding to each pixel in the detection image. Alternatively, a selected quantity threshold can be set, for example, the selected quantity threshold is set to 100, and the three-dimensional detection data corresponding to 100 pixels are selected from the three-dimensional detection data corresponding to each pixel in the detection image in order from high to low confidence.

In S1062, based on the three-dimensional detection data corresponding to the multiple pixels, multiple candidate three-dimensional detection frame information included in the image to be detected can be determined, wherein the three-dimensional detection data of each pixel corresponds to a candidate three-dimensional detection frame information, and the candidate three-dimensional detection frame information includes the position information and size information of the candidate three-dimensional detection frame.

Exemplarily, when determining multiple candidate three-dimensional detection frame information included in the image to be detected based on the three-dimensional detection data corresponding to multiple pixel points, the position information of the center point of the determined target object can be restored to the real scene through the intrinsic parameter matrix of the acquisition device corresponding to the image to be detected, so as to generate the position information of the center point of the candidate three-dimensional detection frame information corresponding to the target object in the real scene.

In S1063, for each candidate 3D detection frame information, corresponding projection frame information in the bird's-eye view may be generated. The bird's-eye view includes a plurality of projection frame information, and the projection frame information includes the position and size of the projection frame.

In S1064, based on the multiple projection frame information, a three-dimensional detection result of at least one target object included in the image to be detected is determined, including:

Step 1: Based on the confidence and center corresponding to the target category indicated by each candidate 3D detection frame information, determine the target confidence of the projection frame corresponding to the candidate 3D detection frame; wherein the center is used to characterize the degree of proximity between the pixel point corresponding to the candidate 3D detection frame and the center point of the corresponding object to be detected;

Step 2: Based on the target confidences corresponding to the projection frames, determine the three-dimensional detection result of at least one target object included in the image to be detected.

In step 1, the centrality is used to characterize the closeness between the pixel point corresponding to the candidate three-dimensional detection frame and the corresponding center point of the object to be detected. The larger the centrality, the closer the pixel point is to the center of the object to be detected, the higher the reliability of the feature information of the pixel point, and the higher the reliability of the three-dimensional detection frame information generated for the pixel point; conversely, the smaller the centrality, the farther the pixel point is from the center of the object to be detected, the lower the reliability of the feature information of the pixel point, and the lower the reliability of the three-dimensional detection frame information generated for the pixel point. Therefore, the centrality of the pixel point indicated by the three-dimensional detection result can be used to screen the background points, avoid the low-quality prediction of the pixel point far from the center of the target object, and improve the detection efficiency.

Therefore, based on the confidence and center corresponding to the target category indicated by each candidate three-dimensional detection frame information, the target confidence of the projection frame corresponding to the candidate three-dimensional detection frame can be determined; and then the target confidence can be used to more accurately determine the three-dimensional detection result of at least one target object included in the image to be detected.

Exemplarily, the confidence corresponding to the target category indicated by each candidate 3D detection frame information may be multiplied by the center degree to determine the target confidence of the projection frame corresponding to the candidate 3D detection frame.

In step 2, a non-maximum suppression (NMS) method may be used to determine a three-dimensional detection result of at least one target object included in the image to be detected based on target confidences corresponding to each projection frame.

In an optional embodiment, a first quantity threshold of the three-dimensional detection results can also be set. After using the NMS method to determine the three-dimensional detection results of at least one target object included in the image to be detected, when the number of obtained three-dimensional detection results is greater than the set first quantity threshold, the obtained three-dimensional detection results can be screened according to the target confidence.

Referring to FIG. 6 , it is a flow chart of a driving control method provided by an embodiment of the present disclosure, which includes:

S601, acquiring a road image collected by the driving device during driving;

S602, using the target neural network trained by the neural network training method described in the above embodiment to detect the road image, and obtain target three-dimensional position data of the target object included in the road image;

S603: Control the traveling device based on the target three-dimensional position data of the target object included in the road image.

Exemplarily, the driving device may be an autonomous vehicle, a vehicle equipped with an Advanced Driving Assistance System (ADAS), or a robot. The road image may be an image collected in real time by the driving device during driving. The target object may be any object and/or any object that may appear on the road. For example, the target object may be an animal, a pedestrian, or other vehicles (including motor vehicles and non-motor vehicles) on the road.

When controlling the traveling device, the traveling device may be controlled to accelerate, decelerate, turn, brake, etc., or a voice prompt message may be played to prompt the driver to control the traveling device to accelerate, decelerate, turn, brake, etc.

Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

Based on the same concept, the embodiment of the present disclosure further provides a neural network training device. Referring to FIG. 7 , it is a schematic diagram of the architecture of the neural network training device provided by the embodiment of the present disclosure, including a first acquisition module 701, a first determination module 702, a second determination module 703, and a training module 704. Specifically:

A first acquisition module 701 is used to acquire training samples;

A first determination module 702 is used to determine, based on the scale of the two-dimensional annotation box of the object to be detected in the training sample, a sub-network structure in the target neural network that matches each pixel point in the two-dimensional annotation box; wherein different sub-network structures are used to extract features of pixel points in the two-dimensional annotation box of different scales in the training sample;

A second determination module 703 is used to determine the three-dimensional annotation data of the object to be detected corresponding to at least one pixel point in the two-dimensional annotation box in the training sample;

The training module 704 is used to train each of the sub-network structures in the target neural network based on at least one pixel point having the three-dimensional annotation data corresponding to each of the sub-network structures, so as to obtain a trained target neural network.

In a possible implementation manner, the first determination module 702, when determining the subnetwork structure matching each pixel point in the two-dimensional annotation box based on the scale of the two-dimensional annotation box of the object to be detected in the training sample, is used to:

For each pixel point in the two-dimensional annotation box, based on the scale of the two-dimensional annotation box, determine a first longest distance between the pixel point and each edge of the two-dimensional annotation box;

Based on the longest first distance and a preset distance range corresponding to each sub-network structure, a sub-network structure matching the pixel point is determined.

In a possible implementation manner, the second determination module 703, when determining the three-dimensional annotation data of the object to be detected corresponding to at least one pixel point in the two-dimensional annotation box in the training sample, is used to:

Determining target pixel points in the two-dimensional annotation box in the training sample that belong to foreground points based on the sub-network structures that are respectively matched with the pixel points in the two-dimensional annotation box;

Determine the three-dimensional annotation data of the object to be detected corresponding to the target pixel point.

In a possible implementation manner, the second determination module 703, when determining the target pixel points belonging to the foreground point in the two-dimensional annotation box in the training sample based on the sub-network structure respectively matched with the pixel points in the two-dimensional annotation box, is used to:

For each pixel point in the two-dimensional annotation frame, determining a second distance between the pixel point and a center point of the object to be detected in the two-dimensional annotation frame;

Determining a distance threshold corresponding to the pixel point based on a stride of the subnetwork structure matched with the pixel point and a preset radius parameter;

When the second distance is smaller than the distance threshold, the pixel point is determined to be the target pixel point.

In a possible implementation manner, the three-dimensional annotation data includes at least one of the following data:

The offset used to characterize the deviation between the pixel point and the corresponding center point of the object to be detected, the depth of the center point of the object to be detected corresponding to the pixel point, the size of the three-dimensional detection frame, the orientation of the three-dimensional detection frame, the orientation category, the speed of the object to be detected, the centrality used to characterize the degree of proximity between the pixel point and the corresponding center point of the object to be detected, the target category of the object to be detected, and the attribute category characterizing the state of the object to be detected.

In a possible implementation manner, before determining the sub-network structure in the target neural network matching each pixel point in the two-dimensional annotation box based on the scale of the two-dimensional annotation box of the object to be detected in the training sample, the method further includes: a third determination module 705, which is used to:

When there is a same pixel point in the training sample that is located in multiple two-dimensional annotation boxes, determining a third distance between the pixel point and a center point of the object to be detected in each of the two-dimensional annotation boxes;

The two-dimensional annotation box corresponding to the smallest third distance is used as the two-dimensional annotation box corresponding to the pixel point.

In a possible implementation, each sub-network structure in the target neural network corresponds to a regression index trained using the sample data, and the regression index is used to perform a multiple adjustment on the predicted regression data in the three-dimensional predicted data output by the detection head network connected to the sub-network structure;

Among them, the detection head network includes a classification network and a regression network. The detection head network is used to determine the three-dimensional prediction data corresponding to the sub-network structure based on the feature map output by each sub-network structure. The three-dimensional prediction data includes the predicted category data output by the classification network and the predicted regression data output by the regression network.

In a possible implementation, after obtaining the trained target neural network, the method further includes: a prediction module 706 for:

Acquire the image to be detected;

The trained target neural network is used to detect the image to be detected, and a three-dimensional detection result of at least one target object included in the image to be detected is determined.

In a possible implementation manner, the prediction module 706, when detecting the image to be detected using the target neural network to determine a three-dimensional detection result of at least one target object included in the image to be detected, is configured to:

Detecting the image to be detected using the target neural network to generate three-dimensional detection data corresponding to a plurality of pixel points in the image to be detected;

Determine, based on the three-dimensional detection data respectively corresponding to the plurality of pixel points, a plurality of candidate three-dimensional detection frame information included in the image to be detected;

Based on the information of the multiple candidate three-dimensional detection frames, determining projection frame information of the multiple candidate three-dimensional detection frames in the bird's-eye view corresponding to the image to be detected;

Based on the plurality of projection frame information, a three-dimensional detection result of at least one target object included in the image to be detected is determined.

In a possible implementation manner, the prediction module 706, when determining the three-dimensional detection result of at least one target object included in the image to be detected based on the plurality of projection frame information, is configured to:

Based on the confidence and center degree corresponding to the target category indicated by each candidate 3D detection frame information, determine the target confidence degree of the projection frame corresponding to the candidate 3D detection frame; wherein the center degree is used to characterize the degree of proximity between the pixel point corresponding to the candidate 3D detection frame and the center point of the corresponding object to be detected;

Based on the target confidences respectively corresponding to the projection frames, a three-dimensional detection result of at least one target object included in the image to be detected is determined.

Based on the same concept, the embodiment of the present disclosure further provides a driving control device. Referring to FIG. 8 , it is a schematic diagram of the architecture of the driving control device provided by the embodiment of the present disclosure, including a second acquisition module 801, a detection module 802, and a control module 803. Specifically:

The second acquisition module 801 is used to acquire the road image collected by the driving device during the driving process;

A detection module 802 is used to detect the road image using a target neural network trained by the neural network training method proposed in the present disclosure to obtain target three-dimensional position data of a target object included in the road image;

The control module 803 is used to control the driving device based on the target three-dimensional position data of the target object included in the road image.

In some embodiments, the functions or templates contained in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

Based on the same technical concept, an embodiment of the present disclosure also provides an electronic device. Referring to FIG9 , a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure includes a processor 901, a memory 902, and a bus 903. Among them, the memory 902 is used to store execution instructions, including a memory 9021 and an external memory 9022; the memory 9021 here is also called an internal memory, which is used to temporarily store the calculation data in the processor 901, and the data exchanged with the external memory 9022 such as a hard disk. The processor 901 exchanges data with the external memory 9022 through the memory 9021. When the electronic device 900 is running, the processor 901 communicates with the memory 902 through the bus 903, so that the processor 901 executes the following instructions:

Get training samples;

Based on the scale of the two-dimensional annotation box of the object to be detected in the training sample, determine the sub-network structure in the target neural network that matches each pixel point in the two-dimensional annotation box; wherein different sub-network structures are used to extract features of pixel points in the two-dimensional annotation box of different scales in the training sample;

Determine three-dimensional annotation data of the object to be detected corresponding to at least one pixel point within the two-dimensional annotation box in the training sample;

Based on at least one pixel point having the three-dimensional annotation data corresponding to each of the sub-network structures, each of the sub-network structures in the target neural network is trained to obtain a trained target neural network.

Based on the same technical concept, an embodiment of the present disclosure also provides an electronic device. Referring to FIG10 , a schematic diagram of the structure of the electronic device provided in the embodiment of the present disclosure includes a processor 1001, a memory 1002, and a bus 1003. Among them, the memory 1002 is used to store execution instructions, including a memory 10021 and an external memory 10022; the memory 10021 here is also called an internal memory, which is used to temporarily store the calculation data in the processor 1001, and the data exchanged with the external memory 10022 such as a hard disk. The processor 1001 exchanges data with the external memory 10022 through the memory 10021. When the electronic device 1000 is running, the processor 1001 communicates with the memory 1002 through the bus 1003, so that the processor 1001 executes the following instructions:

Acquire road images collected by the driving device during driving;

Using the target neural network trained by the neural network training method proposed in the present disclosure, the road image is detected to obtain target three-dimensional position data of the target object included in the road image;

The travel device is controlled based on target three-dimensional position data of the target object included in the road image.

In addition, the embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the neural network training method and the driving control method described in the above method embodiment are executed. The storage medium may be a volatile or non-volatile computer-readable storage medium.

The disclosed embodiments also provide a computer program product, which carries a program code. The program code includes instructions that can be used to execute the steps of the neural network training method and the driving control method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims (14)

1. A neural network training method, comprising:

Obtaining a training sample;

Determining a sub-network structure in a target neural network matched with each pixel point in a two-dimensional labeling frame based on the scale of the two-dimensional labeling frame of the object to be detected in the training sample; the method comprises the steps of obtaining a training sample, wherein different sub-network structures are used for extracting characteristics of pixel points in the two-dimensional labeling frame with different scales in the training sample;

Determining three-dimensional labeling data of the object to be detected, which corresponds to at least one pixel point in the two-dimensional labeling frame, in the training sample;

Training each sub-network structure in the target neural network based on at least one pixel point which corresponds to each sub-network structure and has the three-dimensional labeling data, so as to obtain a trained target neural network;

The determining the three-dimensional labeling data of the object to be detected, which corresponds to at least one pixel point in the two-dimensional labeling frame in the training sample, comprises the following steps:

Determining target pixel points belonging to foreground points in the two-dimensional labeling frame in the training sample based on the sub-network structures respectively matched with the pixel points in the two-dimensional labeling frame;

and determining the three-dimensional annotation data of the object to be detected, which corresponds to the target pixel point.

2. The method according to claim 1, wherein the determining the sub-network structure for matching each pixel point in the two-dimensional labeling frame based on the scale of the two-dimensional labeling frame of the object to be detected in the training sample comprises:

Determining the longest first distance between each pixel point and each side of the two-dimensional labeling frame based on the scale of the two-dimensional labeling frame aiming at each pixel point in the two-dimensional labeling frame;

And determining the sub-network structure matched with the pixel point based on the longest first distance and a preset distance range corresponding to each sub-network structure.

3. The method according to claim 1, wherein the determining, based on the sub-network structures respectively matched with the pixels in the two-dimensional labeling frame, the target pixels belonging to the foreground point in the two-dimensional labeling frame in the training sample includes:

determining a second distance between each pixel point in the two-dimensional labeling frame and a center point of the object to be detected in the two-dimensional labeling frame;

determining a distance threshold corresponding to the pixel point based on the stride of the sub-network structure matched with the pixel point and a preset radius parameter;

and under the condition that the second distance is smaller than the distance threshold value, determining the pixel point as the target pixel point.

4. A method according to any one of claims 1 to 3, wherein the three-dimensional annotation data comprises at least one of:

The method comprises the steps of representing offset of deviation between a pixel point and a center point of a corresponding object to be detected, depth of the center point of the object to be detected corresponding to the pixel point, size of a three-dimensional detection frame, orientation of the three-dimensional detection frame, orientation type, speed of the object to be detected, centrality representing similarity degree between the pixel point and the center point of the corresponding object to be detected, target type of the object to be detected and attribute type representing state of the object to be detected.

5. A method according to any one of claims 1-3, further comprising, prior to determining a sub-network structure in the target neural network for which each pixel point within the two-dimensional labeling frame matches based on the dimensions of the two-dimensional labeling frame of the object to be detected in the training sample:

Under the condition that the same pixel point exists in a plurality of two-dimensional labeling frames in the training sample, determining a third distance between the pixel point and the center point of an object to be detected in each two-dimensional labeling frame;

and taking the two-dimensional labeling frame with the minimum corresponding third distance as the two-dimensional labeling frame corresponding to the pixel point.

6. A method according to any one of claims 1 to 3, wherein each sub-network structure in the target neural network corresponds to a regression index trained by the training sample, and the regression index is used for performing multiple adjustment on prediction regression data in three-dimensional prediction data output by a detection head network connected with the sub-network structure;

The detection head network comprises a classification network and a regression network, wherein the detection head network is used for determining three-dimensional prediction data corresponding to the sub-network structure based on a feature map output by each sub-network structure, and the three-dimensional prediction data comprises prediction category data output by the classification network and prediction regression data output by the regression network.

7. A method according to any one of claims 1-3, further comprising, after obtaining the trained target neural network:

Acquiring an image to be detected;

And detecting the image to be detected by using the trained target neural network, and determining a three-dimensional detection result of at least one target object included in the image to be detected.

8. The method according to claim 7, wherein detecting the image to be detected using the target neural network, determining a three-dimensional detection result of at least one target object included in the image to be detected, includes:

Detecting the image to be detected by using the target neural network, and generating three-dimensional detection data corresponding to a plurality of pixel points in the image to be detected;

Determining a plurality of candidate three-dimensional detection frame information included in the image to be detected based on three-dimensional detection data corresponding to the pixel points respectively;

determining projection frame information of a plurality of candidate three-dimensional detection frames in a bird's eye view corresponding to the image to be detected based on the plurality of candidate three-dimensional detection frame information;

And determining a three-dimensional detection result of at least one target object included in the image to be detected based on the plurality of projection frame information.

9. The method according to claim 8, wherein the determining a three-dimensional detection result of at least one target object included in the image to be detected based on a plurality of the projection frame information includes:

Determining the target confidence of a projection frame corresponding to each candidate three-dimensional detection frame based on the confidence and the centrality corresponding to the target category indicated by the information of each candidate three-dimensional detection frame; the centrality is used for representing the similarity degree between the pixel points corresponding to the candidate three-dimensional detection frames and the central points of the corresponding objects to be detected;

and determining a three-dimensional detection result of at least one target object included in the image to be detected based on the target confidence degrees respectively corresponding to the projection frames.

10. A running control method, characterized by comprising:

Acquiring a road image acquired by a running device in the running process;

Detecting the road image by using the target neural network trained by the neural network training method according to any one of claims 1 to 9 to obtain target three-dimensional pose data of a target object included in the road image;

And controlling the driving device based on the target three-dimensional pose data of the target object included in the road image.

11. A neural network training device, comprising:

The first acquisition module is used for acquiring training samples;

The first determining module is used for determining a sub-network structure in a target neural network matched with each pixel point in the two-dimensional labeling frame based on the scale of the two-dimensional labeling frame of the object to be detected in the training sample; the method comprises the steps of obtaining a training sample, wherein different sub-network structures are used for extracting characteristics of pixel points in the two-dimensional labeling frame with different scales in the training sample;

the second determining module is used for determining three-dimensional annotation data of the object to be detected, which corresponds to at least one pixel point in the two-dimensional annotation frame in the training sample;

the training module is used for training each sub-network structure in the target neural network based on at least one pixel point which corresponds to each sub-network structure and has the three-dimensional annotation data, so as to obtain a trained target neural network;

The second determining module is configured to, when determining three-dimensional labeling data of the object to be detected, where the three-dimensional labeling data corresponds to at least one pixel point in the two-dimensional labeling frame, where the three-dimensional labeling data corresponds to at least one pixel point in the training sample: determining target pixel points belonging to foreground points in the two-dimensional labeling frame in the training sample based on the sub-network structures respectively matched with the pixel points in the two-dimensional labeling frame; and determining the three-dimensional annotation data of the object to be detected, which corresponds to the target pixel point.

12. A travel control device characterized by comprising:

the second acquisition module is used for acquiring road images acquired by the driving device in the driving process;

The detection module is used for detecting the road image by using the target neural network trained by the neural network training method according to any one of claims 1 to 9 to obtain target three-dimensional pose data of a target object included in the road image;

and the control module is used for controlling the running device based on the target three-dimensional pose data of the target object included in the road image.

13. An electronic device, comprising: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory in communication over the bus when the electronic device is running, the machine-readable instructions when executed by the processor performing the steps of the neural network training method of any one of claims 1 to 9, or the steps of the travel control method of claim 10.

14. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a computer program which, when executed by a processor, performs the steps of the neural network training method according to any one of claims 1 to 9, or performs the steps of the travel control method according to claim 10.