BR112021015232A2 - SHADOW AND CLOUD MASKING FOR REMOTE SENSING IMAGES IN AGRICULTURE APPLICATIONS USING MULTI-LAYER PERCEPTRON - Google Patents

Abstract

SHADOW AND CLOUD MASKING FOR REMOTE SENSING IMAGES IN AGRICULTURE APPLICATIONS USING MULTI-LAYER PERCEPTRON. It is a masking method. cloud and shadow for remote sensing images of a field agriculture using multilayer perceptrons, which includes receiving electronically an observed image, perform, with the use of at least a processor, an image segmentation of the observed image to split the observed image into a plurality of image segments or superpixels, extract features for each of the image segments using the at least one processor, and determine, by a module of cloud mask generation running on at least one processor, one classification for each of the image segments using the features extracted for each of the image segments, where the image module cloud mask generation applies a classification model including a junction of multilayer perceptrons to generate a mask of cloud to the observed image so that each pixel within the image observed has a corresponding classification.

Description

“SHADOW AND CLOUD MASKING FOR REMOTE SENSING IMAGES IN AGRICULTURE APPLICATIONS USING MULTI-LAYER PERCEPTRON” RELATED ORDERS

[0001] This application claims priority of US Interim Application No. 62/801,008, filed on February 4, 2019, entitled “SHADOW AND CLOUD MASKING FOR REMOTE SENSING IMAGES IN AGRICULTURE APPLICATIONS USING A MULTILAYER PERCEPTRON”, and incorporated herein by reference in its entirety.

TECHNICAL FIELD

[0002] This invention describes a method and system applicable to satellite imagery for agricultural applications, which uses a cloud and shadow detection algorithm.

BACKGROUND

[0003] Satellite images are often affected by the presence of clouds and their shadows. Because clouds are opaque at the wavelength of visible light, they often hide the ground surface from Earth observation satellites. The lightening and darkening effects of clouds and shadows influence data analysis, causing imprecise atmospheric corrections and land cover classification impedance. Its detection, identification and removal are, therefore, the first steps in the processing of satellite images. Clouds and cloud shadows can be filtered manually, but automating masking is important where there may be thousands of images to process.

[0004] Related technique systems for detecting clouds and shadows in satellite images focus on images that have numerous bands and a wealth of information to work with. For example, some related technique systems use a morphological operation to identify potential shadow regions, which are darkest in the near-infrared spectral range. The related technique addresses how, given a cloud mask, a sweep is taken through a range of cloud heights and addresses how the places where projected shadows would fall are calculated geometrically. The area of greatest overlap between the projections and the possible shadow regions is considered the cloud mask. The related technique, however, uses a large number (eg, 7, 8, 9, etc.) of spectral bands (ie, “bands”) to accomplish this cloud masking task. Satellite images including a greater number of bands are generally not available at high frequency (eg, more than once a day) and therefore images may be less useful in indicating agronomic decisions. It would be useful to perform cloud masking using high frequency images, which include fewer bands, to better inform the producer for agronomic decision making.

SUMMARY

[0005] Sometimes less satellite bands than necessary for the successful operation of agricultural applications to inform the farmer about agricultural field management decisions are available and therefore the related technique procedures are inadequate. Systems and methods are disclosed herein for cloud masking, where fewer bands of information are available than are required for processing by related art systems (e.g., one, two, three, four, or five). In some embodiments, the systems and methods disclosed herein apply to a satellite image including a near-infrared ("NIR") band and a visible red-green-blue ("RGB") band. Using a reduced number of bands allows cloud masking to be performed on satellite images obtained from a greater number of satellites.

[0006] In some embodiments, the systems and methods disclosed in the present document perform cloud masking using a limited number of bands, performing unsupervised image segmentation, extracting resources for each image segment, performing cloud masking using supervised model learning of cloud masking together and evaluating cloud masking models using a confusion matrix.

[0007] According to one aspect, a method for masking cloud and shadow for remote sensing images of an agricultural field using multilayer perceptrons is provided. The method includes electronically receiving an observed image, performing, using at least one processor, an image segmentation of the observed image to divide the observed image into a plurality of image segments or superpixels, extracting features for each of the image segments using the at least one processor, and determining, by a cloud mask generation module running on the at least one processor, a classification for each of the image segments using the extracted resources for each of the image segments, wherein the image module Cloud mask generation applies a classification model including a junction of multilayer perceptrons to generate a cloud mask for the observed image so that each pixel within the observed image has a corresponding classification. The classification can be selected from a set including a cloud classification, a shadow classification and a field classification. Classification of each of the image segments can be performed using five or fewer bands of the observed image. Bands can include a visible band (eg, red spectral band, green spectral band, blue spectral band), a near-infrared band, and a red edge band. The method may also include applying the cloud mask to the observed image. The method may further include using a resulting image to generate a yield forecast for the agricultural field or for other purposes. The method may further include using the cloud generation module running on one or more processors to train the classification model. The method may further include using the cloud generation module running on one or more processors to evaluate one or more classification models. The assessment can be performed using a confusion matrix.

[0008] According to another aspect, a system for masking cloud and shadow for remote sensing images of an agricultural field is provided. The system includes a computing system with at least one processor to run a cloud mask generation module, wherein the cloud mask generation module is configured to: receive an observed image, apply a segmentation model to the observed image to divide the observed image into a plurality of superpixels (or image segments), extract features from each of the superpixels and determine a classification for each of the superpixels, using the extracted features for each of the superpixels and applying a classification model including a junction of multilayer perceptrons, to generate a cloud mask for the observed image so that each pixel in the observed image has a corresponding classification. The classification can be selected from a set including a cloud classification, a shadow classification and a field classification. Classification of each of the superpixels can be performed using a limited number of color bands of the observed image, such as five, four, three, two or one (e.g. visible (RGB), near infrared, red border, etc.) . The computing system can be further configured to apply the cloud mask to the observed image and use a resulting image to generate a yield forecast for the agricultural field or for other purposes. The cloud generation module can be further configured to train the classification model and to evaluate one or more classification models.

[0009] According to another aspect, a method for cloud and shadow masking for remote sensing images of agricultural fields using multilayer perceptron may include receiving an observed image of an agricultural field from a data store, segmenting the observed image field in superpixels, determine a feature set of the observed image using the superpixels, determine a classification for each superpixel, insert the superpixels into a multilayer perceptron to train a model, and evaluate the multilayer perceptron using a evaluation.

[0010] According to another aspect, a method for cloud and shadow masking for remote sensing images of agricultural fields using multilayer perceptron may include receiving a request to generate a cloud map, requesting an observed image, segmenting the image observed on superpixels and determine a feature set, insert the superpixels into a junction, determine a rank for each superpixel, determine an aggregate rank for each superpixel, and apply a rank mask to the image to obtain the necessary information.

BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The disclosed modalities have other advantages and characteristics that will be more easily evident from the detailed description, the appended claims and the accompanying figures (or drawings). A brief introduction of the figures is below.

[0012] Figure 1 illustrates a system environment to generate a cloud map for an agricultural field, according to an example modality.

[0013] Figure 2A illustrates an observed image, according to an example embodiment.

[0014] Figure 2B illustrates a first layer of a cloud map, according to an example embodiment.

[0015] Figure 2C illustrates a second layer of a cloud map, according to an example embodiment.

[0016] Figure 3 illustrates a segmented image, according to an example modality.

[0017] Figure 4 illustrates a multilayer perceptron junction configured to determine a classification for each superpixel in an observed image, according to an example embodiment.

[0018] Figure 5 illustrates a method for training a classification model, according to an example modality.

[0019] Figure 6 illustrates a method to generate a cloud mask, according to an example modality.

[0020] Figure 7 illustrates an example computing system, according to an example embodiment.

[0021] Figure 8 illustrates an overview of data ingestion in the MLP cloud masking model, according to an example modality.

[0022] Figure 9 illustrates a confusion matrix for a cloud masking task, according to an example modality.

DETAILED DESCRIPTION

[0023] The Figures (FIGS.) and the following description refer to preferred embodiments by way of illustration only. It should be noted that from the discussion that follows, alternative embodiments of the structures and methods disclosed in this document will be readily recognized as viable alternatives that can be employed without departing from the principles disclosed. It is noted that, where practicable, similar or similar reference numbers may be used in the figures and may indicate similar or similar functionality. The figures represent embodiments of the system (or method) disclosed for purposes of illustration only.

SYSTEM ENVIRONMENT

[0024] Figure 1 illustrates a system environment to generate a cloud map for an agricultural field. Within the system environment (100), a client system (110) includes a cloud mask generation ("CMG") module (112) that generates a cloud map. A cloud map is an image of an agricultural field in which a rating for each pixel in the image has been determined by the CMG module (112). The classifications can be, for example, “cloud”, “shadow” or “field”. In other examples, a cloud map is some other visualization or data structure that indicates clouds, shadows, and sorted fields in an observed image.

[0025] The CMG module (112) employs a classification model (116) to generate a cloud map from an observed image of an agricultural field. The client system (110) can request observed images over the network (150) and the network system (120) can provide the observed images in response. The network (150) is normally a cell tower, but can be a mesh network or power line. The network system (120) is normally the Internet, but can be any network including, but not limited to, a LAN, a MAN, a WAN, a wired or wireless mobile network, a private network, a virtual private network or a combination thereof. A network system (120) accesses observed images from an observation system (140) via a network (150).

[0026] In various embodiments, the system environment (100) may include additional systems or fewer systems. Furthermore, the capabilities assigned to a system within the environment can be distributed to one or more other systems within the system environment (100). For example, the CMG module (112) may run on the network system (120) instead of the client device (110).

[0027] The CMG module (112) accepts an observed image of the network system (120) and sends a cloud map to a user of the client system (110). The CMG module (112) can also accept an observed image from the observation system (140). Image data can consist of an image or photograph taken from a remote sensing platform (airplane, satellite or drone). Images are a raster dataset; each scan being composed of pixels. Each pixel has a specific pixel value (or values) that represents soil characteristics. Observed images include some pixels. Each pixel includes information on multiple data channels (eg 3, 4, 5), each channel associated with a particular spectral band ("band information"). The CMG module (112) uses the bandwidth information to generate the cloud map.

[0028] In one example, an observed image is an image taken of an agricultural field from a satellite or network of satellites. Space-based satellites use Global Positioning System (GPS) data, which can consist of coordinates and time signals to help track assets or entities. Figure 2A illustrates an example of an observed image, according to an example embodiment. In the illustrated example, the observed image (210) is an RGB satellite image of an agricultural field. More particularly, in this example, the observed image is a GeoTIFF image including geographic information associated with the image. The band information of the observed image (210) includes three channels of data, including a red spectral band, a green spectral band, and a blue spectral band.

[0029] In various modalities, the observed images may have different band information. For example, an observed image may be an image with multispectral bands (eg, six or more bands) taken by a satellite. Some examples of satellite imagery with multispectral bands include images from the LANDSAT™ and SENTINEL™ satellites. In other examples, a satellite image may only have four or five bands. Some examples of five-band satellite imagery are images from the PLANETSCOPE™ and RAPIDEYE™ satellites. In these examples, the five bands include R, G, B, RED EDGE and NIR bands.

Some examples of four-band satellite imagery include DOVE™ images from PLANETSCOPE™. In these examples, the four bands include R, G, B and NIR.

[0030] The CMG module (112) generates a cloud map. Figure 2B and Figure 2C illustrate two layers of a cloud map, according to an exemplary embodiment. To generate the cloud map (220), the CMG module (112) determines a classification for each pixel in the observed image (210). Figure 2B illustrates a first layer of the cloud map (e.g. cloud map (220B)) showing groups of pixels (230B) classified as clouds, and Figure 2C illustrates a second layer of the cloud map (e.g. map (220C)) illustrating groups of pixels (230C) classified as shadows. Notably, the cloud map is an image of the same type as the input, such as a GeoTIFF image, having the same size and shape as the observed image (210), so the sorted pixels of the cloud map (210) correspond to similarly positioned pixels in the observed image (210).

[0031] There are several benefits of this system for producers and agronomists. For example, a cloud map can be applied to multiple downstream projects. Examples include yield forecasting, crop type classification, and crop health. In these applications, the objective is to eliminate non-informative pixels related to cloud and shadow, thus focusing on field information.

[0032] To illustrate, for example, a field manager may wish to predict a yield for his agricultural field using an observed image. If the observed image includes pixels that represent clouds and shadows, the model that predicts the yield of the agricultural field can generate erroneous results. This can be caused by clouds and shadows adversely affecting the detection of healthy and unhealthy areas of plant matter in the field. As such, the cloud map can be used as a mask for the observed image. In other words, pixels that are identified as clouds or shadows can be removed from an observed image before using the observed image to generate a yield prediction for the agricultural field. Masking the cloud and shadow pixels of the observed image increases the robustness of the yield prediction model.

CLOUD MASKING MODEL

[0033] In general, the collected data are processed to derive values that can drive functions such as visualization, reporting, decision making and other analyses. Created roles can be shared and/or distributed to authorized users and subscribers. Data modeling and analysis can include one or more application programs configured to extract raw data that is stored in the data repository and process that data to achieve the desired function. It will be understood by those skilled in the art that the functions of application programs as described herein may be implemented by means of a plurality of programs or separate program modules configured to communicate and cooperate with one another to achieve the functional results. desired.

[0034] In one embodiment, data modeling and analysis can be configured or programmed to pre-process data that is received by the data repository from multiple data sources. Incoming data can be preprocessed with techniques for removing noise and distortion effects, removing unnecessary data that distorts other data, filtering, data smoothing data selection, data calibration, and error accounting. All of these techniques must be applied to improve the overall data set.

[0035] In one embodiment, modeling and data analysis generate one or more pre-configured agronomic models using data provided by one or more of the data sources and which are ingested and stored in the data repository. Modeling and data analysis may comprise an algorithm or a set of institutions to program different elements of a precision agriculture system. Agronomic models can comprise calculated agronomic factors derived from data sources that can be used to estimate specific agricultural parameters. In addition, agronomic models may contain recommendations based on these agricultural parameters. In addition, data modeling and analysis can comprise agronomic models created specifically for sharing external data that is of interest to third parties.

[0036] In one embodiment, modeling and data analysis can generate predictive models. Forecasting models can comprise one or more mathematical functions and a set of weights, coefficients, learned critical values, or any other similar numerical or categorical parameters that together convert the data into an estimate. These may also be referred to as “calibration equations” for convenience. Depending on the modality, each of these calibration equations can refer to the equation to determine the contribution of a data type, or some other arrangement of equations can be used.

[0037] The CMG module (112) generates a cloud map in two stages: (i) unsupervised image segmentation of an observed image and (ii) supervised classification into segments determined from the observed image. In one example, the CMG module (112) uses a multi-layered perception join to classify the segments as “cloud”, “shadow”, or “field”.

[0038] The CMG module (112) performs unsupervised image segmentation on an observed image. Segmentation of the observed image results in a matrix of superpixels representing the observed image (“segmented image”). The CMG module (112) can employ a segmentation model (114) to perform segmentation on the observed image. In one example, the segmentation model (114) is a simple linear iterative clustering algorithm (SLIC), but could be other models. Image segmentation groups pixels in an image with homogeneous properties (for example, color) to form a superpixel. In this process, the CMG module (112) generates superpixels by grouping pixels in an observed image into a number “n” of homogeneous regions (“clusters”).

[0039] When generating clusters, the CMG module (112) uses a distance metric to determine the similarity between two pixels in a multidimensional plane. Similarity is based on RGB values, REDEDGE (if available), NIR band values and their respective spatial coordinate locations. In one example, the distance metric is Euclidean distance, but it could be another distance metric. To illustrate, the CMG module (112) determines that a pixel α is homogeneous with pixel b if and only if the distance metric (e.g. Euclidean distance) calculated between α and b is the smallest among the calculated distance metrics between α and all other pixels b. The CMG module determines that the homogeneous pixels are clusters using the iterative K-means clustering procedure. The CMG generates “n” clusters, where “n” is determined based on the row length and column length of the observed image. For example, the value of “n” has a positive correlation with an observed image size and, in some embodiments, is determined by the total number of pixels.

[0040] To illustrate, Figure 3 is an example of a segmented image. The segmented image (300) is an observed image processed into segments by the CMG module (112). The segmented image includes several segments (310), with each segment illustrated as a small geometric shape. Each segment of the segmented image is a superpixel whose pixels have similar characteristics (eg color, hue, etc.). Similar characteristics between pixels in a superpixel may indicate that the pixels represent a cloud, shadow, and/or field.

[0041] The CMG module (112) generates a classification for each segment in the segmented image (300). A classification is a type identified for a segment. For example, some classifications for segments in the observed image may include “cloud” (e.g. segment (310A)), “shadow” (e.g. segment (310B)), and/or “field” (e.g. segment (310C) )). In other words, the CMG module (112) identifies groups of pixels in an observed image that represent a cloud, a shadow and/or a field.

[0042] The CMG module (112) determines, or extracts, a set of resources for each identified superpixel. A resource set includes any information that can be determined and/or derived from the RGB and/or NIR information included in the pixels of a superpixel. A feature set can include, but is not limited to, Hue, Saturation, and Intensity (HSV) color space, along with XYZ and CaCICb color space. A feature set can also include vegetation indices such as Normalized Difference Vegetation Index (NDVI), Simple Ratio (SR), Green Chlorophyll Index (Green CF), and Meris Terrestrial Chlorophyll Index (MTCI) or any other linear transformation significant nonlinear or non-linear value of the raw pixel value. Some modalities of the segmentation step would also use derived features. In some cases, additional bands may be used to determine a feature set. For example, a RED EDGE band can be used if the observed image is a RAPIDEYE™ image. In addition, a resource set can include regional and soil information, as well as meteorological information, such as average daily cloud cover, average daily rainfall, average daily normal irradiance, average daily relative humidity, and average daily solar radiation.

[0043] The CMG module (112) determines, or extracts, a set of features using a variety of techniques. For example, the CMG module (112) performs color space computation/conversion to determine features for a set of features using the RGB and NIR data in observed images. To illustrate, the CMG module 112 converts RGB information from a superpixel to CaCICb information for the same superpixel. In another example, the CMG module (112) performs statistical analysis to determine specific superpixel features for a set of features. To illustrate, the CMG module 112 calculates a particular statistical property for an entire image and compares it to the particular property of an individual superpixel. The CMG module (112) uses the difference in the particular property of the superpixel and the particular property of an image to determine a characteristic (eg, contrast).

[0044] The CMG module (112) determines a classification for each superpixel using RGB, NIR and/or resource set for each superpixel. The CMG module (112) employs a classification model (116) to determine a classification for each superpixel. In one example, the classification model (116) is a supervised classification model. Some examples of supervised classification models may include, but are not limited to, multilayer perceptron, deep neural networks, or joining methods. Given any of these models, the CMG module 112 learns, without being explicitly programmed to do so, how to determine a rating for a superpixel using the feature set for that superpixel. In various embodiments, the CMG model (112) can determine a classification using bands (eg, RGB, NIR, RED EDGE) for the superpixel. Furthermore, the CMG module (112) can determine a rank using sums and ratios between bands.

[0045] One factor that makes the disclosed system unique is a specific architecture to determine a rating for each superpixel. In an example embodiment, the CMG module (112) employs a classification model (116) with an architecture, which may include the set of multilayer perceptrons or other types of models, such as gradient-increased trees, designed to accurately minimal pre-processing of input observed images. For example, a CMG module employing a multilayer perceptron join has the lowest misclassification rate among cloud masking models (e.g. 3.93%), while a traditional algorithm like XGBoost has a error rate of 8.45% and a convolutional neural network has an error rate of 10.09%. In another exemplary embodiment, the CMG module (112) employs a classification model (116) with an architecture including a junction of multilayer perceptrons and XGBoost. In this case, crop pixels wrongly classified as cloud or shadow is 3.39%.

[0046] Figure 4 is an illustration of a multilayer perceptron junction (“junction”) configured to determine a rating for each superpixel in an observed image. In another embodiment, a junction of multilayer perceptrons and XGBoost can be configured to determine a classification for each superpixel in an observed image. This combination provides a good overview of shadows. In yet another embodiment, any standard methodology for supervised learning techniques can be used, such as tree-based methods with increased gradient, Random Forest Classifiers, Support Vector Machines, KNN, neural network and so on. The junction (400) determines an aggregate classification (402) (eg, "cloud") of a superpixel based on the bandwidth and feature set information for that superpixel. Band information and feature set are input to the junction as an input vector. In one example, feature selection is used to select key features, including the values used to describe features in this document. Some of the values include the following: segment_R_mean, image_R_mean, segment_NDVI_mean, image_NDVI_mean, etc. In the present document, “segment_” features provide information on color/vegetation indices within segments, and “image_” features provide information with the full range of the image.

[0047] The junction (400) includes a number (eg, 10, 20, 30, etc.) of multilayer perceptrons (410) with each multilayer perceptron trained to individually determine a classification (412) of the superpixel. Each multilayer perceptron (410) can be individually trained to determine a rating (412) for a superpixel instead of each multilayer perceptron being clones of a single multiplayer perceptron trained to determine a rating for the superpixel. In some examples, the number of multilayer perceptrons (410) at the junction (400) may be based on the type of image input observed in the cloud masking model. For example, the junction may include ten multilayer perceptrons for RAPIDEYE™ observed images and twenty multilayer perceptrons for PLANETSCOPE™ observed images.

[0048] To determine a rating for a superpixel, the CMG module (112) inputs the band information and feature set for the superpixel into each multilayer perceptron (410) of the junction (400). Each multilayer perceptron (410) determines a classification (412) of the superpixel. The CMG module (112) determines that an aggregate classification (402) for the superpixel is the classification determined by the plurality of multilayer perceptrons. Other methods for determining an aggregate rating (402) for a superpixel based on individual ratings (412) for that superpixel determined by multilayer perceptrons at a junction are also possible.

[0049] Another factor that makes this method unique is that the CMG module (112) determines a classification for superpixels using information only from the RGB and NIR (and RED EDGE) bands. Existing cloud masking methods for agricultural applications use many multispectral and NIR bands, which are useful for detecting cloud features. Satellites used for agricultural purposes may only have four or five bands. The algorithm in these methods works with this limited number of bands.

TRAINING THE MASKING MODEL OF A CLOUD

[0050] In order to train the model and determine values for the model parameters (ie for the calibration equations), certain data can be collected as inputs to train the model. The type of modeling function may vary depending on the implementation. In one embodiment, regression techniques such as ElasticNet, linear, logistic or otherwise can be used. Other techniques can also be used, examples of which include Random Forest Classifiers, Neural Networks, Support Vector Machines, and so on. Once trained, the resulting prediction model can be used to predict a specific type of data. Datasets of predictions versus performance standards are validated by taking test cases and relating them back to the true values of a traditional model in an effort to validate that the model is working. Validation involves relating the model's predictions to the actual values that were collected.

[0051] Figure 8 is an example data ingestion overview for a junction cloud masking model based on MLP and XGBoost. The image inputs (x,y,4) (801) are ingested into the model. In these examples, the four bands include R.G, B, and NIR bands. An unsupervised image segmentation approach (eg simple linear iterative clustering algorithm (SLIC)) is used to segment a given image into superpixels. Cloud/Shadow/Field pixels are generally well divided into different slices due to their different appearance, but not consistently. Only pure cloud/shadow/field segments are chosen to train the model.

[0052] Feature sets are extracted from each segment or superpixel (803). Image resources are calculated which are useful in cloud/shadow/field differentiation. Local band characteristics within each segment (e.g. R/G/B average, R/G/B ratios and various vegetation indices such as NDVI, EVI, etc.) are extracted from the different color bands and spaces of transformed colors (e.g. HSV color, CICaCb color, etc.). The overall color capabilities in the whole image, for example, R/G/B whole image average, etc. are also included as resources. Some field-specific meteorological data include average cloud cover, average normal irradiance, etc. to include them as resources for the classification task.

[0053] The characteristics of each segment then become a row of the training data. A feature selection approach (805) is used to select the main features of the MLP model as well as for the XGBoost model. Other feature selection techniques can also be used, such as, but not limited to, recursive feature elimination, boundary elimination, or simple Pearson correlation.

[0054] For the supervised learning stage, the data is fed into the junction model (806) of several MFP and XGBoost models. Multiple MFP and XGBoost models are trained on a subsampled dataset with relatively even data distribution, so that each model has seen enough data in each class. Each model is trained using the important features selected using the feature selection approach. When making predictions, the prediction probabilities for each model are guaranteed. The average of the two probabilities of the MFP and the extended gradient tree model are obtained and then the appropriate class (807) for each segment is selected. Finally, the most voted class is returned as the final prediction for that segment (808). A cloud/shadow mask (809) is then created.

[0055] The CMG module (112) trains a classification model (116) (eg, a multilayer perceptron) using a series of superpixels with a predetermined classification (“indicator”). In one example, an indicator is a superpixel with a rating determined by a human. To illustrate, the pixels of a superpixel are shown to a human and the human identifies the superpixel pixels as a cloud, shadow, or field. The bandwidth information and feature set for the superpixel is associated with the classification and can be used to train a classification model. In another example, an indicator is a superpixel with a rating determined by a previously trained model (“previous model”). To illustrate, the bandwidth information and feature set for a superpixel are fed into a trained model to determine a classification for the superpixel. In this example, the previous model produces a classification and the band information and feature set for the superpixel is associated with the classification. The bandwidth information and feature set for the superpixel is associated with the classification and can be used to train another classification model.

[0056] The CMG module (112) trains the classification model (116) using indicators (eg, previously labeled observed images). Each indicator has a unique rating and is associated with the band information and feature set for that indicator. The classification model (116) enters a series of indicators (eg 400,000) and determines that the latent information included in the band and feature set information for the indicators is associated with specific classifications.

[0057] The CMG module (112) can evaluate the precision and/or accuracy of a trained classification model (116) using an evaluation function. For example, the evaluation function can determine an accuracy and precision for the trained classification model (116). Precision is a measure of true positives with respect to all data points and precision is a measure of true positives with respect to the combination of true positives and false positives. In one example, the evaluation function determines that the accuracy of a trained classification model is 99.5% when using a set of input observed images (eg, PlanetScope image data).

[0058] Some other examples of an evaluation function are a confusion matrix or an error matrix. The output of the evaluation function can allow visualization of the performance of a machine learning algorithm. In one example, the confusion matrices and evaluation scores used to evaluate models are based on Leave-One-Farm-Out cross-validation and are at the pixel level, although other grouping mechanisms, e.g. k-folds, can be used. . In Leave-One-Farm-Out cross-validation, the observed images are divided into a training set and a test set. The classification model is trained on the training set and tested on the test set. In this example, observed images from the same agricultural farm will end up in either the training set or the test set. Thus, in training, the classification model will never see an observed image of the test set, preventing information leakage and inaccurate training. Figure 9 illustrates a confusion matrix for our cloud masking task. The rows of the confusion matrix represent the instances of the actual classes and the columns represent the instances of the predicted classes. The correctly identified classes are diagonally across the table. For example, in Figure 9 the number of correctly identified crop, cloud and shadow pixels are 4.82E+08, 7.41E+07 and 1.33E+07, respectively. From each row, we can also calculate the percentage of pixels that are misclassified as other classes. For example, the percentage of crop pixels detected as a cloud or shadow is 3.4%, the percentage of cloud pixels detected as a crop is 10.6%, and the percentage of shadow pixels detected as a crop is 11, two%. We can also derive the precision for each class from the confusion matrix. In multiclass classification, precision is the fraction of the correctly predicted class of all instances where the algorithm predicted that class (correctly and incorrectly). In this case, the precision of crop, cloud, and shadow are 0.98, 0.85, and 0.63.

[0059] Figure 5 illustrates a process to train a classification model, according to an example modality. In an exemplary embodiment, the client system (110) executes the process (500). In one exemplary embodiment, the CMG module (112) employs the classification model (e.g., classification model (116)) to determine a classification for superpixels of an observed image as "cloud", "shadow", or "field". ”.

[0060] A CMG module (112) of the client system (110) segments, in step 510, an observed image (e.g., observed image (210)) into a segmented image (e.g., segmented image (300)) including a series of segments (eg segments (310)). In one example, the CMG module (112) segments the observed image (210) based on RGB color similarity and pixel distance in the observed image (210). Segmentation results in pixels representing clouds, shadows and fields being grouped into superpixels. Generally, only superpixels including only cloud, shadow, or field pixels are selected to train the model.

[0061] The CMG module (112) determines, in step (520), a set of resources for each superpixel. The CMG module (112) calculates several features that can differentiate between pixels representing clouds, shadows and fields. These features can include local color features within each superpixel (e.g. R/G/B average and R/G/B ratios), global color features within the full image to provide information on the contrast between segments and its source image (eg average of full R/G/B image, etc.), transformed color spaces and vegetation indices, HSV color, LAB color, NDVI, Enhanced Vegetation Index (EVI), etc. ; and the day of the year the image is observed.

[0062] An actor determines, in step (530), a rating for each superpixel. The actor can be a human or a previously trained classification model.

[0063] The CMG module (112) introduces, in step (540), the superpixels classified into a multilayer perceptron (e.g., multilayer perceptron (410)) to train the multilayer perceptron to identify clouds, shadows and fields based on the bandwidth and feature set information for the superpixel. The CMG module (112) can guide training based on a proportion of the number of cloud, shadow and field superpixels found in the previously classified superpixels.

[0064] The CMG module (112) evaluates, in step (550), the capabilities of the trained multilayer perceptron using an evaluation function. The evaluation function determines the accuracy and/or precision of the multilayer perceptron in correctly determining a classification for a superpixel. Based on the result of the evaluation function, the multilayer perceptron can be trained later.

GENERATION OF A CLOUD MAP

[0065] Figure 6 illustrates a process to generate a cloud map, according to an example modality. In an exemplary embodiment, the client system (110) performs the process, at step (600), for generating a cloud map (e.g., cloud map (220)).

[0066] The client system (110) receives, in step (610), a request to generate the cloud map (for example, cloud map (220)). The client system (110) requests, in step (620), an observed image (e.g., observed image (210)) of the network system (120) via the network (150). The client system (110) receives the observed image (210) from the network system (120) in response. The observed image (210) is a satellite image of an agricultural field obtained by the observation system (140). In some embodiments, the client system (110) requests the observed image (210) from the observation system (140) and receives the observed image (210) from the observation system (140) in response.

[0067] A CMG module (112) of the client system (110) segments, in step (630), the observed image (210) into superpixels based on band information (e.g. RGB and NIR) for each pixel in the observed image (210). The CMG module (112) may employ a segmentation model (eg segmentation model (114)) to segment the observed image. The CMG module (112) determines a feature set for each of the determined superpixels.

[0068] The CMG module (112) introduces, in step (640), the bandwidth information and the resource set for each superpixel in a classification model (eg classification model (116)). In an exemplary embodiment, the classification model includes a junction (eg, set (300)) of multilayer perceptrons (eg, multilayer perceptrons (410)).

[0069] For each multilayer perceptron, the CMG module (112) determines, in step (650), a classification for each superpixel (e.g. classification (412)) based on the band and feature set information for each superpixel. The classification can be “cloud”, “shadow”, or “field”.

[0070] The CMG module (112) determines, in step (660), an aggregate classification (eg aggregate classification (402)) for each superpixel based on the classifications determined by each of the multilayer perceptrons. In one example, the CMG module (112) determines that the aggregate classification is the classification determined by the plurality of multilayer perceptrons and possibly the inclusion of other classification methodologies, such as XGBoost, for example. The CMG module (112) generates a cloud map using the aggregated classifications. The cloud map is the observed image with each pixel classified with its aggregate classification. The CMG model (112) applies, in step (670), the classification mask to an unknown image, in order to obtain the necessary cloud map for the observed image.

[0071] To create a cloud mask, all pixels are set to zero in the original image. Some pixels are subsequently identified as clouds and set to one, and shadows identified are set to two. A cloud mask is created and placed on top of the original image to extract the desired information.

[0072] Cloud pixels distort results by adding high pixel values, thus affecting imaging techniques that utilize all pixels. Shadow pixels decrease in intensity and can affect how the data is interpreted, but they do not have the big effect that cloud pixels have on the average of the data.

[0073] Quantitatively, removing cloud and shadow pixels allows applications that use imaging techniques (eg, crop health, yield prediction, and harvest information) to generate more accurate results. Pixels that affect product calculations are removed and therefore do not drastically change the results. Growers will acquire better information for their applications, which helps them make better agronomic decisions.

[0074] Qualitatively, cloud removal eliminates pixels with extra high values that distract attention from regions of valuable field information. High pixel intensities create poor data scaling, hiding important information and potentially overwhelming small details that can be missed by a producer viewing a monitor. Removing these high-value pixels can ultimately improve the decision-making process. If high-quality data is fed into applications that address crop health or pest health, for example, better agronomic decisions can be made.

EXAMPLE OF COMPUTER SYSTEM

[0075] Figure 7 is a block diagram illustrating the components of an example machine for reading and executing instructions from a machine-readable medium, such as might be used within a computing system. Specifically, Figure 6 shows a schematic representation of the network system (120) and the client device (110) in the form of an example computer system (700). The computer system (700) may be used to execute instructions (724) (e.g., program code or software which may be in the form of a module) to cause the machine to perform any one or more of the methodologies (or processes) ) described here. In alternative embodiments, the machine operates as a standalone device or a connected (eg, networked) device that connects to other machines. In a network deployment, the machine may operate in the capacity of a server machine or a client machine in a client-server system environment (100), or as a peer-to-peer machine in a peer-to-peer system environment (or distributed) (100).

[0076] The machine can be a server computer, a client computer, a personal computer (PC), a tablet PC, a decoder (STB), a smartphone, an internet of things (IoT) device, a router, switch or network bridge, or any machine capable of executing instructions (724) (sequential or not) that specify actions to be taken by that machine. Furthermore, while only a single machine is illustrated, the term “machine” should also be considered to include any collection of machines that individually or together execute instructions (724) to execute any one or more of the methodologies discussed herein.

[0077] The example computer system (700) includes one or more processing units (usually processor (702)). The processor (702) is, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a controller, a state machine, one or more integrated circuits. Specific Applications (ASICs), one or more Radio Frequency Integrated Circuits (RFICs), or any combination thereof. The computer system (700) also includes main memory (704). The computer system may include a storage unit (716). The processor (702), memory (704) and storage unit (716) communicate via a bus (708).

[0078] In addition, the computer system (700) may include a static memory (706), a graphic display (710) (e.g. to drive a plasma display panel (PDP), a liquid crystal display ( LCD) or a projector). The computer system (700) may also include an alphanumeric input device (712) (e.g., a keyboard), a cursor control device (714) (e.g., a mouse, a trackball, a joystick, a motion or other pointing instrument), a signal generating device (718) (e.g., a speaker) and a network interface device (720), which are also configured to communicate over the bus (708).

[0079] The storage unit (716) includes a machine-readable medium (722) on which instructions (724) (e.g., software) are stored that incorporate any one or more of the methodologies or functions described herein. For example, instructions (724) may include the functionality of modules of the client device (110) or network system (120) depicted in Figure 1. Instructions (724) may also reside, completely or at least partially, within the main memory (704) or within the processor (702) (e.g., within a processor cache memory) during its execution by the computer system (700), the main memory (704) and the processor (702) also constituting machine readable media. Instructions (724) may be transmitted or received over a network (726) (e.g., network (120)) via the network interface device (720).

[0080] Although the machine-readable medium (722) is shown in an example embodiment as a single medium, the term "machine-readable medium" should be considered to include a single medium or multiple media (e.g., a bank of centralized or distributed data, or associated caches and servers) capable of storing instructions (724). The term "machine readable medium" shall also be understood to include any medium that is capable of storing instructions (724) for execution by the machine and that causes the machine to execute any one or more of the methodologies disclosed herein. The term “machine-readable medium” includes, but is not limited to, data repositories in the form of solid-state memories, optical media, and magnetic media.

[0081] Although several examples and embodiments have been shown and discussed throughout, the present invention contemplates numerous variations, options and alternatives.

Claims (20)

1. Method for shadow and cloud masking for remote sensing images of an agricultural field using multilayer perceptrons, wherein the method is characterized by the fact that it comprises: electronically receiving an observed image; performing, using at least one processor, an image segmentation of the observed image to divide the observed image into a plurality of image segments; extract resources for each of the image segments using the at least one processor; determine, by a cloud mask generation module running on at least one processor, a classification for each of the image segments using the extracted resources for each of the image segments, where the cloud mask generation module applies a classification model including a junction of multilayer perceptrons to generate a cloud mask for the observed image so that each pixel within the observed image has a corresponding classification. 2. Method according to claim 1, characterized in that the classification is selected from a set comprising a cloud classification, a shadow classification and a field classification. 3. Method according to claim 1, characterized in that the classification of each of the image segments is performed using five or less bands of the observed image. 4. Method according to claim 3, characterized in that the five or fewer bands include a visible band. 5. Method according to claim 4, characterized in that the five or fewer bands also include a near-infrared band. 6. Method according to claim 5, characterized in that the five or fewer bands also include a red border band. 7. Method, according to claim 1, characterized in that it also comprises applying the cloud mask to the observed image. 8. Method, according to claim 1, characterized in that it further comprises applying the cloud mask to the observed image and using a resulting image to generate a yield forecast for the agricultural field. 9. Method, according to claim 1, characterized in that it further comprises using the cloud generation module running on one or more processors to train the classification model. 10. Method, according to claim 1, characterized in that it further comprises using the cloud generation module running on one or more processors to evaluate one or more classification models. 11. Method according to claim 10, characterized in that the evaluation is performed using a confusion matrix. 12. Shadow and cloud masking system for remotely sensing images of an agricultural field, wherein the system is characterized in that it comprises: a computing system having at least one processor to run a cloud mask generation module , where the cloud mask generation module is configured to: receive an observed image; applying a segmentation model to the observed image to divide the observed image into a plurality of superpixels; extract resources from each of the superpixels; determine a classification for each of the superpixels using the features extracted for each of the superpixels and applying a classification model including a multilayer perceptron join to generate a cloud mask for the observed image so that each pixel in the observed image has a corresponding rating. 13. System according to claim 12, characterized in that the classification is selected from a set comprising a cloud classification, a shadow classification and a field classification. 14. System according to claim 13, characterized in that the classification of each of the superpixels is performed using five or fewer bands of the observed image. 15. System according to claim 14, characterized in that the classification of each of the superpixels is performed using a visible band. 16. System according to claim 15, characterized in that the classification of each of the superpixels is performed using a visible band, a near-infrared band and a red border band. 17. System, according to claim 12, characterized in that the computing system is further configured to apply the cloud mask to the observed image. 18. System according to claim 12, characterized in that the computing system is further configured to apply the cloud mask to the observed image and use a resulting image to generate a yield forecast for the agricultural field. 19. System, according to claim 12, characterized by the fact that the cloud generation module is still configured to train the classification model. 20. System according to claim 12, characterized in that the cloud generation model is still configured to evaluate one or more classification models.