CN117794832A - System and method for automatically orienting product containers - Google Patents

Abstract

A computer-implemented method for automatically orienting product containers, comprising: a reference product label image associated with a batch of product containers on a production line is acquired and a perception center of the reference product label image is calculated using a trained deep learning model based on perceptually relevant label features of the reference product label image. The line camera acquires image slices of corresponding product labels of individual product containers in the batch of product containers. A rotation angle for rotating the individual product containers is calculated based on the acquired image slices of the respective product labels and the label center determined based on the calculated perceived center of the reference product label image. The calculated rotation angle is transmitted to a controller to effect rotation of the individual product containers.

Description

Systems and methods for automatically orienting product containers

Technical field

Embodiments of the present disclosure relate to systems and methods for orienting product containers in a desired manner, such as for packaging purposes. In particular, one or more embodiments of the present disclosure relate to a technology that automates the orientation of product containers on a production line based on product label features and graphics.

Background technique

In many industrial production processes, especially in the food and beverage industry, products are transported in containers such as bottles, cans, etc. Product labels are typically attached around the outer surface of product containers and can include graphics/features such as brand logos, Universal Product Codes (UPC), nutritional facts, etc. Labeled containers can arrive at the packaging machine in a random orientation or rotation relative to the corresponding product label. It is desirable to orient the labeled containers delivered to the packaging machine in a visually consistent and/or attractive manner so that the final package or ready product arrangement can be optimally viewed or utilized by the end consumer.

Contents of the invention

Briefly, aspects of the present disclosure provide a technology that automatically orients product containers on a production line based on product label characteristics, reducing operator intervention or configuration.

A first aspect of the present disclosure provides a computer-implemented method for automatically orienting product containers on a production line, wherein each product container has a product label disposed about an exterior surface of the product container. The method includes obtaining a reference product label image associated with a batch of product containers on a production line. The method also includes using a trained deep learning model to calculate the perceptual center of the reference product label image based on the perceptually relevant label features of the reference product label image. The method also includes acquiring, via the production line camera, image slices of corresponding product labels for individual product containers in the batch of product containers. The method also includes calculating the angle of rotation of the individual product container based on the acquired image slice of the corresponding product label and the label center determined based on the calculated perceptual center of the reference product label image. The method also includes transmitting the calculated angle of rotation to the controller to effect rotation of the individual product containers from an initial orientation to a final orientation based on the calculated angle of rotation.

Other aspects of the disclosure implement features of the methods described above in the form of computing systems and computer program products.

Additional technical features and benefits may be achieved through the technology of the present disclosure. Embodiments and aspects of the disclosure are described in detail herein and are considered a part of the claimed subject matter. For better understanding, refer to the detailed description and accompanying drawings.

Description of drawings

The foregoing and other aspects of the disclosure are best understood from the following detailed description when read in conjunction with the accompanying drawings. To easily identify discussion of any element or action, one or more of the most significant digits in a reference number refers to the figure number in which the element or action is first introduced.

Figure 1 is a schematic diagram illustrating a system for automatically orienting product containers on a production line, according to an exemplary embodiment.

Figure 2 is an illustrative embodiment of a reference product label image.

Figure 3 is an illustrative visualization of perceptually relevant label features located on a reference product label image.

Figure 4 is an exemplary representation of the perceptual center of a reference product label image.

5 is a schematic flowchart illustrating an exemplary technique for training a rotation angle classifier deployed for calculating rotation angles at runtime.

Figure 6 is an exemplary representation of using a template matching algorithm at runtime to infer the location of a patch on a reference product label image that matches a captured image slice of the product label.

Figure 7 illustrates a computing system capable of supporting automated orientation of product containers in a production line in accordance with disclosed embodiments.

Detailed ways

Various techniques related to systems and methods will now be described with reference to the accompanying drawings, wherein like reference numerals refer to like elements throughout. The drawings discussed below and the various embodiments used to describe the principles of the disclosure in this patent document are merely illustrative and should not be construed in any way to limit the scope of the disclosure. Those skilled in the art will understand that the principles of the present disclosure may be implemented in any suitably arranged device. It should be understood that functions described as performed by certain system elements may be performed by multiple elements. Similarly, for example, one element may be configured to perform a function described as being performed by multiple elements. Many of the innovative teachings of the present application will be described with reference to exemplary, non-limiting embodiments.

A variety of container orientation systems exist to orient various shapes of cans, bottles, or other containers in a common direction. For example, a servo-based orientation system can perform this task at high speed using sensors (such as a vision system) to detect arbitrary actual container positions and then rotate the container accordingly to the desired orientation.

To configure an orientation system for a new product or label, a production engineer typically specifies a baseline product label/graphic for the orientation system, as well as the desired orientation of the product container relative to that label. This manual engineering step can limit how quickly a production line can switch from one product to another, or introduce new product labels for the same product.

Embodiments of the present disclosure provide an artificial intelligence (AI)-based solution that can identify which part of a product label should form the product label without any prior knowledge of the product label but with high accuracy. or the front of the product label displayed. The AI-based solution is based on the understanding that product labels are typically designed by marketing experts and graphic designers and therefore follow common design patterns and best practices to attract consumers’ attention. For example, certain label features (such as large text or product/brand logos) may often be parts of the product label intended to attract the consumer's eye, and therefore are intended to be positioned on the front of the product label. Additionally, certain label features (e.g. barcode/UPC, Nutrition Facts, etc.) may be designed to be positioned away from the front of the product label, e.g. for scanning/pricing reasons. For example, in a multi-pack product container, you may want the barcode of the individual container to face inward to prevent customers from scanning the individual product barcode instead of the package's barcode for pricing.

In at least some embodiments, the proposed AI-based solution can enable a fully automated system that can automatically determine the perceptual center of a reference product label and, when presented with randomly oriented product containers on a production line, Calculate the angle of rotation to rotate product containers into the correct orientation, for example for packaging purposes, without operator configuration or intervention. In some embodiments, depending on the calculated confidence level in the perceptual center of the product label, the AI-based solution may be assisted by user confirmation or using a user-specified front overlay of the product label.

The disclosed embodiments may reduce the overall engineering effort of automated system integration and label configuration for new product containers arriving on the production line, and thus may provide when switching from one product or product label to another on the production line Faster conversion. Furthermore, the disclosed embodiments can perform tag awareness with high accuracy while providing comparable runtime performance for inference and alignment as existing solutions.

Turning now to the drawings, FIG. 1 illustrates a system 100 for automatically orienting product containers 104 on a production line 102 according to an exemplary embodiment. In particular, the disclosed system 100 may be used to automatically orient labeled product containers 104 based on label features and graphics on the product label.

The product container 104 can have a variety of shapes, typically (but not necessarily) having an axisymmetric outer surface. For example, the most common types of product containers used in the food and beverage industry include cans, bottles, etc. A product label can be attached or otherwise disposed around the outer surface of each product container 104. The product label can extend completely around the outer surface of the product container 104 (i.e., 360 degrees). As a schematic diagram, in FIG. 1, the black and white boxes on the product container 104 depict various label features (e.g., brand logo, UPC code, etc.) of the corresponding product label.

As shown in Figure 1, incoming product containers 104 may have random orientations relative to their product labels. The randomly oriented product containers 104 may be individually rotated via the orientation unit 106 by a corresponding specified rotation angle to ensure that the output product containers 104 have a consistent orientation relative to the product label, as shown. In one exemplary configuration, the orientation unit 106 may include one or more grippers 108 (eg, suction-based grippers) for gripping individual product containers 104 and servo-controlled motors 110 coupled to the respective grippers 108 . The motors 110 may be individually controlled based on corresponding control signals 126 from the controller 124 for rotating the product container 104 held by the clamp by a corresponding designated rotation angle. In embodiments, the output (re)oriented product containers 104 may be conveyed to a packaging machine, where a group of product containers 104 may be packaged into a single package (eg, a six-pack of beverages). The disclosed embodiments are able to determine the angle of rotation of individual product containers based on AI so that the final package or ready product arrangement can be optimally viewed or utilized by the end consumer.

According to the illustrated embodiment, the system 100 may include a production line camera 112 that is arranged to scan each product container 104 transmitted to the production line 102 to obtain an image thereof. Due to the 3D shape of the product container 104, the production line camera 112 may only capture an image slice of the corresponding product label. As used in this specification, the term "image slice" refers to an image of a portion or segment of a product label acquired via a partial (e.g., 180 degrees) scan of the production line camera 112. The production line camera 112 may suitably include a high-speed camera that is configured to capture image slices of the corresponding product labels of individual product containers 104 in a batch of product containers on the production line 102. For example, in one embodiment shown herein, the production line camera 112 and its lighting system for generating a flash can be triggered based on a precisely timed trigger signal 128 from a controller 124, which can also control the position and speed of the product container 104 (e.g., the speed of the conveyor system). The image slices acquired for the individual product containers 104 can be transmitted as data signals 114 to the computing system 116 to calculate the corresponding rotation angles of those individual product containers 104 at runtime.

Computing system 116 may include, for example, an industrial PC, an edge computing device, or any other computing system that includes one or more processors and memory that stores algorithm modules executable by the one or more processors. The algorithm module at least includes a visual label perception engine 118 and a rotation angle calculation engine 120. The various engines described herein, including the visual label perception engine 118 and the rotation angle calculation engine 120 , including components thereof, may be implemented by the computing system 116 in various ways, such as as hardware and programming. Programming of the engines 118, 120 may be in the form of processor-executable instructions stored on a non-transitory machine-readable storage medium, and the engine's hardware may include a processor that executes those instructions. The processing capabilities of the systems, devices, and engines described herein, including the visual tag perception engine 118 and the rotation angle calculation engine 120, may be distributed among multiple system components, such as among multiple processors and memories, optionally Including multiple distributed processing systems or cloud/network elements. An embodiment of a computing system suitable for the present application is described below with reference to FIG. 7 .

Still referring to FIG. 1 , the visual label perception engine 118 may be configured to obtain a reference product label image associated with the batch of product containers 104 on the production line 102 and use trained deep learning based on the perceptually relevant label features of the reference product label image. model to calculate the perceptual center of a reference product label image. The visual tag awareness engine 118 may be implemented as part of an offline system for initial calibration setup. The rotation angle calculation engine 120 may be configured to calculate, at runtime, the center of the individual product container 104 based on the image slice of the corresponding product label acquired via the production line camera 112 and the label center determined based on the calculated perceptual center of the reference product label image. Rotation angle. The calculated perceptual center of the reference product label image may be determined as the label center used to calculate the rotation angle. In some embodiments, the label center used to calculate the rotation angle may be the calculated perceptual center, or may be derived from a user-specified front portion of the label, depending on the confidence level associated with the calculated perceptual center.

Computing system 116 may communicate the calculated angle of rotation of each product container 104 as data signal 122 to controller 124 . In embodiments, the controller 124 may include a programmable logic controller (PLC) or any other type of industrial controller configured to convert the data signal 122 into a control signal 126 and communicate the control signal 126 to the corresponding product container. Appropriate servo-controlled motors 110 are associated with 104 to effect rotation of the product container 104 from an initial orientation to a final orientation based on the calculated angle of rotation.

In one embodiment, as shown in Figure 1, a reference product label image for initial calibration may be obtained via a 360 degree scan of a reference product container associated with the batch of product containers. Often, but not necessarily, product labels can be the same for all product containers in the same batch. The reference product container may be one of the product containers 104 belonging to a new batch of product containers arriving on the production line 102, or a different product container representative of the batch. A reference product container (eg, product container 104) can be mounted on the rotating stage 130 so that the camera 132 can perform a 360-degree scan of the reference product label. A controller (eg, controller 124) may transmit control signals 134 to control the rotation and speed of the turntable 130 and issue appropriately timed trigger signals 136 to trigger the camera 132 and its lighting system for generating synchronized flashes. The 360-degree scanned reference product label image may be transmitted by the camera 132 as a data signal 138 to the computing system 116 to determine the perceptual center of the reference product label image using the visual label perception engine 118 . In an alternative embodiment, the reference product label image may be obtained directly from the product manufacturer.

FIG. 2 shows an illustrative embodiment of a reference product label image 200. For purposes of illustration, angular distances are represented with reference to the horizontal axis of the product label image 200 . As shown, the reference product label image 200 may include a two-dimensional planar or non-distorted image representing a 360-degree layout of the reference product label.

According to the disclosed embodiments, the visual tag perception engine 118 (FIG. 1) may utilize at least one deep learning model to detect perception-related tag features on the reference product tag image 200, which may include one or more pre-trained feature detection models/neural networks. In an exemplary embodiment, the deep learning model may be trained on a training dataset including images of multiple different product labels via a supervised learning process using classification labels for known perception-related tag features (of the dataset) for those images, so that the trained deep learning model may detect and locate perception-related tag features on the reference product tag image. In a suitable implementation, the training dataset may be obtained from an existing OpenLogo dataset that includes product labels for a large number of known products.

For model training, product labels in the training data set may be provided with bounding box annotations of different categories of perceptually relevant label features, which may define classification labels for the deep learning model. Perceptually relevant label features of different categories may include, but are not limited to, large text, brand logos, barcodes (e.g., UPC codes), geometric shapes, coarse or high color contrast features, nutritional information tables, and the like. The visual label perception engine 118 may use a trained deep learning model to detect and locate perceptually relevant label features on a reference product label image, for example, by a bounding box. As shown in FIG3 , an exemplary output obtained from a trained deep learning model may be visualized, wherein the detected perceptually relevant label features, i.e., large text, geometric shapes, nutritional information tables, and UPC codes, are located on the reference product label image 200 by bounding boxes 302 , 304 , 306 , 308 , respectively.

According to disclosed embodiments, perceptually relevant label features or feature categories may be grouped into "interesting" and "not interesting." "Interesting" label features may include those label features that define portions of the product label intended to form the front face of the product label. Examples of "interesting" label features include, but are not limited to, large text, large geometric shapes, high color contrast features, brand logos, etc. "Uninteresting" label features may include those label features that define portions of the product label intended to be positioned away from the front. Examples of "uninteresting" label features include, but are not limited to, barcodes (eg, UPC codes), tables (eg, nutritional facts), and the like.

By identifying the locations of "interesting" and "uninteresting" label features, the visual label perception engine 118 can algorithmically calculate the perceptual center of the reference product label image. In one embodiment, the visual label perception engine 118 may calculate the perceptual center of the reference product label image based on a weighted combination of the locations of perceptually relevant label features detected on the reference product label image. Herein, for example, each detected perceptually relevant label feature may be assigned a corresponding weight, which weight may depend on variables such as the size (eg, angular range) of the detected label feature, the "sense" of the detected label feature. "interesting" tag features or "not interested" tag features and other variables. In an embodiment, to ensure that the calculated perceptual center of the reference product label is positioned further away from the "uninteresting" label features, the "uninteresting" label features may be assigned a negative weight, or relative to the "interesting" Label features are usually assigned lower weights.

To illustrate an exemplary approach, in the embodiment shown in Figure 3, the location of each detected perceptually relevant label feature may be defined by the center of the corresponding bounding box 302, 304, 306, 308 along a horizontal or angular axis. These positions are denoted as X ₁ , X ₂ , X ₃ and X ₄ respectively in Figure 3 . The respective weight assigned to each of the detected perceptually relevant label features may be determined based on the width (ie, angular range) of the corresponding bounding box 302, 304, 306, 308 along the horizontal or angular axis. Additionally, weights associated with "interesting" label features (e.g., corresponding to bounding boxes 302, 304) may include positive values, whereas weights associated with "non-interesting" label features (e.g., corresponding to bounding boxes 306, 308 ) may include negative values. The position of the perceptual center may be determined as a weighted average of the positions X ₁ , X ₂ , X ₃ and X ₄ of each of the detected perceptually relevant label features. In some embodiments, the calculated perceptual center may be located by a bounding box with a defined width surrounding the location of the calculated perceptual center.

An exemplary representation of the calculated perceptual center of a reference product label is depicted in Figure 4. Herein, the perceptual center of the reference product label image 200 is located by a bounding box 402 with a defined width around position _XC . For example, as described above, position X _C may be determined based on a weighted combination of positions X ₁ , X ₂ , X ₃ and X ₄ . As shown, lower or negative weights assigned to "uninteresting" label features (corresponding to bounding boxes 306, 308) may cause the calculated perceptual center to be positioned further away from those "uninteresting" label features.

In an alternative embodiment, the visual label perception engine 118 can use a deep learning model (e.g., one or more neural networks) trained on a dataset including a large number of product-labeled images to calculate the perception center of the reference product label image, where the images of the dataset can each be annotated with a known label center. For example, the model can use an initial layer of a neural network to extract label features (which may include perception-related label features as described herein) and use subsequent layers to learn the spatial relationship between the extracted label features and the annotated label centers.

Thus, the visual label perception engine 118 can identify the location of the perceived center of a new product label without any prior knowledge of the product label or even the product or brand. Thus, the disclosed embodiments provide a completely general method that requires no user input to set up the system, wherein the AI-based visual label perception engine 118 can determine on its own what the "important" part of the image is for a customer to see based on a single image of a reference product label.

Continuing with reference to FIG. 1 , in some embodiments, the computing system 116 may communicate a data signal 142 representing the calculated perceptual center of the reference product label image to a human machine interface (HMI) device 140 . The HMI device 140 may be configured to visualize and display the label front to the operator based on the calculated perceptual center of the reference product label, thereby providing transparency to the underlying orientation process. The HMI device 140 may also be used to assist in the AI-based calculation of perception centers, particularly where there is a low confidence level associated with the calculated perception centers. The confidence level associated with the calculated perceptual center may, for example, be determined based on the confidence score associated with the detected perceptually relevant label features inferred by the deep learning model.

In one embodiment, if the determined confidence level associated with the calculated perceptual center is below a predetermined threshold, user input may be sought via the HMI device 140 for confirmation or overlaying the all with a user-specified label face. Displays the label front-center. User confirmation or override may be communicated by HMI device 140 as data signal 144 to computing system 116 . Based on user input transmitted via data signal 144, the label center used to calculate the rotation angle by the rotation angle calculation engine 120 may be either: the perceptual center calculated by the visual label perception engine 118 or the calculated user-specified label front. center. In at least some embodiments, if the determined confidence level associated with the calculated perceptual center is above a predetermined threshold, the calculated perceptual center can be used to automatically determine the angle of rotation of the individual product container without any operator intervention. .

At runtime, the rotation angle calculation engine 120 may perform an inline inference loop that may involve acquiring image slices of the product labels of each product container 104 on the production line 102 transmitted by the production line camera 112 (via the data signal 114), and calculating the desired offset rotation angle of the product container 104 based on the acquired image slices and the determined label center of the reference product label image. The calculated rotation angle may be used to implement the rotation of the product container 104 via the controller 124 to achieve a desired/consistent orientation relative to the corresponding product label, for example, as described above. An exemplary implementation of the rotation angle calculation engine 120 is described below.

In a first embodiment, the rotation angle calculation engine 120 may use a trained rotation angle classifier to infer the offset rotation angle from the acquired image slices. The rotation angle classifier may comprise a neural network, and suitably a shallow convolutional neural network capable of performing fast inference loops for high speed production lines. A rotation angle classifier can be trained using a dataset generated by performing augmentation of image patches corresponding to different regions of a reference product label image with varying angular offsets from the label center. Angular offsets can be used to define classification labels for image patches. A properly trained rotation angle classifier can provide high accuracy in calculating the required rotation angle for each product container.

5 shows an exemplary process 500 suitable for training a rotation angle classifier such as described above. The various modules described herein, including patcher 504, enhancer/data transformer 506, and offset calculator/label generator 512, including components thereof, can be implemented by a computing system such as computing system 116 in various ways, such as hardware and programming.

The patching program 504 may be used to generate image patches from the reference product label image 502. Each image patch may define an area of the reference product label image 502, for example, corresponding to a 180 degree field of view. In one exemplary embodiment, the image patches may be generated by cropping 360/(x-1) patches from the reference product label image 502, from 0° to (360°-x°) in steps of x°.

Enhancer/data transformer 506 can add noise and warping to image patches to simulate how product labels would look when wrapped around an axis-symmetric (eg, cylindrical) product container. Additionally, the enhancer/data transformer 506 can match extrinsic and intrinsic camera properties (eg, illumination, resolution, etc.) and simulate camera distortion on image patches to simulate actual image slices captured by the production line camera 112 . Enhancer/data transformer 506 can add and combine these modifications to generate a sufficiently large and realistic training data set 508.

The offset calculator/label generator 512 can determine an angular offset for each image patch relative to a label center 510 of the determined reference product label image 502. The determined label center 510 can be the calculated perceived center of the reference product image 502. Alternatively, if the confidence level associated with the calculated perceived center is low (e.g., below a predetermined threshold), as described above, for example, the label center 510 can be determined based on user input. For example, the angular offset can be determined by locating the center of each image patch and calculating the angular distance from the center of the image patch to the label center of the determined reference product label image 502. The angular offset thus determined can be used to define a classification label 514.

The training data set 508 and corresponding labels 514 may be used to train the rotation angle classifier 516 via a supervised learning process. Once trained and validated, the rotation angle classifier 516 can be deployed to the runtime system to automatically infer offset rotation angles from image slices acquired via the production line camera 112 .

In the second embodiment, the rotation angle calculation engine 120 may utilize a template matching algorithm. Template matching is a technique in computer vision for searching and finding the location of a template image within a larger parent image. In the context of this application, template matching algorithms can provide high runtime speed for calculating the required rotation angle for each product container. A suitable embodiment implementation may involve using the template matching functionality provided by the OpenCV library.

Figure 6 is an exemplary representation of using a template matching algorithm at runtime to infer the location of a patch on a reference product label image that matches a captured image slice of the product label. In the illustrated embodiment, image slice 602 represents a distorted image of a 180 degree field of view acquired via production line camera 112 (via data signal 114) having a portion of a product label wrapped around a cylindrical product container. The template matching algorithm may be performed using the image slice 602 as the template image and the reference product label image 200 as the parent image. When executed, the template matching algorithm may slide the template image 602 over the parent image 200 (eg, as in a 2D convolution) and align the template image 602 with a patch of the parent image 200 (eg, 180 degrees in this embodiment). patches) are compared to locate a matching patch 604 on the parent image 200 . In an implementation using OpenCV, depending on the template matching criterion used, the minMaxLoc function in OpenCV can be used to determine the matching patch 604 as a global minimum (when using TM_SQDIFF) or a global maximum (when using TM_CCORR or TM_CCOEFF hour).

The rotation angle may be determined by calculating the angular offset between the matching patch 604 and the determined label center. For example, the angular offset may be determined by locating the center X _M of the matching patch 604 and calculating the angular distance from the center of the matching patch 604 to the determined label center of the reference product label image 200 . In the illustrated embodiment, the determined label center is the calculated perceptual center X _C of the reference product label image 200 . Alternatively, if the confidence level associated with the calculated perceptual center _XC is low (eg, below a predetermined threshold), the label center may be determined based on user input, for example, as described above.

Figure 7 illustrates an embodiment of a computing system 700 capable of supporting automated orientation of product containers in a production line, in accordance with disclosed embodiments. In embodiments, computing system 700 may include one or more of industrial PCs, edge computing devices, and the like. Computing system 700 includes at least one processor 710, which may take the form of a single or multiple processors. Processor 710 may include a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a microprocessor, or any hardware device suitable for executing instructions stored on memory including machine-readable medium 720 . Machine-readable medium 720 may take the form of any non-transitory electronic, magnetic, optical, or other physical storage device that stores executable instructions, such as visual tag sensing instructions 722 and rotation angle calculation instructions 724, as shown in FIG. 7 . As such, machine-readable medium 720 may be, for example, random access memory (RAM) such as dynamic RAM (DRAM), flash memory, spin transfer torque memory, electrically erasable programmable read-only memory (EEPROM), storage drive, optical disk wait.

Computing system 700 may execute instructions stored on machine-readable medium 720 by processor 710 . Execution of instructions (eg, visual label awareness instructions 722 and rotation angle calculation instructions 724 ) may cause computing system 700 to perform any of the technical features described herein, including any features in accordance with visual label awareness engine 118 and rotation angle calculation engine 120 as described above. .

The systems, methods, devices, and logic described above, including the visual tag perception engine 118 and the rotation angle calculation engine 120, may be implemented in many different ways with many different combinations of hardware, logic, circuitry, and executable instructions stored on a machine-readable medium. . For example, these engines may include a controller, microprocessor, or circuitry in an Application Specific Integrated Circuit (ASIC), or may use discrete logic or components, or other types of combinations on a single integrated circuit or distributed among multiple integrated circuits. A combination of analog or digital circuits. A product, such as a computer program product, may include a storage medium and machine-readable instructions stored on the medium that, when executed in an endpoint, computer system, or other device, cause the device to perform an operation in accordance with any of the above descriptions, Operations based on any features of the visual label perception engine 118 and the rotation angle calculation engine 120 are included. The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing/processing device, or to an external computer or external storage device via a network (such as the Internet, a local area network, a wide area network, and/or a wireless network).

The processing capabilities of the systems, devices, and engines including visual tag perception engine 118 and rotation angle calculation engine 120 described herein may be distributed among multiple system components, such as among multiple processors and memories, optionally including Multiple distributed processing systems or cloud/network elements. Parameters, databases, and other data structures can be stored and managed separately, can be combined into a single memory or database, can be logically and physically organized in many different ways, and can be implemented in many ways, including as linked lists , hash table or implicit storage mechanism data structure. A program may be parts of a single program (eg, a subroutine), separate programs, distributed over several memories and processors, or implemented in many different ways, such as in the form of a library (eg, a shared library).

The systems and processes illustrated are not unique. Other systems, procedures, and menus may be derived from the principles of this disclosure to achieve the same purpose. Although the present disclosure has been described with reference to specific embodiments, it is to be understood that the embodiments and variations shown and described herein are for illustrative purposes only. Modifications to the current design may be implemented by those skilled in the art without departing from the scope of this disclosure.

Claims (15)

1. A computer-implemented method for automatically orienting product containers on a production line, wherein each product container has a product label disposed about an outer surface of the product container, the method comprising:

a reference product label image associated with a batch of product containers on the production line is acquired,

calculating a perception center of the reference product label image using a trained deep learning model based on perceptually relevant label features of the reference product label image,

image slices of corresponding product labels of individual product containers in the collection of product containers are acquired via a production line camera,

calculating a rotation angle of the individual product containers based on the acquired image slices of the respective product labels and the label center determined based on the calculated perceived center of the reference product label image, and

the calculated rotation angle is transmitted to the controller to effect rotation of the individual product containers from the initial orientation to the final orientation based on the calculated rotation angle.

2. The method of claim 1, wherein the reference product label image is acquired via a 360 degree scan of a reference product container associated with the collection of product containers.

3. The method of any of claims 1 and 2, wherein the deep learning model is trained on a dataset comprising images of a plurality of different product tags via a supervised learning process using classification tags of known perceptually relevant tag features of the images of the plurality of different product tags, the trained deep learning model being configured to detect and locate perceptually relevant tag features on the reference product tag image.

4. A method according to any one of claims 2 and 3, wherein the perceptually relevant label features include "interesting" label features defining portions of a product label forming a front face of the product label, and "uninteresting" label features defining portions of the product label located away from the front face.

5. The method of claim 4, wherein the "tag feature of interest" comprises one or more features selected from the group consisting of: large text, large geometry, and high color contrast features, and wherein the "uninteresting" tag features comprise one or more features selected from the group consisting of: bar codes and forms.

6. The method of any of claims 2 to 5, wherein the perceived center of the reference product label image is calculated based on a weighted combination of the respective locations of the perceptually relevant label features detected on the reference product label image.

7. The method of claim 6, wherein each detected perceptually relevant label feature is assigned a weight depending on whether the detected perceptually relevant label feature is a "interesting" label feature or a "non-interesting" label feature, and wherein the assigned weights are configured to position the calculated perceptually central away from the detected perceptually relevant label feature as a "non-interesting" label feature.

8. The method according to any one of claims 1 to 7, wherein the calculated perception center is determined as the label center for calculating the rotation angle.

9. The method of any of claims 1 to 7, further comprising:

based on the calculated perceived center of the reference product label, the label front is displayed via a human-machine interface (HMI) device,

if the determined confidence level associated with the computer perceived center is below a predetermined threshold, user input is sought via the HMI device to confirm or override the displayed label front center with the user-specified label front,

wherein the tag center for calculating the rotation angle is determined based on the user input as a calculated center of perception based on the deep learning model or a calculated center of a user-specified tag front face.

10. The method of any of claims 1 to 9, wherein the rotation angle is calculated by feeding the acquired image slices to a trained rotation angle classifier to produce an inferred output indicative of the rotation angle, wherein the rotation angle classifier is trained using a dataset produced by enhancing image patches corresponding to different regions of the reference product label image, the image patches having different angular offsets from the label center, the angular offsets being used as classification labels for the image patches.

11. The method according to any one of claims 1 to 9, wherein the rotation angle is calculated by:

performing a template matching algorithm using the acquired image slices as templates to locate matching patches in the reference product label image, and

an angular offset between the matching patch and the tag center is calculated.

12. A non-transitory computer readable storage medium comprising instructions which, when processed by a computer, configure the computer to perform the method of any one of claims 1 to 11.

13. A computing system, comprising:

at least one processor; and

a memory storing instructions that, when executed by the at least one processor, configure the computing system to perform the method of any one of claims 1 to 11.

14. A system for automatically orienting product containers on a production line, wherein each product container has a product label disposed about an outer surface of the product container, the system comprising:

a production line camera configured to capture image slices of corresponding product labels of individual product containers in a batch of product containers on the production line,

a computing system, comprising:

one or more processors

A memory storing an algorithm module executable by the one or more processors, the algorithm module comprising:

a visual tag perception engine configured to calculate a perception center of a reference product tag image based on perceptually relevant tag features of the reference product tag image using a trained deep learning model, and

a rotation angle calculation engine configured to calculate rotation angles of individual product containers based on the image slices of the respective product labels acquired via the production line camera and a label center determined based on a calculated perceived center of a reference product label image, and

a controller effecting rotation of the individual product containers from an initial orientation to a final orientation based on the calculated angle of rotation.

15. The system of claim 14, further comprising a human-machine interface (HMI) device configured to:

displaying a label front based on the calculated perceived center of the reference product label, and if the determined confidence level associated with the computer perceived center is below a predetermined threshold, seeking user input to confirm or use the user-specified label front to overlay the displayed label front center,

wherein the tag center used by a rotation angle calculation engine to calculate the rotation angle is determined based on the user input as a calculated perceived center based on the deep learning model or a calculated center of a user-specified tag front face.