GB2532075A - System and method for toy recognition and detection based on convolutional neural networks - Google Patents

System and method for toy recognition and detection based on convolutional neural networks Download PDF

Info

Publication number
GB2532075A
GB2532075A GB1419928.5A GB201419928A GB2532075A GB 2532075 A GB2532075 A GB 2532075A GB 201419928 A GB201419928 A GB 201419928A GB 2532075 A GB2532075 A GB 2532075A
Authority
GB
United Kingdom
Prior art keywords
toy
image
elements
learning
digital
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
GB1419928.5A
Other versions
GB201419928D0 (en
Inventor
Velic Marko
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Lego AS
Original Assignee
Lego AS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Lego AS filed Critical Lego AS
Priority to GB1419928.5A priority Critical patent/GB2532075A/en
Publication of GB201419928D0 publication Critical patent/GB201419928D0/en
Priority to EP24159537.0A priority patent/EP4350645A3/en
Priority to EP15791611.5A priority patent/EP3218076B1/en
Priority to EP20187091.2A priority patent/EP3744410B1/en
Priority to DK15791611.5T priority patent/DK3218076T3/en
Priority to US15/524,944 priority patent/US10213692B2/en
Priority to DK20187091.2T priority patent/DK3744410T3/en
Priority to PCT/EP2015/076073 priority patent/WO2016075081A1/en
Publication of GB2532075A publication Critical patent/GB2532075A/en
Priority to US16/258,739 priority patent/US10974152B2/en
Priority to US17/196,598 priority patent/US11794110B2/en
Priority to US18/297,140 priority patent/US12070693B2/en
Withdrawn legal-status Critical Current

Links

Classifications

    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/60—Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
    • A63F13/65—Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor automatically by game devices or servers from real world data, e.g. measurement in live racing competition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional objects
    • G06V20/647—Three-dimensional objects by matching two-dimensional images to three-dimensional objects
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/20—Input arrangements for video game devices
    • A63F13/21—Input arrangements for video game devices characterised by their sensors, purposes or types
    • A63F13/213—Input arrangements for video game devices characterised by their sensors, purposes or types comprising photodetecting means, e.g. cameras, photodiodes or infrared cells
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63H—TOYS, e.g. TOPS, DOLLS, HOOPS OR BUILDING BLOCKS
    • A63H33/00—Other toys
    • A63H33/04—Building blocks, strips, or similar building parts
    • A63H33/06—Building blocks, strips, or similar building parts to be assembled without the use of additional elements
    • A63H33/08—Building blocks, strips, or similar building parts to be assembled without the use of additional elements provided with complementary holes, grooves, or protuberances, e.g. dovetails
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/28—Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/04—Inference or reasoning models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/772—Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/66—Trinkets, e.g. shirt buttons or jewellery items

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Human Computer Interaction (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention concerns a system and method for automatic computer-aided optical recognition of toys, for example, construction toy elements. The aim is to detect toy elements 101,102 in digital images and associate the elements with existing information, possibly as part of a toy-to-life system. The method comprises: reading a digital image, representing a still image or a fragment of a video sequence; creating a learning database of digital images containing toy elements; recognizing a toy element; segmenting the image and detecting one or more toy elements (201,202) present in a digital image; matching the information from a central database or a web source with a recognized element. The learning and recognising steps may be achieved using a deep convolutional neural network. The method may advantageously recognize toy elements of various sizes invariant of distance, rotation, camera angle, background etc..

Description

Intellectual Property Office Application No. GII1419928.5 RTM Date:12 October 2015 The following terms are registered trade marks and should be read as such wherever they occur in this document: LEGO, LEGO technic, LEGO Duplo,K'Nex, Tinkertoys, Playskool Pipeworks, Cleversticks, Zometool, Erector Set, Meccano, Merkur, Steel Tec, Trix, FAC-System, tTherstix, Tog'1, Jovo Click 'N Construct, Zaks, Polydron, Geomag, toyblocks, Anchor Stone Blocks, KEVA planks, Kapla, UnitBricks, Rokenbok, Coco, Rasti, Tente, Mega Bloks, Fischertechnik, Playmobil, LocBlocs, Cobiblocks, BettaBuilda, Oxford, Kre0, Lincoln Logs, GIK, Sticklebricks, EnlightenBrick, Capsela.
Intellectual Property Office is an operating name of the Patent Office www.gov.uk /ipo System and method for toy recognition and detection based on convolutional neural networks
FIELD OF THE INVENTION
Present invention relates to application of computer vision technology in toy-to-life segment particularly the system and method for recognition and detection of construction toys that use modular elements based on dimensional constants and their assemblies based on convolutional neutral networks.
BACKGROUND OF THE INVENTION
Toy-to-life market segment currently involves systems wherein toys must have physical component configured to communicate with a special reader via some form of wireless communication like RFID, NFC etc. Examples of this technology are Skylanders, Disney Infinity or NintentoAmiibo. This invention presents a system where no additional hardware is needed to identify a toy element, and create its virtual digital representation and associate it with additional digital data. There are products which incorporate computer vision to accomplish this task, but these are not robust i.e. not invariant of toy element distance from the image acquiring device for example camera, not invariant of rotation of the toy element, not invariant of angle of the camera, not invariant of background, not invariant of illumination or need a predefined region where a toy element should be placed. Examples of this approach are LEGO Life of George and LEGO FUSION. LEGO FUSION, see web link: hitp://youtu.beAN41xL.VUE1qX8?1=21145s, is an example of the toy-to-life game where assembled model has to be placed on a special part i.e. scanning plate with specific pattern printed on it and image-acquiring device needs to be aligned to the specific angle from the toy element assembly for the recognition to take place. Document US885536B2 discloses method and apparatus for tracking three-dimensional (3D) objects. Method of tracking a 3D object includes constructing a database to store a set of two-dimensional (2D) images of the 3D object using a tracking background, where the tracking background includes at least one known pattern, receiving a tracking image, determining whether the tracking image matches at least one image in the database in accordance with feature points of the tracking image, and providing information about the tracking image in respond to the tracking image matches the at least one image in the database. The method of constructing a database also includes capturing the set of 2D images of the 3D object with the tracking background, extracting a set of feature points from each 2D image, and storing the set of feature points in the database. Present invention does not extract features points, any background or surface can be used and it is rotation invariant. Accordingly it is more robust, enables easier application without additional background elements and it is more accurate. Further according to present invention accuracy of recognition does not depend on rotation, size, scale, illumination or background change. Therefore result achieved by present invention is improvement of state of the art solutions in view of easier and faster object recognition without affecting accuracy based on convolutional neural networks.
Neural networks, especially deep convolutional neural networks are state of the art method for various object recognition tasks and the latest scientific work in this field demonstrates top performance of these kind of models based on results performing on the several recognized benchmarks.
The general purpose computing on graphics processing units (GPGPU) allows for training of very complex deep neural network architectures in reasonable time. This allows for training very accurate models invariant of various common image recognition problems like scale, rotation, illumination, background changes etc.
BRIEF SUMMARY OF THE INVENTION
In accordance with aspects of the invention, a system includes toy elements, an imager device, processor configured to execute detection and recognition tasks and identify one or more objects by utilizing the convolutional neural network. Convolutional neural networks allow for very accurate and robust object recognition and thus are very reliable method for recognition of toy elements and their assemblies invariant to various common problems in computer vision tasks like rotation, size, scale, illumination or background change.
One aspect of the invention provides a learning module i.e. a method for training a deep convolutional neural network to recognize various toy elements.
Another aspect of the invention provides a learning database containing many different variants of images of said toy elements with the accompanying information regarding the identification of toy element to allow for supervised learning.
Another aspect of the invention provides a recognition module, configured to use a learned model to recognize toy element on a new previously unseen image in a real time.
Another aspect of the invention provides a detection module, configured to perform image segmentation and recognize one or more toy elements on a digital image.
Another aspect of the invention provides a processing module configured to match recognized one or more toy parts with the information stored in the central database or on the web.
Another aspect of the invention provides a computer readable program encoded in a computer readable medium (e.g. a disk drive or other memory device). The computer readable program includes an executable computer program code configured to instruct a system to perform steps for learning, detection and object recognition described herein.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
Additional features and advantages will be made apparent from the following detailed description of embodiments that proceeds with reference to the accompanying drawings.
BRIEF DESCRIPTION OF THE FIGURES
Figure 1 presents one possible configuration of objects on the scene. As it can be seen on the left image of the figure there are three objects 101, 102 and 103. All these objects are segmented and correctly classified which is shown on the right image of the figure 1. Bounding boxes are placed around the objects and labels are put on the top left corner of the objects recognized. This is example of segmentation and recognition.
Figure 2 depicts one scenario where the system is used to classify hierarchical objects. As can be seen mini-figure is correctly recognized and then lower hierarchy level objects are recognized such as weapons, more specifically sword 201 and shield 202.
Figure 3 depicts steps needed to implement the method.
DETAILED DESCRIPTION
The following description relates to usage of convolutional neural network trained on central processing units ("CPUs") or graphics processing units ("GPUs") architectures to enable accurate and robust optical recognition of toy elements, particularly construction toys and parts. The one or more GPUs are used to quickly perform series of forward and backward passes on input data in parallel manner, modifying and refining the network parameters on each pass.
Mentioned construction toys are those that use modular elements based on dimensional constants, constraints and matches,with various assembly systems like magnets, studs, notches, sleeves, without interlocking connection etc. Examples of these systems include and are not limited to: LEGO, LEGO technic, LEGO Duplo,K'Nex, Tinkertoys, Playskool Pipeworks, Cleversticks, Zometool, Erector Set, Meccano, Merkur, Steel Tec, Trix, FAC-System, Uberstix, Tog'1, Jovo Click 'N Construct, Zaks, Polydron, Geomag, toyblocks, Anchor Stone Blocks, KEVA planks, Kapla, UnitBricks, Rokenbok, Coco, Rasti, Tente, Mega Bloks, Fischertechnik, Playmobil,LocBlocs, Cobiblocks, BettaBuilda, Oxford, Kre-O, Lincoln Logs, GIK, Sticklebricks,EnlightenBrick and Capsela. Patents LiS3005282 A and USD2 describe one suchinterlocking system and toyfigures.
In various embodiments of the invention, toy elements can be interlocking bricks, parts, accessories, mini-figures, weapons, animals, plants or other pieces that can be physically attached to form a toy assembly.
A system configured to automatically recognize and detect a real world toy elementse.g. LEGO brick, minifigure, minifigure part, weapon, object, accessories, animals or any other construction part or artefact from image or video in real time, the system comprising: one or more toy elements; an imager device; one or more processors configured to execute instructions of computer program modules. Processors further comprising: a reading module configured to receive image from digital image capture device;a learning database of annotated digital images containing toy elements taken in various conditions and altered with transformations to ensure significant variance for training procedure;a learning module configured to learn digital representations of toy elements by training a deep convolutional neural network thus enabling for very accurate and robust recognitions;a recognition module configured to recognize toy elements from a digital image;a detection module configured to detect and recognize one or more toy elements from a digital image; and a processing module configured to match recognized one or more toy parts with the information stored in the central database or on the web.
Reading module is configured to read an image, said image including an image of one or more real world toy elements; extract from said read image, said image of said one or more toy elements. Learning module is configured to learn digital representations of toy elements whose images are stored in the learning database and thus create a model for recognition wherein learning database contains annotated images where images are coupled with information about the toy elements present on said images. The learning database can be expanded by adding additional images of toy elements that exist in the database and new images of toy elements that do not exist in the database. Recognition module is configured to use the model for recognition to recognize toy element on a digital image. Further, detection module is configured to detect one or more toy elements on a digital image and their extract position represented by coordinates of the toy elements and extract the detected elements from the background and recognize the extracted elements. Processing module uses identification of the recognized element and matches that information with additional information stored in the central database or a web source. According to the present invention a learning module is implemented as a deep convolutional neural network comprising of convolutional, rectification, and normalization, pooling and fully connected layers of artificial neurons. The learning module implementation is a set of computer program instructions that can be ran on one or more central processing units ("CPU") or one or more graphics processing units ("GPU") and thus enabling for faster learning process.
A recognition model is configured to feedforward the said digital image data into a learned convolutional neural network causing a network to output prediction consisting of possibilities that said image contains certain objects from the said learning database. A detection module is configured to segment a digital image and detect one or more toy elements on a digital image and recognize each of them thus associating one or more said toy elements with the objects stored in learning database. A processing module is configured to match information about the recognized toy element with the information stored in a central database or on the World Wide Web via a web service or other network protocol. The recognition module is implemented as computer program that runs on a mobile device, or on a personal computer, or on a server.
A toy element is a construction toy part such as a minifigure, a minifigure part, a weapon, an accessory, a tool, an animal, a motor part, an electronic sensor part, or a construction brick, wherein a toy element is an assembly of more toy elements which are physically combinable.
Figure 1 is one possible scenario where various toy elements are placed on the surface and a digital image is captured via an imager device such as digital camera. Left image on the figure 1 depicts original scene with three different objects -barrel 101, goat 102 and a Viking minifigure holding a spear 103. Right image includes bounding boxes surrounding detected objects and labels on top of the rectangles showing results of the recognition task. As can be seen barrel 101 and goat 102 are recognized and regarding the minifigure, specific configuration of the Viking figure holding a spear is recognized 103. Figure 2 depicts a scenario where the same figure is recognized as holding a sword weapon.
Figure 2 depicts a scenario where the system is used to make hierarchical object recognition. At the top most level recognized object is Viking mini-figure holding a sword. At lower hierarchy level there are recognized objects sword 201 and shield 202, more specifically Viking's shield 202.
Figure 3 depicts workflow of a computer implemented method consisting of learning (a) and prediction (b) phases of the recognition task and segmentation (c) workflow for detection and recognition task.
A computer implemented method comprising the steps of: reading a digital image, said image representing still image captured with digital camera or a fragment of a video sequence captured with video camera; creating a learning database of digital images containing toy elements; learning from captured images with a goal to be able to recognize toy element if present on an image; recognizing toy element if the said element is present on an image that has not been used in the learning step; detecting one or more toy elements present on a said digital image; and matching the information from a central database or a web source with a recognized real world toy element.
Reading a digital image from an input device, where said image containing an image of a toy element is done by one or more processors. Creating of the learning database is done by taking many photographs of toy elements, annotating them with an information about which toy element is present on each image and subtracting the mean value to normalize the pixel colour intensities. Photographs of elements contain elements on various locations in an image showing toy elements captured from various angles, distances, rotations, from different cameras and in different illumination conditions. A leaming database is artificially expanded by creating new images from the existing ones by applying horizontal or vertical flips, scaling, rotation, changing colour intensities and affine transformations of existing images. Learning process is achieved by training a deep convolutional neural network whose hyper-parameters are chosen based on the performance on validation portion of the learning database. Training of deep convolutional neural network is conducted by a series of forward and backward passes of input information and gradients respectively, wherein deep convolutional neural network consists of convolutional, rectification, normalization, pooling, interconnected and softmax layers. Top level interconnected layers of the said network can be replaced by another classification algorithm which uses outputs from convolutional layers as inputs for classification. Further, recognizing the toy element from an input digital image is done by conducting a forward pass through trained neural network and output from the network represents probabilities for the existing classes being present on said image. Recognizing can be done as a series of forward passes of different subcrops of an input digital image and outputs from the trained network can be averaged to increase the prediction accuracy. Detection of the one or more toy elements is performed as one or more recognition after the segmentation of a digital image, wherein the segmentation is done by taking different subcrops of a digital image by sliding window or after edge detection and contours extraction or other segmentation algorithm. Matching information from a central database or a web source with a recognized real world toy element is conducted by querying a central database or a web service, wherein the matching information is price of an element, existing colours, construction sets where the element appears, buying locations and other information stored in the central database or a web source.
Learning phase is implemented as a series of training steps in which a deep convolutional neural network is optimized via an optimization algorithm like e.g. stochastic gradient descent. Learning database is created by capturing digital images of the toy elements which should be recognized along with the information regarding which object is present on each image in the database. Set of input images, along with an indication of a correct element each image should be interpreted as, is called annotated database.
Learning database is created by capturing large number ofimages of toy elements that need to be recognized on various surfaces with various backgrounds, different angles, zoom levels, rotations, scene positioning and different illumination conditions. Images could also be created by using more than one camera thus allowing for more variation in the learning database. The bigger and more various learning database is, the more reliable predictions by using the leamed model will be achieved as a measure of preventing neural network model from over-fitting the dataset. To achieve more robustness for different colour intensities, mean colour values from all the images are calculated and subtracted from each image and thus normalize the colour intensities in the learning database.
Next step is data augmentation in which every image from the learning database can be used to artificially generate more images through image processing steps like various distortions including but not limited to horizontal or vertical flipping, skewing (perception transform), resizing, rotating, colour intensity change and cropping. In this way more variance is introduced in the learning database.
During the learning process, each of the images from the learning database is used as an input into the convolutional neural network and the output is calculated to determine how close or far of the proper object recognition network is. Then the error of the network is propagated backwards throughout the network and modification of the network's parameters is done by using an optimization method e.g. stochastic gradient descent. This process is conducted as a series of matrix multiplications and partial derivative calculations.
Neural network that is capable of achieving very accurate recognition of objects on the images it has never seen before has to be assembled of several or many consecutive layers. These layers may include and are not limited to the following layer types: input layer, convolutional layers, pooling layers, rectification, normalization, nonlinear layers, interconnected layers, dropout layers, softmax layers and an output layer. Deep architecture of the neural network provides capabilities of recognizing complex shapes in hierarchical fashion similar to the process of object recognition in the visual cortex of animals or humans. Lower layers specialize for the detection of edges (gradients), colour and texture patterns while deeper layers specialize for recognition of the more complex formations and patterns including elements from lower layers. Consecutive usage of convolutional and pooling layers allows for the position invariance thus enabling object recognition on various locations in the image.
After the learning process has finished, model containing the optimized neural network can be stored on a digital media and be ready to use in recognition tasks. Figure 3.b shows a workflow where mentioned trained model is being used for the recognition task. After capturing an image, digital image is sent to the recognition module which uses learned model to recognize object i.e. toy element on an image. Network outputs probabilities that certain objects appear on the image in question. Once the toy element is recognized, its identity information can be used to acquire additional information about the object from other sources like databases or World Wide Web resources.
Figure 3.c depicts a workflow in which additional step is performed before the actual recognition in order to segment the image if more than one object have to be recognized. Segmentation can be done in several ways like extracting the contours after edge detection with e.g. Canny algorithm and then performing the above described recognition tasks for each image region where contour is found or by using the sliding window approach and gathering top prediction scores and thus identifying regions of the image which contain interesting objects, colour-based segmentation, selective search or any other segmentation method.
Described neural network training process can be achieved on any of a variety of devices in which digital signal processing can be performed. Examples of these devices include and are not limited to: desktop and laptop computers, servers, digital recording devices, gaming consoles, portable gaming devices, mobile devices etc. The mentioned storage for storing the executable files of the described system and for storing the learning database or trained model may be removable or non-removable. This includes magnetic disks, magnetic tapes or cassettes, solid state disks, CD or DVD ROMs, USB disks or any other medium which can be used to store information and which can be accessed within the computing environment.
Convolutional neural networks design involves a lot of hyper-parameters to be specified. These hyper-parameters include and are not limited to: number of neurons, number of layers, types of layers (convolutional, pooling, rectification, normalization, dropout, fully connected, softmax etc.) and their arrangement, learning rates, decay rates, number of learning iterations, batch sizes etc. Exact configuration of these parameters is best found via experiments on the particular problem that is being solved i.e. specific learning database that is being used by monitoring network's performance on the validation and test data sets. The various hyper-parameters of the deep convolutional neural network can be used in combination or independently.
Present invention further concerns a computer readable medium having stored thereon instructions which when executed by one or more central or graphics processing units cause performing the method of: reading a digital image, said image representing still image captured with digital camera or a fragment of a video sequence captured with video camera; creating a learning database of digital images containing toy elements; learning from captured images with a goal to be able to recognize toy element if present on an image; recognizing toy element if the said element is present on an image that has not been used in the learning step; detecting one or more toy elements present on a said digital image; and matching the information from a central database or a web source with a recognized real world toy element.
Although the invention has been described with reference to certain specific embodiments, various modifications thereof will be apparent to those skilled in art without departing from the spirit and scope of the invention as outlined in claims appended hereto.

Claims (42)

  1. Claims 1. A system configured to automatically recognize and detect real world toy construction elements, which can be interconnected or combinedsuch asbrick, minifigure, minifigure part, weapon, object, accessories, animals or any other construction part or artefact from image or video in real time, the system comprising: one or more toy elements; an imager device; one or more processors configured to execute instructions of computer program modules comprising: a reading module configured to receive image from digital image capture device; a learning database of annotated digital images containing toy elements taken in various conditions and altered with transformations to ensure significant variance for training procedure; a learning module configured to learn digital representations of toy elements by training a deep convolutional neural network thus enabling for very accurate and robust recognitions; a recognition module configured to recognize toy elements from a digital image; a detection module configured to segment image, detect and recognize one or more toy elements from a digital image; a processing module configured to match recognized one or more toy elements with the information stored in the central database or on the web.
  2. 2. The system of claim 1, wherein reading module is configured to read an image, said image including an image of one or more real world toy elements, extract from said read image one or more toy elements.
  3. 3. The system of claim 1, wherein learning module is configured to learn digital representations of toy elements whose images are stored in the learning database and create a model for recognition.
  4. 4. The system of claim 1, wherein a learning database contains annotated images wherein images are coupled with information relating to the toy elements present on said images.
  5. 5. The system of claims 1 to4 wherein the learning database is expanded by adding additional images of toy elements that exist in the database and new images of toy elements that do not exist in the database wherein new images of toy elements are annotated.
  6. 6. The system of claim 1 to4, wherein recognition module is configured to use the model for recognition to recognize toy element on a digital image.
  7. 7. The system of claim 1, wherein the detection module is configured to segment image and detect one or more toy elements on a digital image and extract position represented by coordinates of the toy elements and extract the detected elements from the background and recognize the extracted elements.
  8. 8. The system of claim 1, wherein the processing module uses identification of the recognized element and matches that information with additional information stored in the central database or a web source.
  9. 9. The system of preceding claims, wherein a learning module is implemented as a deep convolutional neural network comprising of convolutional, rectification, normalization, pooling and fully connected layers of artificial neurons.
  10. 10. The system of preceding claims, wherein a learning module implementation is a set of computer program instructions that can be ran on one or more central processing units ("CPU") or one or more graphics processing units ("GPU") and thus enabling for faster learning process.
  11. 11. The system of preceding claims, wherein a recognition model is configured to feedforward the said digital image data into a learned convolutional neural network causing a network to output prediction consisting of possibilities that said image contains certain objects from the said learning database.
  12. 12. The system of preceding claims, wherein a detection module is configured to segment a digital image and detect one or more toy elements on a digital image and recognize each of them thus associating one or more said toy elements with the objects stored in learning database.
  13. 13. The system of preceding claims, wherein a processing module is configured to match information about the recognized toy element with the information stored in a central database or on the World Wide Web via a web service or other network protocol.
  14. 14. The system of preceding claims, wherein the recognition module is implemented as computer program that runs on a mobile device.
  15. 15. The system of preceding claims, wherein the recognition module is implemented as computer program that runs on a personal computer.
  16. 16. The system of preceding claims, wherein the recognition module is implemented as computer program that runs on a server.
  17. 17. The system of claim 1, wherein a toy element is a construction toy part.
  18. 18. The system of claim 1, wherein a toy element is a minifigure.
  19. 19. The system of claim 1, wherein a toy element is a minifigure part.
  20. 20. The system of claim 1, wherein a toy element is a weapon.
  21. 21. The system of claim 1, wherein a toy element is an accessory.
  22. 22. The system of claim 1, wherein a toy element is a tool.
  23. 23. The system of claim 1, wherein a toy element is an animal.
  24. 24. The system of claim 1, wherein a toy element is a motor part.
  25. 25. The system of claim 1, wherein a toy element is an electronic sensor part.
  26. 26. The system of claim 1, wherein a toy element is a brick.
  27. 27. The system of claim 1, wherein a toy element is an assembly of more toy elements which are physically combinable.
  28. 28. A computer implemented method for automatic recognition and detection of real world toy construction elements which can be interconnected or combined such asbrick, minifigure, minifigure part, weapon, object, accessories, animals or any other construction part or artefact from image or video in real time, comprising the steps of: reading a digital image, said image representing still image captured with digital camera or a fragment of a video sequence captured with video camera; creating a learning database of digital images containing toy elements; learning from captured images with a goal to be able to recognize toy element if present on an image; recognizing toy element if the said element is present on an image that has not been used in the learning step; segmenting image and detecting one or more toy elements present on a said digital image; matching the information from a central database or a web source with a recognized real world toy element.
  29. 29. The method of claim 28, wherein reading a digital image from an input device, said image containing an image of a toy element is done by one or more processors.
  30. 30. The method of claim 28, wherein creating of the learning database is performed by taking large number ofdigital imagesof toy elements, annotating them with an information about which toy element is present on each image and subtracting the mean value to normalize the pixel colour intensities.
  31. 31. The method of claim 28 to30, wherein digital images of toy elements contain elements on various locations in an image showing toy elements captured from various angles, distances, rotations, from different cameras and in different illumination conditions.
  32. 32. The method of claim 28 to31 wherein a learning database is artificially expanded by creating new digital images from the existing ones by applying horizontal or vertical flips, scaling, rotation, changing colour intensities and affine transformations of existing images.
  33. 33. The method of claim 28, wherein learning process is achieved by training a deep convolutional neural network whose hyper-parameters are chosen based on the performance on validation portion of the learning database.
  34. 34. The method of claim 33, wherein training of deep convolutional neural network is conducted by a series of forward and backward passes of input information and gradients respectively.
  35. 35. The method of claim 33, wherein deep convolutional neural network consists of convolutional, rectification, normalization, pooling, interconnected and softmax layers.
  36. 36. The method of claim 33 and 35, wherein top level interconnected layers of the said network can be replaced by another classification algorithm which uses outputs from convolutional layers as inputs for classification.
  37. 37. The method of claim 28 to 36, wherein recognizing the toy element from an input digital image is done by conducting a forward pass through trained neural network and output from the network represents probabilities for the existing classes being present on said image.
  38. 38. The method of claim 28 to37, wherein recognizing can be done as a series of forward passes of different subcrops of an input digital image and outputs from the trained network can be averaged to increase the prediction accuracy.
  39. 39. The method of claim 28 to 38, wherein detection of the one or more toy elements is performed as one or more recognition after the segmentation of a digital image.
  40. 40. The method of claim 28 to39, wherein the segmentation is done by taking different subcrops of a digital image by sliding window or after edge detection and contours extraction or other segmentation algorithm.
  41. 41. The method of claim 28 to 40, wherein the matching information from a central database or a web source with a recognized real world toy element is conducted by querying a central database or a web service.
  42. 42. The method of claim 41, wherein the matching information is price of an element, existing colours, construction sets where the element appears, buying locations and other information stored in the central database or a web source.42. A computer readable medium having stored thereon instructions which when executed by one or more central or graphics processing units causing performing the method for automatic recognition and detection of real world toy construction elements which can be interconnected or combined such as brick, minifigure, minifigure pad, weapon, object, accessories, animals or any other construction part or artefact from image or video in real time comprising means for: reading a digital image, said image representing still image captured with digital camera or a fragment of a video sequence captured with video camera; creating a learning database of digital images containing toy elements; learning from captured images with a goal to be able to recognize toy element if present on an image; recognizing toy element if the said element is present on an image that has not been used in the learning step; segmenting image and detecting one or more toy elements present on a said digital image; matching the information from a central database or a web source with a recognized real world toy element.
GB1419928.5A 2014-11-10 2014-11-10 System and method for toy recognition and detection based on convolutional neural networks Withdrawn GB2532075A (en)

Priority Applications (11)

Application Number Priority Date Filing Date Title
GB1419928.5A GB2532075A (en) 2014-11-10 2014-11-10 System and method for toy recognition and detection based on convolutional neural networks
PCT/EP2015/076073 WO2016075081A1 (en) 2014-11-10 2015-11-09 System and method for toy recognition
DK15791611.5T DK3218076T3 (en) 2014-11-10 2015-11-09 System and method of toy recognition
EP15791611.5A EP3218076B1 (en) 2014-11-10 2015-11-09 System and method for toy recognition
EP20187091.2A EP3744410B1 (en) 2014-11-10 2015-11-09 System and method for toy recognition
EP24159537.0A EP4350645A3 (en) 2014-11-10 2015-11-09 System and method for toy recognition
US15/524,944 US10213692B2 (en) 2014-11-10 2015-11-09 System and method for toy recognition
DK20187091.2T DK3744410T3 (en) 2014-11-10 2015-11-09 SYSTEM AND METHOD FOR TOY RECOGNITION
US16/258,739 US10974152B2 (en) 2014-11-10 2019-01-28 System and method for toy recognition
US17/196,598 US11794110B2 (en) 2014-11-10 2021-03-09 System and method for toy recognition
US18/297,140 US12070693B2 (en) 2014-11-10 2023-04-07 System and method for toy recognition

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
GB1419928.5A GB2532075A (en) 2014-11-10 2014-11-10 System and method for toy recognition and detection based on convolutional neural networks

Publications (2)

Publication Number Publication Date
GB201419928D0 GB201419928D0 (en) 2014-12-24
GB2532075A true GB2532075A (en) 2016-05-11

Family

ID=52118221

Family Applications (1)

Application Number Title Priority Date Filing Date
GB1419928.5A Withdrawn GB2532075A (en) 2014-11-10 2014-11-10 System and method for toy recognition and detection based on convolutional neural networks

Country Status (5)

Country Link
US (4) US10213692B2 (en)
EP (3) EP3218076B1 (en)
DK (2) DK3218076T3 (en)
GB (1) GB2532075A (en)
WO (1) WO2016075081A1 (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017203262A3 (en) * 2016-05-25 2018-01-04 Metail Limited Method and system for predicting garment attributes using deep learning
WO2018007369A1 (en) * 2016-07-05 2018-01-11 Lego A/S Method for creating a virtual object
WO2018007351A1 (en) * 2016-07-05 2018-01-11 Lego A/S Method for creating a virtual object
EP3272401A1 (en) * 2016-07-22 2018-01-24 PlayFusion Limited Apparatus, system and method for enhancing a gaming experience
CN109190444A (en) * 2018-07-02 2019-01-11 南京大学 A kind of implementation method of the lane in which the drivers should pay fees vehicle feature recognition system based on video
EP3528432A1 (en) * 2018-02-16 2019-08-21 Nokia Solutions and Networks Oy Method and apparatus for monitoring a telecommunication network

Families Citing this family (265)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8677377B2 (en) 2005-09-08 2014-03-18 Apple Inc. Method and apparatus for building an intelligent automated assistant
US9318108B2 (en) 2010-01-18 2016-04-19 Apple Inc. Intelligent automated assistant
US8977255B2 (en) 2007-04-03 2015-03-10 Apple Inc. Method and system for operating a multi-function portable electronic device using voice-activation
US10002189B2 (en) 2007-12-20 2018-06-19 Apple Inc. Method and apparatus for searching using an active ontology
US9330720B2 (en) 2008-01-03 2016-05-03 Apple Inc. Methods and apparatus for altering audio output signals
US8996376B2 (en) 2008-04-05 2015-03-31 Apple Inc. Intelligent text-to-speech conversion
US20100030549A1 (en) 2008-07-31 2010-02-04 Lee Michael M Mobile device having human language translation capability with positional feedback
US8676904B2 (en) 2008-10-02 2014-03-18 Apple Inc. Electronic devices with voice command and contextual data processing capabilities
US10241752B2 (en) 2011-09-30 2019-03-26 Apple Inc. Interface for a virtual digital assistant
US20120311585A1 (en) 2011-06-03 2012-12-06 Apple Inc. Organizing task items that represent tasks to perform
US10276170B2 (en) 2010-01-18 2019-04-30 Apple Inc. Intelligent automated assistant
US8682667B2 (en) 2010-02-25 2014-03-25 Apple Inc. User profiling for selecting user specific voice input processing information
US9262612B2 (en) 2011-03-21 2016-02-16 Apple Inc. Device access using voice authentication
US10057736B2 (en) 2011-06-03 2018-08-21 Apple Inc. Active transport based notifications
US10134385B2 (en) 2012-03-02 2018-11-20 Apple Inc. Systems and methods for name pronunciation
US10417037B2 (en) 2012-05-15 2019-09-17 Apple Inc. Systems and methods for integrating third party services with a digital assistant
US9721563B2 (en) 2012-06-08 2017-08-01 Apple Inc. Name recognition system
AU2014214676A1 (en) 2013-02-07 2015-08-27 Apple Inc. Voice trigger for a digital assistant
US10652394B2 (en) 2013-03-14 2020-05-12 Apple Inc. System and method for processing voicemail
US10748529B1 (en) 2013-03-15 2020-08-18 Apple Inc. Voice activated device for use with a voice-based digital assistant
WO2014197334A2 (en) 2013-06-07 2014-12-11 Apple Inc. System and method for user-specified pronunciation of words for speech synthesis and recognition
WO2014197335A1 (en) 2013-06-08 2014-12-11 Apple Inc. Interpreting and acting upon commands that involve sharing information with remote devices
KR101922663B1 (en) 2013-06-09 2018-11-28 애플 인크. Device, method, and graphical user interface for enabling conversation persistence across two or more instances of a digital assistant
US10176167B2 (en) 2013-06-09 2019-01-08 Apple Inc. System and method for inferring user intent from speech inputs
DE112014003653B4 (en) 2013-08-06 2024-04-18 Apple Inc. Automatically activate intelligent responses based on activities from remote devices
US10296160B2 (en) 2013-12-06 2019-05-21 Apple Inc. Method for extracting salient dialog usage from live data
US9747727B2 (en) 2014-03-11 2017-08-29 Amazon Technologies, Inc. Object customization and accessorization in video content
US9633004B2 (en) 2014-05-30 2017-04-25 Apple Inc. Better resolution when referencing to concepts
US9966065B2 (en) 2014-05-30 2018-05-08 Apple Inc. Multi-command single utterance input method
US10170123B2 (en) 2014-05-30 2019-01-01 Apple Inc. Intelligent assistant for home automation
US9715875B2 (en) 2014-05-30 2017-07-25 Apple Inc. Reducing the need for manual start/end-pointing and trigger phrases
US9430463B2 (en) 2014-05-30 2016-08-30 Apple Inc. Exemplar-based natural language processing
US9338493B2 (en) 2014-06-30 2016-05-10 Apple Inc. Intelligent automated assistant for TV user interactions
US9818400B2 (en) 2014-09-11 2017-11-14 Apple Inc. Method and apparatus for discovering trending terms in speech requests
US10127911B2 (en) 2014-09-30 2018-11-13 Apple Inc. Speaker identification and unsupervised speaker adaptation techniques
US9668121B2 (en) 2014-09-30 2017-05-30 Apple Inc. Social reminders
US10074360B2 (en) 2014-09-30 2018-09-11 Apple Inc. Providing an indication of the suitability of speech recognition
US10152299B2 (en) 2015-03-06 2018-12-11 Apple Inc. Reducing response latency of intelligent automated assistants
US10567477B2 (en) 2015-03-08 2020-02-18 Apple Inc. Virtual assistant continuity
US9886953B2 (en) 2015-03-08 2018-02-06 Apple Inc. Virtual assistant activation
US9721566B2 (en) 2015-03-08 2017-08-01 Apple Inc. Competing devices responding to voice triggers
US10460227B2 (en) 2015-05-15 2019-10-29 Apple Inc. Virtual assistant in a communication session
US10083688B2 (en) 2015-05-27 2018-09-25 Apple Inc. Device voice control for selecting a displayed affordance
US10200824B2 (en) 2015-05-27 2019-02-05 Apple Inc. Systems and methods for proactively identifying and surfacing relevant content on a touch-sensitive device
US9578173B2 (en) 2015-06-05 2017-02-21 Apple Inc. Virtual assistant aided communication with 3rd party service in a communication session
US11025565B2 (en) 2015-06-07 2021-06-01 Apple Inc. Personalized prediction of responses for instant messaging
US11513658B1 (en) 2015-06-24 2022-11-29 Amazon Technologies, Inc. Custom query of a media universe database
US10970843B1 (en) * 2015-06-24 2021-04-06 Amazon Technologies, Inc. Generating interactive content using a media universe database
US20160378747A1 (en) 2015-06-29 2016-12-29 Apple Inc. Virtual assistant for media playback
US20230065252A1 (en) * 2015-08-17 2023-03-02 Lego A/S Toy system and a method of operating the toy system
CN108136257B (en) 2015-08-17 2021-09-21 乐高公司 Method for creating virtual game environment and interactive game system using the same
DK180058B1 (en) * 2018-07-06 2020-02-27 Lego A/S toy system
US10331312B2 (en) 2015-09-08 2019-06-25 Apple Inc. Intelligent automated assistant in a media environment
US10671428B2 (en) 2015-09-08 2020-06-02 Apple Inc. Distributed personal assistant
US10740384B2 (en) 2015-09-08 2020-08-11 Apple Inc. Intelligent automated assistant for media search and playback
US10747498B2 (en) 2015-09-08 2020-08-18 Apple Inc. Zero latency digital assistant
JP2018534953A (en) 2015-09-09 2018-11-29 リーチ ロボティックス リミテッドＲｅａｃｈ Ｒｏｂｏｔｉｃｓ Ｌｉｍｉｔｅｄ Game robot
US11587559B2 (en) 2015-09-30 2023-02-21 Apple Inc. Intelligent device identification
US10691473B2 (en) 2015-11-06 2020-06-23 Apple Inc. Intelligent automated assistant in a messaging environment
US10956666B2 (en) 2015-11-09 2021-03-23 Apple Inc. Unconventional virtual assistant interactions
US10860887B2 (en) * 2015-11-16 2020-12-08 Samsung Electronics Co., Ltd. Method and apparatus for recognizing object, and method and apparatus for training recognition model
US10049668B2 (en) 2015-12-02 2018-08-14 Apple Inc. Applying neural network language models to weighted finite state transducers for automatic speech recognition
US20170161591A1 (en) * 2015-12-04 2017-06-08 Pilot Ai Labs, Inc. System and method for deep-learning based object tracking
US10223066B2 (en) 2015-12-23 2019-03-05 Apple Inc. Proactive assistance based on dialog communication between devices
JP6333871B2 (en) * 2016-02-25 2018-05-30 ファナック株式会社 Image processing apparatus for displaying an object detected from an input image
US10891019B2 (en) * 2016-02-29 2021-01-12 Huawei Technologies Co., Ltd. Dynamic thumbnail selection for search results
US10424072B2 (en) * 2016-03-01 2019-09-24 Samsung Electronics Co., Ltd. Leveraging multi cues for fine-grained object classification
US11222461B2 (en) 2016-03-25 2022-01-11 Outward, Inc. Arbitrary view generation
US10163250B2 (en) 2016-03-25 2018-12-25 Outward, Inc. Arbitrary view generation
US9996914B2 (en) 2016-03-25 2018-06-12 Outward, Inc. Arbitrary view generation
US11232627B2 (en) 2016-03-25 2022-01-25 Outward, Inc. Arbitrary view generation
US11989820B2 (en) 2016-03-25 2024-05-21 Outward, Inc. Arbitrary view generation
US11989821B2 (en) 2016-03-25 2024-05-21 Outward, Inc. Arbitrary view generation
US10163249B2 (en) 2016-03-25 2018-12-25 Outward, Inc. Arbitrary view generation
US11972522B2 (en) 2016-03-25 2024-04-30 Outward, Inc. Arbitrary view generation
US10163251B2 (en) 2016-03-25 2018-12-25 Outward, Inc. Arbitrary view generation
US11062383B2 (en) 2016-05-10 2021-07-13 Lowe's Companies, Inc. Systems and methods for displaying a simulated room and portions thereof
CN106056562B (en) * 2016-05-19 2019-05-28 京东方科技集团股份有限公司 A kind of face image processing process, device and electronic equipment
GB2550911B (en) * 2016-05-27 2021-02-10 Swap Bots Ltd Augmented reality toy
US10579860B2 (en) 2016-06-06 2020-03-03 Samsung Electronics Co., Ltd. Learning model for salient facial region detection
US11227589B2 (en) 2016-06-06 2022-01-18 Apple Inc. Intelligent list reading
US10049663B2 (en) 2016-06-08 2018-08-14 Apple, Inc. Intelligent automated assistant for media exploration
US12223282B2 (en) 2016-06-09 2025-02-11 Apple Inc. Intelligent automated assistant in a home environment
US10586535B2 (en) 2016-06-10 2020-03-10 Apple Inc. Intelligent digital assistant in a multi-tasking environment
DK179415B1 (en) 2016-06-11 2018-06-14 Apple Inc Intelligent device arbitration and control
DK201670540A1 (en) 2016-06-11 2018-01-08 Apple Inc Application integration with a digital assistant
US12197817B2 (en) 2016-06-11 2025-01-14 Apple Inc. Intelligent device arbitration and control
DK179049B1 (en) * 2016-06-11 2017-09-18 Apple Inc Data driven natural language event detection and classification
US10353950B2 (en) 2016-06-28 2019-07-16 Google Llc Visual recognition using user tap locations
US10635973B1 (en) * 2016-06-28 2020-04-28 Amazon Technologies, Inc. Recommendation system using improved neural network
JP6708044B2 (en) * 2016-07-28 2020-06-10 富士通株式会社 Image recognition device, image recognition program, image recognition method, and recognition device
US10810494B2 (en) * 2016-08-08 2020-10-20 EyeEm Mobile GmbH Systems, methods, and computer program products for extending, augmenting and enhancing searching and sorting capabilities by learning and adding concepts on the fly
US20180068329A1 (en) * 2016-09-02 2018-03-08 International Business Machines Corporation Predicting real property prices using a convolutional neural network
US10474753B2 (en) 2016-09-07 2019-11-12 Apple Inc. Language identification using recurrent neural networks
US10043516B2 (en) 2016-09-23 2018-08-07 Apple Inc. Intelligent automated assistant
US10220326B2 (en) * 2016-09-29 2019-03-05 Intel Corporation Projections that respond to model building
US20180114109A1 (en) * 2016-10-20 2018-04-26 Nokia Technologies Oy Deep convolutional neural networks with squashed filters
KR101851169B1 (en) * 2016-11-14 2018-06-11 대한민국 Egg conveyer monitoring system and control method
US11132529B2 (en) * 2016-11-16 2021-09-28 Ventana Medical Systems, Inc. Convolutional neural networks for locating objects of interest in images of biological samples
US11281993B2 (en) 2016-12-05 2022-03-22 Apple Inc. Model and ensemble compression for metric learning
US10593346B2 (en) 2016-12-22 2020-03-17 Apple Inc. Rank-reduced token representation for automatic speech recognition
US11204787B2 (en) 2017-01-09 2021-12-21 Apple Inc. Application integration with a digital assistant
CN106815601B (en) * 2017-01-10 2019-10-11 西安电子科技大学 Hyperspectral Image Classification Method Based on Recurrent Neural Network
WO2018141956A1 (en) 2017-02-06 2018-08-09 Lego A/S Electronic ordering system and method
US10963783B2 (en) * 2017-02-19 2021-03-30 Intel Corporation Technologies for optimized machine learning training
US10339443B1 (en) * 2017-02-24 2019-07-02 Gopro, Inc. Systems and methods for processing convolutional neural network operations using textures
WO2018170512A1 (en) * 2017-03-17 2018-09-20 Neurala, Inc. Online, incremental real-time learning for tagging and labeling data streams for deep neural networks and neural network applications
US10163043B2 (en) 2017-03-31 2018-12-25 Clarifai, Inc. System and method for facilitating logo-recognition training of a recognition model
WO2018195583A1 (en) 2017-04-26 2018-11-01 Emmet Pty Ltd Construction system and method
US10572773B2 (en) * 2017-05-05 2020-02-25 Intel Corporation On the fly deep learning in machine learning for autonomous machines
DK201770383A1 (en) 2017-05-09 2018-12-14 Apple Inc. User interface for correcting recognition errors
US10417266B2 (en) 2017-05-09 2019-09-17 Apple Inc. Context-aware ranking of intelligent response suggestions
US10726832B2 (en) 2017-05-11 2020-07-28 Apple Inc. Maintaining privacy of personal information
DK201770439A1 (en) 2017-05-11 2018-12-13 Apple Inc. Offline personal assistant
DK180048B1 (en) 2017-05-11 2020-02-04 Apple Inc. MAINTAINING THE DATA PROTECTION OF PERSONAL INFORMATION
US10395654B2 (en) 2017-05-11 2019-08-27 Apple Inc. Text normalization based on a data-driven learning network
US11301477B2 (en) 2017-05-12 2022-04-12 Apple Inc. Feedback analysis of a digital assistant
DK179496B1 (en) 2017-05-12 2019-01-15 Apple Inc. USER-SPECIFIC Acoustic Models
DK179745B1 (en) 2017-05-12 2019-05-01 Apple Inc. SYNCHRONIZATION AND TASK DELEGATION OF A DIGITAL ASSISTANT
DK201770429A1 (en) 2017-05-12 2018-12-14 Apple Inc. Low-latency intelligent automated assistant
DK201770431A1 (en) 2017-05-15 2018-12-20 Apple Inc. Optimizing dialogue policy decisions for digital assistants using implicit feedback
DK201770411A1 (en) 2017-05-15 2018-12-20 Apple Inc. MULTI-MODAL INTERFACES
DK201770432A1 (en) 2017-05-15 2018-12-21 Apple Inc. Hierarchical belief states for digital assistants
US20180336275A1 (en) 2017-05-16 2018-11-22 Apple Inc. Intelligent automated assistant for media exploration
US10403278B2 (en) 2017-05-16 2019-09-03 Apple Inc. Methods and systems for phonetic matching in digital assistant services
US20180336892A1 (en) 2017-05-16 2018-11-22 Apple Inc. Detecting a trigger of a digital assistant
DK179560B1 (en) 2017-05-16 2019-02-18 Apple Inc. Far-field extension for digital assistant services
US10311144B2 (en) 2017-05-16 2019-06-04 Apple Inc. Emoji word sense disambiguation
KR102399673B1 (en) * 2017-06-01 2022-05-19 삼성전자주식회사 Method and apparatus for recognizing object based on vocabulary tree
US10657328B2 (en) 2017-06-02 2020-05-19 Apple Inc. Multi-task recurrent neural network architecture for efficient morphology handling in neural language modeling
EP3452955A4 (en) * 2017-06-06 2019-07-03 Midea Group Co., Ltd. Coarse-to-fine hand detection method using deep neural network
EP3270308B9 (en) * 2017-06-14 2022-05-18 Siemens Healthcare GmbH Method for providing a secondary parameter, decision support system, computer-readable medium and computer program product
JP7125983B2 (en) * 2017-07-14 2022-08-25 キャパシティー インコーポレイテッド Systems and methods for creating and displaying interactive 3D representations of real objects
JP7203844B2 (en) * 2017-07-25 2023-01-13 達闥機器人股▲分▼有限公司 Training data generation method, generation device, and semantic segmentation method for the image
US20210134049A1 (en) * 2017-08-08 2021-05-06 Sony Corporation Image processing apparatus and method
US10546387B2 (en) 2017-09-08 2020-01-28 Qualcomm Incorporated Pose determination with semantic segmentation
KR20200073222A (en) 2017-09-18 2020-06-23 엘리먼트, 인크. Method, system and medium for detecting spoofing in mobile authentication
US10445429B2 (en) 2017-09-21 2019-10-15 Apple Inc. Natural language understanding using vocabularies with compressed serialized tries
US10496902B2 (en) * 2017-09-21 2019-12-03 International Business Machines Corporation Data augmentation for image classification tasks
US10755051B2 (en) 2017-09-29 2020-08-25 Apple Inc. Rule-based natural language processing
EP3692470A4 (en) * 2017-10-02 2021-08-11 Sensen Networks Group Pty Ltd SYSTEM AND METHOD FOR OBJECT DETECTION BY MACHINE LEARNING
CN111417442B (en) 2017-10-16 2023-08-08 乐高公司 Interactive game device
CN111492374A (en) * 2017-10-24 2020-08-04 耐克创新有限合伙公司 Image recognition system
CN110348428B (en) * 2017-11-01 2023-03-24 腾讯科技（深圳）有限公司 Fundus image classification method and device and computer-readable storage medium
CN110399929B (en) * 2017-11-01 2023-04-28 腾讯科技（深圳）有限公司 Fundus image classification method, fundus image classification apparatus, and computer-readable storage medium
KR102426491B1 (en) * 2017-11-07 2022-07-28 재단법인대구경북과학기술원 Image data processing apparatus using ensamble and fine tunning and controlling method thereof
JP2019096072A (en) * 2017-11-22 2019-06-20 株式会社東芝 Object detection device, object detection method and program
US10636424B2 (en) 2017-11-30 2020-04-28 Apple Inc. Multi-turn canned dialog
CN107832807B (en) * 2017-12-07 2020-08-07 上海联影医疗科技有限公司 Image processing method and system
US10192115B1 (en) 2017-12-13 2019-01-29 Lowe's Companies, Inc. Virtualizing objects using object models and object position data
WO2019121629A1 (en) 2017-12-19 2019-06-27 Lego A/S Play system and method for detecting toys
CN111557023B (en) * 2017-12-29 2025-03-14 交互数字Vc控股公司 Method and system for maintaining color calibration using common objects
US10733982B2 (en) 2018-01-08 2020-08-04 Apple Inc. Multi-directional dialog
US10733375B2 (en) 2018-01-31 2020-08-04 Apple Inc. Knowledge-based framework for improving natural language understanding
US10872399B2 (en) * 2018-02-02 2020-12-22 Nvidia Corporation Photorealistic image stylization using a neural network model
US10950007B2 (en) * 2018-02-08 2021-03-16 Hasbro, Inc. Color-based toy identification system
US10867214B2 (en) 2018-02-14 2020-12-15 Nvidia Corporation Generation of synthetic images for training a neural network model
KR102595787B1 (en) * 2018-02-27 2023-11-24 삼성전자주식회사 Electronic device and control method thereof
US10789959B2 (en) 2018-03-02 2020-09-29 Apple Inc. Training speaker recognition models for digital assistants
US10909599B2 (en) * 2018-03-08 2021-02-02 Capital One Services, Llc Systems and methods for car shopping using messaging framework
US10592604B2 (en) 2018-03-12 2020-03-17 Apple Inc. Inverse text normalization for automatic speech recognition
US10818288B2 (en) 2018-03-26 2020-10-27 Apple Inc. Natural assistant interaction
USD844394S1 (en) 2018-03-29 2019-04-02 Kraft Foods Group Brands Llc Mold
US10894342B2 (en) 2018-03-29 2021-01-19 Kraft Foods Group Brands Llc System and method for molding comestible building blocks
US10909331B2 (en) 2018-03-30 2021-02-02 Apple Inc. Implicit identification of translation payload with neural machine translation
US11017129B2 (en) 2018-04-17 2021-05-25 International Business Machines Corporation Template selector
US11145294B2 (en) 2018-05-07 2021-10-12 Apple Inc. Intelligent automated assistant for delivering content from user experiences
US10928918B2 (en) 2018-05-07 2021-02-23 Apple Inc. Raise to speak
US10984780B2 (en) 2018-05-21 2021-04-20 Apple Inc. Global semantic word embeddings using bi-directional recurrent neural networks
DK180639B1 (en) 2018-06-01 2021-11-04 Apple Inc DISABILITY OF ATTENTION-ATTENTIVE VIRTUAL ASSISTANT
US10892996B2 (en) 2018-06-01 2021-01-12 Apple Inc. Variable latency device coordination
DK179822B1 (en) 2018-06-01 2019-07-12 Apple Inc. Voice interaction at a primary device to access call functionality of a companion device
DK201870355A1 (en) 2018-06-01 2019-12-16 Apple Inc. Virtual assistant operation in multi-device environments
US11386266B2 (en) 2018-06-01 2022-07-12 Apple Inc. Text correction
US10496705B1 (en) 2018-06-03 2019-12-03 Apple Inc. Accelerated task performance
US11899722B2 (en) 2018-06-20 2024-02-13 Rakuten Group, Inc. Search system, search method, and program
EP3830713A1 (en) 2018-07-31 2021-06-09 Marvell Asia Pte, Ltd. Metadata generation at the storage edge
US20200065706A1 (en) * 2018-08-24 2020-02-27 Htc Corporation Method for verifying training data, training system, and computer program product
US11094079B2 (en) 2018-08-28 2021-08-17 Facebook Technologies, Llc Determining a pose of an object from RGB-D images
EP3620983B1 (en) * 2018-09-05 2023-10-25 Sartorius Stedim Data Analytics AB Computer-implemented method, computer program product and system for data analysis
US11010561B2 (en) 2018-09-27 2021-05-18 Apple Inc. Sentiment prediction from textual data
US11170166B2 (en) 2018-09-28 2021-11-09 Apple Inc. Neural typographical error modeling via generative adversarial networks
US10839159B2 (en) 2018-09-28 2020-11-17 Apple Inc. Named entity normalization in a spoken dialog system
CN110969217B (en) * 2018-09-28 2023-11-17 杭州海康威视数字技术股份有限公司 Method and device for image processing based on convolutional neural network
US11462215B2 (en) 2018-09-28 2022-10-04 Apple Inc. Multi-modal inputs for voice commands
US20200118077A1 (en) * 2018-10-10 2020-04-16 Adroit Worldwide Media, Inc. Systems, Method and Apparatus for Optical Means for Tracking Inventory
US11475898B2 (en) 2018-10-26 2022-10-18 Apple Inc. Low-latency multi-speaker speech recognition
CN109684911B (en) * 2018-10-30 2021-05-11 百度在线网络技术（北京）有限公司 Expression recognition method and device, electronic equipment and storage medium
CN112889084B (en) 2018-11-08 2023-05-23 Oppo广东移动通信有限公司 Method, system and computer readable medium for improving color quality of image
TWI717655B (en) * 2018-11-09 2021-02-01 財團法人資訊工業策進會 Feature determination apparatus and method adapted to multiple object sizes
US10810759B2 (en) * 2018-11-20 2020-10-20 International Business Machines Corporation Creating a three-dimensional model from a sequence of images
US11011257B2 (en) 2018-11-21 2021-05-18 Enlitic, Inc. Multi-label heat map display system
US10755128B2 (en) * 2018-12-18 2020-08-25 Slyce Acquisition Inc. Scene and user-input context aided visual search
CN109829969A (en) * 2018-12-27 2019-05-31 北京奇艺世纪科技有限公司 A kind of data capture method, device and storage medium
US11638059B2 (en) 2019-01-04 2023-04-25 Apple Inc. Content playback on multiple devices
CN111428729A (en) * 2019-01-09 2020-07-17 北京京东尚科信息技术有限公司 Target detection method and device
US10694053B1 (en) 2019-01-22 2020-06-23 Xerox Corporation Wireless location tracking tag for monitoring real time location-tracking apparatus for an electronic device
CN113382790B (en) * 2019-01-23 2023-10-03 乐高公司 Toy system for augmented reality
US10535201B1 (en) * 2019-02-15 2020-01-14 Capital One Services, Llc Utilizing machine learning to generate augmented reality vehicle information for a scale model of a vehicle
US11410078B2 (en) * 2019-03-11 2022-08-09 Nxp B.V. Method and data processing system for making machine learning model more resistent to adversarial examples
US11343277B2 (en) 2019-03-12 2022-05-24 Element Inc. Methods and systems for detecting spoofing of facial recognition in connection with mobile devices
CN111768214B (en) * 2019-03-14 2025-01-17 北京京东尚科信息技术有限公司 Product attribute prediction method, system, device and storage medium
US11348573B2 (en) 2019-03-18 2022-05-31 Apple Inc. Multimodality in digital assistant systems
US10936903B2 (en) * 2019-03-30 2021-03-02 Intel Corporation Technologies for labeling and validating human-machine interface high definition-map data
EP3719697B1 (en) * 2019-04-04 2023-10-18 Aptiv Technologies Limited Method and device for determining whether a hand cooperates with a manual steering element of a vehicle
CN111814514A (en) * 2019-04-11 2020-10-23 富士通株式会社 Number identification device, method and electronic device
US11423908B2 (en) 2019-05-06 2022-08-23 Apple Inc. Interpreting spoken requests
US11307752B2 (en) 2019-05-06 2022-04-19 Apple Inc. User configurable task triggers
DK201970509A1 (en) 2019-05-06 2021-01-15 Apple Inc Spoken notifications
US11475884B2 (en) 2019-05-06 2022-10-18 Apple Inc. Reducing digital assistant latency when a language is incorrectly determined
US11972466B2 (en) * 2019-05-20 2024-04-30 Adobe Inc Computer storage media, method, and system for exploring and recommending matching products across categories
US11140099B2 (en) 2019-05-21 2021-10-05 Apple Inc. Providing message response suggestions
US11496600B2 (en) 2019-05-31 2022-11-08 Apple Inc. Remote execution of machine-learned models
US11289073B2 (en) 2019-05-31 2022-03-29 Apple Inc. Device text to speech
DK201970511A1 (en) 2019-05-31 2021-02-15 Apple Inc Voice identification in digital assistant systems
DK180129B1 (en) 2019-05-31 2020-06-02 Apple Inc. USER ACTIVITY SHORTCUT SUGGESTIONS
US11227599B2 (en) 2019-06-01 2022-01-18 Apple Inc. Methods and user interfaces for voice-based control of electronic devices
US11360641B2 (en) 2019-06-01 2022-06-14 Apple Inc. Increasing the relevance of new available information
US11361505B2 (en) * 2019-06-06 2022-06-14 Qualcomm Technologies, Inc. Model retrieval for objects in images using field descriptors
TW202113681A (en) * 2019-06-07 2021-04-01 德商巴斯夫塗料有限責任公司 Device and method for forming at least one ground truth database for an object recognition system
DE102019208864A1 (en) * 2019-06-18 2020-12-24 Siemens Mobility GmbH Detection system, working procedures and training procedures
GB201908762D0 (en) * 2019-06-19 2019-07-31 Kemp Little Llp Item comparison system device and method therefor
US11205319B2 (en) 2019-06-21 2021-12-21 Sg Gaming, Inc. System and method for synthetic image training of a neural network associated with a casino table game monitoring system
WO2021016923A1 (en) * 2019-07-31 2021-02-04 东北大学 Data enhancement method employing bit plane separation and recombination
US11481633B2 (en) * 2019-08-05 2022-10-25 Bank Of America Corporation Electronic system for management of image processing models
EP4470865A3 (en) 2019-09-17 2025-03-05 Aptiv Technologies AG Method and device for determining an estimate of the capability of a vehicle driver to take over control of a vehicle
US11488406B2 (en) 2019-09-25 2022-11-01 Apple Inc. Text detection using global geometry estimators
EP4055525A4 (en) * 2019-11-08 2024-02-21 Outward, Inc. Arbitrary view generation
WO2021113408A1 (en) * 2019-12-03 2021-06-10 Augustus Intelligence Inc. Synthesizing images from 3d models
CN114730324A (en) 2019-12-13 2022-07-08 马维尔亚洲私人有限公司 Automotive data processing system with efficient metadata generation and derivation
US11507248B2 (en) 2019-12-16 2022-11-22 Element Inc. Methods, systems, and media for anti-spoofing using eye-tracking
CN111091492B (en) * 2019-12-23 2020-09-04 韶鼎人工智能科技有限公司 Face image illumination migration method based on convolutional neural network
US11694333B1 (en) * 2020-02-05 2023-07-04 State Farm Mutual Automobile Insurance Company Performing semantic segmentation of 3D data using deep learning
KR102348179B1 (en) * 2020-03-02 2022-01-06 손준 Image saving method for offline objects
US11244470B2 (en) * 2020-03-05 2022-02-08 Xerox Corporation Methods and systems for sensing obstacles in an indoor environment
CN111569441A (en) * 2020-04-08 2020-08-25 深圳市魔块智能有限公司 Intelligent toy car track system and control method thereof
US11183193B1 (en) 2020-05-11 2021-11-23 Apple Inc. Digital assistant hardware abstraction
US11061543B1 (en) 2020-05-11 2021-07-13 Apple Inc. Providing relevant data items based on context
US20230281865A1 (en) * 2020-06-03 2023-09-07 Slab Dream Lab, Llc Systems and methods for optical recognition and identification of objects, and inventorying of the same
US11597408B2 (en) 2020-07-09 2023-03-07 Aptiv Technologies Limited Vehicle control system
CN111768478B (en) * 2020-07-13 2023-05-30 腾讯科技（深圳）有限公司 Image synthesis method and device, storage medium and electronic equipment
CN111666957B (en) * 2020-07-17 2023-04-25 湖南华威金安企业管理有限公司 Image authenticity identification method and device
US11490204B2 (en) 2020-07-20 2022-11-01 Apple Inc. Multi-device audio adjustment coordination
US11438683B2 (en) 2020-07-21 2022-09-06 Apple Inc. User identification using headphones
US11356800B2 (en) 2020-08-27 2022-06-07 Xerox Corporation Method of estimating indoor location of a device
WO2022101233A1 (en) 2020-11-16 2022-05-19 Lego A/S Toy system for multiplayer social play
US11138690B1 (en) * 2020-12-08 2021-10-05 Photobrick Pty. Ltd. Multi-image brick mosaic and systems and methods for production thereof
IT202100014645A1 (en) * 2021-06-04 2022-12-04 Rome C M S R L 3D modeling method and system
CN113327237B (en) * 2021-06-09 2025-04-08 合肥中科星翰科技有限公司 A visual inspection system for power circuit boards
CN113535155B (en) * 2021-07-16 2022-10-18 上海布鲁可积木科技有限公司 Building block programming system
US11868444B2 (en) 2021-07-20 2024-01-09 International Business Machines Corporation Creating synthetic visual inspection data sets using augmented reality
US11935199B2 (en) * 2021-07-26 2024-03-19 Google Llc Augmented reality depth detection through object recognition
CN113808262B (en) * 2021-10-08 2024-05-24 合肥安达创展科技股份有限公司 Building model generation system based on depth map analysis
US12136484B2 (en) 2021-11-05 2024-11-05 Altis Labs, Inc. Method and apparatus utilizing image-based modeling in healthcare
US20250001320A1 (en) 2021-11-16 2025-01-02 Lego A/S Sensor module for a modular toy construction set
US20230196737A1 (en) * 2021-12-30 2023-06-22 Industrial Technology Research Institute Image recognition method and electronic apparatus thereof
US20230241491A1 (en) * 2022-01-31 2023-08-03 Sony Interactive Entertainment Inc. Systems and methods for determining a type of material of an object in a real-world environment
US12211161B2 (en) 2022-06-24 2025-01-28 Lowe's Companies, Inc. Reset modeling based on reset and object properties
US12189915B2 (en) 2022-06-24 2025-01-07 Lowe's Companies, Inc. Simulated environment for presenting virtual objects and virtual resets
AU2023310357A1 (en) * 2022-07-18 2025-01-23 EMAGINEER Pty Ltd Improved construction system
US12151166B2 (en) * 2022-08-09 2024-11-26 Reuven Bakalash Integrated reality gamified applications
US12220643B2 (en) * 2022-08-09 2025-02-11 Reuven Bakalash Build and design-an integrated-reality educational gaming application
US12216703B2 (en) 2022-10-18 2025-02-04 Google Llc Visual search determination for text-to-image replacement
US20240249476A1 (en) * 2023-01-19 2024-07-25 Palo Alto Research Center Incorporated Method and system for facilitating generation of background replacement masks for improved labeled image dataset collection
CN117058241B (en) * 2023-10-10 2024-03-29 轩创(广州)网络科技有限公司 Electronic element positioning method and system based on artificial intelligence

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120155751A1 (en) * 2010-12-16 2012-06-21 Canon Kabushiki Kaisha Object recognition apparatus, object recognition method, learning apparatus, learning method, storage medium and information processing system
EP2521092A1 (en) * 2009-12-28 2012-11-07 Cyber Ai Entertainment Inc. Image recognition system
US20140294292A1 (en) * 2013-03-29 2014-10-02 Canon Kabushiki Kaisha Image processing apparatus and image processing method

Family Cites Families (45)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3005282A (en) 1958-01-28 1961-10-24 Interlego Ag Toy building brick
USD253711S (en) 1977-08-29 1979-12-18 Interlego Ag Toy figure
US6351265B1 (en) * 1993-10-15 2002-02-26 Personalized Online Photo Llc Method and apparatus for producing an electronic image
US6259815B1 (en) * 1999-03-04 2001-07-10 Mitsubishi Electric Research Laboratories, Inc. System and method for recognizing scanned objects with deformable volumetric templates
US6290565B1 (en) * 1999-07-21 2001-09-18 Nearlife, Inc. Interactive game apparatus with game play controlled by user-modifiable toy
US7092899B2 (en) * 2000-10-31 2006-08-15 Interlego Ag Method and system for generating a brick model
US8224078B2 (en) * 2000-11-06 2012-07-17 Nant Holdings Ip, Llc Image capture and identification system and process
US20020196250A1 (en) * 2001-06-20 2002-12-26 Gateway, Inc. Parts assembly for virtual representation and content creation
JP4509789B2 (en) 2002-10-11 2010-07-21 レゴ エー／エス Computer readable model
PL378138A1 (en) * 2003-05-20 2006-03-06 Lego A/S Method and system for manipulating a digital representation of a three-dimensional object
US7394459B2 (en) * 2004-04-29 2008-07-01 Microsoft Corporation Interaction between objects and a virtual environment display
JP2008502960A (en) * 2004-06-17 2008-01-31 レゴ エー／エス Automatic creation of assembly instructions for assembly block models
US20070097832A1 (en) * 2005-10-19 2007-05-03 Nokia Corporation Interoperation between virtual gaming environment and real-world environments
US9424488B2 (en) * 2007-06-14 2016-08-23 Hewlett-Packard Development Company, L.P. Applying a segmentation engine to different mappings of a digital image
US8452108B2 (en) * 2008-06-25 2013-05-28 Gannon Technologies Group Llc Systems and methods for image recognition using graph-based pattern matching
WO2014055924A1 (en) * 2012-10-04 2014-04-10 Disney Enterprises, Inc. Interactive objects for immersive environment
US8388151B2 (en) * 2009-07-23 2013-03-05 Kenneth J. Huebner Object aware, transformable projection system
US9498721B2 (en) 2009-08-04 2016-11-22 Eyecue Vision Technologies Ltd. System and method for object extraction
US8864589B2 (en) * 2009-10-27 2014-10-21 Activision Publishing, Inc. Video game with representative physical object related content
US10315119B2 (en) 2011-05-17 2019-06-11 Activision Publishing, Inc. Video game with concurrent processing of game-related physical objects
MX339520B (en) 2011-05-23 2016-05-30 Lego As A toy construction system for augmented reality.
MX2013013544A (en) * 2011-05-23 2014-05-27 Lego As Generation of building instructions for construction element models.
US20120304059A1 (en) * 2011-05-24 2012-11-29 Microsoft Corporation Interactive Build Instructions
US10796494B2 (en) * 2011-06-06 2020-10-06 Microsoft Technology Licensing, Llc Adding attributes to virtual representations of real-world objects
US8813111B2 (en) * 2011-08-22 2014-08-19 Xerox Corporation Photograph-based game
WO2013065045A1 (en) * 2011-10-31 2013-05-10 Eyecue Vision Technologies Ltd System for vision recognition based toys and games operated by a mobile device
US8855366B2 (en) 2011-11-29 2014-10-07 Qualcomm Incorporated Tracking three-dimensional objects
US9336456B2 (en) * 2012-01-25 2016-05-10 Bruno Delean Systems, methods and computer program products for identifying objects in video data
US20130217294A1 (en) * 2012-02-17 2013-08-22 Arjuna Ragunath Karunaratne Toy brick with sensing, actuation and control
US8436853B1 (en) * 2012-07-20 2013-05-07 Google Inc. Methods and systems for acquiring and ranking image sets
US20140098991A1 (en) * 2012-10-10 2014-04-10 PixArt Imaging Incorporation, R.O.C. Game doll recognition system, recognition method and game system using the same
US9039532B2 (en) * 2012-10-31 2015-05-26 Activision Publishing, Inc. Interactive video game with toys having functionality that is unlocked through game play
EP2749327A1 (en) 2012-12-26 2014-07-02 Disney Enterprises, Inc. Managing objectives, an environment, and/or a theme associated with a virtual space based on characters made accessible responsive to corresponding tokens being detected
US9076257B2 (en) * 2013-01-03 2015-07-07 Qualcomm Incorporated Rendering augmented reality based on foreground object
JP6134163B2 (en) 2013-03-08 2017-05-24 任天堂株式会社 Information processing apparatus, information processing system, information processing method, and computer program
US9610500B2 (en) * 2013-03-15 2017-04-04 Disney Enterprise, Inc. Managing virtual content based on information associated with toy objects
US9547871B2 (en) * 2013-03-15 2017-01-17 Activision Publishing, Inc. System and method for purchasing physical toys and corresponding virtual toys
US9011194B2 (en) * 2013-03-15 2015-04-21 Disney Enterprises, Inc. Managing virtual content based on information associated with toy objects
US9747307B2 (en) * 2013-11-18 2017-08-29 Scott Kier Systems and methods for immersive backgrounds
US9011246B1 (en) * 2013-11-18 2015-04-21 Scott Kier Systems and methods for immersive backgrounds
US9400924B2 (en) * 2014-05-23 2016-07-26 Industrial Technology Research Institute Object recognition method and object recognition apparatus using the same
US9861882B2 (en) * 2014-09-05 2018-01-09 Trigger Global Inc. Augmented reality gaming systems and methods
US9259651B1 (en) * 2015-02-13 2016-02-16 Jumo, Inc. System and method for providing relevant notifications via an action figure
AU2016250773A1 (en) * 2015-04-23 2017-10-12 Hasbro, Inc. Context-aware digital play
US9989965B2 (en) * 2015-08-20 2018-06-05 Motionloft, Inc. Object detection and analysis via unmanned aerial vehicle

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2521092A1 (en) * 2009-12-28 2012-11-07 Cyber Ai Entertainment Inc. Image recognition system
US20120155751A1 (en) * 2010-12-16 2012-06-21 Canon Kabushiki Kaisha Object recognition apparatus, object recognition method, learning apparatus, learning method, storage medium and information processing system
US20140294292A1 (en) * 2013-03-29 2014-10-02 Canon Kabushiki Kaisha Image processing apparatus and image processing method

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017203262A3 (en) * 2016-05-25 2018-01-04 Metail Limited Method and system for predicting garment attributes using deep learning
US11083968B2 (en) 2016-07-05 2021-08-10 Lego A/S Method for creating a virtual object
WO2018007369A1 (en) * 2016-07-05 2018-01-11 Lego A/S Method for creating a virtual object
WO2018007351A1 (en) * 2016-07-05 2018-01-11 Lego A/S Method for creating a virtual object
US11779846B2 (en) 2016-07-05 2023-10-10 Lego A/S Method for creating a virtual object
CN109641150A (en) * 2016-07-05 2019-04-16 乐高公司 Method for creating virtual objects
US11433310B2 (en) 2016-07-05 2022-09-06 Lego A/S Method for creating a virtual object
CN109641150B (en) * 2016-07-05 2022-06-07 乐高公司 Method for creating virtual objects
EP3272401A1 (en) * 2016-07-22 2018-01-24 PlayFusion Limited Apparatus, system and method for enhancing a gaming experience
US11146344B2 (en) 2018-02-16 2021-10-12 Nokia Solutions And Networks Oy Method and apparatus for monitoring a telecommunication network
CN110166148A (en) * 2018-02-16 2019-08-23 诺基亚通信公司 Method and apparatus for monitoring telecommunication network
EP3528432A1 (en) * 2018-02-16 2019-08-21 Nokia Solutions and Networks Oy Method and apparatus for monitoring a telecommunication network
CN109190444B (en) * 2018-07-02 2021-05-18 南京大学 An implementation method of a video-based toll lane vehicle feature recognition system
CN109190444A (en) * 2018-07-02 2019-01-11 南京大学 A kind of implementation method of the lane in which the drivers should pay fees vehicle feature recognition system based on video

Also Published As

Publication number Publication date
EP4350645A2 (en) 2024-04-10
EP3218076B1 (en) 2020-07-29
WO2016075081A1 (en) 2016-05-19
GB201419928D0 (en) 2014-12-24
EP3744410B1 (en) 2024-02-28
EP3744410A1 (en) 2020-12-02
DK3744410T3 (en) 2024-05-27
US10974152B2 (en) 2021-04-13
US20190184288A1 (en) 2019-06-20
US10213692B2 (en) 2019-02-26
US20230264109A1 (en) 2023-08-24
US20170304732A1 (en) 2017-10-26
US11794110B2 (en) 2023-10-24
US12070693B2 (en) 2024-08-27
EP3218076A1 (en) 2017-09-20
US20210205707A1 (en) 2021-07-08
DK3218076T3 (en) 2020-10-26
EP4350645A3 (en) 2024-07-17

Similar Documents

Publication Publication Date Title
GB2532075A (en) System and method for toy recognition and detection based on convolutional neural networks
Lenc et al. Learning covariant feature detectors
Tang et al. A textured object recognition pipeline for color and depth image data
US10861129B2 (en) Image feature combination for image-based object recognition
Azad et al. 6-DoF model-based tracking of arbitrarily shaped 3D objects
CN106408037A (en) Image recognition method and apparatus
CN112036457B (en) Method and device for training target detection model, and target detection method and device
Yigitbasi et al. Edge detection using artificial bee colony algorithm (ABC)
CN106845494A (en) The method and device of profile angle point in a kind of detection image
Blanc et al. Fish species recognition from video using SVM classifier
Fiaz et al. Convolutional neural network with structural input for visual object tracking
Berral-Soler et al. DeepArUco: marker detection and classification in challenging lighting conditions
Yin et al. Interest point detection from multi‐beam light detection and ranging point cloud using unsupervised convolutional neural network
CN115018886B (en) Motion trajectory identification method, device, equipment and medium
Shen et al. A biological hierarchical model based underwater moving object detection
Ai et al. Geometry preserving active polygon-incorporated sign detection algorithm
Masouris et al. End-to-end chess recognition
Aubry et al. Visual geo-localization of non-photographic depictions via 2D–3D alignment
Tan et al. Research and implementation of sports entity simulation based on heterogeneous binocular vision
Bouachir et al. Visual face tracking: A coarse-to-fine target state estimation
Garrett Real-time generalized convolutional neural network for point cloud object tracking
Teixeira Analysis and evaluation of optimization techniques for tracking in augmented reality applications.
CN118378717A (en) System and method for training machine learning model
Dong et al. A Real-Time Algorithm for Multiple Data Matrix Codes Localization
Datta PROJECT ON ENTROPY BASED PIXEL CLUSTER FOR FACE TRACKING

Legal Events

Date Code Title Description
COOA Change in applicant's name or ownership of the application

Owner name: LEGO A/S

Free format text: FORMER OWNER: MARKO VELIC

WAP Application withdrawn, taken to be withdrawn or refused ** after publication under section 16(1)
OSZAR »