US7725451B2 - Generating clusters of images for search results - Google Patents

Generating clusters of images for search results Download PDF

Info

Publication number
US7725451B2
US7725451B2 US11/337,825 US33782506A US7725451B2 US 7725451 B2 US7725451 B2 US 7725451B2 US 33782506 A US33782506 A US 33782506A US 7725451 B2 US7725451 B2 US 7725451B2
Authority
US
United States
Prior art keywords
images
image
query
search engine
search
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related, expires
Application number
US11/337,825
Other versions
US20070174269A1 (en
Inventor
Feng Jing
Lei Zhang
Mingjing Li
Wei-Ying Ma
Chang-Hu Wang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Corp filed Critical Microsoft Corp
Priority to US11/337,825 priority Critical patent/US7725451B2/en
Assigned to MICROSOFT CORPORATION reassignment MICROSOFT CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: LI, MINGJING, WANG, CHANG-HU, JING, FENG, MA, WEI-YING, ZHANG, LEI
Publication of US20070174269A1 publication Critical patent/US20070174269A1/en
Application granted granted Critical
Publication of US7725451B2 publication Critical patent/US7725451B2/en
Assigned to MICROSOFT TECHNOLOGY LICENSING, LLC reassignment MICROSOFT TECHNOLOGY LICENSING, LLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: MICROSOFT CORPORATION
Expired - Fee Related legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Definitions

  • search engine services such as Google and Overture, provide for searching for information that is accessible via the Internet. These search engine services allow users to search for display pages, such as web pages, that may be of interest to users. After a user submits a search request (also referred to as a “query”) that includes search terms, the search engine service identifies web pages that may be related to those search terms. To quickly identify related web pages, the search engine services may maintain a mapping of keywords to web pages. This mapping may be generated by “crawling” the web (i.e., the World Wide Web) to identify the keywords of each web page. To crawl the web, a search engine service may use a list of base web pages to identify all web pages that are accessible through those base web pages.
  • the keywords of any particular web page can be identified using various well-known information retrieval techniques, such as identifying the words of a headline, the words supplied in the metadata of the web page, the words that are highlighted, and so on.
  • the search engine service may generate a relevance score to indicate how related the information of the web page may be to the search request.
  • the search engine service displays to the user links to those web pages in an order that is based on their relevance.
  • search engine services also provide for searching for images that are available on the Internet.
  • These image search engines typically generate a mapping of keywords to images by crawling the web in much the same way as described above for mapping keywords to web pages.
  • An image search engine service can identify keywords based on text of the web pages that contain the images.
  • An image search engine may also gather keywords from metadata associated with images of web-based image forums, which are an increasingly popular mechanism for people to publish their photographs and other images.
  • An image forum allows users to upload their photographs and requires the users to provide associated metadata such as title, camera setting, category, and description.
  • the image forums typically allow reviewers to rate each of the uploaded images and thus have ratings on the quality of the images.
  • an image search engine service inputs an image query and uses the mapping to find images that are related to the image query.
  • An image search engine service may identify thousands of images that are related to an image query and presents thumbnails of the related images. To help a user view the images, an image search engine service may order the thumbnails based on relevance of the images to the image query. An image search engine service may also limit the number of images that are provided to a few hundred of the most relevant images so as not to overwhelm the viewer.
  • an image search engine service could cluster a search result based on the content of the images and present the clusters, rather than individual images, to the user.
  • clustering techniques include content-based techniques and link-based techniques. The content-based techniques use low-level visual information to identify related images. There are, however, disadvantages to content-based clustering.
  • Content-based clustering is computationally expensive and cannot be practically performed in real time when an image search result contains thousands of images. Moreover, if the clustering is limited to a few hundred of what are thought to be the most relevant images, some very relevant images may be missed because of the difficulties in assessing relevance.
  • the link-based search techniques typically assume that images on the same web page are likely to be related and that images on web pages that are each linked to by the same web page are related. Since this assumption is, however, often not true, unrelated images are often clustered together. It is also difficult for either cluster technique to automatically identify meaningful names for a cluster of images. As a result, a user may not be able to effectively identify relevant clusters.
  • An image search system receives an original image query, identifies phrases related to the original image query, submits to an image search engine each of the identified phrases as an image query, and provides the images of each image search result as a cluster of images for the image search result of the original image query.
  • the search system identifies text associated with the original image query by submitting the original image query to a search engine.
  • the search system identifies phrases from the text of the web page containing the search result.
  • the search system uses each of the identified phrases as an image query and submits the image queries to an image search engine.
  • the search system considers the image search result for each image query to represent a cluster of related images.
  • the search system then presents the clusters of images as the images of the image search result of the original image query.
  • FIG. 1 is a block diagram that illustrates components of the search system in one embodiment.
  • FIG. 2 is a flow diagram that illustrates the processing of the search for images component of the search system in one embodiment.
  • FIG. 3 is a flow diagram that illustrates the processing of the search for text component of the search system in one embodiment.
  • FIG. 4 is a flow diagram that illustrates the processing of the identify clusters component of the search system in one embodiment.
  • an image search system receives an original image query, identifies phrases related to the original image query, submits to an image search engine each of the identified phrases as an image query, and provides the images of each image search result as a cluster of images for the image search result of the original image query.
  • the search system identifies text associated with the image query.
  • the search system may identify the text by submitting the original image query to a conventional web page search engine.
  • the conventional search engine searches for web pages that are related to the original image query and returns a web page that contains links to the related web pages possibly along with a snippet of text from those web pages.
  • the search system Upon receiving the web page containing the search result, the search system identifies phrases from the text of the web page containing the search result. For example, if the original image query is “tiger,” then the search result web page (as part of the links or in the snippets) may contain the phrases “white tiger,” “Tiger Woods,” “crouching tiger,” and so on.
  • the search system uses each of the identified phrases as an image query and submits the image queries to an image search engine.
  • the search system considers the image search result for each image query to represent a cluster of related images. For example, the search system submits to an image search engine the phrase “white tiger” as an image query and receives the image search result.
  • the search system considers images of the image search result to be a cluster of images relating to “white tiger.”
  • the search system also submits to an image search engine the phrase “Tiger Woods” as an image query and receives the image search result.
  • the search system considers the images of the image search result to be a cluster of images related to “Tiger Woods.”
  • the search system may also submit the original image query to the image search engine to generate an “others” cluster.
  • the search system may remove from the “others” cluster those images that are already in another cluster.
  • the search system submits “tiger” to the image search engine, removes from the image search result the images that are also in the “white tiger,” “Tiger Woods,” and “crouching tiger” image search results and forms an “others” cluster of the remaining images.
  • the search system then presents to a user the clusters of images as the images of the image search result of the original image query, such as “tiger.”
  • the search system may use the phrase of the image query as the name of the cluster.
  • one cluster may be named “white tiger,” and another cluster may be named “Tiger Woods.”
  • the search system can generate an image search result that has clusters of images without the computational expense of identifying clusters from a set of images and can provide meaningful names to the clusters.
  • the search system may identify phrases from query logs.
  • a query log contains the queries that have been submitted to a search engine.
  • the search system may identify the most popular queries that contain the original image query and use those as phrases. For example, the most popular queries that contain “tiger” may be “Tiger Woods” and “Detroit Tigers.”
  • the search system then submits each query to an image search engine and names the cluster of image search result with the text of the query.
  • the search system may present the images within each cluster in an order based on relevance as determined by the image search engine service to which the identified phrase was submitted.
  • the search system may also generate sub-clusters of images within each cluster.
  • the search system may identify sub-clusters based on the visual similarities between the images of a cluster. Because of the computational expense of identifying visually similar images, the search system may maintain a mapping of images (e.g., identified by URL) to a feature set of visual features.
  • the search system can then generate the sub-clusters based on the feature sets of the images without having to generate the feature sets at the time of the query.
  • the search system may use various well-known clustering techniques to identify clusters of visually similar images from the feature sets.
  • the search system may identify phrases from the text identified from an image search result provided by an image search engine, such as one adapted to search image forums.
  • an image forum may contain metadata associated with the images that have been uploaded by users of the forum.
  • the search system may submit the original image query to the image search engine for an image forum.
  • the search result may contain the thumbnails of images along with metadata.
  • the search system identifies the text of the metadata for use in identifying phrases for use in generating clusters of images for the image search result.
  • the search system may identify text from search results provided by various web page-based search engines and image search engines.
  • the search system may identify phrases using various techniques for identifying key phrases. For example, the search system may extract all phrases of certain lengths (e.g., n-grams) from the text of the search result and identify the various properties of the phrases such as phrase frequency, the document frequency, phrase length, and so on. The search system may identify the key phrases based on the properties of the phrases. The search system may also filter the key phrases based on those that are unlikely to generate meaningful results (e.g., too many noise words), likely to generate redundant results (e.g., “tiger photo” and “tiger photograph”), and so on. After the search system submits a key phrase and receives the image search result for the cluster associated with that key phrase, the search system may merge, discard, or otherwise modify the clusters.
  • the search system may extract all phrases of certain lengths (e.g., n-grams) from the text of the search result and identify the various properties of the phrases such as phrase frequency, the document frequency, phrase length, and so on.
  • the search system may identify the key phrases
  • the search system may discard clusters that have too many or too few images or that have too many images in common with other clusters.
  • the search system may also merge clusters with too few images that appear to have many images in common.
  • Techniques for identifying phrases from a search result that can be used for clustering the search results are described in Zeng, H., He, Q., Chen, Z., Ma, W., and Ma, J., “Learning to Cluster Web Search Results,” SIGIR 2004 , July 25-29, Sheffield, South Yorkshire, U.K., and U.S. patent application Ser. No. 10/889,841, entitled “Query-Based Snippet Clustering for Search Result Grouping” and filed on Jul. 13, 2004, which are hereby both incorporated by reference.
  • One technique described in these references trains a linear regression model to learn the scores of phrases from a feature set of the phrases.
  • FIG. 1 is a block diagram that illustrates components of the search system in one embodiment.
  • the search system 110 is connected via communications link 120 to web page search engines 131 , image search engines 132 , image forums 133 , and client computing devices 140 .
  • the search system includes a search for images component 111 , a search for text component 112 , an identify clusters component 113 , and a cluster images visually component 114 .
  • the search for images component receives an original image query from a client computing device and invokes the search for text component to identify text from which to extract the phrases
  • the search for text component may submit the original image query to one or more page search engines and image search engines
  • the search for text component then extracts the text along with properties (e.g., font) from the page search results and image search results.
  • properties e.g., font
  • the search for images component invokes the identify clusters component to identify phrases from the extracted text.
  • the identify clusters component identifies key phrases from the extracted text that are to be used to generate the clusters of the image search result.
  • the search for images component then submits each key phrase to an image search engine to identify the images for the cluster represented by that key phrase.
  • the search for images component may optionally invoke the cluster images visually component to generate sub-clusters within a cluster of images.
  • the search system may then present the clusters of images as the image search result to a user who submitted the original image query using the user interface described in U.S.
  • the search system may also include a train regression model component 115 to train a regression model for identifying key phrases as described in U.S. patent application Ser. No. 10/889,841, entitled “Query-Based Snippet Clustering for Search Result Grouping” and filed on Jul. 13, 2004, which is hereby incorporated by reference.
  • the computing devices on which the search system may be implemented may include a central processing unit, memory, input devices (e.g., keyboard and pointing devices), output devices (e.g., display devices), and storage devices (e.g., disk drives).
  • the memory and storage devices are computer-readable media that may contain instructions that implement the search system.
  • the data structures and message structures may be stored or transmitted via a data transmission medium, such as a signal on a communications link.
  • Various communications links may be used, such as the Internet, a local area network, a wide area network, or a point-to-point dial-up connection.
  • the search system may receive original image queries from various computing systems or devices including personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
  • the search system may also provide its services to various computing systems such as personal computers, cell phones, personal digital assistants, consumer electronics, home automation devices, and so on.
  • the search system may be described in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other devices.
  • program modules include routines, programs, objects, components, data structures, and so on that perform particular tasks or implement particular abstract data types.
  • the functionality of the program modules may be combined or distributed as desired in various embodiments.
  • the search system may include a page search engine and/or an image search engine and may not use the services of an external search engine.
  • FIG. 2 is a flow diagram that illustrates the processing of the search for images component of the search system in one embodiment.
  • the component is passed an original image query and returns clusters of images as the image search result.
  • the component invokes the search for text component to identify text associated with the original image query.
  • the component invokes the identify clusters component to identify key phrases from the identified text that are used to identify images of a cluster associated with each key phrase.
  • the component loops submitting each key phrase of a cluster to an image search engine.
  • the component selects the next cluster.
  • decision block 204 if all the clusters have already been selected, then the component completes, else the component continues at block 205 .
  • the component submits the key phrase of the selected cluster to an image search engine to identify images for the selected cluster.
  • the component may optionally generate sub-clusters of the image search result based on visual similarity of the images. The generating of sub-clusters results in a hierarchy of clusters. The component then loops to block 203 to select the next cluster.
  • FIG. 3 is a flow diagram that illustrates the processing of the search for text component of the search system in one embodiment.
  • the component is passed an original image query and returns text associated with the original image query.
  • the component submits the original image query to a page search engine.
  • the component extracts, from the web page containing the search result, titles and snippets as text.
  • the component submits the original image query to an image search engine.
  • the component extracts, from the web pages containing the image search result, titles, snippets, and other metadata as text and then returns the text.
  • the component may submit the original image query to various page search engines and image search engines in parallel and process each search result as it is received.
  • FIG. 4 is a flow diagram that illustrates the processing of the identify clusters component of the search system in one embodiment.
  • the component is passed text and identifies key phrases of the text for use in generating clusters of images.
  • the component loops extracting and scoring the phrases.
  • the component extracts the next phrase from the text.
  • decision block 402 if all the phrases have already been extracted, then the component continues at block 406 , else the component continues at block 403 .
  • the component calculates the properties (e.g., frequency) for the extracted phrase.
  • the component generates a score for the extracted phrase based on the generated properties using, for example, a linear regression model.
  • the component then loops to block 401 to extract the next phrase.
  • the component selects the top scoring phrases to be key phrases. For example, the component may select the top 30scoring phrases as key phrases.
  • the component may remove certain of the selected phrases, such as those phrases with too many noise words or that are duplicates.
  • the component may effectively merge clusters by combining phrases that are similar. The component then completes.
  • the search system may be used to cluster content of various types, such as photographs, drawings, artwork, videos, music, and so on.
  • the search system may also be used an alternate way to cluster web pages of a conventional web page search.
  • the search system may identify phrases for clusters as described above from text derived from the search result by submitting the original page query to a web page search engine. The search can then submit each phrase to a web page search engine to identify the web pages for the cluster and return the clusters as the search result for the original page query.
  • the search system may also submit variations of the original image query, rather than the original image query itself, to identify text. For example, if the original image query is long or does not produce enough results, the search system may divide the original image query into sub-queries and submit each sub-query to a search engine to identify text. Accordingly, the invention is not limited except as by the appended claims.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Library & Information Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

A method and system for generating clusters of images for a search result of an image query is provided. When an original image query is received, the search system identifies text associated with the original image query by submitting the original image query to a search engine. The search system identifies phrases from the text of the web page containing the search result. The search system uses each of the identified phrases as an image query and submits the image queries to an image search engine. The search system considers the image search result for each image query to represent a cluster of related images. The search system then presents the clusters of images as the images of the image search result of the original image query.

Description

BACKGROUND
Many search engine services, such as Google and Overture, provide for searching for information that is accessible via the Internet. These search engine services allow users to search for display pages, such as web pages, that may be of interest to users. After a user submits a search request (also referred to as a “query”) that includes search terms, the search engine service identifies web pages that may be related to those search terms. To quickly identify related web pages, the search engine services may maintain a mapping of keywords to web pages. This mapping may be generated by “crawling” the web (i.e., the World Wide Web) to identify the keywords of each web page. To crawl the web, a search engine service may use a list of base web pages to identify all web pages that are accessible through those base web pages. The keywords of any particular web page can be identified using various well-known information retrieval techniques, such as identifying the words of a headline, the words supplied in the metadata of the web page, the words that are highlighted, and so on. The search engine service may generate a relevance score to indicate how related the information of the web page may be to the search request. The search engine service then displays to the user links to those web pages in an order that is based on their relevance.
Several search engine services also provide for searching for images that are available on the Internet. These image search engines typically generate a mapping of keywords to images by crawling the web in much the same way as described above for mapping keywords to web pages. An image search engine service can identify keywords based on text of the web pages that contain the images. An image search engine may also gather keywords from metadata associated with images of web-based image forums, which are an increasingly popular mechanism for people to publish their photographs and other images. An image forum allows users to upload their photographs and requires the users to provide associated metadata such as title, camera setting, category, and description. The image forums typically allow reviewers to rate each of the uploaded images and thus have ratings on the quality of the images. Regardless of how the mappings are generated, an image search engine service inputs an image query and uses the mapping to find images that are related to the image query. An image search engine service may identify thousands of images that are related to an image query and presents thumbnails of the related images. To help a user view the images, an image search engine service may order the thumbnails based on relevance of the images to the image query. An image search engine service may also limit the number of images that are provided to a few hundred of the most relevant images so as not to overwhelm the viewer.
Unfortunately, the relevance determination may not be particularly accurate because image queries may be ambiguous (e.g., “tiger” may represent the animal or the golfer), the keywords derived from web pages may not be very related to an image of the web page (e.g., a web page can contain many unrelated images), and so on. To help a user view the thousands of images, an image search engine service could cluster a search result based on the content of the images and present the clusters, rather than individual images, to the user. Such clustering techniques include content-based techniques and link-based techniques. The content-based techniques use low-level visual information to identify related images. There are, however, disadvantages to content-based clustering. Content-based clustering is computationally expensive and cannot be practically performed in real time when an image search result contains thousands of images. Moreover, if the clustering is limited to a few hundred of what are thought to be the most relevant images, some very relevant images may be missed because of the difficulties in assessing relevance. The link-based search techniques typically assume that images on the same web page are likely to be related and that images on web pages that are each linked to by the same web page are related. Since this assumption is, however, often not true, unrelated images are often clustered together. It is also difficult for either cluster technique to automatically identify meaningful names for a cluster of images. As a result, a user may not be able to effectively identify relevant clusters.
SUMMARY
A method and system for generating clusters of images for a search result of an image query is provided. An image search system receives an original image query, identifies phrases related to the original image query, submits to an image search engine each of the identified phrases as an image query, and provides the images of each image search result as a cluster of images for the image search result of the original image query. When an original image query is received, the search system identifies text associated with the original image query by submitting the original image query to a search engine. The search system identifies phrases from the text of the web page containing the search result. The search system uses each of the identified phrases as an image query and submits the image queries to an image search engine. The search system considers the image search result for each image query to represent a cluster of related images. The search system then presents the clusters of images as the images of the image search result of the original image query.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a block diagram that illustrates components of the search system in one embodiment.
FIG. 2 is a flow diagram that illustrates the processing of the search for images component of the search system in one embodiment.
FIG. 3 is a flow diagram that illustrates the processing of the search for text component of the search system in one embodiment.
FIG. 4 is a flow diagram that illustrates the processing of the identify clusters component of the search system in one embodiment.
 DETAILED DESCRIPTION
A method and system for generating clusters of images for a search result of an image query is provided. In one embodiment, an image search system receives an original image query, identifies phrases related to the original image query, submits to an image search engine each of the identified phrases as an image query, and provides the images of each image search result as a cluster of images for the image search result of the original image query. When an image query is received, the search system identifies text associated with the image query. The search system may identify the text by submitting the original image query to a conventional web page search engine. The conventional search engine searches for web pages that are related to the original image query and returns a web page that contains links to the related web pages possibly along with a snippet of text from those web pages. Upon receiving the web page containing the search result, the search system identifies phrases from the text of the web page containing the search result. For example, if the original image query is “tiger,” then the search result web page (as part of the links or in the snippets) may contain the phrases “white tiger,” “Tiger Woods,” “crouching tiger,” and so on. The search system uses each of the identified phrases as an image query and submits the image queries to an image search engine. The search system considers the image search result for each image query to represent a cluster of related images. For example, the search system submits to an image search engine the phrase “white tiger” as an image query and receives the image search result. The search system considers images of the image search result to be a cluster of images relating to “white tiger.” The search system also submits to an image search engine the phrase “Tiger Woods” as an image query and receives the image search result. The search system considers the images of the image search result to be a cluster of images related to “Tiger Woods.” The search system may also submit the original image query to the image search engine to generate an “others” cluster. The search system may remove from the “others” cluster those images that are already in another cluster. For example, the search system submits “tiger” to the image search engine, removes from the image search result the images that are also in the “white tiger,” “Tiger Woods,” and “crouching tiger” image search results and forms an “others” cluster of the remaining images. The search system then presents to a user the clusters of images as the images of the image search result of the original image query, such as “tiger.” In addition, the search system may use the phrase of the image query as the name of the cluster. For example, one cluster may be named “white tiger,” and another cluster may be named “Tiger Woods.” In this way, the search system can generate an image search result that has clusters of images without the computational expense of identifying clusters from a set of images and can provide meaningful names to the clusters.
In one embodiment, the search system may identify phrases from query logs. A query log contains the queries that have been submitted to a search engine. The search system may identify the most popular queries that contain the original image query and use those as phrases. For example, the most popular queries that contain “tiger” may be “Tiger Woods” and “Detroit Tigers.” The search system then submits each query to an image search engine and names the cluster of image search result with the text of the query.
In one embodiment, the search system may present the images within each cluster in an order based on relevance as determined by the image search engine service to which the identified phrase was submitted. The search system may also generate sub-clusters of images within each cluster. The search system may identify sub-clusters based on the visual similarities between the images of a cluster. Because of the computational expense of identifying visually similar images, the search system may maintain a mapping of images (e.g., identified by URL) to a feature set of visual features. The search system can then generate the sub-clusters based on the feature sets of the images without having to generate the feature sets at the time of the query. The search system may use various well-known clustering techniques to identify clusters of visually similar images from the feature sets.
In one embodiment, the search system may identify phrases from the text identified from an image search result provided by an image search engine, such as one adapted to search image forums. As described above, an image forum may contain metadata associated with the images that have been uploaded by users of the forum. The search system may submit the original image query to the image search engine for an image forum. The search result may contain the thumbnails of images along with metadata. The search system identifies the text of the metadata for use in identifying phrases for use in generating clusters of images for the image search result. The search system may identify text from search results provided by various web page-based search engines and image search engines.
The search system may identify phrases using various techniques for identifying key phrases. For example, the search system may extract all phrases of certain lengths (e.g., n-grams) from the text of the search result and identify the various properties of the phrases such as phrase frequency, the document frequency, phrase length, and so on. The search system may identify the key phrases based on the properties of the phrases. The search system may also filter the key phrases based on those that are unlikely to generate meaningful results (e.g., too many noise words), likely to generate redundant results (e.g., “tiger photo” and “tiger photograph”), and so on. After the search system submits a key phrase and receives the image search result for the cluster associated with that key phrase, the search system may merge, discard, or otherwise modify the clusters. For example, the search system may discard clusters that have too many or too few images or that have too many images in common with other clusters. The search system may also merge clusters with too few images that appear to have many images in common. Techniques for identifying phrases from a search result that can be used for clustering the search results are described in Zeng, H., He, Q., Chen, Z., Ma, W., and Ma, J., “Learning to Cluster Web Search Results,” SIGIR 2004, July 25-29, Sheffield, South Yorkshire, U.K., and U.S. patent application Ser. No. 10/889,841, entitled “Query-Based Snippet Clustering for Search Result Grouping” and filed on Jul. 13, 2004, which are hereby both incorporated by reference. One technique described in these references trains a linear regression model to learn the scores of phrases from a feature set of the phrases.
FIG. 1 is a block diagram that illustrates components of the search system in one embodiment. The search system 110 is connected via communications link 120 to web page search engines 131, image search engines 132, image forums 133, and client computing devices 140. The search system includes a search for images component 111, a search for text component 112, an identify clusters component 113, and a cluster images visually component 114. The search for images component receives an original image query from a client computing device and invokes the search for text component to identify text from which to extract the phrases The search for text component may submit the original image query to one or more page search engines and image search engines The search for text component then extracts the text along with properties (e.g., font) from the page search results and image search results. The search for images component invokes the identify clusters component to identify phrases from the extracted text. The identify clusters component identifies key phrases from the extracted text that are to be used to generate the clusters of the image search result. The search for images component then submits each key phrase to an image search engine to identify the images for the cluster represented by that key phrase. The search for images component may optionally invoke the cluster images visually component to generate sub-clusters within a cluster of images. The search system may then present the clusters of images as the image search result to a user who submitted the original image query using the user interface described in U.S. patent application Ser. No. 11/337,945, entitled “User Interface for Viewing Clusters of Images” and filed concurrently, which is hereby incorporated by reference. The search system may also include a train regression model component 115 to train a regression model for identifying key phrases as described in U.S. patent application Ser. No. 10/889,841, entitled “Query-Based Snippet Clustering for Search Result Grouping” and filed on Jul. 13, 2004, which is hereby incorporated by reference.
The computing devices on which the search system may be implemented may include a central processing unit, memory, input devices (e.g., keyboard and pointing devices), output devices (e.g., display devices), and storage devices (e.g., disk drives). The memory and storage devices are computer-readable media that may contain instructions that implement the search system. In addition, the data structures and message structures may be stored or transmitted via a data transmission medium, such as a signal on a communications link. Various communications links may be used, such as the Internet, a local area network, a wide area network, or a point-to-point dial-up connection.
The search system may receive original image queries from various computing systems or devices including personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The search system may also provide its services to various computing systems such as personal computers, cell phones, personal digital assistants, consumer electronics, home automation devices, and so on.
The search system may be described in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, and so on that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments. For example, the search system may include a page search engine and/or an image search engine and may not use the services of an external search engine.
FIG. 2 is a flow diagram that illustrates the processing of the search for images component of the search system in one embodiment. The component is passed an original image query and returns clusters of images as the image search result. In block 201, the component invokes the search for text component to identify text associated with the original image query. In block 202, the component invokes the identify clusters component to identify key phrases from the identified text that are used to identify images of a cluster associated with each key phrase. In blocks 203-206, the component loops submitting each key phrase of a cluster to an image search engine. In block 203, the component selects the next cluster. In decision block 204, if all the clusters have already been selected, then the component completes, else the component continues at block 205. In block 205, the component submits the key phrase of the selected cluster to an image search engine to identify images for the selected cluster. In block 206, the component may optionally generate sub-clusters of the image search result based on visual similarity of the images. The generating of sub-clusters results in a hierarchy of clusters. The component then loops to block 203 to select the next cluster.
FIG. 3 is a flow diagram that illustrates the processing of the search for text component of the search system in one embodiment. The component is passed an original image query and returns text associated with the original image query. In block 301, the component submits the original image query to a page search engine. In block 302, the component extracts, from the web page containing the search result, titles and snippets as text. In block 303, the component submits the original image query to an image search engine. In block 304, the component extracts, from the web pages containing the image search result, titles, snippets, and other metadata as text and then returns the text. The component may submit the original image query to various page search engines and image search engines in parallel and process each search result as it is received.
FIG. 4 is a flow diagram that illustrates the processing of the identify clusters component of the search system in one embodiment. The component is passed text and identifies key phrases of the text for use in generating clusters of images. In blocks 401-405, the component loops extracting and scoring the phrases. In block 401, the component extracts the next phrase from the text. In decision block 402, if all the phrases have already been extracted, then the component continues at block 406, else the component continues at block 403. In block 403, the component calculates the properties (e.g., frequency) for the extracted phrase. In block 405, the component generates a score for the extracted phrase based on the generated properties using, for example, a linear regression model. The component then loops to block 401 to extract the next phrase. In block 406, the component selects the top scoring phrases to be key phrases. For example, the component may select the top 30scoring phrases as key phrases. In block 407, the component may remove certain of the selected phrases, such as those phrases with too many noise words or that are duplicates. In block 408, the component may effectively merge clusters by combining phrases that are similar. The component then completes.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. The search system may be used to cluster content of various types, such as photographs, drawings, artwork, videos, music, and so on. The search system may also be used an alternate way to cluster web pages of a conventional web page search. The search system may identify phrases for clusters as described above from text derived from the search result by submitting the original page query to a web page search engine. The search can then submit each phrase to a web page search engine to identify the web pages for the cluster and return the clusters as the search result for the original page query. The search system may also submit variations of the original image query, rather than the original image query itself, to identify text. For example, if the original image query is long or does not produce enough results, the search system may divide the original image query into sub-queries and submit each sub-query to a search engine to identify text. Accordingly, the invention is not limited except as by the appended claims.

Claims (13)

1. A computing device for generating clusters of images, comprising:
a memory containing computer-executable instructions comprising:
a component that receives an image query having search terms;
a component that submits the image query to a web-based search engine and receives search results that include links to web pages and text associated with each link, the text being a snippet describing content of the linked-to web page, the web-based search engine being a search engine that maintains an index of keywords to web pages and when a query is submitted, the web-based search engine identifies web pages for search results using the index of keywords to identify web pages that match the submitted query and returns as search results links to the identified web pages and associated snippets;
a component that identifies phrases associated with the image query by identifying phrases from the snippets of the search results;
a component that identifies images associated with each identified phrase by submitting each identified phrase to an image search engine and receiving identifications of images as search results for each submitted phrase wherein the identified images associated with each submitted phrase represent a cluster of images related to that submitted phrase, the image search engine being a search engine that maintains an index of keywords to images and when a query is submitted, the image search engine identifies images for the search results using the index of keywords to identify images that match the submitted query and returns as search results links to the identified images, the image search engine being different from the web-based search engine; and
a component that displays an indication of each cluster of images as search results for the received image query; and
a processor for executing the computer-executable instructions stored in the memory.
2. The computing device of claim 1 including a component that generates sub-clusters of the cluster of images.
3. The computing device of claim 1 wherein the component that identifies phrases scores phrases based on learned scores for attributes of the phrases.
4. A computer-readable medium storing computer-executable instructions for controlling a computing device to generate clusters of images, by a method comprising:
receiving an image query having search terms;
submitting the image query to a web-based search engine and receives search results that include links to web pages and text associated with each link, the text being a snippet describing content of the linked-to web page, the web-based search engine being a search engine that maintains an index of keywords to web pages and when a query is submitted, the web-based search engine identifies web pages for search results using the index of keywords to identify web pages that match the submitted query and returns as search results links to the identified web pages and associated snippets;
identifying phrases associated with the image query by identifying phrases from the snippets of the search results;
identifying images associated with each identified phrase by submitting each identified phrase to an image search engine and receiving identifications of images as search results for each submitted phrase wherein the identified images associated with each submitted phrase represent a cluster of images related to that submitted phrase, the image search engine being a search engine that maintains an index of keywords to images and when a query is submitted, the image search engine identifies images for the search results using the index of keywords to identify images that match the submitted query and returns as search results links to the identified images, the image search engine being different from the web-based search engine; and
displaying an indication of each cluster of images as search results for the received image query.
5. The computer-readable medium of claim 4 including generating sub-clusters of the cluster of images.
6. The computer-readable medium of claim 4 wherein the identifying of phrases scores phrases based on learned scores for attributes of the phrases.
7. The computer-readable medium of claim 4 wherein a submitted phrase represents a name for the cluster.
8. The computer-readable medium of claim 4 including generating sub-clusters of the clusters of images based on visual similarity of images of a cluster.
9. A method performed by a computing device for generating clusters of images, by a method comprising:
receiving an image query having search terms;
submitting by the computing device the image query to a web-based search engine and receives search results that include links to web pages and text associated with each link, the text being a snippet describing content of the linked-to web page, the web-based search engine being a search engine that maintains an index of keywords to web pages and when a query is submitted, the web-based search engine identifies web pages for search results using the index of keywords to identify web pages that match the submitted query and returns as search results links to the identified web pages and associated snippets;
identifying by the computing device phrases associated with the image query by identifying phrases from the snippets of the search results;
identifying by the computing device images associated with each identified phrase by submitting each identified phrase to an image search engine and receiving identifications of images as search results for each submitted phrase wherein the identified images associated with each submitted phrase represent a cluster of images related to that submitted phrase, the image search engine being a search engine that maintains an index of keywords to images and when a query is submitted, the image search engine identifies images for the search results using the index of keywords to identify images that match the submitted query and returns as search results links to the identified images, the image search engine being different from the web-based search engine; and
displaying an indication of each cluster of images as search results for the received image query.
10. The method of claim 9 including generating sub-clusters of the cluster of images.
11. The method of claim 9 wherein the identifying of phrases scores phrases based on learned scores for attributes of the phrases.
12. The method of claim 9 wherein a submitted phrase represents a name for the cluster.
13. The method of claim 9 including generating sub-clusters of the clusters of images based on visual similarity of images of a cluster.
US11/337,825 2006-01-23 2006-01-23 Generating clusters of images for search results Expired - Fee Related US7725451B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US11/337,825 US7725451B2 (en) 2006-01-23 2006-01-23 Generating clusters of images for search results

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US11/337,825 US7725451B2 (en) 2006-01-23 2006-01-23 Generating clusters of images for search results

Publications (2)

Publication Number Publication Date
US20070174269A1 US20070174269A1 (en) 2007-07-26
US7725451B2 true US7725451B2 (en) 2010-05-25

Family

ID=38286756

Family Applications (1)

Application Number Title Priority Date Filing Date
US11/337,825 Expired - Fee Related US7725451B2 (en) 2006-01-23 2006-01-23 Generating clusters of images for search results

Country Status (1)

Country Link
US (1) US7725451B2 (en)

Cited By (28)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070209025A1 (en) * 2006-01-25 2007-09-06 Microsoft Corporation User interface for viewing images
US20100114933A1 (en) * 2008-10-24 2010-05-06 Vanessa Murdock Methods for improving the diversity of image search results
US20110093361A1 (en) * 2009-10-20 2011-04-21 Lisa Morales Method and System for Online Shopping and Searching For Groups Of Items
US20110194761A1 (en) * 2010-02-08 2011-08-11 Microsoft Corporation Intelligent Image Search Results Summarization and Browsing
US8326845B2 (en) 2007-12-12 2012-12-04 Vast.com, Inc. Predictive conversion systems and methods
WO2012170318A2 (en) * 2011-06-08 2012-12-13 Facebook, Inc. Presenting images as search results
US20130144904A1 (en) * 2010-04-14 2013-06-06 Nhn Corporation Method and system for providing query using an image
US8943074B2 (en) 2013-02-05 2015-01-27 International Business Machines Corporation Generating and displaying an image tour in response to a user search query
US9092458B1 (en) 2005-03-08 2015-07-28 Irobot Corporation System and method for managing search results including graphics
US9104718B1 (en) 2013-03-07 2015-08-11 Vast.com, Inc. Systems, methods, and devices for measuring similarity of and generating recommendations for unique items
US9465873B1 (en) 2013-03-07 2016-10-11 Vast.com, Inc. Systems, methods, and devices for identifying and presenting identifications of significant attributes of unique items
US9600919B1 (en) 2009-10-20 2017-03-21 Yahoo! Inc. Systems and methods for assembling and/or displaying multimedia objects, modules or presentations
US9830635B1 (en) 2013-03-13 2017-11-28 Vast.com, Inc. Systems, methods, and devices for determining and displaying market relative position of unique items
US9843823B2 (en) 2012-05-23 2017-12-12 Yahoo Holdings, Inc. Systems and methods involving creation of information modules, including server, media searching, user interface and/or other features
US10007946B1 (en) 2013-03-07 2018-06-26 Vast.com, Inc. Systems, methods, and devices for measuring similarity of and generating recommendations for unique items
US10120883B2 (en) 2006-01-23 2018-11-06 Microsoft Technology Licensing, Llc User interface for viewing clusters of images
US10127596B1 (en) 2013-12-10 2018-11-13 Vast.com, Inc. Systems, methods, and devices for generating recommendations of unique items
US10268704B1 (en) 2017-10-12 2019-04-23 Vast.com, Inc. Partitioned distributed database systems, devices, and methods
US10296158B2 (en) 2011-12-20 2019-05-21 Oath Inc. Systems and methods involving features of creation/viewing/utilization of information modules such as mixed-media modules
US10303723B2 (en) 2012-06-12 2019-05-28 Excalibur Ip, Llc Systems and methods involving search enhancement features associated with media modules
US10387503B2 (en) 2011-12-15 2019-08-20 Excalibur Ip, Llc Systems and methods involving features of search and/or search integration
US10394882B2 (en) 2014-02-19 2019-08-27 International Business Machines Corporation Multi-image input and sequenced output based image search
US10417289B2 (en) * 2012-06-12 2019-09-17 Oath Inc. Systems and methods involving integration/creation of search results media modules
US10504555B2 (en) 2011-12-20 2019-12-10 Oath Inc. Systems and methods involving features of creation/viewing/utilization of information modules such as mixed-media modules
US20200311126A1 (en) * 2016-03-29 2020-10-01 A9.Com, Inc. Methods to present search keywords for image-based queries
US11048734B1 (en) * 2018-08-20 2021-06-29 Pinterest, Inc. Auto-completion based on content similarities
US11099714B2 (en) 2012-02-28 2021-08-24 Verizon Media Inc. Systems and methods involving creation/display/utilization of information modules, such as mixed-media and multimedia modules
US12106358B1 (en) 2020-07-24 2024-10-01 Vast.com, Inc. Systems, methods, and devices for unified e-commerce platforms for unique items

Families Citing this family (76)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8510283B2 (en) 2006-07-31 2013-08-13 Ricoh Co., Ltd. Automatic adaption of an image recognition system to image capture devices
US8600989B2 (en) 2004-10-01 2013-12-03 Ricoh Co., Ltd. Method and system for image matching in a mixed media environment
US8184155B2 (en) 2007-07-11 2012-05-22 Ricoh Co. Ltd. Recognition and tracking using invisible junctions
US8856108B2 (en) 2006-07-31 2014-10-07 Ricoh Co., Ltd. Combining results of image retrieval processes
US9530050B1 (en) 2007-07-11 2016-12-27 Ricoh Co., Ltd. Document annotation sharing
US8838591B2 (en) 2005-08-23 2014-09-16 Ricoh Co., Ltd. Embedding hot spots in electronic documents
US8156116B2 (en) 2006-07-31 2012-04-10 Ricoh Co., Ltd Dynamic presentation of targeted information in a mixed media reality recognition system
US9405751B2 (en) 2005-08-23 2016-08-02 Ricoh Co., Ltd. Database for mixed media document system
US10192279B1 (en) 2007-07-11 2019-01-29 Ricoh Co., Ltd. Indexed document modification sharing with mixed media reality
US8176054B2 (en) 2007-07-12 2012-05-08 Ricoh Co. Ltd Retrieving electronic documents by converting them to synthetic text
US8195659B2 (en) 2005-08-23 2012-06-05 Ricoh Co. Ltd. Integration and use of mixed media documents
US8335789B2 (en) 2004-10-01 2012-12-18 Ricoh Co., Ltd. Method and system for document fingerprint matching in a mixed media environment
US8949287B2 (en) 2005-08-23 2015-02-03 Ricoh Co., Ltd. Embedding hot spots in imaged documents
US8521737B2 (en) 2004-10-01 2013-08-27 Ricoh Co., Ltd. Method and system for multi-tier image matching in a mixed media environment
US8276088B2 (en) 2007-07-11 2012-09-25 Ricoh Co., Ltd. User interface for three-dimensional navigation
US8868555B2 (en) 2006-07-31 2014-10-21 Ricoh Co., Ltd. Computation of a recongnizability score (quality predictor) for image retrieval
US8156427B2 (en) * 2005-08-23 2012-04-10 Ricoh Co. Ltd. User interface for mixed media reality
US7970171B2 (en) 2007-01-18 2011-06-28 Ricoh Co., Ltd. Synthetic image and video generation from ground truth data
US8369655B2 (en) 2006-07-31 2013-02-05 Ricoh Co., Ltd. Mixed media reality recognition using multiple specialized indexes
US8825682B2 (en) 2006-07-31 2014-09-02 Ricoh Co., Ltd. Architecture for mixed media reality retrieval of locations and registration of images
US7702673B2 (en) 2004-10-01 2010-04-20 Ricoh Co., Ltd. System and methods for creation and use of a mixed media environment
US8144921B2 (en) 2007-07-11 2012-03-27 Ricoh Co., Ltd. Information retrieval using invisible junctions and geometric constraints
US8385589B2 (en) 2008-05-15 2013-02-26 Berna Erol Web-based content detection in images, extraction and recognition
US9384619B2 (en) 2006-07-31 2016-07-05 Ricoh Co., Ltd. Searching media content for objects specified using identifiers
US8332401B2 (en) 2004-10-01 2012-12-11 Ricoh Co., Ltd Method and system for position-based image matching in a mixed media environment
US9171202B2 (en) 2005-08-23 2015-10-27 Ricoh Co., Ltd. Data organization and access for mixed media document system
US9373029B2 (en) 2007-07-11 2016-06-21 Ricoh Co., Ltd. Invisible junction feature recognition for document security or annotation
US7836050B2 (en) * 2006-01-25 2010-11-16 Microsoft Corporation Ranking content based on relevance and quality
US7814040B1 (en) 2006-01-31 2010-10-12 The Research Foundation Of State University Of New York System and method for image annotation and multi-modal image retrieval using probabilistic semantic models
US8026929B2 (en) * 2006-06-26 2011-09-27 University Of Southern California Seamlessly overlaying 2D images in 3D model
US9176984B2 (en) 2006-07-31 2015-11-03 Ricoh Co., Ltd Mixed media reality retrieval of differentially-weighted links
US8201076B2 (en) 2006-07-31 2012-06-12 Ricoh Co., Ltd. Capturing symbolic information from documents upon printing
US8489987B2 (en) 2006-07-31 2013-07-16 Ricoh Co., Ltd. Monitoring and analyzing creation and usage of visual content using image and hotspot interaction
US9063952B2 (en) 2006-07-31 2015-06-23 Ricoh Co., Ltd. Mixed media reality recognition with image tracking
US8676810B2 (en) 2006-07-31 2014-03-18 Ricoh Co., Ltd. Multiple index mixed media reality recognition using unequal priority indexes
US9020966B2 (en) * 2006-07-31 2015-04-28 Ricoh Co., Ltd. Client device for interacting with a mixed media reality recognition system
US7707208B2 (en) * 2006-10-10 2010-04-27 Microsoft Corporation Identifying sight for a location
US7657504B2 (en) * 2006-10-10 2010-02-02 Microsoft Corporation User interface for displaying images of sights
US8131722B2 (en) 2006-11-20 2012-03-06 Ebay Inc. Search clustering
US7954065B2 (en) * 2006-12-22 2011-05-31 Apple Inc. Two-dimensional timeline display of media items
US8276098B2 (en) 2006-12-22 2012-09-25 Apple Inc. Interactive image thumbnails
US9142253B2 (en) * 2006-12-22 2015-09-22 Apple Inc. Associating keywords to media
US8457416B2 (en) * 2007-09-13 2013-06-04 Microsoft Corporation Estimating word correlations from images
US8571850B2 (en) * 2007-09-13 2013-10-29 Microsoft Corporation Dual cross-media relevance model for image annotation
US8572112B2 (en) * 2007-11-02 2013-10-29 Microsoft Corporation Syndicating search queries using web advertising
EP2274711A4 (en) * 2008-03-31 2012-12-19 Univ Southern California ESTIMATING THE INSTALLATION OF PHOTOGRAPHIC IMAGES IN A 3D MODEL OF EARTH WITH HUMAN ASSISTANCE
CN102007492B (en) * 2008-04-14 2016-07-06 Tp视觉控股有限公司 For the method and apparatus searching for the digital picture of several storages
US8051083B2 (en) * 2008-04-16 2011-11-01 Microsoft Corporation Forum web page clustering based on repetitive regions
US8676001B2 (en) 2008-05-12 2014-03-18 Google Inc. Automatic discovery of popular landmarks
US8190623B2 (en) * 2008-06-05 2012-05-29 Enpulz, L.L.C. Image search engine using image analysis and categorization
CN102132274B (en) * 2008-08-26 2015-08-19 国际商业机器公司 Employ searcher and the searching method of disclosed search engine
US10210179B2 (en) * 2008-11-18 2019-02-19 Excalibur Ip, Llc Dynamic feature weighting
US8452794B2 (en) * 2009-02-11 2013-05-28 Microsoft Corporation Visual and textual query suggestion
US8396287B2 (en) 2009-05-15 2013-03-12 Google Inc. Landmarks from digital photo collections
US8385660B2 (en) 2009-06-24 2013-02-26 Ricoh Co., Ltd. Mixed media reality indexing and retrieval for repeated content
US20110041189A1 (en) * 2009-08-13 2011-02-17 Scott Robert Shipman System and method to provide a user-generated image gallery
US8676803B1 (en) 2009-11-04 2014-03-18 Google Inc. Clustering images
JP2011138197A (en) * 2009-12-25 2011-07-14 Sony Corp Information processing apparatus, method of evaluating degree of association, and program
US8625907B2 (en) 2010-06-10 2014-01-07 Microsoft Corporation Image clustering
US9703895B2 (en) * 2010-06-11 2017-07-11 Microsoft Technology Licensing, Llc Organizing search results based upon clustered content
WO2012058794A1 (en) 2010-11-01 2012-05-10 Microsoft Corporation Image search
US9384216B2 (en) * 2010-11-16 2016-07-05 Microsoft Technology Licensing, Llc Browsing related image search result sets
US20120278159A1 (en) * 2011-04-27 2012-11-01 Kumar Gangadharan Method and apparatus for enhancing customer service experience
US9058331B2 (en) 2011-07-27 2015-06-16 Ricoh Co., Ltd. Generating a conversation in a social network based on visual search results
US11068532B2 (en) 2011-09-21 2021-07-20 Horsetooth Ventures, LLC Interactive image display and selection system
US9734167B2 (en) 2011-09-21 2017-08-15 Horsetooth Ventures, LLC Interactive image display and selection system
JP2013179402A (en) * 2012-02-28 2013-09-09 Sony Corp Terminal device, information processor, display method, and display control method
US20130275431A1 (en) * 2012-04-12 2013-10-17 Nainesh Rathod Visual clustering method
KR102072113B1 (en) 2012-10-17 2020-02-03 삼성전자주식회사 User terminal device and control method thereof
US9152652B2 (en) * 2013-03-14 2015-10-06 Google Inc. Sub-query evaluation for image search
US11250203B2 (en) * 2013-08-12 2022-02-15 Microsoft Technology Licensing, Llc Browsing images via mined hyperlinked text snippets
US10430986B2 (en) * 2013-10-10 2019-10-01 Pushd, Inc. Clustering photographs for display on a digital picture frame
US9471695B1 (en) * 2014-12-02 2016-10-18 Google Inc. Semantic image navigation experiences
US10324986B2 (en) 2016-05-27 2019-06-18 Hipmunk, Inc. Search deconstruction, reconstruction, and allocation
US10248306B1 (en) * 2017-12-20 2019-04-02 Motorola Mobility Llc Systems and methods for end-users to link objects from images with digital content
JP7183753B2 (en) * 2018-12-11 2022-12-06 日本電信電話株式会社 LIST GENERATION DEVICE, SUBJECT IDENTIFICATION DEVICE, LIST GENERATION METHOD, AND PROGRAM

Citations (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6119124A (en) * 1998-03-26 2000-09-12 Digital Equipment Corporation Method for clustering closely resembling data objects
US6643641B1 (en) * 2000-04-27 2003-11-04 Russell Snyder Web search engine with graphic snapshots
US6728752B1 (en) * 1999-01-26 2004-04-27 Xerox Corporation System and method for information browsing using multi-modal features
US6823335B2 (en) * 1998-10-05 2004-11-23 Canon Kabushiki Kaisha Information search apparatus and method, and storage medium
US20040249774A1 (en) * 1993-09-20 2004-12-09 Caid William R. Representation and retrieval of images using context vectors derived from image information elements
US6944612B2 (en) * 2002-11-13 2005-09-13 Xerox Corporation Structured contextual clustering method and system in a federated search engine
US20060026152A1 (en) 2004-07-13 2006-02-02 Microsoft Corporation Query-based snippet clustering for search result grouping
US7010751B2 (en) * 2000-02-18 2006-03-07 University Of Maryland, College Park Methods for the electronic annotation, retrieval, and use of electronic images
US7051019B1 (en) * 1999-08-17 2006-05-23 Corbis Corporation Method and system for obtaining images from a database having images that are relevant to indicated text
US7065520B2 (en) * 2000-10-03 2006-06-20 Ronald Neville Langford Method of locating web-pages by utilising visual images
US7099860B1 (en) * 2000-10-30 2006-08-29 Microsoft Corporation Image retrieval systems and methods with semantic and feature based relevance feedback
US20060242178A1 (en) * 2005-04-21 2006-10-26 Yahoo! Inc. Media object metadata association and ranking
US20070174865A1 (en) 2006-01-25 2007-07-26 Microsoft Corporation Normalizing content ratings of content forums
US20070174872A1 (en) 2006-01-25 2007-07-26 Microsoft Corporation Ranking content based on relevance and quality
US20070209025A1 (en) 2006-01-25 2007-09-06 Microsoft Corporation User interface for viewing images
US20080086468A1 (en) 2006-10-10 2008-04-10 Microsoft Corporation Identifying sight for a location
US20080086686A1 (en) 2006-10-10 2008-04-10 Microsoft Corporation User interface for displaying images of sights
US7644373B2 (en) 2006-01-23 2010-01-05 Microsoft Corporation User interface for viewing clusters of images

Patent Citations (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040249774A1 (en) * 1993-09-20 2004-12-09 Caid William R. Representation and retrieval of images using context vectors derived from image information elements
US6119124A (en) * 1998-03-26 2000-09-12 Digital Equipment Corporation Method for clustering closely resembling data objects
US6349296B1 (en) * 1998-03-26 2002-02-19 Altavista Company Method for clustering closely resembling data objects
US6823335B2 (en) * 1998-10-05 2004-11-23 Canon Kabushiki Kaisha Information search apparatus and method, and storage medium
US6728752B1 (en) * 1999-01-26 2004-04-27 Xerox Corporation System and method for information browsing using multi-modal features
US7051019B1 (en) * 1999-08-17 2006-05-23 Corbis Corporation Method and system for obtaining images from a database having images that are relevant to indicated text
US7010751B2 (en) * 2000-02-18 2006-03-07 University Of Maryland, College Park Methods for the electronic annotation, retrieval, and use of electronic images
US6643641B1 (en) * 2000-04-27 2003-11-04 Russell Snyder Web search engine with graphic snapshots
US7065520B2 (en) * 2000-10-03 2006-06-20 Ronald Neville Langford Method of locating web-pages by utilising visual images
US7099860B1 (en) * 2000-10-30 2006-08-29 Microsoft Corporation Image retrieval systems and methods with semantic and feature based relevance feedback
US6944612B2 (en) * 2002-11-13 2005-09-13 Xerox Corporation Structured contextual clustering method and system in a federated search engine
US20060026152A1 (en) 2004-07-13 2006-02-02 Microsoft Corporation Query-based snippet clustering for search result grouping
US20060242178A1 (en) * 2005-04-21 2006-10-26 Yahoo! Inc. Media object metadata association and ranking
US7644373B2 (en) 2006-01-23 2010-01-05 Microsoft Corporation User interface for viewing clusters of images
US20070174865A1 (en) 2006-01-25 2007-07-26 Microsoft Corporation Normalizing content ratings of content forums
US20070174872A1 (en) 2006-01-25 2007-07-26 Microsoft Corporation Ranking content based on relevance and quality
US20070209025A1 (en) 2006-01-25 2007-09-06 Microsoft Corporation User interface for viewing images
US20080086468A1 (en) 2006-10-10 2008-04-10 Microsoft Corporation Identifying sight for a location
US20080086686A1 (en) 2006-10-10 2008-04-10 Microsoft Corporation User interface for displaying images of sights

Non-Patent Citations (39)

* Cited by examiner, † Cited by third party
Title
Brin, Sergey and Lawrence Page, "The anatomy of a large-scale hypertextual (Web) search engine," In the 7th International World Wide Web Conference, 1998.
Broder, Andrei, "A taxonomy of web search," SIGIR Forum 36(2), 2002, 8 pages.
Cai, Deng et al., "Hierarchical Clustering of WWW Image Search Results Using Visual, Textual and Link Analysis," MM'04, Oct. 10-16, 2004 New York, New,York, © 2004 ACM.
Chang, Shi Kuo and Arding Hsu, "Image Information Systems: Where do we go from here?," IEEE Transactions on Knowledge and Data Engineering, vol. 4, No. 5, Oct. 1992, pp. 431-442.
Chen, Hao and Susan Dumais, "Bringing Order to the Web: Automatically Categorizing Search Results," Proceedings of CHI'00, Human Factors in Computing Systems, pp. 145-152.
Dumais, Susan, Edward Cutrell and Hao Chen, "Optimizing Search by Showing Results in Context," SIGCHI'01, Mar. 31-Apr. 4, 2001, Seattle, Washington, © 2001 ACM.
Ferragina et al., "The Anatomy of a Clustering Engine for Web-page Snippets", Jan. 29, 2004, University of Pisa, Tech. Report: TR-04-05. *
Frankel, Charles, Michael J. Swain and Vassilis Athitsos, "WebSeer: An Image Search Engine for the World Wide Web," Aug. 1, 1996, Technical Report 96-14, Computer Science Department, The University of Chicago.
Glance, Natalie S., "Community Search Assistant," IUI'01, Jan. 14-17, 2001, Santa Fe, New Mexico, © 2001 ACM, pp. 91-96.
Google image search, http://images.google.com, [last accessed Jun. 7, 2007].
Google web search, http://www.google.com, [last accessed Jun. 7, 2007].
Hearst, Marti A. and Jan O. Pedersen, "Reexamining the Cluster Hypothesis: Scatter/Gather on Retrieval Results," In the Proceedings of ACM SIGIR, Aug. 1996, Zurich.
http://www.flickr.com, Archived on http://www.archive.org on Feb. 18, 2005, pp. 1-5, available on http://web.archive.org/web/20050218030404/http://flickr.com/. *
Huang, Jing et al., "Image Indexing using color correlograms," In Proc. IEEE Comp. Soc. Conf. Comp. Vis. and Patt. Rec., 1997, pp. 762-768.
Jansen, Major Bernard J. et al., "Real life information retrieval: a study of user queries on the Web," ACM SIGIR Forum 1998, vol. 32, No. 1, pp. 5-17.
Kherfi et al., Image Retrieval from the World Wide Web: Issues, Techniques, and Systems, Mar. 2004, ACM Computing Surveys, vol. 36, No. 1, pp. 35-67. *
Lempel, Ronny and Aya Soffer, "PicASHOW: Pictorial Authority Search by Hyperlinks on the Web," Proceedings of the 10th International WWW Conference, Hong Kong, China, 2001, ACM, pp. 438-448.
Liu, Hao et al., "Effective Browsing of Web Image Search Results," MIR'04, Oct. 15-16, 2004, New York, New York, © 2004 ACM.
Luo, Bo, Xiaogang Wang and Xiaoou Tang, "A World Wide Web Based Image Search Engine Using Text and Image Content Features," Internet Imaging IV, Proceedings of SPIE-IS&T, Electronic Imaging, SPIE, vol. 5018, 2003, © 2003 SPIE-IS&T, pp. 123-130.
MSRA clustering search, http://rwsm.directtaps.net/, [last accessed Jun. 7, 2007].
Mysore et al., DIOGENES: A Distributed Search Agent, May 2003, Technical Reports CSE-2003-24. *
Photosig, http://www.photosig.com, [last accessed Aug. 17, 2006].
Picsearch image search, http://www.picsearch.com, [last accessed Jan. 5, 2006].
Rui, Yong and Thomas S. Huang, "Image Retrieval: Current Techniques, Promising Directions and Open Issues", Journal of Visual Communication and Image Representation, vol. 10, 39-62, Mar. 1999, © 1999 by Academic Press.
Sclaroff, Stan, Leonid Taycher and Marco Lacascia, "ImageRover: A Content-Based Image Browser for the World Wide Web," In IEEE Workshop on Content-based Access of Image and Video Libraries, San Juan, Puerto Rico, Jun. 1997, pp. 2-9.
Shen, Heng Tao et al., "Giving Meanings to WWW Images," Proceedings of ACM Multimedia 2000, Los Angeles, CA, pp. 39-47, © ACM 2000.
Smeulders, Arnold W.M. et al., "Content-Based Image Retrieval at the End of the Early Years," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, No. 12, Dec. 2000, pp. 1349-1380, © 2000 IEEE.
Smith, John R. and Shih-Fu Chang, "Visually Searching the Web for Content," IEEE Multimedia, 1997, pp. 12-20, © 1997 IEEE.
Teevan, Jaime et al., "The Perfect Search Engine Is Not Enough: A Study of Orienteering Behavior in Directed Search," CHI 2004, Apr. 24-29, Vienna, Austria, , pp. 415-422.
U.S. Appl. No. 11/337,945, filed Jan. 23, 2006, Jing et al.
Vivisimo clustering search, http://vivisimo.com, [last accessed Jun. 7, 2007].
Wang, Xin-Jing et al., "Grouping Web Image Search Result," MM'04, Oct. 10-16, 2004, New York, New York, pp. 436-439, © 2004 ACM.
White, David A. and Ramesh Jain, "Similarity Indexing: Algorithms and Performance," 2 / SPIE, Feb. 1-2, 1996, San Jose, California, vol. 2670, pp. 63-73, © 1996 The Society of Photo-Optical Instrumentation Engineers.
Woodruff, Allison et al., "Using Thumbnails to Search the Web," Proceedings of SIGCHI, Mar. 31-Apr. 4, 2001, Seattle, Washington, pp. 198-205, © 2001 ACM.
Yahoo Homepage search, http://www.yahoo.com/, [last accessed Jun. 7, 2007].
Yahoo image search, http://images.search.yahoo.com/, [last accessed Jun. 7, 2007].
Zamir, Oren and Oren Etzioni, "Web Document Clustering: A Feasibility Demonstration," In Proceedings of SIGIR'98, Melbourne, Australia, pp. 46-54, © 1998 ACM.
Zeng, H.J. et al, "Learning to Cluster Web Search Results," SIGIR'04, Jul. 25-29, 2004, Sheffield, South Yorkshire, UK, © 2004 ACM.
Zhang, L. et al., "Enjoy High Quality Photos in Vertical Image Search Engine," Submitted to the SIGCHI 2006 Conference on Human Factors in Computing Systems.

Cited By (63)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9092458B1 (en) 2005-03-08 2015-07-28 Irobot Corporation System and method for managing search results including graphics
US10120883B2 (en) 2006-01-23 2018-11-06 Microsoft Technology Licensing, Llc User interface for viewing clusters of images
US20070209025A1 (en) * 2006-01-25 2007-09-06 Microsoft Corporation User interface for viewing images
US11755598B1 (en) 2007-12-12 2023-09-12 Vast.com, Inc. Predictive conversion systems and methods
US8868572B2 (en) 2007-12-12 2014-10-21 Vast.com, Inc. Predictive conversion systems and methods
US8326845B2 (en) 2007-12-12 2012-12-04 Vast.com, Inc. Predictive conversion systems and methods
US10115074B1 (en) 2007-12-12 2018-10-30 Vast.com, Inc. Predictive conversion systems and methods
US8375037B2 (en) 2007-12-12 2013-02-12 Vast.com, Inc. Predictive conversion systems and methods
US11270252B1 (en) 2007-12-12 2022-03-08 Vast.com, Inc. Predictive conversion systems and methods
US9799000B2 (en) 2007-12-12 2017-10-24 Vast.com, Inc. Predictive conversion systems and methods
US20100114933A1 (en) * 2008-10-24 2010-05-06 Vanessa Murdock Methods for improving the diversity of image search results
US8171043B2 (en) * 2008-10-24 2012-05-01 Yahoo! Inc. Methods for improving the diversity of image search results
US9600919B1 (en) 2009-10-20 2017-03-21 Yahoo! Inc. Systems and methods for assembling and/or displaying multimedia objects, modules or presentations
US20110093361A1 (en) * 2009-10-20 2011-04-21 Lisa Morales Method and System for Online Shopping and Searching For Groups Of Items
US20110194761A1 (en) * 2010-02-08 2011-08-11 Microsoft Corporation Intelligent Image Search Results Summarization and Browsing
US20140321761A1 (en) * 2010-02-08 2014-10-30 Microsoft Corporation Intelligent Image Search Results Summarization and Browsing
US8774526B2 (en) 2010-02-08 2014-07-08 Microsoft Corporation Intelligent image search results summarization and browsing
US10521692B2 (en) * 2010-02-08 2019-12-31 Microsoft Technology Licensing, Llc Intelligent image search results summarization and browsing
US9672282B2 (en) * 2010-04-14 2017-06-06 Naver Corporation Method and system for providing query using an image
US20130144904A1 (en) * 2010-04-14 2013-06-06 Nhn Corporation Method and system for providing query using an image
WO2012170318A3 (en) * 2011-06-08 2013-01-31 Facebook, Inc. Presenting images as search results
KR20140051207A (en) * 2011-06-08 2014-04-30 페이스북, 인크. Presenting images as search results
KR101956691B1 (en) * 2011-06-08 2019-03-11 페이스북, 인크. Presenting images as search results
WO2012170318A2 (en) * 2011-06-08 2012-12-13 Facebook, Inc. Presenting images as search results
US10185769B2 (en) 2011-06-08 2019-01-22 Facebook, Inc. Presenting images as search results
US10387503B2 (en) 2011-12-15 2019-08-20 Excalibur Ip, Llc Systems and methods involving features of search and/or search integration
US10504555B2 (en) 2011-12-20 2019-12-10 Oath Inc. Systems and methods involving features of creation/viewing/utilization of information modules such as mixed-media modules
US10296158B2 (en) 2011-12-20 2019-05-21 Oath Inc. Systems and methods involving features of creation/viewing/utilization of information modules such as mixed-media modules
US11099714B2 (en) 2012-02-28 2021-08-24 Verizon Media Inc. Systems and methods involving creation/display/utilization of information modules, such as mixed-media and multimedia modules
US9843823B2 (en) 2012-05-23 2017-12-12 Yahoo Holdings, Inc. Systems and methods involving creation of information modules, including server, media searching, user interface and/or other features
US10417289B2 (en) * 2012-06-12 2019-09-17 Oath Inc. Systems and methods involving integration/creation of search results media modules
US10303723B2 (en) 2012-06-12 2019-05-28 Excalibur Ip, Llc Systems and methods involving search enhancement features associated with media modules
US8943074B2 (en) 2013-02-05 2015-01-27 International Business Machines Corporation Generating and displaying an image tour in response to a user search query
US10157231B1 (en) 2013-03-07 2018-12-18 Vast.com, Inc. Systems, methods, and devices for identifying and presenting identifications of significant attributes of unique items
US9465873B1 (en) 2013-03-07 2016-10-11 Vast.com, Inc. Systems, methods, and devices for identifying and presenting identifications of significant attributes of unique items
US12141854B1 (en) 2013-03-07 2024-11-12 Vast.com, Inc. Systems, methods, and devices for measuring similarity of and generating recommendations for unique items
US12008626B1 (en) 2013-03-07 2024-06-11 Vast.com, Inc. Systems, methods, and devices for measuring similarity of and generating recommendations for unique items
US10007946B1 (en) 2013-03-07 2018-06-26 Vast.com, Inc. Systems, methods, and devices for measuring similarity of and generating recommendations for unique items
US11886518B1 (en) 2013-03-07 2024-01-30 Vast.com, Inc. Systems, methods, and devices for identifying and presenting identifications of significant attributes of unique items
US9104718B1 (en) 2013-03-07 2015-08-11 Vast.com, Inc. Systems, methods, and devices for measuring similarity of and generating recommendations for unique items
US9710843B2 (en) 2013-03-07 2017-07-18 Vast.com, Inc. Systems, methods, and devices for measuring similarity of and generating recommendations for unique items
US9690857B1 (en) 2013-03-07 2017-06-27 Vast.com, Inc. Systems, methods, and devices for identifying and presenting identifications of significant attributes of unique items
US10572555B1 (en) 2013-03-07 2020-02-25 Vast.com, Inc. Systems, methods, and devices for identifying and presenting identifications of significant attributes of unique items
US10643265B2 (en) 2013-03-07 2020-05-05 Vast.com, Inc. Systems, methods, and devices for measuring similarity of and generating recommendations for unique items
US11423100B1 (en) 2013-03-07 2022-08-23 Vast.com, Inc. Systems, methods, and devices for identifying and presenting identifications of significant attributes of unique items
US9324104B1 (en) 2013-03-07 2016-04-26 Vast.com, Inc. Systems, methods, and devices for measuring similarity of and generating recommendations for unique items
US10942976B2 (en) 2013-03-07 2021-03-09 Vast.com, Inc. Systems, methods, and devices for identifying and presenting identifications of significant attributes of unique items
US11127067B1 (en) 2013-03-07 2021-09-21 Vast.com, Inc. Systems, methods, and devices for measuring similarity of and generating recommendations for unique items
US11651411B1 (en) 2013-03-13 2023-05-16 Vast.com, Inc. Systems, methods, and devices for determining and displaying market relative position of unique items
US10109001B1 (en) 2013-03-13 2018-10-23 Vast.com, Inc. Systems, methods, and devices for determining and displaying market relative position of unique items
US10839442B1 (en) 2013-03-13 2020-11-17 Vast.com, Inc. Systems, methods, and devices for determining and displaying market relative position of unique items
US9830635B1 (en) 2013-03-13 2017-11-28 Vast.com, Inc. Systems, methods, and devices for determining and displaying market relative position of unique items
US10963942B1 (en) 2013-12-10 2021-03-30 Vast.com, Inc. Systems, methods, and devices for generating recommendations of unique items
US10127596B1 (en) 2013-12-10 2018-11-13 Vast.com, Inc. Systems, methods, and devices for generating recommendations of unique items
US11204957B2 (en) 2014-02-19 2021-12-21 International Business Machines Corporation Multi-image input and sequenced output based image search
US10394882B2 (en) 2014-02-19 2019-08-27 International Business Machines Corporation Multi-image input and sequenced output based image search
US20200311126A1 (en) * 2016-03-29 2020-10-01 A9.Com, Inc. Methods to present search keywords for image-based queries
US12061642B2 (en) * 2016-03-29 2024-08-13 A9.Com, Inc. Methods to present search keywords for image-based queries
US10268704B1 (en) 2017-10-12 2019-04-23 Vast.com, Inc. Partitioned distributed database systems, devices, and methods
US11210318B1 (en) 2017-10-12 2021-12-28 Vast.com, Inc. Partitioned distributed database systems, devices, and methods
US11947577B2 (en) 2018-08-20 2024-04-02 Pinterest, Inc. Auto-completion based on content similarities
US11048734B1 (en) * 2018-08-20 2021-06-29 Pinterest, Inc. Auto-completion based on content similarities
US12106358B1 (en) 2020-07-24 2024-10-01 Vast.com, Inc. Systems, methods, and devices for unified e-commerce platforms for unique items

Also Published As

Publication number Publication date
US20070174269A1 (en) 2007-07-26

Similar Documents

Publication Publication Date Title
US7725451B2 (en) Generating clusters of images for search results
US7657504B2 (en) User interface for displaying images of sights
US7707208B2 (en) Identifying sight for a location
Jing et al. Igroup: web image search results clustering
US7702681B2 (en) Query-by-image search and retrieval system
US8370334B2 (en) Dynamic updating of display and ranking for search results
US8117256B2 (en) Methods and systems for exploring a corpus of content
US20090287676A1 (en) Search results with word or phrase index
US7240052B2 (en) Refinement of a search query based on information stored on a local storage medium
US20090254540A1 (en) Method and apparatus for automated tag generation for digital content
US8856125B1 (en) Non-text content item search
US20140032544A1 (en) Method for refining the results of a search within a database
US8606780B2 (en) Image re-rank based on image annotations
TW200842624A (en) Federated search implemented across multiple search engines
JP2017504105A (en) System and method for in-memory database search
EP1428139A2 (en) System and method for extracting content for submission to a search engine
CN109952571B (en) Context-based image search results
CN102262670A (en) Cross-media information retrieval system and method based on mobile visual equipment
EP2192503A1 (en) Optimised tag based searching
WO2014137590A1 (en) Social annotations for enhanced search results
Kennedy et al. Query-adaptive fusion for multimodal search
US8108410B2 (en) Determining veracity of data in a repository using a semantic network
Li et al. Word2image: towards visual interpreting of words
RU2473119C1 (en) Method and system for semantic search of electronic documents
JP2009205588A (en) Page search system and program

Legal Events

Date Code Title Description
AS Assignment

Owner name: MICROSOFT CORPORATION, WASHINGTON

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:JING, FENG;ZHANG, LEI;LI, MINGJING;AND OTHERS;REEL/FRAME:017479/0373;SIGNING DATES FROM 20060313 TO 20060417

Owner name: MICROSOFT CORPORATION,WASHINGTON

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:JING, FENG;ZHANG, LEI;LI, MINGJING;AND OTHERS;SIGNING DATES FROM 20060313 TO 20060417;REEL/FRAME:017479/0373

REMI Maintenance fee reminder mailed
LAPS Lapse for failure to pay maintenance fees
STCH Information on status: patent discontinuation

Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362

FP Lapsed due to failure to pay maintenance fee

Effective date: 20140525

AS Assignment

Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:034543/0001

Effective date: 20141014

OSZAR »