JP2014056503A - Computer packaging method, program, and system for specifying non-text element matching communication in multilingual environment - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a computer packaging method, a program, and a system for effectively obtaining non-text information with high versatility which is suitably used in multilingual environment.SOLUTION: There is provided a computer packaging method for specifying a non-text element matching communication in multilingual environment using a database in which the non-text element can be searched. The method includes the steps of: receiving a search result of the database using a search condition of one language and a search condition of another language corresponding to the search condition, a specific language attribute being related to the non-text element included in the search result; imparting a score to the non-text element included in the search result according to similarity to another element to which a different language attribute is related; and specifying at least one non-text element included in the search result according to the score.

Description

  The present invention relates generally to information processing technology, and more particularly to a computer-implemented method, program, and system for identifying non-text elements suitable for communication in a multilingual environment.

  In recent years, search techniques for images, sounds, and other non-text information have been developed and are widely used. For example, the user can search for related images existing on the Internet by inputting one or more search keywords and executing an image search. The following patent documents and non-patent document lists include background techniques related to a search technique for non-text information.

Japanese Patent No. 4796435 (Japanese Patent Laid-Open No. 2007-299172) Japanese Patent No. 4340367 (Japanese Patent Laid-Open No. 2000-285141) Japanese Patent No. 2601111 (JP-A-6-162104) Special table 2008-547061 JP-A-10-289251

Google Inc., “About GoogleImages-Google Images Help”, [online], [searched September 11, 2012], Internet <URL: http://support.google.com/images/bin/answer.py? hl = en & answer = 112512> Kobi Reiter, Stephen Soderland, and Oren Etzioni, “Cross-LingualImage Search on the Web”, Proceeding of Twentieth International Joint Conference on Artificial Intelligence, published January 6, 2007 "Polyglot Image Search-Automatically translate keywords into 35 languages and search for images from around the world with Google Images", [online], [Search September 11, 2012], Internet <URL: http: // www .crystal-creation.com / software / polyglot-image-search />

  It may be intuitive, flexible and convenient to use non-text information (image, sound, etc.) in communication. However, since non-text information can be interpreted in various ways by people, miscommunication is likely to occur. For example, image icons may have different meanings in communities with different languages and cultures. Therefore, when communicating between people belonging to different communities, it may frequently occur that communication is not successful.

  Accordingly, one object of the present invention is to provide a computer-implemented method, program, and system for identifying non-text elements suitable for communication in a multilingual environment.

  To achieve the above object, a computer-implemented method for identifying non-text elements suitable for communication in a multilingual environment is provided using a database that is searchable for non-text elements. The method includes receiving a database search result using a search condition in one language and a search condition in another language corresponding to the search condition. A specific language attribute is associated with a non-text element included in the search results. The method further includes a step of assigning a score to the non-text element included in the search result according to the similarity with another element associated with a different language attribute, and the method includes the non-text element based on the score. Identifying at least one of the non-text elements to be included.

  Preferably, in the step of giving a score, a high score is given to a non-text element having a high degree of similarity with another element associated with a different language attribute. The method may further include translating a search condition in one language by a translation engine and generating a search condition in another language.

  Preferably, the non-text element is an image and / or sound. The image searched in the present invention includes a still image, a moving image, and any other type of image.

  Preferably, the method includes a step of searching the database using a search condition of a certain language and a search condition of another language corresponding to the search condition, and the language used for the search is set in a non-text element included in the search result. The method further includes associating as a language attribute.

  Preferably, the step of giving a score includes a step of calculating a shape score by performing comparison between a plurality of non-text elements for information on the shape included in the image element. The step of calculating the shape score may include a step of extracting the contour information of the image element and a step of comparing the contour information extracted between the plurality of image elements. More preferably, in the step of comparing the contour information, the contour information subjected to enlargement / reduction and / or orientation change is compared. The step of comparing the contour information may include a step of referring to a shape pattern classification dictionary prepared in advance, and a comparison between a plurality of non-text elements may be performed in consideration of the related shape pattern classification. .

  Preferably, the step of giving a score includes a step of calculating a color component score by comparing a non-text element with another non-text element for information on the color component included in the image image element. The step of calculating a color component score can include comparing representatives of the color component values of the entire image for a plurality of image elements. Preferably, the representative value of the color component is an average value, a median value, a mode value, and / or an upper value of the color component values of the image element.

  Preferably, the method further includes calculating a smoothness score based on the smoothness of the image element. The step of calculating the smoothness score includes the step of dividing the image element into a plurality of regions according to a predetermined rule, the step of calculating an occupation ratio of the same color for each of the plurality of regions, and the same color of each of the plurality of regions Calculating a smoothness score based on the occupation ratio. In addition, the method further includes a step of attempting character recognition of characters included in the image element using an optical character recognition technique, and a step of calculating a character recognition score based on the result of character recognition.

  Preferably, the step of scoring the non-text element includes a step of calculating a total score based on at least one of the shape score, the color component score, the smoothness score, and the character recognition score of the non-text element, The identifying step includes identifying at least one of the non-text elements included in the search result based on the total score.

  The outline of the present invention has been described above as a computer-implemented method for identifying non-text elements suitable for communication in a multilingual environment. However, the present invention is not limited to programs, program products, software, software products, systems or devices, etc. It can also be grasped.

  The program product or software product can include, for example, a storage medium storing the above-described program and software, or a medium for transmitting the program and software. The program may cause a computer to execute the steps of the method described above.

  It should be noted that the above summary of the invention does not enumerate all necessary features of the invention, and combinations or sub-combinations of these components can also be an invention.

It is a functional block diagram of an image search system in an embodiment of the present invention. It is a flowchart expressing the whole operation | movement of the image search system in embodiment of this invention. It is a flowchart expressing the first half of the operation | movement of the outline score calculation process or the color component score calculation across languages in embodiment of this invention. It is a flowchart expressing the second half of the operation | movement of the outline score calculation process or color component score calculation across languages in embodiment of this invention. It is a flowchart expressing the further detail of the contour score calculation process across languages in embodiment of this invention. It is a flowchart expressing the further detail of the color component score calculation process across languages in embodiment of this invention. It is a flowchart expressing the detail of the smoothness score calculation process across languages in embodiment of this invention. It is an image figure for demonstrating the non-text information database in embodiment of this invention. It is a figure for demonstrating an example of the search result in the image search system in embodiment of this invention. It is a figure for demonstrating an example of the search result in the image search system in embodiment of this invention. It is a figure for demonstrating an example of the search result in the image search system in embodiment of this invention. It is the figure which showed an example of the hardware constitutions of the information processing apparatus suitable in order to implement | achieve the image search system in embodiment of this invention.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings. However, the following embodiments do not limit the invention described in the claims, and are described in the embodiments. It should be noted that not all combinations of features that are present are essential to the solution of the invention.

  The present invention can be implemented in many different modes and should not be construed as being limited to the description of the embodiment. The same number is attached | subjected to the same element through the whole description of embodiment.

  In the embodiment of the present invention, an image icon that matches a search condition input by the user is presented to the user through information processing in the image search system. In particular, in the embodiment of the present invention, a highly versatile image icon suitable for use in communication among people belonging to communities having different cultures and languages is presented to the user.

  In the embodiment of the present invention, a query including a search condition (keyword) based on user input is translated into a plurality of languages by a translation engine. Then, an image search is executed based on the original keyword and the translation keyword. In the embodiment of the present invention, a specific language attribute is given to an image included in the image search result.

  Next, in the embodiment of the present invention, character recognition analysis, color component analysis, contour analysis, and smoothness analysis are performed on the image included in the image search result. For contour analysis and color component analysis, feature values are extracted for the contour and color components of each image. Next, the degree to which a similar image exists across languages in terms of contours and / or color components is analyzed using the extracted feature values. When there is a high degree of existence of a similar image to which an attribute of another language is given for a certain image, a high score is given to the certain image.

  In the embodiment of the present invention, character recognition analysis and smoothness analysis are performed to determine whether each image is suitable as a communication means. Specifically, when a character is included, the degree of language dependency increases, and as a result of character recognition, a high score is given to a character that includes a low degree of character. In addition, in order to eliminate those that are not suitable for communication means such as photographs, a high score is given to images with low smoothness as a result of smoothness analysis.

  In the embodiment of the present invention, based on the scores obtained by character recognition analysis, color component analysis, contour analysis, and smoothness analysis, that is, by normalizing these four scores and calculating the average value, An overall score of the image is obtained. Then, according to the total score, an image icon suitable as a communication means that uses the least language in the multilingual environment is specified and presented to the user.

  As a use case of the image search system according to the embodiment of the present invention, for example, an image icon suitable for communication in a multilingual environment obtained by accessing the image search system through a mobile terminal such as a smartphone is displayed on the mobile terminal. By sharing, it is possible to communicate between people who do not speak.

  As another use case, when a message / news is distributed to a large number of people, it is conceivable to distribute an image icon representing a message / news content obtained by an image search system. There are some people who can't read characters among the many people who can be distributed, and for such people, image icons suitable for communication in a multilingual environment provided by an image search system can be an effective source of information. .

  As another use case, for example, when an existing work such as an existing pamphlet designed for use in a local area is developed globally, the image icon included in the work means globally. The image search system according to the embodiment of the present invention can be used to verify whether or not, and to obtain a candidate of an image icon to be replaced if not.

  Hereinafter, embodiments of the present invention will be described in more detail with reference to FIGS.

  FIG. 1 is a functional block diagram of an image search system (100) according to an embodiment of the present invention. Each element shown in the functional block diagram of FIG. 1 is an information processing apparatus having the hardware configuration illustrated in FIG. 12, and stores a computer program such as an operating system and application program stored in the hard disk device 13 or the like in a main memory. 4 is loaded into the main CPU 1 and read by the main CPU 1 so that hardware resources and software cooperate with each other.

  An image search system (100) according to an embodiment of the present invention includes an input / output unit (105), a keyword extraction unit (110), an image search unit (115), a search result storage unit (120), and a search result analysis unit (125). , A total score calculation unit (150), a total score storage unit (155), and a presentation image determination unit (160).

  The input / output unit (105) provides an input / output interface between the image search system (100) and a user or another computer system. In the embodiment of the present invention, for example, the user inputs search conditions for image search through the input / output unit (105).

  The keyword extraction unit (110) according to the embodiment of the present invention interprets the search condition input through the input / output unit (105), extracts one or more search keywords, and delivers them to the image search unit (115). In the embodiment of the present invention, the search condition includes communication contents and language attributes. For example, when the keyword extraction unit (110) is entered as a natural sentence “Earthquake alert. Please be alert.”, Language attribute: Japanese, the search keywords “earthquake” “alarm” “warning” Provides a function to extract.

The keyword extraction unit (110) of the embodiment of the present invention performs keyword extraction using the well-known tf-idf technique. Here, tf is an abbreviation for term frequency, and idf is an abbreviation for inverse document frequency. For example, for each word i obtained by morphological analysis from sentence j, the value obtained by multiplying tf _{i, j} and idf _i obtained according to the following formula is greater than or equal to a predetermined threshold value. If so, the word i may be extracted as a keyword for search.

  Further, the keyword extraction unit (110) of the embodiment of the present invention translates the extracted search condition or the search keyword into one or more predetermined languages, obtains the corresponding search keyword, and the image search unit (with the language attribute) 115). The predetermined language may be a predetermined main language, may be designated each time by the user, or may be determined in combination of both. The predetermined language preferably includes the language of the communication partner.

  The image search unit (115) of the embodiment of the present invention queries the non-text information database (180) using the query including the search keyword and language attribute extracted by the keyword extraction unit (110), and uses the keyword as a keyword. Get multiple matching images. It is preferable to generate a query for each language attribute and to query the database. Also, one or more language attributes are given to the search result image obtained by the image search unit (115). As the language attribute given to the search result image, the language attribute given to the search keyword used when the image search unit (115) performed the search may be given to each image.

  An example of the non-text information database (180) of the present invention will be described in more detail with reference to FIG. FIG. 8 is an image diagram for explaining the non-text information database (180) in the embodiment of the present invention. The non-text information database (180) includes a language list (505), a language-specific first character list (510), a language-specific first character-specific word list (515), a language-specific first character-specific word pointer (520), and a content file storage. Part (525).

  The data stored in the non-text information database (180) of the present invention is obtained by crawling a web server (not shown) on the Internet. Specifically, the non-text information database (180) of the embodiment of the present invention is an alternative text associated with an image file name and image from a crawling engine (not shown) for an image on the Internet. Meta information (index / keyword) is generated by obtaining information such as alt attribute and text near the image. At that time, one or more language attributes are obtained for the image by investigating the language attribute, file name, or encoding method assigned to the image or the web page including the image. Yes.

  The non-text information database (180) of the present invention forms an index for the image to be stored for the image from the meta information and the language attribute. Specifically, the language attribute is stored in the language list (505), the first character of the index / keyword is stored in the first character list by language (510), and the index / keyword is stored in the word list (515) by first character by language. A pointer for the content stored in the content file storage unit (530) is stored in the language-specific first character-specific word-specific pointer (520). Each table (505) to (520) is indicated by a pointer.

  The language-specific first character-specific word-specific pointer (520) includes a ranking score expressing the popularity in a plurality of images having the same keyword index. The ranking score can be obtained by, for example, measuring the number of times the image is finally clicked in the image search result, and those skilled in the art can design various methods for obtaining the ranking score. It will not be described in detail here.

  When the non-text information database (180) of the embodiment of the present invention is queried using a query including a search keyword and a language attribute, first, the language attribute in the query and the language list (505) Matching is performed, and the first character list (510) by language corresponding to the matching language attribute is matched with the first character of the search keyword included in the query. Next, the language-specific first character-specific word list (515) corresponding to the matched first character and the search keyword in the query are matched, and the matched language-specific first character-specific word pointer (520) is specified. Then, the content file stored in the content file storage unit (530) having a high ranking score by a predetermined number is returned as a query response by using the specified language-specific first character-by-word pointer (520). It will be.

  The content file itself can be stored in the content file storage unit (530), or a URL (Uniform Resource Locator) indicating the location on the Internet where the content exists is stored in the content file storage unit (530). You may make it return. In the latter case, the image search unit (115) obtains the corresponding image from a web server on the Internet using the returned URL.

  In the embodiment of the present invention, the language list (505), the language-specific first character list (510), and the language-based first character-specific word list (515) are often used to improve the search speed. Columns can be raised dynamically to the top.

  The search result storage unit (120) of the embodiment of the present invention stores a search result including a plurality of images obtained by the image search unit (115). In the search result storage unit (120), each image of the search result is associated with one or more language attributes.

  The search result analysis unit (125) of the embodiment of the present invention analyzes a search result including a plurality of images searched by the image search unit (115) and stored in the search result storage unit (120). The search result analysis unit (125) includes a contour analysis unit (130), a color component analysis unit (135), a smoothness analysis unit (140), and a character recognition analysis unit (145).

  The contour analysis unit (130) according to the embodiment of the present invention extracts feature points of objects included in each image to acquire contour information, and analyzes the similarity of the contour information between the images. As a result, when the image to be analyzed often has a high degree of similarity between the contour information and another image having a different language attribute, a high “contour score” is given to the image.

  The color component analysis unit (135) of the embodiment of the present invention extracts the color component of each image and then analyzes the similarity of the color component between the images. As a result, a high “color component score” is given to an image that often has a high degree of similarity with an image of a color component that is given a different language attribute. The reason for such scoring is to eliminate images with high locality from the viewpoint of color components to represent a certain event. For example, the color of the post box is blue in the United States, red in Japan and the United Kingdom, green in the People's Republic of China, and yellow in Germany and France. Therefore, when searching for image icons for postboxes, it is necessary to underestimate such images because it is unlikely that it will be meaningful in a multilingual environment if it is an image that does not know if it is a postbox without color. .

  The smoothness analysis unit (140) of the embodiment of the present invention obtains a smoothness score based on the ratio of having a uniform color sub-region for each image. The smoothness analysis unit (140) gives a high “smoothness score” to an image having low smoothness. The reason is that images with high smoothness are often not suitable for information transmission because they are often like landscape photographs and are too intuitive and unclear due to too much information. It is.

  The character recognition analysis unit (145) of the embodiment of the present invention analyzes the degree to which each image of the search result includes language-dependent characters, and gives a high “character recognition score” to an image with a low degree. give. The reason is that if there are a lot of language-dependent characters, the image is not useful as a communication means for a partner who cannot understand the language. The character recognition analysis unit (145) can be implemented by applying an existing optical character recognition (also referred to simply as “OCR”) technology.

  Calculation of the “contour score”, “color component score”, “smoothness score”, and “character recognition score” will be described in detail later.

  The total score calculation unit (150) of the embodiment of the present invention receives “element score” from the search result analysis unit (125), specifically, “smoothness score”, “character recognition score”, “color component score”. The “contour score” is received from the search result analysis unit (125), and based on these, the “total score” is given to each image. The total score calculated by the total score calculation unit (150) of the embodiment of the present invention is stored in the total score storage unit (155). As an example, in the embodiment of the present invention, the “total score” is obtained by normalizing the “smoothness score”, “character recognition score”, “color component score”, and “contour score”, and then calculating an average value. Desired.

  The presentation image determination unit (160) of the embodiment of the present invention determines an image to be presented to the user according to the total score of each image stored in the total score storage unit (155), and the determined image is a search result storage unit. Obtained from (120) and delivered to the input / output unit (105) for presentation to the user. In the embodiment of the present invention, the image with the highest overall score is specified from the images included in the search result and presented to the user.

  FIG. 2 is a flowchart (200) representing the overall operation of the image search system in the embodiment of the present invention. The process starts in step (205), and in step (210), input of a search condition from the user, for example, a keyword or a natural sentence is accepted in a specific language. When the search condition is input in a natural sentence, in step (210), keyword extraction is performed using the tf-idf method described in relation to the keyword extraction unit (110).

  The process proceeds to step (215), and the search keyword in the specific language extracted in step (210) is translated into another predetermined language. The other language preferably includes a language used by a target audience in communication in another language environment, and preferably includes one or more predetermined main languages. Other languages may be specified by the user. In the embodiment of the present invention, a set of keywords in four languages, “English”, “Japanese”, “Chinese”, and “Spanish” is finally obtained.

  Next, in step (220), an image search is executed using a search keyword in a plurality of languages (four languages) obtained in step (215). In the embodiment of the present invention, a query including a search keyword and a corresponding language attribute is formed for each language, an image search is performed, and the language attribute included in the query is obtained for each resulting image. Give and associate as. The result of the image search is stored in the search result storage unit (120).

  In the embodiment of the present invention, after the step (220), the calculation processing of the “contour score”, “color component score”, “smoothness score”, and “character recognition score” is performed in parallel in each of the steps (225) to (240). Executed.

  When the calculation processing of “contour score”, “color component score”, “smoothness score”, and “character recognition score” in steps (225) to (240) is completed, the process proceeds to step (245). In step (245), for each image of the search result, a total score is calculated based on the “smoothness score”, “character recognition score”, “contour score”, and “color component score”. In the embodiment of the present invention, the “smoothness score”, “character recognition score”, “contour score”, and “color component score” are normalized and a simple average value is obtained to calculate the total score.

  Next, the process proceeds to step (250), an image to be presented is determined based on the total score calculated in step (245), and the determined image is presented to the user. In the embodiment of the present invention, an image having the highest total score is presented to the user. The process proceeds to step (255) and ends.

  An example of the contour score calculation process or the color component score calculation process across languages in the embodiment of the present invention will be described in more detail with reference to the flowcharts of FIGS. It should be noted that this contour score calculation process or color component score calculation process corresponds to step (225) or (230) of the flowchart (200) in FIG.

In the flowcharts shown in FIGS. 3 and 4, it is assumed that a contour score calculation process or a color component score calculation process is performed on an image l _ij given a language i as a language attribute. In the embodiment of the present invention, the steps of FIGS. 3 and 4 are executed by the contour analysis unit (130) or the color component analysis unit (135).

  FIG. 3 is a flowchart showing the first half of the operation of the contour score calculation process or the color component score calculation process across languages in the embodiment of the present invention. The process starts at step (305), and at step (310), an attempt is made to specify a language m other than the target language i.

The process proceeds to step (315), and it is determined whether there is another language m that has not been processed yet. If it is determined in step (315) that another language m exists, the process proceeds from the arrow of YES to step (320), and tries to extract one image l _mn given the language attribute of the language m, It is then determined in step (325) whether the image l _mn has been successfully retrieved.

When it is determined in step (325) that the image l _mn has been successfully extracted, the process proceeds from the YES arrow to step (330), and the target image l _ij and the image l _mn extracted in step (320) are processed. The matching score Smn is calculated. Details of specific processing of contour matching or color component matching will be described later in detail with reference to the flowcharts shown in FIGS. 5 and 6, and will not be described further here.

The process proceeds to step (335), and the matching score S _mn obtained in step (330) is the maximum matching score in matching performed in the past with the image having the language m as the language attribute for the target image l _ij. It is determined whether it is greater than Tm. In the embodiment of the present invention, in case the matching relating to the target image l _ij is performed first, the maximum matching score The T _m is assumed to 0 as an initial value is set.

When it is determined in step (335) that the matching score S _mn obtained in step (330) is larger than the maximum matching score T _m in the past matching process, the process proceeds to step (340), and the maximum matching score T _m Is replaced with the matching score S _mn obtained in step (330), and then the process proceeds to step (320). If it is determined in step (335) that the matching score S _mn obtained in step (330) is not larger than the maximum matching score T _m in the past matching process, the process returns to step (320).

After the processing returns to step (320), the processing from step (320) to step (340) described above is repeated until the image l _mn no longer exists. If it is determined in step (325) that the image l _mn does not exist during the repetition, the process exits the loop from the NO arrow and returns to step (310), and further steps (310) and after for other languages. Is repeated. If it is determined in step (315) that there are no other unprocessed languages anymore while the processing from step (310) is repeated, the processing goes from the arrow of NO to the mark (345) of (A). Then, the process proceeds to the flowchart of FIG.

FIG. 4 is a flowchart representing the second half of the operation of contour extraction or color component matching processing across languages in the embodiment of the present invention. The process starts from the mark (345) in (A), and in step (405), the next image l _{ik in} which j ≠ k is extracted from the set of images in which the language i is given as the language attribute.

Next, in step (410), it is determined whether an image l _ik exists. If it is determined in step (410), the process proceeds to step (415), and a matching score Ok between the target image l _ij and the image l _ik extracted in step (405) is calculated. Details of specific processing of contour matching or color component matching will be described later in detail with reference to the flowcharts shown in FIGS. 5 and 6, and will not be described further here.

The process proceeds to step (420), a target image l _ij, the accumulated value P of the past match scores same language i is given as a language attribute, it adds the O _k calculated in step (415). Thereafter, the process returns to step (405) and is repeated until it is determined in step (410) that the image l _ik does not exist. If it is determined in step (410) that the image l _ik does not exist, the process proceeds to step (425).

In step (425), a color component score or contour score for the image l _ij is calculated by weighting T _m and P, respectively. The process proceeds to step (430) and ends.

  The “contour score” calculation process in the embodiment of the present invention will be described in more detail with reference to FIG. FIG. 5 is a flowchart expressing further details of the contour score calculation process across languages in the embodiment of the present invention. Note that the processing according to this flowchart corresponds to step (330) or (415) of the flowchart of FIGS. The steps shown in FIG. 5 are executed by the contour analysis unit (130).

The process starts from step (505), and feature points of two images (here, images A and B) are extracted in step (510). Here, images A and B are images l _ij and l _mn at step (330), and images l _ij and l _ik at step (415). The feature points of each image can be calculated according to a method known as the Gabor Filter method, and specifically, can be obtained according to the following equation.

  The process proceeds to step (515), and the minimum distance dist1 between the feature point of image A and the feature point of image B extracted in step (510) is calculated. Note that the minimum distance here refers to the combination of the feature points of the image A and the feature points of the image B without leaking any feature points, and the feature points of the combined image A and the feature points of the image B. The sum of the distances in the case of forming the sum of the distances to be the minimum is said. Next, in step (525), the image A is reversed left and right, and the minimum distance distR between the inverted feature point of the image A and the feature point of the image B is calculated.

  Next, the process proceeds to step (530), and the optimal enlargement / reduction ratio of the images A and B is calculated. The optimum enlargement / reduction ratio is a ratio for enlarging / reducing one image so that the images A and B have the same size (or the same width or height when the aspect ratio is different). Shall. In step (535), a minimum distance distX between the feature point of image A and the feature point of image B that has been subjected to the enlargement / reduction processing using the enlargement / reduction ratio calculated in step (530) is calculated.

  Next, in step (540), the image A is horizontally reversed, and the optimal enlargement / reduction ratio is calculated for the horizontally reversed image A and image B. Next, in step (545), the minimum of the feature point of image A and the feature point of image B that have been horizontally reversed and enlarged / reduced using the enlargement / reduction ratio calculated in step (540) Calculate the distance distXR.

  The process proceeds to step (550), and the lowest value among dist1, distR, distX and distXR is selected as distOpt. Note that distR, distX, and distXR take into account that if an image is simply a horizontally reversed and / or enlarged / reduced image of another image, both should be treated as similar images. Next, in step (555), distOpt is stored as a contour matching score, and the process ends in step (560).

  The “color component score” calculation process in the embodiment of the present invention will be described in more detail with reference to FIG. FIG. 6 is a flowchart expressing further details of the color component score calculation processing across languages in the embodiment of the present invention. Note that the processing according to this flowchart corresponds to step (330) or (415) of the flowchart of FIGS. The steps shown in FIG. 6 are executed by the color component analysis unit (135).

The processing starts from step (605), and a predetermined color component (here, RGB component) is extracted for each pixel of two images (referred to as images A and B) in step (610). Here, images A and B are images l _ij and l _mn at step (330), and images l _ij and l _ik at step (415).

  Next, in step (615), an average value of each color component (that is, R (red), G (green), and B (blue)) in the images A and B is calculated. In step (620), The difference (absolute value) of the average value of each color component calculated in (615) is calculated, and then the total value (colorVal) of the difference of the average value of each color component is calculated in step (625). In step 630, colorVal is stored as a matching score for the color component, and the process ends in step (635).

  Next, the “smoothness score” calculation process in the embodiment of the present invention will be described in detail with reference to the flowchart shown in FIG. It should be noted that the processing according to the flowchart shown in FIG. 7 corresponds to step (235) of the flowchart (200) in FIG. Further, the steps of this flowchart are executed by the smoothness analysis unit (140).

  First, after the execution of step (220), the process starts from step (705), and in step (710), one image is extracted from the images included in the result of the image search in step (220). Next, in step (715), the image extracted in step (710) is divided into sub-regions (here, 3 × 3 pixel regions) according to a predetermined rule. The process proceeds to step (720), where one of the regions divided in step (715) is extracted, and in step (725), the color components (in this case, RGB values) of the region extracted in step (720) are obtained. Extract.

  The processing further proceeds, and in step (730), based on the color component extracted in step (725), it is determined whether or not there are more points of the same color in the target area than a predetermined ratio (here, 50%). Is done. If it is not determined in step (730) that the ratio is greater than the predetermined ratio, the process proceeds from the NO arrow to step (740). If it is determined in step (730) that the ratio is greater than the predetermined ratio, the process proceeds from the arrow of YES to step (735). In step (735), the variable “number of smooth regions” is incremented by 1, and the flow proceeds to step (740).

  In step (740), it is determined whether or not the unprocessed area obtained by the division in step (715) still remains. If it is determined in step (740) that the process remains, the process returns to step (720) through the YES arrow, and the subsequent processes are repeated. If it is determined in step (740) that the image does not remain, the process proceeds to step (745) through the NO arrow, and the smoothness score of the image to be processed is calculated according to the following equation.

  After the smoothness score is calculated in step (745), the process proceeds to step (750), and it is determined whether there are any unprocessed images remaining in the image search in step (220). If it is determined in step (750) that the process remains, the process returns from the YES arrow to step (710), and the subsequent processes are repeated. If it is determined in step (750) that the process remains, the process proceeds to step (755) from the NO arrow and ends (proceeds to step 245).

  Next, the “character recognition score” calculation process in the embodiment of the present invention will be described. As already described, the “character recognition score” calculation process is executed by the character recognition analysis unit (145). Using existing optical character recognition technology, it is possible to identify an area in the image that may contain characters (called "character area"), and match the image in that area with character font data prepared in advance. By performing the above, the probability that a character exists in the character region can be obtained. In the embodiment of the present invention, the “character recognition score” is calculated by calculating the reciprocal of the probability that a character exists in the character area.

  As already described, the total score calculation unit (150) of the embodiment of the present invention performs the “contour score”, “color component score”, “smoothness score”, and “character recognition score” in step (245). The total score is obtained by normalizing and calculating the average value. The normalization of each score may be performed such that each value falls within a predetermined range according to a predetermined rule. For example, the normalized contour score of image i can be obtained according to the following equation:

  For example, the normalized color component score of the image i can be obtained according to the following equation.

  Hereinafter, some examples of search results in the image search system according to the embodiment of the present invention will be described with reference to FIGS. Although FIGS. 9-11 are shown in black and white drawings due to the restriction that color drawings cannot be used in the application drawings, it should be noted that the actual example images are color icons. FIG. 9 is a diagram for explaining an example of a search result in the image search system according to the embodiment of the present invention. The example shown in FIG. 5 shows a case where “earthquake” in English is selected as an input keyword. Note that in this case, the corresponding translation keywords are Japanese “earthquake”, Chinese “earthquake”, and Spanish “terremoto”. In this example, the image obtained as the fourth candidate when searching using “earthquake” in English has the highest overall score (overall score = 0.88), so this image is presented to the user. It will be.

  FIG. 10 is a diagram for explaining another example of the search result in the image search system according to the embodiment of the present invention. The example shown in FIG. 6 shows a case where “tsunami” in English is selected as an input keyword. In this case, it should be noted that the corresponding translation keyword is “tsunami” in Japanese, “Kaien” in Chinese, and “tsunami” in Spanish. In this example, the image obtained as the fourth candidate when searching using “tsunami” in English has the highest overall score (overall score = 0.96), so this image is presented to the user. It will be.

  FIG. 11 is a diagram for explaining still another example of the search result in the image search system according to the embodiment of the present invention. In the example shown in FIG. 7, “exit” in English is selected as the input keyword. In this case, it should be noted that the corresponding translation keyword is “exit” in Japanese, “exit” in Chinese, and “salida” in Spanish. In this example, the image obtained as the fifth candidate when searching using the Japanese “exit” has the highest overall score (overall score = 0.96), so this image is presented to the user. The

  FIG. 12 is a diagram illustrating an example of a hardware configuration of an information processing apparatus suitable for realizing the image search system according to the embodiment of this invention. The information processing apparatus includes a CPU (central processing unit) 1 and a main memory 4 connected to the bus 2. The hard disk devices 13 and 30 and the CD-ROM devices 26 and 29, the flexible disk device 20, the MO device 28, and the removable storage (external storage system capable of exchanging recording media) such as the DVD device 31 are floppy disk controllers 19. Are connected to the bus 2 via the IDE controller 25, the SCSI controller 27, and the like.

  A storage medium such as a flexible disk, MO, CD-ROM, or DVD-ROM is inserted into the removable storage. In these storage media, the hard disk devices 13 and 30, and the ROM 14, instructions of a computer program for carrying out the present invention can be recorded by giving instructions to the CPU or the like in cooperation with the operating system. The computer program is executed by being loaded into the main memory 4. The computer program can be compressed or divided into a plurality of pieces and recorded on a plurality of media.

  The information processing apparatus receives input from an input device such as a keyboard 6 or a mouse 7 via the keyboard / mouse controller 5. The information processing apparatus is connected via a DAC / LCDC 10 to a display apparatus 11 for presenting visual data to the user.

  The information processing apparatus can connect to a network via a network adapter 18 (Ethernet (R) or the like) and communicate with other computers or the like. Although not shown in the figure, it is possible to connect to a printer via a parallel port or a modem via a serial port.

  As described above, the information processing apparatus suitable for realizing the image search system according to the embodiment of the present invention is realized by an information processing apparatus such as a normal personal computer, workstation, mainframe, or a combination thereof. It will be readily understood that However, these constituent elements are examples, and not all the constituent elements are essential constituent elements of the present invention.

  Various modifications such as combining a plurality of machines with each hardware component of the information processing apparatus used in the embodiment of the present invention, and allocating and executing functions to them can be easily assumed by those skilled in the art. Of course. These modifications are naturally included in the concept of the present invention.

  The image search system according to the embodiment of the present invention includes a Windows (R) operating system provided by Microsoft Corporation, a MacOS (R), UNIX (R) system provided by Apple Computer Incorporated (for example, an international system). Employing an operating system that supports a GUI (Graphical User Interface) multi-window environment, such as AIX® provided by Business Machines Corporation.

  From the above, it can be understood that the image search system used in the embodiment of the present invention is not limited to a specific operating system environment.

  The present invention may be implemented as a system, method or computer program product as will be appreciated by those skilled in the art. Therefore, the present invention may be implemented entirely as hardware, entirely as software (including firmware, resident software, microcode, etc.), or, here, generally “circuit”, “module”. Alternatively, it may take the form of an embodiment combining software and hardware called a “system”. Furthermore, the present invention may take the form of a computer program product implemented on a tangible expression medium having program code usable by a computer implemented on the medium.

  Combinations of computer usable / computer readable media may also be utilized. The computer usable / computer readable medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. A non-exhaustive list of more specific examples of computer readable media includes: Electrical connection with conductors, portable computer diskette, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber Portable compact disk read-only memory (CD-ROM), optical storage device, transmission medium supporting the Internet or intranet, magnetic storage device.

  The program is obtained electronically, for example, by optically scanning paper or other media, and then compiled, interpreted, and otherwise processed as appropriate, as required by computer memory. It should be noted that the computer-usable / computer-readable medium may be paper or other suitable medium on which the program is printed. A computer-usable / computer-readable medium is any medium that can contain, store, communicate, propagate, or carry a program for use by or in connection with an instruction execution system, apparatus or device. It may be a medium. A computer usable medium may include a propagated data signal including program code usable by a computer implemented together in baseband or as part of a carrier wave. Program code that can be used by a computer can be transmitted using any suitable medium including, but not limited to, wireless, wireline, fiber optic cable, RF, and the like.

  The computer program code that implements the present invention includes one or more object-oriented programming languages such as Java, Smalltalk, C ++, etc. and a conventional procedural programming language such as C programming language or similar programming language. Can be described using a combination of programming languages. Program code is entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on the remote computer, entirely on the remote computer Or it can run on the server. In the latter scenario, the remote computer can connect to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be an external computer. (E.g., via the internet using an internet service provider).

  The embodiments of the present invention have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatuses (systems) and computer program products. It will be understood that each block of the flowchart illustration and / or block diagram, and combinations of blocks in the flowchart illustration and / or block diagram, can be performed by computer program instructions. Means for instructions executed through the processor of a general purpose computer, special purpose computer, or other programmable data processing device to perform the functions / activities specified in one or more blocks of the flowcharts and / or block diagrams These computer program instructions can be provided to the processor of the computer or other programmable data processing device.

  These computer program instructions are stored in a computer or other programmable data processing device, the functions / operations in which instructions stored on a computer readable medium are specified in one or more blocks of a flowchart and / or block diagram. Can also be stored on a computer readable medium that can be operated in a particular manner to produce a product that includes instruction means for executing.

  Computer program instructions are provided such that instructions executing on a computer or other programmable data processing device provide a process for performing the functions / activities specified in one or more blocks of the flowcharts and / or block diagrams. In addition, to create a process for execution on a computer, the computer or other programmable data processing device may be loaded to perform a series of operational steps on the computer or other programmable data processing device. it can.

  The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and execution operations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code that includes one or more executable instructions for performing a specified logical function. It should also be noted that in certain alternative implementations, the functions shown in the blocks may be performed in a different order than that shown in the drawings. For example, depending on the functionality involved, two blocks shown in sequence may actually be executed almost simultaneously, or the blocks may be executed in reverse order. Each block shown in the block diagrams and / or flowcharts, and combinations of the blocks shown in the block diagrams and / or flowcharts, is a special purpose hardware-based system or special purpose hardware that performs a special function or activity. It can be executed by a combination of wear.

  It will be apparent to those skilled in the art that various modifications or improvements can be made to the above-described embodiment. For example, the embodiment of the present invention has been described on the assumption that a single image is presented to the user. For example, a plurality of images having a high overall score are presented based on the overall score. Is possible. In the embodiment of the present invention, the overall score is obtained by normalizing the “contour score”, “color component score”, “smoothness score”, and “character recognition score”, and then calculating the average value. However, those skilled in the art will be able to obtain an overall score from the “contour score”, “color component score”, “smoothness score” and “character recognition score” according to various techniques.

  In calculating the “contour score”, a shape pattern classification dictionary is prepared in advance, a related shape pattern classification is identified with reference to such a dictionary, and a plurality of patterns are considered in consideration of the identified pattern classification. It is possible to calculate the similarity between non-text elements. Further, in calculating the “color component score”, the average of the color component values of the image is obtained, but instead, the median, mode and / or upper value of the color component values of the image are used. In addition, the representative values of all color components can be used as appropriate. Furthermore, in calculating the “character recognition score”, the character recognition score can be lowered when more characters are included in the image element.

  As an example of non-text information to be searched, still images are taken up as an example. However, they can be made into a database so that they can be searched by indexing moving images, sounds, etc., and similar to each other Any non-text information or a combination thereof that can calculate the degree can be targeted. Further, for example, when a sound (sound) is targeted, an existing speech recognition technology can be used instead of the optical character recognition technology. Needless to say, embodiments to which the above modifications or improvements are made are also included in the technical scope of the present invention.

  As described above, according to the embodiment of the present invention, it can be understood that highly versatile non-text information suitable for use in a multilingual environment can be effectively obtained.

Claims (20)

A computer-implemented method for identifying non-text elements suitable for communication in a multilingual environment using a database capable of searching non-text elements, comprising:
Receiving a search result of the database using a search condition of a language and a search condition of another language corresponding to the search condition, wherein a specific language attribute is associated with a non-text element included in the search result Step, and
Giving a score according to the degree of similarity between the non-text element included in the search result and another element associated with a different language attribute;
Identifying at least one non-text element included in the search result based on the score;
Including a method.   The method of claim 1, wherein in the step of giving a score, a high score is given to non-text elements that are highly similar to other elements with different language attributes associated with them.   The method according to claim 1, further comprising translating the search condition of the certain language by a translation engine to generate the search condition of the other language.   The method of claim 1, wherein the non-text element is an image.   The method of claim 1, wherein the non-text element is speech. Searching the database using a search condition of the certain language and a search condition of another language corresponding to the search condition;
Associating the language used for the search with the non-text element included in the search result as a language attribute;
The method of claim 1, further comprising:   The method of claim 4, wherein the scoring comprises calculating a shape score by performing a comparison between a plurality of non-text elements for information about the shape contained in the image element.   8. The method of claim 7, wherein providing the shape score includes extracting contour information of image elements and comparing contour information extracted between a plurality of image elements.   9. The method according to claim 8, wherein in the step of comparing the contour information, the contour information subjected to enlargement / reduction and / or orientation change is compared.   The step of comparing the contour information includes a step of referring to a shape pattern classification dictionary prepared in advance, and performing comparison between a plurality of non-text elements in consideration of related shape pattern classification. The method described.   The step of providing the score includes calculating a color component score by comparing a non-text element with another non-text element for information on the color component included in the image image element. The method described.   The method of claim 11, wherein calculating a color component score includes comparing representatives of color component values of the entire image for a plurality of image elements.   The method according to claim 12, wherein the representative value of the color component is an average value, a median value, a mode value, and / or an upper value of the color component values of the image element.   The method of claim 4, further comprising calculating a smoothness score based on the smoothness of the image element.   The step of calculating the smoothness score includes the step of dividing the image element into a plurality of regions according to a predetermined rule, the step of calculating an occupation ratio of the same color for each of the plurality of regions, The method of claim 14, comprising calculating a smoothness score based on an occupancy ratio of each of the same colors.   The method according to claim 4, further comprising: trying to recognize characters of the character included in the image element using an optical character recognition technique; and calculating a character recognition score based on the result of character recognition.   17. The method of claim 16, wherein the character recognition score is lowered when more characters are included in the image element. Scoring the non-text element includes calculating a total score based on at least one of a shape score, a color component score, a smoothness score, and a character recognition score of the non-text element;
The step of identifying includes identifying at least one of non-text elements included in the search result based on the total score;
The method of claim 4.   A system comprising means for performing the steps of the method according to any of claims 1-18.   The program which makes a computer perform each step of the method as described in any one of Claims 1-18.