CN102236719A - Page search engine based on page classification and quick search method - Google Patents

Abstract

The invention discloses a search engine with classification display and a fast retrieval method. The search engine includes a classification module at the server end, which is used to classify each web page according to the Chinese library book classification method, and store the index of the classification results into the web page Index library; the result display module displays the web page index matching the keyword and the web page classification related to the keyword by column. The search engine better helps the user find the webpage that the user is interested in more quickly and accurately by the webpage classification information through the webpage classification information.

Description

Webpage Search Engine Based on Webpage Classification and Its Fast Finding Method

technical field

The invention belongs to the technical field of search engine optimization, and in particular relates to a web page search engine and a fast search method based on web page classification.

Background technique

With the rapid development of Internet technology, a large number of new web pages appear every day, and the number of Internet web pages increases dramatically. Internet webpages contain a wealth of information, and how to quickly search for the information you are interested in from the massive information contained in a large number of webpages is particularly important. If useful information cannot be found from Internet web pages within a limited and tolerable time, the development of the Internet will not be so rapid and its influence will not be so extensive.

Internet search engines extract information contained in web pages from Internet web pages, and store the information in a database. After the user enters keywords on the Internet search engine search webpage, the search engine finds out from the database the webpages related to the input keywords and provides them to the user. The earliest search engine can be traced back to the Archie system designed and realized by three students of McGill University in Canada in 1990. At that time, there was no so-called Internet, and the Archie system was not a search engine in the current sense. By inputting the file name, the user can use the Archie system to search which FTP server has the file with the file name available for download. With the emergence of the Internet, many Internet search engines in different senses such as WebCrawler, Excite, Infoseek, AltaVista, Yahoo, etc. have appeared one after another. WebCrawler is the first full-text search engine implemented by Brian Pinkerton of Washington University in 1994. It can search any word on any web page. And Yahoo is actually more accurately a directory search engine rather than a full-text search engine. Its category directory page collects a large number of classified websites. The webpage owner can register the webpages he owns on the category catalog webpage by himself. Other users can find interested webpages from the classified directory webpage according to the classified directory. Other webpage classification directory webpages also have such as Sina classification directory webpage, DMOZ (www.dmoz.org) and so on. In order to make better use of the Internet to promote their products, a large number of business users began to study the ranking rules of websites on category pages, and tried to use some methods to make their product information pages appear in the search results of search engines. To adjust the ranking of your website on the category page to make your website easier to be searched by search engines, and correspondingly appear in a higher position in the search results, this is search engine optimization. In addition, when crawling webpages of search engines, webpages are often crawled from these catalog webpages. The webpages included in these catalog webpages are more likely to be crawled by search engines, and thus displayed in the search results of search engines. These category pages are not real search engines, but these category pages can be artificially used to manipulate search engine page search results. Different from directory search engines, Google uses PageRank technology, which combines the fact that each webpage is linked by other webpages (especially some frequently visited webpages) into the priority level of the results searched by the search engine, so that it is recognized by other webpages. Web pages with more web links are more likely to be searched by search engines. This greatly improves the correlation between the search results of the search engine and the input keywords. Commercial search engines such as Google and Baidu have adopted this principle.

A search engine generally includes the following steps: web crawling, web word segmentation, web indexing, and web search. Each web page is associated with the participle it contains and the frequency of participle occurrence, and stored in the index database for use by search engines. In this way, when searching with a search engine, the webpages associated with the input keywords are generally arranged according to the degree of relevance as the search results. The most relevant web pages are arranged in front of the search results, and are more likely to be searched by users. When a user uses a search engine to search, generally speaking, the input keywords are limited, and the situation of two to three keywords is very common. In this way, the webpage results searched by the webpage search engine are often not necessarily the webpage that the user really wants to search for. Frequently, users cannot find the webpage they really want to search for by browsing the searched webpages page by page. In order to better provide users with the search results they really need, search engine companies have adopted some methods to provide users with better search results. Baidu search engine and Biying search engine can output search results according to pictures, videos, news, etc., so that users can better and more accurately search for the webpages that users are interested in in the categories of pictures, videos, and news. Google search engine and Youdao search engine not only include these, but also can classify the search result pages by time. For example, only pages that have been updated within the past day, week, month, or year are displayed. In addition, the search engine also points to webpages similar to the displayed result webpage by adding a link next to each displayed search result webpage, or by giving a summary or snapshot of each search result webpage, so that users do not have to go to each search result webpage. By browsing the webpage, we can know whether the webpage contains the results that the user is interested in. These methods improve the possibility of providing users with better and more accurate search results, but the result pages displayed by search engines may still contain a lot of uninteresting search results, and these information need to be filtered so as to provide users with the most interesting results as much as possible. pages of interest.

Contents of the invention

The purpose of the present invention is to provide a search engine with classification and display, which solves the problems in the prior art that the displayed results of the search engine often fail to realize the purpose of the user's search or the retrieved information is too complicated to make it difficult for the user to find accurate information.

In order to solve these problems in the prior art, the technical solution provided by the invention is:

A search engine with classified display, including on the server side:

The webpage crawling and preprocessing module is used to automatically collect webpages from the Internet, perform preprocessing to convert webpage information into computer-readable formatted text information, and regularly update webpage information and capture new webpage information in real time;

The indexing module is used to segment the formatted text information processed by the webpage crawling and preprocessing module, and establishes a webpage index library with correlation characteristics for each webpage and the word segmentation contained in it and the frequency of occurrence of the word segmentation;

The query module is used to respond to the query request of the user terminal, and search the web page index library established by the index module to obtain a list of search results matching the query request of the user terminal;

and on the client side:

The result display module is used for the user to input a keyword query request, and obtain a list of search results matching the keyword from the server-side query module, and display it to the user after being arranged in descending order of relevance;

It is characterized in that the search engine also includes a classification module on the server side, which is used to classify each web page according to the Chinese Library Book Classification, and store the classification result index into the web page index library; the result display module displays the results in columns Indexing of webpages matching keywords and classifying webpages related to keywords.

Preferably, each web page at the server end of the search engine is classified according to the Chinese Library Book Classification.

Preferably, the classification adopts manual classification or machine learning classification using particle swarm optimization algorithm.

Preferably, the result display module includes two columns of display windows, the first display window is used to display a list of webpages matching keywords; the second display window is used to display categories of webpages related to keywords.

Preferably, the two-column display windows are arranged left and right, the left side is the second display window, and the right side is the first display window; when the first display window focuses on the currently displayed webpage list item, the corresponding webpage category of the second display window is bolded or add color.

Preferably, the web page list item in the first display window has a hyperlink linked to the corresponding web page website, and when the user stays on the web page list item with the mouse, the web page category corresponding to the corresponding web page list item is bolded in the second display window or change color, and the right side of the corresponding webpage list item presents a webpage snapshot related to the corresponding webpage list item.

Preferably, when the user selects a webpage list item in the first display window, the user terminal will directly open the corresponding webpage website link for the user to browse; when the user selects the webpage classification in the second display window, the first display window clusters and displays List of webpages of the same webpage category selected by the user.

Preferably, the second display window further includes a webpage update time option, and the webpage update time option is set at the bottom of the webpage classification for users to choose.

The present invention also provides a method for quickly searching webpage search results based on webpage classification, characterized in that the method includes the following steps:

(1) Receive the query request from the client;

(2) In response to the query request, perform a query on the webpage index library that indexes the cumulative association degree of each webpage with the participle it contains and the frequency of occurrence of the participle and classifies each webpage to obtain the search that matches the query request result list;

(3) Arranging the search list in descending order according to the degree of relevance and dividing the sorted search list and the webpage categories of the search list into two columns to generate a webpage and present it to the user.

The present invention can better provide users with more accurate search results required by users. The search engine search results are displayed according to the webpage categories through the webpage classification information, thereby providing users with more accurate search results, enabling users to search for webpages of interest to users more quickly and better. The method for quickly finding webpage search results based on webpage classification can be combined with existing commercial search engines to provide users with the information they need to quickly find from the webpage search results provided by these commercial search engines.

After crawling webpages, word segmentation, indexing, and establishing the correlation between each webpage and the word segmentation it contains and the frequency of word segmentation, each webpage is classified according to the Chinese library book classification method, and each webpage can correspond to In one or several categories, store all the information in the database and use it when searching the webpage.

When searching the webpage, the search engine display page is divided into left and right grids, the left side displays the web page classification information, and the right side displays the search results. When the user enters keywords, the initial search results are displayed on the right side of the web page. The category of the webpage to which the webpage ranked first is displayed on the left. The left side also contains direct links back to the original search results and page number. Select other webpages in the search results displayed on the right, and the corresponding webpage category of the webpage will be displayed on the left. Double-selecting any web page on the right will open that web page for the user to browse this web page. Select the web page category displayed on the left, and the right will display the search results belonging to this web page category among all the search results of the search engine. In this way, the webpage links displayed on the right belong to the search results of the webpage category that the user is interested in, and the user can find the information that the user needs to find more quickly, better, and more accurately. Of course, some methods currently used by search engines can be combined to better serve users, such as by providing a summary or a snapshot of each search result web page.

While crawling webpages, word segmentation, indexing, and establishing correlations between each webpage and its contained word segmentation and the frequency of occurrence of word segmentation, as shown in Figure 1, add each webpage according to the book classification method of the Chinese library. This step of classification is stored in the database. The Chinese library book classification method adopted in this patent can be replaced by other webpage classification methods equally. Webpage classification can be done manually or by machine learning methods such as particle swarm optimization algorithm to classify webpages.

When searching a webpage with a search engine, the webpage search results given by the search engine are displayed as shown in FIG. 2 . The search engine search results are displayed on the right, and the webpage category is displayed on the left by bolding or adding color to the corresponding webpage category. The left side can display both the initial search results and a direct link to the page number in the upper part. The direct link to display the initial search results and page number is for the right to return directly to the search engine search results at any time. If the second page is selected, the second page directly from the search engine search results will be displayed on the right. When the search results are displayed for the first time, the category of the webpage to which the first webpage belongs is displayed on the left. If one of the webpages is selected from the webpage links displayed on the right, the webpage category corresponding to the webpage will be displayed on the left. Select a category on the left, and the right will display links to all webpages belonging to this webpage category in all search results of the search engine. Double selection of a webpage link on the right will directly open this webpage for users to browse. If the user knows the category corresponding to the webpage to be searched, the user can directly browse to the corresponding webpage classification category through the left webpage classification category to select the webpage category. After selecting this webpage category, only the webpage links belonging to this webpage category in the search engine search results will be displayed on the right. In this way, users can quickly and directly find the web pages they are interested in.

Some methods currently used by search engines can be combined to better serve users at the same time, such as by giving a summary or snapshot of each search result page, and only displaying the results of the past day, week, month or a year. Pages updated during the year, etc.

Compared with the scheme in the prior art, the advantages of the present invention are:

The invention can better help the user find the webpage that the user is interested in more quickly and accurately through the search engine through the webpage classification information.

Description of drawings

The present invention will be further described below in conjunction with accompanying drawing and embodiment:

Fig. 1 is the work flow chart that the present invention has the search engine that classification shows at server end;

Fig. 2 is a flow chart of the search engine with classified display in the user end of the present invention.

Detailed ways

The above solution will be further described below in conjunction with specific embodiments. It should be understood that these examples are used to illustrate the present invention and not to limit the scope of the present invention. The implementation conditions used in the examples can be further adjusted according to the conditions of specific manufacturers, and the implementation conditions not indicated are usually the conditions in routine experiments.

Example

The present invention has a search engine for classification and display, and the server side includes a webpage grabbing and preprocessing module, which is used to automatically collect webpages from the Internet, perform preprocessing to convert webpage information into computer-readable formatted text information, and regularly and real-time Update web page information and new web page information capture; index module, used to segment the formatted text information processed by the web page capture and preprocessing module, and establish each web page with the word segmentation it contains and the frequency of occurrence of the word segmentation A web page index library with the characteristics of relevance; a query module, used to respond to the query request of the user terminal, and search the web page index library established by the index module to obtain a list of search results matching the query request of the user terminal; a classification module, used to classify each The web pages are classified according to the book classification method of the Chinese library, and the index of the classification results is stored in the web page index library;

As shown in Figure 1 and Figure 2, web crawling uses the Sina classification directory webpage (http://dir.iask.com/) as the starting webpage, and continuously and orderly crawls new webpages according to the hyperlink list on the webpage , and add new hyperlinks on each crawled webpage to the hyperlink list for subsequent crawling of new webpages to provide webpage links. Each crawled webpage is marked in the hyperlink list as having been crawled this time, avoiding repeated crawling of crawled webpages or even entering an endless loop. Perform preprocessing on the captured webpage information, such as removing markup symbols in HTML files to obtain webpage text information. When crawling for the first time, all captured webpages and their corresponding webpage text information are stored in the database for use in subsequent steps. Web pages that are not crawled are also stored in the database but the marked web pages do not exist. After the first crawl, the web pages will be crawled regularly (such as at 10 o'clock every night). If the information on the web page is updated, the information of the web page is captured and processed and stored in the database. If there is no update, the information on the web page is not captured, and the content of the web page in the database remains unchanged. For webpages with updated webpage information, check whether the hyperlinks contained in it contain new webpages, and if so, add hyperlink lists to these new webpages to crawl new webpage information. In the process of web crawling, the number of times each web page is linked by other web pages is recorded and stored in the database. Finally, all stored webpages are classified according to the webpage information according to the Chinese library book classification method, and the classification categories are stored in the database.

In the database, the newly captured web pages are segmented according to the established word segmentation dictionary (such as 2-word dictionary, 3-word dictionary, proper noun dictionary, etc.), and then the frequency of occurrence of each word segmentation in the web page is established. The degree of correlation between a web page and each word that appears can further establish the degree of association between each word and all web pages that contain the word, which can be used to search web pages based on keywords when searching for web pages. In this way, when the user inputs the keywords to be searched, the query module will find out the webpages related to the keywords and sort them. The ranking of webpages is determined according to the degree of relevance between the webpage and the input keyword, the number of associated keywords and the number of times linked by other webpages. The more keywords associated with a web page, the higher the degree of association and the more times it is linked by other web pages, the higher the ranking of the web page.

The client includes a result display module, which is used for the user to input a keyword query request, and obtains a list of search results matching the keyword from the query module on the server side, and displays them to the user after being arranged in descending order of relevance; The result display module displays the web page index matching the keyword and the web page classification related to the keyword by columns.

Each webpage on the server side is classified according to the book classification method of the Chinese library, and the classification adopts manual classification. The user terminal result display module includes two display windows. The first display window is used to display the list of webpages matching the keyword; the second display window is used to display the classification of webpages related to the keyword. The two-column display windows are arranged left and right, the left side is the second display window, and the right side is the first display window; when the first display window focuses on the currently displayed webpage list item, the corresponding webpage category of the second display window is bolded or colored . The webpage list item in the first display window has a hyperlink linked to the corresponding webpage website, and when the user stays on the webpage list item by the mouse, the corresponding webpage category of the corresponding webpage list item is bolded or discolored in the second display window, A webpage snapshot related to the corresponding webpage list item is displayed on the right side of the corresponding webpage list item.

When the user selects the webpage list item in the first display window, the user terminal will directly open the corresponding webpage website link for the user to browse; A list of web pages in the same web category. The second display window also includes a webpage update time option, and the webpage update time option is set at the bottom of the webpage category for users to choose.

When performing a query, the server first receives the query request from the client; then responds to the query request, and indexes the web page index library that has the cumulative association degree between each web page and the words it contains and the frequency of occurrence of the word, and classifies each web page Execute a query on the search engine to obtain a search result list that matches the query request; finally, sort the search list in descending order according to the degree of relevance and divide the sorted search list and the web page category of the search list into two columns to generate a web page and present it to the user.

The above examples are only to illustrate the technical conception and characteristics of the present invention, and its purpose is to allow people familiar with this technology to understand the content of the present invention and implement it accordingly, and cannot limit the protection scope of the present invention. All equivalent changes or modifications made according to the spirit of the present invention shall fall within the protection scope of the present invention.

Claims (9)

1. A search engine with classified display, including on the server side: The webpage crawling and preprocessing module is used to automatically collect webpages from the Internet, perform preprocessing to convert webpage information into computer-readable formatted text information, and regularly update webpage information and capture new webpage information in real time; The indexing module is used to segment the formatted text information processed by the webpage crawling and preprocessing module, and establishes a webpage index library with correlation characteristics for each webpage and the word segmentation contained in it and the frequency of occurrence of the word segmentation; The query module is used to respond to the query request of the user terminal, and search the web page index library established by the index module to obtain a list of search results matching the query request of the user terminal; and on the client side: The result display module is used for the user to input a keyword query request, and obtain a list of search results matching the keyword from the server-side query module, and display it to the user after being arranged in descending order of relevance; It is characterized in that the search engine also includes a classification module at the server end, which is used to classify each web page according to the Chinese Library Book Classification, and store the classification result index into the web page index library; the result display module displays the results in columns Indexing of webpages matching keywords and classifying webpages related to keywords. 2. The search engine with classification display according to claim 1, characterized in that each webpage at the server end of the search engine is classified according to the Chinese Library Book Classification. 3. The search engine with classification display according to claim 2, characterized in that said classification adopts manual classification or machine learning classification using particle swarm optimization algorithm. 4. the search engine with classification display according to claim 1, characterized in that said result display module comprises two column display windows, the first display window is used to display the list of webpages matched with keywords; the second display window It is used to display the classification of webpages related to keywords. 5. The search engine with classification display according to claim 4, characterized in that said two column display windows are left and right settings, the left side is the second display window, and the right side is the first display window; the first display window focuses When the webpage list item is currently displayed, the corresponding webpage category in the second display window is bolded or colored. 6. The search engine with classification display according to claim 4, characterized in that the webpage list item in the first display window has a hyperlink with the corresponding webpage website link, when the user stays on the webpage list item by mouse, The webpage category corresponding to the corresponding webpage list item is bolded or changed in color in the second display window, and a webpage snapshot related to the corresponding webpage list item is displayed on the right side of the corresponding webpage list item. 7. The search engine with classification display according to claim 6, wherein when the user selects the webpage list item in the first display window, the user terminal will directly open the corresponding webpage website link for the user to browse; when the user selects the first When the category of webpages in the second display window is displayed, the first display window clusters and displays the list of webpages of the same category of webpages selected by the user. 8. The search engine with classification display according to claim 4, characterized in that said second display window further includes a web page update time option, and said web page update time option is set at the bottom of the web page classification for users to choose. 9. A method for quickly searching webpage search results based on webpage classification, characterized in that said method comprises the following steps: (1) Receive the query request from the client; (2) In response to the query request, perform a query on the webpage index database that indexes the cumulative correlation between each webpage and its participle and the frequency of occurrence of the participle, and classifies each webpage to obtain the search that matches the query request result list; (3) The search list is sorted in descending order according to the degree of relevance, and the sorted search list and the web page category of the search list are divided into two columns to generate a web page and present it to the user.

Images