CN109344355A - Automatic returning detection and Block- matching adaptive approach and device for Web evolution - Google Patents
Automatic returning detection and Block- matching adaptive approach and device for Web evolution Download PDFInfo
- Publication number
- CN109344355A CN109344355A CN201811124012.XA CN201811124012A CN109344355A CN 109344355 A CN109344355 A CN 109344355A CN 201811124012 A CN201811124012 A CN 201811124012A CN 109344355 A CN109344355 A CN 109344355A
- Authority
- CN
- China
- Prior art keywords
- webpage
- block
- matching
- web evolution
- new
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/71—Version control; Configuration management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides the adaptive approach and device of a kind of automatic returning detection and Block- matching for Web evolution, can dynamically detect target webpage variation, and provide amending advice.Described to be characterized in that including the following steps: to detect Web evolution for the automatic returning detection of Web evolution and the adaptive approach of Block- matching, whether the webpage for detecting new and old goal systems changes and provides report；Content Block- matching is carried out, webpage is analyzed after receiving the report for detecting variation, finds out webpage content blocks corresponding with the webpage of the old goal systems part of fresh target system, content Block- matching includes: character analysis step, obtains semantic information and character area；Graphical interfaces analytical procedure obtains graphics field；Mapping step carries out similarity mode to semantic information achieved above, character area and graphics field, is then based on the variation of corresponding content blocks part, provides the code revision suggestion for existing web data extraction system tool.
Description
Technical field
The present invention relates to web data extractive technique fields, and in particular to for the automatic returning detection of Web evolution and block
Matched adaptive approach and device.
Background technique
With the development of Web technology and the arrival of big data era, Web page system starts to include more and more letters
Breath, this makes the miscellaneous technology for extracting data (carrier of information) from Web page system grow up.And it will be from webpage
Middle extract data just needs to position the data needed in webpage first, the positioning skill of common webpage interior element
Art includes the positioning based on XPath (XML Path Language, XML Path Language), is based on CSS (Cascading Style
Sheets, cascading style sheets) selector positioning and other simply according to id attribute, positioning of name attribute etc..These
Location technology is all highly dependent on the structure of webpage, after structure of web page changes, the data developed using these methods
Extracting tool has great failure likelihood, it is therefore desirable to manually re-start exploitation maintenance, this process is to expend very much
Manpower and time cost.
In order to solve the problems, such as Web evolution cause data with existing extracting tool fail, it is most important seek to find out it is new and old
The mapping relations of content blocks and back end between version webpage, so if these mappings of program Automatic-searching can be passed through
Relationship, so that it may greatly reduce artificial investment, and save time cost.For this problem, there is portion of techniques at present
Scheme is suggested, they can be divided into three directions substantially: one, being based on DOM (Document Object Model, text object
Model) structure；Two, based on the feature of data item；Three, it is based on machine learning.
Then method based on DOM structure utilizes calculating tree first by the web analysis of new and old two versions at dom tree
The algorithm of similarity finds the subtree closest with the target sub-tree structure of legacy version, the subtree in the dom tree of new version
Just it is taken as content blocks corresponding with legacy version webpage in new version webpage.However the complexity of the similarity calculating method of tree is inclined
Height is cured under gradually complicated trend in webpage, and overall performance is not high；Meanwhile the result only obtained only in accordance with DOM information is not
It is very accurate.
Method based on data item feature is based on the assumption that, practical although structure of web page is changed
Partial Feature (such as syntactic property, the hyperlink, annotating information) maximum probability for the data for needing to extract is constant.Therefore, may be used
To find out these features in the case where ensuring that the data extracted are correct situation, after webpage changes, existed using these features
Actual data item is matched in the page of new version, a configuration then provided according to user regenerates data extraction
Tool.This method needs user to provide a configuration file to data format first, and the configuration is complex cumbersome；Meanwhile
This method limits the generating mode of data extraction tool, and versatility is not high.
Finally, the method based on machine learning is to extract the feature of webpage by the analysis to a large amount of webpages, thus right
A variety of different webpages are classified, and it is similar that the webpage of each classification, which is considered data extracting mode,.Pass through the engineering
Learning method, after page variation, it is only necessary to which new number can be automatically generated by carrying out primary analysis to the webpage of new version again
According to extracting tool.But this method needs the support of mass data, while having higher requirements to the computing capability of equipment.
Summary of the invention
The technical problem to be solved in the present invention
In view of the foregoing, technical problem to be solved by the present invention lies in provide a kind of automatic time for Web evolution
The adaptive approach and self-reacting device for returning detection and Block- matching being capable of dynamics by the adaptive approach or self-reacting device
Ground detects the variation of target webpage system (being also referred to as " goal systems " sometimes below), and can be dynamically according to goal systems
Variation adjust existing data extraction tool, make its still be able to extract from the target webpage system of new version it is correct
Data.
Solve the technological means of technical problem
In order to solve the above-mentioned technical problem, present inventors have developed a kind of automatic returning inspections for Web evolution
Survey the adaptive approach with Block- matching, it is characterised in that include the following steps: to detect Web evolution, for detecting new and old target system
Whether the webpage (below otherwise referred to as " new and old edition webpage ") of system changes and provides report；Carry out content blocks (i.e. net
Some column part on page) matching, webpage is analyzed after receiving the report for detecting variation, finds out fresh target system
Webpage (following to be otherwise referred to as " new version webpage ") and the webpage of old goal systems (are otherwise referred to as " old edition Home Network below
Page ") corresponding content blocks part, content Block- matching includes: character analysis step, obtains semantic information and character area；Figure
Interface analysis step obtains graphics field；Mapping step, to semantic information achieved above, character area and graphics field into
Row similarity mode is then based on the variation of corresponding content blocks part, provides for available data extraction system or tool
Code revision suggestion.
The step of above-mentioned detection Web evolution of the invention preferably further includes: when new and old goal systems be not detected
When webpage changes, indeclinable report is provided, and Web evolution is detected to the webpage of the new and old goal systems inputted again.
Preferably, in the above-mentioned detection Web evolution of the invention the step of, the webpage can be divided into data acquisition
Class webpage and data submit class webpage.
In the above-mentioned detection Web evolution of the invention the step of, detect whether the data acquisition class webpage changes
Method be preferably, by the html code analysis of new and old edition webpage at DOM tree structure after, need test section according to what is inputted
The location information divided, which is got, needs the corresponding subtree of detection part, then compares whether sub-tree structure changes.
At this point, comparison the whether changed method of sub-tree structure be preferably, according to node id, class and tag it is suitable
Ordered pair ratio id, class and tag attribute value, if wherein have a value be it is identical, it is corresponding to be considered as the two nodes, then
The value for comparing another two attribute, judges whether they change.
In the above-mentioned detection Web evolution of the invention the step of, it is preferred that detect the data and submit the class webpage to be
No changed method is, by the html code analysis of new and old edition webpage at DOM tree structure after, according to the needs inputted
The location information of detection part, which is got, needs the corresponding subtree of detection part, then compares the son from new and old edition webpage
Whether tree construction changes.
At this point, comparison is preferably from the whether changed method of sub-tree structure of new and old edition webpage, for new
The sub-tree structure of legacy version webpage compares id, class and tag attribute value according to the sequence of id, class and tag of its node,
If wherein have a value be it is identical, be considered as that the two nodes are corresponding, so in comparison node other all properties value,
Compare the text information in the two corresponding nodes simultaneously, judges whether they change.
In the above-mentioned detection Web evolution of the invention the step of, it is preferred that detect the data and submit the class webpage to be
After no changed method is also possible that the html code analysis by new and old edition webpage at DOM tree structure, do not need to input
The location information for needing detection part compares whether they occur by extracting form list whole in DOM, input label
Variation.
In above-mentioned graphical interfaces analytical procedure of the invention, it is preferred that multiple rectangle frames are obtained by graphics process,
The hierarchical structure of rectangle frame is obtained according to the inclusion relation of rectangle frame, and combines the graphics field of the structural generation tree construction of DOM.
Of the invention includes such as lower unit: net for the automatic returning detection of Web evolution and the self-reacting device of Block- matching
Whether page variation detection unit, the webpage for detecting new and old goal systems change and provide report；Content Block- matching list
Member, for analyzing after receiving the report for detecting variation webpage, find out fresh target system webpage and old goal systems
The corresponding content blocks part of webpage, content blocks matching unit includes: character analysis module, for obtaining semantic information and text
Region；Graphical interfaces analysis module, for obtaining graphics field；Mapping block, for semantic information achieved above, text
Region and graphics field carry out similarity mode, are then based on the variation of corresponding content blocks part, provide for available data
The code revision suggestion of extraction system or tool.
Invention effect
The automatic returning for Web evolution through the invention detects and the adaptive approach of Block- matching or adaptive dress
It sets, can detect the variation that webpage generates in time, and alarm to related personnel；Variation for generation is analyzed automatically,
Finally give relevant the developer prompt and suggestion how to modify to the code of available data extraction system or tool.
Therefore, the present invention enables to the variation of webpage to be found in time, reduces as produced by the failure of existing data extraction tool
Economic loss；The manpower and time cost being adjusted to data extraction tool are considerably reduced simultaneously, reduces exploitation
The threshold that personnel safeguard tool；And the collection for not needing mass data, to equipment computing capability without too big requirement.
Detailed description of the invention
Fig. 1 is the adaptive side of automatic returning detection and Block- matching of an embodiment of the invention for Web evolution
The flow chart of method.
Fig. 2 is the flow chart for the step of an embodiment of the invention detects Web evolution.
Fig. 3 is the flow chart for the step of an embodiment of the invention carries out content blocks matching.
Fig. 4 is in an embodiment of the invention, and character analysis template is pressed in position and the structure for confirming text
Different grain size distinguishes obtained schematic diagram to text.
Fig. 5 be an embodiment of the invention in, graphical interfaces analysis module confirmation graphical interfaces position and
When structure, each obtained schematic diagram of phase process is carried out to graphical interfaces.
Fig. 6 is corresponding content blocks in the new and old edition webpage for detect in an embodiment of the invention variation.
Specific embodiment
Description of specific embodiments of the present invention with reference to the accompanying drawing.
Automatic returning for Web evolution of the invention detects the adaptive approach with Block- matching as shown in Figure 1, including
Following steps: whether detection Web evolution, the webpage for detecting new and old goal systems change and provide report；Progress in
Hold Block- matching, receive and webpage is analyzed after detecting the report of variation, find out fresh target system webpage and old target system
The corresponding content blocks part of the webpage of system, is then based on the variation of corresponding content blocks part, provides and extracts for available data
The code revision suggestion of system or tool.
Of the invention includes such as lower unit: net for the automatic returning detection of Web evolution and the self-reacting device of Block- matching
Whether page variation detection unit, the webpage for detecting new and old goal systems change and provide report；Content Block- matching list
Member, for analyzing after receiving the report for detecting variation webpage, find out fresh target system webpage and old goal systems
The corresponding content blocks part of webpage, be then based on the variation of corresponding content blocks part, provide for available data extract be
The amending advice of system or instrumentation code.
The present invention by using the above-mentioned automatic returning detection for Web evolution and Block- matching adaptive approach or from
Adaptive device is able to detect structure of web page variation, and corresponding content blocks and tool in the webpage of the new and old goal systems of Auto-matching
Volume data item, so that being automatically based upon variation provides amending advice.
It can be seen from the above, the automatic returning detection for Web evolution of the invention is suitable with the adaptive approach of Block- matching
In including Web evolution detection part and content blocks compatible portion.The two parts are further described below.
1. Web evolution detection part
Web evolution detection part, that is, Web evolution detection unit, it is main to use for the step of executing detection Web evolution
Whether changed in the new and old two version webpages of detection.Before new and old two version webpages for example can be five minutes
Webpage and the webpage after five minutes, specific time interval are adjustable.
In order to be detected, various methods can be chosen, such as compares data item feature or calculates DOM tree structure similarity
The methods of.It is detected in adaptive approach or device with Block- matching in the automatic returning for Web evolution of the invention, preferably
DOM is extracted in goal systems to indicate the integrally-built change detecting method of client.
For example, as shown in Fig. 2, DOM is converted to a tree dom tree (DOM tree), then state in system
Variation can be executed by system in the variation of this dom tree embody.If detecting variation, Web evolution detection part
Content blocks compatible portion after will start completes adaptive adjustment.If not detecting variation, Web evolution detection
Part can directly report unconverted result and be detected again after waiting the webpage of next version to input.
Whether certain a part that this part is responsible for detection webpage is changed.Certain solution is made to variation first
It releases, variation here is the variation of the variation of the DOM tree structure of webpage or certain attributes of DOM node, for some spies
The webpage for determining type further includes the variation of content of pages.
As specific detection mode, webpage can be divided by two big classifications: data acquisition according to the function of webpage
Class and data submit class.For different classifications, different detection methods is taken, as shown in Figure 3.
For data acquisition class webpage, html (the HyperText Markup of legacy version webpage can be inputted
Language, hypertext markup language) code, the html code of new version webpage, and the corresponding net in part for needing to detect
Page location information, the information that can be used for carrying out web page joint positioning includes XPath, CSS selector, id attribute, name attribute etc..
This method uses HtmlCleaner (the Html document resolver of Java language) tool by the html code solution of legacy version webpage first
DOM tree structure is analysed into, is then got according to the location information of input and needs the corresponding subtree of detection part.Then to new version
The html code of webpage does same operation, is likely encountered two kinds of situations here: the first is the html code of new version webpage
Specified subtree cannot be found in the dom tree of generation according to the location information of input, or has found multiple corresponding subtrees,
At this moment the report of " can not find corresponding blocks " or " finding multiple corresponding blocks " is just directly given, subsequent content Block- matching module is waited
Processing；Another kind is that unique subtree can be found in the dom tree of new version webpage, this begins to subsequent to subtree internal structure
Comparison algorithm.
How the structure compared from the subtree of new and old edition webpage changes, and just first to determine in two subtrees
Node how to correspond to.It can be based on id, class and tag attribute of the node, according to the sequence of id, class and tag, such as
Fruit have a value be it is identical, being considered as the two nodes is corresponding node.It is (following to be otherwise referred to as to find corresponding node
Node to) after, so that it may whether other attribute values for comparing the node pair identical.It, can be with for the webpage of data acquisition class
Only comparison id, class and tag attribute whether change, if variation if provide attribute change as a result, this is for correspondence
The detection of node attribute values.Following algorithm can be used: only such as the increase and deletion of node as the variation on tree construction
Only the same level of tree is compared, the case where rearranging of same level node is not considered yet, for same level
Node borrows the thought for seeking two string editing distances, finds out the additions and deletions for how passing through node and modification so that a node
Sequence becomes another sequence node, this process needs to judge whether two given nodes are equal, and the foundation of judgement is also
It is that node as described above looks for corresponding method.
Class webpage is submitted for data, two kinds of inspection policies can be provided, one of (data submit the inspection of class Web evolution
Survey mode one) it is essentially identical with the inspection policies of data acquisition class webpage, firstly, in the sub-tree structure of new and old edition webpage,
Id, class and tag attribute value are compared according to the sequence of id, class and tag of its node, if it is identical for wherein having a value
, it is considered as the two nodes and corresponds to, whether other the whole attribute values for then comparing the node pair are identical, the whole attributes
Value is including href, placeholder, value, name etc., not only id, class, these three attributes of tag；Can also it compare simultaneously
Compared with the specific text information of webpage.Another strategy (data submit class Web evolution detection mode two), inputs only new
The html code of legacy version, algorithm meeting automatic identification need to detect the part of variation, and algorithmic elaboration is as follows.
It is first still to convert dom tree for the webpage of new and old edition using HtmlCleaner, because being that data submit class
Webpage, so need to detect whether that the content occurred is exactly form list and a series of input label, as input,
select.Algorithm can traverse the corresponding dom tree of legacy version webpage first, extract whole form, input and select marks
Label, are respectively formed form list and input list.For form list, pair of node is first found out according to id, name, tag attribute
It should be related to, method is the sequence according to above-mentioned attribute, has an attribute value is equal to be taken as corresponding node, then to comparison
Action, entype and the method attribute for answering node pair detect whether there is variation, if it is, quoting corresponding variation.
The then variation of report " addition node " or " deletion of node " respectively of corresponding node is not found in list.The comparison of Input list
Then first can according to tag information, type attribute and partial text information (such as the text in previous node, either
Text in placeholder attribute) node corresponding relationship is found, processing later is also the variation of similar detection attribute value
With the additions and deletions of node.
2. content blocks compatible portion
Content blocks compatible portion, that is, content blocks matching unit is the core of Adaptable System of the invention, is used to execute
The step of content blocks match, major function is some features that variation is not likely to happen found on legacy version webpage, and with this
The matching that new and old edition webpage is carried out as standard, so where is changed in location client system, and have occurred why
The variation of sample.The two information are only obtained, corresponding region can be found in new version webpage, needed for obtaining or submitting
The information wanted, and modification prompt is provided according to the Web evolution of matching content block.
Three modules of content blocks compatible portion point: character analysis module, interface layout analysis module, mapping block, mapping
Module can also be divided into semantic mapping module and area map module further directed to the analysis result of two above module, such as scheme
Shown in 3.
The present invention utilizes OCR (Optical Character ReCognition, optical character by character analysis module
Identification) technology confirmation text position and structure.The result that this step obtains is distinguished by different grain size, such as word, sentence, section etc.,
As shown in Figure 4.
By character analysis module, the coordinate at four angles of all boxes and box on available interface, in figure
Box is exactly the block in the webpage identified comprising text, i.e. character area one by one；It can also be further in webpage
Text carries out semantic analysis.
Specific semantic analysis algorithm be exactly found out in a series of html codes for not quoting page variation it is constant (interior
Hold and position it is all constant) text information.For example, the page comprising headline list, daily news is different, so
The content of list is also different, but the page may include title as similar " XX news ", its position in the different pages
It is the same with text, these constant information show semanteme expressed by webpage different masses.
The present invention confirms position and the structure of graphical interfaces by graphical interfaces analysis module, as shown in Figure 5.
Graphical interfaces analysis module, which is handled, to be completed using OpenCV (open source computer vision library) kit.It is first
Limb recognition first is done to original image using the Canny algorithm provided in OpenCV；The edge identified is obscured later
Change processing, makes the edge that may do not connect in back be connected to one piece, obtains the profile diagram in each region；Then into
The processing of row grid profile, seeks each profile its boundary rectangle；Containerization processing is finally carried out, each rectangle is analyzed, is led to
It crosses graphics process and obtains multiple rectangle frames, the hierarchical structure of rectangle frame is obtained according to the inclusion relation of rectangle frame, and combine DOM's
The graphics field of structural generation tree construction, these graphics fields on webpage picture or character area it is corresponding.
Next, being carried out in conjunction with character analysis module and the calculated data of graphical interfaces analysis module as feature
The comparison of new and old edition webpage.For example, carrying out semantic similarity matching by semantic mapping module for semantic information；For
Character area and graphics field carry out structural similarity matching by area map module.Thus new version webpage is navigated to obtain
The region of information is taken and submitted, completes content Block- matching, and modification prompt is provided according to Web evolution therein.
The specific method of content Block- matching is illustrated below with reference to a webpage example, for other webpages, is also regarded
Actual conditions take specific method appropriate.In numerous webpages, to detect certain trip of new and old edition Web evolution shown in Fig. 6
For remembering webpage, the left side is legacy version webpage, and the right is new version webpage, and the content for needing to pay close attention to is divided into inside black surround, i.e., interior
Hold block.
For detecting the new and old edition webpage of variation in variation detecting step, its text is obtained by character analysis module
Block domain and semantic information.Confirmed using OCR (Optical Character ReCognition, optical character identification) technology
The position of text and structure are distinguished acquired results by different grain size, such as word, sentence, section, obtain including text in webpage
Block, i.e. character area.Semantic information is obtained simultaneously.For the text in webpage travel notes shown in Fig. 6, as " popular travel notes ", " newest
Deliver ", " temporally " these and every travel notes have oneself title, author, abstract etc. because each travel notes part is
Dynamic change, there is its content of text if the travel notes newly delivered that will change, so being not suitable as text feature；And picture
" popular travel notes " these texts not will become complete different content generally, it has reacted this Web page content block in fact
Semantic information, so it is suitable for as feature text information.The specific method for finding out these feature text informations is to mention first
Then these texts are spliced into a character string by a series of text information for taking legacy version webpage wholes, that acquires out is multiple
The longest common subsequence of character string, the result obtained are exactly the constant feature text information for including, these features in webpage
It is used in the matching that content blocks are carried out in mapping block later.In addition to text information, (href of such as a label belongs to link information
Property) generally also will not change, because this is relevant to the file structure of website, and the file structure of a website becomes in webpage
Changing front and back also can largely remain unchanged.
On the other hand, graphics field is obtained by graphical interfaces analysis module.Due to the content for including on a webpage
Structure generally changes less, travel notes webpage as shown in Figure 6, each travel notes be all a picture, a title, one section abstract,
And other some data, and entire content block is exactly to be made of a series of travel notes substantially.So this constant knot
Structure feature may be used for the matching that content blocks are carried out in mapping block.These structural informations can be obtained from two approach
Take --- dom tree and image analysis.Dom tree itself is the structure of a stratification, and is known that by each label each
The content of a node is picture, title or other texts, while can also very easily identify list.But dom tree is deposited
In a problem, granularity is too thin if carrying out cutting according to its node, for example abstract part may be cut into multiple sections
Point in this way can lose the information of certain semantic aspects of structure；And it is based on image analysis, available bigger region,
For example an entirety, the available complete holding of the semantic information in region each in this way will be maintained into if abstract part.
So the available more perfect structure of web page information of the two is combined, to greatly improve subsequent similarity calculation
Accuracy rate.
Followed by mapping block.After obtaining features described above information, the calculating of similarity is exactly in two structures (i.e. region)
In, if there is an identical matched part (if there is picture, there is title text), then some weighting of increase of being scored
Value (picture, title and general text weight may be different), that highest part of last matching degree is exactly new version page
The content blocks for needing to find in face.The biggish matched part that compares is found first.But inside this content blocks
Structure is it can also happen that certain variations, for example in Fig. 6 two corresponding content block pages change.Then content blocks are found out
Internal each data item (being exactly the information for needing to extract, such as the information such as the title of every travel notes, author, abstract in webpage) it
Between mapping relations.Specific mode is the statistical nature based on every data item, such as text size, digital density, character
Whether density is some certain moduli formulas (such as the text for indicating DEG C feature great in this way of temperature) etc., in the new version page
Middle searching and that closest data item of these statistical natures, using it as corresponding with original data item one
?.
It, can be to the variation that this region generates to deployment after navigating to new version webpage and obtaining and submit the region of information
Personnel propose amending advice, then manually modify.For example, if being positioned based on Xpath, when navigating to new edition Home Network
Behind region in page, the new XPath that each data item has in new version webpage can be obtained.Then, current number is scanned
According to the existing code of extracting tool, analysis wherein may the sentence comprising XPath (the specific method of this block is to check that code is each
Whether sentence is comprising function call related with XPath calculating is carried out, for example presets some function names as evaluateXPath), so
XPath of the prompt code in this should the XPath to be modified in new version webpage afterwards.
Therefore, through the invention the adaptive approach of the automatic returning detection for Web evolution and Block- matching or from
Adaptive device can timely and accurately detect that the variation of webpage generation provides amending advice to related personnel's alarm and automatically,
Significantly reduce manpower and time cost.
Claims (10)
1. a kind of adaptive approach of automatic returning detection and Block- matching for Web evolution, it is characterised in that including walking as follows
It is rapid:
Web evolution is detected, whether the webpage for detecting new and old goal systems changes and provide report；
Content Block- matching is carried out, webpage is analyzed after receiving the report for detecting variation, finds out the webpage of fresh target system
Content blocks corresponding with the webpage of old goal systems part, content Block- matching include: character analysis step, obtain semantic information with
Character area；Graphical interfaces analytical procedure obtains graphics field；Mapping step, to semantic information achieved above, character area
Similarity mode is carried out with graphics field, is then based on the variation of corresponding content blocks part, provides and is extracted for available data
The code revision suggestion of system or tool.
2. the adaptive approach of the automatic returning detection and Block- matching according to claim 1 for Web evolution, special
The step of sign is, detects Web evolution further comprises: when the webpage that new and old goal systems is not detected changes, giving
Indeclinable report out, and Web evolution is detected to the webpage of the new and old goal systems inputted again.
3. the adaptive approach of the automatic returning detection and Block- matching according to claim 1 or 2 for Web evolution,
In the step of being characterized in that, detecting Web evolution, the webpage is divided into data acquisition class webpage and data submit class webpage.
4. the adaptive approach of the automatic returning detection and Block- matching according to claim 3 for Web evolution, special
Sign is that detecting the whether changed method of data acquisition class webpage is, by the html of the webpage of new and old goal systems
After code analysis is at DOM tree structure, need the location information of detection part to get to need detection part pair according to what is inputted
Then the subtree answered compares whether sub-tree structure changes.
5. the adaptive approach of the automatic returning detection and Block- matching according to claim 4 for Web evolution, special
Sign is, the comparison whether changed method of sub-tree structure is, according to the sequence comparison id of id, class and tag of node,
Class and tag attribute value, if wherein have a value be it is identical, it is corresponding to be considered as the two nodes, then compares another two
The value of attribute, judges whether they change.
6. the adaptive approach of the automatic returning detection and Block- matching according to claim 3 for Web evolution, special
Sign is that detecting the data submission whether changed method of class webpage is, by the html of the webpage of new and old goal systems
After code analysis is at DOM tree structure, need the location information of detection part to get to need detection part pair according to what is inputted
Whether the subtree answered, the sub-tree structure then compared from the webpage of new and old goal systems change.
7. the adaptive approach of the automatic returning detection and Block- matching according to claim 6 for Web evolution, special
Sign is that the whether changed method of sub-tree structure compared from the webpage of new and old goal systems is, for new and old mesh
The sub-tree structure of the webpage of mark system compares id, class and tag attribute according to the sequence of id, class and tag of its node
Value, if wherein have a value be it is identical, be considered as the two nodes correspondence, and then other all properties in comparison node
Value, while comparing the text information in the two corresponding nodes, judge whether they change.
8. the adaptive approach of the automatic returning detection and Block- matching according to claim 3 for Web evolution, special
Sign is that detecting the data submission whether changed method of class webpage is, by the html of the webpage of new and old goal systems
After code analysis is at DOM tree structure, the location information that input needs detection part is not needed, it is only whole in DOM by extracting
Form list, input label compare whether they change.
9. the adaptive approach of the automatic returning detection and Block- matching according to claim 1 or 2 for Web evolution,
It is characterized in that, in graphical interfaces analytical procedure, multiple rectangle frames is obtained by graphics process, according to the inclusion relation of rectangle frame
It obtains the hierarchical structure of rectangle frame, and combines the graphics field of the structural generation tree construction of DOM.
10. a kind of self-reacting device of automatic returning detection and Block- matching for Web evolution, it is characterised in that including as follows
Unit:
Whether Web evolution detection unit, the webpage for detecting new and old goal systems change and provide report；
Content blocks matching unit finds out fresh target system for analyzing after receiving the report for detecting variation webpage
Webpage content blocks corresponding with the webpage of old goal systems part, content blocks matching unit includes: character analysis module, for obtaining
Obtain semantic information and character area；Graphical interfaces analysis module, for obtaining graphics field；Mapping block, for obtaining above
Semantic information, character area and the graphics field obtained carries out similarity mode, is then based on the variation of corresponding content blocks part,
Provide the code revision suggestion for available data extraction system or tool.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811124012.XA CN109344355B (en) | 2018-09-26 | 2018-09-26 | Automatic regression detection and block matching self-adaption method and device for webpage change |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811124012.XA CN109344355B (en) | 2018-09-26 | 2018-09-26 | Automatic regression detection and block matching self-adaption method and device for webpage change |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109344355A true CN109344355A (en) | 2019-02-15 |
| CN109344355B CN109344355B (en) | 2022-03-15 |
Family
ID=65306539
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811124012.XA Active CN109344355B (en) | 2018-09-26 | 2018-09-26 | Automatic regression detection and block matching self-adaption method and device for webpage change |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109344355B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110968761A (en) * | 2019-11-29 | 2020-04-07 | 福州大学 | A method for adaptive extraction of web page structured data |
| CN111079043A (en) * | 2019-12-05 | 2020-04-28 | 北京数立得科技有限公司 | Key content positioning method |
| CN111158973A (en) * | 2019-12-05 | 2020-05-15 | 北京大学 | Web application dynamic evolution monitoring method |
| CN112115111A (en) * | 2019-06-20 | 2020-12-22 | 上海怀若智能科技有限公司 | OCR-based document version management method and system |
| CN112887381A (en) * | 2021-01-15 | 2021-06-01 | 中国地质大学（武汉） | Method and device for detecting and converging new content facing specific network entrance |
| CN113626028A (en) * | 2020-05-07 | 2021-11-09 | 腾讯科技（深圳）有限公司 | Page element mapping method and device |
Citations (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| AU2001268674B2 (en) * | 2000-06-22 | 2007-04-26 | Microsoft Technology Licensing, Llc | Distributed computing services platform |
| CN101026503A (en) * | 2006-02-24 | 2007-08-29 | 国际商业机器公司 | Unit detection method and apparatus in Web service business procedure |
| CN101127044A (en) * | 2007-06-08 | 2008-02-20 | 北京大学 | Blocking Method for Dynamic Web Pages |
| CN101141449A (en) * | 2007-10-22 | 2008-03-12 | 珠海金山软件股份有限公司 | Apparatus and method for implementing Web client terminal software self-adaptive running |
| CN101174899A (en) * | 2007-11-26 | 2008-05-07 | 中兴通讯股份有限公司 | Automatic test method for service protection and recovery in ASON network |
| CN101178708A (en) * | 2006-11-07 | 2008-05-14 | 北京酷讯科技有限公司 | Automatic moulding plate information locating method for structured web page |
| CN101207639A (en) * | 2007-12-03 | 2008-06-25 | 华为技术有限公司 | Method and apparatus of classifying for user |
| CN101251855A (en) * | 2008-03-27 | 2008-08-27 | 腾讯科技（深圳）有限公司 | Equipment, system and method for cleaning internet web page |
| CN101266603A (en) * | 2007-03-12 | 2008-09-17 | 北京搜狗科技发展有限公司 | Webpage information sorting method, system and service system applying the classification |
| US20080281834A1 (en) * | 2007-05-09 | 2008-11-13 | Microsoft Corporation | Block tracking mechanism for web personalization |
| CN101408877A (en) * | 2007-10-10 | 2009-04-15 | 英业达股份有限公司 | Tree node loading system and method thereof |
| CN101477571A (en) * | 2009-01-07 | 2009-07-08 | 华天清 | Method and apparatus for marking network contents semantic structure |
| CN101546261A (en) * | 2008-10-10 | 2009-09-30 | 华中科技大学 | Secure web page tag library system supported by multiple strategies |
| CN101593184A (en) * | 2008-05-29 | 2009-12-02 | 国际商业机器公司 | The system and method for self-adaptively locating dynamic web page elements |
| CN101655862A (en) * | 2009-08-11 | 2010-02-24 | 华天清 | Method and device for searching information object |
| US20100287132A1 (en) * | 2009-05-05 | 2010-11-11 | Paul A. Lipari | System, method and computer readable medium for recording authoring events with web page content |
| CN102004805A (en) * | 2010-12-30 | 2011-04-06 | 上海交通大学 | Webpage denoising system and method based on maximum similarity matching |
| CN102663023A (en) * | 2012-03-22 | 2012-09-12 | 浙江盘石信息技术有限公司 | Implementation method for extracting web content |
| CN102662969A (en) * | 2012-03-11 | 2012-09-12 | 复旦大学 | Internet information object positioning method based on webpage structure semantic meaning |
| CN102890681A (en) * | 2011-07-20 | 2013-01-23 | 阿里巴巴集团控股有限公司 | Method and system for generating webpage structure template |
| CN102955854A (en) * | 2012-11-06 | 2013-03-06 | 北京中娱在线网络科技有限公司 | Webpage presenting method and device based on HTML5 (Hypertext Markup Language 5) protocol |
| US20130332451A1 (en) * | 2012-06-06 | 2013-12-12 | Fliptop, Inc. | System and method for correlating personal identifiers with corresponding online presence |
| US20140123186A1 (en) * | 2002-05-10 | 2014-05-01 | Convergent Media Solutions Llc | Method and apparatus for browsing using alternative linkbases |
| CN108345687A (en) * | 2018-03-09 | 2018-07-31 | 沈文策 | A kind of 3D web page display method and apparatus |
-
2018
- 2018-09-26 CN CN201811124012.XA patent/CN109344355B/en active Active
Patent Citations (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| AU2001268674B2 (en) * | 2000-06-22 | 2007-04-26 | Microsoft Technology Licensing, Llc | Distributed computing services platform |
| US20140123186A1 (en) * | 2002-05-10 | 2014-05-01 | Convergent Media Solutions Llc | Method and apparatus for browsing using alternative linkbases |
| CN101026503A (en) * | 2006-02-24 | 2007-08-29 | 国际商业机器公司 | Unit detection method and apparatus in Web service business procedure |
| CN101178708A (en) * | 2006-11-07 | 2008-05-14 | 北京酷讯科技有限公司 | Automatic moulding plate information locating method for structured web page |
| CN101266603A (en) * | 2007-03-12 | 2008-09-17 | 北京搜狗科技发展有限公司 | Webpage information sorting method, system and service system applying the classification |
| US20080281834A1 (en) * | 2007-05-09 | 2008-11-13 | Microsoft Corporation | Block tracking mechanism for web personalization |
| CN101127044A (en) * | 2007-06-08 | 2008-02-20 | 北京大学 | Blocking Method for Dynamic Web Pages |
| CN101408877A (en) * | 2007-10-10 | 2009-04-15 | 英业达股份有限公司 | Tree node loading system and method thereof |
| CN101141449A (en) * | 2007-10-22 | 2008-03-12 | 珠海金山软件股份有限公司 | Apparatus and method for implementing Web client terminal software self-adaptive running |
| CN101174899A (en) * | 2007-11-26 | 2008-05-07 | 中兴通讯股份有限公司 | Automatic test method for service protection and recovery in ASON network |
| CN101207639A (en) * | 2007-12-03 | 2008-06-25 | 华为技术有限公司 | Method and apparatus of classifying for user |
| CN101251855A (en) * | 2008-03-27 | 2008-08-27 | 腾讯科技（深圳）有限公司 | Equipment, system and method for cleaning internet web page |
| CN101593184A (en) * | 2008-05-29 | 2009-12-02 | 国际商业机器公司 | The system and method for self-adaptively locating dynamic web page elements |
| CN101546261A (en) * | 2008-10-10 | 2009-09-30 | 华中科技大学 | Secure web page tag library system supported by multiple strategies |
| CN101477571A (en) * | 2009-01-07 | 2009-07-08 | 华天清 | Method and apparatus for marking network contents semantic structure |
| US20100287132A1 (en) * | 2009-05-05 | 2010-11-11 | Paul A. Lipari | System, method and computer readable medium for recording authoring events with web page content |
| CN101655862A (en) * | 2009-08-11 | 2010-02-24 | 华天清 | Method and device for searching information object |
| CN102004805A (en) * | 2010-12-30 | 2011-04-06 | 上海交通大学 | Webpage denoising system and method based on maximum similarity matching |
| CN102890681A (en) * | 2011-07-20 | 2013-01-23 | 阿里巴巴集团控股有限公司 | Method and system for generating webpage structure template |
| CN102662969A (en) * | 2012-03-11 | 2012-09-12 | 复旦大学 | Internet information object positioning method based on webpage structure semantic meaning |
| CN102663023A (en) * | 2012-03-22 | 2012-09-12 | 浙江盘石信息技术有限公司 | Implementation method for extracting web content |
| US20130332451A1 (en) * | 2012-06-06 | 2013-12-12 | Fliptop, Inc. | System and method for correlating personal identifiers with corresponding online presence |
| CN102955854A (en) * | 2012-11-06 | 2013-03-06 | 北京中娱在线网络科技有限公司 | Webpage presenting method and device based on HTML5 (Hypertext Markup Language 5) protocol |
| CN108345687A (en) * | 2018-03-09 | 2018-07-31 | 沈文策 | A kind of 3D web page display method and apparatus |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112115111A (en) * | 2019-06-20 | 2020-12-22 | 上海怀若智能科技有限公司 | OCR-based document version management method and system |
| CN110968761A (en) * | 2019-11-29 | 2020-04-07 | 福州大学 | A method for adaptive extraction of web page structured data |
| WO2021103557A1 (en) * | 2019-11-29 | 2021-06-03 | 福州大学 | Adaptive extraction method for webpage structured data |
| CN110968761B (en) * | 2019-11-29 | 2022-07-08 | 福州大学 | A method for adaptive extraction of web page structured data |
| CN111079043A (en) * | 2019-12-05 | 2020-04-28 | 北京数立得科技有限公司 | Key content positioning method |
| CN111158973A (en) * | 2019-12-05 | 2020-05-15 | 北京大学 | Web application dynamic evolution monitoring method |
| CN111158973B (en) * | 2019-12-05 | 2021-06-18 | 北京大学 | A method for monitoring the dynamic evolution of web applications |
| CN111079043B (en) * | 2019-12-05 | 2023-05-12 | 北京数立得科技有限公司 | Key content positioning method |
| CN113626028A (en) * | 2020-05-07 | 2021-11-09 | 腾讯科技（深圳）有限公司 | Page element mapping method and device |
| CN112887381A (en) * | 2021-01-15 | 2021-06-01 | 中国地质大学（武汉） | Method and device for detecting and converging new content facing specific network entrance |
| CN112887381B (en) * | 2021-01-15 | 2022-07-19 | 中国地质大学（武汉） | Method and apparatus for new content detection and aggregation for specific network portals |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109344355B (en) | 2022-03-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109344355A (en) | Automatic returning detection and Block- matching adaptive approach and device for Web evolution | |
| US11256856B2 (en) | Method, device, and system, for identifying data elements in data structures | |
| US5794257A (en) | Automatic hyperlinking on multimedia by compiling link specifications | |
| US8799772B2 (en) | System and method for gathering, indexing, and supplying publicly available data charts | |
| US7941420B2 (en) | Method for organizing structurally similar web pages from a web site | |
| Kovbasistyi et al. | Method for detection of non-relevant and wrong information based on content analysis of web resources | |
| US8359294B2 (en) | Incorrect hyperlink detecting apparatus and method | |
| US8676814B2 (en) | Automatic face annotation of images contained in media content | |
| WO2008021561A2 (en) | Joint optimization of wrapper generation and template detection | |
| US20090125529A1 (en) | Extracting information based on document structure and characteristics of attributes | |
| CN102662969B (en) | A Method for Locating Internet Information Objects Based on Webpage Structural Semantics | |
| Papadakis et al. | Stavies: A system for information extraction from unknown web data sources through automatic web wrapper generation using clustering techniques | |
| CN109325201A (en) | Generation method, device, equipment and the storage medium of entity relationship data | |
| CN106960058B (en) | Webpage structure change detection method and system | |
| CN111079043A (en) | Key content positioning method | |
| US20060200457A1 (en) | Extracting information from formatted sources | |
| CN103853738A (en) | Identification method for webpage information related region | |
| US20100185684A1 (en) | High precision multi entity extraction | |
| JP2019032704A (en) | Table data structuring system and table data structuring method | |
| CN111158973B (en) | A method for monitoring the dynamic evolution of web applications | |
| US20080015843A1 (en) | Linguistic Image Label Incorporating Decision Relevant Perceptual, Semantic, and Relationships Data | |
| CN105279249B (en) | A method and device for determining the confidence of point of interest data in a website | |
| CN105160032B (en) | The determination method and device of the confidence level of interest point data in a kind of website | |
| CN115186240A (en) | Social network user alignment method, device and medium based on relevance information | |
| CN114238735A (en) | Intelligent internet data acquisition method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder | ||
| CP01 | Change in the name or title of a patent holder |
Address after: No. 826, building 12345, Phoenix legend, Hanbang, Jingyue Development Zone, Changchun City, Jilin Province Patentee after: Intel Technology Co.,Ltd. Address before: No. 826, building 12345, Phoenix legend, Hanbang, Jingyue Development Zone, Changchun City, Jilin Province Patentee before: Changchun interui Software Co.,Ltd. |
|
| CP03 | Change of name, title or address | ||
| CP03 | Change of name, title or address |
Address after: No. 826, building 12345, Phoenix legend, Hanbang, Jingyue Development Zone, Changchun City, Jilin Province Patentee after: Changchun interui Software Co.,Ltd. Address before: Room 1626, No. 65, North Fourth Ring West Road, Haidian District, Beijing 100080 Patentee before: BEIJING INTERNETWARE Ltd. |
|
| CP03 | Change of name, title or address | ||
| CP03 | Change of name, title or address |
Address after: 130117, 30th floor, Building A2, Mingyu Plaza, No. 3777 Ecological Street, Jingyue High tech Industrial Development Zone, Changchun City, Jilin Province Patentee after: Shenqi Digital Co.,Ltd. Country or region after: China Address before: No. 826, building 12345, Phoenix legend, Hanbang, Jingyue Development Zone, Changchun City, Jilin Province Patentee before: Intel Technology Co.,Ltd. Country or region before: China |