CN106570060B - Random extraction method and device for data in information flow - Google Patents

Abstract

Embodiments of the present invention provide a method and device for randomly extracting data in an information stream, so as to effectively reduce memory and database bandwidth consumption and improve data reading performance. The random extraction method of data in the information flow, including: setting random extraction parameters according to the business requirements of randomly extracting data in the information flow, including: the total amount of the latest dynamic data and the number of random extractions; determined according to the total amount of the latest dynamic data and the number of random extractions The number of pages, and the logical page number is randomly selected according to the determined number of pages; according to the logical page number, the corresponding cache key value is queried whether there is cache data, and if so, the cache data corresponding to the cache key value is directly read in the cache for display. ; Otherwise, read the paging data in the database for display according to the logical page number and the randomly selected quantity, and store the read paging data in the cache, and the cache key value includes the service unique identification code and the logical page number.

Description

Method and device for random extraction of data in information flow

technical field

The invention belongs to the technical field of computers, and in particular relates to a method and device for randomly extracting data in an information stream.

Background technique

With the advancement of Internet technology, the development of social network (Social Network Service, SNS) is eye-catching. Social networks play an important role in people's lives, and have become the network business with the widest coverage, the greatest influence, and the highest commercial value.

At present, the production and dissemination of news feeds in social networks is very fast. Taking the Weibo platform as an example, the big screen of Weibo (also known as Weibo Wall) is used in exhibitions, concerts, industry summits, live TV, weddings, etc. It is usually necessary to randomly select part of the data from the latest dynamic data of the information flow for display in the large screen of microblogs with a specific theme) and other interactive products in places such as the scene. For example, randomly select 10 pieces of data from the latest 1,000 pieces of data in the information flow for display, or randomly select 50 pieces of data from the latest 10,000 pieces of data in the information flow for display, and so on.

In this embodiment of the present invention, the actual requirement of randomly extracting part of the data from the latest dynamic data of the information flow for display may be referred to as "the business requirement of randomly extracting data from the information flow" for short. Assume that the business requirement of randomly extracting data in the information flow is: randomly extract n pieces of data from the latest N pieces of data in the information flow for display. The solutions provided in the prior art are:

Step 1. Read the latest N pieces of data from the information flow in the database according to business requirements, and obtain the total number N of reads; in order to improve performance, the latest N pieces of data generally read will be cached.

Step 2. Use a random function to generate n random numbers (n is less than N) required in the range of (0, N), and query the n pieces of data to be extracted from the N pieces of cached data according to the random n serial numbers for display.

In the solutions provided by the prior art, there are the following problems:

1. In the process of reading the latest N pieces of data from the information flow in the database, the memory usage is large and the database bandwidth transmission cost is high.

2. After randomly generating n random numbers, the process of obtaining n pieces of data one by one from the latest N pieces of data in a circular manner is relatively expensive.

3. Since the latest N pieces of data read from the information flow will be cached, the latest dynamic data of the information flow during the buffering period cannot get exposure opportunities when the flow speed is high.

SUMMARY OF THE INVENTION

In view of this, the present invention provides a method and device for randomly extracting data in an information stream, so as to effectively reduce memory and database bandwidth consumption and improve data reading performance.

In order to achieve the above object, the present invention provides a method for randomly extracting data in an information flow, including:

Set random extraction parameters according to the business requirements of randomly extracting data in the information flow, and the random extraction parameters include: the total amount of the latest dynamic data and the number of random extractions;

Determine the number of paging according to the total amount of the latest dynamic data and the number of random extractions, and randomly select the logical page number according to the determined number of paging;

According to the randomly selected logical page number, query whether the corresponding cache key has cached data, and if so, directly read the cached data corresponding to the cached key in the cache for display;

Otherwise, read the paging data in the database according to the randomly selected logical page number and the randomly selected quantity for display, and store the read paging data in the cache, where the cache key value corresponding to the cached paging data includes the unique business identifier code and the logical page number.

Further, the random extraction parameter further includes the cache time of each page; and the expiration time corresponding to the cached paging data is set according to the cache time of each page.

Further, determining the number of paging according to the total amount of the latest dynamic data and the number of random extractions is specifically implemented by the following formula:

P=int(N/n)+1

Among them, P represents the number of pages, N represents the total amount of the latest dynamic data, n represents the number of random extractions, and int() is the rounding function.

Further, randomly selecting the logical page number according to the determined paging quantity is specifically realized by the following formula:

p=rand(0, P)

Among them, p represents a randomly selected logical page number, P represents the number of pages, and rand() is a random function for generating random numbers.

Further, the page data is read and displayed in the database according to the randomly selected logical page number and the randomly selected quantity, which specifically includes:

The index position is determined according to the randomly selected logical page number and the randomly selected number, which is specifically realized by the following formula:

S=(p-1)*n

Among them, S represents the index position, p represents the randomly selected logical page number, and n represents the number of random selections;

Determine that the read length is consistent with the random sampling number;

According to the determined index position and read length, the paging data is read in the database for display.

Based on the same technical concept, the present invention also provides a device for randomly extracting data in an information stream, including:

a setting unit, configured to set random extraction parameters according to the business requirements of randomly extracting data in the information flow, and the random extraction parameters include: the total amount of the latest dynamic data and the number of random extractions;

The unit for determining the number of pages is used to determine the number of pages according to the total amount of the latest dynamic data and the number of random extractions;

The page number random selection unit is used to randomly select the logical page number according to the determined number of pages;

The cache query unit is used to query whether the corresponding cache key value has cache data according to the randomly selected logical page number. If there is, directly read the cache data corresponding to the cache key value in the cache for display; otherwise, trigger the database reading unit ;

The database reading unit is used to read the paging data in the database for display according to the randomly selected logical page number and the randomly selected number, and store the read paging data in the cache, wherein the cache key corresponding to the cached paging data The value includes the business unique identification code and the logical page number.

Further, the random extraction parameter set by the setting unit also includes the cache time per page; and,

The expiration time corresponding to the paged data cached by the database reading unit is set according to the cache time of each page.

Further, the unit for determining the number of pages is specifically implemented by the following formula:

P=int(N/n)+1

Among them, P represents the number of pages, N represents the total amount of the latest dynamic data, n represents the number of random extractions, and int() is the rounding function.

Further, the page number random selection unit is specifically realized by the following formula:

p=rand(0, P)

Among them, p represents a randomly selected logical page number, P represents the number of pages, and rand() is a random function for generating random numbers.

Further, the database reading unit specifically includes:

The index position determination subunit is used to determine the index position according to the randomly selected logical page number and the randomly selected number, which is specifically realized by the formula S=(p-1)*n, where S represents the index position, p represents the randomly selected logical page number, n represents the number of random draws;

a read length determination subunit, used to determine that the read length is consistent with the random extraction quantity;

The reading subunit is used to read the dynamic data in the database for display according to the determined index position and the reading length;

The storage subunit is configured to store the read paging data in the cache, wherein the cache key value corresponding to the cached paging data includes the service unique identification code and the logical page number.

The method and device for randomly extracting data in the information flow provided by the present invention are implemented by paging and dividing the latest dynamic data in the information flow at the logical level according to the business requirement of randomly extracting data in the information flow. First, determine the number of pages according to the total amount of the latest dynamic data and the number of random extractions, and randomly select the logical page number according to the determined number of pages; directly read the paged data in the database for display according to the logical page number and the number of random extractions, and read The paging data can be stored in the cache, and reading the paging data from the database does not need to traverse all the latest dynamic data, and only caches the paging data read out, thereby reducing memory and server bandwidth consumption; The output paging data is cached and stored. If the corresponding cache key value has cached data according to the logical page number query, the cached data can also be directly read for display, which greatly improves the reading performance. Compared with the prior art, this solution does not need to read all the latest dynamic data in the information flow in the database into the cache; it also does not need to use a circular method to retrieve the latest dynamic data from the cache after generating corresponding random numbers according to the randomly selected quantity. Each piece of data is extracted one by one, which can effectively reduce memory and database bandwidth consumption and improve data read performance.

Other features and advantages of the present application will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the present application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description, claims, and drawings.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts. The accompanying drawings are used to provide a further understanding of the present invention, and constitute a part of the specification, and are used to explain the present invention together with the embodiments of the present invention, and do not constitute a limitation to the present invention. In the attached image:

1 is a schematic diagram of a business requirement solution for randomly extracting data from an information flow in an embodiment of the present invention;

Fig. 2 is the flow chart of the random extraction method of data in the information flow in the embodiment of the present invention;

3 is a flowchart of a method for randomly extracting n pieces of data from the latest N pieces of data in an information flow in an embodiment of the present invention;

4 is a block diagram of a random extraction device for data in an information flow in an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a possible structure of a database reading unit in an embodiment of the present invention.

Detailed ways

Embodiments of the present invention provide a method and device for randomly extracting data in an information stream, so as to effectively reduce memory and database bandwidth consumption and improve data reading performance.

The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, but not to limit the present invention. And the embodiments in this application and the features in the embodiments may be combined with each other without conflict.

In order to effectively reduce the consumption of memory and database bandwidth, and improve the data reading performance, the embodiment of the present invention is aimed at the actual requirement of randomly extracting part of the data from the latest dynamic data of the information flow for display, that is, "the business requirement of randomly extracting data from the information flow". Corresponding solutions are provided, and the schematic diagram is shown in Figure 1. This scheme divides the latest dynamic data in the information flow into logical pages, then randomly selects the logical page number, and finally reads the paging data according to the randomly selected logical page number, so as to randomly extract part of the data from the latest dynamic data of the information flow the goal of.

The method for randomly extracting data in an information flow provided by an embodiment of the present invention, as shown in FIG. 2 , includes:

S201. Set random extraction parameters according to the business requirements of randomly extracting data in the information flow; the random extraction parameters include: the total amount of the latest dynamic data and the number of random extractions.

S202. Determine the number of pages according to the total amount of the latest dynamic data and the number of random extractions.

S203: Randomly select a logical page number according to the determined number of pages.

S204: Query whether the corresponding cache key value has cached data according to the randomly selected logical page number, and if so, execute S205, otherwise, execute S206.

S205, directly reading the cache data corresponding to the cache key value in the cache for display.

S206. Read the paging data in the database for display according to the randomly selected logical page number and the randomly selected quantity, and store the read paging data in the cache, wherein the cache key value corresponding to the cached paging data includes the service unique identifier code and logical page numbers;

Preferably, the random extraction parameter may also include the cache time of each page, and accordingly, the expiration time corresponding to the cached paging data is set according to the cache time of each page; if the random extraction parameter does not include the cache time of each page, then the cached page The expiration time corresponding to the data can be set to the system default value.

Those of ordinary skill in the art can understand that the "reading and displaying paging data in the database according to the randomly selected logical page number and the randomly selected quantity" refers to the information flow in the database according to the randomly selected logical page number and the randomly selected quantity. Read the paging data from the latest dynamic data for display. It should be noted that when reading data in the database, only the index position and read length can be specified. In the specific implementation, it is necessary to first convert the randomly selected logical page number and the randomly selected number into the index position and read number that the database can recognize. length, and then read the paging data. Specifically, the index position is determined according to the randomly selected logical page number and the randomly selected number, and the read length is determined to be consistent with the randomly selected number.

Assume that the business requirement of randomly extracting data in the information flow is: randomly extract n pieces of data from the latest N pieces of data in the information flow for display, and describe in detail the method for randomly extracting data in the information flow provided by the embodiment of the present invention, as shown in FIG. 3 . display, including:

S301. Set random extraction parameters according to business requirements for randomly extracting data in the information flow; the random extraction parameters include: the total amount of latest dynamic data N, the number of random extractions n, and the cache time t per page.

S302: Determine the number of pages P according to the latest dynamic data total amount N and the random extraction number n, which is specifically implemented by the following formula [1]:

P=int(N/n)+1 [1]

Among them, int() is the rounding function, which is a function that rounds down a value to the nearest integer.

S303. Randomly select the logical page number p according to the determined page number P, which is specifically implemented by the following formula [2]:

p=rand(0,P)[2]

Among them, rand() is a random function used to generate random numbers.

S304, according to the randomly selected logical page number p, query whether the corresponding cache key (key) value has cached data, if so, execute S305, otherwise, execute S306;

Since the cache key value includes the service unique identification code and the logical page number p, and since the service unique identification code is preset, the corresponding cache key value can be easily deduced according to the randomly selected logical page number p, so as to query the cache key value Whether the cached data is stored correspondingly.

S305, directly reading the cache data corresponding to the cache key value in the cache for display.

S306, read the paging data in the database according to the randomly selected logical page number p and the randomly selected number n; specifically, the following steps are included:

Step 1. Determine the index position S according to the randomly selected logical page number p and the randomly selected number n, which is specifically realized by the following formula [3]:

S=(p-1)*n [3]

Step 2. Determine that the read length L is consistent with the random sampling number n, that is, L=n;

Step 3. According to the determined index position S and the read length L, the paging data is read and displayed from the latest dynamic data of the information flow in the database.

S307, storing the read paging data in the cache; wherein, the cache key value corresponding to the cached paging data includes a service unique identification code and a logical page number p, and the expiration time corresponding to the cached paging data is based on the cache time of each page tset;

The expiration time refers to the length of time that the cached data is stored in the cache (cache); the service unique identification code refers to the unique identification code of the randomly extracted data service in the information flow, which can be flexibly set in advance, as long as it can be compatible with other services in the cache. The corresponding cache key value can be distinguished. For example, if the service unique identification code is set to "***", the cache key value corresponding to the cached paging data can be "***-p";

Those of ordinary skill in the art can understand that since the business requirement of randomly extracting data from the information flow is continuously repeated, the cached data corresponding to the cached key value involved in S304 and S305 is essentially read from the database before and cached, and paginated data that has not yet reached the expiration time.

S308, displaying the paging data corresponding to the logical page number p read in the database.

Based on the same technical concept, an embodiment of the present invention provides a device for randomly extracting data in an information stream. Since the principle of the device for solving problems is consistent with the method for randomly extracting data in an information stream, the implementation of the device can be referred to in the description of the method. Implementation, the repetition will not be repeated.

As shown in FIG. 4 , an apparatus for randomly extracting data in an information stream provided by an embodiment of the present invention includes:

The setting unit 401 is configured to set random extraction parameters according to the business requirements of randomly extracting data in the information flow, and the random extraction parameters include: the total amount of the latest dynamic data and the number of random extractions;

a paging quantity determining unit 402, configured to determine the paging quantity according to the total amount of the latest dynamic data and the random extraction quantity;

The page number random selection unit 403 is used to randomly select the logical page number according to the determined page number;

The cache query unit 404 is configured to query whether the corresponding cache key value has cache data according to the randomly selected logical page number, and if so, directly read the cache data corresponding to the cache key value in the cache for display; otherwise, trigger the database to read unit 405;

The database reading unit 405 is configured to read the paging data in the database for display according to the randomly selected logical page number and the randomly selected number, and store the read paging data in the cache, wherein the cache corresponding to the cached paging data The key value includes the business unique identification code and the logical page number.

In a specific implementation, the device for randomly extracting data in the information flow is usually set in the service server on the network side.

In a specific implementation, the random extraction parameter set by the setting unit 401 further includes the cache time per page; and the expiration time corresponding to the paged data cached by the database reading unit 405 is set according to the cache time per page.

In a specific implementation, the page number determining unit 402 is specifically implemented by the following formula: P=int(N/n)+1, where P represents the number of pages, N represents the total amount of latest dynamic data, n represents the number of random extractions, and int () is the rounding function.

In a specific implementation, the page number random selection unit 403 is specifically implemented by the following formula: p=rand(0, P), where p represents a randomly selected logical page number, P represents the number of pages, and rand() is used to generate a random number random function.

In a specific implementation, a possible structure of the database reading unit 405, as shown in FIG. 5, specifically includes:

The index position determination subunit 501 is used to determine the index position according to the randomly selected logical page number and the randomly selected number, which is specifically realized by the formula S=(p-1)*n, where S represents the index position, and p represents the randomly selected logical page number , n represents the number of random draws;

a read length determination subunit 502, configured to determine that the read length is consistent with the random extraction quantity;

The reading subunit 503 is used to read the paging data in the database for presentation according to the determined index position and the reading length;

The storage subunit 504 is configured to store the read paging data in the cache, wherein the cache key value corresponding to the cached paging data includes the service unique identification code and the logical page number.

The method and device for randomly extracting data in an information stream provided by the embodiments of the present invention are implemented by paging and dividing the latest dynamic data in the information stream at a logical level according to the business requirement of randomly extracting data in the information stream. First, determine the number of paging according to the total amount of the latest dynamic data and the number of random extractions, and randomly select the logical page number according to the determined number of pages; directly read the paging data in the database for display according to the logical page number and the number of random extractions, and then read The paging data can be stored in the cache, and reading the paging data from the database does not need to traverse all the latest dynamic data, and only caches the paging data read out, thereby reducing memory and server bandwidth consumption; The output paging data is cached and stored. If the corresponding cache key value has cached data according to the logical page number query, the cached data can be directly read for display, which greatly improves the read performance. Compared with the prior art, this solution does not need to read all the latest dynamic data in the information flow in the database into the cache; it also does not need to use a circular method to retrieve the latest dynamic data from the cache after generating corresponding random numbers according to the randomly selected quantity. Each piece of data is extracted one by one, which can effectively reduce memory and database bandwidth consumption and improve data read performance.

Further, since the cache time of each page is set, the expiration time of each page data stored in the cache is different, which can improve the probability of exposure of the latest dynamic data.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

Although preferred embodiments of the present invention have been described, additional changes and modifications to these embodiments may occur to those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be construed to include the preferred embodiment and all changes and modifications that fall within the scope of the present invention.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the invention. Thus, provided that these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims (10)

1. A method for randomly extracting data from an information stream, comprising:

setting random extraction parameters according to the service requirement of random extraction data in the information flow, wherein the random extraction parameters comprise: the total amount of the latest dynamic data and the random extraction amount;

determining the paging number according to the total amount of the latest dynamic data and the random extraction number, and randomly selecting a logical page number according to the determined paging number;

inquiring whether cache data exist in the corresponding cache key value according to the randomly selected logical page number, and if so, directly reading the cache data corresponding to the cache key value in the cache for display;

otherwise, reading the paging data in the database according to the randomly selected logical page number and the randomly extracted number for displaying, and storing the read paging data into a cache, wherein a cache key value corresponding to the cached paging data comprises the service unique identification code and the logical page number.

2. The method of claim 1, wherein the randomly chosen parameters further include a per-page buffer time; and setting the corresponding expiration time of the cached paging data according to the caching time of each page.

3. The method according to claim 1 or 2, wherein the determining the number of pages according to the latest total dynamic data amount and the random number is implemented by the following formula:

P＝int(N/n)+1

wherein, P represents the number of pages, N represents the total amount of the latest dynamic data, N represents the number of random fetches, and int () is a rounding function.

4. The method according to claim 1 or 2, wherein said randomly selecting a logical page number according to the determined number of pages is implemented by the following formula:

p＝rand(0，P)

where P represents a randomly selected logical page number, P represents the number of pages, and rand () is a random function for generating random numbers.

5. The method according to claim 1 or 2, wherein reading paging data for presentation in the database according to the randomly selected logical page number and the randomly drawn number comprises:

determining an index position according to the randomly selected logical page number and the randomly extracted number, and specifically realizing the following formula:

S＝(p-1)*n

wherein S represents an index position, p represents a randomly selected logical page number, and n represents a random extraction number;

determining that the read length is consistent with the random number;

and reading paging data in the database for display according to the determined index position and the reading length.

6. An apparatus for randomly extracting data from an information stream, comprising:

a setting unit, configured to set a random extraction parameter according to a service requirement of randomly extracting data in an information flow, where the random extraction parameter includes: the total amount of the latest dynamic data and the random extraction amount;

the page quantity determining unit is used for determining the page quantity according to the total quantity of the latest dynamic data and the random extraction quantity;

the page random selection unit is used for randomly selecting a logic page according to the determined paging number;

the cache query unit is used for querying whether cache data exist in the corresponding cache key value according to the randomly selected logical page number, and if so, directly reading the cache data corresponding to the cache key value in the cache for display; otherwise, triggering a database reading unit;

and the database reading unit is used for reading the paging data in the database for display according to the randomly selected logical page numbers and the randomly extracted number, and storing the read paging data into a cache, wherein cache key values corresponding to the cached paging data comprise the service unique identification codes and the logical page numbers.

7. The apparatus according to claim 6, wherein the random extraction parameter set by the setting unit further includes a per-page buffer time; and the number of the first and second groups,

and setting the expiration time corresponding to the paging data cached by the database reading unit according to the caching time of each page.

8. The apparatus according to claim 6 or 7, wherein the page number determining unit is specifically implemented by the following formula:

P＝int(N/n)+1

wherein, P represents the number of pages, N represents the total amount of the latest dynamic data, N represents the number of random fetches, and int () is a rounding function.

9. The apparatus according to claim 6 or 7, wherein the page number random selection unit is implemented by the following formula:

p＝rand(0，P)

where P represents a randomly selected logical page number, P represents the number of pages, and rand () is a random function for generating random numbers.

10. The apparatus according to claim 6 or 7, wherein the database reading unit specifically includes:

an index position determining subunit, configured to determine an index position according to the randomly selected logical page number and the randomly extracted number, specifically by using a formula S ═ p-1 × n, where S denotes the index position, p denotes the randomly selected logical page number, and n denotes the randomly extracted number;

a read length determining subunit, configured to determine that the read length is consistent with the random number;

the reading subunit is used for reading the dynamic data in the database for display according to the determined index position and the reading length;

and the storage subunit is used for storing the read paging data into a cache, wherein a cache key value corresponding to the cached paging data comprises a service unique identification code and the logical page number.

Images