CN113126920B

CN113126920B - Object storage batch reading and writing methods, clients, storage gateways and media

Info

Publication number: CN113126920B
Application number: CN202110361851.9A
Authority: CN
Inventors: 陈坚; 匡浩; 马四; 文刘飞
Original assignee: Shenzhen Sandstone Data Technology Co ltd
Current assignee: Shenzhen Sandstone Data Technology Co ltd
Priority date: 2021-04-02
Filing date: 2021-04-02
Publication date: 2025-02-07
Anticipated expiration: 2041-04-02
Also published as: CN113126920A

Abstract

The embodiment of the invention relates to an object storage batch reading and writing method, a client and a storage gateway, wherein the reading method is applied to the client and the storage gateway, the client can read fragmented data of at least one appointed object in an appointed range from the storage gateway, batch reading is realized, multiple interactions between the client and the storage gateway are avoided, the writing method is applied to the client and the storage gateway, and the writing method can realize all or part of writing of at least one object and the fragmented data, realize writing of massive objects, and effectively reduce the interaction times between the client and the storage gateway. According to the embodiment of the invention, the batch read-write interface based on the POST request is provided, so that the request interaction times between the client and the storage gateway are reduced, the read-write efficiency is improved, and the resource consumption of the client is reduced.

Description

Method for batch reading and writing of object storage, client, storage gateway and medium

Technical Field

The embodiment of the invention relates to the field of object storage, in particular to a method for batch reading and writing of object storage, a client, a storage gateway and a medium.

Background

The Object store (Object store) was originally the S3 (Simple Storage Service) introduced by amazon 2006. Object storage is object/file-oriented, mass internet storage, which may also be referred to directly as "cloud storage". The object is a file, which is already encapsulated (the object in programming has the feature of encapsulation), that is, the file cannot be directly opened/modified in the object storage system, but the file can be uploaded, downloaded, etc. like ftp.

Taking the distributed object storage as an example, the http type application client is based on the S3 REST API standard when reading and writing. However, in the existing S3 type storage interface, whether it is an AWS product or a public cloud product, if multiple files are to be written, only one file can be written in a request, and the client needs to package the PUT write requests of the multiple files respectively, and then send the requests in a certain concurrency manner, so that the interaction consumes the client resources and the network resources very much. When a file is read, only a partial fragment of the file, i.e., partial range data, is sometimes read, it is common practice to use GET read requests to specify multiple different ranges, and it may be necessary to individually encapsulate GET read requests for different fragments and then send the requests at certain concurrence. Particularly, when the GET is read for the fragments of different files, if the number of the files is large, a large number of requests are constructed, the interaction is many times, the whole system resource is consumed, and the efficiency is low.

Disclosure of Invention

The embodiment of the invention aims to provide a batch reading and writing method for object storage, a client, a storage gateway and a storage medium, by providing batch read-write interfaces based on POST requests, the number of request interaction times between the client and the storage gateway is reduced, the read-write efficiency is improved, and the resource consumption of the client is reduced.

In a first aspect, an embodiment of the present invention provides a method for batch reading of object storage, which is applied to a client, where the method includes:

When a reading operation instruction aiming at least one specified object corresponding to target barrel information is received, packaging a reading POST request, wherein the reading PSOT request comprises a reading request head, target barrel information positioned in the reading request head or the reading request row and a to-be-read fragment range positioned in a reading request body;

The POST reading request is sent to the storage gateway, so that the storage gateway responds according to the POST reading request after receiving the POST reading request;

and reading the plurality of fragment data of the fragment range to be read of the appointed object according to the response of the storage gateway.

In some embodiments, the package read POST request includes:

determining a request method as reading in a read request row;

placing the target bucket information into a read request head;

And adding a multi-level tag into the text of the read request body to identify the specified object and the fragment data of the specified range of the specified object, wherein the number of the specified objects is 1.

In some embodiments, the package read POST request further comprises:

And packaging the POST reading request according to an API interface specification of storage definition, wherein the API interface is based on an S3 REST API protocol standard.

In some embodiments, if the number of the specified objects is greater than 1, adding a multi-level tag to the body of the request body to identify the specified object and the specified range of the specified object, including:

And adding multi-level labels into the text of the read request body, and adding different identifications to different specified objects to identify the specified objects and the fragmented data of the specified objects within the specified range.

In some embodiments, the method further comprises:

If the read POST request comprises at least two specified objects, the number of the specified objects is at most 200, and the number of the sliced data of each specified object is at most 200, and if the read POST request comprises only one specified object, the number of the sliced data of the specified object is at most 1000.

In a second aspect, an embodiment of the present invention provides a method for batch writing of object storage, which is applied to a client, where the method includes:

creating a POST writing request, wherein the POST writing request comprises a writing request head, target bucket information positioned at the request head or a request row, at least two objects to be processed positioned at a request body and fragment data of the objects to be processed;

The POST writing request is sent to a storage gateway, so that the storage gateway analyzes the writing request head after receiving the POST writing request, so as to respond to the request of the target barrel information, the object to be processed and the fragment data, obtain a response result, write the target barrel information, the object to be processed and the fragment data according to the response result, and return the response result to the client;

After the response result is received, if the response result contains the objects to be processed passing through the request, analyzing the processing result of each object to be processed passing through the request in the response result.

In some embodiments, the creating a write POST request, the write POST request including a write request header, target bucket information located at a request header or a request line, at least two pending objects located at a request body, and fragmented data of the pending objects, includes:

determining a request method as writing in a writing request row;

placing the target bucket information into a write request header or a write request row;

setting an object dividing line in the write request header;

starting with an object dividing line in a text of a write request body;

Packaging a first object to be processed, wherein the object to be processed comprises an object name, fragment data of the object to be processed and additional attributes;

continuing to encapsulate a second object to be processed, and separating the first object to be processed from the second object to be processed through the object dividing line until encapsulation of all the objects to be processed is completed;

and adding an ending parting line to finish the encapsulation of the writing POST request.

In some embodiments, the creating the write POST request further comprises:

And encapsulating the POST writing request according to an API interface specification of storage definition, wherein the API interface is based on an S3 REST API protocol standard.

In some embodiments, the method further comprises:

if the response result contains the object to be processed, which is not passed by the request, receiving an error code of the object to be processed, which is not passed by the request and is sent by the storage gateway;

And judging the failed reason according to the error code, and correcting the failed object to be processed according to the failed reason so as to create a POST writing request again according to the corrected object to be processed.

In a third aspect, an embodiment of the present invention further provides an object storage batch reading method, applied to a storage gateway, where the method includes:

After receiving a read POST request sent by the client, responding according to the read POST request, wherein the read POST request comprises a read request head, target bucket information positioned at the read request head or a request row, at least one designated object positioned at a read request body and fragment data of the designated object, and the designated object is a to-be-processed object passing through the request.

In some embodiments, the responding according to the read POST request includes:

Carrying out validity check on the POST reading request;

When the read POST request passes the validity check, analyzing the text of the read request body to obtain the specified object and the specified range of the specified object;

Searching a position corresponding to the specified range in the stored fragment data;

integrating and/or sequencing the storage sequence of the stored fragment data according to the positions to obtain a reading sequence;

reading the fragment data of the specified range according to the reading sequence;

and packaging the fragmented data in the appointed range in a response result, and sending the response result to the client.

In some embodiments, if the number of the specified objects is 1, the encapsulating the specified range of the fragmented data in the response result includes:

Placing the response result in a response request row;

Setting a slicing dividing line in the response request head;

starting with a slicing parting line in the text of the response request body;

Encapsulating first sliced data of the specified range of the specified object, wherein the first sliced data comprises a sliced range and data;

packaging second sliced data in a designated range until all sliced data in the designated range are packaged, and separating adjacent sliced data by a sliced dividing line;

And adding a slicing dividing line to finish the encapsulation of the slicing data of the specified range.

In some embodiments, if the number of the specified objects is greater than 1, then encapsulating the specified range of fragmented data in a response result includes:

Placing the response result in a response request row;

Setting a slicing dividing line in the response request head;

starting with a slicing parting line in the text of the response request body;

Encapsulating the sliced data of the specified range of the first specified object, and dividing the sliced data of the specified range of the first specified object by a slicing dividing line;

packaging the fragmented data of the specified range of the second specified object until the fragmented data of the specified range of all the specified objects are packaged, wherein the fragmented data of the specified range of the second specified object are partitioned by a fragmented partition line;

object dividing lines are added to end the encapsulation of the specified range of the fragmented data of all the specified objects.

In a fourth aspect, an embodiment of the present invention further provides an object storage batch writing method, applied to a storage gateway, where the method includes:

The method comprises the steps of receiving a POST writing request sent by a client, wherein the POST writing request comprises a writing request head, target bucket information positioned at a request head or a request row, at least two objects to be processed positioned at a request body and fragment data of the objects to be processed;

Analyzing the writing request head to request and respond to the target barrel information, the object to be processed and the fragment data, obtaining a response result, writing the target barrel information, the object to be processed and the fragment data according to the response result, and returning the response result to the client.

In some embodiments, the parsing the write request header to respond to the target bucket information, the object to be processed, and the tile data, to obtain a response result, writing the target bucket information, the object to be processed, and the tile data according to the response result, and returning the response result to the client includes:

analyzing the writing request head, and performing validity check on the writing POST request to obtain a response result corresponding to the object to be processed;

Placing the response result in a response request row;

setting an object dividing line, and placing the object dividing line in the response request head;

starting with a response split line in the body of the response request body;

Packaging a response result of the first object, wherein the response result comprises an object name, a response code and an additional attribute;

Packaging the response result of the second object, and separating the response result of the first object from the response result of the second object through a response dividing line until the packaging of the response results of all objects is completed;

and adding a finishing dividing line to complete the encapsulation of the response result.

In some embodiments, the performing validity check on the write POST request to obtain a response result corresponding to the object to be processed includes:

Verifying the validity of the request interface of the write-in POST request, wherein when the validity of the request interface of the write-in POST request fails, a first response result that the validity of the request interface of the write-in POST request fails is obtained;

When the validity check of the request interface of the write-in POST request passes, a second response result that the validity check of the request interface of the write-in POST request passes is obtained, and the target bucket information and the authority of the object to be processed are checked;

When the authority of the target bucket information and the appointed object passes the verification, a third response result passing the verification is obtained, and the target bucket information, the object to be processed and the fragment data are stored;

And when the target barrel information and the authority of the object to be processed are not checked to pass, a fourth response result that the verification is not passed is obtained.

In a fifth aspect, an embodiment of the present invention provides a client, where the client includes:

at least one first processor, and

A first memory communicatively coupled to the first processor, the first memory storing instructions executable by the at least one first processor, the instructions executable by the at least one first processor to enable the at least one processor to perform the method of any one of the above.

In a sixth aspect, an embodiment of the present invention provides a storage gateway, including:

At least one second processor, and

A second memory communicatively coupled to the second processor, the second memory storing instructions executable by the at least one second processor, the instructions executable by the at least one second processor to enable the at least one processor to perform the method of any one of the above.

In a seventh aspect, embodiments of the present invention provide a non-volatile computer-readable storage medium storing computer-executable instructions that, when executed by a client or storage gateway, cause the client to perform a method as described above or the storage gateway to perform a method as described above.

According to the object storage batch reading and writing method, the client, the storage gateway and the storage medium, at least one object can be written in during writing, so that when the client reads the sliced data, similarly, one appointed object or a plurality of appointed objects can be read from the storage gateway, a POST reading request is sent to the storage gateway, so that the storage gateway responds according to the POST reading request, and the sliced data of the appointed objects are read according to the response of the storage gateway, and batch reading of the sliced data is achieved.

And when the client writes batch objects into the storage gateway, creating a write POST request, wherein the write POST request comprises a write request head, target bucket information positioned at a request head or a request row, at least two objects to be processed positioned at a request body and sliced data of the objects to be processed, one target bucket information corresponds to at least two objects to be processed and the sliced data thereof, namely, one write POST request comprises a plurality of objects to be processed and the sliced data thereof, sending the write POST request to the storage gateway, analyzing the write request head by the storage gateway to request and respond to the target bucket information, the objects to be processed and the sliced data, obtaining a response result, writing the target bucket information, the objects to be processed and the sliced data according to the response result, and returning the response result to the client, and writing the objects to be processed and the sliced data according to the response result, thereby realizing the complete writing or partial writing of the objects to be processed and the sliced data. When the method and the device are used for writing in the mass objects and reading the fragment data of the mass objects, the interaction times can be effectively reduced, the access efficiency is improved, and the resource consumption of the client is reduced.

Drawings

One or more embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like references indicate similar elements, and in which the figures of the drawings are not to be taken in a limiting sense, unless otherwise indicated.

FIG. 1 is a flow diagram of one embodiment of an object store batch write method of the present invention;

FIG. 2 is an interactive schematic diagram of one embodiment of an object store batch write and read method of the present invention;

FIG. 3 is an interactive schematic diagram of one embodiment of an object store batch write method of the present invention;

FIG. 4 is a schematic flow chart diagram illustrating one embodiment of an object store batch read method of the present invention;

FIG. 5 is a schematic diagram of interactions for reading an object in one embodiment of an object store batch read method of the present invention;

FIG. 6 is a flow diagram of one embodiment of an object store batch read method of the present invention for reading multiple objects;

FIG. 7 is a flow diagram of one embodiment of an object store batch write method of the present invention;

FIG. 8 is a flow chart of yet another embodiment of an object store batch write method of the present invention;

FIG. 9 is a schematic diagram illustrating the structure of one embodiment of a first object store batch writing apparatus of the present invention;

FIG. 10 is a schematic diagram illustrating the construction of one embodiment of a first object store batch read device of the present invention;

FIG. 11 is a schematic diagram illustrating the construction of one embodiment of a second object store batch writing apparatus of the present invention;

FIG. 12 is a schematic diagram illustrating the construction of one embodiment of a second object store batch read device of the present invention;

FIG. 13 is a schematic diagram of the hardware architecture of the first controller in one embodiment of the client of the present invention;

fig. 14 is a schematic diagram of a hardware structure of the second controller in an embodiment of the storage gateway of the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The object storage batch reading/writing method provided by the embodiment of the invention can be applied to a client. The client may be an intelligent terminal device, such as a smart phone, a computer, an Ipad, and other terminal devices.

The method for batch reading/writing of object storage provided by the embodiment of the invention can be applied to a storage gateway, and the storage gateway can be a server and other devices.

It can be understood that the client is provided with a first controller as a main control center of the client, and the storage gateway is provided with a second controller as a main control center of the storage gateway. By providing batch read-write interfaces based on POST requests, the number of request interaction times between the client and the storage gateway is reduced, the read-write efficiency is improved, and the resource consumption of the client is reduced.

Referring to fig. 1, fig. 1 is a flowchart of an object storage batch writing method according to an embodiment of the present invention, where the method may be performed by a first controller 13 in a client, and as shown in fig. 1 and fig. 2, the method is applied to the client, and the method includes:

And 101, creating a writing POST request, wherein the writing POST request comprises a writing request head, target bucket information positioned at the request head or the request row, at least two objects to be processed positioned at a request body and fragment data of the objects to be processed.

Specifically, writing a POST request refers to an HTTP request message with a POST request method, where the request message includes a request row, a request header, and a request body, the request row includes a request method, a URL, and a protocol, the request header includes a header field name, a value, and target bucket information may be placed in the request row or the request header, and the request body includes request data, that is, at least two objects to be processed and fragment data of the objects to be processed.

Target bucket information, which can be understood as a bucket (storage space), is all objects within the bucket, each object having a plurality of sliced data. Further, since an object, i.e., one file in the client service, the object storage generally takes the stored file as an object, in this case, the object, i.e., the file.

It can be understood that the target bucket information in the present case refers to a bucket, at least two objects to be processed exist in a bucket, each object has a unique file name, each object may also have separate metadata (additional attribute) and fragment data, and when a write POST request is constructed, 200 or more objects to be processed may be written into the storage gateway at the same time.

In some embodiments, the creating a write POST request may include:

Specifically, S3 is Amazon S3, which is collectively referred to as Amazon Simple Storage Service, REST, and is a Web service architecture. Briefly, a resource is identified and located by a URI, and the operation of the resource is defined by HTTP methods, including GET, PUT, POST, DELETE and HEAD, that constitute actions in REST, which generally match CRUD actions as follows:

Create:POST;

Read:GET;

Update:PUT/PATCH;

Delete:DELETE。

However, the matching is not strictly limited, and sometimes the PUT can be used to create new resources, the POST is used to update resources, and the non-idempotent nature of the POST request makes it a very flexible method.

The most critical in a write POST request is how to transfer multiple file data together. In order to improve efficiency, a write POST request is limited to write files of the same barrel in batches, so that only one barrel name is needed, the barrel name is directly placed in a request head or a request row, then the file name of an object to be processed, fragment data and the like are placed in a request text, and a dividing line is defined in the request head for dividing different file data. The request text starts with a division line and ends with a division line, and different files are divided according to the division line, and different objects are identified according to the division line when the processing data is stored (the same applies to the response to divide different file response information by the division line). After the request is generated, the client sends the request to the storage gateway.

In some embodiments, the creating the write POST request, where the write POST request includes a write request header, target bucket information located in the request header or the request line, at least two objects to be processed located in the request body, and the tile data of the objects to be processed may include:

determining a request method as writing in a writing request row;

setting an object dividing line in the write request header;

starting with an object dividing line in a text of a write request body;

Specifically, as shown in fig. 3, the procedure for constructing a write POST request by the client is as follows:

1. filling put representation write in URL of request line;

2. Placing the target bucket (bucketname) in the request header;

3. setting an object dividing line (boundary) in the request header;

4. adding blank rows;

5. The request body starts with a "- -split line";

6. Encapsulating a first object, namely an object name (Key 1), an Additional attribute (Additional) and file Content (Content);

7. Add "- -split line" indicates that there is a next object;

8. Encapsulating a second object, namely an object name (Key 2), an Additional attribute (Additional) and file Content (Content);

9. Sequentially packaging other objects;

10. The addition "- -split line- -" indicates the end, with no other objects.

The method comprises the steps of packaging a plurality of objects to be processed into a POST writing request, sending the objects to a gateway for storage, and responding after the whole POST writing request is processed, so that the aim of writing a plurality of objects in batches is fulfilled.

102, Sending the POST writing request to a storage gateway so that the storage gateway analyzes the writing request head to respond to the request of the target barrel information, the object to be processed and the fragment data, obtaining a response result, writing the target barrel information, the object to be processed and the fragment data according to the response result, and returning the response result to the client.

After creating the POST writing request, the client sends the POST writing request to the storage gateway. After receiving the POST writing request, the storage gateway analyzes the writing request head to respond to the request of the target barrel information, the object to be processed and the fragment data, obtains a response result, writes the target barrel information, the object to be processed and the fragment data according to the response result, and returns the response result to the client.

In some embodiments, the parsing the write request header to respond to the target bucket information, the object to be processed, and the fragment data, to obtain a response result, writing the target bucket information, the object to be processed, and the fragment data according to the response result, and returning the response result to the client includes:

Placing the response result in a response request row;

starting with a response split line in the body of the response request body;

Specifically, as shown in fig. 3, the storage gateway responds to the write POST request of the client by the following encapsulation process:

1. placing the request response result in a request row;

2. setting an object dividing line (boundary) in the request header;

3. Adding blank rows;

4. the response text starts with a "- -split line";

5. Encapsulating a first object write response, namely an object name (Key), an error code (Result) and an Additional attribute (Additional);

6. adding "- -split line" indicates that there is a next response;

7. Encapsulating a second object write response, namely an object name (Key), an error code (Result) and an Additional attribute (Additional);

8. sequentially packaging other objects;

9. add "- -split line- -" indicates end, no other responses.

In some embodiments, the storage gateway performs validity verification on the write POST request to obtain a response result corresponding to the object to be processed, and stores the target bucket information, at least two objects to be processed, and fragment data, which may include:

when the target bucket information and the authority of the object to be processed pass through verification, a third response result of passing through verification is obtained, and the target bucket information, the object to be processed and the fragment data are stored;

After receiving a POST writing request, a storage gateway firstly performs validity check on a request interface of the POST writing request, stores and supports batch POST interfaces (unsupported storage can be regarded as illegal requests), namely when the validity check on the request interface of the POST writing request fails, a first response result that the validity check on the request interface of the POST writing request fails is obtained, then when the validity check on the request interface of the POST writing request fails, a second response result that the validity check on the request interface of the POST writing request passes is obtained, rights of target barrel information and objects to be processed are checked, such as existence, correctness and other rights of a target barrel information check barrel are extracted, and otherwise, request errors are directly returned. And finally, carrying out batch storage business of the files, namely obtaining a third response result passing verification when the target bucket information and the authority of the object to be processed pass verification, storing the target bucket information, the object to be processed and the fragment data, specifically extracting different file names and corresponding metadata and data, and writing the files into a bottom storage medium according to a general storage method in storage. For example, small files can be combined into large file writing, so that the performance of mass files is ensured. Regardless of the storage mode, after each object to be processed is processed, a single corresponding result is obtained, and the result is reflected in the returned error code executed by each object.

It can be understood that when the target bucket information and the authority of the object to be processed do not pass, a fourth response result which does not pass the verification is obtained, that is, whether the verification is passed or not, a corresponding response result is returned, so that the client side obtains the response result.

By responding to each object to be processed independently, the overall request failure caused by a small number of file writing failures is avoided, and the overall failure means that the client needs to write again, which is not friendly to the client and is not beneficial to performance experience. After the storage gateway finishes processing each to-be-processed file written in the POST request, the storage gateway stores all response results and performs response encapsulation.

And 103, after the response result is received, if the request passing object to be processed exists in the response result, analyzing the processing result of each request passing object to be processed in the response result.

After receiving the response result returned by the storage gateway, the client checks the response result to judge whether the request is wholly successful or partially successful or wholly failed, and when the request passing object to be processed exists in the response result, the processing result of the request passing is analyzed.

In some of these embodiments, the method further comprises:

Specifically, when a part of verification fails, the response result can have an object to be processed, which is required to fail, and the response result can return a corresponding response result no matter whether the verification fails or not, and the request fails and generates an error code, the client can judge the failed reason according to the error code, so that the failed object to be processed can be corrected according to the failed reason, for example, the request is illegal, or the authority is insufficient, if the number of objects exceeding the single limit is exceeded, the client can reduce the number of objects, so that the verification can pass, furthermore, the client can rewrite a POST request which can be encapsulated, avoid the rewriting of the object which is already successful, cover the original object, encapsulate the object with other objects to be written, initiate the POST request, and for the storage gateway, the write is stateless, and the successfully written file is not affected.

The object storage batch processing method, the client and the storage gateway of the embodiment of the invention create a write POST request when the client writes batch objects into the storage gateway, wherein the write POST request comprises a write request head, target bucket information positioned at the request head or the request row, at least two objects to be processed positioned in a request body and fragment data of the objects to be processed, one target bucket information corresponds to at least two objects to be processed and fragment data thereof, namely one write POST request comprises a plurality of objects to be processed and fragment data thereof, the write POST request is sent to the storage gateway, the storage gateway analyzes the write request head to request response to the target bucket information, the objects to be processed and the fragment data, a response result is obtained, the target bucket information, the objects to be processed and the fragment data are written according to the response result, the response result is returned to the client, and all or part of the objects to be processed and the fragment data are written according to the response result, thereby realizing writing of all or part of the objects to be processed and the fragment data.

Referring to fig. 4, after writing the object to be processed and the fragment data, the client may perform a read operation on the object and the fragment data stored in the storage gateway, so that an embodiment of the present invention further provides an object storage batch reading method, which is applied to the client, and the method further includes:

104, packaging a POST reading request when a reading operation instruction aiming at least one specified object corresponding to the target bucket information is received, wherein the reading PSOT request comprises a reading request head, target bucket information positioned in the reading request head or the reading request row and a to-be-read fragment range positioned in a reading request body;

105, sending the POST reading request to the storage gateway so that the storage gateway responds according to the POST reading request after receiving the POST reading request, wherein the designated object is an object to be processed passing through the request in a processing result;

And 106, reading the plurality of pieces of fragment data of the specified object according to the response of the storage gateway.

The method comprises the steps that when a client receives a reading operation instruction of at least one designated object corresponding to target bucket information, a reading POST request is packaged, then the reading POST request is sent to a storage gateway, the storage gateway responds according to the reading POST request after receiving the reading POST request, the designated object is an object to be processed, the object is requested to pass through in a processing result, and finally, a plurality of pieces of data of the designated object are read according to the response of the storage gateway.

When the client needs to read the object stored in the storage gateway, the client can request to read a plurality of pieces of data of a single object in a single barrel or request to read a plurality of pieces of data of a plurality of objects in a single barrel, the object to be read is defined as a designated object, and the designated object is a corresponding object to be processed in a processing result. And the information of the plurality of fragment data is packaged in the read POST request, and the read POST request corresponding to at least one designated object is sent to the storage gateway.

When a request is made to read a plurality of pieces of sliced data of a single object in a single bucket, for example, the size of an object is 100 bytes, and the current read operation instruction corresponds to the sliced data in the range of 0-5 and 10-25, then the read POST request encapsulates the target bucket information, the name of the specified object and the range of 0-5 and 10-25 to be read of the sliced data, that is, the read POST request does not contain actual data, the content type of the request is fixed, in order to simplify the generation of the request and improve the efficiency of the generation of the request, the file name of the specified object and the sliced range to be read can be represented by using a request message body in xml format, and the maximum number of the sliced can reach 1000.

In some of these embodiments, the client package read POST request may include:

determining a request method as reading in a read request row;

placing the target bucket information into a read request head;

Specifically, as shown in fig. 5, the procedure for the client package to read the POST request is as follows:

1. filling get in URL of request line to represent read;

2. Placing the target bucket (bucketname) in the request header;

3. Adding blank rows;

4. the request text is in xml format;

5. first-level tag < Get >;

6. secondary label < Object >;

7. Three-level label < Key >, intermediate package object name (Key 1);

8. three level tag < VersionId >, intermediate package version number (VersionId);

9. three-level label < Range >, range of multiple slices (bytes=), comma separated, e.g., 0-9,100-200,500-600.

After the POST reading request is packaged, the POST reading request is sent to the storage gateway, and the storage gateway responds to the POST reading request after receiving the POST reading request.

Correspondingly, when the POST request is packaged, the POST request is packaged according to an API interface specification defined by storage, wherein the API interface is based on an S3 REST API protocol standard.

In some embodiments, after receiving the POST reading request, the storage gateway responds according to the POST reading request, which may include:

Carrying out validity check on the POST reading request;

Specifically, the verification of the validity may be to first use a unified request parameter verification process to ensure that the request is legal, and then analyze the specific file request in the message body. When acquiring a plurality of pieces of data of a single designated object, in order to improve efficiency and reduce redundancy of data response, searching the stored pieces of data for the position of at least one designated object in the read POST request, then reading the pieces of data corresponding to the designated object, and when reading, integrating and/or sorting the storage sequence of the stored pieces of data according to the position to obtain a reading sequence, for example, the range to be read is the pieces 0-5 and 4-10, and the pieces of data can be combined into one piece 0-10, so that the pieces of data of 4-5 do not need to be repeatedly read or packaged, and for example, the range to be read is 20-50 and 10-15, if the range to be read 10-15 is read first, then the range to be read 20-50 is more friendly to the storage bottom layer of the storage gateway, and when reading, the range to be 10-50 can be completely avoided to be read, namely, the reading sequence can be adjusted, the original reading sequence is optimized, and after the optimized reading sequence is obtained, the pieces of data can be read according to the reading sequence, so that the reading efficiency is higher.

In some embodiments, when the number of specified objects is 1, the storing gateway encapsulates the fragmented data of the specified range in a response result, which may include:

Placing the response result in a response request row;

Setting a slicing dividing line in the response request head;

starting with a slicing parting line in the text of the response request body;

Specifically, the storage gateway needs to respond to the client when reading the fragment data, and when the number of the specified objects is 1, as shown in fig. 5, the corresponding process of responding to the client by the storage gateway is as follows:

1. placing the request response result in a request row;

2. setting a slicing dividing line (boundary) to be placed in a request head;

3. Adding blank rows;

4. the response text starts with a "- -split line";

5. encapsulating the first piece of data, namely Range and data (Content);

6. adding "- -a split line" indicates that there is next piece of data;

7. encapsulating the second piece of data, namely Range and Content;

8. sequentially packaging other fragments;

9. the addition "- -split line- -" indicates the end, with no other sliced data.

And finally, after receiving the response, the client analyzes the segmented data according to the sequence, and if the combined segmented data exist, the client needs to determine which segmented data are contained in the requested data according to the segmented range, and extracts the data to the upper business layer according to the own business requirement. The reading of the fragmented data may also fail, and retry requests or associated operation and maintenance positioning problems may be performed according to the error code.

When the multi-slice data of a plurality of objects of a single barrel are read, the different slice data of the objects are contained in a reading POST request and are sent to a storage gateway together, and the storage gateway responds to the request after processing the whole reading POST request, so that the aim of batch reading the multi-slice data of the objects is fulfilled.

In some embodiments, the number of the specified objects is greater than 1, that is, when the number of the specified objects to be read by the client is multiple, the client adds a multi-level tag in the body of the request body to identify the specified objects and the fragmented data of the specified range of the specified objects, including:

Referring to fig. 6, in the same bucket, the reading of multi-object multi-slice data is an extension of the reading of multi-slice data by a single object, so the client needs to confirm whether the requested POST file belongs to the same bucket, otherwise, the request fails.

Specifically, different file identifiers are added to different specified objects relative to a single specified object, different specified objects can be distinguished by adopting a newly added xml secondary label, and then a plurality of pieces of data of each specified object are contained in respective tertiary labels < Range >. And the storage interface can support the acquisition of 200 specified objects and 200 pieces of data of each specified object at most, and the storage interface sends the request to the storage gateway after the POST reading request is packaged.

As shown in fig. 6, the client constructs a read POST request procedure for a plurality of fragmented data for a plurality of specified objects within a bucket as follows:

1. filling get in URL of request line to represent read;

2. Placing the target bucket (bucketname) in the request header;

3. Adding blank rows;

4. the request text is in xml format;

5. first-level tag < Get >;

6. secondary label < Object >;

7. Three-level label < Key >, intermediate package object name (Key 1);

9. Three-level label < Range >, a Range (bytes=) encapsulating multiple slices, commas separated, e.g., 0-9,100-200,500-600;

10. Sequentially encapsulating other objects key2.

Then, after receiving the POST reading request, the storage gateway needs to analyze different specified objects, and then analyzes a plurality of pieces of data of each specified object. Similarly, if there is duplication of files or overlap of fragments, the stores are merged to increase read speed and reduce response packet size. For example, file a reads 0-5 and 20-25, file a reads 2-10, 40-50 would be combined into file a reads 0-10, 20-25, 40-50. The reading of each file is separated, the storage position of the corresponding file data is searched firstly, then the data of different files are read concurrently, and the files are packaged respectively after the completion of the reading. Because of the plurality of objects, two split lines, an object split line and a slice split line, need to be implemented in the request body. Object partition lines are defined in the request header, and the request body uses the object partition lines to identify the section of each object. Each object section also contains a plurality of slices, so that the slice dividing lines are defined in each object section, the slice dividing lines of each object are different, and each individual slice is distinguished by the individual dividing line of the object to which each individual slice belongs. And after the packaging is finished, replying the response to the client.

When the number of the specified objects is greater than 1, the storing gateway encapsulates the fragmented data of the specified range in a response result, which may include:

Placing the response result in a response request row;

Setting a slicing dividing line in the response request head;

starting with a slicing parting line in the text of the response request body;

Specifically, referring to fig. 6, the corresponding procedure of storing the response client is as follows:

1. placing the request response result in a request row;

2. setting an object dividing line (boundary 1) in the request header;

3. Adding blank rows;

4. The response text starts with "- -object split line";

5. Encapsulating first object data, namely an object name (Key 1), an error code (Result) and a fragmentation dividing line (boundary 2);

6. data starts with a "- -shard split line";

7. Encapsulating the first piece of data, namely Range and data (Content);

8. adding "- -a split line" indicates that there is next piece of data;

9. Encapsulating the second piece of data, namely Range and Content;

10. sequentially packaging other fragments;

11. Add "- -shard split line- -" indicates end, this object has no other shard data.

12. Sequentially packaging other objects;

13. the addition "- -object split line- -" indicates the end, with no other objects.

And finally, after receiving the response, if the response is successful, the client sequentially analyzes the object names in the message body, and analyzes the data of different fragments under each object name. Because the file and the fragment may be returned after being merged, the file and the fragment cannot be in one-to-one correspondence, so that the required data is acquired according to the file name and the fragment data range. For failed files or fragments, the error code may be identified and retried, etc.

According to the object storage batch processing method, at least one object can be written in during writing, so that when a client reads the sliced data, similarly, one appointed object or a plurality of appointed objects can be read from a storage gateway, a POST reading request is sent to the storage gateway, the storage gateway responds according to the POST reading request, and the sliced data of the appointed objects are read according to the response of the storage gateway, so that batch reading of the sliced data is realized. When the method and the device are used for writing in the mass objects and reading the fragment data of the mass objects, the interaction times can be effectively reduced, the access efficiency is improved, and the resource consumption of the client is reduced.

Referring to fig. 7, fig. 7 is a flowchart of an object storage batch writing method according to an embodiment of the present invention, where the method may be performed by the second controller 23 in the storage gateway, and as shown in fig. 7, the method is applied to the storage gateway, and the method includes:

And 201, receiving a POST writing request sent by a client, wherein the POST writing request comprises a writing request head, target bucket information positioned at the request head or a request row, at least two objects to be processed positioned at a request body and fragment data of the objects to be processed.

And 202, analyzing the writing request head to respond to the request of the target barrel information, the object to be processed and the fragment data, obtaining a response result, writing the target barrel information, the object to be processed and the fragment data according to the response result, and returning the response result to the client.

The object storage batch writing method, the client and the storage gateway of the embodiment of the invention create a writing POST request when the client writes batch objects into the storage gateway, the writing POST request comprises a writing request head, target barrel information positioned at a request head or a request row, at least two objects to be processed positioned in a request body and fragment data of the objects to be processed, one target barrel information corresponds to at least two objects to be processed and fragment data thereof, namely one writing POST request comprises a plurality of objects to be processed and fragment data thereof, the writing POST request is sent to the storage gateway, the storage gateway analyzes the writing request head to request for response to the target barrel information, the objects to be processed and the fragment data, a response result is obtained, the target barrel information, the objects to be processed and the fragment data are written according to the response result, the response result is returned to the client, and all writing of the objects to be processed and the fragment data is realized according to the response result.

Placing the response result in a response request row;

starting with a response split line in the body of the response request body;

The invention also provides a batch reading method for object storage, when the client reads data from the storage gateway, the method is applied to the storage gateway, and the method can comprise the following steps:

And 200, responding according to the read POST request after receiving the read POST request sent by the client, wherein the read POST request comprises a read request head, target bucket information positioned at the read request head or a request row, at least one designated object positioned at a read request body and fragment data of the designated object, and the designated object is a to-be-processed object through which the request passes.

In some embodiments, as shown in fig. 8, the responding according to the read POST request includes:

203, carrying out validity check on the POST reading request;

204, when the read POST request passes the validity check, analyzing the text of the read request body to obtain the specified object and the specified range of the specified object;

205, searching the position corresponding to the specified range in the stored fragment data;

206, integrating and/or sorting the storage sequence of the stored fragment data according to the positions to obtain a reading sequence;

207, reading the fragment data of the specified range according to the reading sequence;

And 208, packaging the fragmented data with the designated range in a response result, and sending the response result to the client.

In some embodiments, if the number of the specified objects is 1, the encapsulating the fragmented data of the specified range in the response result includes:

Placing the response result in a response request row;

Setting a slicing dividing line in the response request head;

starting with a slicing parting line in the text of the response request body;

In some embodiments, if the number of the specified objects is greater than 1, the encapsulating the specified range of the fragmented data in the response result includes:

Placing the response result in a response request row;

Setting a slicing dividing line in the response request head;

starting with a slicing parting line in the text of the response request body;

Correspondingly, as shown in fig. 9, the embodiment of the present invention further provides a first object storage batch writing device, which may be used for a client, where the first object storage batch reading and writing device 700 includes:

a write request creation module 701, configured to create a write POST request, where the write POST request includes a write request header, target bucket information located in the request header or the request line, at least two objects to be processed located in the request body, and fragment data of the objects to be processed;

The sending module 702 is configured to send the write POST request to a storage gateway, so that the storage gateway, after receiving the write POST request, parses the write request header to perform a request response on the target bucket information, the object to be processed, and the fragment data, obtain a response result, write the target bucket information, the object to be processed, and the fragment data according to the response result, and return the response result to the client;

And the response receiving module 703 is configured to, after receiving the response result, parse a processing result of each object to be processed passing through the request in the response result if the object to be processed passing through the request exists in the response result.

In the embodiment of the invention, when a client writes batch objects into a storage gateway, a write POST request is created, and the write POST request comprises a write request head, target bucket information positioned at a request head or a request row, at least two objects to be processed positioned at a request body and fragment data of the objects to be processed, so that one target bucket information corresponds to at least two objects to be processed and fragment data thereof, namely, one write POST request comprises a plurality of objects to be processed and fragment data thereof, the write POST request is sent to the storage gateway, the storage gateway analyzes the write request head to request and respond to the target bucket information, the objects to be processed and the fragment data, a response result is obtained, the target bucket information, the objects to be processed and the fragment data are written according to the response result, the response result is returned to the client, and the objects to be processed and the fragment data are written according to the response result, so that all or part of writing of the objects to be processed and the fragment data is realized.

In other embodiments, the write request creation module 701 is further configured to:

determining a request method as writing in a writing request row;

setting an object dividing line in the write request header;

starting with an object dividing line in a text of a write request body;

And encapsulating the POST writing request according to an API interface specification of storage definition, wherein the AP I interface is based on an S3 REST API protocol standard.

In other embodiments, the first object storage batch read/write device 700 further includes a correction module 704 for:

As shown in fig. 10, the embodiment of the present invention further provides a first object storage batch reading device, which may be used for a client, where the first object storage batch reading device 600 includes:

A package read request module 601, configured to package a read POST request when a read operation instruction for at least one specified object corresponding to the target bucket information is received;

the read request sending module 602 is configured to send the read POST request to the storage gateway, so that the storage gateway responds according to the read POST request after receiving the read POST request, where the specified object is an object to be processed passing through a request in a processing result;

And a reading module 603, configured to read the plurality of fragment data of the specified object according to the response of the storage gateway.

In other embodiments, the package read request module 601 is further configured to:

determining a request method as reading in a read request row;

placing the target bucket information into a read request head;

In other embodiments, if the number of specified objects is greater than 1, the package read request module 601 is further configured to:

In other embodiments, in the first object storage batch reading device 600, if the read POST request includes at least two specified objects, the number of the specified objects is at most 200, and the slice data of each of the specified objects is at most 200, and if the read POST request includes only one specified object, the number of the slice data of the specified objects is at most 1000.

Accordingly, as shown in fig. 11, the embodiment of the present invention further provides a second object storage batch writing device, which may be used for a storage gateway, where the second object storage batch reading and writing device 800 includes:

the write-in request receiving module 801 is configured to receive a write-in POST request sent by a client, where the write-in POST request includes a write-in request header, target bucket information located in a request header or a request line, at least two objects to be processed located in a request body, and fragment data of the objects to be processed;

And a write request response module 802, configured to parse the write request header, so as to respond to the target bucket information, the object to be processed, and the fragment data, obtain a response result, write the target bucket information, the object to be processed, and the fragment data according to the response result, and return the response result to the client.

In other embodiments, the write request response module 802 is further configured to:

Placing the response result in a response request row;

starting with a response split line in the body of the response request body;

Correspondingly, as shown in fig. 12, the embodiment of the present invention further provides a second object storage batch reading device, which may be used in a storage gateway, where the second object storage batch reading device 900 includes:

a read request response module 901, configured to:

The read request response module 901 is further configured to:

Carrying out validity check on the POST reading request;

In other embodiments, the number of the specified objects is 1, and the read request response module 901 is further configured to:

Placing the response result in a response request row;

Setting a slicing dividing line in the response request head;

starting with a slicing parting line in the text of the response request body;

In other embodiments, the number of specified objects is greater than 1, and the read request response module 901 is further configured to:

Placing the response result in a response request row;

Setting a slicing dividing line in the response request head;

starting with a slicing parting line in the text of the response request body;

It should be noted that, the device can execute the method provided by the embodiment of the application, and has the corresponding functional modules and beneficial effects of executing the method. Technical details which are not described in detail in the device embodiments may be found in the methods provided by the embodiments of the present application.

Fig. 13 is a schematic hardware structure of the first controller in one embodiment of the client, and as shown in fig. 13, the first controller 13 includes:

one or more first processors 131, a first memory 132. In fig. 13, a first processor 131 and a first memory 132 are taken as an example.

The first processor 131 and the first memory 132 may be connected by a bus or other means, and in fig. 13, connection by a bus is exemplified.

The first memory 132 is used as a non-volatile computer readable storage medium, and may be used to store a non-volatile software program, a non-volatile computer executable program, and a module, such as program instructions/modules corresponding to the object storage batch read/write method in the embodiment of the present application (for example, a write request creation module 701, a transmission module 702, a response reception module 703, a correction module 704 shown in fig. 9, and a package read request module 601, a read request transmission module 602, and a read module 603 shown in fig. 10). The first processor 131 executes various functional applications of the controller and data processing, that is, implements the object storage batch read/write method of the above-described method embodiment, by running nonvolatile software programs, instructions, and modules stored in the first memory 132.

The first memory 132 may include a storage program area that may store an operating system, an application program required for at least one function, and a storage data area that may store data created according to the use of the first object storage batch read-write device, and the like. Further, the first memory 132 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the first memory 132 optionally includes memory remotely located relative to the first processor 131, which may be connected to the client via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The one or more modules are stored in the first memory 132 and when executed by the one or more first processors 131 perform the object store batch processing method of any of the method embodiments described above, e.g., performing method steps 101-103 in fig. 1, and method steps 104-106 in fig. 4 described above, and implementing the functions of modules 701-704 in fig. 9, and modules 601-603 in fig. 10.

The product can execute the method provided by the embodiment of the application, and has the corresponding functional modules and beneficial effects of the execution method. Technical details not described in detail in this embodiment may be found in the methods provided in the embodiments of the present application.

Fig. 14 is a schematic diagram of a hardware structure of the second controller in one embodiment of the client, and as shown in fig. 14, the second controller 23 includes:

One or more second processors 231, a second memory 232. In fig. 14, a second processor 231 and a second memory 232 are taken as examples.

The second processor 231, the second memory 232 may be connected by a bus or otherwise, for example in fig. 14.

The second memory 232 is used as a non-volatile computer readable storage medium for storing a non-volatile software program, a non-volatile computer executable program, and modules, such as program instructions/modules corresponding to the object storage batch writing/reading method in the embodiment of the present application (for example, the writing request receiving module 801, the writing request responding module 802, and the reading request responding module 901 shown in fig. 11). The second processor 231 performs various functional applications of the controller and data processing, i.e., implements the object storage batch write/read method of the above-described method embodiment, by running nonvolatile software programs, instructions, and modules stored in the second memory 232.

The second memory 232 may include a storage program area that may store an operating system, an application program required for at least one function, and a storage data area that may store data created according to the use of the second object storage batch write/read device, and the like. In addition, the second memory 232 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the second memory 232 may optionally include memory remotely located relative to the second processor 231, which may be connected to the client via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The one or more modules are stored in the second memory 232 and when executed by the one or more second processors 231 perform the object store batch processing method of any of the method embodiments described above, e.g., performing method steps 201-202, 203-208 of fig. 7-8 described above, and implementing the functions of modules 801-802 of fig. 11 and 901 of fig. 12.

Embodiments of the present application provide a non-volatile computer-readable and writable storage medium storing computer-executable instructions that are executed by one or more processors, such as one of the first processor 131/second processor 231 in fig. 13 and 14, to enable the one or more first processor 131/second processor 231 to perform the object storage bulk read/write method in any of the method embodiments described above, such as performing method steps 101 through 103 in fig. 1, method steps 104-106 in fig. 4, and method steps 201 through 202 in fig. 7 described above, to implement the functions of modules 701-704 in fig. 9, modules 601-603 in fig. 10, modules 801-802 in fig. 11, and 901 in fig. 12.

The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

From the above description of embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus a general purpose hardware platform, but may also be implemented by means of hardware. Those skilled in the art will appreciate that all or part of the processes implementing the methods of the above embodiments may be implemented by a computer program for instructing relevant hardware, where the program may be stored in a computer readable storage medium, and where the program may include processes implementing the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a random access Memory (RandomAccessMemory, RAM), or the like.

It should finally be noted that the above embodiments are only intended to illustrate the technical solution of the present invention and not to limit it, that the technical features of the above embodiments or of the different embodiments may be combined in any order, and that many other variations in the different aspects of the present invention as described above exist, which are not provided in details for the sake of brevity, and that although the invention has been described in the detailed description with reference to the foregoing embodiments, it should be understood by those skilled in the art that it may still make modifications to the technical solution described in the foregoing embodiments or equivalent to some of the technical features thereof, where these modifications or substitutions do not depart from the essence of the corresponding technical solution from the scope of the technical solution of the embodiments of the present invention.

Claims

1. An object storage batch reading method, which is applied to a storage gateway, the method comprising:

After receiving a POST reading request sent by a client, responding according to the POST reading request, wherein the POST reading request comprises a reading request head, target bucket information positioned at the reading request head or a request row, an appointed object positioned at a reading request body and fragment data of the appointed object, the appointed object is an object to be processed through which the request passes, and the number of the appointed objects is more than 1;

the responding according to the POST reading request comprises the following steps:

Carrying out validity check on the POST reading request;

reading the plurality of pieces of data of the specified range according to the reading sequence;

Placing the response result in a response request row;

Setting a slicing dividing line in the response request head;

starting with a slicing parting line in the text of the response request body;

and adding an object dividing line to finish the encapsulation of the fragmented data of the specified range of all the specified objects, and sending the response result to the client.

2. An object storage batch processing writing method, which is applied to a storage gateway, comprising:

Analyzing the writing request head, and checking the validity of the request interface of the writing POST request, wherein when the validity check of the request interface of the writing POST request fails, a first response result that the validity check of the request interface of the writing POST request fails is obtained;

When the target barrel information and the authority of the object to be processed are not checked to pass, a fourth response result that the verification is not passed is obtained;

Placing the response result in a response request row;

Setting an object dividing line, and placing the object dividing line in a response request head;

starting with a response split line in the body of the response request body;

3. A storage gateway, the storage gateway comprising:

At least one second processor, and

A second memory communicatively coupled to the second processor, the second memory storing instructions executable by the at least one second processor, the instructions executable by the at least one second processor to enable the at least one processor to perform the method of claim 1 or 2.

4. A non-transitory computer readable storage medium storing computer executable instructions which, when executed by a terminal, cause the storage gateway to perform the method of claim 1 or 2.