DE202014010886U1 - Programmatic selection of preferred storage parameters for files in large, distributed storage systems - Google Patents

Abstract

A system for programmatically determining the preferred storage parameters for files stored in a large distributed storage system, comprising: one or more processing devices and one or more storage devices storing instructions that when executed by one or more processing devices, one or more processing devices do: receiving the request for desired file reliability or availability; receiving the system properties including statistics on system failures in a data center; predicting future failures of the storage system based on the statistics of the system failures; receiving the file encryption or replication information for at least one file in the storage system; predicting file reliability and maintenance costs based on past system failures and file encryption and replication information; and determining the preferred storage parameters for at least one file to reduce costs while achieving desired file reliability or availability based on the received request.

Description

BACKGROUND

As in 1 Large, distributed storage systems shown provide networked online storage and allow multiple computing devices to store, access, and share their files through online storage. Distributed storage systems may use a client / server architecture in which one or more central servers store files and provide file access to network clients. Be provided under protection and subject of the utility model are, according to the provisions of the utility model law, only devices as defined in the appended claims, but no method. Wherever in the description, if appropriate, reference is made to methods, these references are merely illustrative of the device or devices set forth in the appended claims.

Storage devices can include servers in a data center with flash, drives, and / or RAM, and can provide various forms of access to the files. For example, some storage devices provide faster access to files or could allow for increased availability.

Data files stored in large, distributed storage systems can be very large and potentially reach several gigabytes in size. To manage these files, they are typically divided into blocks of a fixed size, as illustrated in 2a and 2 B , Multiple data blocks can make up a data file. Code blocks are additional blocks that are calculated from the data blocks using a mathematical formula. Normally, when the user reads out data stored in a large, distributed storage system, that data is read out of pieces of data, called systematic blocks. However, if some of the data blocks are unavailable, the storage system could read the code blocks to mathematically recalculate missing data blocks over the available data blocks and code blocks. The number of failed or unavailable data blocks that can be recovered depends on the amount of code blocks available and the encryption system used. Even though 2 B shows a file split over blocks of one byte or one character; In general, blocks are a contiguous part of a file. For example, the contiguous part could be the first 1 MB of the file. Using a 4 + 2 encryption, similar to the 2 B For example, a stripe might comprise 4MB of a data file, the data being spread over multiple blocks (0-3) and all blocks containing one megabyte of the file. Two blocks of code can be constructed from blocks 0-3 using a mathematical operation. If the file is larger than 4 MB, a second strip could be created, which can also be up to 4 MB with two blocks of code.

When storing files in large, distributed storage systems, blocks can be split across multiple storage devices. This data striping is a technique for decomposing logically sequential files so that consecutive segments can be stored on different physical storage devices. A stripe is a segment of a file that is written to a storage device. The size of each stripe, or block, can be configured so that each storage device can provide the maximum amount of data in as short a time as possible. The stripe width is the number of parallel stripes that can be written or read simultaneously. Striping is useful when a client device requires access to a file that may need to be faster than a single storage device could provide. By striping the file across different storage devices, a client device can simultaneously access multiple file segments.

In addition to splitting into blocks and stripes across multiple storage devices, files in a large, distributed storage system can also be encrypted to correct errors that occurred during file transfer or storage. For example, Reed-Solomon encryption could add extra "redundant" bits to the files that could be used to restore the original file if those bits are lost during file transfer or storage.

Once a file is saved, clients can query the file from the large, distributed storage system. When a client requests access to a data file or part of a data file, a large, distributed storage system server will respond with the location of the corresponding file. In some large systems, the client could acquire a lease of the file and hold that lease until the client releases the file. Although multiple read accesses to the file can still be granted during the lease period, no other client can describe the file. In addition, it may be desirable for multiple clients at the same time to access the same file to replicate the data to provide sufficient power to access more than one client for that particular file block To allow time. Additionally or alternatively, files may be replicated across multiple storage devices to allow for multiple file blocks even if one of the storage devices fails or is unavailable. The replication level of a given block is the number of storage devices on which a duplicate file is stored. For example, a triple-replicated strip would mean placing three blocks of the exact copies of each other on three different storage devices. Saving files to multiple systems in multiple datacenters could be quite costly. Saving files to a single system or datacenter, however, could cause problems if the system or datacenter fails and the file is not available.

SUMMARY

This specification describes the technologies associated with cloud storage in general and, more particularly, the methods and systems for programmatically selecting preferred configuration parameters for storing and providing files in a large, distributed storage system.

In general, one aspect of the subject matter described in this specification may be included in a system and method for programmatically determining the preferred storage parameters for files stored in a large, distributed storage system. An exemplary system could include one or more processing devices, as well as one or more storage devices that include instructions for performing the steps of an exemplary method. An exemplary method could include: receiving the desired file reliability or availability requirements; Receive system properties that include statistics on system failures in data centers; the prediction of future memory system failures based on the statistics of the system failures; Receiving the file encryption or replication information for at least one file in the file system; Predicting file reliability and maintenance costs based on the previous system failure, file encryption and replication information; and determining the preferred storage parameters for at least one file to reduce costs while achieving the desired file reliability or availability based on the requests received.

A second exemplary method could include: receiving requests for desired file reliability or availability; Receiving the system properties including information about the storage system, off-site backups for the storage systems, and file access patterns for information stored in the storage system; Determining the historical reliability of the system, the reliability of off-site storage system backups, and the reliability of files stored in the system through one of the possible file encryption or replication levels; Predicting the future reliability of the system based on the determination of the reliability of the storage system; Predicting the future reliability of the files stored in the storage system for each file encryption or replication level; Determining the likelihood of losing both the large system and the off-site backup using the predicted reliability of the storage system and the off-site backup; and determining a preferred storage parameter for at least one file based on the determined probability of losing both the storage system and the off-site backup, the predicted reliability of each encryption and replication level, and the received requests for the desired file reliability or availability.

These and other embodiments may optionally include one or more of the following features. A storage parameter could be a stripe width, stripe size, an encryption or replication level, a backup schedule for large, distributed storage systems, or a location for storing a file. A location could define the location of a file on a device in a large, distributed storage system, or the location could define the location of a file on a backup device of a large, distributed storage system. The file properties can be viewed on a dashboard. A user could be informed of a preferred storage parameter. A created, preferred storage parameter could be used to store a file with or without the user's consent.

In general, one aspect of the subject matter described in this specification may be included in a system and method for programmatically determining the preferred storage parameters for files stored in a large, distributed storage system. An exemplary system could include one or more processing devices, as well as one or more storage devices that include instructions for performing the steps of an exemplary method. An exemplary method could include receiving trace data that provide access to information about files stored in large, distributed storage systems; Analysis of trace data to identify file access patterns; Receiving the metadata information associated with the data stored in the large distributed storage system; and creating a preferred storage parameter for at least one of the files based on the receiving information and identified file access patterns.

These and other embodiments may additionally include one or more of the following functions: at least one property of a memory system could be obtained in addition to the received information and identified file access patterns and used to create a preferred memory parameter; a system property could be a system failure history, maintenance failure history or schedule, system equipment availability, system equipment performance, or I / O capacity; and a file reliability and availability requirement may be received and used in addition to the obtained information and identified file access patterns to create a preferred storage parameter. Analyzing the trace data to identify the file access patterns could include receiving trace data for a subset of files in a large, distributed storage system; the determination of the properties of the subset; and assessing the distribution of the properties for the files in the large distributed storage system based on the particular properties of the subset. A property for the files could be the age of each file at the time of the last access to the file. Creating a preferred storage parameter for at least one file could be the definition of at least one fileset based on the identified file access patterns, creating a preferred storage parameter for the fileset properties; and include using the created storage parameters to store at least one file associated with the fileset. Metadata information associated with the file can be obtained along with file reliability or file availability information. This information can be compared to the information of a defined filegroup in the large, distributed storage system. If the file information matches the information of a defined filegroup, the file is saved with the storage parameters of that filegroup. If the file information does not match the information of a defined filegroup, a new filegroup is created based on the file information, and the preferred storage parameters can be created for the new filegroup based on the filegroup properties. Before implementing a filegroup policy for allocating storage parameters to files based on the generated preferred storage parameters for a new filegroup, the user's consent is required. Storage parameters could include: stripe width, stripe size, encryption or replication level, a backup schedule, or a location. A location could define the location of a file on a device in a large, distributed storage system, or the location could define the location of a file on a backup device of a large, distributed storage system. The properties of the fileset may be displayed on a dashboard, and a user may be notified of the preferred storage parameters. A created, preferred storage parameter could be used automatically or, upon the user's consent, to store a file. A user may be informed that the application of preferred storage parameters improves the cost of file storage. Preferred storage parameters can be created for a variety of files, and users can be informed of the preferred parameters for a subset of the entire files, with the application of preferred storage parameters improving the availability, reliability, or memory cost of files.

The details of one or more embodiments of the invention are set forth in the accompanying drawings, which are given by way of illustration, and in the description below. Other features, aspects and advantages of the invention will be apparent from the description, drawings and claims. Corresponding reference numbers and markings in the various drawings indicate corresponding elements.

BRIEF DESCRIPTION OF THE DRAWINGS

1 FIG. 10 is a block diagram illustrating an example configuration of a large distributed storage system. FIG.

2a is a block diagram illustrating an example of block files.

2 B is a block diagram illustrating an example of block files.

3 FIG. 10 is a flow chart of an exemplary method for programmatically creating preferred storage parameters for files stored in large, distributed storage systems.

4 FIG. 10 is a flowchart of an exemplary method of analyzing the trace data to identify the file access patterns in a large, distributed storage system.

5 FIG. 10 is a flowchart of an exemplary method for determining the preferred storage parameters for files stored in large, distributed storage systems.

6 FIG. 10 is a flowchart of an exemplary method for determining the preferred storage parameters for files stored in large, distributed storage systems.

7 is a block diagram illustrating an example of a data center.

8th is a block diagram illustrating an example computing device.

DETAILED DESCRIPTION

Replication, encryption level, stripe width, stripe length, storage system backup schedule, and file location are all configurable storage parameters that can affect the performance of the large, distributed storage system in terms of file access and query. Large, distributed storage systems should use these parameters to store files, so that system resources can be optimized and files can be retrieved as effectively and quickly as possible.

Optimizing large, distributed storage systems might involve balancing the cost of file access, overall system performance, and the availability and reliability of files. Optimization could be achieved by changing the file storage parameters, including replication, encryption level, file strip width, stripe size, storage system backup schedule, and file location based on factors such as file properties and storage system properties.

Although custom systems allow application developers to select file storage parameters, application developers may not understand the preferred storage parameters to select or are unaware of how storage parameters interact. Aspects of innovative concepts may automatically or programmatically select or recommend preferred storage parameters for a particular set of files or workload in a distributed storage system based on the characteristics of the file and / or the storage system to optimize the large storage system.

In particular, when attempting to optimize a large storage system, the type of files the system stores and the file access patterns are important. The file type can be specified by parsing the metadata associated with the files. Metadata can include file properties such as file size, file name, file age, and actual file type.

An exemplary embodiment could use the telemetry or monitoring systems to obtain telemetry and / or system monitoring data (hereafter referred to as trace data). These obtained trace data can be used to identify file access patterns for files stored on large, distributed storage systems. Surveillance systems could provide an exemplary storage system with trace data in log files or other formats. Individual events and sequences of events with timestamp in chronological order, so-called traces, can be recorded in the trace log files. Trace data may include information about file access in a large memory system, such as read and write access, updates, users or applications accessing a file, the frequency of file access, the length of file access, the sequence in which different parts the file being accessed, the user or application that last accessed the file, and the date the file was last accessed with other information.

Trace data may be used by an exemplary storage system to determine file access patterns for individual files or sequences of file accesses. The sequence of accessing data might be useful for caching analysis, which could help determine, for example, whether storage of files on a server with significant RAM cache or on flash memory is preferred. An exemplary method of creating preferred storage parameters for files stored in large, distributed storage systems might begin with the receipt of trace data representing access information about files stored in the large distributed storage system, as in FIG 3 ( 301 ) discussed. The trace data can be analyzed to identify the file access patterns ( 303 ). The system could also obtain metadata information associated with the files that are stored in the large, distributed storage system ( 304 ) are stored. A preferred storage parameter could then be created for at least one of the files based on the obtained one Information and the identified file access patterns ( 305 ).

In some embodiments, analyzing trace data to identify the file access patterns also includes receiving trade data for a subset of the files of the large, distributed storage system. This allows a property of the subset to be determined. An exemplary system may then judge the distribution of the properties or the properties even for the files in the large distributed storage system, based on the property intended for the subset. For example, the age of each file at the time of the most recent access to the file could be assessed based on a subset of storage system files. 4 shows that an exemplary method using the access patterns of a subset of the file population ( 401 ) Analyze trace data to identify file access patterns for calculating the rated age of a file from the subset when the file was last accessed ( 402 ). The estimated access age of each file from the subset can be used to derive the distribution of the age at which a file from the entire file population was last accessed ( 403 ). This distribution can be used to identify the access pattern of a file based on the age of the file and the particular distribution ( 404 ).

Telemetry and / or surveillance systems may also provide an exemplary system with information regarding the characteristics of the large, distributed storage system. In one embodiment, an exemplary method could receive at least one storage system property and use that storage system property in addition to other received information and identified file access patterns to generate the preferred storage parameter. Storage system properties could include the availability, capacity, and performance of storage devices, including servers in a large, distributed storage system. An exemplary embodiment could obtain statistics on device failures and reliability of storage devices in a storage system as well as off-site backups. Off-site backups could be secondary instances of a distributed storage system located in datacenters other than the data center where the primary distributed storage system resides. A site-external backup could also be site-external cooling storage devices. A preferred storage system could be the location of the file in the large, distributed storage system. The location could include determining a device in a large, distributed storage system to which a file is to be stored; or determining file storage on a backup device of the large, distributed storage system. Alternatively, a preferred storage parameter could be a backup schedule that determines how often data from the large, distributed system is stored on the backup systems and whether these are full or incremental backups.

An exemplary embodiment could obtain the desired file reliability and availability requirements as input by a user or an application. An exemplary system might need to understand if the file requires extremely reliable access or if some probability of failure is allowed. The file availability or reliability requirement could be used along with other received information or identified file access patterns to create a preferred storage parameter.

Monitoring system trace data could provide information so that an exemplary system can determine name patterns and / or other types of files for files from a particular application or process. Naming patterns can be combined with other file properties, such as the current user or application accessing the file, the time of day the file was created, whether the user or application specified an encryption or replication level, encryption or encryption Replication level, if one is used, and / or the permissions associated with the file. File properties and / or file access patterns can be used to find files that could be grouped together. The grouping could be done manually or by machine learning classification or clustering algorithm.

Two techniques for grouping files are: (1) classifying files and (2) clustering files. Classification involves grouping files based on known criteria, such as specific metadata functions. Clustering involves grouping files with similar access patterns or properties. A number of properties can be used to cluster files. An example of a clustering technique is "k-means", but there are many other classifications of machine learning and clustering techniques for data mining. This technique can be used to cluster files in k groups with similar properties by creating heuristic functions that define the distance between files based on their properties. Classification techniques, such as classification trees or Regression trees "could be used to form filesets that predicted to have some properties based on the patterns previously seen. For example, groups of files may be subject to intensive access for a period of three months, which will then cease altogether, as these access patterns for this filegroup have been observed in the past. The regression tree classifier could get some files that have this access pattern, as well as others that do not have this pattern. The classifier can then come up with a criterion that can predict this access pattern behavior from the outset.

Files can be classified into specific categories, including: (1) important, frequently accessed files with high availability and low latency requirements; (2) important, infrequently accessed files with high availability requirements but no latency requirements; (3) unimportant, frequently accessed files; and (4) unimportant, infrequently accessed files. Files can be grouped using the file metadata and / or file access patterns. These groups can be associated with the preferred memory parameters. For example, a high file availability requirement could map a larger stripe width of file data that provides greater redundancy in the case of storage device failure. Low latency requirements for a file may reflect a higher replication level of file data, rather than Reed-Solomon encryption, which, while providing redundancy, requires computational power and thus increases latency to recover missing data in the event of a storage device failure. A category of infrequently accessed, unimportant files could indicate encryption capabilities with lower replication costs to reduce the cost of file storage. For example, a file could be replicated across multiple datacenters on different continents. The file could be accessed multiple times from each continent. Accessing a file from different continents is expensive, so replication makes sense. As soon as the access rates for a file go down, or it can be determined that access to a particular file will be low, the file in the data centers can be moved to Reed Solomon encryption. This encryption gets availability even if a datacenter is not available; this leads to higher costs per access but lower storage costs.

Files can also be clustered in groups with similar access properties. Depending on the user or application accessing a particular file, there might be patterns in file usage. For example, in an electronic book-building application, there could be a number of text and image files that will eventually be converted into an eBook format, such as * .epub, for marketing. After conversion, the original files are rarely accessed. Similarly, a database running on a large, distributed storage system could store intermediate formats of files with certain recognizable name patterns. These files may be used for a limited period of time after creation, but intensive use may occur during this period. Clustering involves selecting a function or set of functions to group or cluster the files into. Access patterns for a particular file in the cluster can be assessed based on the particular access patterns of other files in the cluster. Preferred storage parameters such as encryption or replication level and location are determined based on the assessed access patterns, as well as system properties such as server reliability and speed.

For example, an application might create multiple temporary files during execution, all stored in a large, distributed storage system. These temporary files can be prefixed with "TEMP.". Using trace data from the monitoring system, an exemplary storage system could analyze the trace data and determine that these temporary files can be used during application execution. However, once the application finishes running, the temporary files are no longer needed. Trace data may inform an exemplary system that temporary files will only be used for a certain period of time after they are created, but that use in that limited time period may be common. For example, temporary files can often be used on the day they were created, but they may never be used again after that day. However, the application might not be able to delete the temporary files from the storage system.

An example technique might cluster these temporary files using file properties such as filename, creation date, last access, and access patterns. Files with filenames that have the prefix "TEMP." And whose last access date is within one day of the creation date can be assigned to this cluster. The cluster could specify a location storage parameter that keeps the file connected to the cluster in fast memory at the file's creation date, as this file is frequently accessed. After this Creation date, the cluster can specify that the temporary file should be moved to a slower memory, since the access probability after the creation date is reduced.

In another example, if it is determined on the basis of trace data that an example system actively uses the application files for the first week, and then the data is hardly ever used, an example system might need to provide only enough memory to cache one Week to find new files for the application space in memory. The exemplary system could move all files over a week old to a slower memory; based on the file access patterns, after which the system has determined that files are rarely used after one week.

When an exemplary system observes that some file patterns have temporal access patterns, such as: For example, if files are no longer called after a certain period of time (for example, 30 days), the system may determine that the encryption of those files must match the file pattern once the file reaches a certain age. An exemplary system might observe the access pattern behavior through the trace data. It might also be possible to set up monitoring for certain conditions. The monitor could count and report how many hits a particular set of files receives if the group definition (or file pattern) is known in advance.

If a file was created with a stripe length N, but the file was completed and marked as immutable, an exemplary system can compare the total size of the file with the given stripe length and re-encrypt the data to use a stripe length of the total size of the file, if the total size of the file is less than the default stripe length.

In some embodiments, creating a preferred storage parameter for at least one file includes defining at least one fileset based on the identified file access patterns, creating a preferred storage parameter for the properties of the fileset; and using the created storage parameters to store at least one file associated with the filegroup.

Upon subsequent storage of the new file, an exemplary system could receive metadata information associated with the file and obtain information / requirements about file reliability or availability associated with the file. The system could then determine if the received file information matches the information of a defined filegroup in a large, distributed storage system. If the properties of the new file match the properties of a defined group, the new file can be saved with the storage parameters of that corresponding filegroup. If the file properties do not match the properties of a defined filegroup, a new filegroup is created based on the file information, and the preferred storage parameters can be created for the new filegroup based on the filegroup properties. In some embodiments, prior to implementing a filegroup policy for allocating storage parameters to files based on the generated preferred storage parameters for a new filegroup, the user's consent may be required.

In some embodiments, an exemplary system could determine the storage parameters, such as file encryption or replication, to minimize the cost of file storage while at the same time providing adequate file reliability / availability based on the receiving or particular file reliability / availability requirements. An exemplary method of determining the preferred storage parameters for files stored in the large distributed storage system begins with obtaining the desired reliability and / or availability requirements for a file or set of files, as illustrated in FIG 5 ( 501 ). The method could also include preserving the properties of the storage system that include statistics on system failures in a data center ( 502 ). Storage system properties could include the availability, capacity, and performance of storage devices, including servers in a large, distributed storage system. For example, the selected file encryption could be limited by the amount of available space on the storage system because encryption determines how much space the file uses. The historical information can be obtained from the trace data provided by a monitoring system that monitors the health of the large, distributed storage system. Future failure of the distributed storage system could be predicted using these statistics ( 503 ). An exemplary technique might assume that individual storage devices will continue to fail at a similar rate in the future as storage devices have failed in the past. The probability of simultaneous, related Failure of storage devices could also be considered at a constant rate compared to the previously observed rate. Alternatively, a more pessimistic predictive model could be used that adds a factor to increasing the expected number of future failures designed to age the large, distributed storage system equipment.

In other embodiments, an exemplary system could receive file encryption or replication information for the information stored in the system ( 504 ). This encryption or replication information could come from received file metadata. Actual file reliability / availability and maintenance costs can be predicted based on past or historical system failure and file encryption and replication information ( 505 ). A preferred storage parameter, such as file encryption or replication, for at least one file could be determined to reduce the cost of the file store while achieving the desired file reliability (FIG. 506 ).

To achieve the desired file reliability / availability per file, a systematic approach could use system properties, file properties, and the desired file reliability / availability to determine the preferred storage parameters, such as encryption and replication levels. The process begins with obtaining the desired file reliability or file availability according to 6 ( 601 ). The method could then receive system properties ( 602 ). System properties could include traces and other data that help an exemplary storage system determine the reliability of the system in general, determine the historical reliability of off-site backups for the storage system, and determine the reliability of files that can be accessed using any file encryption or Replication level are stored in the distributed storage system ( 603 ). The method could predict the future reliability of the storage system based on the particular reliability of the system ( 604 ). The process could also predict the future reliability of off-site storage system backups based on the reliability of the backup ( 605 ). In addition, the method could also predict the future reliability / availability of files stored in the storage system for each possible level of encryption and replication (FIG. 606 ). The likelihood of losing the large, distributed storage system and storage system backup at the same time could be determined by using the predicted reliability of the storage system and its backups ( 607 ). The method may then determine a preferred storage parameter, such as encryption or replication level, for at least one file based on (1) the determined likelihood of both the storage system and the off-site backups being lost; (2) the predicted reliability of all encryption or replication levels , and (3) the received file reliability or availability requirements ( 608 ).

Other custom procedures, like those in "Designing Dependable Storage Solutions for Shared Application Environments", Shravan Gaonkar et al., 2006 , can use abstract workload models, devices, and reliability techniques (e.g., redundancy and backups) that can be applied to determine good conditions for storing files in large, distributed storage systems. However, these custom methods do not suggest the use of trace data and / or system monitoring to determine preferred storage parameters for files stored on large, distributed storage systems.

Recommendations and / or automatic adjustments of memory parameters resulting from the exemplary methods could be displayed to a user via the dashboard or other interface or API. A user could also view the predicted reliability, distribution of age-related access patterns, and other file and system properties through a dashboard, interface, or API. In particular, a dashboard could display the user-relevant access properties and patterns related to the files so that the user can understand and improve access behavior. For example, the user or the user application could run large amounts of very small read accesses or create many small files, both of which could be costly. A user might also observe that the user or an application in conjunction with the user stores large amounts of old but unused data on a large, distributed storage system. This behavior and / or the access patterns can be costly and the user could find ways to minimize these patterns of behavior. The user could delete files based on information received through the dashboard.

Recommendations can be issued additionally or alternatively as notifications. In addition to recommendations regarding configuration parameters, an exemplary Method also recommend deleting a file to a user if the method determines that the file is no longer needed or used. An exemplary system could use the date of the most recent file access to instruct a user to consider deleting a file. For example, if a file was no longer used for a period of 180 days, the system might ask the user for the need to continue saving the file. When the data of a particular file or file pattern is rarely read; that is, if, for example, there were no accesses in the last 90 days, an exemplary system could notify a user or an application that the data should be re-encrypted in a more space-efficient manner. For example, for replicated data, an exemplary system might reduce the number of replicas. Once the data has been encrypted with Reed-Solomon, an exemplary system can determine new encryption with fewer redundancies, reducing the amount of disk space required for the file.

Exemplary notifications could include: (1) if a file has not been opened in a year, the user will be notified to delete or archive the file; (2) if a file has not been opened in a month but is stored on an expensive storage medium, such as a flash memory; the user is notified that the file is stored on an expensive storage medium but is not used; (3) if a file was not opened in a month but is stored in a relatively expensive format, such as: For example, with four replicas of the file, the user is notified that a more space-efficient encryption of the file is recommended. These notifications are only example alerts and are not intended to limit the scope of this application in any way.

An exemplary system could notify a user of the preferred storage parameters to alert a user to files that currently have bad storage configurations. By using the preferred storage parameters, a user could significantly improve the cost of file storage or better meet compliance with the desired availability / reliability requirements. An exemplary system could alternatively inform a user only about the preferred storage parameters for a subset of the files. Basically, there could be so many files with possibly better configurations that a user or an exemplary system would limit the fixes to only those files that could achieve the most significant improvements. An exemplary system could compare the potential improvements for files if preferred storage parameters are to be used to determine the subset of files that a user should be informed about. A user could be notified of a subset of files for which the application of the preferred storage parameters would result in a significant improvement in file availability and reliability as well as storage cost.

Preferred memory parameters can be automatically implemented to resolve memory configuration issues in a large, distributed storage system. In some embodiments, the preferred memory parameters may only be used to resolve memory issues after obtaining the appropriate approval. Preferred memory parameters could be used to store a file at file creation or at a later time once the file has been stored in a large, distributed storage system.

Large storage systems and access can be handled in a distributed file processing and storage system, such as a data center or a network of data centers. For example, large-scale Internet services and the massive, parallel computer infrastructure that supports these services could use warehouse-sized computer systems consisting of thousands or even tens of thousands of compute nodes.

7 is a block diagram illustrating an example of a data center ( 700 ). The data center ( 700 ) is used to store files, perform computer-aided tasks, and transfer files to other systems outside the data center. For example, it uses a network connected to the datacenter. The data center ( 700 ) could, above all, perform large file processing for the programmatic selection of the preferred storage parameters for stored files.

The data center ( 700 ) contains several racks ( 702 ). While only two racks are displayed, the datacenter might ( 700 ) have many more racks. Each rack ( 702 ) could include a frame or housing in which components, such as processing modules ( 704 ). In principle, every processing module ( 704 ) include a printed circuit board and a motherboard on which a plurality of computer-related components for file processing is mounted. The processing modules ( 704 ) within each rack ( 702 ) are connected to each other via a rack switch and the racks ( 702 ) within each data center ( 700 ) are also connected to each other, for example via a data center switch.

In some implementations, the processing modules ( 704 ) take over the role as master or slave. The master modules control the scheduling and file distribution tasks within the master and slave modules. A rack could include a memory (eg, one or more network storage devices) that may be stored by one or more processing modules (e.g. 704 ) and / or each processing module ( 704 ) has its own memory. In addition or alternatively, a remote memory could also be connected to the racks via the network.

The data center ( 700 ) could include dedicated optical links or other dedicated communication channels, as well as supporting hardware such as modems, bridges, routers, switches, wireless antennas, and towers. The data center ( 700 ) could include one or more Wide Area Networks (WANs) and multiple local area networks (LANs).

8th FIG. 4 is a block diagram illustrating a computing device example (FIG. 800 ) for one or more processing modules ( 704 ) of a data center could be used. In a very basic configuration ( 801 ) contains the computing device ( 800 ) typically one or more processors ( 810 ) and system memory ( 802 ). A memory bus ( 830 ) can be used for communication between the processor ( 810 ) and the system memory ( 820 ) be used.

Depending on the desired configuration, the processor ( 810 ) of any type, including, but not limited to, a microprocessor (μP), a microcontroller (μC), a digital signal processor (DSP), or any combination thereof. The processor ( 810 ), another cache level, such as a level 1 cache ( 811 ) and a level 2 cache ( 812 ), a processor core ( 813 ) and a register ( 814 ). The processor core ( 813 ) may comprise an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core (DSP core), or any combination thereof. A memory controller ( 816 ) can also be used with the processor ( 810 ), or in some implementations, the memory controller ( 815 ) an internal part of the processor ( 810 ) be.

Depending on the desired configuration, the system memory ( 820 ) of any type, including but not limited to volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. System memory ( 820 ) typically includes an operating system ( 821 ), one or more applications ( 822 ) and program files ( 824 ). The application ( 822 ) programmatically selects the preferred storage parameters for storing files in large, distributed storage systems. The program file ( 824 ) could include data from telemetry or surveillance systems as well as storage system metadata. The operating system ( 821 ) typically involves procedures for handling various basic system services and performing tasks that depend on the hardware. In some embodiments, the application may ( 822 ) to be set up on an operating system ( 821 ) to work.

The computing device ( 800 ) may have additional features or functionalities and additional interfaces to facilitate communication between the basic configuration ( 801 ) and all necessary equipment and interfaces.

The system memory ( 820 ) is an example of electronic storage media. Electronic storage media include but are not limited to RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, tapes, disk storage or other magnetic storage devices or any other medium can be used to store the desired information pointed to by the computing device ( 800 ) can be accessed. Any such storage medium may be part of the device ( 800 ) be.

The telemetry / monitoring system data ( 802 ) and the storage system metadata ( 803 ) provide information, including access patterns and properties, regarding the files stored in the large storage system. The telemetry and / or surveillance file could be obtained from telemetry and / or surveillance systems.

The computing device ( 800 ) may be used as part of a small form factor mobile electronic device such as a mobile phone, smartphone, PDA (minicomputer), a personal media player, tablet computer, a wireless web-watch device, a personal headset Device, an application-specific device or a hybrid device can be implemented, which includes one of the above functions. The computing device ( 800 ) can also be implemented as a personal computer that includes both portable computers and non-portable computer configurations.

The foregoing detailed description has defined various embodiments of the devices and / or processes through the use of block diagrams, flowcharts, and / or examples. To the extent that such block diagrams, flowcharts, and / or examples include one or more functions and / or operations, they will be understood by those in the science that any function and / or operation can be implemented with such block diagrams, flowcharts, or examples, individually and / or or collectively, through a wide range of hardware, software, firmware of any virtual combination thereof. In one embodiment, some portions of the subject matter described herein may be implemented via application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), or other integrated formats. However, those skilled in the art will appreciate that some aspects of the embodiments set forth herein may be implemented, in whole or in part, in integrated circuits as one or more computer programs executing on one or more computers as one or more programs running on one or more computers or multiple processors, as firmware or any combination thereof; and that a design of the circuits and / or the writing of the code for the software and / or firmware taking into account the present publication would be a great achievement. In addition, those skilled in the art will understand that the mechanisms of this subject matter described herein are capable of being distributed as a program product in a variety of forms, and that an illustrative embodiment of the subject matter described herein applies regardless of the particular nature of the non-volatile signal carrying medium which is used for the actual distribution. Examples of a non-transient signal bearing medium include, but are not limited to: a receptive medium, such as a medium; A floppy disk, a hard disk drive, a compact disk (CD), a digital video disk (DVD), a digital tape, a computer memory, etc .; and a transmission medium such as a digital and / or analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communication link, a wireless communication link, etc.)

With respect to the use of substantially any plural and / or singular expressions, those skilled in the art can translate from plural to singular and / or singular to plural, as appropriate to the context and / or application. The various singular / plural permutations may be expressly set forth herein for the sake of clarity.

Thus, certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the acts described in the claims may be performed in a different order and still achieve desirable results. In addition, the processes illustrated in the attached figures do not necessarily require the particular order shown or sequential order to achieve desired results. In certain implementations, multitasking and parallel processing may be beneficial.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited non-patent literature

• "Designing Dependable Storage Solutions for Shared Application Environments", Shravan Gaonkar et al., 2006 [0046]

Claims (51)

A system for programmatically determining the preferred storage parameters for files stored in a large distributed storage system, comprising: one or more processing devices and one or more storage devices storing instructions that, when executed by one or more processing devices, one or more processing devices: receiving the request for desired file reliability or availability; receiving the system properties including statistics on system failures in a data center; predicting future failures of the storage system based on the statistics of the system failures; receiving the file encryption or replication information for at least one file in the storage system; predicting file reliability and maintenance costs based on past system failures and file encryption and replication information; and determining the preferred storage parameters for at least one file to reduce costs while achieving desired file reliability or availability based on the received request. The system of claim 1, wherein the storage parameter is the stripe width. The system of claim 1, wherein the storage parameter is the strip size. The system of claim 1, wherein the storage parameter is the location for file storage. The system of claim 4, wherein the location defines storage of the file on a device in the large distributed storage system. The system of claim 4, wherein the location defines storage of the file on a backup device for the large distributed storage system. The system of claim 1, wherein the storage parameter is an encryption or replication level. The system of claim 1, wherein the storage parameter is a backup schedule for the large distributed storage system. The system of claim 1, further comprising displaying the file properties on a dashboard. The system of claim 1, further comprising notifying a user of preferred storage parameters. The system of claim 1, further comprising using the preferred storage parameters to store the file. The system of claim 1, further comprising using the created preferred storage parameters to save the file upon receipt of the user consent. A system for programmatically determining the preferred storage parameters for files stored in a large distributed storage system, comprising: one or more processing devices and one or more storage devices storing instructions that, when executed by one or more processing devices, one or more processing devices: receiving the request for desired file reliability or availability; receiving the system properties including information about the storage system, the off-site storage system backups, and the file access patterns for files stored in the storage system; determining the historical reliability of the system, the reliability of off-site storage system backups, and the reliability of files stored in the distributed storage system by any possible file encryption or replication layer; predicting the future reliability of the storage system based on the particular reliability of the system; predicting the future reliability of files stored in the storage system for each encryption and replication layer; determining the likelihood of losing both the large system and the storage system backup by using the predicted reliability of the storage system and its off-site backups; and determining a preferred storage parameter for at least one file based on the determined likelihood of losing both the storage system and the off-site backups, the predicted reliability of all encryption or replication levels, and the desired file reliability or availability requirements. The system of claim 13, wherein the storage parameter is the stripe width. The system of claim 13, wherein the storage parameter is the strip size. The system of claim 13, wherein the storage parameter is the location for file storage. The system of claim 16, wherein the location defines storage of the file on a device in the large distributed storage system. The system of claim 16, wherein the location defines storage of the file on a backup device for the large distributed storage system. The system of claim 13, wherein the storage parameter is an encryption or replication level. The system of claim 13, wherein the storage parameter is a backup schedule for the large distributed storage system. The system of claim 13, further comprising displaying the file properties on a dashboard. The system of claim 13, further comprising notifying a user of preferred storage parameters. The system of claim 13, further comprising using the preferred storage parameters to store the file. The system of claim 13, further comprising using the created preferred storage parameters to save the file upon receipt of the user consent. A system for programmatically creating the preferred storage parameters for files stored in a large distributed storage system, comprising: one or more processing devices and one or more storage devices storing instructions that, when executed by one or more processing devices, one or more processing devices: receiving trace data representing access information about files stored in the large distributed storage system; analyzing the trace data to identify the file access patterns; receiving metadata information associated with the files stored in the large distributed storage system; and creating a preferred storage parameter for at least one of the files based on the received information and the identified file access patterns. The system of claim 25, further comprising: receiving at least one storage system property; and using the storage system property in addition to the receiving information and identifying file access patterns to generate the preferred storage parameter. The system of claim 26, wherein the system property is the system failure history. The system of claim 26, wherein the system property is the failure history or the maintenance schedule. The system of claim 26, wherein the system property is the system device availability. The system of claim 26, wherein the system property is the system device power. The system of claim 26, wherein the system property is the I / O capacity. The system of claim 25, further comprising: receiving the request for file reliability or availability; and using the received requests in addition to the received information and identified file access patterns to generate the preferred storage parameter. The system of claim 25, wherein analyzing the trace data to identify the file access patterns comprises: receiving trace data for a subset of files in the large distributed storage system; determining a property of the subset; and evaluating the distribution of the property for the files in the large, distributed storage system based on the property intended for the subset. The system of claim 33, wherein a property is the age of each file at the time of the last access to the file. The system of claim 25, wherein creating a preferred storage parameter for at least one file comprises: defining at least one fileset based on the identified file access patterns; creating a preferred filegroup storage parameter based on the properties of the filegroup; and using the created storage parameters to store at least one file associated with the filegroup. The system of claim 35, further comprising: receiving the metadata information associated with a file; receiving the file reliability or availability information with the file; determining whether the obtained file information corresponds to the information of a defined filegroup in the large distributed storage system; and in response to determining that the file information corresponds to the information of a defined filegroup, storing the file with the storage parameters of that corresponding filegroup. The system of claim 36, further comprising: in response to determining that the file information does not correspond to the information of a defined filegroup, creating a new filegroup based on the information of the file; and Creating the preferred storage parameters for the new filegroup based on the properties of the filegroup. The system of claim 37, further comprising: requesting user consent prior to implementing a filegroup policy for allocating storage parameters to files based on the generated, preferred storage parameters for a new filegroup. The system of claim 25, wherein the storage parameter is the stripe width. The system of claim 25, wherein the storage parameter is the strip size. The system of claim 25, wherein the storage parameter is the location for file storage. The system of claim 41, wherein the location defines storage of the file on a device in the large distributed storage system. The system of claim 41, wherein the location defines storage of the file on a backup device for the large distributed storage system. The system of claim 25, wherein the storage parameter is an encryption or replication level. The system of claim 25, wherein the storage parameter is a backup schedule. The system of claim 36, further comprising displaying the file properties on a dashboard. The system of claim 25, further comprising notifying a user of the preferred storage parameters. The system of claim 25, further comprising automatically using the preferred storage parameters to store the file. The system of claim 25, further comprising using the created preferred storage parameters to save the file upon receipt of the user consent. The system of claim 49, further comprising notifying the user that the application of the preferred storage parameters improves the cost of file storage. The system of claim 23, wherein the preferred storage parameters are created for a plurality of files, further comprising: notifying a user of the preferred parameters for a subset of the plurality of files, wherein the application of the preferred storage parameters would improve the availability, reliability, or memory cost of files.

Images