Images

Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
  • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
  • G06F11/16—Error detection or correction of the data by redundancy in hardware
  • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
  • G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
  • G06F11/2023—Failover techniques
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
  • G06F11/30—Monitoring
  • G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
  • G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
  • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
  • G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
  • G06F11/0793—Remedial or corrective actions
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
  • G06F11/30—Monitoring
  • G06F11/3055—Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
  • G06F11/30—Monitoring
  • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
  • G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
  • G06F11/30—Monitoring
  • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
  • G06F11/3466—Performance evaluation by tracing or monitoring
  • G06F11/3495—Performance evaluation by tracing or monitoring for systems
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
  • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
  • G06F11/16—Error detection or correction of the data by redundancy in hardware
  • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
  • G06F2201/82—Solving problems relating to consistency

Definitions

• Computing devices may include any number of internal components such as processors, memory, and persistent storage. Each of the internal components may operate with other components of the computing devices. For example, some processors store generated data in a persistent storage and may utilize capacity of the memory to perform computations.
• multiple computing devices may cooperate to accomplish a task.
• multiple computing devices may perform different computations that may be used, in turn, to generate a final result.
• a support engine for managing computing clusters in accordance with one or more embodiments of the invention includes a persistent storage and a processor.
• the persistent storage includes monitoring policies.
• the processor monitors a computing cluster of the computing clusters and identifies a potential component failure of the computing cluster based on the monitoring and the monitoring policies.
• the processor identifies an error state of the computing cluster; obtains solution level state information from the computing cluster based on the identified error state; generates a support package comprising the solution level state information; and initiates a support session by sending the generated support package to a support manager.
• a method for managing computing clusters in accordance with one or more embodiments of the invention includes monitoring a computing cluster of the computing clusters and identifying a potential component failure of the computing cluster based on the monitoring and monitoring policies. The method further includes, in response to identifying the potential component failure, identifying an error state of the of the computing cluster; obtaining solution level state information from the computing cluster based on the identified error state; generating a support package comprising the solution level state information; and initiating a support session by sending the generated support package to a support manager to correct the potential component failure.
• a non-transitory computer readable medium in accordance with one or more embodiments of the invention includes computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for managing computing clusters, the method includes monitoring a computing cluster of the computing clusters and identifying a potential component failure of the computing cluster based on the monitoring and monitoring policies. The method further includes, in response to identifying the potential component failure, identifying an error state of the of the computing cluster; obtaining solution level state information from the computing cluster based on the identified error state; generating a support package comprising the solution level state information; and initiating a support session by sending the generated support package to a support manager to correct the potential component failure.
• FIG. 1 shows a diagram of a system in accordance with one or more embodiments of the invention.
• FIG. 2 shows a diagram of an example support manager in accordance with one or more embodiments of the invention.
• FIG. 3 shows a diagram of an example solution manager in accordance with one or more embodiments of the invention.
• FIG. 4 shows a diagram of an example support engine in accordance with one or more embodiments of the invention.
• FIG. 5A shows a diagram of example monitoring policies in accordance with one or more embodiments of the invention.
• FIG. 5B shows a diagram of example actions of monitoring policies in accordance with one or more embodiments of the invention.
• FIG. 6 shows a diagram of a flowchart of a method of correcting a solution in accordance with one or more embodiments of the invention.
• FIG. 7 shows a diagram of a flowchart of a method of managing computing clusters of a solution in accordance with one or more embodiments of the invention.
• FIG. 8 shows a diagram of a flowchart of a method of selecting replacement components in accordance with one or more embodiments of the invention.
• FIG. 9A shows a diagram of a malfunctioning computing cluster.
• FIG. 9B shows a diagram of the computing cluster of FIG. 9A after corrective actions have been performed.
• FIG. 10 shows a diagram of a computing device in accordance with one or more embodiments of the invention.
• any component described with regard to a figure in various embodiments of the invention, may be equivalent to one or more like-named components described with regard to any other figure.
• descriptions of these components will not be repeated with regard to each figure.
• each and every embodiment of the components of each figure is incorporated by reference and assumed to be optionally present within every other figure having one or more like-named components.
• any description of the components of a figure is to be interpreted as an optional embodiment, which may be implemented in addition to, in conjunction with, or in place of the embodiments described with regard to a corresponding like-named component in any other figure.
• a solution may include one or more computing clusters. Each of the computing clusters may include computing devices. Each of the computing devices may include individual components.
• the solution may orchestrate the computing devices to perform predetermined actions to accomplish a goal.
• failure of any component of the solution may impact the performance of any portion of the solution.
• failure of a data storage device of a computing device may manifest itself as a decrease in the transaction rate of a database application hosted by a second computing device.
• a system in one or more embodiments of the invention, includes a support engine that monitors the components of a solution.
• the support engine may detect potential component failures based upon the decrease in performance of a portion of the solution.
• the detected potential component failures may be real or may be the manifestations of failures of other components.
• the support engine may automatically obtain solution level state information.
• the solution level state information may include various performance metrics regarding the overall functionality of the solution.
• the solution level state information may be used to determine a corrective action to be performed. In this manner, embodiments of the invention may quickly and automatically obtain all of the information that may be necessary to identify a component that has truly failed. Doing so reduces the amount of time required to identify a corrective action that actually remediates the failure of the component in the solution.
• FIG. 1 shows a system in accordance with one or more embodiments of the invention.
• the system may include a support manager ( 100 ) that manages a solution ( 115 ) by diagnosing problems with the supported solution ( 115 ) and performing corrective actions.
• the system may include a support engine ( 120 ) that obtains information regarding the state of computing clusters ( 130 ).
• the support engine ( 120 ) may provide the obtained information to the support manager ( 100 ) to enable it to perform its management functions.
• the support manager ( 100 ) may identify replacement hardware via a solution manager ( 110 ).
• Each of the aforementioned components may be operably connected by any combination of wired and wireless networks. Each components is discussed below.
• the support manager ( 100 ) diagnoses and remediates problems of the computing clusters ( 130 ) that may otherwise disrupt the functionality of the computing clusters ( 130 ). For example, computing cluster A ( 130 A) may report a decrease in the ability of one computing devices of the cluster to communicate with other computing devices of the cluster. In such a scenario, the support manager ( 100 ) may identify a problem with computing cluster A ( 130 A), identify a solution, and initiate corrective actions in computing cluster A ( 130 A) to address the identified problem.
• the support manager ( 100 ) may obtain state information of the computing clusters ( 130 ) from the support engine ( 120 ), or another entity. Similarly, when a hardware problem with a computing cluster is identified, the support manager ( 100 ) may identify a replacement component via the solution manager ( 110 ). For additional details regarding the support manager ( 100 ), See FIG. 2 .
• the support manager ( 100 ) is a computing device.
• the computing device may be, for example, a mobile phone, a tablet computer, a laptop computer, a desktop computer, a server, or a cloud resource.
• the computing device may include one or more processors, memory (e.g., random access memory), and persistent storage (e.g., disk drives, solid state drives, etc.).
• the persistent storage may store computer instructions, e.g., computer code, that when executed by the processor(s) of the computing device cause the computing device to perform the functions of the support manager ( 100 ) described in this application and/or perform all or portion of the methods illustrated in FIGS. 6-8 .
• FIG. 10 See FIG. 10 .
• the support manager ( 100 ) is a logical device.
• a logical device may be a virtual device that utilizes the computing resources of any number of computing devices to perform its functions.
• the logical device may be implemented as computer instructions, e.g., computer code, that when executed by the processor(s) of one or more computing devices cause the computing devices to perform the functions of the support manager ( 100 ) described in this application and/or all or portion of the methods illustrated in FIGS. 6-8 .
• the solution manager ( 110 ) identifies replacement hardware for components of the computing clusters ( 130 ). Due to underlying hardware or software requirements, each of the components may only be compatible with specific hardware components. Further, due to finite limitations, not all types of hardware may be available at all points in time. The solution manager ( 110 ) may identify replacement hardware for a computing cluster by identifying hardware that is both compatible with the computing cluster and available for deployment. For additional details regarding the solution manager, See FIG. 3 .
• the solution manager ( 110 ) is a computing device.
• the computing device may be, for example, a mobile phone, a tablet computer, a laptop computer, a desktop computer, a server, or a cloud resource.
• the computing device may include one or more processors, memory (e.g., random access memory), and persistent storage (e.g., disk drives, solid state drives, etc.).
• the persistent storage may store computer instructions, e.g., computer code, that when executed by the processor(s) of the computing device cause the computing device to perform the functions of the solution manager ( 110 ) described in this application and/or perform all or portion of the methods illustrated in FIGS. 6-8 .
• FIG. 10 See FIG. 10 .
• the solution manager ( 110 ) is a logical device.
• a logical device may be a virtual device that utilizes the computing resources of any number of computing devices to perform its functions.
• the logical device may be implemented as computer instructions, e.g., computer code, that when executed by the processor(s) of one or more computing devices cause the computing devices to perform the functions of the solution manager ( 110 ) described in this application and/or perform all or portion of the methods illustrated in FIGS. 6-8 .
• the support engine ( 120 ) monitors the computing clusters ( 130 ). Based on the monitoring, the support engine ( 120 ) identifies an error state of the computing clusters ( 130 ). The support engine ( 120 ) may perform monitoring based on instructions or configurations received from the support manager ( 100 ). For example, the support manager ( 100 ) may provide thresholds that when exceeded trigger the support engine ( 120 ) to obtain detailed state information from the computing clusters ( 130 ). The support engine ( 120 ) may provide the obtained information to the support manager ( 100 ). For additional details regarding the support engine, See FIG. 4 .
• the support engine ( 120 ) is a computing device.
• the computing device may be, for example, a mobile phone, a tablet computer, a laptop computer, a desktop computer, a server, or a cloud resource.
• the computing device may include one or more processors, memory (e.g., random access memory), and persistent storage (e.g., disk drives, solid state drives, etc.).
• the persistent storage may store computer instructions, e.g., computer code, that when executed by the processor(s) of the computing device cause the computing device to perform the functions of the support engine ( 120 ) described in this application and/or perform all or portion of the methods illustrated in FIGS. 6-8 .
• FIG. 10 See FIG. 10 .
• the support engine ( 120 ) is a logical device.
• a logical device may be a virtual device that utilizes the computing resources of any number of computing devices to perform its functions.
• the logical device may be implemented as computer instructions, e.g., computer code, that when executed by the processor(s) of one or more computing devices cause the computing devices to perform the functions of the support engine ( 120 ) described in this application and/or perform all or portion of the methods illustrated in FIGS. 6-8 .
• the computing clusters are physical devices that perform computations.
• Each of the computing clusters may operate independently to perform different functions or may be orchestrated to cooperatively perform a predetermined function. In such a scenario, each of the computing clusters may perform similar or different functions cooperatively to accomplish a shared goal.
• a first computing cluster may provide computing functionality while a second computing cluster may provide storage functionality.
• Any number of computing clusters may perform similar or different functions without departing from the invention.
• each computing cluster includes a number of computing devices.
• Each computing device may be, for example, a mobile phone, a tablet computer, a laptop computer, a desktop computer, a server, or a cloud resource.
• the computing device may include one or more processors, memory (e.g., random access memory), and persistent storage (e.g., disk drives, solid state drives, etc.).
• the persistent storage may store computer instructions, e.g., computer code, that when executed by the processor(s) of the computing device cause the computing device to perform the functions of the computing clusters ( 130 A, 130 N) described in this application.
• FIG. 10 See FIG. 10 .
• FIG. 2 shows a diagram of an example support manager ( 200 ) in accordance with one or more embodiments of the invention.
• the example support manager ( 200 ) may: (i) monitor the computing clusters, (ii) identify potential component failures, (iii) obtain solution level state information in response to identifying potential component failures, and (iv) initiate the performance of corrective actions to reduce the likelihood that the identified potential component failures will impact the functionality of the computing clusters.
• the example support manager ( 200 ) may include a session manager ( 210 ) and a persistent storage ( 220 ). Each component of the example support manager ( 200 ) is discussed below.
• the session manager ( 210 ) configures support engines to monitor computing clusters, obtains alerts from the support engines when a cluster is identified as having a potential component failure, obtains solution level state information in response to an identified potential component failure, and initiates performance of a corrective action.
• the corrective action may be a solution level corrective action, i.e., based on solution level state information, rather than a component level corrective action, i.e., based on component level state information.
• the session manager ( 210 ) may initiate a support session with the support engine.
• the support session may be associated with computing cluster, e.g., a triggering computing cluster, that exceeded a threshold or other trigger criteria.
• the session may be placed in a queue for analysis by the system. After being placed in the queue, the session manager ( 210 ) may obtain solution level state information from the triggering computing cluster.
• the solution level state information may include a solution type of the computing cluster, e.g., a predetermine function of the computing cluster.
• the solution level state information may include a hardware state of the computing device hosting, i.e., host computing device, the component that triggered the potential component failure.
• the hardware state may be a listing of the physical component of the aforementioned computing device.
• the solution level state information may include the hardware state of other computing devices of the triggering cluster that are operably connected to the host computing device.
• the solution level state information may include a software state of the host computing device, e.g., a listing of the applications, firmware, and associated settings for the applications and firmware of the host computing device.
• the solution level state information may include the software state of other computing devices of the triggering cluster that are operably connected to the host computing device.
• the solution level state information may include a transaction log of the host computing device, e.g., a listing of the actions performed over a predetermined period of time.
• the solution level state information may include transaction logs of other computing devices of the triggering cluster that are operably connected to the host computing device.
• the session manager ( 210 ) may determine that a solution level corrective action involves replacing of a physical component of a computing cluster. In such a scenario, the session manager ( 210 ) may send a request to a solution manager to identify replacement hardware. The request may specify the component to be replaced. The solution manager may respond by providing an identify of a replacement hardware that is available to facilitate the replacement of the component.
• the session manager ( 210 ) is a hardware device including circuitry.
• the session manager ( 210 ) may be, for example, digital signal processor, a field programmable gate array, or an application specific integrated circuit.
• the session manager ( 210 ) may be other types of hardware devices without departing from the invention.
• the session manager ( 210 ) is implemented as computing code stored on a persistent storage that when executed by a processor performs the functionality of the session manager ( 210 ).
• the processor may be hardware processor including circuitry such as, for example, a central processing unit or a microcontroller.
• the processor may be other types of hardware devices for processing digital information without departing from the invention.
• the session manager ( 210 ) may perform all or a portion of the methods illustrated in FIGS. 6-8 .
• the session manager ( 210 ) may utilize data structures stored in the persistent storage ( 220 ).
• the persistent storage ( 220 ) is a storage device that stores data structures.
• the persistent storage may be a physical or virtual device.
• the persistent storage ( 220 ) may include solid state drives, solid state drives, tape drives, and other components to provide data storage functionality.
• the persistent storage ( 220 ) may be a virtual device that utilizes the physical computing resources of other components to provide data storage functionality.
• the persistent storage ( 220 ) stores sessions ( 220 A), support engine settings ( 220 B), a software repository ( 220 C), and a knowledge base ( 220 D).
• the persistent storage ( 220 ) may store additional data structures without departing from the invention.
• the sessions ( 220 A) may be data structures that include information regarding support sessions, the support sessions may be ongoing or previously completed.
• the sessions ( 220 A) may include information obtained from a support engine used by the session manager ( 210 ) to identify a corrective action.
• the sessions ( 220 A) may include component state information and/or solution state information used by the session manager ( 210 ) to identify a solution level corrective action.
• Each session may be associated with a place in a support queue and a potential component failure that caused the session to be initiated.
• Each session may be generated by the session manager ( 210 ) when a support engine notifies the example support manager ( 200 ) that a component of a computing cluster has triggered a potential component failure.
• the support engine settings ( 220 B) may be a data structure that includes settings used by the session manager ( 210 ) to configure support engines. Specifically, the support engine settings ( 220 B) may be used to configure component policies of the support engines that are used to determined whether a component is potentially in a failure state. For additional details regarding component policies, See FIGS. 5A and 5B .
• the software repository ( 220 C) may be a data structure that includes copies of software used by the session manager ( 210 ) to initiate the performance of solution level corrective actions. For example, in a scenario in which the session manager ( 210 ) makes a determination that a solution level corrective action involves a software replacement, the session manager ( 210 ) may obtain a copy of the replacement software and/or software settings and send the copy to a computing device for implementation, i.e., replacing existing software and/or software settings.
• the software may be applications and/or firmware.
• the knowledge base ( 220 D) may be a data structure used by the session manager ( 210 ) to identify a solution level corrective action.
• the knowledge base ( 220 D) may map component level and/or solution level state information to solution level corrective actions.
• the mappings may be specified at any of granularity.
• the knowledge base ( 220 D) may be generated based on previous sessions. In other words, the contents of the knowledge base ( 220 D) may be generated heuristically.
• the knowledge base ( 220 D) may be automatically updated by the session manager ( 210 ) upon completion of a support session. In other words, the session manager ( 210 ) may generate a new mapping between component level and/or solution level state information to a solution level corrective action that resulted in the elimination of the potential component failure, i.e., restored the performance of the computing cluster.
• mappings of the knowledge base ( 220 D) are unconventional because the mappings assume that a potential component failure is not necessarily based on the component. Rather, the aforementioned mappings make an assumption that the identification of a potential component failure is merely a symptom of a solution level defect. Thus, the knowledge base ( 220 D) mappings are between a solution state and a corrective action, not necessarily a component state and a corrective action.
• data structures of the persistent storage ( 220 ) are illustrated as separate data structures, the aforementioned data structures may be combined with each other and/or other data without departing from the invention. Additionally, while the aforementioned data structures are illustrated as being stored on the example support manager ( 200 ), the data structures may be stored on persistent storage of other devices without departing from the invention.
• FIG. 3 shows a diagram of an example solution manager ( 300 ) in accordance with one or more embodiments of the invention.
• the example solution manager ( 300 ) may identify replacement components for computing clusters. In other words, the solution manager may identify whether a replacement component both exists and is available, i.e., in stock or otherwise obtainable.
• the example solution manager may include a replacement component identifier ( 310 ) that identifies replacement components in response to request for replacements and a persistent storage that stores data structures used by the replacement component identifier ( 310 ) to identify replacement components.
• a replacement component identifier 310
• Each component of the example solution manager is discussed below.
• the replacement component identifier ( 310 ) obtains requests from the support manager to identify a replacement component for a component.
• the replacement component identifier ( 310 ) may use the data structures stored in the persistent storage ( 320 ) to identify a replacement component.
• the data structure may enable a replacement component that both exists and is available to be identifier.
• the replacement component identifier ( 310 ) may notify the support manager of the identifier replacement component in response to the request.
• the replacement component identifier ( 310 ) may perform all or a portion of the method illustrated in FIG. 7 .
• the replacement component identifier ( 310 ) is a hardware device including circuitry.
• the replacement component identifier ( 310 ) may be, for example, digital signal processor, a field programmable gate array, or an application specific integrated circuit.
• the replacement component identifier ( 310 ) may be other types of hardware devices without departing from the invention.
• the replacement component identifier ( 310 ) is implemented as computing code stored on a persistent storage that when executed by a processor performs the functionality of the replacement component identifier ( 310 ).
• the processor may be hardware processor including circuitry such as, for example, a central processing unit or a microcontroller.
• the processor may be other types of hardware devices for processing digital information without departing from the invention.
• the persistent storage ( 320 ) is a storage device that stores data structures.
• the persistent storage ( 320 ) may be a physical or virtual device.
• the persistent storage ( 320 ) may include solid state drives, solid state drives, tape drives, and other components to provide data storage functionality.
• the persistent storage ( 320 ) may be a virtual device that utilizes the physical computing resources of other components to provide data storage functionality.
• the persistent storage ( 320 ) stores component to replacement mappings ( 320 A) and a replacement availability repository ( 320 B).
• the persistent storage ( 320 ) may store additional data structures without departing from the invention.
• the component to replacement mappings ( 320 A) may be data structures that maps components to replacement components. Over time, a particular component may not longer be produced and, consequently, become unavailable.
• the component to replacement mappings ( 320 A) may specify mappings between a component and all other types of components that are suitable replacements.
• the replacement availability repository ( 320 B) may specify the availability for each replacement component specified by the component to replacement mappings ( 320 A).
• the component to replacement mappings ( 320 A) may be used to identify all possible replacement components for a given component and the replacement availability repository ( 320 B) may specify the availability for each of the possible replacement components.
• the aforementioned data structures may specify the availability for each possible replacement component.
• the data structures of the persistent storage ( 320 ) are illustrated as separate data structures, the aforementioned data structures may be combined with each other and/or other data without departing from the invention. Additionally, while the aforementioned data structures are illustrated as being stored on the example solution manager ( 300 ), the data structures may be stored on persistent storage of other devices without departing from the invention.
• FIG. 4 shows a diagram of an example support engine ( 400 ) in accordance with one or more embodiments of the invention.
• the example support engine ( 400 ) may: (i) monitor computing clusters, (ii) identify potential component failures, (iii) notify a support manager of the potential component failures to start support sessions, (iv) collect solution level state information, (v) generate a support package including the solution level state information, and (vi) send the generated support package to the support manager to initiate the support session.
• the example support engine ( 400 ) may perform solution level corrective actions to address the potential component failures.
• the solution level corrective actions may be specified by the support manager.
• the example support engine ( 400 ) may include a cluster monitor ( 410 ) that performs both component level and solution level monitoring of computing clusters via cluster interfaces ( 420 ).
• the example support engine ( 400 ) may also include a persistent storage ( 430 ) that stores data structures used by the cluster monitor ( 410 ) when monitoring the computing clusters.
• a cluster monitor 410
• the example support engine ( 400 ) may also include a persistent storage ( 430 ) that stores data structures used by the cluster monitor ( 410 ) when monitoring the computing clusters.
• the cluster monitor ( 410 ) monitors the computing clusters based on monitoring policies ( 430 A) stored in the persistent storage ( 430 ).
• the cluster monitor ( 410 ) may monitor the operations of the computing clusters.
• the monitoring may be performed by, for example, periodically obtaining performance statistics of the respective components of the computing clusters.
• the performance statistics may be, for example, the compression ratio of data stored in the computing clusters, the transaction rate of applications executing on the clusters, the bandwidth between computing devices of the computing clusters, etc. Other performance statistics may be used without departing from the invention.
• the cluster monitor may initiate a support session with a support manager, perform solution level monitoring to obtain solution level state information, and perform solution level corrective actions.
• the cluster monitor ( 410 ) may perform all or a portion of the method illustrated in FIG. 6 .
• the cluster monitor ( 410 ) is a hardware device including circuitry.
• the cluster monitor ( 410 ) may be, for example, digital signal processor, a field programmable gate array, or an application specific integrated circuit.
• the cluster monitor ( 410 ) may be other types of hardware devices without departing from the invention.
• the cluster monitor ( 410 ) is implemented as computing code stored on a persistent storage that when executed by a processor performs the functionality of the cluster monitor ( 410 ).
• the processor may be hardware processor including circuitry such as, for example, a central processing unit or a microcontroller.
• the processor may be other types of hardware devices for processing digital information without departing from the invention.
• the cluster interfaces ( 420 ) may be operable connections between the example support engine and one or more computing clusters.
• the operable connections may be supported by any combination of wired and/or wireless networking.
• the example support engine ( 400 ) only supports a single computing cluster, only a single cluster interface may be present.
• the example support engine ( 400 ) may include any number of cluster interfaces without departing from the invention.
• the persistent storage ( 430 ) is a storage device that stores data structures.
• the persistent storage ( 430 ) may be a physical or virtual device.
• the persistent storage ( 430 ) may include solid state drives, solid state drives, tape drives, and other components to provide data storage functionality.
• the persistent storage ( 430 ) may be a virtual device that utilizes the physical computing resources of other components to provide data storage functionality.
• the persistent storage ( 430 ) stores monitoring policies ( 430 A).
• the persistent storage ( 430 ) may store additional data structures without departing from the invention.
• the monitoring policies ( 430 A) may be data structures that specify how the monitoring is to be performed and actions that are to be taken in response to the occurrence of a predetermined event identified by the monitoring.
• the monitoring policies ( 430 A) may specify that the monitoring is to be performed by monitoring a transaction rate of a database executing on a cluster and if the transaction rate falls below a predetermined threshold a solid state disk storing a cache used by the database is to be identified as having potentially failed.
• FIGS. 5A and 5B See FIGS. 5A and 5B .
• the data structure of the persistent storage ( 430 ) is illustrated as a solitary data structure, the aforementioned data structure may be combined with other data without departing from the invention. Additionally, while the aforementioned data structure is illustrated as being stored on the example support engine ( 400 ), the data structure may be stored on persistent storage of other devices without departing from the invention.
• FIG. 5A shows a diagram of example monitoring policies ( 500 ) in accordance with one or more embodiments of the invention.
• the example monitoring policies ( 500 ) may specify monitoring to be performed by a support engine and actions to be taken when predetermined condition occurs that is identified based on the monitoring.
• the example monitoring policies ( 500 ) may specify the policies at any level of granularity without departing from the invention.
• the example monitoring policies ( 500 ) include a number of entries ( 502 , 504 ). Each of the entries may specify a monitoring policy. Each entry may include an entity identifier ( 502 A, 504 A), a state ( 502 B, 504 B), and actions ( 502 C, 504 C).
• the entity identifier ( 502 A, 504 A) may be an identifier of entity to be monitored.
• the entity identifier ( 502 A, 504 A) may identify a component, a computing device, a computing cluster, or a solution, i.e., group of computing devices or group of computing clusters.
• the state ( 502 B, 504 B) may specify a condition of the component identifier by the entity identifier ( 502 A, 504 A) that, when met, triggers the performance of the actions ( 502 C, 504 C) of the respective entry.
• the state ( 502 B, 504 B) may be a condition such as, for example, a threshold or a range.
• the state ( 502 B, 504 B) may be other types of conditions without departing from the invention.
• the policy associated with entry may invoke the performance of the actions associated with the respective policy.
• the actions ( 502 C, 504 C) may be actions to be performed by the support engine in response to the conditions specified by the state being met.
• the actions may be, for example, the performance of solution level monitoring, obtaining of predetermined information from a computing cluster, modifying the operation of a computing cluster, or any other type of action.
• See FIG. 5B See FIG. 5B .
• example monitoring policies ( 500 ) are illustrated as a list of entries, the information of the example monitoring policies ( 500 ) may be stored in other formats, may include additional, less, and/or different information, and/or may be broken down into multiple data structures without departing from the invention.
• FIG. 5B shows a diagram of example actions ( 520 ) in accordance with one or more embodiments of the invention.
• the example actions ( 520 ) may specify one or more actions to be performed.
• the example actions ( 520 ) include obtaining of a hardware state of a component host ( 522 ), obtaining of a software state of the component host ( 522 ), obtaining of host settings ( 524 ), validation of the component ( 526 ), initiation of a new support session ( 528 ), and generation of a log ( 530 ).
• the component host may be the computing device that hosts the component that was identified as potentially failed.
• the validation may be to compare the functionality of the component to a baseline, e.g., performance test the component.
• the generated log may include all of the actions performed by a support engine and the response from a computing cluster.
• example actions ( 520 ) are shown as included a limited number and selection of actions that may be performed, the example actions ( 520 ) may include any quantity and type of actions without departing from the invention.
• the example actions ( 520 ) may include a triggering of another policy.
• the example actions ( 520 ) may cause a condition specified by a state of another action to be met and, thus, trigger the performance of another set of actions.
• monitoring policies may be nested and/or recursively trigger any number of monitoring policies upon the triggering of any one monitoring policy.
• FIG. 1 performs the methods of FIGS. 6-8 .
• FIG. 6 shows a flowchart of a method in accordance with one or more embodiments of the invention.
• the method depicted in FIG. 6 may be used to manage a computing cluster in accordance with one or more embodiments of the invention.
• the method shown in FIG. 6 may be performed by, for example, an support engine ( 120 , FIG. 1 ).
• Other component of the system illustrated in FIG. 1 may perform the method of FIG. 6 without departing from the invention.
• Step 600 computing clusters are monitored using monitoring policies.
• the monitoring may be accomplished by obtaining state information from the computing clusters and comparing the state information to the state policies.
• the monitoring may be performed, periodically, in accordance with a schedule, or via other regimes.
• the monitoring may obtain component level state information from components of computing devices of the computing clusters.
• the component level state information may be, for example, performance characteristics of the components such as, for example, a processing rate, bandwidth, or storage rate.
• the performance characteristics may be other types of characteristics without departing from the invention.
• Step 602 a potential component failure is identified based on the monitoring.
• the potential component failure is identified by comparing the obtain component level state information to monitoring policies. If the component level state information exceeds a threshold specified by the monitoring policies, the component associated with the component level state information may be identified as a potential component failure.
• Step 604 an error state of the computing cluster is identified in response to the identified potential component failure.
• the error state of the computing cluster is identified based on the potential component failure. For example, each type of component failure may be associated with an error state of the computing cluster. In contrast to traditional approaches that may focus on component level failures, embodiments of the invention may use the presence of a potential component failure as an indicated of a cluster level error state.
• Step 606 solution level state information is obtained from the computing cluster based on the identified error state of the cluster.
• the solution level state information is obtained by characterizing the performance of each component of each computing device of a cluster associated with the cluster level error state.
• embodiments of the invention may automatically obtain solution level state information in response to the identification of a potential component level failure.
• Step 608 a support package is generated that includes the solution level state information.
• the support package may include an identifier of the solution, an identifier of the computing cluster, identifiers of the computing devices of the computing cluster, identifiers of the components of the computing devices, the potential component failure, an identifier of the potentially failed component, and/or other information regarding the solution.
• Step 610 a support session is initiated by sending the generated support package to a support manager.
• Step 612 a solution level corrective action is obtained.
• the solution level corrective action may be obtained from the support manager or another entity.
• Step 614 the solution level corrective action is performed.
• the solution level corrective action is replacing the component with a replacement component selected based on the solution level state information.
• the replacement component may be identified by identifying a solution type of the computing cluster, e.g., data storage, processing, etc.
• a certified component e.g., a component that is known to be compatible with the solution type, may be selected.
• the selected certified component may be used as the replacement component.
• the solution level corrective action is modifying firmware associated with the potentially failed component.
• the modification may be a replacement of the firmware or a modification of the settings of the firmware.
• the modification may be obtained from a support manager.
• the solution level corrective action is modifying the firmware of a second component that is not the potentially failed component.
• the second component and the potentially failed component may be hosted by the same computing device.
• the solution level corrective action is modifying the firmware of a component of a computing device that does not host the potentially failed component.
• the method may end following Step 614 .
• FIG. 7 shows a flowchart of a method in accordance with one or more embodiments of the invention.
• the method depicted in FIG. 7 may be used to manage a computing cluster in accordance with one or more embodiments of the invention.
• the method shown in FIG. 7 may be performed by, for example, a support manager ( 100 , FIG. 1 ).
• Other component of the system illustrated in FIG. 1 may perform the method of FIG. 7 without departing from the invention.
• Step 700 a support package associated with a support session is obtained.
• the support package includes solution level state information.
• Step 702 the support package is analyzed using a knowledge base to identify a corrective action.
• the support package is analyzed by matching the solution level state information to similar information in the knowledge base. A corrective action associated with the matched solution level state information may be identified.
• the corrective action is replacement of the potentially failed component with a replacement component.
• the replacement component may be selected by a solution manager.
• the corrective action is a replacement of an application or firmware.
• the replacement may change a version of the application or firmware.
• the replacement may be reinstallation of an already installed version.
• the corrective action is a changing of setting of an application or firmware.
• Step 704 the corrective action is sent to a support engine associated with the support session.
• Step 706 an outcome associated with the corrective action is obtained from the support engine.
• Step 708 the knowledge base is updated based on the outcome.
• the knowledge base is updated by storing an associated between the solution state information and the outcome in the knowledge base.
• the method may end following Step 708 .
• FIG. 8 shows a flowchart of a method in accordance with one or more embodiments of the invention.
• the method depicted in FIG. 8 may be used to select a replacement component in accordance with one or more embodiments of the invention.
• the method shown in FIG. 8 may be performed by, for example, a solution manager ( 110 , FIG. 1 ).
• Other component of the system illustrated in FIG. 1 may perform the method of FIG. 8 without departing from the invention.
• Step 800 a replacement hardware identification request is obtained.
• the request is obtained from a support manager.
• the request may include an identifier of a potentially failed component.
• Step 802 replacement hardware associated with a component specified by the request of Step 800 is selected.
• the selected replacement hardware is any hardware that is certified to work with the solution which hosts the potentially failed component.
• Step 804 it is determined whether the selected replacement hardware is available. If the replacement hardware is available, the method proceeds to Step 806 . If the selected replacement hardware is not available, the method proceeds to Step 808 .
• Step 806 the selected replacement hardware is sent.
• Step 808 an alternative replacement hardware that is both certified and available is sent.
• the method may end follow Step 806 or Step 808 .
• FIG. 9A shows a diagram of a system including a computing cluster ( 900 ) that includes two computing devices ( 910 , 920 ).
• a malfunctioning solid state drive ( 912 ) may start to impact the performance of the computing cluster ( 900 ) that manifests in a deterioration of the performance of an interconnect ( 930 ).
• a support engine may identify that the interconnect ( 930 ) is potentially failing due to its reduced performance. In response to the identified potential component failure, the support engine may obtain solution level state information.
• the solution level state information may include the iops of the malfunctioning solid state drive ( 912 ).
• the solution level state information may be analyzed based on information in a knowledge base (not shown).
• the analysis shows that the malfunctioning solid state drive ( 912 ) is malfunctioning. Based on the analysis, it is determined that the malfunctioning solid state drive ( 912 ) should be replaced.
• FIG. 9B shows a diagram of the system of FIG. 9A after the malfunctioning solid state drive is replaced with a replacement solid state drive ( 916 ).
• a replacement solid state drive 916
• the malfunctioning solid state drive rather than component that facilitated the interconnect ( 930 ) such as the network interface cards ( 914 , 924 ), the performance of the system was restored without replacing component that were operating properly.
• FIG. 10 shows a diagram of a computing ( 1000 ).
• the computing device ( 1000 ) may include one or more computer processors ( 1002 ), non-persistent storage ( 1004 ) (e.g., volatile memory, such as random access memory (RAM), cache memory), persistent storage ( 1006 ) (e.g., a hard disk, an optical drive such as a compact disk (CD) drive or digital versatile disk (DVD) drive, a flash memory, etc.), a communication interface ( 1012 ) (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), input devices ( 1010 ), output devices ( 1008 ), and numerous other elements (not shown) and functionalities. Each of these components is described below.
• the computer processor(s) ( 1002 ) may be an integrated circuit for processing instructions.
• the computer processor(s) may be one or more cores or micro-cores of a processor.
• the computing device ( 1000 ) may also include one or more input devices ( 1010 ), such as a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device.
• the communication interface ( 1012 ) may include an integrated circuit for connecting the computing device ( 1000 ) to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, mobile network, or any other type of network) and/or to another device, such as another computing device.
• a network not shown
• LAN local area network
• WAN wide area network
• the computing device ( 1000 ) may include one or more output devices ( 1008 ), such as a screen (e.g., a liquid crystal display (LCD), a plasma display, touchscreen, cathode ray tube (CRT) monitor, projector, or other display device), a printer, external storage, or any other output device.
• a screen e.g., a liquid crystal display (LCD), a plasma display, touchscreen, cathode ray tube (CRT) monitor, projector, or other display device
• One or more of the output devices may be the same or different from the input device(s).
• the input and output device(s) may be locally or remotely connected to the computer processor(s) ( 1002 ), non-persistent storage ( 1004 ), and persistent storage ( 1006 ).
• the computer processor(s) 1002
• non-persistent storage 1004
• persistent storage 1006
• Embodiments of the invention may improve the performance of solutions that utilize distributed computations performed by computing clusters or other groups of computing devices. Embodiments may improve the performance of the solutions by improving the uptime of the solutions. Specifically, embodiments of the invention may provide a method of determining corrective actions based on solution level state information rather than state information of a component that is potentially failing. Doing so reduces the number of corrective actions that may need to be performed to repair the solution when a portion of the solution fails. For example, a failure of a component may manifest itself throughout a solution in unexpected ways and impact the performance of other components. Traditional approaches to repair solutions focus on relating decreases in the performance of individual components to corrective actions.
• Embodiments of the invention may prevent this problem by performing corrective actions that are based on solution level state information rather than component level state information. In this manner, a solution that is impacted by a malfunctioning or otherwise failed component may be corrected without needlessly replacing components, application, or firmware. Thus, embodiments of the invention may improve an uptime of a solution and decrease a cost of supporting a solution.
• one or more embodiments of the invention address the problem of detecting and correcting component failure in a distributed system. Since the failure of a component in a distributed system may cause unpredictable performance penalties, embodiments of the invention necessarily address problems that are rooted in computing technology. That is the identification of component failure and remediation of failed components in a distributed environment that might otherwise mask the true cause of a performance decrease of a distributed system.
• a data structure may include a first element labeled as A and a second element labeled as N.
• This labeling convention means that the data structure may include any number of the elements.
• a second data structure also labeled as A to N, may also include any number of elements. The number of elements of the first data structure and the number of elements of the second data structure may be the same or different.
• One or more embodiments of the invention may be implemented using instructions executed by one or more processors of the data management device. Further, such instructions may correspond to computer readable instructions that are stored on one or more non-transitory computer readable mediums.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• General Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Quality & Reliability (AREA)
• General Physics & Mathematics (AREA)
• Computing Systems (AREA)
• Computer Hardware Design (AREA)
• Mathematical Physics (AREA)
• Debugging And Monitoring (AREA)

Abstract

Description

Claims (20)

Priority Applications (1)

Applications Claiming Priority (1)

Publications (2)

Family

ID=68237867

Family Applications (1)

Country Status (1)

Families Citing this family (3)

Citations (163)

• 2018
  • 2018-04-24 US US15/961,237 patent/US11086738B2/en active Active

Patent Citations (169)

Non-Patent Citations (12)

Also Published As

Similar Documents

Legal Events