Dynamic thresholds solve the issues associated with fixed thresholds since they can use behavior profiles describing the dynamic nature of normal behavior (e.g. how a variable is different at different times of the day, days of the week, and special days like the first of the month). Dynamic thresholds provide the basis for accurate real time alerts by providing an unlimited set of context relevant thresholds. The reason that dynamic thresholds aren’t in widespread use is that they can’t be calculated manually. For example to define the dynamic threshold for variable like DB2-locking-rate or CPU usage would require at least 8700 thresholds (one per hour) for a single system for single year. Just consider how difficult it would be to define the same for the hundreds and thousands of values of each system variable, manually.

While dynamic thresholds are much better than fixed thresholds, on their own they still aren’t accurate enough for performance alerts. Not every breach of a dynamic threshold is a reason for an alert – that is where performance models come into play. These models understand semantics of system and application performance, and can differentiate between a threshold breach that should generate an alert and a threshold breach that should be ignored in the current context. The best performance models are layered models (e.g. generic performance models, virtualization performance models, mainframe performance models) where the models are combined to provide the most accurate alerts possible for a specific installation.

Read more about this in our downloadable technical brief.

As Jeff points out in his comments to my last post – you would expect your cloud provider to provide an application level SLA protecting you against application performance issues caused by the infrastructure  - but they don’t. At best you’ll get a SLA covering  infrastructure reliability and robustness (at worse you won’t get even that)  - but you certainly won’t get an application performance SLA. If you think about it – that isn’t really surprising. The cloud provider is certainly using some type of virtualized shared infrastructure. That means unexpected application\resource contention and conflicts will happen - but the cloud infrastructure provider really can’t know about them, because they have no visibility into the application – and it is easy for them to assume that any problem with application performance is the application’s fault. So the only way to catch those problems is for you to infer them from your application’s performance anomalies.

So what can you do if you find an anomaly? The answer is use the cloud to obtain additional resources for the component that is acting anomalously and provide it with “capacity on demand”.  It will cost you extra, but it will solve the immediate problem (even the noisy neighbor problem) – and then go try and find a cloud provider that will  provide an application level SLA…

That blindness means that you don’t know if your apps are running on machine with 10 other apps, or if your VM has just been migrated to another physical server. In a perfect world that wouldn’t matter – but the world isn’t perfect. Your apps will be affected by their physical neighborhood – for example take the “noisy neighbors” problem that I mentioned in “Noisy Neighbors, Amazon Cloud and the Mainframe”. All of a sudden, through no fault of your own, your production applications may start acting erratically. So now you’ll need to understand that your performance and SLA problems may be caused by things that you can’t see, and can’t access – and will never be able to access.

So how do you manage something you can’t see or measure? The same way doctors and psychologists do it every day – but watching behavior and looking out for anomalies. Once an anomaly has been detected you need to look further and use benchmarks, best practices and models to correlate the visible behavior to the underlying causes. This capability will critical to performance management in the cloud. In my next post I’ll cover how to use behavioral information for cloud performance management.

First off – most enterprises aren’t going to move to a public cloud infrastructure, but rather to a private cloud. I know there is a lot of contention over what exactly a private cloud means – but I found a short description that I believe is right on the mark by James Urquhart - “Cloud computing is an application-centric operations model.” So for enterprise IT it means that applications become king – and the whole of the IT will be focused around serving applications and application users. Even though sounds almost intuitive to most business folks (I mean what else is IT except a way to get applications to users?) it isn’t how most IT departments operate. In most IT departments – infrastructure is king, not applications. You want to deploy a new application or you need more resources for an existing application – well then wait a few months for the infrastructure folks to requisition, provision, integrate and provide you those resources. The cloud will play havoc with that model. There is a reason for this mismatch in paradigms. It is because for the infrastructure folks, cloud or not,  compute and storage resources are not flexible or infinite – even though they may appear that way to the application folks. So how will  infrastructure operations manage in a “cloudy” world?

I think there is a lesson to be learned here from the mainframe. Sub-capacity pricing and capacity-on-demand have been around for a while. They essentially mean that organizations acquire over provisioned machines, but are charged only for the base capacity with the extra capacity available as needed  at a premium price. What this does is put a limit on the capacity available to applications not by technical complexity or bureaucracy, but rather by economics –  an application can get the unplanned extra capacity it needs, but it will be much more expensive then planned capacity. I expect cloud infrastructures (especially private clouds) to move to a similar model.

This means that both the IT infrastructure folks  and the application owners will need a good baseline of what resources an application actually uses. The infrastructure folks need to statistically aggregate that information across all applications to make sure that it has the reosurce  buffers needed to handle extra demand, and the application ownerswill needs  to ensure that there is cost justification for the extra capacity requests (and isn’t caused by something other than actual user demand).

Today many IT departments use peak capacity to define thier baselines. In a cloudy world that will be just too expensive, behavioral performance analysis is the only way to really understand what these baselines should be.

Distributed operations management teams are barely able to keep up with complexity of managing their own systems, and now with mass virtualization and the cloud coming down the pipe things will only get worse. The same hold true for the ever shrinking mainframe operations staff which have been struggling with the issues of managing a highly transactional virtual infrastructure forever. It isn’t that they aren’t capable – there is just too much data, too much complexity and too much interdepency for a human to manage. So the solution can’t be to glom the two together on a shared dashboard creating another layer of complexity for human operators and technical support (unless they also plan to use genetic engineering to create a superman operator!

The only viable solution to converged mainframe and cloud performance management is an automated APM brain that can do the heavy lifting of aggregating and filtering the reams of performance data and distilling it into human friendly, actionable information. Anyone that has been to a large data center knows there two types -those that regularly ignore alerts(caused by thresholds set too tightly) and ignoring real problems; or those with no alerts (caused by thresholds set too loosely) and missing real problems. In many, many cases it is the users that catch production performance problems first – not IT.

I believe that using behavioral analysis and predictive analytics to create the “brain” needed analyze converged performance data is  the only way to really enable convergence of mainframe and cloud performance management.   We have been doing this at ConicIT  for the mainframe (and discussing it here on our blog) – but believe that it is time to start applying the same technologies to the cloud, and that is squarely in our roadmap. That is the only workable way to converge mainframe and cloud performance management.

I think he hit a number of issues right on the money. One piece he missed is the need for good “brain” that will manage the automation behaviors he mentions (to be honest I think we are a long ways off from automated problem resolution – the first steps will need to be much better automated problem identification and guided manual problem resolution).

Rules aren’t the solution – there needs to some level of self-learning and behavioral analytics, otherwise the uncertainty level of whether to take an action, or what action to take is too high. This “brain” also needs to link SLA management to cost (for cost aware SLA management). The sharing and dynamic infrastructure capabilites that virtualization provides didn’t exist before (except on mainframes) and they are a double edged sword, since adding more virtual capacity to blindly meet SLAs can be costly. If SLA’s don’t become cost aware I forsee the same kind of cost overruns as with the Mainframe capacity-on-demand and sub-capacity pricing models.

The Gartner report and Bernd’s blog made me noodle about the fact that APM needs to get a lot smarter before it can get to the nirvana of  automated problem resolution. We are a long way from there - though it is clear their is a burning need for semi-automated problem identification at a minimum - since there is just too much performance data flying around for humans to manage. Operations need help and I believe this is next frontier for APM (and especially APM and the Cloud). I am guessing we are about to see a round of consolidation in APM – with the larger players picking up capabilities that they lack – first to meet Gartner’s APM criteria, and then to add a brain:

1. Large monitoring players like HP, Opnet, Quest will need to start completing their APM stack through acquistion or partnerships with the niche APM players.

2. VMWare will start highlighting the ”brain” they acquired through Integrien, and BMC will actively position ProActivenet as their APM “brain”, resulting in the legacy APM players CA, ASG, IBM needing to acquire or partner for a “brain” for their APM – I am betting the first move is CA buying Netuitive. My other guess is that IBM will build something itself.

1. Anyone using a cloud is probably using a private cloud (by a wide margin)

2. Performance bottlenecks, capacity planning and storage- i\o issues are the top challenges for the cloud. I predict this will continue to be the case in the forseeable future. It also shows that companies are moving to production cloud environments (otherwise why care about performance so much)

Bottom line – production private clouds (with a dash of hybrid) are coming.  Performance and SLA\capacity management issues surrounding the cloud will be top of mind for IT.

The graph shows average utilization, peak utilization and capacity. There isn’t a lot of information in the report describing the graph. For starters- no surprise, utilization goes up on regular basis, with cyclical peaks at the beginning and end of the year. For some reason 2011 has much bigger increase at the beginning of the year than 2010 – maybe a lot more people paid taxes?? It reminds me of a complaint I hear from some of our customers – “I don’t have very many new customers – but somehow I need to keep increasing capacity…”.

But what really struck me was how well the capacity tracks peak utilization – it looks like exactly the right amount of capacity was added at exactly the right time to manage new peaks. I am guessing this either happened because someone in capacity planning is clarvoyant, or because they employed some workload capacity cap. If they are using a capacity cap – then it would seem that at least some of the workload isn’t time critical – I wonder if it would be possible to offload that work to times when the machine is under utilized?

The reason I like this graph is that it is a good coarse-grain  illustration of a raw “behavior graph” for the system. It measures only a single parameter (utilization) and at large time intervals (a day) – but it does a good job of illustrating the value of a behavior graph. Just think of the value of being able to do that for hundreds of parameters (and synthesized parameters) that effect key performance indicators – and being able to that on a much finer time scale. The information would be overwhelming for a human (which is why we tend to distill it down into simpler graphs like the ones in the report) – but a machine could continually monitor and analyze those graphs – using the information to build behavioral profiles, and find anomalies.

That is the heart of what ConicIT does.

Until now most clouds in the enterprise have been used for development\QA and other non-production uses. As IT becomes more proficient and comfortable with the notion of private cloud computing, it will start being used for production – and that means taking applications and moving them over to a more automated, virtualized environment. So how do you prepare?

In many cases the only preparation that is done is to test existing applications in the new environment to make sure they run. The thinking goes that since the OS is the same, the application doesn’t need to be tested beyond that. That is a mistake – what is needed is a performance behavior baseline so that when a problem occurs in th enew cloud environment there is some hope of finding the real root cause.

So before you decide to move production to the cloud – make sure you really understand the behavior of the application,so that you have a performance behavior baseline to compare against when problems occur.

If you look closely at components of APM as Gartner describes them (end-user experience monitoring;     runtime application architecture discovery, modeling and display; user-defined transaction profiling;    component deep-dive monitoring in application context and analytics) they all make sense from a distributed system viewpoint – but not from a mainframe perspective, or a cloud perspective. Right now vendor focus is on getting more performance data (e.g. end-user experience data, end-to-end transaction data). My prediction – once cloud computing comes to the forefront in production systems (2-3 years), we will see another dramatic shift in the APM marketplace. The capture of data will be a commodity – just as it has become on the mainframe (on a mainframe there is no real technical reason to switch from TMON, OMEGAMON, SYSVIEW or Mainview). The future of APM will be all about intelligent APM – tools that can make sense out of commodity data provided by the current batch of APM tools.

• There is just too much data for humans to process – too many arms gathering data, not enough brains making sense of the data. The need for real time performance management is growing, and APM tools are generating all the performance data anyone could want (e.g. a complete profile of every single transaction in the system). At the moment this wealth of data is not helping – it is hurting performance management – alerts are ignored on a regular basis, reams of logs are never even looked at. The reason is that there is just too much data for a human to process. What we need are brains that distill the important essence from the sea of available performance data, and give it to IT in a way that provides complete, accurate, actionable alerts – along with domain specific root cause analytics.
• In a highly virtualized and shared environment (like a mainframe, SOA or cloud) – interactions between unrelated applications (at least from a user and business perspective) take onnew importance. The APM stack as defined by Gartner is very application oriented and seems to make the assumption that performance of an application is only dependent on the components of that application – not noisy neighbors or other interferences dictated by the application or deployment topology. Once virtualization becomes predominant in the data center again (either as a private or hybrid cloud) we will cycle back to the same performance issues problems that we see on mainframe. As I said – everything old is new again…
]]> http://conicit.biz/2011/10/19/application-performance-monitoring-apm-%e2%80%93-everything-old-is-new-again/feed/ 1