The Stonemont Quality KPI

What is KPI?

KPI stands for Key Performance Indicator, which is a type of performance measurement commonly used by an organization to evaluate its success of a particular function or activity.  It is common for a company to have several performance measurements or KPIs.  It is important that a company balance KPIs across several processes rather than have one overriding goal.  For example, if KPIs are centered only on production goals then several other worthwhile goals, such as quality, will likely be sacrificed resulting in unbalanced performance measures and hence unbalanced company performance.
 
Introduction
 
Stonemont software is widely used across the construction material industry to manage quality information.  Hence, it made sense to develop a Quality KPI that could be used effectively by aggregate, asphalt, and concrete producers.  The Stonemont Quality KPI is intended to provide an indication of quality; a single value that can be used to quickly assess or compare quality from the parameter level to the company level.  Our primary goal with the Stonemont Quality KPI is to provide an indication of future risk regarding conformance to specifications.  For this purpose we roll-up four (4) statistical measures.  Certain measures have a greater impact on the KPI than others.  In fact, one could question why we use these four measures when clearly some are more important than others.  In some cases, producers would rather rely entirely on conformance to specifications (CTS).  In other cases, producers would rather rely entirely on percent within specifications (PWS).  However, we decided to utilize these four statistical measures because we feel they provide more insight into quality when viewed together as a group than when viewed by themselves.  For those users that wish to only use PWS we provide a capability report that is similar to the KPI report.
Each of the performance measures are easily computed and most utilize the mean and standard deviation of a given dataset.  The arithmetic mean (average) is the expected value or central tendency of a random variable, which in this case is our dataset of a quality parameter.  The standard deviation (SD) is a measure of variation or deviation around the mean (Figure 1).  A low standard deviation generally indicates less variation and hence more product consistency.  A higher standard deviation indicates more variation and hence less product consistency.  Although standard deviation can provide some insight into product consistency, it should be viewed relative to the specifications and the ability of the process to produce material in specification.  For example, consider the same aggregate product that is produced at a traditional plant versus a fractionated plant.  In general, material at the fractionated plant may show a lower standard deviation and hence more product consistency.  However, if the material at the traditional plant is produced near the center of the specification while the material at the fractionated plant is produced near a specification boundary it may in fact be that the material at the traditional plant results in less risk of producing material out of specification.
 
Figure 1. Standard Deviation.
Each of these statistical measures (except CTS) comes with the standard statistical assumptions regarding normality, sample size, in control processes, etc.  The assumptions may not be valid in all cases and for all parameters.  However, in many cases they will provide assistance in indicating the risk of future non-conformance to specifications.  One of our reasons for using these performance measures for the Quality KPI is that the mean and standard deviation of datasets can be quickly and easily computed using standard database functions.  This is important when you consider large companies with hundreds of plants and thousands of products.  Although the Stonemont KPI is a quick tool that can be used by management as a relative measure of quality between products or plants, for example, it is no substitute for the routine use of run charts or statistical process control charts to properly assess the performance of each product/process.
The Stonemont KPI can be applied across the aggregate, asphalt, and concrete businesses and therefore includes the flexibility to assign different parameters to comprise the KPI for any given product.  For example, an aggregate producer may specify certain gradation sieves and possibly FM to be included in the KPI for a concrete stone.  An asphalt producer may consider gradation sieves and asphalt and voids content as KPI parameters for an asphalt mix.  A concrete producer would consider values such as compressive strength and possibly other field measured values like unit weight and slump as KPI parameters on a concrete mix.  KPIs are most useful when thought is put in regarding what parameters best represent the quality of a product. 
 
Performance Measures
 
PWS – PWS is equivalent to the Transportation Research Board (TRB) definition of PWL which is as follows: Percent within limits (PWL). The percentage of the lot falling above the LSL, beneath the USL, or between the LSL and the USL. [PWL may refer to either the population value or the sample estimate of the population value. PWL = 100 – PD].  PD stands for percent defects and is commonly 5 or 10 resulting in an acceptable PWL of 90 or 95 and greater.  The reason we refer to PWS rather than PWL is because we can compute this statistical measure against specifications (PWS), targets (PWT), and limits (PWL).
PWS is considered a “measure of choice” statistic by the FHWA, meaning that it is the recommended statistic to be used by agencies to measure the quality of pavement material and hence then acceptance of pavement material.  PWS is the most important measure in the KPI and can in fact be used by itself as a KPI.  We chose to include other statistics in the KPI because they lend insight into why a PWS may be low.  PWS is a type of variable acceptance procedure that is based on computed statistical parameters (mean and standard deviation). PWS uses the sample mean and the sample standard deviation to estimate the percentage of the population that is within the specification limits. It is similar to determining the area under the normal distribution curve (Figure 1).  A sample is a subset of the data taken from a larger population or process.  In other words, typically you can’t effectively measure the entire population so you measure a subset of the population and you use statistics to predict the quality of the entire process.  PWS is an estimate of the probability of conformance to the specification…or the capability of the process to produce material in specification.  As such, PWS can be used to estimate the future conformance to specifications as long as the process has not changed.
 
CTS – CTS stands for conformance to specifications and is a measure of the percentage of sample conformance to specification relative to all samples in the data set.  This is a type of pass/fail attribute acceptance procedure that is based on measures that are counted rather than computed.  CTS is simple to compute and simple to understand and is not subject to the same assumptions regarding normality as PWS.  However, attribute acceptance requires a larger dataset to achieve a similar efficiency regarding the estimate of conformance.  Another disadvantage of CTS is that it can easily be manipulated and therefore is not considered a strong predictor of future conformance.  So why did we include CTS in the Stonemont KPI?  The answer is that we have found that CTS is commonly used and understood in the industry and when used in conjunction with PWS can provide useful information.  In general, one would not expect CTS and PWS to significantly differ from one another if the dataset is large enough.  However, we have observed cases where PWS is 85% and CTS is 100% indicating that CTS may not be representative of the true estimate of the population conformance to specifications.  This can happen for reasons including the dataset being too small or failing samples being removed from the dataset.  Regardless of the reason, we consider large discrepancies an indicator of a potential problem that may warrant an investigation to better understand the cause of the discrepancy.  We consider a 10% difference between measured (CTS) and predicted (PWS) conformance to indicate a large discrepancy.  The KPI is weighted in favor of PWS rather than CTS as the difference between the two numbers increases, unless CTS is lower than PWS.  This ensures that a high CTS doesn’t bias the KPI.
 
Ppk – Ppk is a process capability ratio that will give an indication of where the process mean is located relative to the specifications.  Ppk is another type of variable acceptance procedure that is based on computed statistical parameters mean and standard deviation.  Ppk = 1: the mean is 3xSD away from the closest specification; Ppk = 0.67: the mean is 2xSD away from the closest specification; Ppk = 0.33: the mean is 1xSD away from the closest specification; Ppk = 0: the mean is on a specification; Ppk < 0: the mean is outside of specifications.  We include Ppk in the Stonemont KPI because where the mean is located relative to the specifications is the primary reason for a low PWS and because it is easy to understand.
The KPI uses a Ppk value of 0.67, which corresponds to the process mean being two (2) standard deviations away from the closest specification boundary.  Statistically this is nearly equivalent to 95% compliance.  Ppk values greater than or equal to 0.67 contribute 100% to the KPI.  Ppk values less than 0.67 contribute less than 100 % to the KPI.  Remembering these key Ppk values can provide important information regarding how your process is performing relative to specifications.  Values less than 0.67 indicate your closest specification is less than 2xSD away from the mean, which indicates that your risk of producing nonconforming material or defects is greater than 5% and therefore your KPI is lower.
 
Ppk/Pp – Pp is a measure of the spread of the specifications relative to the spread of the process.  Used by itself it doesn’t indicate where the process mean is located relative to specifications.  As a result, Ppk/Pp is used to give an indication of how off-center the process mean is located relative to the specifications.  In general, the more off-center the process mean, the higher the risk of non-conformance.  Ppk/Pp = 1: The mean is centered within the specifications; Ppk/Pp = .5: The mean is half way between the center of the specification and the closest specification; Ppk/Pp = .25: The mean is a quarter way from the closest specification relative to the center of the specification.  Values of Ppk/Pp greater than 1 are possible and indicate the process is off-center but isn’t a concern when the boundary specification on percent values is 0 or 100.  The KPI uses a Ppk/Pp value of 0.5.  Ppk/Pp >= .5 contribute 100% to the KPI.  Values of Ppk/Pp < .5 start to lower the KPI.  Ppk/Pp is not a critical measure of quality on its own.  It was included only because it provides a measure that indicated where the mean is relative to the specification.  Ppk/Pp is different than just Ppk because although Ppk may indicate that you are less than two standard deviations away from the closest specification, it doesn’t indicate whether you are near the center of the specification (indicating that your process isn’t capable of producing material at 95% conformance due to a large standard deviation or overly tight specifications); or if the mean is too far off-center to produce conforming material. 
 
Summary
 
Stonemont Software is widely used to manage aggregate, asphalt, and concrete quality control information and as a result we developed a Quality KPI that can be used across these industries. Many companies use KPI measures to evaluate performance for different sectors of their business.  These different KPI values can be combined into a single KPI value but more importantly they can be viewed individually to identify underperforming sectors of the business.  The approach we took with the Stonemont Quality KPI had a similar goal in mind in that it can be viewed as a single KPI value but that the components that go into the KPI provide additional information and insight into underperforming KPI values.  By presenting all of this information together, the KPI value is better understood.  Figures 2-7 show simulated dataset distributions with specification boundaries and 2xSD limit boundaries.  Figures 2-4 demonstrate how the KPI changes when a mean is centered relative to the specifications but standard deviation increases.  Figures 5-7 show how the KPI changes when a mean is off-center relative to the specifications for different standard deviations.  These charts show that as the standard deviation increases the risk of producing non-conforming material can increase but that this risk is much greater when the distribution mean is sufficiently off-center relative to the specifications.  Our primary goal with the Stonemont Quality KPI is to provide an indication of future risk regarding conformance to specifications.  Note that the Stonemont KPI is useless on small datasets and should not be used for daily or even weekly process control, which is the intended function of run charts and control charts.  However, the Stonemont KPI is a measure that producers can use to improve their processes by helping to identify material that has an increased risk of non-conformance.
 

Figure 2.
Centered mean with a low standard deviation.
 
Figure 5.
Off-center mean with a low standard deviation.
 
Figure 3.
Centered mean with a moderate standard deviation.
 
Figure 6.
Off-center mean with a moderate standard deviation.
 
Figure 4.
Centered mean with a high standard deviation.
 
Figure 7.
Off-center mean with a high standard deviation.
 

 

Adrian Field

 

Facebooktwitterredditlinkedinmail