Solving real life Multi-Criteria Decision Making problems using Criteria Analysis Technique

Systematic choice or ranking approach

Multicriteria decision making

Very often we need to choose something we desire from a few promising choices. While renewing your wardrobe, you need to

  • decide first which type of new dresses you would require and then,
  • actually buy the clothing items from the market.

Overall objective of renewing your wardrobe was not very well defined. You only felt that some of your clothing items are old and not anymore you would like to use them as also you would like to get hold of a few items that are in fashion and missing in your wardrobe.

At the first step you had inspected your available dresses and decided on the types and numbers of new dresses to buy by using your judgment and conditions of the items available with you. This is an important process of defining your target objective more specifically, and is a necessary step for any kind of detailed buying activity.

Thus we follow the steps in buying or choice cases:

  • Analyse and define target objective in as much detail as possible. For daily life buying or choosing cases, this may be a short step, though for organizational choice activity, this may be a long drawn out detailed step. Generally speaking this step of deciding what to buy is the first sub-step under the category of Problem definition. The better you define your problem before going into solving process, more are the chances of your success.
  • Knowing the type of items you would buy, now you think a bit and decide when and wherefrom you would buy the items. This is the initial information analysis step to further focus on the actual buying process. In daily life buying activities this won’t take a long time, But in organizational procuring and acquiring cases, a detailed analysis and decision making usually will be required. Generally prospective buying sources will be more than one. Though we are specifying a separate step for clarity, this in fact is a second sub-step in the category of Problem definition.
  • Choosing or selecting: This is the main thing you have to do now. You already know what to buy and wherefrom to buy. You arrive at a particular shop and start selecting (to keep things simple we assume your target shop to be only one).

Choosing or selecting activity is abundant in daily life, organizational life and also in academic life and form the most substantial portion of a very important class of problems known as Multi-Criteria Decision Making Problems (Or MCDM problems in short).

In an MCDM problem, you have thus three major items to consider,

  • Goal or objective: you have to analyze and decide on your goal or objective as precisely as possible. There are real life situations where objectives may be more than one. Such problems are classified as Multi-objective Multi-criteria Decision Making problems. This first step is important because if you are not too sure of your objective to start with, your chances of reaching the desired solution will be bleak.
  • Set of criteria or preferences: based on which you would select from the set of choices to fulfill your objective. You need to form this set of preferences as clearly as possible. Additionally the set of preferences should be comprehensive, should not be too large in number and most importantly, should not depend on each other. In other words, the set of evaluation criteria should be exhaustive and independent of each other as also should not be too many to make the evaluation process confusing. In real life, a golden rule is to select the few most important criteria – not too many. Already the problem is subjective and complex. With too many criteria, the choice evaluation may go out of hand altogether.
  • Set of Choices or possibilities: from which you would choose: in the same way that you identify the most suitable set of evaluation criteria, you would choose the promising set of choices exhaustively but not too large in number. The final evaluation of each choice against the set of evaluation criteria should be simple and clear enough to understand the process of selection as also the suitability of the final selection. To reduce the number of choices, in many cases an intermediate step or technique of short-listing is applied where the number of choices is large.

MCDM problem cases

Wardrobe renewal:

We have already touched the problem of renewing your wardrobe. As a first activity you need to decide what and how many to buy. This may not be the final decision but it is necessary to devote some time on examination of your existing wardrobe and analyzing the new requirements to form an idea of your requirements before you go to the shop to buy.

When in a shop, as you have already identified the choice types as also your preferences on choosing, you would go through a process of one or two stages of short-listing before choosing the items fulfilling your desire.

While going through the selection process, you may further refine your preferences and even modify your objective to some extent.

In real life problem solving, the steps may well be iterative, and not just sequential only. Thus the final choices may not be exactly as you have thought in the beginning, but nevertheless fulfilled your modified desire because while evaluating choices you went back and felt the need to change your objective and preferences a little bit.

The short-listing technique invariably introduces the important element of exhaustive approach by leaving no promising category unexamined as also speeding up the whole process. In short-listing, the important technique of elimination is applied. Lastly, you may visit a few shops and not just one, to finalize your purchases.

Selecting employees:

in any organization this is the job of HR department. Depending on the job specification, suitable candidates are selected out of many aspirants. If applicants are large in number, selection is carried out through a number of stages using elimination and short-listing techniques.

The final stage is always the most thorough in identifying evaluation criteria and the actual evaluation process.

Where creativity is the crucial desired ability, advanced organizations may ignore academic performance and directly judge the creativity of the candidates. Taking up a routine path, another organization with same requirement of creativity may eliminate candidates not having a minimum academic performance thus missing potential candidates with great creativity. This is interdependence of the real objective and identifying the criteria of choice. Without a clear understanding of this interdependence, a sub-par selection is probable.

Choosing a subject in undergraduate stage:

For a student, choosing a subject in her undergraduate academic stage of education is of paramount importance. Choice will depend not only on her interest in a subject but also on future prospects and availability of seats in a desired college.

The student needs to have a clear idea of what she likes to do finally in job situation to work back and select a set of options from which the final choice will be made.

Both the objective analysis and the criteria analysis against the available options will form the final set of criteria. Thus criteria analysis and identification is not really an activity independent from the objective analysis and option status.

Selecting a winner:

In selection and ranking type of problem solving, objective definition, evaluation criteria formation and final evaluation are in most cases highly subjective processes that are based on judgments or opinions of one or more than one person. Exceptions are few and can be found in the area of sports where the winner of a race takes the minimum time to complete the race. Selection of the single winner is thus based on a quantity (that of time to finish the race) having a numerical value. No judge determines this criterion. Thus the winner of a 100m race is the best performer in the activity area of the particular race.

Unfortunately even in this constrained environment, the winner of today may not win tomorrow. The fact of winning is valid for only that particular race. The result is bound tightly in the context of time and place.

Normal perception in case of examinations on a specific subject is to declare the examinee getting the highest marks as the best student. Unfortunately, the perception is quite misleading because the course content, the questions set, the teaching processes, the health condition on the day of the examination and many other factors determine the score. If the test is based on descriptive questions, judgments of the multiple examiners complicate the outcome further. To eliminate the subjectivity in examiner evaluation, Multiple Choice Question system was introduced that has its own set of pros and cons in judging the mental ability of a student.

In short, any statement declaring someone or something as the “best” is inherently misleading. Choosing a winner is at best an approximate process.

In daily life we may accept the approximations as a routine without being aware of the implications or consequences, but in organizations where critical decision making depends on goal evaluation, criteria analysis and choice evaluation in MCDM problem situations, the need for systematic and scientific methods to deal with the undependable personal judgments is sometimes considered crucial. Thus was born a new discipline “MCDM problem solving systems” which deviated from quantified optimization approach to qualitative judgment based evaluation approach in a systematic manner.

Among the few such mature systems, Analytic Hierarchic Process (or AHP in short) introduced by Prof. A L Saaty, is still the most popular and leading system used in many countries.

Case example of multi-criteria decision making problem solving

Saturday being an off-day for both of us, me and my wife decided to have a dinner out. We do not do it often but from our previous experiences we knew most of the places.

“Where to go?”, She asked. “Let’s go to a place with good ambiance”. I replied. “But what about food, won’t you like to have the best food?” She asked back. I smiled, “What’s that best food dear?” "Well, I mean very good eating experience.” She tried to explain.

“Let’s take stock. According to our overall joint preferences, we should go to one of the three places – Gourmet, Foodie or Chipmunk – agreed?” I made an attempt to short-list the choices.

I use this strategy many times when we need to reach a concluding point. This is Technique of Elimination coupled with Technique of affirmation.

From a large set of probable choices, it is much better to eliminate a large number of unsuitable choices to arrive at a small set of suitable choices based on preferences.

All the people concerned in the decision making though have to agree to the short-listed choices at this early stage of analysis. The best way to get an agreement (or disagreement) is to use the Technique of Affirmation, by forming the probing or concluding question in such a way that answer can only be a “yes” or a “no’. If my wife didn’t agree to my proposed short-list, I would have had to go back to the earlier stage of informal preference analysis once more.

I would have added or modified a few important preferences also possibly targeting a wider set of short-listed choices.

It is important to quickly form the first set of preferences and make a short-list on the basis of evaluation of these short-listed choices informally. This step enables us to focus on detailed analysis and evaluation based on preferences covering a small number of choices. If the number of choices were large, it would have been too costly in terms of time and other resources required for detailed analysis. This in short is a pragmatic approach.

As it happened, my wife agreed, “Yes these three places seem to be alright. But how costly are the places? I remember only vaguely about these restaurants – we should not blunder into the priciest one unknowingly.” She was trying to introduce another new criterion.

At this point I boldly suggested, “Let’s do a quick criteria analysis and include also the price consideration. This should take no more than 10 minutes time, I assure you. You would know your mind much better, you would find.”

Before she could react strongly I pulled the laptop towards us and in a few minutes produced the list of criteria in a spreadsheet. Sometimes you need to hurry on before your partner can react. This is a direct application of Inventive principle rushing through.

Short-listed Criteria

  • Ambience
  • Food quality
  • Price level

“Now tell me, would you like to add, delete or modify any of these?” I asked her.

It is better to go through a judgmental evaluation of the criteria at this stage to improve the quality of the final decision.

“Oh yes, I had forgotten, last time the place we went to was too noisy. As you are scientifically evaluating our choices this time, I should add noise level as a new criterion.” Though I could feel her hidden barb, keeping an impassive face, just like an analyst I replied with assumed authority, “But a place of good ambience shouldn’t be noisy, isn’t it? I mean, noise level, if at all to be used as a criterion should be put under ambience as a dependent criterion.”

I had two motives: first to limit the number of criteria and second to ensure that the short-listed criteria are independent of each other.

The second requirement is of paramount importance in any kind of multi-criteria analysis. The powerful method of AHP needs this requirement to be fulfilled.

From my earlier experiences of problem solving, I believed in this requirement. If my possible paths are dependent or linked with each other, confusion reigns supreme and it would be impossible for me to proceed along any path with any kind of certainty. It is called an Entangled situation. To some extent such a situation resembles a bowl of spaghetti with all strands enmeshed with all others. Separating strands from the whole entangled mass is tough.

For cases of reasoning and analysis, the same principle holds true. You must have possible paths, or broken down sub-problems or evaluation criteria independent of each other. This follows commonsense.

Thus, at this stage we should have a set of evaluation criteria that are independent of each other (or more technically, mutually exclusive of each other) and also be comprehensively exhaustive.

This principle is so important that a leading business consulting firm McKinsey Consulting reportedly coined a name MECE (if you give a name to something you attach more importance to it) and uses it unfailingly in all their analyses. We won’t though use this catchy name. Instead we would express it as application of Principle of interdependence and Principle of exhaustivity. Usually it is better to work from the more basic level.

Prof. Saaty was aware of this problem of criteria dependence and formed a modified and more complex method of Analytic Network Process or ANP to deal with interdependent criteria. It is more complex and we won’t go into it. We would rather try to ensure criteria independence.

Today my wife was quite clear in her reasoning and countered my attempt to reduce the set of criteria, “What would you say about a quiet, apparently classy but a little bit shabby place?” “Okay I agree. We delink noise level from ambience for the moment and include it as a separate criterion.” I wasted no time in argument. “I am happy that you have suggested it. I also feel comfortable in a quiet place.” A little praise should increase the interest level I thought.

Without any more delay, I formed the inter-criteria judgment matrix quickly.

 

  Ambience Food quality Price level Noise level
Ambience        
Food quality        
Price level        
Noise level        

 

What we did all this time was a detailed analysis of evaluation criteria exhaustively together with short-listing of favourable choices.

But under the given circumstances I felt the number of criteria to be too large for a quick evaluation. It is high time that I prune it. She should also be happy.

I said, “See dear, the criteria are too many for a quick evaluation. Let’s not consider the price level as a criterion this time. Further we consider that a place with good ambience should also be quiet. Let’s drop noise level also. Let’s focus on the two most important criteria – Ambiance and Food quality.” As expected, my wife readily agreed. “So, the pruned inter-criteria judgment matrix is somewhat like this”, I finished.

 

  Ambiance Food quality
Ambience    
Food Quality    

 

“Now let me ask you, between the two criteria which one is more important, and how much more important to you? Just think for a moment to answer.” I took the first step in putting judgmental values to the judgment matrix.

Without hesitation she came out with her response, “Well both are important, but I think food quality is more important – rather, somewhat more important but not extremely more important – than ambiance.” “Great, we will put the value of 1 to ambiance and 2 to food quality forming a ratio of 1/2 in the cell ‘Ambiance - Food quality’ and 2/1 the cell ‘Food quality – Ambience’ to indicate that Food quality is twice more important than Ambience. I happily went ahead and formed the inter-criteria judgment matrix quickly. After all, we already had reduced it to only a 2 by 2 matrix.

 

  Ambiance Food quality
Ambience 1/1 1/2
Food quality 2/1 1/1

 

This is where mathematics starts by assigning numerical values to our judgment on the relative importance between two and only two criteria. As recommended by Prof. Saaty in AHP, we can make fair judgments on relative importance between criteria only if we restrict the comparison between two criteria.

Pair-wise comparison is much more reliable than comparison between more than two things simultaneously.

To aid assigning numbers to the judgmental values of phrases such as “Equally important”, “Somewhat more important” or “Extremely more important”, Prof. Saaty devised an assignment table specifying values of 1, 2, 3....9 to the comparison phrases in a 9 point scale. Using this table I converted my wife’s judgment of 'Food quality is somewhat more important than Ambiance' to a value of 2 is to 1.

Evaluation of the judgment matrix will finally produce the inter-criteria importance in the form of a decimal number for each criterion the total of which will be 1. Effectively we would distribute the value of 1 between the criteria being judged. More the value share of a criterion is, more will be its importance while evaluating the choices against all the criteria. This decimal share is thus called the inter-criteria weight.

In this process we break down the judgmental process into smaller related judgmental problems all bound into a system.

Steps to produce the inter-criteria-weights

  • Step 1: Convert the ratio fractions in each cell to its equivalent decimal number.
  • Step 2: Sum up the total of the values in each row, thus forming a new column with heading ‘Row sums’.
  • Step 3: Sum up the row totals to produce a single total of all values in the matrix. We may call it ‘Grand total’.
  • Step 4: Divide each row total by the grand total to form a new column containing the results as ‘Weighted row totals’.

These values should total up to the value of 1 and finally should form the inter-criteria weights.

 

  Ambience Food quality Row totals Weight
Ambience 1 0.5 1.5 0.33
Food quality 2 1 3 0.67
    Grand total: 4.5  

 

Without going through further steps, because of the simplicity we straightway assumed these values to be our final inter-criteria weights.

“Your analysis is over?” My wife asked apparently relieved. She didn’t notice that we have not evaluated the relative importance of the choices against the criteria yet.

“Not yet. But it’s only a small step more. Please have patience. I will show you the Choice versus Criteria table and you will quickly put values for each choice against each criterion in no time using your judgment. Just take care to allot a decimal number to each choice – criterion cell so that the total of these values for the three choices against a criterion sums to the value 1. You have only to spell out six relative importance values now.” Grudgingly she looked into the new table and in no time spelled out six judgmental values.

 

Criteria Unweighted choice values
  Gourmet Foodie Chipmunk
Ambiance      
Food quality      

 

She considered that food in Gourmet was the best of all the three but its ambiance is a bit less than Foodie the second ranker. It was a quick but good decision.

Having all the values I needed, producing the final ranking values for the choices was a breeze for me.

 

Criteria weights Criteria Unweighted choice values Weighted choice values
    Gourmet Foodie Chipmunk Gourmet Foodie Chipmunk
0.33 Ambience 0.3 0.4 0.3 0.099 0.132 0.099
0.67 Food quality 0.5 0.3 0.2 0.335 0.210 0.134
        Final values: 0.434 0.342 0.233

 

“So let’s go to Gourmet to have our fill tonight.” Triumphantly I announced.

As it turned out ambience in Gourmet had improved a lot. Most importantly, its food was great. It was the priciest of the three and if we had considered the price level as a criterion of evaluation, Gourmet might have been relegated to the second place.

Back home she confessed. “All along I wanted to go to Gourmet...but it was so costly.” I smiled.

It is all a matter of judgment, you know.

 

This is a story that we cooked up for you and can take place anytime except the multi-criteria analysis part of course. After all, we don’t think any wife anywhere in her right mind would agree to undergo that kind of torturous experience on the eve of a much awaited evening out.

Coming this far though, we should not leave our analytical exercise incomplete. You may wonder, ‘Still something left out?’ Well, if you notice carefully, while evaluating we skipped a part of the method altogether.

The step missed

According to AHP, in this case example of ours, we should have formed one judgment matrix of dimension 3 by 3 for each criterion for evaluation of unweighted value (this is called utility value) for each choice. The method of evaluation of a inter-choice judgment matrix would have been the same as the method by which we had evaluated the inter-criteria weights from inter-criteria judgment matrix. For each criterion at this second stage, we have to evaluate one 3 by 3 judgment matrix, as number of choices compared against each other for the particular criterion is three.

Out of each inter-choice judgment matrix we would have got three values – for two criteria a total of six values.

Choice judgment matrix for criterion Ambience:

  Gourmet Foodie Chipmunk Row totals Relative utility
Gourmet         0.3
Foodie         0.4
Chipmunk         0.3
      Grand total:    

 

Instead of going through these formal steps, the hubby took a short-cut and straightway asked his wife to assign judgmental utility values to each choice against each criterion.

The last step is simple – we need to multiply the unweighted utility values by corresponding criteria weights and total up the weighted utility values for the choices. The choice with the highest weighted total utility value wins.

Being a pragmatist, the hubby used a hybrid method.

Any advantage in this rather involved method?

There is no advantage at all in following this popular MCDM method if you can easily make the choice using your own judgment framework and your experience. In fact that’s what we do in our daily life MCDM problem solving.

But yes, there can be big advantages of being very clear in your mind regarding your own preference framework and how much importance you actually attach to each of the preferences in evaluating the choices.

If you spend some time on criteria analysis alone, you would come to know your mind.

To us, criteria analysis is the most important component in any MCDM problem solving scenario.

In analyzing the relevant criteria, the best approach is to,

  • Exhaustively list all the criteria or preferences that you can think of. Here you need to apply the Principle of exhaustivity. Thinking exhaustively at this stage often brings out hidden and potentially important criteria. At this stage you must not think of inter-relationships or relative importance of the criteria and also how you would evaluate a choice against a criterion at all. You would focus wholly on what can all be the criteria relevant for the choice making.
  • In the second stage, it is time to consider possibilities of interdependence of the criteria carefully. If you find any, you can try to eliminate it by putting a dependent criterion as a sub-criterion of the parent criterion. This technique forms a hierarchy of criteria. For example, in our case story we could have considered breaking up the criterion Ambience into three second layer sub-criteria of Decor, Cleanliness and Noise level. But if we do that we would also have to distribute the weight 0.33 of Ambience suitably over its three sub-criteria using the judgment matrix again. During the choice evaluation in this case, we need to evaluate the set of three choices against these three sub-criteria rather than against the criterion Ambience. Thus there would be then 3 plus 1, that is four numbers of 3 by 3 choice judgment matrices at the last stage of evaluation.

Criteria-hierarchy

 

  • In the last stage if you want to skip the judgment matrices for choice evaluation, you may directly assign the utility values for each choice against each last level criterion for calculation of weighted utility values and final ranking values as a sum of weighted utility values for each choice. Otherwise, you may mentally evaluate the choices against the by now clear set of preferences.

In a number of daily life MCDM problem situations we have found that if we go through at least the activity of criteria analysis, our judgmental decision making tended to produce much better results.

Though it may seem to be overkill in daily life problems, popular MCDM methods such as AHP are extensively used from National Policy making to Defence related purchases in many countries across the world.

 

Note: abundant literature exists in the Web on MCDM methods specially on AHP.