ݺߣ

ݺߣShare a Scribd company logo

APRESENTAÇÃO BALAS TESE (1) (1).PDF

•
•
A
Alexandre Bentes

This document describes a study that integrated the Balanced Scorecard and Analytic Hierarchy Process methodologies to assess the multidimensional performance of three organizational units within a Brazilian telecommunications company. The study assessed the units across four performance perspectives using nine indicators, and ranked the units based on manager judgments of the importance of the perspectives and indicators. The study aimed to test whether this integrated approach could provide a useful assessment of relative unit performance, while also validating the frameworks in a new industry and context.

1 of 30
Download to read offline
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 1 Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies in a Brazilian telecom company Alexandre Bentes, PUC-Rio, Brazil Jorge Carneiro, PUC-Rio, Brazil Herbert Kimura, Mackenzie, Brazil Jorge Ferreira da Silva, PUC-Rio, Brazil IAG Business School, Pontifical Catholic University of Rio de Janeiro March 24 – 26, 2010 Barcelona, Spain
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 2 Appropriate assessment of organizational performance is essential for firms The multidimensional nature of organizational performance poses challenges to researchers and practitioners The performance of functional areas within an organization is an instance of the concept of organizational performance in general Several studies have used BSC to assess the performance of firms and of functional units within organizations used AHP to rank decision alternatives But few have used both This is what is new and contributive in our study Introduction
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 3 Verify whether an integrated approach – that combines the multidimensional perspective of the Balanced Scorecard (BSC) with the framework for multi-criteria ranking of decision alternatives provided by the Analytic Hierarchy Process (AHP) – can be useful for the assessment of the (relative) performance of functional units within an organization. Extend the external validity of both the BSC and the AHP frameworks by applying those frameworks to another industry (telecommunications) and another setting (Brazilian environment and managers) not often researched Objectives of the study Multiple perspectives of organizational performance Multi-criteria ranking of decision alternatives Multidimensional assessment and ranking of organizational performance
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 4 In this study the overall performance of three functional areas of the financial department of a Brazilian Telecommunications company was rank-ordered by taking into consideration: the relative degree of importance of (four) distinct perspectives of organizational performance, the relative degree of importance of performance indicators within each perspective, and the (relative) performance of the (three) functional areas in each performance indicator Objectives of the study (cont.)
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 5 Organizational performance and the Balanced Scorecard (BSC) Comparison and decision-making based on multiple criteria The Analytic Hierarchy Process (AHP) Theoretical Framework
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 6 Organizational performance and the Balanced Scorecard (BSC) Organizational performance is a complex and multi-faceted phenomenon (Cameron, 1986; Chakravarthy, 1986; Venkatraman and Ramanujam, 1986) Kaplan and Norton (1996) proposed that organizational performance be simultaneously assessed from distinct, but complementary, perspectives, which were originally detailed as: Financial Customer Internal Business Processes Innovation and Learning Theoretical Framework Financial Vision and Strategy Innovation and Learning InternalBusiness Processes Customer
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 7 Organizational performance and the Balanced Scorecard (BSC) (cont.) While BSC provides a balance (Kaplan and Norton, 1992) between short-term and long-term objectives, between financial and non-financial measures, between lagging and leading indicators, and between internal and external performance perspectives” BSC approaches, nonetheless, also present some challenges: Overload of information Necessity to reach some synthesized judgment Theoretical Framework
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 8 Comparison and decision-making based on multiple criteria Some decision-making problems involve the choice among alternatives which must take into account multiple criteria which may present some degree of inconsistency with one another The values (performance) achieved in some criteria may suggest a distinct choice (decision) from the values in some other criteria Improvement of results under a given criterion may be obtainable only at the expense of another This may be particularly true in the assessment of organizational performance and the choice of the best performing unit (or the rank-ordering of the units under evaluation) Theoretical Framework
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 9 The Analytic Hierarchy Process (AHP) A multi-criteria decision-making tool developed by Saaty (1980) Decomposition, comparative judgment, and synthesis Decomposition of a complex problem into a multilevel hierarchic structure of objectives, criteria and sub-criteria, and alternatives Series of pair-wise comparisons of judgments (of child items below a parent node, i.e., the relative contributions of the child items to the parent node) Theoretical Framework Objective Criteria Sub-criteria Alternatives
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 10 The Analytic Hierarchy Process (AHP) (cont.) Synthesis in an eigenvector (priority vector) Judgments express the relative impact / importance of the elements in the hierarchy and are translated into numbers, which indicate the relative importance (weight, priority) of an element over another in the determination of their respective higher hierarchical level Final weights of any given lowest element are calculated from a weighted (by degree of importance) average over all levels of the hierarchy and synthesize the multiple judgments (often not mutually consistent) Rank-ordering of alternatives Overall importance represents the relative (percent) contribution of a given indicator with respect to the global goal Theoretical Framework
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 11 The Analytic Hierarchy Process (AHP) (cont.) Given the particularities of the present study, AHP has some advantages over other decision-making tools: Synthesis of multiple viewpoints (criteria) into a single unified result A consistency index can be calculated to assess the transitivity in judgments given by the paired comparisons Involvement of decision-makers who have to explicitly reveal their preferences Communication among decision-makers tends to lead to better decisions (reached through a process of conflict resolution until some consensus or, at least, agreement is reached) Theoretical Framework
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 12 Units of analysis: three functional areas of the financial department of a Brazilian telecom company – Fraud unit, Collection unit, and Revenue Assurance unit Data collection procedures Data collected in 2009 Interviewees and participants three senior mangers (one for each unit; all with long experience in the company), three supervisors (one for each unit), nine senior analysts (three for each unit), and the financial operations director. The diversity in hierarchical levels was deemed important in order to reach a more consensual and “department-wide” opinion about performance that would somehow better represent the variety of views in the financial department. Methods and data
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 13 Iterative and interactive process of data collection For the selection of dimensions and measures of organizational (functional) performance − four dimensions and nine performance indicators chosen For the definition of the weights (degree of importance) of each performance dimension and of each indicator (within each dimension) − Paired comparisons: How much more (less) important is this dimension / indicator as compared to the other one? Five rounds of discussion until some agreement was reached Role played by the investigator Convergent validity check Methods and data
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 14 Identify the best performing unit objective criteria Financial perspective Ind A Ind B Ind C Fraud unit Customer perspective Innovation and learning perspective Internalbusiness processperspective Ind D Ind E Ind F Ind G Ind H Ind I Collection unit Revenue assurance unit Fraud unit Collection unit Revenue assurance unit Fraud unit Collection unit Revenue assurance unit Fraud unit Collection unit Revenue assurance unit sub-criteria alternatives Methods and data The AHP tree
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 15 Training and capability-building activitiesI Improvement in employees’ capabilitiesH Innovation and Learning Number of new projects fully delivered in the year G Number of improvement projects implemented in the year F Internal Business Processes Satisfaction level of external clientsE Satisfaction level of internal clientsD Customer Decrease in operating costsC ROI of implemented projectsB Recovered value or avoided lossA Financial DefinitionIndicatorBSC Perspective Methods and data Indicators selected for the present study
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 16 Financial Ind A Ind B Ind C Ind A 1 4 7 Ind B 1/4 1 3 Ind C 1/7 1/3 1 0.71 0.21 0.08 Customer Ind D Ind E Ind D 1 1/7 Ind E 7 1 0.12 0.88 Processes Ind F Ind G Ind F 1 3 Ind G 1/3 1 0.75 0.25 Innovation and Learning Ind H Ind I Ind H 1 5 Ind I 1/5 1 0.83 0.17 CR = 0.03 CR = 0.00 CR = 0.00 CR = 0.00 Application of the Analytic Hierarchy Process Relative importance, Normalized weights, and Consistency ratios at the performance indicators (sub-criteria) level
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 17 BSC perspectives Finance Customer Processes Innovation and Learning Finance 1 5 4 5 Customer 1/5 1 3 3 Processes 1/4 1/3 1 2 Innovation and Learning 1/5 1/3 1/2 1 0.59 0.21 0.12 0.08 CR 0.09 Application of the Analytic Hierarchy Process Relative importance, Normalized weights, and Consistency ratios at the BSC perspectives (criteria) level
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 18 0.71 0.21 0.08 x 0.59 0.42 0.12 0.05 Indicators A,B and C = 0.12 0.88 x 0.21 0.03 0.18 Indicators D and E = local weights global weights local weights global weights 0.75 0.25 x 0.12 0.09 0.03 Indicators F and G = local weights global weights 0.83 0.17 x 0.08 0.07 0.01 Indicators H and I = local weights global weights Application of the Analytic Hierarchy Process From local weights to global weights of each indicator weight of parent node weight of parent node weight of parent node weight of parent node
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 19 Ind. A Fraud Collection Revenue assurance Fraud 1 1/7 1/2 Collection 7 1 5 Revenue assurance 2 1/5 1 0.09 0.74 0.17 Ind. B Fraud Collection Revenue assurance Fraud 1 4 3 Collection 1/4 1 1/3 Revenue assurance 1/3 3 1 0.61 0.12 0.27 Ind. C Fraud Collection Revenue assurance Fraud 1 1/4 1/2 Collection 4 1 4 Revenue assurance 2 1/4 1 0.13 0.66 0.21 CR = 0.01 CR = 0.06 CR = 0.05 Application of the Analytic Hierarchy Process Relative degree of success in (performance indicators of) the financial perspective by each functional unit
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 20 Ind D Fraud Collection Revenue assurance Fraud 1 1/2 3 Collection 5 1 7 Revenue assurance 1/3 1/7 1 0.19 0.73 0.08 Ind E Fraud Collection Revenue assurance Fraud 1 1/4 2 Collection 4 1 3 Revenue assurance 1/2 1/3 1 0.22 0.63 0.15 CR = 0.06 CR = 0.09 Application of the Analytic Hierarchy Process Relative degree of success in (performance indicators of) the customer perspective by each functional unit
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 21 Ind. F Fraud Collection Revenue assurance Fraud 1 1/2 1/5 Collection 2 1 1/4 Revenue assurance 5 4 1 0.12 0.20 0.68 Ind. G Fraud Collection Revenue assurance Fraud 1 1/2 1/6 Collection 2 1 1/3 Revenue assurance 6 3 1 0.11 0.22 0.67 CR = 0.02 CR = 0.00 Application of the Analytic Hierarchy Process Relative degree of success in (performance indicators of) the internal business process perspective by each functional unit
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 22 Ind. H Fraud Collection Revenue assurance Fraud 1 1/5 1/3 Collection 5 1 3 Revenue assurance 3 1/3 1 0.10 0.64 0.26 Ind. I Fraud Collection Revenue assurance Fraud 1 1/4 1 Collection 4 1 2 Revenue assurance 1 1/2 1 0.18 0.58 0.23 CR = 0.03 CR = 0.05 Application of the Analytic Hierarchy Process Relative degree of success in (performance indicators of) the innovation and learning perspective by each functional unit
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 23 Fraud 0.09 0.61 0.13 Collection 0.74 0.12 0.66 Revenue assurance 0.17 0.27 0.21 Ind. A 0.42 Ind. B 0.12 Ind. C 0.05 x Fraud 0.13 Collection 0.35 Revenue assurance 0.11 vectorial product Ind. A Ind. B Ind. C 0.59 = Fraud 0.19 0.22 Collection 0.73 0.63 Revenue assurance 0.08 0.15 Ind. D 0.03 Ind. E 0.18 Fraud 0.05 Collection 0.14 Revenue assurance 0.02 Ind. D Ind. E 0.21 x = vectorial product Finance Customer Application of the Analytic Hierarchy Process Partial contribution of each functional unit to the overall performance objective
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 24 Fraud 0.12 0.11 Collection 0.20 0.22 Revenue assurance 0.68 0.67 Ind. F 0.09 Ind. G 0.03 Fraud 0.01 Collection 0.03 Revenue assurance 0.08 Ind. F Ind. G 0.12 x vectorial product = Fraud 0.10 0.18 Collection 0.64 0.58 Revenue assurance 0.26 0.23 Ind. H 0.07 Ind. I 0.01 Fraud 0.01 Collection 0.05 Revenue assurance 0.02 Ind. H Ind. I 0.08 Processes Innovation and Learning x = vectorial product Application of the Analytic Hierarchy Process Partial contribution of each functional unit to the overall performance objective (cont.)
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 25 0.240.240.020.080.020.11 Revenue assurance unit 1.001.000.080.120.210.59Total Finance Customers Internal Business Processes Innovation and Learning TotalTotal Fraud unit 0.13 0.05 0.01 0.01 0.190.19 Collection unit 0.35 0.14 0.03 0.05 0.570.57 Application of the Analytic Hierarchy Process Final results for assessment of the best performing functional unit
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 26 Results indicate that the Collection unit performed better than the other units in three perspectives – finance, customer, and innovation and learning – and had the highest contribution (57%) to final objective. The Revenue Assurance unit performed better in the internal business processes perspective, but its contribution (23%) to the final decision regarding the (overall) best performing unit is not too different from the contribution of the Fraud unit (20%). Discussion
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 27 BSC explicitly incorporates several perspectives (besides the usual financial viewpoint) to organizational performance assessment Evaluating performance requires some agreement on the relative importance of performance perspectives and performance indicators in the way to reach some aggregated (synthesized) measure or ranking AHP handles multiple dimensions and measures with distinct degrees of importance and translates the overall result into a unified metric, explicitly accounting for incongruence among objectives and mutually inconsistent evaluations AHP circumvents the pitfalls of having managers use a simpler, ad hoc, weighing approach to make sense of the multiplicity of performance measures from a balance scorecard AHP provides more than an ordinal ranking; it also informs about the magnitude of the difference between alternatives, that is, how much more (less) a functional area is performing better (worse) than another Conclusions
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 28 AHP provides insights to managers as to which functional areas are more likely to contribute to overall success and which deserve more attention but all in relative basis; note that if all areas improve, some will still be (relative) underperformers…) so, both a relative and an absolute assessment should be conducted Results of the AHP + BSC can assist managers in deciding about: bonus distribution, incentive schemes identification (not covered in this study) of the possible reasons for poor (or good) performance Conclusions
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 29 Academic and managerial implications: Such an integrated approach (between BSC and AHP) was shown to have convergent validity with overall assessments by a senior director while providing much more fine-grained information for future managerial actions. The interactive and iterative process employed in this study has the additional advantage of leading managers to apprehend the diverse perspectives of performance assessment and to understand the possible trade-offs thereof. Conclusion
Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 30 Questions and comments are very welcome! Alexandre Bentes (alexvb@globo.com) Jorge Carneiro (jorgemtc@iag.puc-rio.br) Herbert Kimura (herbert.kimura@gmail.com) Jorge Ferreira da Silva (shopshop@iag.puc-rio.br)

More Related Content

APRESENTAÇÃO BALAS TESE (1) (1).PDF

  • 1. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 1 Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies in a Brazilian telecom company Alexandre Bentes, PUC-Rio, Brazil Jorge Carneiro, PUC-Rio, Brazil Herbert Kimura, Mackenzie, Brazil Jorge Ferreira da Silva, PUC-Rio, Brazil IAG Business School, Pontifical Catholic University of Rio de Janeiro March 24 – 26, 2010 Barcelona, Spain
  • 2. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 2 Appropriate assessment of organizational performance is essential for firms The multidimensional nature of organizational performance poses challenges to researchers and practitioners The performance of functional areas within an organization is an instance of the concept of organizational performance in general Several studies have used BSC to assess the performance of firms and of functional units within organizations used AHP to rank decision alternatives But few have used both This is what is new and contributive in our study Introduction
  • 3. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 3 Verify whether an integrated approach – that combines the multidimensional perspective of the Balanced Scorecard (BSC) with the framework for multi-criteria ranking of decision alternatives provided by the Analytic Hierarchy Process (AHP) – can be useful for the assessment of the (relative) performance of functional units within an organization. Extend the external validity of both the BSC and the AHP frameworks by applying those frameworks to another industry (telecommunications) and another setting (Brazilian environment and managers) not often researched Objectives of the study Multiple perspectives of organizational performance Multi-criteria ranking of decision alternatives Multidimensional assessment and ranking of organizational performance
  • 4. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 4 In this study the overall performance of three functional areas of the financial department of a Brazilian Telecommunications company was rank-ordered by taking into consideration: the relative degree of importance of (four) distinct perspectives of organizational performance, the relative degree of importance of performance indicators within each perspective, and the (relative) performance of the (three) functional areas in each performance indicator Objectives of the study (cont.)
  • 5. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 5 Organizational performance and the Balanced Scorecard (BSC) Comparison and decision-making based on multiple criteria The Analytic Hierarchy Process (AHP) Theoretical Framework
  • 6. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 6 Organizational performance and the Balanced Scorecard (BSC) Organizational performance is a complex and multi-faceted phenomenon (Cameron, 1986; Chakravarthy, 1986; Venkatraman and Ramanujam, 1986) Kaplan and Norton (1996) proposed that organizational performance be simultaneously assessed from distinct, but complementary, perspectives, which were originally detailed as: Financial Customer Internal Business Processes Innovation and Learning Theoretical Framework Financial Vision and Strategy Innovation and Learning InternalBusiness Processes Customer
  • 7. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 7 Organizational performance and the Balanced Scorecard (BSC) (cont.) While BSC provides a balance (Kaplan and Norton, 1992) between short-term and long-term objectives, between financial and non-financial measures, between lagging and leading indicators, and between internal and external performance perspectives” BSC approaches, nonetheless, also present some challenges: Overload of information Necessity to reach some synthesized judgment Theoretical Framework
  • 8. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 8 Comparison and decision-making based on multiple criteria Some decision-making problems involve the choice among alternatives which must take into account multiple criteria which may present some degree of inconsistency with one another The values (performance) achieved in some criteria may suggest a distinct choice (decision) from the values in some other criteria Improvement of results under a given criterion may be obtainable only at the expense of another This may be particularly true in the assessment of organizational performance and the choice of the best performing unit (or the rank-ordering of the units under evaluation) Theoretical Framework
  • 9. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 9 The Analytic Hierarchy Process (AHP) A multi-criteria decision-making tool developed by Saaty (1980) Decomposition, comparative judgment, and synthesis Decomposition of a complex problem into a multilevel hierarchic structure of objectives, criteria and sub-criteria, and alternatives Series of pair-wise comparisons of judgments (of child items below a parent node, i.e., the relative contributions of the child items to the parent node) Theoretical Framework Objective Criteria Sub-criteria Alternatives
  • 10. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 10 The Analytic Hierarchy Process (AHP) (cont.) Synthesis in an eigenvector (priority vector) Judgments express the relative impact / importance of the elements in the hierarchy and are translated into numbers, which indicate the relative importance (weight, priority) of an element over another in the determination of their respective higher hierarchical level Final weights of any given lowest element are calculated from a weighted (by degree of importance) average over all levels of the hierarchy and synthesize the multiple judgments (often not mutually consistent) Rank-ordering of alternatives Overall importance represents the relative (percent) contribution of a given indicator with respect to the global goal Theoretical Framework
  • 11. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 11 The Analytic Hierarchy Process (AHP) (cont.) Given the particularities of the present study, AHP has some advantages over other decision-making tools: Synthesis of multiple viewpoints (criteria) into a single unified result A consistency index can be calculated to assess the transitivity in judgments given by the paired comparisons Involvement of decision-makers who have to explicitly reveal their preferences Communication among decision-makers tends to lead to better decisions (reached through a process of conflict resolution until some consensus or, at least, agreement is reached) Theoretical Framework
  • 12. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 12 Units of analysis: three functional areas of the financial department of a Brazilian telecom company – Fraud unit, Collection unit, and Revenue Assurance unit Data collection procedures Data collected in 2009 Interviewees and participants three senior mangers (one for each unit; all with long experience in the company), three supervisors (one for each unit), nine senior analysts (three for each unit), and the financial operations director. The diversity in hierarchical levels was deemed important in order to reach a more consensual and “department-wide” opinion about performance that would somehow better represent the variety of views in the financial department. Methods and data
  • 13. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 13 Iterative and interactive process of data collection For the selection of dimensions and measures of organizational (functional) performance − four dimensions and nine performance indicators chosen For the definition of the weights (degree of importance) of each performance dimension and of each indicator (within each dimension) − Paired comparisons: How much more (less) important is this dimension / indicator as compared to the other one? Five rounds of discussion until some agreement was reached Role played by the investigator Convergent validity check Methods and data
  • 14. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 14 Identify the best performing unit objective criteria Financial perspective Ind A Ind B Ind C Fraud unit Customer perspective Innovation and learning perspective Internalbusiness processperspective Ind D Ind E Ind F Ind G Ind H Ind I Collection unit Revenue assurance unit Fraud unit Collection unit Revenue assurance unit Fraud unit Collection unit Revenue assurance unit Fraud unit Collection unit Revenue assurance unit sub-criteria alternatives Methods and data The AHP tree
  • 15. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 15 Training and capability-building activitiesI Improvement in employees’ capabilitiesH Innovation and Learning Number of new projects fully delivered in the year G Number of improvement projects implemented in the year F Internal Business Processes Satisfaction level of external clientsE Satisfaction level of internal clientsD Customer Decrease in operating costsC ROI of implemented projectsB Recovered value or avoided lossA Financial DefinitionIndicatorBSC Perspective Methods and data Indicators selected for the present study
  • 16. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 16 Financial Ind A Ind B Ind C Ind A 1 4 7 Ind B 1/4 1 3 Ind C 1/7 1/3 1 0.71 0.21 0.08 Customer Ind D Ind E Ind D 1 1/7 Ind E 7 1 0.12 0.88 Processes Ind F Ind G Ind F 1 3 Ind G 1/3 1 0.75 0.25 Innovation and Learning Ind H Ind I Ind H 1 5 Ind I 1/5 1 0.83 0.17 CR = 0.03 CR = 0.00 CR = 0.00 CR = 0.00 Application of the Analytic Hierarchy Process Relative importance, Normalized weights, and Consistency ratios at the performance indicators (sub-criteria) level
  • 17. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 17 BSC perspectives Finance Customer Processes Innovation and Learning Finance 1 5 4 5 Customer 1/5 1 3 3 Processes 1/4 1/3 1 2 Innovation and Learning 1/5 1/3 1/2 1 0.59 0.21 0.12 0.08 CR 0.09 Application of the Analytic Hierarchy Process Relative importance, Normalized weights, and Consistency ratios at the BSC perspectives (criteria) level
  • 18. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 18 0.71 0.21 0.08 x 0.59 0.42 0.12 0.05 Indicators A,B and C = 0.12 0.88 x 0.21 0.03 0.18 Indicators D and E = local weights global weights local weights global weights 0.75 0.25 x 0.12 0.09 0.03 Indicators F and G = local weights global weights 0.83 0.17 x 0.08 0.07 0.01 Indicators H and I = local weights global weights Application of the Analytic Hierarchy Process From local weights to global weights of each indicator weight of parent node weight of parent node weight of parent node weight of parent node
  • 19. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 19 Ind. A Fraud Collection Revenue assurance Fraud 1 1/7 1/2 Collection 7 1 5 Revenue assurance 2 1/5 1 0.09 0.74 0.17 Ind. B Fraud Collection Revenue assurance Fraud 1 4 3 Collection 1/4 1 1/3 Revenue assurance 1/3 3 1 0.61 0.12 0.27 Ind. C Fraud Collection Revenue assurance Fraud 1 1/4 1/2 Collection 4 1 4 Revenue assurance 2 1/4 1 0.13 0.66 0.21 CR = 0.01 CR = 0.06 CR = 0.05 Application of the Analytic Hierarchy Process Relative degree of success in (performance indicators of) the financial perspective by each functional unit
  • 20. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 20 Ind D Fraud Collection Revenue assurance Fraud 1 1/2 3 Collection 5 1 7 Revenue assurance 1/3 1/7 1 0.19 0.73 0.08 Ind E Fraud Collection Revenue assurance Fraud 1 1/4 2 Collection 4 1 3 Revenue assurance 1/2 1/3 1 0.22 0.63 0.15 CR = 0.06 CR = 0.09 Application of the Analytic Hierarchy Process Relative degree of success in (performance indicators of) the customer perspective by each functional unit
  • 21. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 21 Ind. F Fraud Collection Revenue assurance Fraud 1 1/2 1/5 Collection 2 1 1/4 Revenue assurance 5 4 1 0.12 0.20 0.68 Ind. G Fraud Collection Revenue assurance Fraud 1 1/2 1/6 Collection 2 1 1/3 Revenue assurance 6 3 1 0.11 0.22 0.67 CR = 0.02 CR = 0.00 Application of the Analytic Hierarchy Process Relative degree of success in (performance indicators of) the internal business process perspective by each functional unit
  • 22. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 22 Ind. H Fraud Collection Revenue assurance Fraud 1 1/5 1/3 Collection 5 1 3 Revenue assurance 3 1/3 1 0.10 0.64 0.26 Ind. I Fraud Collection Revenue assurance Fraud 1 1/4 1 Collection 4 1 2 Revenue assurance 1 1/2 1 0.18 0.58 0.23 CR = 0.03 CR = 0.05 Application of the Analytic Hierarchy Process Relative degree of success in (performance indicators of) the innovation and learning perspective by each functional unit
  • 23. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 23 Fraud 0.09 0.61 0.13 Collection 0.74 0.12 0.66 Revenue assurance 0.17 0.27 0.21 Ind. A 0.42 Ind. B 0.12 Ind. C 0.05 x Fraud 0.13 Collection 0.35 Revenue assurance 0.11 vectorial product Ind. A Ind. B Ind. C 0.59 = Fraud 0.19 0.22 Collection 0.73 0.63 Revenue assurance 0.08 0.15 Ind. D 0.03 Ind. E 0.18 Fraud 0.05 Collection 0.14 Revenue assurance 0.02 Ind. D Ind. E 0.21 x = vectorial product Finance Customer Application of the Analytic Hierarchy Process Partial contribution of each functional unit to the overall performance objective
  • 24. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 24 Fraud 0.12 0.11 Collection 0.20 0.22 Revenue assurance 0.68 0.67 Ind. F 0.09 Ind. G 0.03 Fraud 0.01 Collection 0.03 Revenue assurance 0.08 Ind. F Ind. G 0.12 x vectorial product = Fraud 0.10 0.18 Collection 0.64 0.58 Revenue assurance 0.26 0.23 Ind. H 0.07 Ind. I 0.01 Fraud 0.01 Collection 0.05 Revenue assurance 0.02 Ind. H Ind. I 0.08 Processes Innovation and Learning x = vectorial product Application of the Analytic Hierarchy Process Partial contribution of each functional unit to the overall performance objective (cont.)
  • 25. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 25 0.240.240.020.080.020.11 Revenue assurance unit 1.001.000.080.120.210.59Total Finance Customers Internal Business Processes Innovation and Learning TotalTotal Fraud unit 0.13 0.05 0.01 0.01 0.190.19 Collection unit 0.35 0.14 0.03 0.05 0.570.57 Application of the Analytic Hierarchy Process Final results for assessment of the best performing functional unit
  • 26. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 26 Results indicate that the Collection unit performed better than the other units in three perspectives – finance, customer, and innovation and learning – and had the highest contribution (57%) to final objective. The Revenue Assurance unit performed better in the internal business processes perspective, but its contribution (23%) to the final decision regarding the (overall) best performing unit is not too different from the contribution of the Fraud unit (20%). Discussion
  • 27. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 27 BSC explicitly incorporates several perspectives (besides the usual financial viewpoint) to organizational performance assessment Evaluating performance requires some agreement on the relative importance of performance perspectives and performance indicators in the way to reach some aggregated (synthesized) measure or ranking AHP handles multiple dimensions and measures with distinct degrees of importance and translates the overall result into a unified metric, explicitly accounting for incongruence among objectives and mutually inconsistent evaluations AHP circumvents the pitfalls of having managers use a simpler, ad hoc, weighing approach to make sense of the multiplicity of performance measures from a balance scorecard AHP provides more than an ordinal ranking; it also informs about the magnitude of the difference between alternatives, that is, how much more (less) a functional area is performing better (worse) than another Conclusions
  • 28. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 28 AHP provides insights to managers as to which functional areas are more likely to contribute to overall success and which deserve more attention but all in relative basis; note that if all areas improve, some will still be (relative) underperformers…) so, both a relative and an absolute assessment should be conducted Results of the AHP + BSC can assist managers in deciding about: bonus distribution, incentive schemes identification (not covered in this study) of the possible reasons for poor (or good) performance Conclusions
  • 29. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 29 Academic and managerial implications: Such an integrated approach (between BSC and AHP) was shown to have convergent validity with overall assessments by a senior director while providing much more fine-grained information for future managerial actions. The interactive and iterative process employed in this study has the additional advantage of leading managers to apprehend the diverse perspectives of performance assessment and to understand the possible trade-offs thereof. Conclusion
  • 30. Alexandre Bentes Jorge Carneiro Herbert Kimura Jorge Ferreira da Silva Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies 30 Questions and comments are very welcome! Alexandre Bentes (alexvb@globo.com) Jorge Carneiro (jorgemtc@iag.puc-rio.br) Herbert Kimura (herbert.kimura@gmail.com) Jorge Ferreira da Silva (shopshop@iag.puc-rio.br)