Barbara Befani. Barbara Befani -

Transcription

1 Barbara Befani International Workshop Case Studies in Development Evaluation: Validity, Generalisation and Learning University of Copenhagen May 21-23, 2012

2 My QCA Background + some references G. DE MEUR & B. RIHOUX (2002), L'Analyse Quali- Quantitative Comparée (AQQC-QCA). Approche, techniques et applications en sciences humaines, Louvain-la-neuve: Academia-Bruylant H. GRIMM & B. RIHOUX (eds.) (2005) Innovative Comparative Methods for Policy Analysis: Beyond the Quantitative-Qualitative Divide, Springer B. BEFANI, S. LEDERMANN & F. SAGER (2007) Realistic Evaluation and QCA: Conceptual Parallels and an Empirical Application, Evaluation 13(2) F. SAGER & C. ANDEREGGEN (2012) Dealing With Complex Causality in Realist Synthesis: The Promise of Qualitative Comparative Analysis, AJE current issue

3 Outline Application of QCA Context of the study Research Questions and Design Introduction to QCA Preliminary findings What can QCA do for us? Validity, Generalization, Learning Software Tools and solutions Generalization: the logical cases (a.k.a. remainders, counterfactuals) Validity: the Fuzzy-Set option

4 Context of the Study EU Agriculture and Rural Development Fund (EARDF) European Network of National Rural Networks (NRNs) Identification and dissemination of good practices and innovation in RD and RD policy implementation Subgroup (Youth) National Prize called New Success Factors Fattori in Italian is an old word for farmers

5 National Prize New Success Factors 40 young farmers (self-perceived as successful) applied on a voluntary basis Beneficiaries of EARDF funds ( / ) 11 were considered ineligible by the Ministry Did not meet formally required standards 29 candidates, 28 interviews Success was defined as presenting a practice that was: Effective (see next slide) Sustainable (survives without public funding) Transferable (applies to other sectors / contexts)

6 What is a successful farmer? An effective practice meets standards of: Economic performance Environmental performance Use of human resources Innovation (product / process / market analysis) Quality (typical / healthy products) Impact on local networks of actors / institutions A score interval for each dimension / sub-dimension Final ranking based on sums of scores

7 Research Design What makes young farmers successful? What is the role played by RD policy, if any? Explaining Factors The linear way: measure their contribution to the outcome; average, fixed, depends only on the coefficient (independent of other factors), proportional to factor quantity The comparative way: analyze combinations of factors and study the contribution of packages to the outcome; and of conditions within packages

8 Theory: Success Factors Education Family background Work of family members Production Supply of raw products, HR, quality of demand Local Competition (Perceived) Impact of EARDF funding Access to private funds (savings / bank loans) Cultural traits (economic behaviour) Attitudes toward work/achievement, frugality and prosperity, entrepreneurship, risk propensity, competition, innovation, advancement

9 Introduction to QCA Systematic Cross-Case Comparison Set Theory: a case is a set Of boolean / dichotomous conditions = 1 A b c D E f g h I = O Spot patterns / typologies of cases Visual representation Truth Table (Boolean matrix) Geometric representation Boolean minimization

10 Spotting patterns / typologies of cases Cases are complex, combine many different conditions Difficult to describe in their essential features Truth Table (a boolean matrix) Boolean Minimization (merged rows 3 & 4) (merged rows 3 & 5)

11 Minimization Procedure Case ONE: A*B*C*D Case TWO: A*B*c*d Case THREE: A*B*C*d Case FOUR: A*B*c*D A*B*C (merged Cases ONE and THREE) A*B*c (merged Cases TWO and FOUR) A*B (merged all four cases) (prime implicant)

12 B C c Case THREE Case TWO a D Case ONE Case FOUR A d b

13 Necessity & Sufficiency Let us assume the above 4 cases were all associated with a positive outcome (O) And the minimized expression associated with the negative outcome is A*b = o A*B is necessary for a positive outcome (O) In particular A is also necessary A*B sufficient, B sufficient But A by itself is not sufficient for a positive outcome Consistency and Coverage of (terms of) solutions Sufficiency & Necessity

14 o A O B A*B is necessary for O But also sufficient A is necessary But not sufficient for O b parsimonious Complex = A*B*c Intermediate = A*B intermediate complex Parsimonious = A

15 The INUS cause In previous case A*B was N&S for O Let s see what happens when we have: A*B*C = O A*B*c = o A*B*C is sufficient for O but A*B is not; it is C that makes A*B sufficient for O because if I take C away O becomes o C still not INUS because A*B*C is also necessary for O and because C is also sufficient in itself When we have what follows, C is INUS A*B*C = O A*B*D = O (this makes A*B*C unnecessary) A*B*c = o A*b*C = o (this makes C insufficient in itself)

16 Young Farmers: findings Two interview phases First mostly about performance The second mostly about success factors Delays, suspension of 2 nd phase Most data you will see here are fictitious It s what we have at the moment! Interviews are scheduled to be completed by September Publication of findings is planned for end of 2012

17 LEAST SUCCESSFUL MOST SUCCESSFUL The Truth Table Case ID Education Culture Family Background Private Funds Public Impact Local Comp Family Work Production

18 Spotting Patterns Sufficient Configurations for Most Successful (12 cases) (Consistency = 1) Most Successful? 1 Education Culture Coverage (10/12) 2 Education Culture Private Funds Coverage(9/12) 3 Education Culture Private Funds Public Impact Coverage (8/12) 4 Education Culture Private Funds Public Impact Local Competition Coverage (5/12) ~Local Competition Coverage(3/12) Sufficient Configurations for Least Successful (6 cases) (Consistency = 1) Least Successful? 5 ~Education ~Culture Coverage (6/6) 6 ~Education ~Production Coverage(6/6) 7 ~Culture ~Production Coverage(6/6) 8 Either of the above ~Public Impact Coverage(5/6) 9 Either of the above ~Public Impact Local Competition Coverage(4/6)

19 Spotting Patterns ~education*~culture*~production is necessary and sufficient for ~successful The three conditions are also singularly necessary BUT they are not sufficient Education*Culture is sufficient (but not necessary) for successful Neither culture nor education are sing. necessary for S Family Background is necessary (but not sufficient) for successful Public Impact is neither necessary nor sufficient

20 Spotting Patterns 2 Local Competition in itself seems irrelevant to success but it s interesting when seen as context Whenever I have LocalComp (competition), I also need Public Impact (public support to work) in order to succeed In a specific context Public Impact is necessary (INUS) In competitive contexts, if I take PubImp away, I become less successful LocalComp*PubImp => Success LocalComp*~PubImp => ~Success PubImp is also INUS in another context: lack of private funds. All successful cases which lacked private funds had PubImp When I have production problems family work and public impact are necessary. Are they INUS? NO because the combination is insufficient - in one case it leads to failure

21 The contribution of Public Funding: is it INUS? You can use QCA with a Pub condition = access to public funding, and have cases that lack this (~Pub) Is Pub INUS? Is it necessary, not in general (there will nonfunded paths to success) but within a particular set, context, combination of conditions? Such that, if I take public funding away, that same combination leads to failure instead of success? That s quite a statement to make about the importance, relevance, impact of public funding The ability to produce success instead of failure in a specific set of contextual conditions, to make a difference

22 Another example: the evaluation of Swiss EIA Outcome = Quality of EIA implementation Two most important conditions / mechanisms DEF = clarity of project definition DISC = whether a discussion among stakeholders had taken place at an early stage in the process QUA = DEF*SIZE*(MAN + SENS) + DISC*size qua = def*size + disc*size WHAT MAKES THE DIFFERENCE? When the project was big, clarity of project definition When the project was small, early discussion

23 What can QCA do for us? Learning We start from theory, we go back to theory Not necessarily just once: iterative We can change our model, the conditions we consider Increasing levels of clarity on what the cases tell us Validity Dichotomization might not be fine-grained enough Fuzzy-Set QCA (fsqca)

24 What can QCA do for us? Generalization / Synthesis Limited diversity, limited generalization No cases with all possible combinations of conditions What to do when you can only access the safe areas, when your sample is biased - for QCA this is natural : your sample has limited diversity Middle-range (parallel with Realistic Evaluation) Construct typologies: what can we say about a specific categories of cases? emergency / development Different degrees of decentralization Size, scope, budget, beneficiary range of Country Office operations Logical cases / remainders / counterfactuals

25 Is there an alternative to staring at the Truth Table? QCA for MS-DOS, fsqca for Windows, Tosmana Quine-McCluskey minimization algorithm Merging can happen in different ways Continuum from COMPLEX to PARSIMONIOUS No merging takes place Software tools All possible merging takes place Sometimes the simplest solution is not the most informative (not the most relevant info)

26 The logical cases (a.k.a. remainders) A solution to problems of limited generalization Frequently even the simplest solution is too complex Making assumptions on the outcome of specific theoretical configurations we don t have data about Specific because they need to help, not make matters worse! Transparency: QCA for MS-DOS and Tosmana list the needed assumptions Fun Fact: Ragin calls this counterfactual analysis!

27 Simplifying the solution in fsqca As far as I know, fsqca is not as transparent as Tosmana in listing the simplifying assumptions This is partly understandable because: with many conditions and few cases you may need hundreds, thousands standard procedure : obtaining three types of solution Parsimonious All logical cases that help are associated with the outcome Complex No logical case is associated with the outcome Intermediate Only a specific category of remainders is associated with the outcome: the easy counterfactuals A subset of the parsimonious and a superset of the complex solution Adding a redundant condition to a combination associated to outcome

28 Easy vs. difficult counterfactuals Assumptions on the outcome of given configurations If I know that A*B*C => O And I assume that D is also associated with O (would be redundant when combined with A*B*C Then A*B*C*D is an easy counterfactual Conversely If from theory or common sense I know that D is unlikely associated with the outcome A*B*C Then A*B*C*D is a difficult counterfactual Problem when I make statements on D I don t know in advance what it will combine with Causal power of D as a single condition

29 The Fuzzy-Set option Solution to Validity problems 0s and 1s are considered membership scores to a set 0 is fully out and 1 is fully in Also other scores: 0.17 (mostly but not fully out); 0.25 (more out than in); 0.33 (more or less out); 0.5 (crossover: neither in nor out); 0.67 (more or less in); 0.75 (more in than out); 0.83 (mostly but not fully in). Score to an intersection is MIN (scores to a condition) Score to a sum is MAX (scores to a condition)

30 The Fuzzy-Set option fsqca computes m. scores of cases to all possible combinations of conditions Clouds of points near a corner in the multi-dim space Configurations are selected on the basis of: having empirical cases in their corner having consistently sufficient cases Necessity & Sufficiency are analyzed with subset/superset analysis M.score to configuration mostly =< or >= m.score to the outcome

31 0,1 1,1 0,0 1,0 Number = number of cases in each square Coverage = number of cases having the outcome which are covered by that configuration; roughly an indicator of necessity

32 Membership to outcome 1 0 Membership to configuration 1 Consistency = proportion of consistent cases for each configuration Roughly an indicator of sufficiency

33 Advantages of QCA Multiple-Conjunctural Causality Causes are not easily modeled as factors that top-up (or take away from) the outcome consistently, proportionally to their magnitude An outcome can be produced by many different combinations of causes Each combination can be highly sensitive to even small changes (e.g. changes in only one cause in the package: INUS) CPPB: it seems you are on the right path to build peace, but all of a sudden something happens and you relapse into conflict! Sample size csqca is ideal for small n research (the 5-30 case no man s land) but can also handle large N samples fsqca is ideal for large N samples; it produces indicators of relevance, consistency, and coverage of configurations

34 Advantages of QCA (cont.) Any situation when cases need to be compared and synthesized (realist syntheses, syntheses of evaluations) rather than counting cases in which specific things happen QCA discovers patterns and typologies and asks how causes and factors combine Iterative: allows dialogue between theory and data (based on information-rich case studies), increasing levels of clarity (LEARNING) Qualitative data are transformed into Boolean Algebra and compared according to Set Theory (maths, even more rigorous than stats!) It brings together strengths of quali and quanti methods