UNDP United Nations Development Programme ÈÑäÇãÌ ÇáÃãã ÇáãÊÍÏÉ ÇáÅäãÇÆí
Programme on Governance in the Arab Region ÈÑäÇãÌ ÅÏÇÑÉ ÇáÍßã Ýí ÇáÏæá ÇáÚÑÈíÉ POGAR
Publications: Judiciary

- Introduction

- Assessments
- Elements of an Overview Assessment
- Monitoring
- Evaluation
- Research
- Conclusions

- References
- Annex I: List of Basic Statistics for an Assessment
- Annex II: A Proposed Instrument for Assessing Judicial Operations

Assessments, Monitoring, Evaluation, and Research: Improving the Knowledge Base for Judicial Reform Programs
By
Linn Hammergren

Research:
Because judicial reforms have tended to become very routinized (a mixture of training, administrative restructuring, legal change, automation, ADR, and subsidized services for the poor), there is a danger that they will only lead to improving a second-best approach. They are very much aimed at making an existing process work better, but rarely address the question of whether that process represents the best use of resources in addressing high priority needs. They also have tended to take their impact on extra-system objectives (economic growth, equity, political stability, and citizen security) on faith because of the obvious difficulties of tracking extra-system changes. Better assessments, monitoring and evaluation are unlikely to tackle these broader issues. Reform, however radical it may propose to be, like every other human activity tends to follow patterns that have already been set. Whether or not training has ever accomplished any improvement, it is far easier to mount a reform program highlighting training than one excluding it.

Additionally, certain elements are added largely because they are popular –judges want new laws, equipment, training, and buildings. External assistance agencies tend to feature activities conventionally included in all their programs. And finally, there are the various preconceived notions about what is wrong with judicial systems which are again far easier to incorporate in a program than to contest. For all these reasons, knowledge building based only on actual reform programs is inherently limited. The full range of possibilities, whether of solutions or underlying causes, is unlikely to be explored because they have never become part of the standard reform repertoire.

Research is thus a vital fourth element in the knowledge management program. It can examine conventional understandings, test dominant hypotheses, and even attempt novel solutions which would not be possible in an ordinary reform. Research on judiciaries and judicial reforms is far from nonexistent, but as currently conducted suffers from some potentially serious problems:

  • Research itself is not very coordinated in the sense of pursuing a set of common themes and questions, building on prior work, or confronting differing findings.
  • Many researchers and most practitioners appear to lack a good overview of the relevant body of existing studies.
  • Research is rarely policy relevant in the sense of exploring themes which might improve reform programming.

The explanations for the first two phenomena largely lie in the absence of a well-established academic discipline focusing on the judiciary and its reform. Researchers come from a variety of disciplines. This in itself is desirable, but some of the negative side effects are that content, methodologies, and eventual dissemination are usually driven by the parent discipline’s priorities and that published articles are scattered through a vast number of journals, only a few of which focus specifically on legal and judicial reform. The fact that academic careers, access to funding, and publication depend on a disciplinary community less interested in judicial reform for its own sake creates two interlocking vicious circles which also explain a good part of the third observation. Researchers select topics, questions, and methodologies defined as important by their disciplines, and so tend to keep cross-disciplinary, thematically determined issues off the agenda, thereby guaranteeing they will continue to be seen as less significant. Practitioners’ inherently lesser interest in reviewing research is thus reinforced because much research is not relevant to their activities.

The cycles tend to affect even research sponsored by assistance agencies, which often seems more influenced by academic than practitioner concerns. For example, much of the recent work sponsored by the World Bank has involved the statistical analysis of relationships between aggregate indicators of judicial and economic performance. Its apparent purpose is to demonstrate the need for judicial reform, but beyond that its impact on reform content is dubious. Even discounting questions about concept validity [ 15 ], the further problem is in unpacking the indicators to understand the nature (and direction) of the relationships. Efforts to breech that gap by linking major variations in judicial structure or legal tradition to performance face similar criticisms. The construction of both the dependent and independent variables often involve some fairly arbitrary distinctions [ 16 ], but even if we accept their findings, the implicit recommendations (change to a common law system, or a parliamentary as opposed to presidential one) will not be easy to enact. While the grand distinctions (civil versus common law, inquisitorial versus accusatory criminal proceedings) may matter, the emphasis on tracking their impact fails to recognize that they are the least amenable to change. It also overlooks the potentially greater variations within the grand categories (e.g. between the inquisitorial justice system as practiced in Haiti and in France). Such within-category variations are intrinsically less interesting to academic researchers because they hinge on a multitude of differences of detail, most of which will not be statistically significant. The obvious conclusion is that if practitioners are going to benefit from research, they must help define the agenda and finance those studies most likely to have an impact on their own work. Inevitably these will involve more intensive comparisons of a limited number of cases. As they will be more costly to conduct [ 17 ], it is also important that the themes be selected with care.

As a start toward a new research agenda, the following general topics are suggested, ordered from least to most ambitious:

  • A review (possibly as cross-project single theme evaluations) of the activities (training, automation, law reform) common to most projects to identify variations in approaches, difficulties encountered, costs, and results. This is the most basic form of knowledge generation needed, and while it might not substantially alter the cafeteria approach to programming, it might reduce redundancy and increase efficiency of actions. Despite a recent aversion to the notion of best practices [ 18 ], at the level of individual activities, the concept still warrants consideration.
  • A review (via cross-project single theme evaluations) of progress in advancing the common subgoals (delay reduction, increased access, anti-corruption measures) found in most programs. While more complicated than the above (because it usually includes multiple activities), the value of this approach lies in its emphasis on affecting outcomes.
  • Research on the factors commonly blamed for poor judicial performance and giving rise to many standard reform policies. Among the high priority issues are salaries (for their impact on corruption, quality of judges, efficiency and efficacy), size and control of budgets (as related to quality of service), variations in appointment systems and criteria, tenure and lateral entry, variations in forms of judicial governance; and equipment and infrastructure [ 19 ].
  • Research on some of the overlooked candidate causes of poor performance – for example the role of the private bar and variations in its organization, power, or the sheer number of private professionals; the influence of educational institutions and patterns in legal training; the effects of judicial fees, prohibitions on pro se (self) representation, legal (il)literacy, and the types of services provided in limiting access to courts; the adequacy of and disincentives for using alternative and community dispute resolution mechanisms.
  • A simple examination of how courts and other sector institutions really operate and what incentive patterns and other constraints shape their output. Most reforms are based on uncontested conventional wisdom (there are delays, large amounts are commonly at stake, large users avoid the judiciary because they don’t’ trust it ad so on). Many of these do not hold up in fact and to the extent they can be replaced with a more accurate picture of what is occurring, overall reform planning will benefit.
  • Finally, because of the use of extrajudicial impacts as the principal justification for reform programs, more attention needs to go to exploring them. This is the most ambitious part of the research agenda, and the one least directly related to the immediate interests of reformers. However, it is critical to ensuring the value of their work. Here the current emphasis on aggregate, multi-country analysis can be abandoned in favor of more intensive exploration of the themes within one or several countries. The macro-studies are suggestive of where one might look; what is now required is an exploration of the specific linkages. If delay is critical, what kinds of delay and critical for whom and for what? What kinds of compensatory mechanisms are adopted and with what results? Do the answers vary across and within different types of legal systems and societies?

Donors will have to sponsor this research, but to the extent they can help build a body of relevant work they may also help legitimate it within the academic community. An obvious first priority is that they reorient their in-house research programs to coincide more closely with programming needs. A second is a larger research budget both for in-house and sponsored work. And a third is that research be reviewed and utilized. That is has not been so far may be a disguised blessing, but it is time to replace the two vicious circles with a virtuous one. If reform planners are forced to incorporate research findings in their own designs, they will pay more attention to what is proposed and what is delivered. When the results are, as is currently the case, contradictory [ 20 ], someone will have to resolve the differences.

____________________________
[15] These are legion. Most involve the judicial performance indicators which are usually based either on surveys or expert opinions. Even in a single country, there are question as to what is really being measured (public perceptions, the perceptions of a small group of citizens, experiences respondents are willing to report, and so on). However, cross-national comparisons are still more problematic because of the different perceptions of what is normal. One researcher using this approach notes the perplexing finding that residents of London reported more exposure to street crime than those of Caracas. Further interviewing revealed that the threshold for reporting incidents was far lower in England than in Venezuela. (Juan Jose Toharia in a presentation at the Law and Society Association meetings, Miami, 2000.) A 2000 Gallup poll on citizen satisfaction with government handing of crime is also illustrative. The second highest ranking went to Nigeria, the fourth to the Philippines, while France, the Netherlands, Finland, and Sweden received only half the positive responses of the two former countries.
[16] A series of studies (la Porta et al, 1998) focusing on differences between civil and common law systems have been criticized (Pistor, 1995) for overlooking the enormous variation within and cross fertilization between the two traditions. Another Bank study (Buscaglia and Dakolias) relating delay to such traits as computerization of courts, investments in infrastructure, and proactive judges raises many questions as to how the last three variables were constructed.as well as to the posited causal links. Courts which invest in computers may simply be courts interested in reducing delay, and it may be that interest, not the computers, which explains their success in shortening time to disposition.
[17] Aside from the disciplinary biases of the researchers doing macro studies (most of whom are economists) another explanation for their methodological focus is that they can use data bases that already exist. This means that they can do a study covering 80 countries for less than it costs to do original research in one.
[18] For some reason certain donors (most notably the World Bank) have become extremely critical of the term. Taken as blueprint solutions, the criticism may be well placed. However, as more flexible rules of thumb or basic principles for conducting activities, “best practices” are demonstrably effective. For example, we know that training is best done with a needs assessment; that the focus should be not on simple knowledge transfer, but rather on improving concrete practices and behaviors; and that repetition and complementary organizational changes substantially enhance training’s effect.
[19] Equipment and infrastructure often constitute the major part of investments in reform programs. Ad hoc arguments and some empirical work (Buscaglia and Dakolias) suggest an impact on performance. However despite (or because) of their popularity, critics have expressed doubts which are worth pursuing.
[20] For example two recent Bank publications found that salaries were 1) unimportant to and 2) highly predictive of levels of judicial corruption. The different sources of data easily explain the discrepancies but to my knowledge no one has even noticed them.

Top of this page