ORGAPET Section B1:

Evaluating the Process of Programme Design and Implementation

Johannes Michelsen
University of Southern Denmark

Version 6, April 2008

B1-1      Introduction

The aim of this section is to identify issues and evaluation methods concerning the design and management of policy programmes such as organic action plans. These cover not just the content of programmes, but also the process by which the programme has been developed, resourced and prioritised. ORGAPET Section A5 also considers some of these issues, based on the MEANS approach (EC, 1999) and the Evalsed update. Some issues, including risks of implementation failure and internal and external conflicts and synergies, are also addressed in ORGAPET Section B2. The evaluation of stakeholder involvement in the design, decision-making, implementation and evaluation processes is considered in more detail in ORGAPET Section B3.

Evaluation is about assessing impacts or merits of a policy decision/programme (Vedung, 1997). Evaluation thus presupposes identification and description of a policy decision/programme. To serve the purpose of evaluation, the description must include intended aims, objectives or merits, the means selected to fulfil them, and the processes and organisations involved in realisation of intentions. Evaluation feeds back into political and administrative decision-making and may thus be seen as a separate stage of a policy process. This process begins with setting the agenda of policy problems and is succeeded by policy formulation and political decision-making, arriving at implementation as the last stage before evaluation and after evaluation, a feedback process may take place during which issues arising from evaluation may be brought back into the political agenda to start a new policy process (see Figure B1-1).

This model of the policy process implies that the content of the policy decision and the management of its realisation is decided and shaped in separate stages prior to evaluation. One part of evaluation is thus to reconstruct the background for and the current conditions under which the policy decision or programme in question operates. The reconstruction must be able to serve as basis for systematic evaluative analysis.

Figure B1-1: The policy process

There are deliberately no arrows pointing from one stage to the next, because the stages are not necessarily consecutive or separate from each other.

Source: Premfors (1979) and others

In some instances, it may be very easy to reconstruct the background and current conditions for a policy decision, where policy issues, decisions and actors involved in the process are clearly defined and easy to identify. One example is that of a policy which is decided after overt public or parliamentary debate, where various actors openly participate in clearly announced and defined processes of decision-making and implementation. However, policy-making is far from always undertaken in this way. Even when analysing a seemingly well-defined issue like that of organic action plans which might be expected to build on relatively clear, formal policy decisions, clarity may, in reality, be lost in narrow processes inside public administration. Here, it may appear very difficult to disentangle separate stages of agenda-setting, policy formulation, decision-making and implementation. It might even appear close to impossible to clearly identify relevant policy decisions, and implementation may even have taken place without any visible basis in policy decisions.

Returning to Figure B1-1, these insights draw attention to the fact that agenda-setting for an issue does not always lead to policy decision-making and implementation, and that not all policy decisions are actually implemented – or implemented properly. In fact, political and administrative needs to investigate actual impacts of policy decisions are one of the main historical reasons for the development of policy evaluation. The reconstruction of the background and current condition for a policy decision is thus an important tool for the evaluator when identifying what to evaluate, the goals to be assessed, the actors involved and the essential evaluation criteria involved in the analysis.

Separate theories and theoretical discussions have developed in attempts to analyse each stage of the model of the policy process, while attempts to analyse the policy process as a whole are much fewer and less developed. Together, Winter’s summary of implementation research (2003) and Vedung’s (1997) suggestions regarding process evaluation make it possible to point at elements in the policy process which are of major importance to the success or failure of a policy programme – and hence worth considering in an evaluation. For simplicity, the analysis here is split into only two main stages: policy design and decision on the one hand, focusing on the occasional aspects of policy-making – and implementation on the other hand, focusing on the continuous activities of public administration and its interplay with the groups targeted by policy decision, i.e. targeted by organic action plans. Section B1 ends with a discussion of issues relevant for analysing the implementation of the EU’s organic action plan in member states.

B1-2       Policy design and decision-making

Political scientists have ascertained that the content of any political decision depends on the way the issue appears on the political agenda, the policy options introduced to decision-makers, the general history of the policy area (how problems and solutions were perceived in the past), the level of political conflict and how conflict is coped with (Parsons, 1995). Each of these aspects may have direct implications for the content of the policy decision in terms of aims and measures and the expected causalities between them (Pawson and Tilley, 1997) or theory of action (EC, 1999: 82; see also Evalsed section on programme theory and logic models). A programme theory is an expression of how actions included in an action plan are expected to produce impacts in terms of linking specific causes to specific effects. An example of a programme theory is that a 10% increase in conversion payment rates to farmers will lead to a 25% increase in converted agricultural land. A more detailed discussion of programme theory with respect to organic action plans is contained in Section A3).

An obvious illustration of the need for clear programme theories relating to organic action plans is that, in some instances, promotion of organic food and farming appears on the policy agenda as part of the solution to agriculture’s environmental problems while, in other instances, organic action plans develop in response to consumer demand. Although promotion of organic food and farming may be the main purpose of policy action in both cases, political attempts to solve agriculture’s environmental problems implies preferences for policy options designed to increase the use of organic practices among farmers and the quantity of organically-grown areas, with only little consideration given to the issue of whether the additional production might fit into existing or future organic food markets. Conversely, political attempts to satisfy consumer demand may give preference to policy options which optimise the fit between types and quantities of produce supplied and demanded, without paying much attention to the consequences for the environment.

In addition, affected stakeholders (actors) may be very different in the two cases. In the case of solving agriculture’s environmental problems, the affected stakeholders may include ministries of agriculture and environment together with organisations of mainstream and organic farmers, and their political supporters. In the case of responding to consumer demand, the affected stakeholders may include ministries of agriculture, food and industry together with organisations of food suppliers and consumers, and their political supporters. The opposing views promoted in the policy process may vary similarly, such that environmental problems in agriculture may involve views that emphasise or deny agriculture’s environmental problems, while consumer demand may involve views that prefer either push or pull mechanisms on the food market. If decision- makers attempt to cope with these conflicts through consensus, policy decisions may be broadly accepted but, in addition, the content may be ambiguous. A conflictual approach to decision-making may, on the other hand, lead to clear decisions while acceptance may appear limited outside the circles of decision-makers. Ambiguity may also result from attempts to combine several purposes (such as concern for the environment, consumer demand, rural development) or several types of policy options in one policy decision/action plan. A further experience of political science is that initial perceptions of problems and solutions tend to design future perceptions (Thelen and Steinmo, 1992). Hence, if organic food and farming was introduced as part of environmental policy, then difficulties should be expected in changing the main direction of the policy towards market orientation later on.

The evaluator’s reconstruction of the background and current condition for the policy decision or programme to be evaluated thus involves collection of several types of data. The following long list is only intended to be indicative for issues to be dealt with in reconstruction of the background information. The most important issues to consider are the main points: prior policy initiatives, agenda-setting, policy formulation and decision-making, while all the sub-points are suggestions regarding how to deal with the main points. Not all sub-points are relevant in all situations and, in some situations, sub-points other than those mentioned might be more relevant.

·       The existence and content of any policy initiatives prior to the one to be evaluated, if any. Lampkin et al., (1999) includes a list of initiatives regarding support for organic food and farming where a main distinction may be made between prior political recognition of organic food and farming in terms of standards. In addition, the political orientation of prior policy initiatives are of interest, for example:

o       Promotion or hindrance of organic food and farming distribution

o       Policy affiliation: agriculture, environmental, health, consumer or other types of policy or mixes of policies.

·        Agenda-setting of the policy decision/action plan to be evaluated. Strength and orientation of actions leading to policy decision:

o      Possible themes include:

o       Policy affiliation: agriculture, environmental, health, consumer or other types of policy or mixes of policies

o       Forums for discussing the agenda: private/third sector organisations; public media; parliament (committee); government; public administration (committee); interplay between forums

o       Importance to public policy: importance in media and public discussions; importance to government; importance to policy sector (public administration and NGOs).

·        Policy formulation: policy options and programme theories presented for discussion:

o       Policy options (legal, financial, communicative, organisational or packages, combining more than one policy option)

o       Programme theories involved in policy options i.e. expressions of how actions included in an action plan are expected to link specific causes to specific effects.

·        Decision-making: who made the decision(s); what were the main discussion points; what was the content of the decision; what was the level of conflict?

o       Decision-makers: parliament; government; single minister; public administration (level) or other type – indication of decision’s authority

o       Discussion points: all or only parts of the policy options suggested – outcome of discussions

o       Content: final composition of objectives, policy options and implementation issues: ambiguity or clarity, coherence

o       Level of conflict among decision-makers: consensus vs. clearly opposed views (Michelsen, 2008a; 2008b), classified as:

Policy design and decision-making may involve varying interest groups and stakeholders (see Sections A4 and B3). By definition, stakeholders include both public and private interests involved in the policy field (EC, 1999 vol. 6:37). One type of policy-making is to only include public administration and public interest groups while keeping all or most private organisations out of all or most aspects of policy design and decision-making. Another type of policy-making is to involve selected private interest groups and organisations so much that the policy sector may be seen as 'captured' by these private interests. In many EU member states, farmers’ organisations were said to have captured agriculture policy until the 1990s. Hence, the type and extent of involvement of stakeholders – whether public or private - is an important issue in reconstructing the background and current condition for the policy decision/programme to be evaluated.

It is a general experience of policy analysis that stakeholders gaining early access in the policy process secure more influence on the final decision than those involved later on. Hence, it is important to know which stakeholders were involved earlier or later in designing and deciding organic action plans and what interests and views they promoted. In addition, it is important to know the level of conflict involved in the policy process – for instance, whether stakeholders of mainstream and of organic food and farming consider themselves antagonists or co-operators, and are considered so by other – not least public – stakeholders (see the above distinction of four levels of conflict (Michelsen, 2008a; 2008b)). The analysis of stakeholder involvement is specified in ORGAPET Section B3 below. It helps in explaining the content of the policy decision/programme to be evaluated but, also, in detecting potential problems in implementation. If, for instance, suggestions made by stakeholders important for organic food distribution have not been followed by policy decision-makers, these stakeholders cannot be expected to co-operate enthusiastically in implementation and this, in turn, may influence implementation and policy impacts negatively.

B1-3       Policy implementation

Policy formulation and decision-making constitute background conditions for a policy decision/programme while implementation constitutes the current conditions. Decisions are made occasionally, while implementation goes on continuously. It is the task of public agencies to implement public policy, and implementation may involve one or several public agencies. Policy decisions – not least decisions like organic action plans - are to influence the behaviour of target groups, whether they be governmental actors or non-governmental actors such as citizens, organisations and business firms. The impact or merit of a policy decision/programme may be measured in terms of policy delivery/performance of public agencies in their interplay with target groups or in terms of outcome where the effects of policy are measured (Winter, 2003). Hence, implementation performs policy decisions and is a central process for realising the aims of a policy decision/programme.

One important precondition for successful implementation is the content of the policy decision/programme to be implemented. It is well known from implementation and organisation research that it is difficult to realise ambiguous goals and decisions based on high levels of conflict (Matland, 1995), but many policy decisions/programmes – not least the action plan type – are ambiguous regarding goals, objectives and aims. Similar experience goes for conflict – a high level of conflict in decision-making may be implanted into implementation and influence the behaviour of the actors involved in opposite directions. Nevertheless, both ambiguity and conflict lie at the very heart of policy-making. Finally, the design of policy, i.e. the selection and combination of policy options, may not combine into a valid programme theory. Hence, implementation is often hampered by the very policy decision/programme that it is to realise (Winter, 2003). Part of evaluation is therefore to judge merits/impacts against the ‘implementability’ of the policy decision/programme.

Implementation takes place through the interactions of a diversity of public and private organisations and individuals. Winter (2003) summarises implementation research by distinguishing three types of behaviour which are decisive in the output and outcome of implementation and, therefore, important to evaluations of impact or merit of policy decisions/programmes. The first one is organisational and inter-organisational behaviour. It involves interaction between organisations of distinct policy sectors, such as environmental/agriculture ministries and regional/local governments, or within one policy sector, such as various agencies of the ministry of agriculture. This type of interaction may be shaped by bureaucratic power games concerning the survival and expansion of the organisations involved, and the implementation of any policy decision may be influenced positively or negatively from these power games.

The second type of behaviour is that of so-called street level bureaucrats (Lipsky, 1980), i.e. the front persons with direct contact to target groups. The behaviour of street level bureaucrats may be influenced by their individual preferences and working conditions as well as by the intentions of policy decisions/programmes to be implemented. The third type of behaviour is the behaviour of target groups. It may be more or less in line with the aims of the policy decision/programme, and their reactions to policy decisions/programmes may depend very much on their dependency on policy support and the types of incentives involved.

Much implementation research focuses on implementation deficit and attempts to explain it by combining negative aspects of the policy decisions with negative aspects of the three types of behaviour involved in implementation. One type of analysis focuses on the loss of consistency between policy intentions and implementation outcomes when a policy decision is moving down from the political top decision-makers through numerous administrative levels, each of which has discretionary powers, to the street level bureaucrats at the bottom of public administration far away from policy-making (Pressman and Wildawsky, 1973). This top-down approach has been challenged by a bottom-up approach that suggests a fertile interrelationship between street level bureaucrats who attempt to help target groups in resolving local problems. Here, local problems are solved by street level bureaucrats using their discretionary power more or less decoupled from perceptions of the problem and policy decisions made at the political top. The debate between the two approaches has never been resolved and implementation research make various attempts to combine them (Winter, 2003; Hill and Hupe, 2002).

Elaborating on the findings of Winter (2003), Vedung (1999) suggests that implementation results may be summarised, depending on the level of three qualities associated with any type of actor and any level of implementation:

Implementation thus depends to a large extent on stakeholders’ involvement – regarding both public agencies and their employees and the target groups of the policy decision/programme in question. It is therefore important that the stakeholders involved in implementation show comprehension and willingness in accordance with the policy decision/programme in question, and that they are well-disposed in terms of the capabilities relevant for implementation.

Regarding organic food and farming, it might be difficult to select stakeholders who combine all three characteristics because organic food and farming was defined and developed in opposition to mainstream food and farming. Opponents may therefore have control over or access to different but complementary resources. Organisations and firms in the organic food and farming sector may possess comprehension and willingness, while the capability (in terms of access to the main economic levers of agriculture policy; farmers and firms to take up organic practices and products; and expertise in developing agriculture and the food sector) lies with the opponent. As with implementation, this complexity calls for a careful balance in policy-making between consensus and ambiguity on the one hand, and conflict and clarity on the other. Ambiguity may attract all relevant stakeholders but be difficult to implement (and evaluate); clarity based on a conflictual approach may be easy to implement and evaluate, but implementation is undermined when conflict reduces a group of stakeholders’ comprehension, capability and willingness to act in the way anticipated by policy-makers and managers.

Stakeholder involvement and behaviour in implementation are therefore important issues in reconstructing the current conditions for operation of a policy decision/programme. One aspect concerns which stakeholders to involve in implementation; another aspect is to shape their interaction, for instance by attempting to cope with conflict in common boards/advisory groups or other fora for discussion and co-ordination - both these issues are considered in detail in ORGAPET Section A4. A third aspect, with special regard to the introduction of comprehensive action plans, is whether the actions suggested are followed by organisational and behavioural changes among stakeholders and their interaction, in support of the purpose of the action plan. These issues are dealt with in ORGAPET Section B3.

B1-4       Implementing the EU action plan in member states

Organic action plans have developed in various national policy contexts in Europe. When taking the EU Organic Action Plan into account, Europeanisation research suggests looking for processes parallel to those of implementation research (Michelsen, 2008a). The EU decision-making process is then considered as an additional decision-making level that increases the overall complexity of decision-making and of implementation and evaluation.

In EU decision-making, member states act as stakeholders in the common process, while other stakeholders may be organised at a European level and thus may seek influence on both member states, through national organisations, and at the European level (the Commission, the Council or the Parliament) through European organisations (such as COPA and COCEGA). Regarding implementation, each member state has a general record of implementing EU-binding decisions in national law, i.e. transposing EU directives (Giuliani, 2003). Among others, Denmark and Austria have a positive record in terms of timely transposition of EU directives, while Italy and Germany have negative records as transposition is often delayed. These general differences are explained primarily by variation in the formal transposition procedures used in member states. Policy impact of EU policies in member states is an issue that Europeanisation research generally abstains from analysing.

On the basis of a combination of Europeanisation and implementation research, Michelsen (2008a; 2008b) has compared transposition and policy impact of the EU regulations defining organic production and allowing financial support to organic farmers in eleven new and old member states. The findings are relevant to evaluation of the implementation of the EU action plan in member states although they are not fully applicable, since member states are only obliged to implement those parts of the EU action plan that appear as new regulations or directives, such as the new regulation on organic production.

Regarding transposition of EU regulations on organic agriculture, Michelsen’s (2008b) findings suggest that this is primarily an issue of the formal procedures in each member state. Member states’ timing of transposition follows the general pattern of timeliness, and member states transpose the content in a way that fits existing patterns of public agricultural policy administration. There is one important exception, however. The level of financial support paid to organic farmers is clearly related to the level of conflict over organic agriculture within the agricultural policy subsystem and between organic and mainstream agriculture organisations. In Austria, a truly low level of conflict – where organic and mainstream agriculture are considered as having common economic interests – has led to a high level of financial support to organic farmers. In the UK and Belgium, low financial support related to a high level of conflict, and the level of financial support was equally low in member states such as Greece and Poland, characterised by a low-ignore level of conflict, where mainstream and organic agriculture avoided recognising the agricultural values of the other.

Regarding policy impacts of the EU regulations, Michelsen (2008a) suggests that the EU regulations have no direct impact on the growth of organic agriculture in member states. Rather, they are parts of a stream of choice opportunities (see Kingdon, 1995) available to member states and the national stakeholders on issues of promoting organic agriculture and food. Once again, the level of conflict between mainstream and organic agriculture is decisive to the outcome. A truly low level of conflict, in terms of common economic interests, correlated with high policy impacts in terms of a high national ratio of organic farms while low policy impacts, in terms of a low ratio of organic farms, is associated with both a high and a low-ignore level of conflict in terms of agricultural values. An additional factor is the level of stakeholder activities involved in realising the potentials represented by the regulations. Aside from situations with a high level of conflict, the policy impacts of activities with a broad scope, combining political actions with activities in the food market and in the farming community, are higher than those with a narrow scope – and policy impact grows, the more frequent are the attempts to combine activities.

These results point at additional elements in the evaluation of the European organic action plan. Regarding transposition, timing may become an issue since member states can delay or eventually refrain completely from implementing any non-obligatory part of the EU organic action plan. Delays are, however, not expected to relate to member states’ formal procedures of transposition but to other aspects such as the level of conflict over organic agriculture policy, or to stakeholders’ comprehension, willingness and capability in relation to the non-obligatory aspects of the EU organic action plan. The content of the transposed, non-obligatory elements of the action plan may either fit the existing administrative structure or – especially regarding financial aspects – reflect the level of conflict over organic agriculture within the agricultural policy subsystem.

Regarding policy impact, the results suggest attention towards the level of conflict between mainstream and organic agriculture, and to the scope and frequency of initiatives (if any) taken by national stakeholders that may originate in the opportunities offered by the EU action plan. The results even suggest specific expectations as to policy impacts, depending on the level of conflict in member states. In member states characterised by a truly low level of conflict between organic and mainstream agriculture, stakeholders will use the action plan as an opportunity to activate many of the actions suggested rather frequently and with a broad scope. Conversely, stakeholders in member states characterised by a low-ignore level of conflict may be expected to use none or only few of the actions included in the EU action plan, and any activity is expected to have a rather narrow scope. Finally, in member states with a high level of conflict, stakeholders may be able to initiate frequent activities with a broad scope but these are not expected to materialise in terms of policy impact.

As part of the ORGAP project, the perceptions of stakeholders with respect to the implementation of the EU organic action plan in a national context were investigated in a workshop framework (Annex B1-1). The theoretical background for the workshops was implementation research and its focus on conflicts between various actors as the main explanation for implementation success or failure as influenced by stakeholders’ willingness, capability and comprehension in relation to the policy to be implemented. As the realisation of the EU organic action plan was still in its initial phase, implementation was a rather hypothetical issue and could only be assessed in qualitative terms. The workshops took place in eight EU member states with an organic action plan, which implied that participants in the focus groups had some knowledge on the problems associated with implementing organic action plans, but that they on the other hand should have a systematically more positive attitude to the implementation of the EU organic action plan than expected in all EU member states. All the workshops discussed implementation problems and coping strategies in relation to the EU Commission’s proposal for a new regulation on organic production, a major part of the EU organic action plan expected to be implemented by all member states by 2009. Two focus groups discussed implementation problems and coping strategies in relation to the recommendations on using the Rural Development Plans as basis for financing the national policies. This is also a rather concrete discussion topic since all member states had to specify the distribution of subsidies for Rural Development Plans for 2007 and following years about the time when the focus groups were held (November 2006 to February 2007). The second discussion topic in six focus groups was the recommendations on a more transparent market development included in the EU organic action plan, which partly overlapped the new regulation and partly included many specific recommendations on ways to obtain market transparency. When the results of the discussions of the three topics were combined, it appeared that findings were complementary and that they seem representative for discussions of the full EU and national organic action plans.

The main conclusion from the analysis in the ORGAP workshops is that the level of implementation success in any member state is a matter of balances between positive and negative aspects of all three main dimensions of implementation: willingness, capability and comprehension of all stakeholders involved. These balances are unique to each member state and within each dimension. The main expectation is that more weight on positive aspects on all three dimensions will lead to more successful implementation, while there are no clear expectations with regard to the interplay between the balances of the three dimensions. The workshops were characterised by a theoretically unexpected lack of conflict. Conflict was thus not an issue in the direct data analysis. However, the theoretically informed, final analysis showed the major importance of the conflict between the organic food and farming sector on the one hand and various threats against it from the socio-economic context, from the ideas behind the EU organic action plan and from the focus on its unintended impacts on the other hand. This demonstrates that a certain level of conflict is involved in the organic sector’s comprehension of the socio-economic context that includes the mainstream and non-organic part of the food sector and agricultural policy.

As it is a rather complex issue to measure the level of conflict in any member state, it may be appropriate to focus more on analyses of comprehension, willingness and capability among main stakeholders. The findings also emphasise the issue of including stakeholders representing both mainstream and organic agriculture and food sectors in the analysis.

B1-5       Checklist

The list of questions presented in the following checklist is intended as a summary of the issues covered in this section. It is envisaged that the questions should be addressed by the evaluators in order to examine the process of designing and implementing the action plan. In some cases, more detailed assessments may be required as described in subsequent sections – these are indicated where appropriate. The list is not intended to be exclusive, equally it may not be possible to address all the questions in every case. The commissioner of an evaluation should therefore consider what, if any, modifications should be made prior to the evaluation, consistent with the purpose of the evaluation and other considerations raised in ORGAPET Section A5.

  1. What was the extent (type, scale and policy orientation) of prior policy initiatives in support of organic food and farming (if any)?

  2. How was the agenda for the policy process defined and what were its characteristics (describe the policy process adopted)?

  3. What was the specific occasion/problem leading to the policy initiative, if any?

  4. What kind of analysis (if any) of the organic food sector and its needs was carried out in preparing the decision?

  5. Were results of previous evaluations available? If so, how were the results of these evaluations applied?

  6. Summarise the programme content (e.g. action plan action points) and relevant regulatory (legislative) framework (a more detailed analysis of the programme will be conducted in subsequent sections).

  7. Were clear and specific (SMART) objectives defined (see also ORGAPET Section C1)?

  8. What was the programme theory developed to support the objectives and measures proposed (see also ORGAPET Section A3) and was it relevant in solving the original problem?

  9. Was an analysis of conflicts and synergies (coherence) and risks of implementation failure carried out (see ORGAPET Section B2)?

  10. What were/are the relevant alternative (complementary or exclusive) policy options that could have been implemented (could the desired results have been achieved by different or additional policies, such as a prohibition on nitrogen or pesticide use, by supporting integrated farming or by taxation)?

  11. What relevant counter-factual situations can be described?

  12. What kind of changes in public or private organisations within the agricultural sector were required and/or made with specific relevance to organic food and farming?

  13. Was the strategy/programme finally developed relevant to the original problem and for how long did it remain relevant? If no longer relevant, what factors have changed to cause this?

  14. Who were the main actors involved in decision-making (see also ORGAPET Section B3)?

  15. What was the level of political conflict? What were the main discussion points? What proposals were rejected and why?

  16. How strong was the level of political commitment to the plan (very low, low, moderate, high, very high)?

  17. Which institution(s) was (were) responsible for the implementation of the plan (describe type and affiliation, e.g. governmental/non-governmental, and main characteristics of the institution(s), and assess the institution(s) comprehension, willingness and capabilities regarding policy objectiveness)?

  18. Were a separate budget and staff resources allocated to the action plan (details are needed for ORGAPET Section C3)?

  19. Which specific administrative issues/constraints could have influenced the implementation?

  20. Were monitoring and evaluation issues addressed appropriately from the outset (see ORGAPET Section A5)?

B1-6        References

EC (1999) Evaluating Socio-economic Programmes. MEANS Collection Vols. 1-6. Office for Official Publications of the European Communities, Luxembourg.

Hill, M. and P. Hupe (2002) Implementing Public Policy: Governance in Theory and in Practice. Sage Publications, London.

Giuliani, M. (2003) Europeanization in comparative perspective: institutional fit and national adaptation. In: Featherstone, K. and C.M. Radaelli (eds) The Politics of Europeanization. Oxford University Press, Oxford, pp. 134–57.

Kingdon, J. W. (1995) Agendas, Alternatives and Public Policies. 2nd ed. Harper Collins, New York.

Lampkin, N., C. Foster, S. Padel and P. Midmore (1999) The Policy and Regulatory Environment for Organic Farming in Europe. Organic Farming in Europe: Economics and Policy, Vols. 1 and 2, University of Hohenheim, Stuttgart.

Lipsky, M. (1980) Street-Level Bureaucracy. Dilemmas of the Individual in Public Services. Sage Foundation, New York.

Matland, R. E. (1995) Synthesising the Implementation Literature: The Ambiguity-Conflict Model of Policy Implementation. Journal of Public Administration Research and Theory, 5(2):145-174.

Michelsen, J. (2008a) A Europeanisation deficit? The impact of EU organic agriculture regulations on new member states. Journal of European Public Policy 1/2008.

Michelsen, J. (2008b) The Europeanisation of organic agriculture and conflicts over agricultural policy. Food Policy, in press.

Parsons, W. (1995) Public Policy. An Introduction to the Theory and Practice of Policy Analysis, Edward Elgar, Cheltenham.

Pawson, R. and N. Tilley (1997) Realistic Evaluation. Sage Publications, London.

Premfors, R. (1979) Policy analysis. Studentlitteratur, Lund.

Pressman, J. L. and A. Wildawsky (1973) Implementation. How Great Expectations in Washington Are Dashed in Oakland; Or, Why It’s Amazing that Federal Programs Work at All, This being a Saga of the Economic Development Administration as Told by Two Sympathetic Observers Who Seek to Build Morals on a Foundation of Ruined Hopes. The Oakland Project. Berkeley: University of California Press.

Thelen, K. and S. Steinmo (1992) Historical institutionalism in comparative politics. In Steinmo S., Thelen K., and Longstreth F. (eds.), Structuring Politics: Historical Institutionalism in Comparative Analysis.  University Press, Cambridge.

Vedung, E. (1999) Public Policy and Program Evaluation. Transaction Publishers, New Brunswick, New Jersey.

Winter, S. C. (2003) Implementation Perspectives: Status and Reconsideration. In Peters, B. G. and Pierre, J. (eds.), Handbook of Public Administration, pp. 212-222, Sage Publications, London.

B1-7             Annexes

Annex B1-1:    Michelsen, J., and A.-M. Tyroll Beck (2007) Implementing the European Organic Action Plan in EU member states. Stakeholders perceptions of implementation problems and coping strategies. ORGAP project deliverable D7. University of Southern Denmark.