Getting Real about Unknowns in Complex Policy Work

As with all public policy work, education policies are demanding. Policy workers need to ‘know’ a lot—about the problems they are addressing, the people who need to be engaged, the promises they can make in response, the context they are working in, and the processes they will follow to implement. Most policy workers answer questions about such issues within the structures of plan and control processes used to devise budgets and projects. These structures limit their knowledge gathering, organization and sense-making activities to up-front planning activities, and even though sophisticated tools like Theories of Change suggest planners ‘know’ all that is needed for policy success, they often do not. Policies are often fraught with ‘unknowns’ that cannot be captured in passive planning processes and thus repeatedly undermine even the best laid plans. Through a novel strategy that asks how much one knows about the answers to 25 essential policy questions, and an application to recent education policy interventions in Mozambique, this paper shows that it is possible to get real about unknowns in policy work. Just recognizing these unknowns exist—and understanding why they do and what kind of challenge they pose to policy workers—can help promote a more modest and realistic approach to doing complex policy work.

practices to undertake in pursuing such objectives (like teacher training, free textbook provision, the use of targeted grants to schools, capacity building of parent committees, use of electronic information and payment systems, and more) (Adhikary 2014, Hallinger 2010. These practices are included in many countries' projects and policies, as referenced in a completion report of the World Bank's 2011-2018 Education Sector Support Project in Mozambique: "The project development objective (PDO) and supporting project activities are based on internationally recognized research" and "The technical design of the project drew from globally recognized approaches for developing effective basic education interventions that are aligned with international best practice" (World Bank 2020, p.9, p.35).
The tenor of this statement (and others like it) suggests that policymakers know exactly what is needed to reach education policy goals, and even how to do so. Actual policy results do not fully endorse this view, however, with policy engagements across the world consistently failing to generate intended improvements in key parts of the work. The introductory Mozambique story provides an example: Three decades of policies have succeeded in delivering new school buildings and raising enrollment rates but not in improving learning outcomes, especially for girls and in poorer parts of the country (World Bank 2020a, p.3 and p.5).
Policy evaluation documents commonly explain these mixed results by pointing to the insufficient preparation of planners with respect to the poor performing policy dimensions. The completion review for a 1999-2006 World Bank project, for instance, faults policymakers for not 'foreseeing' or 'sufficiently preparing' or 'paying attention to' various issues related to girls education, and making "no attempts to form a line of reasoning that would lead from inputs to outcomes" when designing interventions intended to improve equity (World Bank 2007, pp.21-22). Similarly, in reflecting on the way an unexpected dearth of teachers in poorer regions of the country undermined efforts to promote equal access, a 2020 project evaluation asked why planners didn't simply develop "a system to better identify the staffing needs of schools and deploy/distribute teachers based on the real needs of schools" (World Bank 2020, p. 43).
These explanations and recommendations suggest that the education community knows or should know how to do better in the policy areas where failure is commonplace, and policies will succeed more in these areas if planners and implementers just did a better job of identifying and using what is known. I challenge such assertions in this paper, positing that it is often an incorrect explanation of policy failure and generates the wrong remedy-"plan more, pretending to know more than you do, and implement based on the plan, hoping for the best." I suggest that recurrent failures are often the result of policy professionals not knowing the answers to key questions and that pretending 'to know' will not work in such cases. Instead, policy workers should recognize what is unknown in their policy work and get real about the limits they face to know such things.
I offer this argument in three sections. The first section identifies a list of 25 questions one might consider 'essential' to doing policy work. This list shows how much policy workers need to know and how hard the task of knowing is. A second section then describes the plan and control approach most modern policy organizations use to do their policy work (including education departments in developing countries and entities like the World Bank). I note that this approach depends on long, passive planning processes for knowledge gathering, and that failures are often seen as failures to know, which are then seen as failures to plan. Section three offers a new way of thinking about 'failures to know', given a breakdown of the different kinds of 'unknowns' policymakers face. I suggest that many answers to policy questions are unknown, because of ambiguity, indeterminacy or bias, and policies often fail because of such unknowns (which cannot be rectified by better passive planning). I refer to the Mozambique example throughout the paper, showing the relevance of questions I pose, the dominance of plan and control methods, and-in section three-how repeated policy failure relates to the degree of unknown in a key part of the country's education work, focused on girl's education.

What should policy workers know?
Public policy work is complex: it involves many actors doing various things, often over long periods of time. Definitions of concepts like 'public policy' and 'public policy implementation' help to identify key aspects of this complexity, capturing what is most essential to consider when doing such work. Kraft and Furlong (2004, 4), for instance, define public policy, as "a course of government action (or inaction) taken in response to social problems," which points to the importance of identifying the social problems prompting policy responses, decisions needed to activate these responses, parts of government to involve in such decisions, and the 'course(s) of action (or inaction)' required for response. A definition by George Kousoulas (1982, 313-314) i refers similarly to four essential concerns, including which governmental authorities to engage, what decisions and plans are required, how state revenues and other resources will be disposed, and which public and social problems will be addressed.
I identified over 300 similar concerns about policy work 'essentials' from over 100 definitions of 'public policy', 'public policy analysis', 'public policy implementation' and related concepts, organized such into categories, and translated the categories into 25 questions for policy workers to ask when doing policy work (where I define policy workers broadly as anyone involved in policy engagements, from authorizers to planners, implementers, community mobilizers, and more). Table 1 shows these questions in 5 categories-reflecting the essential focal areas of policy purpose, people, promise, fit (to place, people, preparedness and period), and process.
This list implies the kinds of things policy workers should know about when they do policy work: the problem they are addressing (and if it is on relevant agendas); the people who need to be engaged (and if they are engaged and capable); the promises they can make in response to any focal problem (and if these are realistic); the context they are working in (and if policy ideas 'fit' such); and the processes they will follow to get things done. Studies suggest that knowledge about such things can determine success or failure in policy work, which is why 'knowing' about them is essential (Aldayel et al. 2011, Ika et al. 2012, Tezera 2019. Studies in the education arena also call for improved knowledge about such things-including problems facing girls at school, opportunities to expand access for students with disabilities, the challenges of promoting STEM courses for girls, and more (Croft 2013, Phillips 1998, Pilot and Bulte 2007. Studies in Mozambique raise similar questions, even three decades into the education reform journey (Fox et al. 2012). Table 1. Essential questions policy workers should ask when doing policy work 1. Is the policy purpose established? 1.i. Is the policy motivated by an agreed, clear and consequential public problem? 1.ii. Is the motivational problem receiving active attention and included in relevant policy agenda(s)? 1.iii. Are the values and interests informing this policy work clear, accepted, and non-conflictual? 1.iv. Are the potential winners and losers of this policy work clearly identified, accepted and being managed? 1.v. Is the problem considered manageable (with identifiable, agreed and treatable causes and interactions)?
2. Are needed policy people identified and engaged in the work? 2.i. Are political, resource and administrative authorizers actively engaged in (and committed to) the work? 2.ii. Are appropriately skilled planners and designers actively engaged in (and committed to) the policy work? 2.iii. Are appropriately skilled implementers actively engaged in (and committed to) the policy work? 2.iv. Are public beneficiaries and other affected public parties actively engaged with (and committed to) the work? 2.v. Are appropriate discourse processes actively fostering needed interaction between agents? 2.vi. Are processes in place to manage competing interests, opposition, and resistance to the policy work?
3. Is the policy promise (solution and results commitment) clear and deliverable? 3.i. Is there a clear and agreed vision of what the state of the world will look like when the problem is 'solved'? 3.ii. Are there clear, agreed metrics to signal the state of the world will look like when the problem is 'solved'? 3.iii. Is there a clear and agreed response to the problem (ideas that will solve the problem)? 3.iv. Are there clear and agreed periodic milestones and deliverables associated with this response? 3.v. Are risks of non-delivery of milestones and deliverables clear and manageable? 4. Is the work fitted to the policy place, people, preparedness, and period? 4.i. Is the policy work actively fitted to realities of the 'place' of engagement (especially laws, geography, politics)? 4.ii. Is the policy work actively fitted to story of the 'people' affected (especially history and culture)? 4.iii. Is the policy work actively fitted to realities of contextual 'preparedness' (especially existing capacity)? 4.iv. Is the policy work actively fitted to the 'period' of engagement (the time and timing of action)? 5. Is there a policy process in place to foster effective implementation? 5.i. Is there a clear and agreed program of action to deliver the proposed policy response? 5.ii. Is the program being (or likely to be) actively 'carried out'-turned from intent to action? 5.iii. Are agents being (or likely to be) effectively mobilized to execute programmed actions? 5.iv. Are resources (including finances) being (or likely to be) effectively mobilized to execute programmed actions? 5.v. Are decisions needed to execute programmed actions occurring as and when required (or likely to be)?
Source: Author's analysis of essential concerns to consider in public policy work, reflected in 100 definitions of 'public policy', 'public policy implementation', and related concepts.
The length of this list of questions is just one thing that makes policy work hard. The challenge of 'knowing' all 25 essential questions is also complicated by the many agents involved in such work.
Answers to the questions raised in Table 1 must be known across groups, not just in the heads of individuals. This is implied in the idea of 'agreement' on key issues (Lempert et al, 2003) and is important to note because policies-especially implemented policies-are the products of multiple agents' interactive engagement. Once again, we see this communal 'knowing' need in the education sector as well, and in Mozambique, where studies find that many policies only work if multiple parties agree on the problem, solution, process, and more (including entities like the Ministries of Finance and Education, Provincial Governors, Mayors, principals, teachers, parents, and more) (Andrews 2013). ii

'Knowing' in plan and control processes
Public policies tend to be identified, prioritized, designed and implemented through plan and control processes used to develop budgets and projects. Such processes impose an "orderly modernist perspective" on policy-making activities (Geyer and Rihani 2010, p.64) iii and structure policy work into two main parts-planning and implementation (as shown simply in Figure 1).
Planning typically involves identifying problems, objectives, ideas for action, and programs of work for a period to come (often a year or multiple years). Implementation involves mobilizing resources (including street-level actors) and executing activities according to planned programs, constantly monitoring such activity to ensure compliance with the plan. iv  A key step always happens between planning and implementation of public budgets and projects: The 'lock-in' event. This is the occasion when policy programs are approved and legally set-in-place (project contracts are signed or budget laws are passed). Another key step happens after implementation: Evaluation (or audit). This typically involves an ex-post analysis of execution, especially to determine if work following the lock-in event complied with the plan or budget approved (and its objectives).
These two steps impact when policy workers get to answer the kinds of questions posed in Table   1 (and come to 'know' what is needed to do their work). In particular, the lock-in step significantly limits knowledge gathering, organization and sense-making to the early planning stage. Knowledge gained after lock-in is hard to act upon, given that the work program has already been approved and set-in-place. As a result, significant changes to post lock-in programs are often avoided (because they tend to require new legal approval, which is costly and time consuming). In addition to this, policy evaluations typically ask how implementers acted on the program approved at lockin-not how they added to, adapted or improved such-which disincentivizes new knowledge gathering and appropriation after planning is done.  The two main stages of work (planning and implementation) are obvious in this figure, as are the lock-in and evaluation events. Project documents also show the tendency to limit knowledge gathering to planning, given the limited changes to objectives, activities, and implementation modalities after the 1999 project approval. One sees a similar process of policy development in other education projects in the country, including the 2011 to 2020 Education Sector Support Project. There were more changes to this project after the initial approval and lock-in, in 2011, because of high-level political shifts and shocks (like the country's 2015 financial crisis). These changes required new approvals and even the creation of new project vehicles, which took time and added costs to the work. Even with these changes, however, most of the project's activities were not altered from the initial plan and it is fair to say that the knowledge gained in the first years of planning drove policy work predominantly throughout the decade of work (World Bank 2020). v

Planning tools to facilitate 'knowing'
As presented, the early planning stage is absolutely vital in plan and control processes. This is where policy workers do most of their knowledge gathering, organization and sense-making.
Implementation work after this stage focuses mostly on enacting ideas locked-in through approved plans (with implementers not commonly focused on facilitating new knowledge gains).
As such, policy organizations commonly set high expectations for planners, whose work is often required to meet stringent principles of rational decision-making and smart policy design-being well justified and offering logical and feasible solutions to clearly defined policy problems (Simon 1976, Viennet andPont 2017, p.7).
Public policy organizations like the World Bank-and many governments-often require their planners to use tools designed to help meet such expectations. The Logical Framework (LogFrame) is an example. This tool has long been used across the world to depict the relationship between policy inputs (money, time, people and skills), activities (processes), outputs, outcomes (short-and medium-term results) and impacts (long-term results) (Auriacombe 2011, p.42;Brown 2017, p.3;Stein and Valters 2012, p.7;Uwizeyimana 2020, p.8). The LogFrame encourages policy planners to simplify (or reduce) the policy challenges they are working on into small, do-able parts and show how these can be delivered in linear sequences of activity. This helps to operationalize high-level policy interventions (and guide implementers on how to specifically execute plans) and informs how policies should be monitored and evaluated (a key concern in plan and control processes). To illustrate the influence of such tool, Figure 3 shows the percentage of OECD countries that evaluate policies by referencing LogFrame concepts like 'inputs', 'outputs', 'outcomes', and 'impact' and broader concepts emerging as interactions between these concepts: 78% of OECD countries emphasize efficiency, for instance, which is measured as the relationship between outcomes or outputs and inputs; 85% of these countries monitor effectiveness, which is the relationship between outcomes and objectives. The LogFrame tool thus helps planners gather knowledge and answer questions like those raised in Table 1, and to present this knowledge as rational and 'smart' (or at least logical, measurable and operational) policy proposals that facilitate control. Such tools are commonly used in devising education policies, as described by Johnson (2008, p. 5-6): "Currently, many methods of investigating the educational outcomes of individual schools ... are based on linear algorithms that simplify and break down systems into isolated, component parts. The premise of such linear models is that inputs into the system will result in predictable outcomes." Similarly, Biesta and Osberg (2010, p. 2) discuss the way these tools are used by "many policy makers" intent on "making education into a perfectly controllable and perfectly predictable technology." Critics of the LogFrame question whether it does help policy workers answer all the questions they face, however. Biesta and Osberg (2010, p. 2), for instance, suggest that "many educational practices and processes are not able to achieve a perfect match between "input" and "output"." Similarly, Johnson (2008, p. 5-6) questions whether the education system is as linear and predictable as the LogFrame 'logic' demands: "While appropriately predictive of some static, closed systems, these models fail to adequately predict the behavior of or capture the essence and emergent properties of complex systems." Organizations like the World Bank have responded to such critique by devising more sophisticated tools to help planners ensure they 'know' what they need to know to get policy work done (or to communicate that they do). The most prominent recent addition is the Theory of Change tool.
This still focuses planners on breaking policy work down into parts and showing the linear strategies they propose to achieve goals in certain parts (using LogFrame concepts like inputs, activities, outputs and outcomes) but it also allows planners to illustrate more complex potential pathways (that may not always be linear) and to record the thinking behind such, through a "set of assumptions that explain both the mini-steps that lead to a long-term goal and the connections between these activities and the outcomes of an intervention or program" (Anderson 2004; Monaghan and King 2018, 369). As described, therefore, it offers a "way of thinking about how a project is expected to work" (Stein and Valters 2012) that is not necessarily linear and that can include the possibility of multiple and even overlapping cause and effect pathways.
Theory of Change tools are increasingly used in project and budget preparation, monitoring and evaluation. They inform discussions about risk and implementation flexibility requirements in ways that prior tools could not (given the important role of knowledge about assumptions, which help planners determine potential challenges and even prepare for responses to such). These contributions make Theory of Change tools useful additions to the planner's toolbox, allowing her to plan and control with even more confidence than she ever had before. This is because the tool is incredibly knowledge intensive, at least when used properly. When used in this way (or presented as being used in this way), a planner implies that she knows ( what the potential pathways and interactions between these inputs, activities, outputs and outcomes look like, who will be involved in every part of the work, where and when these parts of the work will occur, and-on top of it all-what assumptions underlie everything that is known, what risks are associated with those assumptions, and what pivots will be needed if risks transpire.
If a policy planner knows this, she is probably satisfying all the questions asked in Table 1 and should be confident that her policy proposal is likely slated for success.

Policy failures, and the constant call to 'do better next time'
It would be wonderful if all policy interventions went according to plan and succeeded, following the Theory of Change pathways from activity to output, objective and impact. That is seldom the case, however, and reality usually involves mixed results: where some parts of an intervention achieve objectives, some do not; some parts finish on time, others do not; some are over budget, others are in budget and yet others fail to absorb a penny (never advancing from paper to action).
Post Education Sector Support Program). Mixed results are evident in both projects-with stronger performance on school construction (promoting access) contrasted with weak performance in activities promoting equity (involving access and quality concerns for girls, special education learners, and students in hard-to-reach and poorer central and northern provinces), for instance.
In explaining the weaker results associated with work on equity, evaluation documents blame policymakers for not knowing enough, noting that they 'did not foresee' or 'sufficiently prepare' or 'pay attention' to key things that they should-apparently-have known. These are all good recommendations, offering ideas that sound 'correct' and 'rational' and could certainly be included into new projects. They were confidently offered in 2007 in response to the limited successes of certain policy areas (especially with regard to quality issues with hard-to-reach groups). In keeping with a plan and control mindset, the assumption behind such recommendations seems to be that policy workers involved in these prior engagements simply did not 'know' the right things needed to improve education quality and that future policymakers would have more success if they just acted on this new knowledge and proposed different activities. Unfortunately, I don't see the assumption holding. Some of these ideas were acted upon in World Bank sponsored policy interventions following 2007, but with limited success, and education equity and quality improvements remain elusive (as evidenced in the focus of a proposed new project targeting girls education in 2020 (World Bank 2020a)).

Getting real about unknowns in complex policy work
The experience with girls education policies in Mozambique (and initiatives to promote educational equity more broadly) is fascinating. A 1999 project included a full component dedicated to "Girls Education Initiatives" that did not, ultimately, disburse one cent (and failed to mobilize any activities). Evaluation documents blamed this failure on weak planning. Trying again, a 2011-2018 project started out listing gender parity as one of a handful of objectives, only to discontinue the focus in 2015 (after which there was no explicit focus on gender issues).
The change was explained as the result of altered views on the best indicator of equity. Another initiative was slated to begin in 2021, responding to an obvious, festering, problem that seems to garner attention but fails to motivate action: "Half of Mozambican women are illiterate compared to 27 percent of men … Women achieve only 1.4 years of schooling, two years below the average schooling among men" (World Bank 2020a, p.3).
It seems too simple to blame this kind of recurring failure on poor planning or implementation, and to hope that better planning will lead to better future performance. That was, in a nutshell, the argument offered in 2007, and we are now 14 years removed from that time. A new argument is required, explaining why this work is so hard (and perhaps impossible) to do. This section offers such argument, positing that the challenge of improving girls education in Mozambique (and many other challenges that suffer from recurring failure) may be too 'unknown' to allow successful treatment through plan and control processes.

Do different policy unknown profiles matter?
My argument builds on an emerging literature that points to differences in the extent and type of 'knowns' and 'unknowns' implicit in different policy challenges. Such perspective suggests that some policy workers can answer essential questions about their policy challenge with certainty, being clear about the policy's problems, people, promises, contexts and process demands (given questions listed in Table 1) whereas others are less certain in answering such questions, faced with nagging unknowns about one or more essential concern related to their policy challenge.
Additionally, studies posit that policy challenges with different unknown profiles need to be addressed using different methods and tools. Han et al. (2011, p.829) note, for instance, that "multiple varieties of uncertainty [exist] … and warrant different courses of action" in policy work.
Similarly, adherents to the 'VUCA' concept posit that unknowns arising from volatility, uncertainty, complexity and ambiguity (hence, VUCA) are different and need to be dealt with using different policy and management mechanisms (Bennett and Lemoine 2014, p.313).
This perspective is interesting in the context of public policy work, where there is arguably little attention paid to the extent or type of unknown posed by any policy challenge; and where every challenge-no matter its unknowns profile-is addressed through similar plan and control processes (using LogFrame and Theory of Change tools). Could this perspective help explain why Mozambique's girls education initiatives are repeatedly failing? Is it because girls education challenges pose sufficiently more-and more demanding-unknowns than can possibly be addressed using plan and control mechanisms?

Policy unknowns and the plan and control approach
These questions cannot be addressed without a strategy to assess the unknowns profile of any policy challenge. I draw on the literature on uncertainty in devising such strategy. This literature suggests that 'varieties' of uncertainty manifest as degrees of unknown that differ more by type than extent (Funtowicz and Ravetz 1990, p.4;Knight 1921;Spiegelhalter and Riesch 2011;Stirling 1993;Walker et al. 2012;Wynne 1992 • 0 th Degree unknown involves full certainty; we are sure of every fact and cause-and-effect relationship involved in our policy work and there is one possibility that we are 100% confident in (with full probability). ix We succeed in such situations by simply following what we know.
• 1 st Degree unknown involves fully calculated risk, where many possibilities exist (not one certain possibility) but we know about all the possibilities and we know the probability of each possibility. We can thus confidently model options to make decisions, forecast, and plan.
• 2 nd Degree unknown involves strict uncertainty, where we again face many possible facts and know about all potential possibilities of these facts, but we do not know the probabilities of all possibilities. We cannot, therefore, calculate our risks but do know what we do not know about those risks. We thus have to monitor and manage for what we know is unknown (often by preparing redundancies or planning structured flexibility into our strategies).
• 3 rd Degree unknown involves recognized ignorance. This broad category x relates to situations where we lack knowledge that we think we could know. We often think we 'do not know' because we have not personally sourced what we believe to be existing or latent knowledge, for instance, or that we 'do not know' because existing or latent knowledge is fragmented and needs to be combined, or that we 'do not know' because our existing belief systems impede the acceptance or appropriation of existing or latent knowledge. This is a common category in many policy engagements, where policymakers say, "I do not have the knowledge but I am sure it exists." In response, policymakers need to source the knowledge, coordinate collection and interpretation, or engage with belief systems that run counter to the knowledge.

th •
Degree unknown involves ambiguity, where we face multiple non-equivalent representations of possibilities and probabilities that cannot be reconciled. Essentially, there are many plausible perspectives or sets of knowledge or interpretations of knowledge about the same things; and we cannot choose between these sets. The ambiguity arises for various reasons; including because some knowledge can be understood in multiple ways (given differences in language or terms), different evidence exists about the same knowledge, and different social and political narratives exist about the same thing (Best 2008(Best , 2012(Best , 2018 Kovacic and Di Felice 2019). xi As a result of such ambiguity, it is impossible for policymakers to decide which of the known possible facts or ideas is correct or will be realized. They need to pursue sense making, relationship building and experimentation to achieve reconciliation (which is an uncertain and risky process in itself).
• 5 th Degree unknown involves indeterminacy or systemic uncertainty, where we are missing knowledge that is essentially not possible to know in any fixed way (through time or across contexts) because it "lack(s) definition within a cause-effect system … [or] interact(s) dynamically with multiple other developments and occurrences [such that it is constantly changing] … (or is) dependent on the particular social context [such that it differs from situation to situation]" (Butler et al. 2015, 668). These unknowns are extremely hard to work with in policymaking, because we do not know what we do not know and we could not readily know it anyway. We cannot, therefore, plan or prepare or act on any assumption of order but rather prepare for and even engage in the disordered or chaotic world of the unknown (to promote or respond to forces of emergence). As Butler et al. (2015, 669) argue, "Conventional expert-led approaches [to policymaking] are limited" in such situations, "and alternatives to understanding and decision-making are required to anticipate outcomes and build resilience with regards to uncertainty." • 6 th Degree unknown involves what Walker et al. (2012) call total ignorance; where we lack knowledge and do not know what it is that we do not know. This is often because of blind spots and biases in our knowledge sourcing and organizing processes and in our belief systems. An example comes in the psychological explanations Bent Flyvbjerg deploys to explain cost overruns in large projects. Flyvbjerg argues that cost inaccuracies often arise because of "optimism bias" which he explains as a "cognitive predisposition found with most people to judge future events in a more positive light than is warranted by actual experience" (Flyvbjerg 2008, p.6).
He notes that this is not an intentional act, but rather a kind of 'self-deception' the policymaker falls prey to because of implicit bias. Policymakers need to promote an 'outside view' to curb this kind of unknown (where they facilitate an external, unbiased view of their challenge, to identify where blind spots of total ignorance exist). This requires acknowledging the potential ignorance, however, which is unlikely when total ignorance prevails. Possibilities and probabilities are unknown but there is a belief that they can be known Yes, and managed through research and 'homework' in planning 2 nd Strict uncertainty All possibilities are known but probabilities are not known (some risk) Yes, and managed through process design (monitoring and flexibility) 1 st Quantifiable Risk All possibilities and probabilities are known (limiting risk) Yes, and managed through risk analysis 0 th Full certainty The certain possibility is known Yes, and preferred Source: Author's work.
As summarized, the plan and control approach is most suited to 0 th and 1 st degree unknowns-full certainty and risk-where the sum of the challenge is clearly known (implying agreement of a group), its parts are also known, and we know how to put the parts together to create the whole (with certainty, or reliably, given known odds). "The policy maker's role here is to assemble the requisite minds," develop an understanding of the challenge, "undertake interventions" based on this knowledge, and replicate again and again (Snyder 2013, p.7-8). Plan and control methods can also fit with 2 nd and 3 rd degree unknowns, strict uncertainty and recognized ignorance, where we know what knowledge is missing and can monitor and manage accordingly (in 2 nd degree situations) or source, coordinate and better align beliefs to learn what we do not know (in 3 rd degree situations). Plan and control approaches do not work well with 4 th , 5 th and 6 th degrees of unknown, however, where we face ambiguity, indeterminacy or total ignorance and do not know (or agree as a group) what the whole is, what the parts are, how the parts interact with each other to yield the whole, or if the whole is even the sum of its parts (to borrow from Aristotle). xii It is folly to pretend one can 'know' what is needed in such situations, or to develop a LogFrame or Theory of Change. Instead, the key to making any progress is to recognize and even celebrate the unknown. In the words of Fenyman (1956, p.21): "To make progress in understanding, we must remain modest and allow that we do not know. Nothing is certain or proved beyond all doubt. You investigate for curiosity, because it is unknown, not because you know the answer."

Unknowns and the mixed results of education policy work in Mozambique
I asked earlier if girls education programs in Mozambique were failing, repeatedly, because their implied challenges pose sufficiently more-and more demanding-unknowns than can possibly be addressed using plan and control mechanisms?
One way of investigating such question is to apply the unknowns framework in Table 2 to all 25 questions identified in Table 1 as 'essential' for doing policy work-testing the degree of unknown associated with each answer. This is not an easy exercise, partly because documents describing the girls education work are limited in detail (and were not written with this kind of activity in mind). Also, different types of 'unknown' described in the framework can be hard to differentiate-something that seems unknown could be an example of recognized ignorance (3 rd degree unknown) or indeterminacy (5 th degree unknown)-depending on the information one possesses. Additionally, all policy actors (including myself as an academic observer) have subjective views on any challenge and bring bias to the work. Walker et al. (2012, p.1) point to such subjectivity in saying that any perspective on "the extent of uncertainty [will be] related to the satisfaction with existing knowledge, which is colored by the underlying values and perspectives of the policymaker." Regardless of these difficulties, I offer Table 3 as my personal-distant-analysis of the 'unknowns profile' for girls education work in Mozambique. The analysis is based on information in project evaluation documents and is provided as a demonstration of how one can assess unknowns in a policy challenge and use such assessment to reflect on why the policy keeps failing. I provide a 'degree of unknown' in respect of the 5 main areas in Table 1, adopting a conservative bias in my scoring (choosing higher scores if there is any question about a type of unknown). Past policy workers believed this issue is on the national agenda but repeatedly found it was not actually supported.
This suggests a bias to believe support is present. 2. Are needed policy people identified and engaged in the work?

-Indeterminate
There is no evidence that the needed people have ever been identified or mobilized to work on this issue.

-Indeterminate
There is no evidence that policy workers know much about these contextual variables and how girls education programs could fit with contextual realities.
5. Is there a policy process in place to foster effective implementation?

-Indeterminate
Past projects proposed programs of action to follow that were not even initiated; it is unclear why this was the case or what kinds of lessons can be learned from the experience.
Source: Authors' analysis based on limited information provided in World Bank (2007) and World Bank (2020). * Given past experiences in World Bank projects tackling such issue ^The degree of unknown is an estimation, based on documented evidence, with a conservative bias (to score the highest degree of unknown implied) The analysis shows that-based on the limited information offered in project evaluations documents to explain past policy failures-girls education is fraught with unknowns. This is a subjective assessment and has not been done in collaboration with policy workers in Mozambique. Their involvement would certainly foster robust debate and-perhaps-a different set of scores and unknowns profile. As it stands, however, the assessment offers a new explanation as to why Mozambique's girls education initiatives persist in failing: The work is fraught with unknowns that cannot be addressed using plan and control mechanisms that the Government (and donors) keep on deploying. Better planning will not address bias, ambiguity, or indeterminacy. Another policy approach is needed to do this.
Getting real with and facing up to unknowns I imagine that some readers are actively asking (or at least wondering), 'So, what kind of approach can one use in the face of unknowns, given that the plan and control approach (and its call for more and better planning in the face of repeated policy failures) often won't work?' It is a valid and important question, and certainly follows from the discussion. Unfortunately, however, it falls beyond the scope of this paper, which focuses on making two more fundamental points: First, policy unknowns exist and can be assessed in any policy engagement and are likely present in many key engagements (like Mozambique's girls education initiatives); Second, policy challenges subject to significant unknowns cannot be effectively addressed through the same policymaking approaches one uses when there are no unknowns.
Beyond making such points, the paper intends to offer practical new ways for policy workers to engage with their policy challenges. It presents these workers with a list of 25 'essential questions' to ask when doing such work, for instance, and with a 'policy unknowns framework' to use in answering these questions. When combined, these mechanisms provide a potentially powerful tool policy makers can use to perform due diligence on their policy challenges-determining how much they know and do not know and why-before they embark on policy journeys and at every juncture they may find along journeys that have already begun.
My hope is that policy workers who use such tools will start learning more about the unknowns that probably plague much of their work and initiate their own searches for new approaches to doing this work. These applied searches-and the new approaches they spawn-are where we as a policy community will learn new ways of pursuing policy objectives in the presence of unknowns. These searches will only come from policy workers who have undergone a real shift in thinking about their work, however, and are so convinced that unknowns frustrate their progress that they are willing to give up the instrumental mindset of the linear LogFrame optimizer that asks, "where do we want to be and how do we get there?" and embrace a more experimental, risky mindset bent on learning "how … we move in a desirable direction in the face of uncertainty?" (Innes and Booher 2010, p.207).
This kind of shift may seem impossible in the public policy world-and especially in the international development context-where it seems implausible to think of organizations moving away from plan and control processes (even in the face of evidence that unknowns exist and cannot be planned for or controlled). Readers in such situations might feel it necessary to continue using such processes, even when they have unknowns in their work, given that they know how to do Theories of Change and LogFrames and these seem the only approaches at hand. But I warn these readers against what Wiggins (2015) calls 'Stupidification'; "(the) deadly illness in which [we reduce] intricate issues and processes to simplistic, rigid, and mandated policies, in the impatient quest for quick fixes to complex problems." Ignoring our ignorance and pretending we know what we do not know may help us define and sell a project or policy today, but it will also ensure we are still working on the same policy challenges in years to come. Hopefully this will not continue to happen with girls education in Mozambique.
iii Referring to this kind of approach, Geyer (2012) notes, "For much of the twentieth century UK public policy [in the education and health sectors] has been based on a strong centralist, rationalist and managerialist framework," which contributed "to the development of 'evidence-based policy making' (EBPM) and the 'audit culture'." iv While simplified, I believe this figure captures the basic structure of budget processes across the world, including Europe (Saliterer et al 2018), Organization for Economic Cooperation and Development (OECD) countries (OECD 2018(OECD , 2019(OECD , 2020, and developing countries (PEFA 2016, Schiavo-Campo 2017, Kristensen et al. 2019). The figure is also a simple but effective description of project planning processes used by bi-lateral, multi-lateral and non-governmental organizations working on public policy (especially in developing countries) (Golini et al. 2015, Golini et al. 2018, Matos et al. 2019. Finally, the figure also captures the basic structure of budgeting, planning and auditing and evaluation processes commonly used to devise, inform, and implement education policy across the world (Chang 2008, Rogers and Demas 2013, SABER 2013, UNESCO (2006, Viennet and Pont (2017), World Bank (2017). v The World Bank evaluation of this project (World Bank 2020) discusses all changes that were made after the initial lock-in. Most changes were made at new lock-in events (where additional finances were made available in new projects and targets were most commonly adjusted-for time, scope and level) but even these did not alter key aspects of the work program. As noted in World Bank (2020, pp.12-16), for instance, project development objectives "remained the same throughout the AFs of April 2012, July 2015, and May 2017" and "modifications did not affect the original theory of change, but rather enhanced the ability to measure progress toward achievement of the project's goals." vi PDO relates to Project Development Objectives. vii The authors' findings are stated in a negative light here, as related to policy failures. The actual findings are stated in a positive manner in the source article, however, to explain successes. viii Additional references include Ben-Haim (2006), Funtowicz and Ravetz (1994), Hillier (2016), Kovacic and Di Felice (2019), Luce and Raifa (1957), Quade and Marshall (1989), Ravetz (2019), and Scoones (2019). ix The distinction between possibilities and probabilities is possibly best expressed in discussing the 'four dimensions of incertitude' model proposed by Stirling (Leach et al. 2010, p. 53). In my understanding, however, possibilities are potential facts about a policy (in part or whole), including cause and-effect relationships and behaviors, and probabilities center on the likelihood of those facts being true.
x There is a large literature on ignorance and its types, which I do not do justice to, but useful references include Rescher (2009), Faber et al. (1992, Buratti and Allwood (2018), Smithson (2015), and El Kassar (2018). xi A growing literature explores the way in which ambiguity is fostered and used in bureaucratic and political settings, to foster decision-making, reduce conflict, and more (see Matland 1995, for instance). I personally think that Bent Flyvbjerg's "political-economic explanations" of project cost overruns fit into this category. Flyvbjerg (2008, p.6) describes this as "strategic misrepresentation … when … forecasters and planners deliberately and strategically overestimate benefits and underestimate costs in order to increase the likelihood that it is their projects, and not the competition's, that gain approval and funding. Strategic misrepresentation can be traced to political and organizational pressures, for instance competition for scarce funds or jockeying for position, and to lack of incentive alignment." This misrepresentation depends on a certain level of ambiguity in the system, which allows officials to interpret numbers and likelihoods differently. xii Aristotle's famous comment is commonly used in discussing complexity, as in the following description of complex systems by Herbert Simon (Simon 1991, p.468).): "Roughly, by a complex system I mean one made up of a large number of parts that interact in a non-simple way. In such systems, the whole is more than the sum of the parts, not in an ultimate, metaphysical sense, but in the important pragmatic sense that, given the properties of the parts and the laws of their interaction, it is not a trivial matter to infer the properties of the whole."