A Deficiency in Chapter 9

As we noted on the splash page, the book was a collaborative effort with the text entirely written within five days. Inevitably this means that after we left some of the ideas continued to bubble and new ways of organizing and summarizing the concepts came to light.

In Chapter 9 we discuss the challenges and possibilities for evaluating universities’ progress on the journey to becoming an Open Knowledge Institution. This introduces a theory of change with three stages (Aspiration/Narrative, Action/Investment and Outcomes/Evaluation) but this is not connected to a framing device that provides a way to categorise and organise these. As a result the two tables in the Chapter form something of a laundry list of activities and objectives.

An Improved Framework

Following the book sprint workshop the project team was looking at this issue and realised that the answer was already implicit in the book and its structure. Chapters 5-7 cover the main substance of what it is we propose that universities do to act as a OKIs. Seeing this, it became clear that we could organise the laundry lists in Chapter 9 under these three headings.

From a theoretical perspective this is appealing because these three areas of activity and intent map neatly onto those implied by social theories of knowledge. If we see generalisable knowledge being created in the contact between groups then diversity is a first order principle. The more diverse the groups, the more generalised the knowledge created in productive contact. But not all contact will be productive. To achieve this we need coordination; activities and platforms that support the bringing together of the right groups at the right scale at the right time with the right supports to make contact productive. Communication is key both to transmission of the generalised knowledge but also the cycle of supporting and engaging diverse groups and coordination between them.

With these categories in place we can build an improved version of Table 1 in Chapter 9.

To understand this division we will now examine these various areas in more detail.

Communication

Communication is the most straightforward and familiar of the three categories. We are concerned with effective communication to target audiences in the broad sense. This includes aspects of archiving and preservation (communication to the future) as well as many of the familiar policy areas such as open access and data sharing. It also includes effective public communication.

Policy initiatives and aspiration in this area are generally signaled by policies such as those on open access and data management, as well as aspirations for public engagement through effective communication. Action and investment will take the form of platforms and resources such as repositories and services (including open access) and the resourcing of support for public communication. Outcomes that could be measured include proportion of OA outputs, effective data sharing, re-use of outputs, and public and target audience use of outputs.

Diversity

In a social model of knowledge production that depends on the interaction between groups to create generalised knowledge diversity will be a first order principle. The greater the diversity of groups in contact producing knowledge, the more generalised and generalisable that knowledge is. In our current social context we are particularly concerned with diversities relating to gender, ethnicity, race, socio-economic status, sexuality, cognitive diversity and geographical origin. That is there is a focus on the demographic diversity of humans involved in knowledge production. This is long over-due and has the potential to massively increase the value of our scholarship by helping us to ask better and more relevant questions and provide improved and better contextualised answers.

Other important forms of diversity include that of disciplinary expertise, experience and techniques, as well as in the forms of output. Some of these, such as disciplinary diversity, universities do reasonably well at. In others, especially output format and media diversity, they have a distinct weakness. Another area that is worth of consideration is expanding cognitive diversity to considering machines more fully as actors in knowledge production with their own needs.

Aspirations in this area will include equal opportunity and inclusion policies, as well as statements of the scope of an organisation. It might, as suggested above include requirements for machine readability. Investments might take the form of support programs, child-care provision, and training for incumbent staff. It will also include investment in interdisciplinary programs and subsidies to maintain disciplinary diversity across the organisation. Outcomes might include staff diversity measures, but should be extended to retention of under-represented minorities as well as of their experience. Diversity in outputs (discipline, audience, formats) as well as of inputs (revenue, people) might provide valuable insights.

Coordination

Coordination is in some ways the least familiar of the three categories in the framework. It is also however critical to achieving the goals of Open Knowledge Institutions. Diversity is a first order principle. However, it is not sufficient to throw diverse groups into contact. Indeed this is a standard failure mode for diversity and inclusion programs that do not deliver sustained culture change and provide an environment that retains under-represented groups. Crucial to success is the thoughtful coordination of the contact between and within groups.

Part of that coordination is providing a safe environment, an institutional form in which some risks may be taken, because others do not need to be. Creating environments in which women, people of colour, indigenous and other under-represented groups are able to bring criticism based on their experience without taking risks with their physical and mental safety remains a substantial issue and a burden which is largely shouldered by exactly those groups. A successful OKI will institutionalise sufficient safety, in the form of rules, support, salaries and employment protections, to enable those risks to be taken.

Many universities already do this work effectively in other arenas, including the development of collaborations with industry and civil society organisations, or across disciplinary boundaries. Here negotiating the ground rules for ‘safety’ will include elements such as non-disclosure agreements, and IP protection, or formal contracts and commitments. Here the institutional and cultural forms are well established, indeed perhaps not sufficiently questioned. The connection between these platforms and institutional forms and their goals and those for supporting under-represented groups are rarely considered. Much could probably be learned in both directions.

Aspirations in this area will often be signaled in collaboration policies as well as in intellectual property controls. They may also be present in Reconciliation Plans (particularly seen in the Australian system) and goals for active engagement of specific communities. Investment takes the form of systems and platforms to support interactions, usually focused on a very narrow set of interactions (an ‘International Office’, an ‘Industrial Support Unit’) rather than broadly on the skills and tools to support productive group interactions more generally.

Institutionalising Open Knowledge

In the original version of Chapter 9 we discuss institutionalising open indicators. However because we don’t have the full framework we talk less about institutionalising open knowledge, and what that means. In this sense we don’t get to the question of how evaluation might help, although we touch on what the risks are. Our new categorisation offers a route towards this.

In Table 1 above there is a challenge as we move from left to right. It becomes increasingly difficult to know which row a particular outcome belongs in. This problem actually points us towards a much more rigorous theory of change which maps well onto the models that we develop in Chapter 3. When we start on the journey of change we will naturally engage with deficit models. What are we not doing? What do we need to change or do better? Policy efforts respond by targeting specific areas, ideally with as much focus as possible. Open Access policies never mention diversity. Inclusion and Diversity policies never mention open access. But we cannot achieve the aspirations of open access in delivering more usable research outputs unless we address both the needs and affordances of our communication to diverse audiences.

The consideration of how these different areas relate to each comes into focus when we move to the next phase of our theory of change. As soon as a university invests resources, whether time or money, there are choices to be made about where those resources are deployed. Do you invest in paying APCs to deliver open access or is that money better spent on child-care provision? While this may appear a contrived example, these choices are often made implicitly and without consideration of an overall strategy. How can the university change so as to find synergies between these investments? How might investment in child-care also provide a connection to user communities for relevant research? How does the provision of open access support or child care build and strengthen those connections?

This leads us to the final phase. An imagined organisation where it is the culture and institutional forms that hold all of these issues in tension. There is no correct solution, but rather behaviours and practices that help to optimise the overall position as a whole. Just as in Chapter 3 we talk about a shift in that optimum as a result of societal and technological change, we see here how culture and institutions (in the political economy sense) need to built and sustained that work to hold this conflicting requirements in tension.

The key questions we need to ask therefore are how policy, investment, internal evaluation and environmental change are contributing to this institution and culture building? How is this complex of competing factors being harnessed towards those goals? What works? What does not? This places evaluation firmly within the process of change, but also illustrates how on its own – as with policy making and investment – it is not enough.