Having presented the key to the mental map (a set of tools for thinking), we now briefly introduce the mental map components, before focusing in detail on the first component: the match between claims and warranting in arguments. How do the components relate to the tools? The authors whose work you study will have employed the tools for thinking in order to develop a convincing argument. The four components of the mental map will help you evaluate a range of factors that contribute to the content and robustness of that argument (Table 7.1).

In short, your mental map will enable you to home in on what authors were trying to do, why and with what success. In this chapter, we discuss the component that focuses on the match between authors’ claims and the quality of the evidence supporting them.

Two dimensions of variation among knowledge claims

In Part One, we saw that an argument is constructed from one or more claims to knowledge – assertions that something is, or normatively should be, true.

Table 7.1 Components of the mental map

Mental map component

Aspect of authors’ argument that it helps you examine Two dimensions of variation amongst

Two dimensions of variation amongst knowledge claims about the social world, affecting their vulnerability to criticism

Authors’ tentativeness or certainty about their claims and their willingness to generalize, relative to the amount of appropriate evidence available

Three kinds of knowledge that are generated by reflecting on, investigating and taking action in the social world

The basis of their claims, as relevant to theory, new research evidence or experience

Four types of literature that inform understanding and practice

Whether the account aims to inform theory, research knowledge, practice or policy, and some common weaknesses that can render each type less than convincing

Five sorts of intellectual project that generate literature about the social world

Authors’ reasons for undertaking their work: aiming to understand, evaluate, change others’ action directly or through training, or improve their own action

These claims form the conclusion, which is one half of the argument. The claims are supported by some form of warranting: the half of the argument that justifies why the conclusion should be accepted. Claims vary along two important dimensions, according to the amount of appropriate evidence contained in the warranting. If there is a mismatch, we see the warranting as inadequate.

In Chapter 3, we saw how warranting can be provided, and be appropriate, but still be inadequate – not sufficiently convincing for the critical reader. An inadequately warranted claim often fails to convince because:

• It is based on an insufficient amount of robust evidence to support the degree of certainty with which this claim is made.
• The evidence does not justify the extent to which the claim is generalized beyond its immediate context.
• Or both.

Certainty about a claim

Knowledge claims are made with varying degrees of certainty and it is possible to question whether the degree of certainty that the author asserts is justified. The academic literature is not short of highly speculative claims to knowledge of the social world, made with enormous confidence that they are certain truths. Yet, as we have already seen, no knowledge of the social world can ever be beyond all doubt. It is always appropriate for the critical reader to ask whether there is sufficient evidence to support the degree of certainty with which a claim has been made.

How sure authors are that they really have found out what they claim will be reflected in the degree of certainty with which they make the claim. An example of a highly certain claim is: ‘Trainee managers demonstrably learn more effectively when they are praised than when their efforts are criticized’. Often the degree of certainty expressed in a claim is left implicit, as in the version we saw in Table 3.1: ‘Trainee managers learn more effectively when they are praised than when their efforts are criticized’. Removing the word ‘demonstrably’ means there is no explicit indication of the high degree of certainty. But the certainty of the claim is still there: the authors simply state that praise does help trainee managers learn more than criticism – not that it may do so, or that it may sometimes do so in particular circumstances.

From time to time you will probably come across claims made with a level of certainty that you feel is unwarranted. Such claims are vulnerable to being rejected once you scrutinize the match between the evidence provided in the warranting for them and the certainty with which they are proposed. The more confident a claim, the stronger the evidence required adequately to warrant it. The more tentative the claim, the less the evidence required, because much less is actually being claimed. As a critical reader, you can scrutinize any argument by first checking that it actually has both a claim and a warranting, and then checking the match between the degree of certainty of the claim and the strength of the evidence offered as warranting. Remember each time to ask yourself:

• Is there strong enough evidence to support the degree of certainty adopted (or implied) for this claim?

In our example above, the certainty of the claim could be justified if the researcher had studied a very large number of trainee managers and always got this result. Some kinds of claims are compatible with strong certainty. The claim that ‘the earth is round’ could be warranted by the evidence that whenever you fly westwards for long enough you end up in the east, and that satellite photographs of the earth reveal its curvature. Little knowledge about the social world is that certain, so you are unlikely to find experienced social science researchers stating that their evidence proves a claim. You are much more likely to come across authors who state that their evidence suggests or is consistent with a claim. They may fine-tune such an explicit indication that they are not wholly certain by saying may suggest or strongly suggests.

There are other ways in which authors may signal their own lack of certainty. One is by stating that their claims are tentative or cautious. A formal means of signalling tentativeness is through hypotheses. A hypothesis is a claim consisting of a proposition or statement that something is the case but which is as yet unproven. It will often be predictive (as we saw earlier in Figure 2.1), implying that a particular outcome will flow from a particular action. An enquiry into an aspect of the social world might begin with a hypothesis, the validity of which is then tested by checking whether evidence supports it or not. Alternatively, an enquiry may produce hypotheses as outcomes, amounting to predictions that could be tested in future. However, many hypotheses in the study of the social world are so general that they are not amenable to straightforward testing. For instance, how could we convincingly test the hypothesis that ‘learning how to learn is a more effective preparation for adult life than learning lots of facts’? What would count as sufficient evidence warranting the conclusion that the hypothesis was supported or should be rejected?

Generalizing a claim

Claims are also made with varying degrees of generalization. The issue here for you, as a critical reader, is checking the extent to which findings from within the context studied also apply to other contexts. Some level of generalization is normally expected in research: one examines a phenomenon in a limited way in order to find out something that is likely to be true in other similar circumstances. Generalization, in part, is about how one judges what counts as a similar circumstance. A claim about, say, the effectiveness of an approach to supporting teenage mothers might be made on the basis of studying five social service units which offer such support in the UK. A judgement must then be made about whether it holds true for all UK social service units providing such support, and whether it might be extended to all social service units and other support arrangements for school-age mothers anywhere.

Frequently, when sweeping generalizations are made, the author is not explicit about the range of contexts to which the claim applies. Rather, the extent of the claim is implied rather than stated, as in our example ‘Trainee managers learn more effectively when they are praised than when their efforts are criticized’, which is not only presented with high certainty but is also implicitly highly generalized. By implication, the claim is asserted to have universal applicability – to all trainee managers everywhere, past, present or future. But generalizations are, in themselves, just claims that something is known, not proof that it is known. If you scrutinize the evidence offered for warranting this claim (in Table 3.1), you are likely to find it unconvincing. It comes from a survey of only female trainee managers in just one sector: retail. So, as a critical reader it is always appropriate for you to check the match between the degree of generalization of the claims made and amount of evidence used to back that generalization. Make a habit of asking yourself:

• Is there sufficient evidence to support the degree of generalization adopted (or implied) for this claim?

As a critical reader, you can scrutinize any argument by first checking it actually has both a claim and a warranting, and then checking the match between the degree of certainty of the claim and the strength of the evidence offered as warranting for the degree of generalisation.

The broader the range of contexts to which a claim is generalized, the more it may affect the level of abstraction. The issue here is the extent to which the intricate details of the specific context that was directly examined can be set aside, so that a greater range of contexts becomes eligible for the generalization. The broader the generalization, the more likely it is to be at a high level of abstraction, glossing over details of individual contexts to make a claim about some quite abstract feature that is supposedly common to them all. The generalization ‘learning how to learn is a more effective preparation for adult life than learning lots of facts’ glosses over the multiplicity of details that may vary between different contexts. They include learning environments (does it matter if you have a computer-equipped classroom or just an open space?), the characteristics of learners (is the claim equally true of adventurous and quietly reflective learners?) or purposes for promoting learning (can the learning be for its own sake or must it be aiming to contribute to society?)

When claims are generalized to many or all contexts, they are likely to be made at a high level of abstraction. The authors abstract – or ‘zoom out’ – from the details of the context studied, to capture something common to a much wider range of contexts. The issue here is the extent to which, in this particular case, the intricate details of the specific context that was directly examined really can be set aside, so that a greater range of contexts becomes legitimately eligible for the generalization.

An assumption underlying any generalized, abstract claim is therefore that the detailed factors differentiating each context are not significant enough to affect the applicability of the claim across a wide range of contexts. As a critical reader evaluating such a claim, you have to judge how far this assumption is warranted by the evidence presented. (You will have to take into account all the evidence that the authors provide, which may include both their own empirical findings and their account of other relevant literature.) The claim that ‘trainee managers learn more effectively when they are praised than when their efforts are criticized’ glosses over the possibility that contextual differences (such as trainees’ gender, age, work sector, past training experience or cultural background) might affect how far praise works better than criticism. The claim also implies that different kinds and amounts of praise and criticism, or the balance between them, have no major impact either. The degree of abstraction entailed in this highly generalized claim leaves it vulnerable to being rejected, because the authors have not shown that various contextual differences actually have no significance. (Contextual factors tend to affect the findings of social science research. Two researchers, dealing with different contexts, could easily get different results for this reason. If both over-generalized their claims, the critical reader might find that each had predicted incorrectly what the other would find.) Overall, the more generalized the claim, the more warranting it needs, to indicate how the claim applies across different contexts. Conversely, the more specific the claim, the less of such warranting is needed.

Scrutinizing the certainty and generalization of claims together

Figure 7.1 shows how the degree of certainty and generalization of claims interact. Each operates along a continuum, thus varying gradually: either from low to high certainty or from low to high generalization. The degree of certainty is independent of the degree of generalization so any combination is possible. The situation depicted in the bottom right-hand corner of Figure 7.1 is that of our example: a claim made with a high degree of certainty and a high degree of generalization. We have already seen how, for such a claim to convince a critical audience, it must be warranted by evidence which is adequate to justify the boldness of the claim.

So, as a critical reader, be alert to high certainty, high generalization claims – whether explicit or left implicit. Not every such claim will begin ‘It is always the case that …’, and the generality or certainty associated with the claim may not be stated close to the main concluding statements. The signal might be a brief remark near the end of the text, or something said or not said in the abstract. Subtler cases might build certainty and generalization into a new claim, as in: ‘The results of our study demonstrate that future training policy should focus on promoting the praise of trainee managers and minimizing criticism’. Here, the policy proposal is sweeping. It reflects the authors’ assumption that the study’s evidence adequately warrants a high degree of generalization and certainty. Whenever you identify a claim, first evaluate its degree of certainty and generalization. Then set your expectations accordingly about the extent of warranting you must find in the account if you are to accept the claim.

Figure 7.1 Dimensions of knowledge claims and their vulnerability to rejection

HIGH-RISK WRITING: HIGH CERTAINTY, HIGH GENERALIZATION

If as a critical reader you require extensive warranting to be convinced when claims are made with high certainty and high generalization, the same is likely to be true of the critical readers who assess your written work. Beware of making such claims unless you are sure that you have adequate warranting, whether from your own research or the wider literature. Otherwise your claims will be vulnerable to rejection as unconvincing. One way of reducing the vulnerability of high certainty claims is to make them conditional, as in ‘If these results are reliable, this is definitely the case’. Similarly, high generalization claims can be made conditional, as in ‘What I have discovered may also apply in other contexts, to the degree that this one is similar to them’. But critical readers might then question why you don’t seem sure that your results are reliable or that other contexts are like yours. A more effective writing strategy is to judge for yourself just how reliable and generalizable you consider your findings to be, and then to adopt a clear and defensible position along each continuum.

The top left-hand corner of Figure 7.1 represents low certainty and low generalization, where a claim is tentative and is not held to apply to contexts other than the specific context studied. Such claims have low vulnerability to being rejected because of inadequate warranting. Since they are so tentative, only a modest amount of evidence is needed to warrant them. Further, as the claims are not generalized to other contexts, evidence can come solely from the context investigated.

You may already have noticed that a low certainty, low generalization claim is not vulnerable to the criticism of being under-warranted precisely because there is not much of a claim in the first place – a modest claim needs only modest warranting. But a claim that is both tentative and confined to a specific context is not likely to be of much interest to most readers, because they are trying to establish the reliability of the claim in relation to the contexts that they work in themselves. Having identified a low certainty, low generalization claim, you may set your expectations about warranting correspondingly low. But of course the claim will not be much use in establishing what is definitely known, let alone how far this tentative knowledge can be applied to diverse contexts.

So, as a critical reader, be alert to low certainty, low generalization claims. Telltale signs are qualifiers like ‘it may possibly be the case that …’, ‘it might be applicable only here’, but authors may be more subtle. Sometimes only the absence of a more confident or generalized claim indicates how limited specific claims must be. Yet even when the claims are of low vulnerability, they still bear checking for adequacy of warranting. Unusually, you may judge that the authors have been more modest than they need be. Perhaps you know from your reading of the literature that evidence from other studies corroborates their findings. Collectively, that information may enable you, when you come to write, to express a greater level of certainty and generalization than they did.

UNDER-AMBITIOUS WRITING: LOW CERTAINTY, LOW GENERALIZATION

As a self-critical writer developing your argument, beware of playing too safe. You risk your work being dismissed as under-ambitious and so failing to find out anything important. Critical readers will be most interested in claims with wide significance for the area of enquiry. If you are writing a paper for presentation at an academic conference or your dissertation, you may be expected to demonstrate robust claims to important new knowledge. So your claims need to be every bit as certain and generalized as you can adequately warrant using the evidence you can produce in support, both from your own investigation and from other literature. Gathering that evidence, of course, involves designing a study to take into account from the beginning your eventual need to make claims with the highest warrantable degree of certainty and generalization (addressed in Part Three of this book). But bear in mind that you can also predetermine the level of vulnerability of your eventual claims to an extent by careful choices in the way you word claims, since the degree of vulnerability changes as one moves along the continuum in each dimension.

We have already noted how the two dimensions portrayed in Figure 7.1 vary independently. So a claim could conceivably be of low certainty and high generalization (‘might be the case and applies universally …’), or high certainty and low generalization (‘is clearly the case in this context but its wider applicability has yet to be demonstrated’). Such claims are moderately vulnerable to rejection because they are ambitious along one dimension and play safe on the other. Equally, claims may reflect other positions, such as moderate generalization.

SIGNALLING THE DEGREE OF CERTAINTY AND GENERALIZATION

Wallace (Appendix 2) makes explicit both the level of certainty with which he makes his claims about effective sharing of school leadership and the extent to which he is prepared to generalize beyond the few settings in his research.

Early on, he raises questions about the risks that headteachers face when sharing leadership and about the justifiability of headteachers varying the extent to which they share according to the evolving situation (page 227). He then states: ‘The remainder of the paper seeks a tentative answer to these questions … ’. Tentativeness implies a relatively low degree of certainty over his claims to knowledge.

Later, having presented his findings, Wallace refers (page 234–5) to ‘… two features of the real world, at least in Britain’. Further, ‘The research implies that prescriptions for school leadership should be informed by evidence, and so rest on principles that are context-sensitive: the approach advocated will therefore be contingent on circumstances’. He articulates three such principles for the UK, then claims that: ‘These principles would justify British headteachers working toward the most extensive, equal sharing of leadership possible to maximize potential for synergy, while allowing for contingent reversal to hierarchical operation to minimize the risk of disaster’. Wallace generalizes to all schools in the UK but not to those in other countries, nor to any organizations other than schools.

In combination, his claims are tentative and are only moderately generalized beyond his own research settings – to other schools that he judges to be affected by the same contingent circumstances. His assessment about the limit of this generalization reflects his belief that central government reforms affecting all UK schools (but not, of course, non-UK schools or any organizations other than schools) played a critical role in the outcomes he observed. In Figure 7.1 his position might be located between the upper and lower left-hand cells: low certainty, but a moderate degree of generalization.

Claims to watch out for are those embodying recommendations for improving practice. They tend to make the strongest claims to knowledge, often combining a high degree of certainty with a high degree of implicitly or explicitly expressed generalization, at a high level of abstraction. As you would by now expect, they fall into the lower right-hand cell of the diagram. Popular ‘how to do it’ management books typically make high certainty, high generalization claims along the lines of ‘effective managers are visionaries who inspire others to go the extra mile to realize corporate objectives’. But whatever the position of a particular claim along the two continua, it will have some level of vulnerability to rejection. So, in sum, an incisive way to evaluate the major claims you come across as a critical reader is to:

• identify the degree of certainty and generalization of claims; and
• check how well this degree of certainty and generalization matches up with the amount of appropriate evidence employed in warranting them.

Conversely, as a self-critical writer, you will wish to make your writing robust to the demands and expectations of the critical readers of your work. Be cautious about asserting greater certainty over your claims to knowledge than you have evidence to support and about making broad generalizations – except perhaps at a high level of abstraction.

Armed with a sense of how to check for the match between claims in a conclusion and the warranting used to try and make them convincing, it is time now to get out of an argument. Let us move on to describing the other three components of your mental map.