BAM responds to REF consultation on draft assessment criteria and working methods

As part of its remit to advocate on behalf of the business and management research community, the British Academy of Management has submitted it's reponse to the consultation on the REF draft assessment criteria and working methods.

The final panel criteria and working methods will be published in early 2012.

For more information on the draft, and links to download the documentation, please see http://www.hefce.ac.uk/research/ref/pubs/2011/03_11/.

Please read on to see BAM's responses (in blue):
 

 

Generic questions

1a
Overall draft panel criteria and working methods
The generic and four main panel statements achieve an appropriate balance between consistency across the exercise and allowing for justifiable differences between the four main panels.

Agree

1b
Are there particular aspects of the criteria and working methods that should be more consistent across all the main panels? Are there differences between the disciplines that justify further differentiation between the main panel criteria? Where referring to particular main panels, please state which one(s).

We congratulate HEFCE and the four panels. Overall there is good balance between consistency and recognition of differences between subjects covered by each main panel. However, in our view the treatment of reserve item between main Panel C and D is not justifiable. We suggest reconsideration of treatment of "reserve item" by Panel C.

2a
Individual staff circumstances
The proposals for determining the number of outputs that may be reduced without penalty, for staff with a range of individual circumstances, are appropriate (Part 1, Tables 2 and 3).

Agree

2b
Please comment on these proposals. Respondents are also invited to comment specifically on:

  • whether Tables 2 and 3 are set at appropriate levels
  • the proposed options for taking account of pregnancy and maternity (Part 1, paragraph 62)
  • whether a consistent approach across the exercise is appropriate, or whether there are any specific differences in the nature of research that justify differences in the approach between UOAs or main panels.

If commenting in respect of particular panels or disciplines, please state which.

We welcome the recognition that circumstances may constrain the ability of staff to produce 4 output during the period covered by REF. This will address "equal opportunity" issues. We are however concerned by the proposal that a woman with less than 14 months of maternity leave would not normally be entitled to any reduction in cycle. In our view this discriminates against those who have to return to work following expiration of statutory maternity leave for economic or family circumstances.
 
We are also not clear how circumstance related to a complicated pregnancy or double pregnancy may be taken into consideration. We urge a reduction in tariff and greater clarity. In our view reducing the number of output by 1 for each maternity leave has the advantage of simplicity provided this does not prejudice also submitting more complex circumstances that might further reduce the submission (e.g. complication around birth). We assume lengthy paternity leave will be considered under other circumstances.
 
We also believe an ECR need more time to produce their first output and suggest revision to Table 2 offering more time for ECRs to produce their first output. This is particularly important because ECRs are not likely to contribute to impact and by setting high output tariff they may be excluded from REF. This will have a detrimental impact on discipline's future research capacity and career of ECR.
 
We are also concerned with the definition of ECR as it only includes colleagues who are taking up their first appointment as a lecture. It discriminates against those colleagues who are "new to research", but have been employed (mainly as teachers in HE or FE) for a longer time. A potential indicator for this group of staff may be the date that they complete their PhDs.
 
We are concerned with paragraph 64 these are complex issues and the penalty proposed is too severe. As there is no penalty for submitting only a proportion of academics from any one institution perhaps if the panel decides the reduction on output submitted is not appropriate, they should discount the individual as a whole rather than (effectively) reducing the institution's quality rating.

 

Main panel C criteria and working methods

C3a
Main panel criteria and working methods
The main panel statement achieves an appropriate balance between consistency and allowing for discipline-based differences between the sub-panels.

Agree

C3b
Please comment on the balance between consistency and allowing for discipline-based differences between the sub-panels within this main panel. Please state the UOA(s) on which you are commenting.

In our view there is a good balance. We also welcome the effort to minimise cross referrals and instead use additional panel members. In our view this approach will produce a more consistent and robust outcome.

C4a
Submissions and units of assessment (Section 1)
Do the UOA descriptor and boundary statements provide a clear and appropriate description of the disciplines covered by the UOAs? Please include any suggestions for refining the descriptors and state which UOA(s) you are commenting on.

As the consultation document recognises there is likely to be significant overlap between UOA 18 and UOA 19. Having one individual serving on both panels is likely to enhance consistency. On the other hand in subjects like business history ontological and epistemological difference s are likely to play significant role and output may be judged differently legitimately. We welcome the suggestion that final decision lies with the primary UoA and suggest this point to be strengthened. In our view boundaries of UOA 19 and UOA 26 also overlap closely and we believe there is a need for its recognition in advance.

C4b
Please comment on the main panel's criteria in relation to multiple submissions in its UOAs.

We welcome the criteria. There is need for greater clarification, for example, potential justification for such submissions. Also greater clarity if an output is authored by 3 scholars can the HEI submit it three times? In short greater clarity.

C5a
Assessment criteria: outputs (Section 2)
Overall, the main panel criteria relating to outputs are clear and appropriate.

Agree

C5b
Please comment on the criteria in Section 2, in particular on where further clarification is required or where refinements could be made.

We are concerned with implications of para 44 and 45. Publication lead-times are considerably longer in B &M compared to stem subjects. It is not unusual for researchers to produce working papers. We strongly suggest clarity here. In particular working papers by single authors which appeared before 2008 but were not submitted to RAE 2008 should be eligible for submission to REF 2014 as final paper. By the same token if an author did not submit a working paper to RAE 2008, then they may submit the final paper to REF 2014, irrespective of what co-authors at other institutions did. Finally, examples of publications that may be discounted is helpful.

We welcome the proposal for recognising "double-weighted" outputs. However, we suggest that there is a need for In our view further clarification and definition of double weighted out is required. The current "exceptional research output" is too vague. We also suggest any further clarification should recognise the potential for double weighting of "research monographs", fragmented texts on bases similar to that of UOA 17. We suggest allowing of reserve in line with Panel D.

Finally, we think there should be a clearer statement in terms of work that will be read, that monographs and chapters will be judged on their merits and all this is tied to a clear an unambiguous statement and no journal ranking will play any role in assessment of output.

C6a
Assessment criteria: impact (Section 3)
Overall, the main panel's criteria relating to impact are appropriate and helpful to institutions in preparing submissions.

Agree

C6b
Please comment on the criteria in Section 3, in particular on where further clarification is required or where refinements could be made.

In our view the criteria for the assessment of impact is appropriate. However, in our view it would be useful if further information on the range and type of impact was provided. In addition examples will be useful. We suggest that a protocol similar to that of Panel D. As I read it, it is possible for an institution to claim impact for a researcher who conducted the research at that institution but left before impact was evaluated but not claim impact for a researcher who conducted the research before coming to that particular institution, even when imp[act occurred in the relevant period. That doesn't strike me as quite fair because you cannot otherwise submit an academic who is not actually at the institution in question? Further clarification would help.

If the panels are going to leave it open for institutions to decide themselves what might constitute impact then there is (as above) a problem with any penalties for institutions who may have conceived impact "wrongly" (from the panel's point of view).
We also believe there is a need for clarity as how the Panel will address the possibility that evidence may no longer exist.

C7a
Assessment criteria: environment (Section 4)
Overall, the main panel criteria relating to environment are clear and appropriate.

Strongly Agree

C7b
Please comment on the criteria in Section 4, in particular on where further clarification is required or where refinements could be made.

We welcome the structured approach.

C8a
Working methods (Section 5)
Overall, the working methods of the main panel and its sub-panels are clear and appropriate.

Agree

C8b
Please comment on the working methods, in particular on where further clarification is required or where refinements could be made.

Except, as above, not clear how judgements will be made to downgrade where UOAs may have made an honest mistake in knowing what to submit.

Upcoming BAM Events



  See all BAM Events

 

Job Advertisements

_____________________

     
University of Ulster - 8 Positions Available
Closing: 21st May 2012

_____________________
Advertise here

Facebook