Two Experiences of REF ‘From the Other Side’

The results of REF 2021 will be published on 12th May. In advance of this, we caught up with two Westminster researchers who sat on REF panels this year, Roland Dannreuther and Ian Locke. Each share an interesting insight into the other side of the REF process.

Roland Dannreuther is a Professor in the School of Social Sciences:

I have now had the experience of being on two REF panels for Politics and International Studies – for REF2014 and REF2021. So, I thought I could reflect on some of the differences and similarities in the REF processes over these two cycles.

The first major difference was that REF2014 took place in the prelapsarian conditions when all meetings were physical, there was no
option to attend virtually or in hybrid format, and the panel met regularly in hotels across the country of distinctly varying quality. REF2021 was almost all conducted through Zoom and I only attended one meeting physically – the sole meeting prior to the Pandemic.

There were some advantages of doing business via Zoom. It was more efficient and cut out a lot of unnecessary travel and administration. But it did make the process less personal and more technocratic. There was a sociability to REF2014 – the discussions in the corridors and over lunch; the satisfaction of the post-meeting drink and a gossip; and getting to know a
diverse group of colleagues from across the discipline. There also seemed to be more humour and jokes in the physical meetings as compared to those on Zoom.

Overall, though, the processes and dynamics of discussion and decision-making were pretty similar over the two REF cycles. And I did make good connections with the other members of the panel in REF2021 and the debates and discussions were always well-informed and engaging.

Another difference between the two cycles was in the variation of outputs. In REF2014, everyone had to submit 4 outputs and, in
practice, staff were often excluded if they did not have sufficient outputs. In REF2021, everyone had to be included but could submit from 1-5 outputs. As a result, only a minority of individuals submitted four or more outputs and there was great variation at the indvidual level. This did have an impact on the nature of assessment we did in REF2021. There was a distinct shift from
assessing individuals to assessing units of submission. The result was certainly more inclusive and many more individuals were submitted, but it does create de-facto inequalities within submissions which was not present in REF2014.

The other changes were relatively minor. In REF2014, we spent a disproportionate amount of time discussing how we should assess impact case studies since this was new to the exercise. By REF2021, the impact agenda were less novel and the panel was more comfortable about the criteria for assessment. There was, unlike in REF2014, no Impact template and instead we received institutional statements which were rarely other than turgid to read.

In the Environment Statements, there was more attention on EDI, which was commendable and significantly less tokenistic than in REF2014. There was also more focus on interdisciplinarity, though it is questionable whether this has resulted in a greater consensus of how this is to be defined across the different disciplines.

Overall, though, there was much that similar between the two REF cycles. Reading and evaluating the roughly 400 outputs (90 of which were books) was a slog and appeared at times Sisyphean in scope. REF2021, with its 3-month delay, meant that much of this had to be done over the summer. But hard as this was, there was also a privilege of reading such a volume of high quality work and learning so much about the research undertaken within the discipline. There was also enjoyment in the discussions you had with your co-markers about how to grade the outputs. REF output reading is generally more satisfying than reading and marking student essays – the REF outputs are usually well-written and you are not required to give any feedback or even feed forward.

The Impact Case Studies and the Environment Statements were less numerous but were more difficult to evaluate and grade. This is where collective discussions across the panel were most useful since all of us felt that we were sometimes being asked to compare apples and oranges. It was through quite an extensive deliberative process at the panel level that procedures and practical guidelines emerged for how to adopt the formal criteria to such a diverse range of cases and environments.

This will be my last REF cycle. Like the US Presidency, you are only permitted two terms. Although there are lots of reasons why being on the Panel is a significant burden, for which you are poorly compensated, it is ultimately a rewarding experience which requires you to read high quality work and brings together scholars in your discipline area who would otherwise have little reason to be together.

Ian Locke is a Principal Lecturer in the School of Life Sciences:

As a research active member of staff in the Biomedical Sciences, I was first submitted to the last ever RAE in 2001, not long after joining the university of Westminster, and then to subsequent REFs (including this one) over the last 20 years. As such I suppose I was an ‘end user’ of the REF process although I had acted as an internal output reviewer as part of Westminster’s preparation for REF 2014 (and was doing so again for REF2021) but that changed when subpanel 3, Allied Health Professions, Dentistry, Nursing and Pharmacy, found itself in need of more assessors and I was nominated and offered an appointment to the panel.

Once appointed to the panel I had to excuse myself from the internal preparations to ensure impartiality so apologies to colleagues who ended up with more outputs to look at but if I remember rightly, we had nearly finished the process at that point so hopefully the extra workload was not too onerous!

As with so many things over the past couple of years, Covid made it’s mark on REF 2021 too. Firstly the submission date was delayed by three months, shifting the whole time frame of the process somewhat. Secondly, all though there were regular expressions of hope that we may be able have ‘physical’ panel meetings at some point, Covid failed to ‘play ball’ and these did not happen, with the whole process taking place online over Zoom. As this was my first experience on a REF panel, and online meetings had become pretty ubiquitous by then, I found the ability to join the meetings from home all very convenient and thought that the format worked really quite well with plenty of breakout rooms available to engage in smaller group discussions when needed. Colleagues on the panel who were seasoned veterans of previous REFs lamented the loss of the social aspect of in-person meetings but I however, knew nothing else!

Despite the limitations of being online, we were still however able to have spirited full panel discussions and in depth discussions over individual outputs in the smaller breakout rooms and I think we did all the submissions justice. From my point of view it was a privilege to read some outstanding contributions to knowledge in the Biomedical Sciences field (just over 600 of them in fact) and then discuss with others who were just as passionate about the subject area. On the face of it, sub-panel 3 seems to cover quite a broad subject area, and whilst that is the case to some extent, the diversity of expertise on the panel more than matched the diversity of the submissions and to me this was an added bonus. As an academic I feel it does me good to move a little out of my comfort zone once in a while and discuss things with those that are looking at the same data but from a different perspective.

All in all, being a part of REF 2021 was an extremely rewarding experience, despite the significant workload in a somewhat restricted time-frame. If there is another REF (a perennial topic for debate), and if I’m invited to join the panel again, I certainly will!