Next week, thousands of final year medical students will sit a Situational Judgement Test (SJT) as part of the application for their first medical jobs. This will be the second year that the Foundation School Application System (FPAS) has used the SJT, which was developed by the Improving Selection to the Foundation Programme group (ISFP), and replaced the resource-intensive and perceived-unreliable short-answer questionnaires of the previous application process.
The process of allocating jobs is one which matters greatly to the finalists – it’ll spell out the promises and difficulties of two pivotal years in their training, and both the content and location of those posts are critical. I know, because this time last year, I was in exactly their situation, as part of the first cohort of students subjected to the SJT.
In a search for the perfect application/allocation system, academic performance seems like an obvious consideration. There are many practical difficulties with any ‘universal test’ of medical knowledge – not least that of timing – sit the test any time before the end of medical school, and different schools will have covered different patches of course material, or sit it after graduation and leave only a couple of weeks for the whole process of job allocation, contract-drafting, and life-planning to ensue. These are the banal, but undeniable practical difficulties with academic testing.
A more subtle concern however, arises if you make the assumption that medical students are attracted to work in better hospitals. This is hardly a contentious assertion, but it has significant ramifications. Academic high-performers are perhaps even more likely to want to work in centres of academic prestige. If we allocated jobs on the basis of academic merit – by whichever method – then it is conceivable that good hospitals would get the best applicants, and thrive, and less attractive hospitals would end up with mostly lower-performing doctors, reinforcing the cycle. This isn’t a trivial assertion – it’s already noted that there’s a small but significant increase in hospital mortality rates when junior doctors start in August – allocation of junior doctors likely plays a part in this phenomenon.
However, ISFP tells us that “non-cognitive attributes constitute essential requirements for being a doctor”. That is to say, an academically brilliant medical student could well make a brilliant doctor, but equally, they may have terrible communication skills, poor ethical judgement, and perform badly in their jobs. Academic ability is not sufficient as a criterion for medical job selection.
Enter the SJT. ISFP hail the SJT as an all-round better predictor of job performance. If at this stage you’d like to get a better idea of the format – head over to the FPAS website and try some sample questions. Here’s how they describe it themselves:
- Professional attributes are targeted such as coping with pressure, teamwork and effective communication.
- SJTs … correlate significantly with cognitive ability and also with some elements of personality, including agreeableness, conscientiousness and emotional stability.
It targets key qualities in their ‘person specification’, including:
- Commitment to professionalism
- Displaying honesty
- Knowing how to respond when you make a mistake, e.g. providing wrong medication to patient
- Identifying that a patient’s views and concerns are important
- Taking responsibility for own actions
All of which are obviously, and uncontroversially very important qualities for being a junior doctor. And ISFP are pretty sure that the SJT works:
- In selection for postgraduate training in UK General Practice, ongoing evaluation consistently shows that the SJT is reliable and effectively predicts performance in … subsequent assessment.
- There is a great deal of evidence to support their [SJTs’] use as part of selection processes.
Assuming that all this is true – we surely have a catastrophe on our hands. In a world dominated by academic selection, there are good hospitals, populated by academic physicians, some of whom are good doctors, and others who are not – in the SJT-world, there are good hospitals, brimming with communicative, honest, and ethical professionals, and less-good hospitals, filled with demonstrably devious doctors – clinicians for whom there is ‘a great deal of evidence’ to support their ineptitudes (relatively speaking).
For ISFP however, this wasn’t enough, and the current system is a fantastic chimera of both academic performance[1] and SJT results. I needn’t spell out the implications.
Where ISFP seem to have missed the mark is their apparent misappraisal of the allocation process. ISFP are very vocal about the fact that SJT is a tried and tested selection method in other industries – including the FBI and the police force. They note, correctly, that it has been a useful adjunct to post-graduate medical selection also – GP and public-health trainees have already been subject to SJTs, and this year sees its introduction to core medical training. In each of these instances, there are many more applicants than there are places, and there is good reason to choose the best whilst eliminating the worst.
FPAS, however, is the allocation of x number of jobs to x number of applicants[2] – it is not a selection process, it is an allocation process, and the criteria are different. If you accept my reasoning with regards to medical students wanting to work in better hospitals, then it follows that any allocation system which combines preference with some correlate of ‘doctoring ability’ will fall foul of the problems described above. It is worth emphasising at this point the fact that admission criteria to the foundation programme are competence-based, i.e. medical students must have reached a threshold degree of competence, as outlined by the GMC, and judged by the medical school, e.g. on the basis of passing finals. One could argue that the SJT could highlight applicants who require further training, but this seems like a task more suited to the local level.
There are two main stakeholder groups in the allocation process:
- The public – who would like an equally good chance of getting the best care, and for whom a randomised allocation system would serve well
- The applicants – who want choice over both the location, and content, of their foundation years. For some, location will be most important, for others, the rotations will trump location. There’ll be specialities they must experience, and others they want to avoid at all costs.
ISFP seem to emphasise the forward-looking aspects of application, talking extensively about ‘predictive validity’. However, the nub of the matter for medical students is more backwards-looking than that. On what basis can they compete with their peers to fill the most attractive placements?
I’d be a fan of combining preferences with randomisation[3]. I know very well that no matter which selection method was used, medical students up and down the country would be awake night after night revising, or practising tiddlywinks, or perfecting their poker skills, if it meant that they had a better chance at their ideal foundation placement. Medical students are fiercely competitive and fully immunised against the arbitrary. Randomisation[4] might be healthy during a myopic process so otherwise governed by box-ticking and inane fact-collecting.
The SJT may be good at determining which students will eventually make the best doctors. Being machine-marked instead of hand-marked, it is also assumedly cheaper than the previous short-answer process. These features however do not seem to justify its use in the allocation system as it stands today. I’d be interested to hear thoughts about how the SJT is currently being employed, and any alternatives people have to offer?
– – –
Footnotes
[1] Academic performance is given as a student’s decile-ranking within their medical school – a whole host of issues arise, none of which I’d like to address here.
[2] Unfortunately, in recent years, there have been slightly more applicants than places in the first round of applications, but the hope is that every eligible candidate is eventually allocated a job. Even so, the number of applicants per post is still much closer to 1:1 than the post-graduate and non-medical applications mentioned.
[3] As for my two-cents on an alternative process, assuming that location and specialities are the primary considerations, three measurements of preference are needed:
- Specialities they absolutely want to do, and those they really need to avoid
- Locations they’d want to work, in order of preference
- Some marker of whether they value speciality or location more highly, e.g. on a scale of 1 – 10
Given this preference data, applicants then need ranking – as I explain above, I think randomisation would be the healthiest option. With the addition of the third criterion to the preference system, the increased granularity should decrease the overall effect of randomisation on the allocation process.
[4] Some would see randomisation as a significant disempowerment of the applicant – however, this only matches ISFP’s assertion that the SJT can not be meaningfully prepared for.
Dear Christian,
I hope that cardiology isn’t too traumatic!
Back to the subject, may I suggest that the SJT is highly random in its function and, indeed, that is its intent? Reading the reports as to its success seemed to pivot on the spread function rather than correlate to any standardised measurement of personality or future practice. The re-allocation of scores based on students’ selections of certain answers reinforced the view that it had relatively little to do with objectively ‘correct’ answers and more to do with regression to the mean, whatever the mean may be. The determined misuse of mathematics, such as describing the SJT vs academia balance as ‘equal’, illustrates either a tragic grasp of basic maths or that they think we have a tragic grasp.
Of course, it does seem to have some sensitivity to students with poor judgement in my experience (disclosure: my SJT score was 38 with an academic of 47). But the massive effective score allocation (50) versus 16 for ‘academic’ seems to reinforce the limitation on academia.
As for preventing a cycle of concentration of the best to the best hospitals and the opposite, surely one can design the job rotations for this (and, I believe, they already have)?
Look at my rotation: cardiology/geriatrics/psychiatry in a trust listed in the Keogh report, then neurosurgery/orthopaedics/paediatrics in the JR, one of the UK’s most famous hospitals. Practically every F1 I know in Buckinghamshire will be going to Oxford next year, and the few rotations which put you in Oxford for 2 years were notably general in scope; if you want Oxford, you’ll have to pay for it one way or another.
None of this is rigorous research, just a synopsis of misquotations, rumours, tales of ‘the good old days’ and some sporadic Internet trawlings for data!
Thanks – on a practical level, I agree with you. My biggest miff with FPAS is the flagrant mis-selling of the SJT as ‘equally balanced’, or somehow a systematic way of allocating places. In reality, it does feel like a randomiser, though it will be interesting to see what follow-up will reveal down the line. My point was less practical than that, and I guess relates to general questions of how a selection process differs to an allocation one. However, again, I agree, with FPAS, it does seem as though the problem can be mitigated by way of rotation-selection. A real-world result of the general problem would be that less competitive fields might have a higher proportion of frankly incompetent individuals working in them, and therefore progress at a much slower rate. Not naming any particulars, you can guess which specialities I’m thinking about.
In any case, you’d be on MFoP at the moment – how is it? Cardiology sure is a change of pace from psychiatry…
Comments are closed.