Author and test writer Pauline Cullen speaks with Senior Teacher: Assessment, Megan Yucel

Pauline Cullen is a freelance test writer and author of 5 IELTS preparation books including the Official Cambridge Guide to IELTS and two Cambridge Vocabulary for IELTS books. In 2016 Pauline launched her own IELTS Vocabulary teaching apps.

Thank you for your time today, Pauline. Test item writing is a very specialised field. How did you first become interested in it?

I was originally a French and Spanish teacher in a UK high school and so had to prepare classes for O levels and A levels, and I was also put in charge of teaching ESL within the school. When I moved to Australia I began teaching ESL fulltime, and I was naturally drawn to the exam preparation classes. This was in 1988, so pre-IELTS, and the main tests at the time were FCE and CPE, or TOEFL and TOEIC. The language centre where I worked steadily grew in size and I was very involved in testing and placement right from the start. I really liked that side of my role, so I was writing in-house tests for a long time before I was writing them professionally. As a teacher, I like the structure that a test gives to a class, as well as the fact that a formal test gives a course some sense of an ‘end’. We would often have students studying with us for 6 months or more, and so this idea of having completed a stage in the learning process was something I found really useful.

In the early 1990s, I became an oral examiner for Cambridge Main Suite exams, and I can remember being really interested in the idea of IELTS right from the beginning, because we had a lot of EAP classes, and we knew that IELTS would bring to those courses what FCE and CAE had brought to our GE courses. I trained as an IELTS examiner in the very early days of the test. I’d been an examiner for about 2 years when I was approached to join the Australian IELTS test writing team in 1995. For my first commission, I had to write an Academic reading section. That first passage is now in Cambridge IELTS Test Book 3. I’ve
been involved in test writing since then.

There seems to be some controversy in testing circles about whether item writing can be seen as an art or science (see the discussion in Green and Hawkey, 2012, for example). On the one hand, item writing requires creativity and flexibility, while still keeping within the guidelines. On the other hand, item writers need to be fairly prolific, and able to quickly produce items that are standardised and reliable. What’s your perspective on this?

I think that this is a really interesting question and discussion to have. When it comes to perspective, in the last few years I’ve become more and more aware of just how important it is to understand the perspective of the person who is speaking or writing. Even when we’re writing an academic textbook, the ideas and materials will always in some way reflect the writer’s personal perspective - their background and experiences. So, I do think the different perspectives of people working in the language assessment field are interesting, and I think you will get different answers to that question depending on whether you are talking to a test writer or, for example, a statistician who has never had to write test materials. I can only give my test writer perspective, which is that writing reliable test materials requires both science and art. If there wasn’t a creative aspect, I certainly wouldn’t have been able to stick with it for over 20 years. The science comes from the technical aspects and, as you say, the need for standardisation. It takes years to learn and hone the skills to be able to produce test materials that are fair, valid, and reliable. The art (and perhaps the issue that is least understood) comes from the fact that the specifications and guidelines which item writers are given, are not a set of instructions or some sort of recipe that can easily be replicated with just a few variations. They are closer to a theory of how testing works that then needs to be applied in a skilful way to create something new each time. There is an art to that. We are repurposing texts, but to be able to do that, you need to be able to see the possibility within the original as well as the ‘shape’ of what you can create from it. In listening, of course, the creative aspect is even more apparent, as we have to write and produce credible scenarios and scripts.

What’s your process when you sit down to write a test, for example, when you want to write a reading test? Can you share that with us?

The test writing process cannot begin when you are asked to produce a test. The process has to begin weeks, often months, before. When a writer estimates that they would spend half a day, or a day, finding appropriate materials, they rarely mean that those hours were spent in one sitting. The time it takes is spread over many weeks, when we search habitually. We are never not looking for texts, and possible source materials. Because of that, any guesstimate about how long it takes to find a source text is wildly inaccurate, I think. When I first started, I used to constantly cut out and collect hundreds of articles, but then when it came time sit down to write a test, I would sift through them only to find that they were not as test-friendly as they had appeared at first glance. As a new writer, it is very easy to be initially impressed by a text only to find that it actually contains only one or two testable ideas. So, I soon learned not to print or even save anything unless I could see 12 – 13 testable ideas. The most important work is done at this text selection stage, before the writing even begins. You learn from painful experience that, if you try to rush this phase, then you have a high chance of having your material rejected. What is worse, those are the texts that you need to work on for far too long to be able to produce items that you hope will work, only to have the material ultimately rejected. The main problem with a lot of the test materials we see online nowadays is that the writer has begun with a source text that is not suitable for the task. The writer persists in producing questions, but these simply don’t work, because there aren’t enough salient points within the material to test on. It’s actually much more difficult than people imagine to find useable test materials.

Do you have any advice for teachers who might need to write their own tests?

My first advice would be to use good models. Your aim should always be to produce materials that are fair, valid, and reliable. So, make sure you make a study of test materials that reflect this. Once you have found test materials that you trust, look carefully at the different question types and how they work. It is easy to get the form, or the look, right, but if the questions don’t perform their required function then they aren’t useful – you will just end up with a list of questions rather than a testing tool. So, try to identify, or be aware of, the skills you have to use to get to the correct answer. Then, when trying to replicate that type of question, make sure you focus on testing those same skills, not simply on writing a question. You also need to be very aware of the difference between writing productive questions (where candidates need to write a word or words) and objective test questions (where candidates choose a letter from a list). There are very different skills involved in writing these different question types. New test writers also often produce questions that are what we call ‘tricksy’ or unfair. They often feel that this is necessary in order to create a test that is ‘difficult’. This is often because the source material is not suitable, and so to create an illusion of difficulty, the writer has to rely on trick questions. My final advice would be to practise, practise and practise, but make sure to get meaningful feedback. In order to produce test materials that are fair and reliable, you must get feedback from both students and from other language professionals. The difficulty comes in not seeing this feedback as criticism. A test will only be fair if others can see the same idea within the passage and interpret it in the same way as you. It is often a surprise to learn how others interpret (or misinterpret) your text or your questions. And it is important not to dig in and fight against their interpretations but to look again to see if you can make the idea clearer. Testing is very much about writing in a clear and precise way.

As a published author of IELTS preparation materials, you’re an expert on the IELTS test. I believe you also have an online platform which you use to communicate with students. Do you have any key pieces of advice that you give to students who are preparing to take the IELTS test?

I started to be active on social media in 2012. That was because I was disappointed in the sales of several of my books and I wanted to find a way to help publicise them. My contact with candidates, students, and teachers from all over the world has taught me a lot about the user experience and goes a long way to inform the talks that I give at conferences or in webinars. I quickly realised that there is a lot of confusion and misinformation about IELTS. It will sound naïve, but I really thought that once people could contact an expert, that would solve the problem, and everyone could then focus on teaching and studying in the right way to prepare for the test. I was actually really shocked to find that people were more than ready to tell me I was wrong and point to other ‘experts’ who have attracted a far greater audience, and whose views and theories completely contradict what I tell them. The root of the problem, in my view, is the examples that don’t represent IELTS – what I refer to as inauthentic test materials. These materials often have guessable questions and represent the test as a confusing and tricksy test. People have looked at these materials, which abound online, and drawn conclusions about the real test which are not accurate. They then give advice based on those conclusions, and produce even more examples to support their theories. So, a vicious circle is created, where the bad examples, lead to bad advice, which is supported through the production of more bad examples. Nowadays, we have to be careful to filter what we read or see in terms of news stories, and the same applies to IELTS. So, my main advice to students is always to use only authentic test practice questions. The issue of test validity is an important one when we are writing test materials. It is the validity that produces the desired washback effect to the classroom or the test preparation of the individual student. So, it shouldn’t be surprising that, in my experience, when people use inauthentic materials, where this idea of test validity has not been a factor, their preparation does not go far enough and so their test results do not improve. For example, in my experience, inauthentic reading test materials focus generally on matching vocabulary rather than testing reading skills, so students using these to practise do not develop the reading skills needed for the real test. Similarly, many of the writing tasks I have seen force students to write in a repetitive way and do not require them to take a position, so they don’t develop the writing skills that are essential for the writing test. So, my number one tip for both teachers and students is to use authentic materials as much as you can.

Great advice. Out of all your publications, do you have any particular favourites?

That’s a tricky question! I am probably most proud of my two Vocabulary for IELTS books, because they really reflect my own teaching style and the way that I would teach an IELTS class myself. The two levels focus on building language, and I would love the chance to develop them into an IELTS course book together with the Official Guide to IELTS, which focuses on developing skills. But I am also very proud of my two IELTS vocabulary teaching apps, and the free e-book I am writing at the moment. Over the last 5 years I realised that one of the main problems people have is knowing how to study language at a high level. Many of the people I deal with are doctors, who have not studied language in a formal way for a very long time, yet have to achieve a minimum of band 7.5 in all skills, including writing, which they really struggle with. My free book helps show them how to become a language learner again, and draws on my own experiences when I struggled at B2 myself when I first began studying for my Advanced level exams in French and Spanish. I had to learn how to write essays on serious topics like crime, and the environment, and so on, just as they have to in writing task 2.

Assessment literacy is a topic which seems to be attracting a lot of interest nowadays. In your experience, how much do you think teachers and students know about language assessment?

Again, it’s a very interesting point. When it comes to IELTS, I would say that the focus of the teachers and students I meet, either through my talks or through social media, is almost solely on scores and scoring – this is what I see in the questions I am asked. I
constantly have to try to bring their focus back to language and the skills being assessed. This was the aim of the Official Cambridge Guide to IELTS, to show teachers and students how to prepare for each part of the test and to put the emphasis back on skills and language learning so that genuine progress could be made. The focus for teachers and for students is generally on looking for patterns within questions. This is the wrong approach if you want to make genuine progress. I also think that perhaps there is a tendency to see the theory of language testing as somehow separate to the practice of language testing. So, while teachers may be very familiar with the guiding principles of language assessment from an academic point of view, they don’t always equate this or relate this to their preparation and teaching. It’s one thing to be aware of language assessment theory, but being able to apply this in the classroom in a meaningful way is a completely different matter. Again, that is what the Official Guide to IELTS aims to help teachers do. It’s really important not to see ideas like test validity, washback, and reliability as mere jargon. They really are at the forefront at every stage of IELTS test production. Understanding this means that you can begin to see that there is real benefit in teaching specific skills. Again, a lot of the problems here may have come about from the use of inauthentic test materials, which don’t follow, or reflect, these principles. In my view, those types of materials have had a very damaging effect on trust in the test, and perhaps on testing in general. If people are exposed to confusing and unreliable test materials, it is not surprising that their preparation becomes ineffective and that they stop thinking in terms of the assessment of language and skills.

Pauline, what trends do you expect to see regarding the future of language testing? What about test preparation, in terms of the modes in which instructional materials are delivered? As an author and test writer, how do these changes affect you?

In terms of test delivery, the future will surely lie in more computer-based testing. I welcome that, and I think it adds an extra dimension to language teaching and test preparation. Like a lot of writers, I now have to self-publish, and I’m already working on some materials that will be interactive and only available as e-books rather than a standard print version. That side of writing has always interested me, so it is something
I’m enjoying working on. I suppose that the future trends in language testing are following a pattern similar to other industries, and people will no doubt be hoping to create an algorithm that will take the place of the test writer. While it might be argued that this would produce a more standardised test, I would offer a counter argument that when we try to find some ‘hidden formula’ or a ‘recipe’ for producing any sort of content,
the result will be predictable rather than ‘standardised’. Once you have predictable testing materials, then you have an unreliable test, because the same ‘formula’ that is used to create such a test can be used to master it. What gives IELTS its high credibility now is the fact that there is no formula to follow – you just have to study, and master, the language and skills needed for academic or professional success.

MY: Thank you, Pauline, for an enlightening and thought-provoking discussion.

Originally published in Language Assessment Matters Issue 8, October 2017

Last updated:
12 March 2018