Designing a placement test

Using statistics to prepare an item analysis on a Placement Test
measuring the facility & discrimination of each item

TESTING THE TEST 1:

An ITEM ANALYSIS measures the difficulty level i.e. facility & discrimination of each item in the test.

1. I administered my Placement Test to 24 Swedish Pensioners.

2. I then took the TOP and the BOTTOM third of their scripts: 16 of them in all (8 from the top 33 % of the scripts and 8 from the bottom 33% - the rest are discarded) and prepared the following table to calculate the DIFFICULTY LEVEL and the DISCRIMINATION index for each of the 28 items in my test:

N=16 (N is the number of test scripts: 8 from the top-scorers & 8 from the bottom-scorers)

NOTE: T = from the TOP 8 scripts B = from the BOTTOM 8 scripts (c=correct) (x=wrong) (-= pass)

Item		c	x	-	Facility	Discrimination	Item		c	x	-	Facility	Discrimination
1	T	8	0		0.93	0.12	15	T	1	6	1	0.06	0.12
	B	7	1					B	0	6	2
2	T	8	0		0.81	0.37	16	T	2	4	2	0.12	0.25
	B	5	3					B	0	7	1
3	T	8	0		0.75	0.5	17	T	6	2		0.43	0.62
	B	4	3	1				B	1	4	3
4	T	8	0		0.81	0.37	18	T	5	1	2	0.31	0.62
	B	5	3					B	0	2	6
5	T	1	6	1	0.25	- 0.25	19	T	6	1	1	0.37	0.75
	B	3	5					B	0	3	5
6	T	8	0		0.93	0.12	20	T	3	3	2	0.25	0.25
	B	7	1					B	1	3	4
7	T	3	5		0.19	0.37	21	T	1	5	2	0.06	0.12
	B	0	8					B	0	5	3
8	T	7	1		0.56	0.62	22	T	4	2	2	0.25	0.5
	B	2	5	1				B	0	6	2
9	T	6	2		0.37	0.75	23	T	2	4	2	0.12	0.25
	B	0	5	3				B	0	2	6
10	T	8	0		0.62	0.75	24	T	0	3	5	0	0
	B	2	1	5				B	0	1	7
11	T	8	0		0.69	0.62	25	T	3	4	1	0.19	0.37
	B	2	1	5				B	0	3	5
12	T	7	1		0.56	0.62	26	T	0	3	5	0	0
	B	2	3	3				B	0	1	7
13	T	6	2		0.44	0.62	27	T	3	1	4	0.19	0.37
	B	1	2	5				B	0	0	8
14	T	7	1		0.62	0.5	28	T	3	3	2	0.19	0.37
	B	3	1	4				B	0	3	5

The DIFFICULTY LEVEL of each of the 28 items in the test can be expressed as a decimal (as in the above table).

Ignore the zero and the point and the DIFFICULTY LEVEL can also be expressed as the percentage of the students who answered correctly.

It is calculated by [T(correct)+B(correct)] divided by N (The Number of scripts you are sampling. This means for item 14 of my test you add (7+3) and divide by 16.

The DISCRIMINATION for each item can also be expressed as a decimal (as in the above table) or as a percentage.

It is calculated by [T(correct) - B(correct)] divided by 1/2 N (half the Number of scripts you are sampling. This means for item 14 of my test you subtract (7 - 3) and divide by 8 (which is half the Number of scripts).

Notice that there is a very strange discrimination value for item 5 of the test. Only ONE of the eight top students provided the correct answer, while as many as THREE of the eight bottom students managed to provide the correct answer.

I would be reluctant to scrap this item before administering the test to a broader sample of students. I would expect students at post-elementary level to know that the answer is "at".

Since this placement test is set on an incline of difficulty, items near the beginning and near the end of the test should have a high facility value and a low facility value, respectively. The items in the middle of the test will be more useful for purposes of discrimination. In a placement test, you need a few questions which learners who have done a little English can answer correctly, so they can at least show where they have got to on the scale.

Questions that nobody can answer correctly are of little value, except in showing that you are not in the business of teaching learners with extraordinarily high levels of proficiency in English or who share the idiosyncrasies of the question-setter.

But I would also be reluctant to scrap the two items at the more difficult end of the test, which none of my Swedish pensioners answered correctly:

Item 24: This (the travel card) is usually issued for ________ of one or four weeks.

I would expect a student who has successfully completed the Cambridge First Certificate (an exam set at the higher intermediate level) to get "periods" for item 24 and "increases" or "rises" for item 26.

I have given the same test to a sample of 52 students from mixed nationality classes.

The test was re-drafted five times so that weak items could be scrapped, but items 5, 24 and 26, which appear in the final draft have been kept.

I would not be content with my facility and discrimination percentages if I was administering an end-of-course achievement test to students who were grouped at the same level at the beginning of their course.

If the facility is higher than 0.9 (90 %) for an item in a progress or achievement test, then the item is too easy (unless the aim is to boost students' confidence by providing one or two give-aways at the start of the test. If the facility value is lower than 0.3 (30 %), then the item is probably too difficult and should perhaps be redrafted.

Aim for a minimum discrimination value of 0.15 (15 %), when writing a progress or achievement test.