1. PISA (the Programme for International Student Assessment) has been running since 1997 and the first testing cycle was in 2000, meaning there have been six tests in total so far. It is run and funded by the OECD (Organisation for Economic Co-operation and Development), which currently has around 35 member countries and 37 partner countries.
2. Students who take the PISA tests must be between 15 years and 3 months and 16 years and 2 months old – not necessarily all in one school year. Only students who attend school (not home-schooled students) are tested.
3. PISA testing happens every three years, but the main focus alternates between reading, mathematics and science. Because it’s not feasible to test all pupils in all subjects every year, PISA has an imputation algorithm that estimates student scores to test questions. For example, in 2006, it has been calculated that about half of students did not answer any reading questions at all but data was created for these students using an adapted Rasch model.
4. The PISA test takes samples in two stages: firstly, they use a stratified sample to select schools containing 15-year-olds using probability proportional to size. Then a simple random sample of 35 pupils is chosen from each sample school. If a school does not respond, it is replaced with one of its two nearest neighbours on the list; if a pupil does not respond, they are not replaced.
5. In the 2003 round of testing (mathematics), the UK was disqualified from analysisbecause the OECD felt that the data for England suffered from response bias as they were lower than their acceptable thresholds of 85% for school response and 80% for student response. They also suggested the 2000 results were similarly problematic but further investigation convinced them this was ‘negligible’.
6. In 2014, 83 academics and educators from around the world wrote to Andreas Schleicher, calling for him to skip the next PISA cycle and make six more proposed changes to the system (you can read it here). The tests went ahead as planned in 2015 (results were published this week and can be found here).
7. The OECD have awarded a contract to Pearson (a large commercial education corporation) to define what will be measured in PISA 2018. They have been working with McGraw-Hill Education (another commercial US-based assessment company) since 2013.
8. China does not currently participate as a single country (rather as separate provinces such as Shanghai and Beijing) but is expected to in 2018 ‘as a sign of increased co-operation with the OECD’. Critics have suggested some of Shanghai’s perceived success in PISA rankings is due to its skewed population demographics; Schleicher has suggested 27% of Shanghai’s 15-year-olds are excluded from its school system (as they often return to their hometown for high school due to movement restrictions known as hukou).
9. PISA testing used to be done exclusively on paper but has now moved to a system where computers are mostly used– the OECD says that computer testing is now ‘the default’, but that ‘for the small number of countries who were not ready for computer-based delivery it was possible for them to take the tests on paper’. There have been criticisms of this, suggesting that this change could have had an impact on scores (particularly in East Asian countries) despite the OECD suggesting they have accounted for the change carefully.
10. Countries can exempt students who are chosen on three grounds: physical disability (that limits their ability to write the test); intellectual, mental or emotional disability; or limited proficiency in the assessment language. The UK had the highest national exclusion rate in 2015 with 8.2% of students chosen being exempted. The OECD states that anything up to a 5% level ‘of the relevant population’ is permitted.