Validating survey data: Educational attainment in SDR and the OECD

Educational attainment data

Data on the education structure of the adult population (instead of gross or net enrollment ratios) are not that easy to get. OECD provides data on the shares of populations by educational attainment in the Educational attainment and labour-force status database, code EAG_NEAC. This database includes the indicator “Share of population by educational attainment” (NEAC_SHARE_EA), which is available for several education levels and for different year ranges. I chose “At least upper secondary education” (L3T8) and the age range 25-64 (Y25T64).

The SDR project harmonized selected variables on political attitudes, behavior, and basic sociodemographics, from 22 cross-national survey projects covering 142 countries/territories between 1966 and 2013. From the SDR v.1 dataset I selected only whole country samples (i.e., excluding split-ups like West and East Germany or Flanders and Wallonia). For calculating shares of samples with at least upper secondary education I only used the target variable “Education level” (t_edu), not “Schooling years” (t_school_yrs), which is also available in the SDR v.1 dataset. I used respondents in the 25-64 age range and applied calibrated case weights (t_weight_l1u_2). More information about how these variables were constructed is available in the SDR documentation.

Respondents with at least upper secondary education were identified with the following condition: (t_edu == 30 & c_edu_incomplete == 0) | (t_edu > 30 & t_edu < 90), where t_edu (the harmonized “education level” target variable) takes the following values:

0 = less than primary
10 = primary
20 = lower secondary
30 = upper secondary 40 = post-secondary non-tertiary
50 = short-cycle tertiary
60 = bachelor’s or equivalent level
70 = master’s or equivalent level
80 = doctoral or equivalent level
90 = not elsewhere classified

and c_edu_incomplete is a flag indicating whether the given level of education has been completed (c_edu_incomplete == 0) or not (c_edu_incomplete == 1).

After matching the SDR v.1 and OECD datasets (and losing some cases on both sides) I ended up with 542 cases (national samples) from 15 projects: Asian Barometer (ABS), Afrobarometer (AFB), Americas Barometer (AMB), Asia-Europe Survey (ASES), Consolidation and Democracy in Central Eastern Europe (CDCEE), Comparative National Elections Project (CNEP), European Quality of Life Survey (EQLS), European Social Survey (ESS,) European Values Study (EVS), International Social Justice Project (ISJP), International Social Survey Programme (ISSP), Latinobarometro (LB), Life in Transition Survey (LITS), New Baltic Barometer (NBB), and the World Values Survey (WVS), and for 38 countries (that the OECD dataset only has information for a limited number of countries).

library(OECD) # getting OECD data
library(dataverse) # getting data from Dataverse
library(tidyverse) # cleaning and reshaping data
library(countrycode) # switching between country codes
library(ggplot2) # plots
library(knitr) # for pretty tables (kables)
library(kableExtra) # to customize tables (kables)

dataset_list <- get_datasets()
search_dataset("education", data = dataset_list)

dataset <- "EAG_NEAC"

oecd <- get_dataset(dataset = dataset) %>% 
  filter(INDICATOR == "NEAC_SHARE_EA", 
                     ISC11A == "L3T8", 
                     obsTime != "9999",
                     MEASURE == "VALUE",
                     SEX == "T",
                     AGE == "Y25T64",
                     !COUNTRY %in% c("E22", "OAVG", "G20", "NMEC")) %>%
  mutate(iso3c = COUNTRY, year = as.numeric(obsTime)) %>%
  select(iso3c, year, obsValue)

master.raw <- get_file(3006244)
tmp <- tempfile(fileext = ".dta")
writeBin(as.vector(master.raw), tmp)
master <- haven::read_dta(tmp)

merge <- master %>% filter(t_age >= 25 & t_age <= 64) %>%
  mutate(hs = ((t_edu == 30 & c_edu_incomplete == 0) | 
                 (t_edu > 30 & t_edu < 90)) ) %>%
  group_by(t_survey_name, t_survey_edition, t_country_l1u, t_country_set) %>%
  summarise(mean_hs = 100 * weighted.mean(hs, t_weight_l1u_2, na.rm = TRUE),
            year = mean(t_country_year),
            prop_edu_mis = mean(is.na(t_edu) | t_edu == 90)) %>%
  filter(!is.na(mean_hs), nchar(t_country_l1u) == 2) %>%
  mutate(iso2c = substr(t_country_l1u, 1, 2),
         iso3c = countrycode(iso2c, 'iso2c', 'iso3c'),
         continent = countrycode(iso2c, 'iso2c', 'continent')) %>%
  inner_join(oecd, by = c("iso3c", "year")) %>%
  mutate(off = (mean_hs < obsValue * 0.5 | mean_hs * 0.5 > obsValue )) %>%
  filter(!is.na(obsValue))

Results

The correlation (Pearson’s r) between the SDR percentage of respondents with at least upper secondary education and the corresponding OECD share is 0.67. The scatter plot in Figure 1 visualizes this association. While most of the samples are relatively close to the 45-degree line (as expected), there is a worrying group of observations with survey proportions substantially lower than the proportions from OECD data. Red points indicate that the survey proportion is less than half of the OECD proportion (the 50 percent threshold is of course arbitrary).

ggplot(merge, aes(x=obsValue, y=mean_hs, col = off)) + geom_point(show.legend=F) + 
  scale_color_manual(values=c("black", "red")) +
  ggtitle("Figure 1: Educational attainment from SDR v.1 and OECD") + 
  xlab("OECD: % of population with at least upper secondary education") +
  ylab("SDR: % of population with at least upper secondary education") +
  xlim(0,100) + ylim(0,100) +
  geom_abline(intercept = 0, slope = 1)

Figure 2 presents separate plots for each survey project, showing that most of the problematic surveys are in the International Social Survey Programme (ISSP), but also in ASES, CDCEE, EVS, and WVS. This Shiny app enables protting the data by survey projects and makes available the data: https://mkolczynska.shinyapps.io/sdr_oecd_education/.

ggplot(merge, aes(x=obsValue, y=mean_hs, col = off)) + geom_point(show.legend=F) + 
  scale_color_manual(values=c("black", "red")) +
  ggtitle("Figure 2: Educational attainment from SDR v.1 and OECD by project") + 
  xlab("OECD: % of population with at least upper secondary education") +
  ylab("SDR: % of population with at least upper secondary education") +
  xlim(0,100) + ylim(0,100) +
  geom_abline(intercept = 0, slope = 1) + 
  facet_wrap("t_survey_name")

The curious case of ISSP Switzerland

ISSP surveys in Switzerland are a particularly interesting case. As shown in Table 1 below, the time series starts with survey proportions of respondents with at least upper secondary education (column “SDR % HS+”) of around 90% in 1996, drops to around 40% in 1998, stays low until 2010, and then increases back to 85% in 2011. Note that only selected waves of ISSP are included in SDR v.1. Also note that survey proportions from 2006 and 2007, and from 2008 and 2009 are identical; this is because these two waves were carried out together on the same sample.

merge %>% filter(t_survey_name == "ISSP" & t_country_l1u == "CH") %>%
  arrange(t_survey_edition) %>% select(1,2,3,5,10) %>%
  kable(digits = 3, align=c("c", "c", "c", "c"), 
        caption = "Educational attainment data from ISSP Switzerland and the OECD",
        col.names = c("Project", "Wave", "Country", "SDR % HS+", "OECD % HS+")) %>%
  kable_styling(full_width = F, position = "left") %>%
  column_spec(1:5, width = "10em")

Further investigation required digging into the original data and documentation, which benefitted from the help of Przemek. It turns out that some ISSP waves have two variables capturing respondent’s education level: one is country-specific and the other is recoded to a common scheme across countries by ISSP. For example, in ISSP/2007, the country-specific variable for Switzerland (ch_degr) has 14 categories (including one for missing values). The degree variable is the recoded one and has 7 categories (also including one for missing values). The frequencies in both variables are nice enough to make a relatively safe bet as to how ch_degr was mapped onto degree by the ISSP team. Przemek and I checked the SDR v.1 documentation and the code used to harmonize education variables from ISSP. It turns out that the source variable used for harmonization if ISSP/2007/Switzerland was the recoded degree variable. The scheme in Table 2 shows how the three variables are related.

Table 2: Mapping scheme of ISSP/2007/Switzerland: degree, ch_degr, t_edu

It seems that the problem lies in the recoding from ch_degr to degree, especially in the value labels “Lowest formal qualification”, which suggests primary school completion, as it was coded in SDR. Meanwhile the category “Lowest formal qualification” in degree includes both “Primary school” and “Secondary school” from ch_degr, which would be assigned different ISCED codes. As a consequence of the misclassification of “Lowest formal qualification”, “Above lowest qualification” was understood as levels above primary, i.e., lower secondary, but not necessarily completed upper secondary.
An alternative (hypothetical) mapping presented in Table 3 uses the country-specific ch_degr variable as the source and maps it to categories of SDR’s t_edu. In this scenario all the categories highlighted in blue would be assigned target codes corresponding to upper secondary education or more, resulting in the share of respondents with at least upper secondary education of 86%, i.e., matching the OECD value.

Table 3: Hypothetical mapping scheme of ISSP/2007/Switzerland

Data dictionaries of the source variables for t_edu (i.e., the non-country-specific education variables) from other ISSP waves (see Appendix) show that the “Lowest formal qualification” category is present in all those ISSP waves (2004, 2006, 2007, 2008, 2009, 2010), where ca. 40% of the Swiss sample in SDR has at least upper secondary education. The earliest and latest wave of ISSP (1996, 2011) have different sets of response categories in the recoded education variable, which did not result in such errors when harmonizing them in SDR. The relevant variable in ISSP/1998 does not have the “Lowest formal qualification” or “Above lowest qualification” categories, and yet the share of respondents with at least upper secondary education based on the harmonized t_edu variable in SDR is less than 40. This requires further investigation.

Conclusion

Comparing the distribution of survey variables to data from non-survey sources can help identify errors in the recoding or harmonization of survey data and potentially reveal other surprises. A simple comparison of sample proportions of respondents with at least upper secondary education from surveys in the SDR v.1 dataset to analogous data from OECD led to the identification of samples with large differences between these two measurements. The examination of data and documentation from ISSP/Switzerland suggests that errors might have originated from the conversion of country-specific education variables to an education variable with uniform coding for the whole ISSP wave. Investigations of other surveys and projects can provide new insight.
This situation also proves the importance of clearly identifying the source data and variables and all transformation in harmonization projects, so that users can trace back the error to its source – in the case of educational attainment in ISSP/Switzerland it seems that the problem is not in the original measurement, but in the subsequent transformations.

Appendix

Data dictionary of source variables for t_edu from ISSP (selected waves).

ISSP_1996 v205
R: Education II: categories
1|None;still at school,uni
2|Incpl primary
3|Primary compl
4|Incpl secondary
5|Secondary compl
6|Semi-higher,Incpl uni.
7|University compl
99|NA, others

ISSP_1998 v205
R: Education II: categories
1|None;still at school
2|Incpl primary
3|Primary compl
4|Incpl secondary
5|Secondary compl
6|Semi-higher,incompl uni,other
7|University compl
99|NA

ISSP_2004 v205
R: Education II-highest education level (DEGREE)
0|No formal qualification
1|Lowest formal qualification
2|Above lowest qualification
3|Higher secondary completed
4|Above higher sec level,other educ
5|University degree completed
8|Don't know
9|NA

ISSP_2006 DEGREE
R: Education II-highest education level
0|No formal qualification, incomplete primary
1|Lowest formal qualification
2|Above lowest qualification
3|Higher secondary completed
4|Above higher secondary level,other qualification
5|University degree completed, graduate studies
8|Dont know
9|NA

ISSP_2007 DEGREE
R: Education II-highest education level
0|No formal qualification
1|Lowest formal qualification
2|Above lowest qualification
3|Higher secondary completed
4|Above higher secondary level
5|University degree completed
9|Dont know, NA

ISSP_2008 DEGREE
R: Education II-highest education level
0|No formal qualification
1|Lowest formal qualification
2|Above lowest qualification
3|Higher secondary completed
4|Above higher secondary level, others
5|University degree completed
8|Don't know
9|No answer

ISSP_2009 DEGREE
R: Education II-highest education level
0|No formal qualification
1|Lowest formal qualification
2|Above lowest qualification
3|Higher secondary completed
4|Above higher secondary level, other qualification
5|University degree completed
8|Dont know
9|NA

ISSP_2010 DEGREE
Education II: Highest education level: categories
0|No formal qualification
1|Lowest formal qualification
2|Intermediate secondary completed (qualifications above the lowest qualification)
3|Higher secondary completed (usual entry requirements for universities)
4|University degree incomplete (qualifications above higher secondary level)
5|University degree completed, BG: lower tertiary level, upper tertiary level
8|Don't know
9|No answer, other qualification, education

ISSP_2011 DEGREE
Highest completed education level: Categories for international comparison
0|No formal education
1|Primary school
2|Lower secondary (secondary completed does not allow entry to
3|Upper secondary (programs that allows entry to university
4|Post secondary, non-tertiary (other upper secondary programs
5|Lower level tertiary, first stage (also technical schools at
6|Upper level tertiary (Master, Dr.)
9|No answer