# Application of cognitive diagnosis models to competency-based situational judgment tests

←

→

**Page content transcription**

If your browser does not render page correctly, please read the page content below

Pablo Eduardo García, Julio Olea and Jimmy De la Torre ISSN 0214 - 9915 CODEN PSOTEG Psicothema 2014, Vol. 26, No. 3, 372-377 Copyright © 2014 Psicothema doi: 10.7334/psicothema2013.322 www.psicothema.com Application of cognitive diagnosis models to competency-based situational judgment tests Pablo Eduardo García1, Julio Olea2 and Jimmy De la Torre3 1 Instituto de Ingeniería del Conocimiento (IIC-UAM), 2 Universidad Autónoma de Madrid and 3 Rutgers, The State University of New Jersey Abstract Resumen Background: Profiling of jobs in terms of competency requirements Aplicación de los modelos de diagnóstico cognitivo a tests de juicio has increasingly been applied in many organizational settings. Testing situacional basados en competencias. Antecedentes: muchas these competencies through situational judgment tests (SJTs) leads to organizaciones definen sus puestos de trabajo en base a las competencias validity problems because it is not usually clear which constructs SJTs profesionales que requieren. La medición de tales competencias mediante measure. The primary purpose of this paper is to evaluate whether the tests de juicio situacional (TJS) presenta problemas de validez, en tanto application of cognitive diagnosis models (CDM) to competency-based no suele estar claro los constructos que miden. El objetivo principal de SJTs can ascertain the underlying competencies measured by the items, este estudio es evaluar si la aplicación de los modelos de diagnóstico and whether these competencies can be estimated precisely. Method: cognitivo (MDC) a estos tests permite clarificar y estimar de forma The generalized deterministic inputs, noisy “and” gate (G-DINA) model precisa las competencias medidas. Método: se aplicó el modelo G-DINA was applied to 26 situational judgment items measuring professional (generalized deterministic inputs, noisy “and” gate) a 26 ítems de juicio competencies based on the great eight model. These items were applied to situacional que medían competencias profesionales fundamentadas en el 485 employees of a Spanish financial company. The fit of the model to the modelo great eight. Se aplicó el test a 485 trabajadores de una entidad data and the convergent validity between the estimated competencies and financiera española. Se examinó el ajuste del modelo a los datos, y la personality dimensions were examined. Results: The G-DINA showed a validez convergente entre las competencias estimadas y dimensiones de good fit to the data and the estimated competency factors, adapting and personalidad. Resultados: G-DINA mostró un buen ajuste a los datos, y coping and interacting and presenting were positively related to emotional los factores competenciales estimados adaptarse y aguantar, e interactuar stability and extraversion, respectively. Conclusions: This work indicates y presentar mostraron una relación positiva con estabilidad emocional y that CDM can be a useful tool when measuring professional competencies extraversión, respectivamente. Conclusiones: este trabajo muestra que los through SJTs. CDM can clarify the competencies being measured and MDC pueden ser una herramienta útil para la medición de competencias provide precise estimates of these competencies. profesionales a través de TJS, aclarando las competencias que miden y Keywords: Cognitive diagnosis models (CDM), G-DINA model, obteniendo estimaciones precisas de las mismas. situational judgment tests (SJT), great eight model. Palabras clave: modelos de diagnóstico cognitivo (MDC), modelo G-DINA, tests de juicio situacional (TJS), modelo great eight. Competency modeling is a useful tool that guides the specification middle tier is made up of 20 competency dimensions (e.g., deciding of repertories of behaviors needed for effective performance at and initiating action); and the top tier is made up of eight broad work. Many competency models (i.e., defined sets of competencies) competency factors (e.g., leading and deciding), which are usually have been proposed, most of them pertaining to the managerial area, referred to as the great eight. which aim to reach a judicious equilibrium between the generality This research is focused on the measurement of competencies and specificity of work demands. One influential model that covers (middle tier) through situational judgment tests (SJT). SJT present both managerial and non-managerial positions is the great eight actual work-related situations using various formats (e.g., paper, model (Bartram, 2005; Kurz & Bartram, 2002). This competency video), and ask respondents how they would or should deal with model consists of a three-tier structure. The bottom tier is made up those situations, usually having to choose from various response options. These tests have become very popular among industrial of 112 component competencies (e.g., acting with confidence); the and organizational (I/O) psychologists in the last twenty years because they are cheaper than interviews, can be applied in Received: November 25, 2013 • Accepted: April 23, 2014 large-scale hiring contexts, and include work-related skills not Corresponding author: Pablo Eduardo García García easily measured by traditional cognitive and personality tests. Instituto de Ingeniería del Conocimiento (IIC-UAM) Several studies have shown the advantages of this type of tests, Universidad Autónoma de Madrid 28049 Madrid (Spain) which include predictive and incremental validity over work e-mail: pablo.garcia@icc.uam.es performance criteria, incremental validity over measures of 372

Application of cognitive diagnosis models to competency-based situational judgment tests cognitive ability and personality, reduced adverse impact, better Several CDM have been proposed in the last few years (Rupp, face validity perceptions of applicants, and greater resistance to Templin, & Henson, 2010), and they vary in the way attributes faking (Lievens, Peeters, & Schollaert, 2008). However, “it would are combined and formalized to estimate the probability of appear that SJT do not strongly relate to any particular construct item responses. To synthesize the various CDM, de la Torre but are moderately related to many different constructs” (Ployhart (2011) has proposed a general model, the G-DINA (generalized & MacKenzie, 2011, p. 244). Validity studies are still needed to deterministic inputs, noisy “and” gate) model, which allows clarify which constructs are being measured when applying these reformulating many of the existing CDM as special cases of this tests. general model. For each item, the G-DINA model partitions the latent classes Cognitive diagnosis models K into 2K*j latent groups, where K *j = q jk represents the number of k=1 Cognitive diagnosis models (CDM), also called diagnostic required attributes for item j. For example, item 3 of Table 1 leads classification models (DCM) by other researchers (e.g., Rupp to K *3 = 3 and 23 = 8 possible latent groups. We let α*lj be the reduced & Templin, 2008), are multidimensional categorical-latent trait attribute vector whose elements are the required attributes for item models. j (e.g., for item 3, the reduced vector is α*l3 = (αl1αl2αl5)). For Although CDM share several features with other models such as notational convenience but without loss of generality, we can let item response theory (IRT) and confirmatory factor analysis (CFA), the first K *j attributes be the required attributes for item j. This will one of the most important differences is the conceptualization of aid in the discussion of the properties of the model. In the G-DINA the latent trait as categorical (usually referred to as attributes), model, the probability that a respondent from latent class l with rather than continuous. attribute pattern α*lj will answer item j correctly is denoted by P(Xj CDM are formulated to specify the probability of responding = 1|α*lj) = P (α*lj). to an item in a specific way, given a vector of attributes that The formulation of G-DINA model in its saturated form (i.e., indicates which ones a respondent has mastered and not mastered. no restrictions are made) is: For existing work, item responses in CDM are categorical. This is one characteristic that differentiates CDM and IRT from CFA, K *j K *j K *j 1 K *j ( ) which models the probability distribution of continuous responses. Although item responses can be both dichotomous or polytomous, P lj* = j 0 + jk lk + jkk ' lk lk '… + lk j12…K *j k=1 k=1 k '=k+1 k=1 (1) for didactic reasons we will focus on CDM for dichotomous responses, as in responding correctly or incorrectly to an item, or where δj0 is the intercept for item j. It represents the baseline endorsing or not endorsing a particular statement. probability (i.e., the probability of a correct or most effective In CDM, the attribute vectors are referred to as attribute response option when none of the required attributes is mastered); patterns or latent classes, and they are denoted by αl = (αl1… δjk is the main effect due to αk. It is the change in the probability α1K), for l = 1 to L, where L represent the total number of attribute of a correct (or most effective) response as a result of mastering patterns (i.e., possible combinations of attributes). The kth element a single attribute (i.e., αk); δjkk’ is the interaction effect due to of αl (i.e., αlk) is equal to 1 if the respondents in latent class l have αk and αk’ (first-order interaction effect). It is the change in the mastered the kth attribute, and 0 if they have not. For example, the probability of a correct response due to the mastery of both αk and attribute vector αl (11010) means that respondents in latent class αk’, and it represents the impact that is over and above the additive l have mastered attributes 1, 2 and 4, but not attributes 3 and 5. impact of the mastery of the same two attributes; and δjl2…K*j is Theoretically, there are 2K possible latent classes. For example, the interaction effect due to α1, …, αK*j. It represents the change in with K =5, there would be L = 25 = 32 possible latent classes, with the probability of a correct response due to the mastery of all the αl (11010) being only one of them. required attributes that is over and above the impact of the main The inputs needed for CDM to be estimated are item responses, and lower-order interaction effects. and what is known as a Q-matrix (Tatsuoka, 1983). As confirmatory The G-DINA model parameters can be estimated using models, CDM require specifying which attributes are needed marginalized maximum likelihood estimation (MMLE). The to respond to each item correctly. This is a task that is typically algorithm proposed by de la Torre for performing this estimation conducted by domain experts. The item-attribute mapping leads is largely similar to that described in detail for the DINA model to the J (number of items) × K (number of attributes) binary (de la Torre, 2009), and is written in Ox (Doornik, 2003). Q-matrix. The element in row j and column k of the Q-matrix, q jk, is equal to 1 if the kth attribute is required to answer item j correctly; otherwise it is equal to zero. Table 1 gives a Q-matrix Table 1 for five items and five attributes (where the kth attribute is denoted Example of Q-matrix by αk). Based on the third row of the Q-matrix, attributes 1, 2 and Attribute 5 are required to answer item 3 correctly, whereas attributes 3 and 4 are not. Item α1 α2 α3 α4 α5 The main output of CDM for each respondent is a vector of estimates indicating the probability that the respondent has 1 1 1 1 1 0 mastered each of the attributes. These probability can be converted 2 0 1 0 0 1 3 1 1 0 0 1 in dichotomous scores (i.e., mastery or non-mastery) by comparing 4 1 1 0 0 0 them to a cutscore (usually .5; de la Torre, Hong, & Deng, 2010; 5 0 1 1 1 1 Templin & Henson, 2006). 373

Pablo Eduardo García, Julio Olea and Jimmy De la Torre The current study presents an application of the G-DINA model (Aguado, Lucia, Ponte, & Arranz, 2008) administered to the to a competency-based SJT. The two main predictions of the work same participants through the eValue system was used. Bartram are: a) When applied competency-based SJTs, which are based on (2005) showed empirical relations between the top-tier factors of the great eight model (Bartram, 2005), the G-DINA model can his model and the Big Five dimensions. Table 2 indicates which provide a good model-data fit; and b) Based on the theoretic model factors are related to which dimensions. predictions, convergent validity evidence between the G-DINA estimates and measures of personality dimensions can be obtained. Procedure and data analysis Method Q-matrix specification. In addition to operational definitions of the seven competencies indicated in Table 2, the four experts were Participants also provided with the response option considered most effective for each item and were then asked to decide which competency or Four hundred eighty five employees of a Spanish financial competencies were necessary to choose those responses. The task company were tested in several professional competencies, as part was based on a Delphi method, conducted over three rounds. In of a developmental process of the company, through computer- Round 1, each expert performed the task individually. In Round 2, based controlled tasks administrations. They were junior the experts were anonymously provided with the decisions of the employees (less than two years working for the company), aged other experts and were told they could (not should) change their between 25 and 31 years old and 50.7% of them being women. initial specifications. Finally, in Round 3 the four experts met in Four industrial/organizational psychologists (i.e., three person, and they discussed in detail their opinions and remaining university professors of organizational psychology and a senior differences. consultant at a research and consulting firm) with expertise in Because a total consensus at the end of Round 3 was not competency modeling specified the 0/1 entries of the Q-matrix. mandatory, different Q-matrices were examined to determine the one most appropriate for the data. To accomplish this, the fit Instruments indexes of the different Q-matrices were compared. Assuming a CDM can fit the data, Chen, de la Torre, and Zhang (2013) show Situational judgment test. The employees were tested through that Schwarz’s (1976) Bayesian information criterion (BIC) can eValue, a computer-based system for measuring competencies. The determine the best fitting Q-matrix when used in conjunction with system includes inbox exercises, auto-reports, tasks execution tests, a general CDM such as the G-DINA model. The matrix with the and also situational judgment tasks. This system was developed smallest BIC was selected and used in succeeding analyses. and is currently marketed by the Knowledge Engineering BIC index, model parameters estimated through the MMLE Institute - Instituto de Ingeniería del Conocimiento (IIC; www. algorithm, as well as the statistics presented onwards are all iic.uam.es/en/). For the current study, 26 items, which measure outputs provided by the code written by de la Torre (2011) in seven competencies of Bartram’s middle-tier competencies, were Ox. chosen from the situational judgment tasks. See Table 2 for these Comparison of G-DINA with other simpler models. Given competencies, and the top-tier factors to which they belong. The that G-DINA model is a general model in its saturated form situational items were structured as follows: (a) applicant must read (Equation 1), we also examined whether a simpler model can a critical incident; (b) this is followed by a question about how he/ be applied to data without significant loss in statistical fit. Two she would act in that situation; and (c) applicant must choose one of of these simpler CDM are the DINO (deterministic input, noisy three response options. The correct or most effective response was “or” gate; Templin & Henson, 2006) and DINA (deterministic determined by nine experts who participated in the development input, noisy “and” gate; de la Torre, 2009; Haertel, 1989; Junker of these situational tasks. The interjudge agreement of the nine & Sijtsma, 2001) models. These models can be formulated as experts for the 26 items was .79. special cases of the G-DINA model by imposing restrictions Personality questionnaire. To examine the validity of the to Equation 1 (see de la Torre, 2011). They represent pure CDM-derived SJT scores, the Big Five personality questionnaire compensatory and non-compensatory models, respectively. For an item, both models can only differentiate between two possible latent groups in determining the probability of a correct response. Table 2 The middle-tier competencies (Bartram, 2005) measured by the items The DINO model differentiates between those who master at least one of the required attributes from those who do not master Competency Top-tier factor Big Five dimension any of them (i.e., not mastering one required attribute can be Relating and networking Interacting and presenting Extraversion compensated by mastering another of those required); the DINA Persuading and influencing Interacting and presenting Extraversion model differentiates between those who master all the required Presenting and communicating information Interacting and presenting Extraversion attributes from those who do not master at least one of them (i.e., Following instructions and procedures Organizing and executing Conscientiousness the absence of one required attribute cannot be compensated for Adapting and responding to change Adapting and coping Emotional stability by the presence of other attributes). Coping with pressure and setbacks Adapting and coping Emotional stability Because the DINO and DINA models are nested in the G-DINA Deciding and initiating action Leading and deciding Extraversion model, the likelihood ratio test (LR) was conducted to determine whether using one of these reduced models led to a significant loss Note: Top-tier factor = the top-tier factor the competency belongs to; Big Five dimension of fit (i.e., LR ≠ 0). LR is computed as = the personality dimension the top-tier factor is positively related with, according to Bartram (2005) LR = [-2LL reduced model] – [-2LL generalmodel] (2) 374

Application of cognitive diagnosis models to competency-based situational judgment tests and it is approximately χ2 distributed, with degrees of freedom whether people who had mastered all competencies included in the equal to the difference between the numbers of parameters of the top-tier factor had a significantly higher score in the corresponding saturated and reduced models. personality dimension than people who had not mastered any of Absolute fit. In addition to the relative fit analyses presented those competencies. above to select the best fitting Q-matrix and CDM for the current data, an absolute fit analysis was also conducted to determine Results whether the selected model fit the data adequately (i.e., the best model is not a bad model). Statistics based on the residuals between Q-matrix Specification the observed and predicted Fisher-transformed correlations and between the observed and predicted log-odds ratios of item Table 3 shows the experts’ decisions for each item across the pairs can be used to statistically test model misfit (Chen et al., three rounds. First, it can be observed that Competency 1 (Relating 2013). If the evaluated model fits the data, these statistics should and networking) was almost not selected in the last round despite be close to zero for all the items. Considering the tests involve being one of the seven competencies expected to be measured. a large number of item pairs (26 items would lead to 325 pairs Thus, it was eliminated from the list of competencies, and only the for each test statistic), Chen et al. propose to examine only the six remaining competencies were considered thereafter. residual with the maximum z-score for each statistic. Rejecting Secondly, the table shows that for those six competencies, a total any z-score indicates that the model does not fit at least one item consensus was not reached at the end of the last round for items pair adequately. 3, 5, 10, 13, 24 and 26. So three Q-matrices were compared: the The Fisher-transformed correlations and log-odds ratios elements were equal to 1 if all the experts considered a competency residuals were computed as follows: necessary (Q-matrix 1), if at least two experts considered it necessary (Q-matrix 2), and if any expert considered it necessary ( ) ( ) rjj ' = Z Corr X j , X j ' Z Corr X j , X j ' , and (3) (Q-matrix 3). The G-DINA model estimated for each one of these three Q-matrices led to BIC indices of 15782.75 (Q-matrix 1), 15749.79 (Q-matrix 2) and 15840.37 (Q-matrix 3). Because N N N N Q-matrix 2 (see Table 4) had the lowest BIC, it was the Q-matrix l jj ' = log 11 00 log 11 00 used for the subsequent analyses. As can be seen from the table, N 01N10 N 01N 10 (4) most items (15) involved three competencies, 6 items involved two competencies, and 5 items involved only one competency. where j’ ≠ j, Corr is the Pearson’s product-moment correlation, Z is the Fisher transformation, Xj, and X̃j are the observed and model- Comparison of G-DINA with Other Simpler Models generated data for item j, respectively, and Nyy’ and Ñyy’ are the number of observed and predicted examinees, respectively, who The two χ2 tests, each one with 102 degrees of freedom, scored y (0 or 1) on item j, and y' (0 or 1) on item j'. corresponding to the likelihood ratio tests resulting from The approximate standard errors for these statistics, which are needed for obtaining the z-scores, are computed as follows: Table 3 Experts’ task. Competencies that were considered necessary to choose the most SE[rjj'] = [N –3]1/2, and (5) effective response option in each item SE[ljj'] = [Ñ(1/ Ñ11 + 1/ Ñ00 + 1/ Ñ01 + 1/ Ñ10) / N]1/2 (6) Round Round The resulting z-score is assumed to be approximately normally Item 1 2 3 Item 1 2 3 distributed. A Bonferroni correction is also suggested by Chen et 1 4,5,6 4,5,6 4,5,6 14 4,5 4,5 4,5 al. (2013) so as not to inflate the Type I error due to the multiple 2 2,3,4,5,6 3,4,5,6 3,5,6 15 2,3,7 2,3,7 2,3,7 comparisons. 3 2,3,5,6,7 2,3,7 2,3a,7 16 3 3 3 Validity analysis. The G-DINA model provides a vector of 4 1,2,3,6,7 2,6 2,6 17 4,5,6 4,6 4 estimates per respondent representing his/her expected a posteriori 5 1,2,3,5,6 1,3,6 3,6b 18 1,2,3 2 2 (EAP) probabilities of mastering each one of the competencies. 6 4,5,6,7 4,5,6 4,5,6 19 1,2,3,6 2,3,6 2,3 Using a cutscore of .5, participants were classified as either 7 2,3,6 2,3,6 2 20 2,3,6,7 2,3,7 2,3,7 mastering or not mastering those competencies. 8 4,5,6,7 4,5,6 4,5,6 21 1,4,5,6 1,4,5 1a,4,5 Once the 485 participants were classified, we examined the 9 4,5,6 4,5,6 4,5,6 22 1,4,5,6 4,5,6 4,5,6 relations between these classifications and the Big Five personality 10 1,2,3 1,2,3 1,2a,3 23 1,4,5,6 4,5,6 4,5,6 dimensions as described by Bartram (2005). Given that Bartram 11 2,3,4,5,7 2,5,7 5,7 24 2,3,5,6,7 2,3,6,7 2,3,6b,7 showed that relations exist between the personality dimensions 12 1,2,3,7 2,3,7 2,3,7 25 1,2,3,7 1,2,3,7 1b,2,3,7 and the eight top-tier factors, only those stated for the factors 13 4,5,6,7 4,5,6,7 4,5b,6,7 26 1,2,3,5,7 1,2,3,7 1,2,3a,7 Adapting-and-coping and Interacting-and-presenting could be Note: Competencies in boldface were considered necessary by the four experts. properly examined in this study, because all of their aggregated Competencies 1 = Relating and networking; 2 = Persuading and influencing; 3 = Presenting middle-tier competencies had to be estimated (see Table 2). and Communicating information; 4 = Following instructions and procedures; 5 = Adapting Bartram showed that the top-tier factor Adapting-and-coping is and responding to change; 6 = Coping with pressure and setbacks; 7 = Deciding and positively related to emotional stability, and the top-tier factor initiating action a Interacting-and-presenting is positively related to extraversion. Two experts considered the competency necessary b One expert considered the competency necessary To examine these relations, two t tests were conducted to contrast 375

Pablo Eduardo García, Julio Olea and Jimmy De la Torre comparing the G-DINA model with the DINO (LR = 328.05) mastered both Persuading-and-influencing and Presenting-and- and DINA (LR = 319.17) models were both significant (p

Application of cognitive diagnosis models to competency-based situational judgment tests it must be highlighted that the application of CDM involves through the pre-specification of the Q-matrix, would make this considering competencies as categorical, a statement with which second evaluation unnecessary. several authors may not agree. Finally, a limitation of the CDM application presented in For future works involving CDM, it would be interesting the current study is that the specification of the competencies to evaluate CDM-based SJT predictions against empirical measured by each item of the SJT was done after the test was work performance data. It would also be interesting to apply developed. A more optimal approach is to apply these theory- these models to other organizational areas where ratings of based specifications during the test development itself (de la Torre, competencies are involved. In many assessment centers, judges Tjoe, Rhoads, & Lam, 2013). usually have to perform a two-step evaluation. In the first step, judges have to evaluate how well the participants perform on Acknowledgements each exercise. In the second step, judges have to determine the participants’ mastery of several competencies based on their This research was supported in part by the UAM-IIC Chair for performance across multiple exercises. The application of CDM, Psychometric Models and Applications. References AERA, APA, & NCME (1999). Standards for educational and Haertel, E.H. (1989). Using restricted latent class models to map the skill psychological testing. Washington, DC: American Psychological structure of achievement items. Journal of Educational Measurement, Association. 26, 333-352. Aguado, D., Lucia, B., Ponte, G., & Arranz, V. (2008). Análisis inicial Junker, B.W., & Sijtsma, K. (2001). Cognitive assessment models with de las propiedades psicométricas del Cuestionario BFCP Internet few assumptions, and connections with non parametric item response para la Evaluación de Big-Five [Initial analysis of the psychometric theory. Applied Psychological Measurement, 25, 258-272. properties of the BFCP Internet questionnaire for the evaluation of Kurz, R., & Bartram, D. (2002). Competency and individual performance: Big-Five]. Revista Electrónica de Metodología Aplicada, 13(2), 15- Modeling the world of work. In I.T. Robertson, M. Callinan, & D. 30. Bartram (Eds.), Organizational effectiveness: The role of psychology Bartram, D. (2005). The Great Eight Competencies: A criterion-centric (pp. 227-255). Chichester: Wiley. approach to validation. Journal of Applied Psychology, 90, 1185- Lievens, F., Peeters, H., & Schollaert, E. (2008). Situational judgment 1203. tests: A review of recent research. Personnel Review, 37, 426-441. Chen, J., de la Torre, J., & Zhang, Z. (2013). Relative and absolute fit Ployhart, R.E., & MacKenzie, W.I. (2011). Situational judgment tests: evaluation in Cognitive Diagnosis Modeling. Journal of Educational A critical review and agenda for the future. In S. Zedeck (Ed.), APA Measurement, 50, 123-140. handbook of industrial and organizational psychology (Vol. 2): Christian, M.S., Edwards, B.D., & Bradley, J.C. (2010). Situational Selecting and developing members for the organization. Washington, judgment tests: Constructs assessed and a meta-analysis of their DC: American Psychological Association. criterion-related validities. Personnel Psychology, 63, 83-117. Rupp. A.A., & Templin, J.L. (2008). Unique characteristics of diagnostic de la Torre, J. (2009). DINA model and parameter estimation: A didactic. classification models: A comprehensive review of the current state-of- Journal of Educational and Behavioral Statistics, 34, 115-130. the-art. Measurement, 6, 219-262. de la Torre, J. (2011). The generalized DINA model framework. Rupp, A.A., Templin J.L., & Henson, R.A. (2010). Diagnostic Psychometrika, 76, 179-199. measurement. Theory, methods, and applications. New York, NY: de la Torre, J., Hong, Y., & Deng, W. (2010). Factors affecting the item The Guilford Press. parameter estimation and classification accuracy of the DINA model. Schwarz, G. (1976). Estimating the dimension of a model. Annals of Journal of Educational Measurement, 47, 227-249. Statistics, 6, 461-464. de la Torre, J., Tjoe, H., Rhoads, K., & Lam, D. (2013). Conceptual and Tatsuoka, K.K. (1983). Rule space: An approach for dealing with theoretical issues in proportional reasoning. International Journal for misconception based on item response theory. Journal of Education Studies in Mathematics Education, 6, 21-38. Statistic, 20, 345-354. Doornik, J.A. (2003). Object-Oriented Matrix Programming using Ox Templin J.L., & Henson, R.A. (2006). Measurement of psychological (Version 3.1) [Computer software]. London: Timberlake Consultants disorders using cognitive diagnosis models. Psychological Methods, Press. 11, 287-305. 377

You can also read