Artikelen Talent Assessment
Talent Assessment (PSMDB-2)
Rijksuniversiteit Groningen
Aanbevolen voor jou
Reacties
Andere studenten bekeken ook
Gerelateerde Studylists
Talent Development and CreativityPreview tekst
Inhoudsopgave
- General note: geel gemarkeerd is opmerking vanuit college, niet artikel..........................................
- Artikelen Talent Assessment: Lecture 1: Signs & Samples.................................................................
- Bergkamp et al (2019): Methodological issues in soccer talent identification research.....................
- Methodological issues for talent identification, fidelity, range restriction, base rate, utility.........
- Callinan & Robertson (2000): Work sample testing............................................................................
- Work sample tests, adverse impact................................................................................................
- Dawes (1979): The robust beauty of improper linear models in decision making..............................
- Linear models.................................................................................................................................
- Wernimont & Campbell (1968): Signs, samples and criteria..............................................................
- !!Too harsh on signs!!, conclusion is outdated, only use for distinction.........................................
- Bergkamp et al (2019): Methodological issues in soccer talent identification research.....................
- Artikelen Talent Assessment: Lecture 2: Signs & Samples.................................................................
- restriction........................................................................................................................................... Huffcutt et al (2014): Reanalyzing the validity of employment interviews with indirect range
- Valid interviews..............................................................................................................................
- Lievens & De Soete (2012): Simulations.............................................................................................
- Inhoud............................................................................................................................................
- Niessen et al (2018): Admission testing for higher education............................................................
- Inhoud............................................................................................................................................
- Sackett et al (2017): Assessment center vs. cognitive ability tests.....................................................
- AC and cognitive ability..................................................................................................................
- saturation, predictive validity and applicant perceptions.................................................................. Schäpers et al (2019): Role of situations in Situational Judgment Tests: effects on construct
- SJT criterion-validity, situation...................................................................................................
- Schmidt & Hunter (1998): Validity and utility of selection methods in personnel psychology...........
- Predictive and incremental validity of signs and samples..............................................................
- Van Iddekinge et al (2019): Meta-analysis of criterion-related validity of prehire work experience..
- Pre-hire experience (not past performance)..................................................................................
- restriction........................................................................................................................................... Huffcutt et al (2014): Reanalyzing the validity of employment interviews with indirect range
- Artikelen Talent Assessment: Lecture 3: Cognitive and noncognitive traits.......................................
- Borghans et al (2016): What grades and achievement tests measure...............................................
- Grades, intelligence, achievement tests.........................................................................................
- Morgeson et al (2007): Reconsidering the use of personality tests in personnel selection................
- Faking, self-report, personality tests..............................................................................................
- Niessen et al. (2017): Measuring non-cognitive predictors in high-stakes contexts...........................
- High-stakes, non-cognitive, faking..................................................................................................
- assessment......................................................................................................................................... Christiansen et al (2005): Reconsidering forced-choice item formats for applicant personality
- Forced-choice vs normative............................................................................................................
- Kuncel & Hezlett (2010): Fact and fiction in cognitive ability testing for admission and hiring..........
- Inhoud............................................................................................................................................
- Van Iddekinge et al (2012): The criterion-related validity of integrity tests.......................................
- Integrity tests.................................................................................................................................
- Borghans et al (2016): What grades and achievement tests measure...............................................
- Artikelen Talent Assessment: Lecture 4: Opinions of stakeholder and effects in practice..................
- Anderson et al (2010): Applicant reactions in selection.....................................................................
- Organizational context...................................................................................................................
- differences across predictors of job performance............................................................................ Dahlke & Sackett (2017): Relationship between cognitive-ability saturation and subgroup mean
- Adverse impact in job performance prediction............................................................................
- König et al (2010): Reasons for being selective when choosing personnel measures......................
- Inhoud..........................................................................................................................................
- Ployhart & Holtz (2008): Diversity-validity dilemma.........................................................................
- Adverse impact, race....................................................................................................................
- Niessen & Meijer (2017): On the use of broadened admission criteria in higher education............
- Inhoud..........................................................................................................................................
- Niessen et al (2017): Applying organizational justice theory to admission in high education..........
- Educational context......................................................................................................................
- Sackett et al (2017): Predictor weighting methods on incremental validity.....................................
- Only understand conclusion and table application Inhoud..........................................................
- impact.............................................................................................................................................. Sackett & Ellingson (1997): Effect of multi-predictor composites on group differences and adverse
- Only understand conclusion and table application reducing adverse impact..........................
- selection........................................................................................................................................... Taylor & Russell (1939): Relationship of validity coefficients to practical effectiveness of tests in
- Only understand conclusion and table application: dichotomized criterion................................
- Anderson et al (2010): Applicant reactions in selection.....................................................................
- Practitioner Gap (weights)............................................................................................................... Artikelen Talent Assessment: Lecture 5: Clinical vs Actuarial Prediction and the Scientist-
- Dana et al (2013): Belief in the unstructured interview...................................................................
- Dilution effect, interview..............................................................................................................
- Dietvorst et al (2018): Overcoming algorithm aversion....................................................................
- Inhoud..........................................................................................................................................
- Highhouse (2008): Stubborn reliance on intuition and subjectivity in employee selection..............
- Inhoud..........................................................................................................................................
- Dana et al (2013): Belief in the unstructured interview...................................................................
Callinan & Robertson (2000): Work sample testing............................................................................
Work sample tests, adverse impact................................................................................................
Work sample tests: good criterion-related validity, positive applicant reaction and job preview capability, highest predictive validity, less adverse impact (rejection rate of sub-groups by comparison with rest of assessors). Limitations work sample tests: not suitable for applicants without job experience, costly and complex Work sample tests can complement other selection instruments
Cognitive ability is best general predictor, but represents only a small part of the domain. Work sample approach is predicated on people displaying behavioural consistency Content validity is very important (job-relatedness), when lower relatedness, lower fairness and higher race-bias.
Types of Work-Sample Testing: - Hands-on performance test: psychomotor tests can only be used when there is similar experience, lower adverse impact - Trainability Tests: also suitable for no experience applicants. Minicourse approach is a very brief training of the job. - Situational tests: describe how one would behave in a given situation some say this is job knowledge test of even multi-faceted. - Job knowledge test: good predictors of work performance - Assessment Centre Exercises: in-basket, leaderless group, discussions, role-based group predict performance on a level, not a specific job. No discriminant validity. Bandwidth: degree to which the job performance is represented by the tasks in the measure Fidelity: hand-on performance tests has high fidelity, written description of work situation is low fidelity.
Dawes (1979): The robust beauty of improper linear models in decision making..............................
decision making
Linear models.................................................................................................................................
Clinical/Holistic judgment: you process the information you have in your mind. Actuarial/mechanical/statistical judgment: use a decision rule or formula how to combine the information. outperforms clinical judgment. So it’s both about how you combine information.
Proper linear model: weights given to predictor variables re chosen to optimize the relationship (correlation) between prediction and criterion, discrepancy between two groups or correlation with criterion in new dataset. Improper linear model: weights chosen by nonoptimal method (equal, intuition of random). However, this can lead to interesting results (example of intercourse – fighting in marital couples). A lot of examples where actuarial judgment is better than clinical judgment (clinical judgment is never better than actuarial).
People are important because they don’t do the integration of the model, but they choose the variables and code them. Proper linear models are good in integration when the predictions have a conditionally monotone relationship to the criterion.
Proper linear models cannot be used when there I no decent ratio of observations or when there are no measurable criterion variables (for instance: measuring professional self-acitualization is not measurable beforehand, but we know that it probably relates to intelligence, past abilities. you cannot do a standard regression analysis Solutions: - Bootstrapping: use expert’s judgment to build a model and use the model. no proper linear model, but better than judges. - Paramorphic: representation of judges can be simulated by a weighting?? equal to random linear models Bootstrapping eliminates unrealibility Equal weighting is superior to models based on judges behavior
Objections to using linear models (because they are better, but are often not used) - Outstanding judges are better, you did not find the best judges is not supported by research - I know someone who (but this is not a replicated finding) - Low validity of linear models, so another method is better
Wernimont & Campbell (1968): Signs, samples and criteria..............................................................
!!Too harsh on signs!!, conclusion is outdated, only use for distinction.........................................
Signs: measure relevant traits/characteristics (constructs as theoretical determinants of performance and behavior. Theory and explanation-oriented (understanding) more useful for theory development. It is objective because you use standardization (however, you also need this in samples) With signs there is faking and discrimination in testing (some items are not related to work required and most signs are for middle-class).
Samples: use a sample of relevant behavior or performance (behavioral consistency: past or current behavior is the best predictor for future behavior). Are developed and most often used in the context of personnel selection. Samples do not reduce to test scores (which is the danger of using signs they say) however also in samples you need standardization. Samples are practice-oriented and prediction-oriented and therefore according to some a- theoretical. We know it predicts it, but often don’t know why (can be downside when you want to understand)
Artikelen Talent Assessment: Lecture 2: Signs &
Samples
restriction........................................................................................................................................... Huffcutt et al (2014): Reanalyzing the validity of employment interviews with indirect range
interviews with indirect range restriction
Valid interviews..............................................................................................................................
Focus on components and levels of structure and conclusions. Two factors on which you can classify interviews: 1. Question 2. Response
- How you respond to SJT is mostly not result of interaction between trait, knowledge and environment just traits and knowledge
- Criterion-validity: you can only predict global job performance, not related to situation
SJT less fakeable when higher cognitive load, more fakeable when more transparent items. Should do/would do matters.
Schmidt & Hunter (1998): Validity and utility of selection methods in personnel psychology...........
personnel psychology
Predictive and incremental validity of signs and samples..............................................................
Used an often cited table gain of added predictor(look at Multiple R for validity after adding) GMA is one of best predictors of later job performance
Several of the articles from lecture 2 are replications of Schmidt & HUnter
Van Iddekinge et al (2019): Meta-analysis of criterion-related validity of prehire work experience..
of prehire work experience
Pre-hire experience (not past performance)..................................................................................
Pre-hire experience: amount, duration or type of work experience workers have acquired prior to entering a new organization Basic ideas: 1. Tasks and KSAOs needed can vary greatly between jobs (even over organizations with same job) 2. Several theories suggest that experience in one organization may not help (or even impede) in another. 3. Measures of prehire experience do not assess KSAOs acquired from these experiences or effectiveness of behavior in those situations Findings: 1. Prehire experience is a weak predictor of future performance supported 2. Prehire experience is more strongly related to training performance than job performance not supported 3. Prehire experience and performance are stronger related for measures for relevant experience than for general experience not supported 4. Prehire experience and performance are stronger related for qualitative then quantitative measures not supported 5. Prehire experience and performance are stronger when experience measured at specific levels en weaker at general levels not supported Interpretation: Skills obtained in one organization may not be transferable to another Sample of experience no predictive validity so be careful to select based on experience
Artikelen Talent Assessment: Lecture 3: Cognitive and
noncognitive traits
Borghans et al (2016): What grades and achievement tests measure...............................................
Grades, intelligence, achievement tests.........................................................................................
They use grades & achievement test scores (these also measure noncognitive) richer than only IQ better predict life outcomes than IQ (or maybe because then predictor and criterion are more alike). 1. Grades, achievement test scores and IQ strongly positively correlated, but not perfectly (can be used interchangeably) 2. Grades, achievement test scores influenced by IQ and personality (grades more heavily) 3. All three predict some important life outcomes, but IQ is lowest of the three 4. Grades and achievement test scores are more predictive of life outcomes, because they capture aspects of personality has independent predictive power.
Morgeson et al (2007): Reconsidering the use of personality tests in personnel selection................
personnel selection
Faking, self-report, personality tests..............................................................................................
Faking cannot be avoided (deliberate or impression management) Self-report personality tests should be reconsidered. Personality may be useful when not self-report. Validities: relatively low (.10)
When using personality measures, direct your measure toward the outcome you have (more validity).
Niessen et al. (2017): Measuring non-cognitive predictors in high-stakes contexts...........................
stakes contexts
High-stakes, non-cognitive, faking..................................................................................................
Difficult to measure non-cognitive (intra, interpersonal skills, character-based) In low-stakes conditions: Likert scale questionnaires are fine (cheap, reliable, sometimes valid) In high-stakes conditions/specific outcomes: problematic
Are personality questionnaires useful in selection/matching (admission vs research contect) The effect of faking on personality measures is low (predictor-criterion relations). But: more faking in admission context with impact on validity so questionable if you need to use noncognitive measures in high-stakes admission context.
assessment......................................................................................................................................... Christiansen et al (2005): Reconsidering forced-choice item formats for applicant personality
for applicant personality assessment
Forced-choice vs normative............................................................................................................
Ipsative scoring: paired items on scale roughly comparable on social desirability Scoring is more complicated than Likert.
Anderson et al (2010): Applicant reactions in selection
Organizational context
Favorable measures: work samples, interviews Reasonably favorable: resumes, cognitive ability, biodata, personality, honesty Least favorable: personal contacts, graphology High Validity: work samples, GMA test Reasonably validity: interviews, biodata, personality, honesty No validity: graphology
Dahlke & Sackett (2017): Relationship between cognitive-ability
saturation and subgroup mean differences across predictors of job
performance
Adverse impact in job performance prediction
Higher cognitive load more ethnicity adverse impact - Cognitive tests: o Female academic/job performance is underpredicted o Ethnic academic/job performance is overpredicted - Predictions are biased (not always caused by test bias) Because: bias in tests, criterion or omitted variables/selection system bias
When there are no prediction (outcome) differences then no bias.
König et al (2010): Reasons for being selective when choosing
personnel measures
Inhoud
Ployhart & Holtz (2008): Diversity-validity dilemma
Adverse impact, race
Consequential validity: positive or negative social consequences of assessment practices Also: adverse impact: systematic differences in scores, expressed by Cohen’s d Bias: systematic differences in measurement/prediction
Level of adverse impact GMA: Ethnicity (large), gender: personnel (little), education (some) Personality: almost none (in low-stakes) Interviews: ethnicity (some) Simulations: Ethnicity (moderate), gender (depends)
Reducing adverse impact: validity-diversity dilemma: most valid assessment methods have largest adverse impact Assessing all KSAOs (in low-stakes): lower adverse impact, the same validity Without reducing validity a lot: reduce verbal skills, improve applicant perceptions, enhance minority recruitment
Niessen & Meijer (2017): On the use of broadened admission criteria
in higher education
Inhoud
Niessen et al (2017): Applying organizational justice theory to
admission in high education
Educational context
Favorable: interviews, curriculum-sampling Reasonably favorable: cognitive ability, subject tests, biodata, motivation, personality Least favorable: high school grades, lottery High Validity:
Sackett et al (2017): Predictor weighting methods on incremental
validity
Only understand conclusion and table application Inhoud
In practice: optimal weights unknown, advice: unit weights.
Adding predictors with incremental validity, can reduce validity when suboptimal weights are used: When the additional predictor have lower validity than the first
Every base rate has they own table Selection ratio: horizontal Predictive validity of the test(s): vertical. Result: process of success ratio and you can compare this percentage with the selection ratio.
So success ratio depends on: base rate, selection ratio and predictive validity
Utility model: added value of assessments to decisions given the predictive validity and incremental validity (and context). Base rate: percentage successful in applicant pool (who are suitable) Base-line performance: continuous. Selection ratio: Percentage selected out of applicant pool Success ratio: percentage successful in selected group
Low utility when: extreme base rates, high selection ratio (even with high predictive validity) High utility when: base rate +/- .50, low selection ratio (even with relatively low predictive validity)
Artikelen Talent Assessment: Lecture 5: Clinical vs
Actuarial Prediction and the Scientist-Practitioner Gap
(weights)
HOW is the information combined Clinical (holistic, intuitive, expert): decision-maker combines or processes information in his head Actuarial (mechanical, statistical, algorithm-based): decision rules, based on relations between data and the condition often better than clinical
Dana et al (2013): Belief in the unstructured interview
Dilution effect, interview
Common belief is: more information cannot hurt dilution effect: adding invalid information to valid information dilutes the valid information (less predictive, when holistic) Dilution: availability of irrelevant information reduces use of relevant information
Validity of clinical decision is lower when unstructured interview is added. Actuarial GPA is higher (with .04) than clinical with no interview.
A random, real or natural interview does not matter for validity of reported informativeness.
Participants prefer a random interview over no interview at all and sense-making (seeing patterns and make sense of virtually anything the interviewee says): even when we know the information is invalid.
Dietvorst et al (2018): Overcoming algorithm aversion....................................................................
Inhoud..........................................................................................................................................
Inhoud............................................................................................................................................
The autonomy: use a little bit of both makes predictions worse, but still better than clinical judgment, and people are more likely to use it than actuarial judgment.
Highhouse (2008): Stubborn reliance on intuition and subjectivity in employee selection..............
employee selection
Inhoud
One must be willing to accept error to make less error. People are not very predictable, but selection decision aids help (assessment, higher scoring group larger chance of acceptance).
You have different objections: technical, psychological and ethical objections. Technical: - You used poor judges (experience makes confident, not accurate) - Overconfidence in judgments (I saw it with my own eyes) - Lack of feedback & confirmation feedback Psychological: - I know someone who (proven low validity when 1 does not work well, does not mean that every other one is better) Ethical (Dawes, 1979) - Dehumanizing (reducing people to numbers) - Scores underrepresent reality - ‘How can they tell what I’m like) But: not making best possible judgment and decision is unethical!
Arguments in favor of holistic: - Unique candidates - Interaction of traits - Dynamic interpretation - But still: model of man is better than man itself (Brunswick, 1955)
The failure of psychology has been the inability to convince practitioners to use evidence-based methods! Most agree that test is effective, but also most agree that informal discussions can help you learn more.
Kahneman & Klein (2009): Conditions for intuitive expertise: Failure
to disagree
Developing expertise
Conditions for developing expertise: opportunity to learn, immediate & clear feedback, high-validity contexts. We do not judge better when we have more experience!
Artikelen Talent Assessment
Vak: Talent Assessment (PSMDB-2)
Universiteit: Rijksuniversiteit Groningen
- Ontdek meer van:
