The scientific basis

Evidence-Centred Design: building the test backwards

The usual way to build a test is to write questions that "look relevant". Evidence-Centred Design reverses that, in three steps:

1. A model of what is measured

   Define exactly what the skill means, broken into observable behavioural indicators. Not "empathy", but "asks before assuming what the other person needs".

2. A model of the evidence

   Define what will be accepted as proof that the indicator is present. Which choice, at which moment, counts and how much.

3. A model of the situation

   Only then is the scenario written — designed to produce exactly that evidence rather than something adjacent to it.

The consequence is practical: every item points at a specific indicator. There is no item that "measures generally".

Why scenarios rather than a questionnaire

A questionnaire asks "how good a listener are you?". What it measures is self-image, not skill — and self-image drifts upward precisely where the skill is absent.

A situational judgement test gives you the scene and asks for a choice. A colleague has just presented your work as their own, in front of everyone, and you have ten seconds. What do you do? Your opinion of yourself is not scored. The choice is.

How scoring actually works

Each option carries points on specific behavioural indicators, and credit is relative to what was achievable in that scene. A scene where every option scores the same on an indicator carries no information about it and does not count.

While we have seen few of your decisions, the estimate shrinks toward the middle and its range is wide. As they accumulate, it narrows. That is why every number here travels with its range: "62" on its own would be a claim to precision we do not have.

What it cannot do

Being honest about the limits is part of the method:

• A diagnostic is not a diagnosis. It does not assess mental health and does not replace a professional.
• It measures judgement in situation, which predicts behaviour better than self-assessment but not perfectly.
• One measurement is a photograph, not an identity. Its value shows up in the second measurement.

The bibliography

• Ajzen, I. (1991). The theory of planned behavior

  Ajzen's paper presents the Theory of Planned Behavior, according to which human behavior is determined chiefly by the individual's intention to act. That intention is shaped by personal attitudes, social norms and perceived behavioral control. The theory is widely used in psychology, education, health and operations research.

  doi:10.1016/0749-5978(91)90020-T
• Amabile, T. M. (1996). Creativity in Context

  Teresa Amabile analyzes creativity as the product of the interaction between personal characteristics and environmental factors. She argues that creativity is strengthened where there is intrinsic motivation, freedom of expression and a supportive environment. The work is a foundation for research on creative thinking and innovation.

  doi:10.4324/9780429501234
• American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for Educational and Psychological Testing

  This book is the leading international standard for the development, evaluation and use of psychological and educational tests. It sets out guidance on reliability, validity, fairness and the ethical use of measurement in education and psychology.

  Source on apa.org
• Baddeley, A. D. (2000). The Episodic Buffer: A New Component of Working Memory?

  Baddeley introduces the episodic buffer, a new component of the working memory model. This component allows information from different cognitive systems to be integrated, and gives a better account of how the brain combines experience, language and memory.

  doi:10.1016/S1364-6613(00)01538-2
• Baker, R. S., & Inventado, P. S. (2014). Educational Data Mining and Learning Analytics

  The authors present the field of educational data mining and learning analytics. They explain how the data students generate can be used to personalize learning, predict performance and improve educational systems.

  doi:10.1017/CBO9781139519526.007
• Bandura, A. (1977). Social Learning Theory

  Learning does not always require personal trial and error: observing another person act is enough to acquire a new behavior.

  Source on search.worldcat.org
• Bandura, A. (1991). Social Cognitive Theory of Moral Thought and Action

  The study examines how moral thought and moral behavior develop. Bandura explains that self-regulation, social influences and cognitive processes work together in shaping moral decisions.

  Source on taylorfrancis.com
• Bandura, A. (1997). Self-Efficacy: The Exercise of Control

  This work presents the concept of self-efficacy, that is, a person's belief that they can attain a goal. High self-efficacy is associated with greater persistence, better learning and higher performance.

  Source on macmillanlearning.com
• Bar-On, R. (1997). The Emotional Quotient Inventory (EQ-i)

  Bar-On presents one of the first comprehensive instruments for measuring emotional intelligence. The EQ-i assesses skills such as self-awareness, empathy, stress management and social relationships.

  Source on search.worldcat.org
• Beghetto, R. A., & Kaufman, J. C. (2014). Classroom Contexts for Creativity

  The classroom environment is among the strongest factors determining whether creative potential is expressed or suppressed.

  doi:10.1080/13598139.2014.905247
• Black, P., & Wiliam, D. (1998). Assessment and Classroom Learning

  Interventions that strengthen frequent feedback to students produce substantial learning gains.

  doi:10.1080/0969595980050102
• Boekaerts, M. (1996). Self-Regulated Learning at the Junction of Cognition and Motivation

  Self-regulated learning facilitates the transfer of knowledge and skills to real situations and reduces the learner's dependence on the teacher.

  doi:10.1027/1016-9040.1.2.100
• Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The Concept of Validity

  A test is valid when the attribute it is supposed to measure actually exists and variations in that attribute causally produce variations in the measurement outcome.

  doi:10.1037/0033-295X.111.4.1061
• Bond, T. G., & Fox, C. M. (2015). Applying the Rasch Model

  The Rasch model makes it possible to place item difficulty and test-taker ability on the same scale.

  doi:10.4324/9781315814698
• Brackett, M. A., Rivers, S. E., & Salovey, P. (2011). Emotional Intelligence

  The ability model describes emotional intelligence as four underlying emotional abilities, not as a general personality trait.

  doi:10.1111/j.1751-9004.2010.00334.x
• Bransford, J. D., Brown, A. L., & Cocking, R. R. (2000). How People Learn

  The work brings together the most important findings of the cognitive sciences on how humans learn. It places particular emphasis on the importance of active participation, feedback and prior knowledge.

  doi:10.17226/9853
• Brown, T. A. (2006). Confirmatory Factor Analysis for Applied Research

  The book presents the method of confirmatory factor analysis (CFA), which is used to assess the structural validity of questionnaires and psychometric instruments.

  Source on guilford.com
• Butler, D. L., & Winne, P. H. (1995). Feedback and Self-Regulated Learning

  Feedback is not an external add-on to self-regulation but an integral and decisive element of its processes.

  doi:10.3102/00346543065003245
• Cialdini, R. B. (2001). Influence: Science and Practice

  Cialdini presents the six basic principles of social influence, such as reciprocity, social proof and authority. The book is widely used in the fields of psychology, marketing and leadership.

  Source on search.worldcat.org
• Chi, M. T. H. (2009). Active-Constructive-Interactive

  The terms "active", "constructive" and "interactive" are distinguished on the basis of observable activities and the underlying learning processes.

  doi:10.1111/j.1756-8765.2008.01005.x
• Cronbach, L. J. (1951). Coefficient Alpha and the Internal Structure of Tests

  Coefficient alpha equals the mean of all possible split-half coefficients obtained from the different ways of splitting a test in two.

  doi:10.1007/BF02310555
• Crone, E. A., & Dahl, R. E. (2012). Understanding Adolescence as a Period of Social-Affective Engagement and Goal Flexibility

  The authors analyse adolescence as a period of intense biological, cognitive and social change. They explain that brain development during adolescence increases sensitivity to social influences, but at the same time creates great potential for learning and adaptation.

  doi:10.1038/nrn3313
• Damasio, A. R. (1994). Descartes' Error

  Antonio Damasio rejects the traditional view that reason operates independently of emotion. He argues that emotions are a basic mechanism for making sound decisions and presents neuroscientific evidence supporting this position.

  Source on penguinrandomhouse.com
• Davidson, R. J., Jackson, D. C., & Kalin, N. H. (2000). Emotion, Plasticity, Context, and Regulation

  Two broad emotional systems, approach and withdrawal, rest mainly on the prefrontal cortex and the amygdala.

  doi:10.1037/0033-2909.126.6.890
• Davis, M. H. (1983). Measuring Individual Differences in Empathy

  Empathy is captured in four distinct subscales: perspective taking, fantasy, empathic concern and personal distress.

  doi:10.1037/0022-3514.44.1.113
• Dehaene, S. (2011). The Number Sense

  Stanislas Dehaene presents the neuroscientific foundations of mathematical thought. He explains that the human brain has an innate mechanism for understanding numbers, which develops through learning and education.

  Source on global.oup.com
• Decety, J., & Jackson, P. L. (2004). The Functional Architecture of Human Empathy

  Empathy requires experiencing something of another person's emotion without losing track of whose emotion it is.

  doi:10.1177/1534582304267187
• Deci, E. L., & Ryan, R. M. (1985). Intrinsic Motivation and Self-Determination in Human Behavior

  The book introduces Self-Determination Theory. The authors argue that autonomy, competence and social relatedness are the three basic psychological needs that strengthen intrinsic motivation.

  doi:10.1007/978-1-4899-2271-7
• Deci, E. L., & Ryan, R. M. (2000). The "What" and "Why" of Goal Pursuits

  Understanding motivation requires positing three innate psychological needs: competence, autonomy and relatedness.

  doi:10.1207/S15327965PLI1104_01
• DeVellis, R. F. (2016). Scale Development

  The book is a standard guide to creating scientific questionnaires. It presents all the stages of developing a measurement scale, from designing the items to testing reliability and validity.

  Source on collegepublishing.sagepub.com
• Diamond, A. (2013). Executive Functions

  Executive functions are made up of three core components: inhibition, working memory, and cognitive flexibility.

  doi:10.1146/annurev-psych-113011-143750
• Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). Grit

  Grit, as perseverance and passion for long-term goals, accounts on average for about 4% of the variance in success outcomes.

  doi:10.1037/0022-3514.92.6.1087
• Edmondson, A. (1999). Psychological Safety and Learning Behavior in Work Teams

  Psychological safety is defined as a shared belief among members that the team is safe for interpersonal risk-taking.

  doi:10.2307/2666999
• Embretson, S. E., & Reise, S. P. (2000). Item Response Theory for Psychologists

  The principles of item response theory can be grasped intuitively through graphs and analogies with familiar psychological principles.

  doi:10.4324/9781410605269
• Evans, J. St. B. T. (2008). Dual-Processing Accounts of Reasoning

  The paper sets out the two-systems theory of thinking. The first operates quickly and intuitively, while the second is slow, logical, and analytical. This theory has had a major influence on cognitive psychology.

  doi:10.1146/annurev.psych.59.103006.093612
• Efklides, A. (2006). Metacognition and Affect

  Anastasia Efklides examines the relationship between metacognition and emotion during the learning process. She shows that emotions affect the way learners monitor and regulate their learning.

  doi:10.1016/j.edurev.2005.11.001
• Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The Role of Deliberate Practice

  Elite performance is explained as the result of prolonged effort to improve under motivational and external constraints, not as an expression of innate talent.

  doi:10.1037/0033-295X.100.3.363
• Flavell, J. H. (1979). Metacognition and Cognitive Monitoring

  Flavell is regarded as the founder of the concept of metacognition. The paper describes how people can monitor and control their own cognitive processes, which leads to more effective learning.

  doi:10.1037/0003-066X.34.10.906
• Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active Learning Increases Student Performance

  Across 225 studies, active learning raised examination performance by 0.47 standard deviations compared with traditional lecturing.

  doi:10.1073/pnas.1319030111
• Goleman, D. (1995). Emotional Intelligence

  The established conception of human intelligence is too narrow and overlooks abilities that weigh decisively on the course of a person's life.

  Source on penguinrandomhouse.com
• Goleman, D., Boyatzis, R., & McKee, A. (2002). Primal Leadership: Realizing the Power of Emotional Intelligence

  The book sets out the decisive role of emotional intelligence in effective leadership. The authors argue that successful leaders stand out not only for their technical knowledge but above all for their ability to recognize, manage, and influence their own emotions and those of their colleagues. Different leadership styles and their effects on team performance are also presented.

  Source on store.hbr.org
• Graen, G. B., & Uhl-Bien, M. (1995). Relationship-Based Approach to Leadership: Development of Leader-Member Exchange (LMX) Theory

  This paper presents Leader–Member Exchange (LMX) theory, according to which the quality of the relationship between leader and employee significantly affects performance, job satisfaction, and commitment to the organization. The theory is widely used in human resource management and organizational psychology.

  doi:10.1016/1048-9843(95)90036-5
• Gross, J. J. (1998). The Emerging Field of Emotion Regulation: An Integrative Review

  Emotion regulation concerns which emotions one has, when one has them, and how one experiences and expresses them.

  doi:10.1037/1089-2680.2.3.271
• Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of Item Response Theory

  The book is one of the most important introductions to Item Response Theory (IRT). It presents the basic principles of developing reliable tests, the estimation of item difficulty, and the more accurate assessment of examinees' abilities.

  Source on uk.sagepub.com
• Hattie, J. (2009). Visible Learning

  The synthesis of more than 800 meta-analyses builds an overall picture of what really influences student achievement.

  doi:10.4324/9780203887332
• Hattie, J., & Timperley, H. (2007). The Power of Feedback

  Feedback is among the most powerful influences on learning, but its effect can also be negative.

  doi:10.3102/003465430298487
• Hmelo-Silver, C. E. (2004). Problem-Based Learning: What and How Do Students Learn?

  The paper presents Problem-Based Learning. Under this method, students develop critical thinking, collaboration and skills for solving complex problems through authentic scenarios and real-world situations.

  doi:10.1023/B:EDPR.0000034022.16470.f3
• Holmes, W., Bialik, M., & Fadel, C. (2019). Artificial Intelligence in Education

  The book examines the role of artificial intelligence in education. It analyses the potential of intelligent educational systems, personalised learning, automated assessment and learning analytics, as well as the ethical issues arising from the use of AI in education.

  Source on curriculumredesign.org
• Kahneman, D. (2011). Thinking, Fast and Slow

  Daniel Kahneman presents the two-system model of thinking. System 1 operates quickly, automatically and intuitively, while System 2 is slower, analytical and logical. The work explains how cognitive biases affect decisions and has had a major influence on psychology, economics and decision-making.

  Source on us.macmillan.com
• Kolb, D. A. (1984). Experiential Learning: Experience as the Source of Learning and Development

  David Kolb introduces Experiential Learning Theory, according to which learning is a cyclical process comprising concrete experience, reflection, the formation of abstract concepts and their active application. The theory is widely used in education, vocational training and skills development, as it holds that knowledge is acquired most effectively through experience.

  Source on pearson.com
• Lazarus, R. S., & Folkman, S. (1984). Stress, Appraisal, and Coping

  Lazarus and Folkman present one of the most important theories of stress management. They argue that stress depends on how the individual appraises a situation and on the coping strategies they use. The theory is widely applied in health psychology and in education.

  Source on search.worldcat.org
• LeDoux, J. (1996). The Emotional Brain

  Emotions — fear, anger, joy — are examined as products of specific brain mechanisms rather than as a single, undifferentiated experience.

  Source on simonandschuster.com
• Lieberman, M. D. (2007). Social Cognitive Neuroscience

  Social cognitive neuroscience is organised into four areas: understanding others, understanding the self, control of the self, and processes at the self–other interface.

  doi:10.1146/annurev.psych.58.110405.085654
• Lord, F. M. (1980). Applications of Item Response Theory to Practical Testing Problems

  Frederic Lord presents practical applications of Item Response Theory (IRT) for the development of modern psychometric tests. The book is regarded as foundational in educational assessment and in the construction of highly reliable examinations.

  doi:10.4324/9780203056615
• Luckin, R., Holmes, W., Griffiths, M., & Forcier, L. B. (2016). Intelligence Unleashed: An Argument for AI in Education

  At the core of artificial intelligence in education lies the aim of making computationally explicit the knowledge that usually remains tacit.

  Source on discovery.ucl.ac.uk
• Mayer, J. D., Caruso, D. R., & Salovey, P. (2008). Emotional Intelligence: New Ability or Eclectic Traits?

  Some individuals process emotion-related information in more complex ways than others and use that information to guide their thinking and behaviour.

  doi:10.1037/0003-066X.63.6.503
• Mayer, J. D., Roberts, R. D., & Barsade, S. G. (2008). Human Abilities: Emotional Intelligence

  Emotional intelligence comprises accurate reasoning about emotions and the use of emotions to enhance thought.

  doi:10.1146/annurev.psych.59.103006.093646
• Messick, S. (1989). Validity

  Samuel Messick formulated one of the most influential theories of validity in psychometrics. He argues that validity is a unified concept covering not only the statistical accuracy of a test but also the consequences of its use. To this day the work remains a basic theoretical foundation for the development and evaluation of educational and psychological instruments.

  Source on doi.org
• Miller, E. K., & Cohen, J. D. (2001). An Integrative Theory of Prefrontal Cortex Function

  Cognitive control arises from the active maintenance, in the prefrontal cortex, of patterns of activity that represent goals and the means of achieving them.

  doi:10.1146/annurev.neuro.24.1.167
• Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003). A Brief Introduction to Evidence-Centered Design

  Evidence-centered assessment design treats a test as a structured evidentiary argument.

  doi:10.1002/j.2333-8504.2003.tb01908.x

The full library holds 113 entries and is available inside the app.