The questions of validity and reliability are often raised with regard to Discus and other similar systems. This section of the site reproduces a South African study, A Reliability and Validity Study on the Discus Personality Profiling System, by Karin Roodt of the prestigious Technikon Natal. Note that this study - known as The Roodt Report - relates not just to DISC in general but specifically to the Discus software.
K. Roodt (Ms)
M Ed (Counselling and Guidance) (UNISA)
Senior Lecturer: Department Human Resources Management, Technikon Natal
The purpose of the investigation was to determine whether the Discus measuring instrument could be considered a reliable and valid instrument. The test-retest method was used in the reliability study and was administered to 90 employees from a variety of companies in Kwa Zulu-Natal and Gauteng. The Pearson's product-moment correlation coefficient was used and correlation scores of 0.728 (Dominance), 0.645 (Influence), 0.730 (Steadiness) and 0.550 (Compliance) were established. The p-value in all the cases was as low as 0.0001. This indicates significance at alpha = 0.001. It can therefore be concluded with 99.9% level of confidence that the Discus instrument is reliable.
In the validity exercise criterion-related validity was used. An exploratory study was undertaken in order to determine which of the 15 Factors (Factor B excluded) of the 16-PF correlated with the four dimensions of the Discus. One hundred and twenty respondents in South Africa were involved for this purpose. The Pearson's product-moment correlation coefficient was applied. It was found that Factors Q1, X=G, L, Q1 and X=Q2, E; E, Q2 and -I show significant correlations with Dominance at the 1% and 5% level of significance. Factors A, -Q2, H, F and -Q3 show significant correlations with Influence at the 1% and 5% level of significance. Factors -E and -Q1 show a correlation with Steadiness at the 5% level of significance. Factors -E, Q2, -H, -G and O show significant correlations with Compliance at the 1% and 5% level of significance. It can therefore be concluded that the correlations were significant.
What follows is the outcome of a reliability and validity study as performed by Karin Roodt, M Ed (Counselling and Guidance) UNISA, Registered Psychologist (SAMDC), Senior Lecturer, Technikon Natal (serving as Project Manager), assisted by Charles Robert, BSc (Hons)(Stats): Statistician, Director of South Africa on Line.
I would like to thank the following companies and individuals for participating in this study and in some way contributing to the final report:
A study of this nature must be approached with caution because no single psychological tool is able to yield everything about a person's personality. A test battery, i.e. a collection of several tests, reveals far more about an individual than does any single test of personality, preference, interests, intelligence or general personality dynamics.
"The Discus personality profiler is a completely computerised assessment tool, designed to describe the different roles a person fulfils in the work environment."
The word 'personality' has been debated for centuries. Everybody has their own idea about exactly what it means. In Discus terms a personality is defined as the sum of all a person's varying response styles to varying stimuli (Swanepoel 1995:1). In practical terms, however, it is impossible to measure and evaluate every one of a person's possible responses to every possible stimulus.
The purpose of this study is to determine whether the Discus instrument may be considered a reliable and valid instrument for assessing personality.
In the reliability exercise, the test-retest reliability technique was used. According to this method the same instrument is applied to the same respondents at a later stage and the correlation between the two scores is then calculated (Huysamen 1980:54; Mulder 1981:211).
The questionnaire was administered by the respective people participating in the exercise. All of these participants are trained in Discus and how to administer the instrument. The instrument was administered for the first exercise to obtain a pretest score. The exercise was then repeated with the same respondents after a period of three months in order to obtain a post-test score.
A statistical evaluation of the raw data, resulting from the exercise, was then obtained by using the SAS system, reflecting Pearson's Product-moment correlation coefficient (coefficiency of stability).
The questionnaire consists of 24 questions each of which presents the respondents with four options. The respondents' task is to select one of the options that most closely resembles themselves, and one that least closely describes them. The respondents are required to focus on the role they fulfil in their work environment and answer all the questions in relation to that role.
For the purpose of this exercise the phrase-based questionnaire was used because it is easier to understand.
Various companies were approached to assist with the exercise as reflected in table 1.
TABLE 1: COMPANIES USED
Edgars Group (Gauteng)
Toyota South Africa (KZN)
The questionnaire was administered to 90 respondents. These respondents were randomly selected from the respective companies reflected in table 1. A statistical evaluation of the raw data resulting from the testing was then obtained by using the SAS system reflecting Pearson's product-moment correlation coefficient.
The correlation analyses are reflected in table 2.
TABLE 2: CORRELATION MATRIX:
BEFORE AND AFTER SCORES
Pearson's Product-Moment Correlation Coefficient
r = values
p = values
The significance level chosen for this instrument is alpha = 5%. Where the p-value is less than 0.05, the scores show a significant correlation. In the reliability analysis the p-value in all the cases is as low as 0.0001. This indicates significance at alpha = 0.001. It can therefore be said the correlation is significant at 1% level.
The reliability coefficient of the measuring instrument is close to 1 and can therefore be seen as reliable.
Content validity of the instrument is determined when the instrument is designed. Content validity refers to the extent in which the instrument measures what it is supposed to measure (De Wet, De K Monteith, Steyn& Venter 1981:146; Huysamen 1980:95; Mulder 1989:219).
Each question in the Discus instrument was evaluated by the designers of the instrument, namely Axiom. Although content validity was done by Axiom, the researcher also decided to measure validity in terms of criterion-related validity.
By applying the method of criterion-related validity, an exploratory study was done by correlating all 15 Factors in Cattell's 16-PF with the four dimensions in the Discus. Factor B was not considered as doubts exist in the literature as to the validity of Factor B (intelligence) within a personality test.
Criterion-related validity was restricted to validation procedures in which the test scores of a group of respondents are compared with ratings of other measurements (Aiken 1994:96).
Nunally (1978) claims that it is unrealistic to expect exceptionally high correlation coefficients and Anastasi (1976) says that coefficients of 0.20 and higher can be significant.
In this exercise employees of the Edgars group, Toyota South Africa and Technikon Natal were used. It was therefore decided to use the Discus and Cattell's 16-PF (Form A) for this exercise.
In an attempt to determine a correlation between the Discus dimensions and Cattell's 16-PF, scores on the 16-PF were obtained from 120 employees employed by the abovementioned companies. These respondents were randomly selected from line managers, middle managers, professionals and junior officials. The sample was drawn from all organisational functions and cultural groups within the organisations.
This questionnaire has already been discussed in paragraph 3.1.1.
The 16-PF is specifically constructed for the purpose of determining individual attitudes, perceptions and personality characteristics. It was developed by R.B. Cattell and published in 1949. The A and B Forms of the test consist of 187 items each and are suitable for adults with at least standard 10 or equivalent education. The 16-PF can be used for the evaluation of personality in people of different population groups because it is culture friendly (Prinsloo 1992:21-22).
Cattell applied the technique of factor analysis and obtained a set of 16 primary factors. The rationale behind the 16-PF is that a questionnaire which is based on revealed traits, obtained through mathematical techniques from a large pool of possible personality descriptions, is capable of measuring reliably and validly the true constructs present in humans.
The general purpose of the 16-PF is to describe testees' personality and predict behaviour using a set of selected, structured items. The test has many practical applications, some of which are mentioned below:
TABLE 3: THE HIGH AND LOW FACTORS OF THE 16-PF
1 2 3 4 5 6 7 8 9 10
For interpretation purposes, factor scores of 1 to 3 and 8 to 10 are considered. The low numbers of each factor are pictured as portraying one extreme of the profile and the high numbers as portraying the other. It should be pointed out that Cattell, in analysing all 16 factors, came up with clusters of several adjectival descriptors for each factor. Table 3 depicts words representative of factor clusters. The Kuder-Richardson 8-method was used to determine the reliability of the 16-PF. Table 4 reflectsthereliabilityfigure for eachfactor.
TABLE 4: RELIABILITY SCORES FOR 15 FACTORS OF 16-PF REFLECTING THE KR-8 SCORE
SOURCE: Prinsloo 1991:23
The Discus and the 16-PF were administered by trained and registered psychologists. The marking and interpretation of the 16-PF questionnaires were done manually by the researcher, a registered psychologist, and by psychologists from the HSRC.
Pearson's product-moment correlation coefficient was computed, using the SAS-system to determine a correlation between the identified fifteen 16-PF factors and the Discus dimensions.
The 120 questionnaires that were returned were statistically analysed. Scores for each of the fifteen 16-PF factors as well as the Discus dimensions were correlated. The individual Discus variable scores were then correlated with all the fifteen factors of the 16-PF, resulting in significant correlations at 1% level of significance and at 5% level of significance. The results of these findings are reflected in tables 5A (p-values) and 5B (r-values).
TABLE 5A: COMPARISON BETWEEN THE DISCUS DIMENSIONS AND THE FIFTEEN FACTORS OF THE 16-PF. PEARSON'S PRODUCT-MOMENT CORRELATION COEFFICIENT
(5% LEVEL OF SIGNIFICANCE)
(1% LEVEL OF SIGNIFICANCE)
1% Level ofSignificance
5% Level ofSignificance
X (Q2, E)
X (G, L, Q1)
TABLE 5B: COMPARISON BETWEEN THE DISCUS DIMENSIONS AND THE 16-PF FACTORS
1% Level of Significance
5% Level of Significance
It was found that the Discus instrument is reliable at a significance level = 0.01.
In the validity exercise it was found that the majority of the factors of the 16-PF show a significant correlation with all four dimensions of the Discus using the p-values. The second order factors of the 16-PF were not used for the purpose of this study.
Suelz (1997) is currently busy with a validation study on the Discus using respondents from the Edgars group in Gauteng as his sample size. He has not yet published his masters thesis. It would be interesting to compare the findings of his research with the findings in this study.
he following recommendations have been identified which could open avenues for further research:
Their advice is gratefully acknowledged.
Graph 1: Reliability Analysis:
Scatter Plot Dominance Before vs. Dominance After
Graph 2: Reliability Analysis:
Scatter Plot Influence Before vs. Influence After
Graph 3: Reliability Analysis:
Scatter Plot Steadiness Before vs. Steadiness After
Graph 4: Reliability Analysis:
Scatter Plot Compliance Before vs. Compliance After