All the news about our revolutionary HR tool, now on LinkedIn!

Research

SHORT SUMMARY


The JobTreffer screening was developed in cooperation with the University of Vienna within the framework of 4 studies. Further studies are in progress as well as in planning. The completed studies focused on the following topics:

  1. Preliminary study: Theoretical conception, construction of an item pool, and identification of normative items.
  2. 2 studies on the test-theoretical analysis of the item pool, calculation of the reilabilities of the scales, and development of preference profiles.
  3. Validation study through a so-called extreme group validation, in which real "top matches vs. non-matches" were examined with Jobtreffer.

Results:
  1. A differentiable item pool could be created with the exclusion of normative items
  2. After item selection, all scales showed good to very good reliabilities. Within the preference profiles, low inter-item correlations were found, confirming that the dimensions are asked in a differentiated manner.
  3. The content-related, discriminant and prospective validity of the procedure could be established. With the JobTreffer screening alone, 83% "top matches and non-matches" could be correctly assigned, which shows a highly significant result.
  4. The scientific basis of the method is constantly being enhanced, and further studies are already i nprogress or in planning.



ABSTRACT OF STUDIES 1 TO 3:

TEST-THEORETICAL CONSTRUCTION OF THE JOBTREFFER PROCEDURE IN COOPERATION WITH UNIV. VIENNA / DEP. WORK & ORGANIZATIONAL PSYCHOLOGY (JobTreffer: Monshi, Benesch et al. in cooperation with Univ. Vienna: Korunka, Senjak, Teksoy, Szapary 2023 ).


1. aim of the studies
In the context of the development of the multidimensional JobTreffer screening 3 studies were executed, where the first study served as pre-study.
One goal of the 3 studies was to construct a test theoretically founded questionnaire, which can seize the fit between employers and employees in several dimensions, scales and preference profiles.
From the resulting match profile an interview guide should be derivable, which gives both sides the possibility of asking informed and precise questions in the recruitment process. For practical purposes, this should thus support a comprehensive and efficient preparation of the initial interview in the recruiting process.


2. milestones for the scientific foundation of the JobTreffer screening
In a period of 2 years of development cooperation between JobTreffer GmbH and the University of Vienna / Department of Work & Organizational Psychology, the following milestones were completed:

  1. Theoretical derivation of factors relevant to occupational science for matchings between employers and employees
    The following matching dimensions were derived and evaluated as relevant:
    • Culture-Fit
    • Leader-Fit
    • Team Fit
    • Role-Fit and
    • Task Fit
  2. Construction of an item pool in reference to the theory of the corresponding dimensions.
  3. Pre-study to check the item pool for "differentiable items".
    The items were given to a first sample (n=111) and then the statistical response distributions per item were checked. In the JobTreffer matching, only items with differentiable response behavior may be used. Items with normative character show statistically too little response distributions, where the effect of "social desirability" might lead to a response bias. Within the recruitment process, the pressure to answer in a socially desirable way increases and therefore items (e.g. fair cooperation) had to be excluded, since they ultimately do not enable a differentiating matching profile in the application process.
  4. Test-theoretical analysis and final item selection: The questionnaire with the preselected dimensions, scales, and items was then given to a sample of 466 subjects aged between 18-68 years from various social and educational backgrounds.
  5. The JobTreffer screening: Based on comprehensive test-theoretical item analyses and reliability measurements (=measurement accuracy) of all individual scales, the current version of the "JobTreffer Screening" was created. It should be noted that the scales were constructed with very few items (2-4) for reasons of efficiency and reasonableness. The small number of items per scale makes it very difficult to obtain high reliability coefficients (Cronbach's alpha). Accordingly, the reliability measurements in all scales can be judged as good to very good.
  6. In addition, so-called preference profiles were developed in order to be able to collect data about preferences with regard to working conditions in a differentiated manner.
  7. Final assessment of the test quality criteria:
    1. Reliability
      In the test-theoretical foundation, the strict benchmarks for differential diagnostic psychological questionnaire procedures were applied. The JobTreffer matching fulfills these criteria and can be regareded as a reliable procedure.
    2. Objectivity
      The main quality criteria of objectivity is fully guaranteed by the digital architecture of the procedure in terms of standardized specification, execution and evaluation.
    3. Validity
      Content-related, discriminant, and prospective validity of the procedure were demonstrated. For more details, see the description of study 4 (not the content of this abstract) about validity testing.


3. concluding remark:
It should be emphasized at this point that the JobTreffer matching procedure is not intended as the only selection criteria, despite very good characteristic values of the digital architecture of the procedure. The procedure serves as a supporting tool during the preparation of the interview and as an instrument to generate hypothesis. This should lead to unprejudiced and well-founded decisions in recruitment and people development.


Appendix

  1. Reliabilities of the individual scales per matching dimension
    1. CULTURE-FIT scales
      • Relationship culture: 0.70
      • Success culture: 0.75
      • Perfection culture: 0.73
      • Innovation culture: 0.68
      • Home office: 0.84
    2. LEADER FIT scales
      • Autonomy: 0.71
      • Direct control: 0.73
    3. TASK FIT scales
      • Task variety: 0.72
      • Cognitive difficulty: 0.81
      • Physical difficulty: 0.74
      • Emotional difficulty: 0.84
  2. Preference profiles
    So-called preference profiles were derived during the development of the JobTreffer screening. Data about preferences of employees regarding the working environment and conditions are collected and compared with the employer's information about these working conditions. Considering preference profiles it was important that the items within a profile (e.g. types of team stability) correlate as little as possible, because a differentiated perspective with independent characteristic values should be represented.
    CULTURE FIT preference profiles
    • Time flexibility
    • Overperformance
    • Development opportunities

    LEADER-FIT preference profiles
    • Responsibility for results
    • Feedback by manager

    TEAM-FIT Preference Profiles
    • Team decisions
    • Team stability

    ROLE-FIT Preference Profiles
    • Problem Solving
    • Relationship work
    • Proactivity

    TASK-FIT Preference Profiles
    • Importance of work
    • Distribution of the amount of work


STUDY 4: EXTREME GROUP VALIDATION:


The central question of this study was:
"Can a multidimensional JobTreffer screening distinguish between individuals who have shown to be very good matches for their jobs in practice ("top-matches") and those who have shown not to be good matches ("non-matches")?"

  1. Methodological approach
    To answer this question two extreme groups were surveyed:
    • "Top matches": individuals as well as their employers are very satisfied with their job and there is no willingness to change job.
    • "Non-matches": individuals, who are very dissatisfied with their job and therefore have a high willingness to change job.
  2. The sample
    A total of 51 people and their employers* answered the JobTreffer screening. Subsequently, their information was evaluated with the Job-Treffer algorithm and the matching profiles were calculated.
  3. The hypotheses
    The following hypotheses were tested:
    1. Proven top matches have higher matching scores than non-matches.
    2. 2) The JobTreffer screening can be used to reliably predict whether a matching result refers to a top match or a non-match.
  4. The results
    The following findings were obtained:
    ad1) "Top matches" have significantly higher matching scores than "non-matches". This speaks for a high discriminant validity of the procedure.
    ad2) By means of a logistic regression analysis, it could furthermore be determined that the JobTreffer matching correctly identified the individuals as "top matches" or "non-matches" by 83.3%. Thus, a high predictive validity of the matching procedure could be shown.
  5. Conclusion
    The results of this study show that the primary quality criteria of validity is very well met. Additionally, it is important to note that the JobTreffer matching procedure is not intended to serve as the sole decision criteria in personnel selection or for people development. By incorporating additional data, such as qualifications, and delving into the matching profile results through a professionally conducted job interview, a high level of practical accuracy can be achieved.
    As a qualitative side effect, it is worth emphasizing that users report that the multidimensional matching profile diminishes the significance of traditional personal attributes, such as age, gender, appearance, or cultural background, in the perception. JobTreffer thus establishes an objective foundation for unbiased personnel selection (considering Unconscious Bias).
    Furthermore, ongoing long-term studies in collaboration with the University of Vienna are already in progress.

Appendix: Study 4:

Ad1) Testing the difference hypothesis ("top matches" vs. "non-matches") using Mann-Whitney U test.


Ad1) Testing the difference hypothesis ("top matches" vs. "non-matches") using Mann-Whitney U test.