Table of Contents

• Executive summary
  

• Introduction
  
  • Background
    

  • Employer Sample
    

  • Rating Scales and Analysis
    

• Description of Employer Respondents
  
  • Type of Employer Organization

  • Employees Within the Organization

  • Hiring Practices
    

• Description of Alumni Who Were Rated
  

• Communication Skills
  

• Technical Skills
  

• Workplace Skills
  

• Professional Traits and Attitudes
  

• Educational Preparation
  

• Conclusions
  

• Appendix 1
  
  

Return to OIRP Survey Page
Return to OIRP Home Page

Return to Table of Contents
Return to OIRP Survey Page

Introduction

Background

In the search for a comprehensive approach to assessing outcomes, alumni and employer assessments have the advantage of ranking high on believability and utility for both formative, faculty-driven assessment purposes as well as for summative evaluations at the system or state level. NC State University developed alumni and employer surveys with the understanding that undergraduate program impact could best be measured by assessing the knowledge, skills, and abilities that connect NC State to the world of work. Two sets of measurement constructs are apparent from other institutions' research: evaluating the relative importance of a knowledge or skill area to a graduate's professional position, and evaluating the actual knowledge and skill level of graduates.

With this theoretical background in mind, NC State surveyed baccalaureate alumni who graduated between December 1990 and August 1993. Items concerning professional preparation, general education goals, further education, and employment were asked. Three full mailings and a telephone follow-up resulted in a 51.2% response rate, yielding 3,179 usable questionnaires. The results are broadly representative of the population of NC State bachelor's graduates for those years. Findings from the alumni survey are published in several reports, available from Office of Institutional Planning and Research.

Employer Sample

On the alumni survey instrument, the researchers requested permission to survey direct supervisors. Complete contact information was provided by 975 of the responding alumni (30.6%). Two mailings to these employers produced 616 usable responses, a 67.8% response rate. Employers were asked questions identical to the professional preparation items on the alumni survey.

No significant differences between the alumni who were rated by their employers and the initial population of alumni surveyed were found by gender, ethnicity, year of graduation, or college - a situation that speaks well for the generalizability of the findings to the population of employers of NC State alumni. In addition, very few items had significant differences by ethnicity or gender. The overall margin of error was 3.8% (given a 95% confidence level), and item reliabilities (Cronbach's Alpha) of .95 or higher were reached.

Rating Scales and Analysis

Employers were asked to indicate the importance of items to the NC State graduate's job. The scale ranged from 5 = very important to 1 = not important. Additionally, employers were asked to rate the graduate's knowledge and skills compared to other employees at the same level and in the same capacity. The scale ranged from 5 = excellent to 1 = poor. Responses in the "not applicable" category were omitted in calculation of the mean score for both importance and skill items. In order to facilitate interpretation of the results of this survey, similar items are grouped together in the discussion below.

In this report, three distinct pieces of information are presented on each survey item concerned with professional preparation. First, mean ratings of the relative importance of each area to graduates' jobs are presented. Next, mean ratings of the graduates' knowledge and skills in each area are shown. For items disaggregated to the college/school level, T-tests of significance between each college/school mean and the university-level mean were run. Certain items were also analyzed by type of employer organization, using ANOVA with Duncan's multiple comparison procedure. Finally, the means of observed differences between levels of importance and levels of knowledge/skills are presented. These means of observed differences are labeled "gap scores" and are included to provide an indicator of the size and direction of areas for improvement (indicated by a negative gap score) or areas in which level of knowledge/skills exceeds perceived workplace requirements (indicated by a positive score). T-tests were conducted to determine whether or not these gap scores were significantly different from zero; significant gap scores are noted in the tables. When viewing the results of this survey, it is important to remember that all three elements should be considered - importance, knowledge/skills, and gap scores - before arriving at a judgment as to the correct interpretation of the findings.

As a guide to interpreting the relevance of the gap score, in any instances where the gap score is larger than -0.5, the item most probably merits further investigation. In other cases, it may be useful to examine the standard deviations for both importance and knowledge/skills. In situations where the gap score is negative and larger than the standard deviation of either the importance or knowledge/skills rating, further investigation may be warranted. In situations where the gap score for a particular item is negative and larger than the standard deviation for both importance and knowledge/skills ratings, a real difference between importance and knowledge/skills is implied and further investigation is indeed warranted. Note: due to rounding and to the differing number of respondents to items on importance and knowledge/skills, mean gap scores may not exactly total the difference between the mean importance and mean knowledge/skills ratings.

Return to Table of Contents
Return to OIRP Survey Page

Description of Employer Respondents

Type of Employer Organization

Employers were asked to classify their organization into one of several types, including an other category. Most of the other responses fit into one of the four original types. The remainder were classified as Religious/Non Profit, except for five responses in which the type could not be ascertained. Of the 606 respondents to this question, 43% classified their organization as an Industry, and an additional 21% considered themselves Professional Service firms. The breakdown by type is presented in Chart 1.

Employees Within the Organization

Employers were asked the approximate number of people employed in the organization at that site. One-fourth (25%) of the respondents reported less than 20 full-time employees, with a total of 55% reporting less than 100 full-time employees. Only 12% of the organizations reported 1,000 or more full-time employees at that site. More than three-fifths (61%) employed less than 10 part-time persons.

Twenty-two percent of employer respondents reported that the NC State graduate was the only person at that site who held that job title. Seventy percent of the organizations employed between one and nine persons (including the graduate) with the same job title as the graduate. Eighty-four percent of respondents said that their organization had hired other NC State graduates.

Approximately one-third (34%) of the respondents reported that only one NC State graduate was employed in their organization at this site, with an additional third (37%) employing between two and four NC State graduates. Only 6% employed 20 or more graduates at that site.

Hiring Practices

From a list of typical resources used in an organization's search for employees, employers were asked to choose the one most useful resource. 497 employers responded to this question. The highest percentage (32%) indicated that direct contact with potential employees was the most useful. Fourteen percent chose newspaper advertisements, followed by twelve percent choosing referral from employees or acquaintances. Results are presented in Table 1.

Employers were asked the three most important things they or their organization looks for when hiring new employees. The most frequent answers given by respondents were:

• educational preparation and knowledge (n = 260)
• general work or job-related experience (n = 180)
• communication skills (n = 128)
• ability to work in teams and with people (n = 123)
• work attitude (n = 115)
• technical knowledge and skills (n = 114)
• motivation/initiative/desire (n = 105).

Other frequent responses include ability and desire to learn (n = 64), ability to adapt to changes and pressures on the job (n = 61), and ability to work independently (n = 41).

Return to Table of Contents
Return to OIRP Survey Page

Description of Alumni Who Were Rated

Demographic characteristics of alumni are reported in Appendix 1. Over three-fourths (78%) of employer respondents reported that the minimum educational requirement for the graduate's position was a Bachelor's Degree, while 18% required less than a four-year degree.

Seventy-one percent of employers responded that they had daily contact with the alumni rated, and an additional 23% reported contact several times per week - which speaks positively for their ability to accurately rate NC State alumni. Over half (54%) had known the graduate less than three years, with 13% reporting knowing them less than one year.

Notably, 52% of the employer respondents stated that the NC State graduate rated was much better or somewhat better educationally prepared than other employees. Only three percent of the alumni were rated as less prepared than other employees. Results are reported in Table 2. Further validating the high preparation level of alumni is the finding that 55% of alumni rated had been promoted since joining the employer's organization.

Return to Table of Contents
Return to OIRP Survey Page

Communication Skills

Employers perceived general communication skills to be very important to NC State graduates' current positions; particularly listening skills. Graduates' communication skills were judged to be good, with employers rating graduates as most skilled in reading and listening. Foreign language skills were rated lowest in importance and skill level. These ratings are detailed in Table 3 and Chart 2.

Moderate negative gap scores exist on communication skills overall and listening skills, which means employers rated graduates as somewhat deficient in these areas compared to what they felt was important to graduates' jobs. However, employers felt that graduates had higher foreign language skills than were needed on their jobs.

Return to Table of Contents
Return to OIRP Survey Page

Technical Skills

Employers perceived technical (computer, scientific, and math-related) skills to be important to graduates' jobs. The most important was overall technical knowledge. Graduates were rated as having good technical skills, especially basic computer skills and ability to apply mathematical skills. Ratings for each skill area are presented in Table 4 and Chart 3.

Employers rated NC State graduates as better skilled on the four specific technical areas than was expected for their jobs, as evidenced by the positive gap scores. Graduates were seen as especially well prepared on technical computer skills, but the large standard deviations indicate a wide range of ratings by the employers. Further analysis indicated that religious/non-profit organizations rated all technical skill areas significantly lower in importance than did the other types of organizations.

Analysis of technical skills by college/school revealed several significant differences between college/school means and university-level means, with Engineering and Physical and Mathematical Sciences rated higher and Humanities and Social Sciences rated lower. Specifically, for technical computer skills, five colleges showed moderately high positive gap scores (with Engineering and Humanities and Social Sciences both significant) while Physical and Mathematical Sciences was rated as slightly underprepared (see Table 4b). Again, wide standard deviations exist for most importance and knowledge/skills means, indicating a diversity of employer opinion even within the academic units. Caution should be exercised when interpreting these analyses for the School of Design, however, because the small number of respondents raises validity concerns about the representativeness of the sample to the population.

Return to Table of Contents
Return to OIRP Survey Page

Workplace Skills

Workplace Skills are defined as skills and abilities that connect NC State to the world of work. They deal with higher order conceptual and analytical abilities. Employers perceived these workplace skills to be important to graduates' jobs, with the highest ratings on areas concerned with problem solving. Graduates' workplace skills were rated as good, particularly using knowledge to solve problems overall. Results are shown in Table 5 and Chart 4.

The gap between the employers' rating of graduates' knowledge and skills and rating of the importance of those skills is also shown in Table 5. The NC State graduates rated had higher skills than expected from previous work, volunteer, or internship experience. However, on the rest of the workplace areas graduates were seen as somewhat deficient. In particular, NC State graduates seemed to have less skills in dealing with problems (using knowledge to solve problems overall, solving problems, defining problems and leadership and management) than were demanded of their jobs.

Because employers rated problem-solving skills as integral to graduates' jobs, further analysis by college/school was performed on several items. Caution should be exercised when interpreting these analyses for the School of Design, however, because the small number of respondents raises validity concerns about the representativeness of the sample to the population.

As revealed in Tables 5a and 5b, employers felt NC State alumni in most colleges/schools were not very well skilled in problem-solving (gap scores were significant for seven of the nine academic units). In particular, Forest Resources had the largest significant negative gap score for both using knowledge to solve problems and solving problems. Engineering and Humanities and Social Sciences also had significant negative gap scores for defining problems and leadership and management skills.

Further analysis of workplace skills by organization type was conducted. Three of the workplace skills with large gap scores were significantly different by organization type for the ratings of importance of those skills to the graduates' job. Religious/Non-Profit organizations rated thinking creatively as more important than did Professional Service firms, Industries, and Government organizations; rated defining problems more important than did Professional Service firms and Industries; and rated solving problems more important than did Industries. These breakdowns are shown in Chart 5.

Return to Table of Contents
Return to OIRP Survey Page

Professional Traits and Attitudes

Information on professional traits, work attitudes, and professional development was gathered through nineteen items. Employers perceived that these traits and attitudes were fairly important to graduates' jobs. Those rated the highest in importance were conducting work activities in an ethical manner, work attitudes and skills overall, understanding and carrying out assignments, and attitude toward work. Graduates' knowledge and skills in these areas were rated good to excellent, with the highest being conducting work activities in an ethical manner, being dependable and punctual, willingness to accept new responsibilities, and attitude toward work. Results are displayed in Table 6 and Chart 6.

Negative gap scores exist for all items, and all but one were significant. In particular, employers felt that graduates seemed to have trouble dealing with pressure and adjusting to new job demands, as evidenced by moderately large negative gap scores in making decisions under pressure, working under pressure, and ability to adjust to new job demands. On a positive note, graduates were rated only slightly less prepared than was needed for their jobs in ability to work with persons from diverse ethnic and cultural backgrounds.

NC State alumni were rated by employers as lacking in skills related to the rigors of adjusting to job demands and pressures. Analysis by college/school (see Tables 6a - c) revealed significant negative gap scores in ability to adjust to new job demands, working under pressure, and making decisions under pressure for Agriculture and Life Sciences, Education and Psychology, Engineering, and Humanities and Social Sciences. Forest Resources had the largest negative gap scores for all three items. The moderately large standard deviations for many of the level of knowledge and skills items should be noted. Caution should be exercised when interpreting these analyses for the School of Design because the small number of respondents raises validity concerns about the representativeness of the sample to the population.

Return to Table of Contents
Return to OIRP Survey Page

Educational Preparation

Employers perceived that educational preparation was moderately important to graduates' jobs. NC State graduates were rated as better educationally prepared overall than might be required for their jobs. All three academic areas had positive gap scores, with the largest for GPA. These results are presented in Table 7 and Chart 7. As expected, academic organizations rated all three education areas as significantly more important to graduates' jobs than did the other four types of organizations.

Return to Table of Contents
Return to OIRP Survey Page

Conclusions

Assessments of alumni professional preparation are generally gathered from the alumni themselves, and occasionally from their employers. Ratings from direct supervisors are quite often considered more believable and trustworthy - after all, they are perhaps the ultimate customers of graduates' knowledge and skills. In general, employers of NC State's alumni in this study report that graduates are fairly well prepared for the important aspects of their jobs. In particular, alumni are highly prepared on technical skills such as computer and mathematical/ scientific areas (as would be expected from a university such as NC State) and better educationally prepared than might be required. Nonetheless, employers feel our graduates are somewhat less than optimally prepared to deal with problems and pressures on the job.

A comparison of ratings by employers and the alumni themselves reveals little agreement for individual students, except on technical items. Moderate agreement at the university level was found on importance of most items to the graduates' jobs. Generally, alumni overall rated themselves as slightly less prepared than did their employers.

Return to Table of Contents
Return to OIRP Survey Page

Return to Table of Contents
Return to OIRP Survey Page
Return to OIRP Home Page