2007 Reaffirmation Teams
18.104.22.168 - 2007 Reaffirmation Teams
The institution identifies expected outcomes, assesses whether it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: educational programs, including student learning.
Each of the academic programs, including majors and academic certificate programs, at The University of Texas at Dallas (UT Dallas) identifies student learning outcomes, annually assesses whether the program has achieved those outcomes, and uses the assessments to improve the program.
UT Dallas applies a distributed approach to assessment, which includes internal auditing, program accreditation, and certification of students. When taken together, this approach ensures an effective evaluation of educational programs that is distributed across the entire university and not housed solely in one administrative office.
As UT Dallas’ program assessment flow chart demonstrates, the review of academic programs at UT Dallas is a dynamic process with multiple feedback loops to identify areas that need improvement and to make adjustments based on the data collected . Academic assessment occurs at two basic levels-academic program level and individual course level with emphasis on undergraduate core curriculum courses.Academic Program Level
Annual Academic Program Assessment: Program assessment is a critical part of UT Dallas’ continued focus on institutional effectiveness. UT Dallas has a rigorous process of program review and assessment that ensures that expected outcomes are clearly defined and measurable and are used for improving education. Each academic degree program as well as each academic certificate program at UT Dallas is assessed annually using UT Dallas’ online assessment tool, AT6 . AT6 is a web-based solution to capture, manage, archive, and track academic and administrative assessment information for regional and disciplinary accreditation, program reviews, annual reporting, and program improvement. By leveraging the power and flexibility of the web, users can capture or retrieve data anytime, anywhere. Deans, program heads, department heads, assessment officers, and other designated faculty work with instructors to determine program objectives generated from program mission statements and to enter the mission statement and program objectives along with an assessment plan into AT6 . These plans are reviewed annually by the faculty and updated in light of the findings in the annual assessment report . This report not only includes measurements and analyses but also evidence of improvement in student learning as well as plans for improvement. Sample assessment process minutes from the master’s and Ph.D. programs in Public Affairs , from the bachelor of science program in Speech-Language Pathology and Audiology , the master’s program in Communication Disorders, and the Ph.D. program in Communication Sciences and Disorders , and from the Erik Jonsson School of Engineering and Computer Science (ECS)  demonstrate the review process and provide evidence of UT Dallas’ use of student learning outcomes in the assessment process.
The university has recently finished its second online assessment cycle (2007), and the third cycle (2008) is currently being entered into AT6. All 117 implemented academic degree programs (the M.S. in psychological sciences has been authorized but not implemented)  and all 24 academic certificate programs  were fully assessed in 2006 and 2007, and the assessments were approved by the Office of the Executive Vice President and Provost’s assessment team  . The team will continue to monitor the progress in the 2008 cycle.
Academic Program Assessment Process: UT Dallas’ deans, program heads, and department heads, with the approval of their faculty constituents, implement the academic assessment process in the following sequence:
- publishing program mission statements,
- identifying several measurable student learning goals,
- specifying how the programs will measure their achievements,
- assessing how well the goals have been met, and
- taking corrective action to improve the program.
To educate the faculty about the assessment process as the university transitioned toward online assessment in 2006, the assessment team asked the deans, program heads, and department heads eight basic questions about how they had improved their programs in the past:
- How you have improved your program? (assessment outcomes)
- How did you know it needed improving? (assessment monitoring)
- What was the basis of your decision to improve? (assessment measures)
- What were your goals for improvement? (learning objectives)
- How you will know that your improvements met your goals? (assessment data)
- How often you will check to see whether the improvements are working? (assessment cycle)
- Who will check up on improvements? (assessment accountability)
- What will you do if improvements are not working the way you want them to work? (closing the loop)
Their responses are documented in the program assessment response from seven schools . This material was used by the faculty as the background information once the formal assessment process went online.
AT6 has six basic steps: (1) mission statement, (2) objectives, (3) measures, (4) findings, (5) closing the loop, and (6) analysis, all of which are used to produce the final report that focuses on improving programs by closing loops discovered in the assessment process. Members of the faculty in each program at UT Dallas meet at the beginning of each assessment cycle to discuss their plan and at the end of each cycle to discuss the performance of their programs with regard to meeting program objectives that might include critical thinking, improved computational skills, more sophisticated senior projects, and/or improved retention and graduation rates. The faculty adjust the objectives and measures based on the material gathered in AT6   . Sample program assessment reports include the B.S. in electrical engineering , M.S. in communication disorders  and Ph.D. in public affairs . The complete suite of program assessment reports generated by AT6 for 2006 and 2007 is available in the AT6 Navigator .
In order to ensure the quality of the program assessments, the assessment team works with every degree program at UT Dallas to determine how programs and courses mesh with good educational practice and the mission and goals of the university. As mentioned above, the faculty in each academic program have chosen a specified slate of student learning objectives (SLO’s) to ensure that, as the result of each student’s participation in the academic major and course work, students achieve the requisite knowledge, skills, and values associated with the discipline; SLO’s are posted on the UT Dallas website . These SLO’s require multiple measures (or assessments) to determine the degree to which students meet those objectives or goals One of the resources most commonly used by academic programs in developing assessment plans and writing assessment reports is UT Dallas’ assessment workbook , which discusses how to use both direct assessments in courses and indirect assessments such as student and employee surveys and employment rates to produce assessment measures and reports and to devise ways to close the loop.
To continue to improve the assessment process, the assessment team in conjunction with the Center for Excellence in Learning and Teaching (CELT) frequently runs learning assessment workshops for program heads, department heads, and faculty members. The general agendas in these workshops cover a variety of topics such as how to write effective assessment plans, how to use AT6, and what to consider in assessing findings and closing the loop . The goal of these workshops is to help programs improve student learning. After the 2006 assessment cycle was completed, the assessment team held a special workshop on “The Importance of Evidence and Progression in Learning Assessment” based on the team’s analysis of what could be done to improve the quality of the assessments  . A similar analysis was done after the 2007 cycle, and the team has decided to focus during the 2008 cycle on improving the format of and data collected for four sections in the process: executive summary, success criteria, closing the loop, and resources needed .
Beginning with fiscal year 2007-08, AT6 will also be used as the official means for deans, department heads, and program heads to document their budget requests -the trial implementation during spring 2007 indicated the process needs to be modified with more specific language so users can identify their needs more simply. Using a newly-developed budgeting form, each budgetary unit develops a budget presentation during the spring semester to address accomplishments of the previous year and priorities of the coming year with related funding requirements . For more information on the university’s budget process, see the response to Principle 2.11.1 .
Academic Certificate Program Review: At UT Dallas, academic certificate programs are defined as certificate programs that use organized, for-credit courses as the basis for the certificate; these programs are reviewed using the same model described above, including meetings to review the data collected in the assessment process and to initiate new actions to close loops. A report from the graduate Certificate in Project Management is included to illustrate some of the changes that the assessment process has helped accomplish .
Of course, not all certificate programs are offered each semester-some may be offered only as demand arises, but nonetheless, all must have assessment plans. To ensure the quality of the various certificate programs, the Academic Senate has established a rigorous approval process. Before the certificate program can be implemented, it must be approved by the faculty in the school, the Graduate Council (or Council on Undergraduate Education as appropriate), the Council on Educational Policy, and the Academic Senate . As part of the approval policy, the faculty proposing the certificate must include a detailed assessment plan with appropriate objectives and measures to assess the performance of the program . The Graduate Certificate in Holocaust Studies underwent this process in 2006 .
Distance Education Program Assessment: UT Dallas offers four online degree programs (Global Leadership Executive MBA, Global MBA, M.S. in Management and Administrative Sciences, and M.A.T. in Science Education) and three online certificate programs (Project Management Certificate, Certificate in Executive and Professional Coaching, and Chess Education). UT Dallas’ distance education policy  ensures that all its online programs have SLO’s and assess them accordingly to measure program quality and drive improvement. Both the distance learning programs and distance learning certificate programs use AT6 in the assessment process. In addition to the syllabus for the online course, each instructor provides the assessment team with comparable sample syllabi from a traditionally delivered course . The two syllabi are compared against each other to ensure that the course content is significantly similar and that the quality of the online course is assured. Beginning with summer 2007, different syllabi templates exist for face-to-face and online courses  , each template tailored to the appropriate means of delivery. All syllabi undergo approval by the assessment team, using a checklist to see that all requirements are met. By having a specialized checklist for distance education courses, UT Dallas ensures the rigor and breadth of both its face-to-face and online courses. The checklist for online course syllabi includes areas such as technical requirements, technical support, and contact information .
Academic Program Review: Series 40307 of The University of Texas System (UT System) Board of Regents’ Rules and Regulations establishes the standards for the approval of new academic programs . As discussed in the response to Principle 2.7.2 , the approval process requires approbation not only by UT Dallas committees and UT System but also by the Texas Higher Education Coordinating Board (THECB). Two specific examples of the approval of new programs (one graduate and one undergraduate) and the content required in the proposals are available in the response to Principle 4.2 .
UT Dallas’ Policy Memorandum 94-III.24-63, Academic Program Review, governs the periodic review of academic programs and charges the review team to provide an “assessment of the goals, plans, staffing, resources, existing and potential strengths, etc., of the unit, and those areas needing improvement .” The Office of the Executive Vice President and Provost maintains the schedule of reviews  and works with the Program Review Committee (PRC) and the unit under review to facilitate the review. The process is peer review oriented and includes a review team which incorporates both internal and external members. Typically, the review team is composed of at least three individuals from other institutions that have programs similar to those of the unit under review and at least two members from the UT Dallas faculty and a member of the program review committee who is not affiliated with the program to be reviewed. One member of the review team, usually not affiliated with UT Dallas, is designated team chair by the provost at the time the team is constituted. The review team evaluates the unit as requested by a written charge prepared by the provost after consultation with the PRC. The charge specifically asks the review team to “[e]valuate the quality, the effectiveness, and the efficiency of the undergraduate and graduate curricula and the delivery of instruction ,” and to evaluate the appropriateness of it assessment plans and student learning outcomes .
In accordance with the guidelines and instructions issued by the provost, the unit undergoing review prepares a comprehensive self-study document (an internal planning document not intended for general distribution) which is sent to the review team prior to its on-campus visit. The PRC also collaborates with the provost in ensuring that the review team’s needs are met during the on-campus audit. Before leaving the campus, the team holds exit interviews with the unit’s faculty and administration, the provost, the president, and other appropriate senior administrators. The review team summarizes its immediate impressions and provides a forecast of its eventual written report. After the chair provides the provost with the finalized report, the unit under review provides its own written response to the review team’s recommendations and conclusions. The provost then prepares final recommendations to the president to complete the review. In the years between reviews of the unit, the results of the program review are used when making decisions on budget, staffing, curricular and degree changes, and allocation of special resources. Additional information regarding program reviews is available in the responses to Principles 2.5  and 2.7.2 . Sample program reviews from the School of Natural Sciences and Mathematics-science/math education  and biology -are included.
Assessments through Professional Accreditation: Several academic programs at UT Dallas are accredited by professional accreditation agencies with a diverse set of accreditation standards. Some of these standards are described briefly below.
The Accreditation Board for Engineering and Technology (ABET) : ABET is the professional accrediting organization that accredits undergraduate engineering programs. UT Dallas has four engineering and computer science undergraduate programs that are ABET accredited: electrical engineering, telecommunications engineering, software engineering, and computer science.
The Association to Advance Collegiate Schools of Business International (AACSB) : The UT Dallas School of Management (SOM) has been granted full accreditation since 2002 of all its programs by the board of directors of AACSB . AACSB is a nonprofit organization consisting of more than eight hundred educational organizations and corporations devoted to the promotion and improvement of higher education in business administration and management. The faculty in SOM has developed a coordinated assessment system that involves multiple written reports that are reviewed at multiple levels . Exit interviews of graduates  and assessment of internship students  serve as examples of measures in such assessments. Both the 1997-2002 SOM strategic plan  and the 2001 self study  demonstrate ongoing assessment management in SOM.
Project Management Institute (PMI) Global Accreditation Center : The project management certificate program in UT Dallas’ SOM has become one of only five such programs in the world to receive PMI accreditation (announced September 10, 2005) . The PMI 2005 self study , PMI site visit team report , and the required 2006 PMI annual report  detail PMI accreditation activities.
American Speech-Language-Hearing Association (ASHA) : Two programs in the School of Behavioral and Brain Sciences (BBS), the doctorate in audiology (Au.D.) and the M.S. in communication disorders, are accredited by ASHA. ASHA is the professional, scientific, and credentialing association for more than 123,000 members and affiliates who are audiologists, speech-language pathologists, and speech, language, and hearing scientists. The complete description of ASHA certification requirements are supplied in the supporting documents  as is the August 8, 2007 letter regarding the latest ASHA accreditation .
The American Chemical Society (ACS) : The ACS Committee on Professional Training evaluates and reevaluates undergraduate chemistry programs with the objective of improving the standards and quality of chemistry education in America. In 2001, the UT Dallas undergraduate (B.S.) chemistry program was authorized by ACS to certify its graduates. Every five years, programs must reapply, and the chemistry program has recently submitted its five-year report as part of its authorization renewal process .
The National Association of Schools of Public Affairs and Administration (NASPAA) : The Master of Public Affairs in the School of Economic, Political and Policy Sciences (EPPS) received accreditation from the National Association of Schools of Public Affairs and Administration  in 2004 . The accreditation is valid for 6 years, after which EPPS will apply for reaccreditation .Individual Course Level
Student Evaluations of Courses: In 1999, the university adopted a new course evaluation tool that allows students to provide feedback to instructors regarding course effectiveness and allows students to benefit from the reviews of other students who have completed the course . UT Dallas posts the course evaluations on the university’s website. Users may search by the professor’s name or the course number .
Two to three weeks before the end of the semester, a 15-20 minute period at the beginning or end of each class is reserved for student evaluations, by prior arrangement with the instructor. The instructor is not present during the evaluation. At the beginning of the evaluation period, students are asked to complete course evaluation forms that detail the efficacy of the instructor and course materials. The Committee on Effective Teaching (CET) in each of the seven schools may add questions to the evaluation forms which are tailored for the type of instruction involved, e.g., large lecture classes, seminars, or labs. The dean of each school appoints this committee to ensure that the university’s policy of effective teaching is met as stated in Policy Memorandum 96-III.21-70, Teaching Evaluation Policy . The individual instructors, department or program heads, and deans receive these results. These evaluations (sample provided ) are discussed in the annual review with the appropriate dean, department head, or program head in the spring semester each year. More information about faculty annual reviews can be read below and in the responses to Principle 3.7.2  and 3.2.9 .
Evaluating Student Learning through Faculty Annual Reports: Every spring, UT Dallas’ faculty members are required to complete an annual report . A new section that addresses learning assessment activities has been added to all 2006 faculty annual reports. Based on the learning objectives in organized courses, faculty are now asked to provide a narrative that summarizes the assessment activities undertaken during the reported year, a brief description of what factors may have influenced the results of those activities, and what actions faculty intend to take to improve student learning. These documents are discussed with the appropriate dean, department head, or program head in a face-to-face review.
Syllabi Checklist: In the fall of 2005, the Academic Senate began discussion of improving syllabi, and in the spring 2006 semester, syllabi templates  were developed that require, among other things, clearly stated student learning objectives for all courses . All UT Dallas’ syllabi follow a common set of guidelines and syllabi templates . To facilitate the process for submitting syllabi, UT Dallas created a new webpage that simplifies submission and allows students to easily search for and view these syllabi online .
After a successful submission, each syllabus is reviewed by the assessment team to ensure that it follows the guidelines, e.g., that it contains a course description, grading scheme, student grievance policy, and SLO’s. Once the syllabus is approved, it is then posted and made available via the online syllabus repository.
Assessment of the General Education Component: Core curriculum courses are courses that constitute UT Dallas’ 42-hour general education curriculum required of all baccalaureate degree graduates. The Core Curriculum Committee (CCC) , composed of seven voting members appointed from the membership of the General Faculty distributed across all seven schools, reviews and decides which courses meet the core curriculum requirements in eight areas: Communication, Mathematics, Natural Science, Humanities, Fine Arts/Performing Arts, U.S. and Texas History, U.S. and Texas Government, and Social and Behavioral Sciences  .
The CCC reviews all courses in the core curriculum and determines the degree to which the core curriculum at UT Dallas meets the goals of general education. In 2005, in consultation with the CCC, the Office of Undergraduate Education reviewed the core curriculum from 1999-2004 and issued a report to the THECB . That report stated that “[f]or 2006-2007, our primary goal is integrating the various evaluation components into a larger system and field testing the assessment instruments and procedures….” Hence, in spring 2006, UT Dallas developed an additional component within the online AT6 system into which instructors currently enter core course assessment plans and reports designed to improve courses. Each spring and each fall, the CCC reviews the core course reports and communicates to the Academic Senate, deans, provost, and the president the overall analysis of the effectiveness of each of the eight components of the core. The core curriculum assessment reports from fall 2006 and spring 2007 are available for review in AT6 , and the CCC’s 2006 annual assessment report is included in the supporting documents . More information regarding the core curriculum process, its findings, and closing the loop procedures are included in the responses to Principles 2.7.3  and 3.5.1 .Other Assessment Processes
Reviews of Deans, Program Heads, and Department Heads: As part of the periodic reviews of deans, program heads, and department heads, the faculty is asked to respond to the following question: “Comment on the academic administrator’s efforts in support of faculty curricular objectives. Does he/she provide leadership and encouragement for developing and implementing effective student learning outcomes ?” Once all of the responses have been collected, the contents of the reviews are discussed and shared by the administrator’s supervisor in order to ensure effective monitoring of implementation and evaluation of student learning outcomes. A full discussion of the evaluation of administrators can be found in the response to Principle 3.2.10