I. INTRODUCTION
Activities of quality assurance and assessment and accreditation of training program (TP) are increasingly interested by the state and society. With the trend of international integration in terms of training quality, universities aim to meet international quality standards through registration, assessment and accreditation of education quality. Evaluation of training programs (TPs) usually begins with the university writing a self-assessment report according to the selected set of quality assessment criteria, improving the quality of the TP, registering for an assessment with an educational quality accreditation organization, welcoming the external assessment team to work directly at the school to survey the curriculum, interview stakeholders, visit the facilities, study records to prove the authenticity of the information written in the self-report evaluates and draw conclusions about the quality of the TP compared to the requirements of the set standards. However, from 2020, when the Covid-19 epidemic spread and had a profound impact on many areas of life, the assessment and accreditation activities of schools and TPs have been evaluated and controlled by organizations. International quality standards such as the Organization for Evaluation of Educational Programs in Applied Science, Computer Science, Engineering and Technology (ABET), Association for the Advancement of Collegiate Business Teaching (AACSB) ), Accreditation Council for Graduate Medical Education (ACGME), AUN-QA, etc. moved from face-to-face to online-distance. This article aims to introduce the form of online- distance, e- learning assessment, compare the differences between the two assessment methods, strengths, weaknesses and propose solutions to improve the effectiveness of assessment activities. After practical implementation experience at Thai Nguyen University (TNU) with online-distance assessment of the quality of TPs of Quality Assurance Organization - Southeast Asian Universities Network (AUN-QA).
II. OVERVIEW OF THAI NGUYEN UNIVERSITY AND ACTIVITIES OF ONLINE- DISTANCE ASSESSMENT OF THE TRAINING PROGRAM OF AUN-QA
1. Introduction of Thai Nguyen University
Thai Nguyen University (TNU) is a regional university established in 1994. Currently, TNU has 11 training institutions including 07 member universities, 01 subordinate university, 01 subordinate faculty, branches in Lao Cai province and College of Economics and Technology. The university currently has 328 training programs at all levels. The quality assurance system in the university is built according to the three levels model: the Testing and Quality Assurance Department, member and subordinate schools with Testing and Quality Assurance Department/Quality Management Department, subordinate departments with testing and quality assurance department, at member schools with staff who are concurrently responsible for this work.
The TNU has become an associate member of AUN-QA since 2015, the first evaluation of the AUN-QA at the TNU in the form of online training for 04 training programs in the pedagogy and medicine sector took place in March 2021 after a period of delay due to the epidemic. In order to carry out the evaluation of the TP, TNU has implemented the following activities:
Develop a strategy for the evaluation and accreditation of TPs and concretize to member universities and subordinate faculties through long-term and short-term plans. The plan for self-assessment of TP is specific to each individual and unit with the implementation time and products to be achieved.
Promulgating internal guidelines on the evaluation of TPs of TNU based on the guiding documents of AUN-QA. The Board of Education Examination and Quality Assurance advise the Steering Committee for the evaluation of TPs - TNUT periodically reviews the implementation to guide and promptly solve difficulties and problems.
In addition, TNU has organized the learning experience of institutions such as Center for Testing and Evaluation of Education Quality, Ho Chi Minh City National University; experience in implementing self-assessment and writing reports of Ho Chi Minh City University of Pedagogy and Technology, Can Tho University, Foreign Trade University, etc.
At TNU, the self-assessment report of the TP is read and appraised through two levels. The Self-Assessment Board and the Education Quality Assurance and Testing Department at the school level; TN level, Examination Board and Education Quality Assurance - TU reads and advises on editing self-assessment reports, supporting documents in Vietnamese and English to clarify the strengths and weaknesses of the curriculum. At the same time, recommending measures to improve the quality of the existing points of the TP.
TNU develops online document storage software to serve the assessment of school quality and TPs. The software has effectively supported assessors and staff of the Self-Assessment Council, especially with the online- distance assessment form.
Overcoming difficulties in approaching the new assessment method, 04 TPs were assessed by AUN-QA experts as meeting requirements and exceeding expectations. In the second half of 2021, TNUT has 03 more TPs in the field of agriculture assessed by AUN-QA, gradually contributing to affirming the training quality of TU in the region.
2. AUN-QA's process for distance/ online evaluation of TP
Evaluating the quality of TPs is one of AUN-QA's activities with the aim of (1) making recommendations on areas the university needs to improve on, and (2) improving the effectiveness of its internal quality assurance system. Higher education institutions to achieve international standards [3], [4], [5]. Evaluation activities are organized according to the following process: The university writes a self-assessment report of the TP – Registers for assessment with the AUN-QA Secretary (online since 2018) – AUN-QA announces the evaluation schedule – The school sends the self-assessment report – The evaluator researches reports and documents – Surveys at the university – Submits the final external assessment report – The school gives feedback on the assessment activities – Grants certificate of meeting quality standards (if the TP meets the requirements).
According to the latest guidance of AUN-QA [5], the distance assessment process of AUN-QA TP and the on-site assessment process are fundamentally different in the field survey activity, if in the past this activity If deployed directly on site, in distance assessment, this survey will be conducted online and distancely on Zoom software.
Activities for preparation and implementation are divided into pre-assessment, during and post-audit stages, specifically as follows:
2.1. Pre-evaluation meeting
There are 04 meetings prior to the online assessment date, implemented on Zoom software.
2.1.1. Orientation meeting with the university and the program
- The orientation meeting was held after the university sent the self-assessment report of the curriculum, the core evidence, the checklist of the self-assessment report and related documents to the AUN-QA Secretary.
2.1.2. Meeting to check the readiness of the university
- The readiness check usually takes place after the e-learning self-assessment report has been verified and approved by the AUN-QA assessor.
2.1.3. Orientation meeting with Zoom management team
- Two weeks before the official assessment date, the AUN-QA Secretary held a meeting to guide on the process and technique of organizing Zoom meetings.
2.1.4. The last review meeting
- Two weeks before the evaluation, AUN-QA Secretary and university representatives meet to review the evaluation program and arrange interviews during the evaluation; resolve other issues related to the evaluation activities.
2.1.5. Họp định hướng của trường đại học với người được phỏng vấn
2.1.5. Meeting of the university with an interviewee
- One week before the assessment or according to the appropriate schedule, the School/Faculty holds an orientation meeting with the interviewees to prepare for the assessment.
2.2. Formal evaluation activities
During 05 days, implementation activities include opening ceremony, working with dean, vice dean, TP manager, department head and self-assessment report writing team; interviewing stakeholders of the TP including students, lecturers, training support staff, alumni and employers; visit the facilities at the school level and the Faculty/TP in the form of live broadcast; In-depth study of evidence records, clarifying self-assessment reports. Initial assessment results will be announced during the closing session.
2.3. Post-assessment activities
After the assessment from 1.5 to 2 months, the assessment secretary will send the final external assessment report of the assessment team together with the feedback form on the assessment process to the university.
The AUN-QA Secretariat completes the External Assessment Report (print version) and issues a certificate to the TP after receiving all documents.
Certification period: 2 to 3 months after the audit.
III. DIRECT AND DISTANCE EVALUATION OF TP: DIFFERENCES, ADVANTAGES, DIFFICULTIES AND CHALLENGES OF EACH FORM
3.1. Differences between the two forms
From AUN-QA's guidelines for on-site and distance assessment of TPs, implementation experience of TNU and units [1], [2], the difference between the two forms About the assessment time, implementation process, preparation of reports, documents, ... are summarized in Table 3.1 by the author.
3.2. Advantages and difficulties, challenges
From the above differences, the advantages and challenges of the two forms of assessment can be seen as follows:
IV. LESSONS LEARNED FROM THE PRACTICE OF DIRECT ASSESSMENT - DISTANCE LEARNING AT THAI NGUYEN UNIVERSITY
TNU has successfully evaluated 04 TPs in the form of online- distance. The evaluator and secretary of AUN-QA's TP highly appreciated the preparation and coordination activities of TNU. On the basis of the differences between the two assessment methods, the advantages and disadvantages of each form, TNU draws some experiences to effectively deploy the online- distance assessment of the e-learning program. AUN-QA is as follows:
4.1. Direction and planning
Develop and implement a strategic plan on assessment and accreditation of TP quality in order to create resources and connect units and departments in the process of implementing evaluation activities.
4.2. Regarding professional training and retraining
Organize training for staff and lecturers to participate in self-assessment activities on report writing skills, collection of evidence records, and use of evidence record software
4.3. Consulting in writing and editing self-assessment reports and collecting supporting information
The staff involved in writing the self-assessment report and collecting evidence should be trained and consulted according to each standard and criterion
4.4. Using online document storage software
This software has shown its superiority when organizing the distance assessment of e-learning programs. All self-assessment reports and proof records are stored on dedicated software for easy access by professionals.
4.5. About developing the script, filming the introduction of the TP's facilities and the School
According to the regulations of AUN-QA, in order to evaluate the facilities for the TP, in addition to the live visit of the school facilities, it is necessary to send the assessor a video introducing the facilities in advance. serving Faculty-level curriculum and an introductory video about the university's facilities.
4.6. Personnel
For Zoom managers, select staff who are fluent in English so that they can talk directly with the AUN-QA Secretary and the auditor; proficiently use Zoom software to serve live streaming of facilities and host interview sessions.
Interpreters: Each TP needs to prepare 02 interpreters who are independent interpreters or can be lecturers in other faculties/schools. AUN-QA discourages the use of interpreters in interviews with learners and former learners
Officials participating in live streaming of facilities: video recording, narrator, person in charge of facilities.
Interviewees: It is necessary to select the right subjects and in sufficient quantity according to the requirements of AUN-QA. The interviewees using English fluently will help the assessment activities to be highly effective.
4.7. Live streaming of facility visits
4.7.1. Selection of equipment
Devices with cameras, anti-vibration devices, headphones, and bluetooth connected microphones or microphones (optional) can use for live streaming.
Units can use phones with 3G/4G sims or use dedicated devices with a fixed internet connection to stream facilities.
The university can choose the option that suits the actual conditions of the university to achieve the highest efficiency.
4.7.2. Development of a scenario for live broadcast of the visit to the facilities
Make sure that the names and order of the facilities that are broadcast live follow the list of facilities submitted to the audit team.
Develop specific plans and scenarios for each point, including time, route, and introductory content and specific tasks of each participating officer.
The narrator and videographer at each location use 02 separate accounts with their own headphones and microphones to avoid unstable internet connection and ensure picture and sound quality.
AUN-QA reviewers can question facility users during a live stream, so it's important to ensure the quality of the sound and microphone operation.
4.7.3. Narrator
The school needs to arrange a narrator for each point of facilities. The narrator should be able to explain, provide information about the facilities, and be able to answer questions from the AUN-QA auditor during the live stream.
4.7.4. Commissioning of the live stream scenario of the visit to the facilities
Before the university's Readiness test and the official assessment, the university should organize a test run of the live-streaming script to visit the facilities
V. CONCLUSIONS
The transformation of e-learning evaluation form from face to face to online during the Covid-19 pandemic has shown its superiority when at the same time limiting the movement of parties involved in the evaluation process, avoiding disease spread, while helping universities and TPs continue to affirm their commitment to ensuring training quality, continuously improving quality, and demonstrating accountability to society, contributing to promoting the quality of training. Strengthen regional and international integration in terms of training quality.
During the assessment process, universities and training institutions need to perform the following activities well:
Proactive and ready at the highest level to provide the required information and evidence to clarify the quality of the TP.
Do not speculate: because the interview and question-and-answer activities may be affected by the quality of the internet connection, the interpreter or the interviewees, the staff supporting the assessment activities do not know the questions and requirements. The expert's request should be re-questioned, avoiding speculation because it may lead to giving incomplete or inaccurate answers.
Work closely with the AUN-QA Secretary. The AUN-QA secretary is the bridge between the expert and the school during the assessment process; they will assist in answering the school's questions and concerns before, during and after the assessment process. Make the most of their support.
There is a difference in the transfer of assessment funding in the form of in-person and distance assessment. For assessor and development fees, payments cannot be made directly. AUN-QA does not have regulations on contract form and liquidation, so the school needs to actively draft and submit to sign to transfer money online to AUN-QA.
In addition to the advantages, we are also well aware of the limitations of the online- distance assessment of e-learning programs to take measures to overcome and limit the disadvantages to ensure the quality of e-learning evaluation activities. Hopefully, this article will make a small contribution to answer questions for units in the selection and implementation of online- distance assessment of e-learning programs according to AUN-QA.
According to Pham Van Hung; Nguyen Thi Thu Huong (2021), Monograph "Assurance and accreditation of higher education quality in the context of university autonomy: Theory and practice", Hanoi National University Publishing House.
