Microsoft Azure Automatic Grading Engine - Oct 2021 Update

Problem

One of the main challenges on teaching cloud technologies is how to assess student. For IT courses, educators can collect some files such as source code and give a grade to students. However, for cloud tech educators, it is hard to use the traditional approach to handle the assessment.

There are two main problems:

1. It is hard to check each student work one by one manually which is time consuming and error-pone. 
2. It is not a good idea to get a grade based on a “snapshot”, as students may use Infrastructure as a Code (IasC) tools such as ARM template or terraform and every student gets full mark, but they know nothing!

 

In fact, for vocational training, my course IT114115 Higher Diploma in Cloud and Data Centre Administration in Department of Information Technology (IT) of the Hong Kong Institute of Vocational Education (Lee Wai Lee).

Our Goal is to ensure our students are skilled to be highly effective cloud engineers. All our students know how to do it well and within the region employers are ready to pay students for the roles but they all want know "who is the best?".

 

Solution

Educators need to have access to the student work and see their code, as part of the Microsoft Learn Educator Programme I have been developing an opensource serverless application – Microsoft Azure Automatic Grading Engine.

The engine includes 3 main functions:

1. Schedule Grader
2. Class Grade Report Generator
3. Student Registration

The grader includes 2 main functions:

1. Azure Function to run NUnit Test in an executable
2. HTTPS Endpoint

Each assignment need to one grader and unique endpoint URL.

 

Architecture Diagram

 

How does Schedule Grader work?

1. A Timer Trigger runs a Durable Orchestration function “Schedule Grader” based on a expression.
2. The “Schedule Grader” is an Orchestration Function.
3. The grader scans "assignments" which are stored in an Azure Table and inspects the resources associated each item the table contains:
   1. PartitionKey: The name of a assignment, 
   2. GraderUrl: The Grader Azure function endpoint and it runs NUnit tests.
   3. TeacherEmail: Email of the teacher and mark report will send to this email address. (Optional)
   4. SendMarkEmailToStudents: true will send mark to all students after running the test.
4. Query "credentials" Azure Table by each assignment name and it returns list of student email and credentials key json. Assignment, student email and credential becomes a grading task. 
5. All grading tasks run in parallel by an independent call to the "GraderUrl" and it returns NUnit test report in XML format. It uses the Azure SDK to collect the Azure settings, the process runs a NUnit test to check the settings it expects or not and return an xml test result output file. For the fairness, it is important to grade all students at the same time. Also, using web services decouple the grader from the solution. This allows the educators to add and update the assignment grader without redeployment of the whole solution. 
6. All the completed test results are saved into a "test result" blob container with path prefix "assignment/email/yyyy/MM/dd/HH/mm/".
   
7. If the students have registered a valid email the "SendMarkEmailToStudents" is true, each student will receive mark report email with NUnit test xml attachment.
   
8. The solution Generates a class mark report in both Excel and JSON and saves it into the "test result" blob container in "assignment/" and "assignment/yyyy/MM/dd/HH/mm/".
   
   
   
9. If  the educator defines a "TeacherEmail" an email containing the Excel and JSON report is sent to the educator. (See the attached sample accumulated marks spreadsheet) 
   
   
   

 

How does Class Grade Report Generator work?

1. When the Grade Report Function HttpTrigger endpoint is called, it checks query 3 parameters “assignment”, "today", and "json".
2. Optionally, if "today" exists, the report only contains the mark of today.
3. Optionally, if "json"  exists, the report changes into json format.
4. The grade reporter sets “assignment” as prefix and gets all text result xml files.
5. By mean of xpath we extract text case full name and test result. If the test is pass, it contains 1 mark.
6. The service then generates the accumulate mark and creates a report in an Excel or JSON file format.

 

How to create a class assignment?

Educators need to complete 3 tasks:

1. Implement a NUnit test for the assignment and note down the https endpoint. You can refer our assignment example project the code within this C# file it is very straight forward.
2. Define a assignment by adding an Entity in assignments table. 
   
   
3. Simply Email the Registration Link to your Student with Mail Merge (Demo https://www.youtube.com/watch?v=CXc7fx6nNJk )
   

   To prevent typos of the assignment name and email address, teacher can use a standard mail merge to send the link to students template. Using the template will result in your creating and issuing a unique URL string for each student. The url string will be in the following format.

   https://somethingunique.azurewebsites.net/api/StudentRegistrationFunction?project=studnetid&email=studnetemailaddress

4. (Optional) Change CRON Expression C# code of "ScheduleGrader" and redeploy the project. (It will be configurable in the coming release! )
   
   The current build is running the grading task every 12 hours of UTC.

 

Students need to know

1. Subscription ID and Email must be unique for each assignment.
2. Don't run "az ad sp create-for-rbac -n "gradingengine" --sdk-auth" repeatedly!
   Students must re-submit the online form again with the new credentials.
3. To ensure you can create the services principal, each student should create their own active directory. Otherwires, you may not be possible to create services principal.
   
   
   If Students are using Azure subscription associated to their academic institution they may NOT have permissions to create service principals.
   
   Students will have to move the subscription to their own AD. The solution is that creates their own Azure AD and move the subscription to their own AD.

 

How does the Azure Grader Function work?

Each assignment has one set NUnitTest, and you can fork the Reference Assignments for the Azure Automatic Grading solution repository.

The solution is based on  Visual Studio 2019 solution with 2 projects.

Add and update the "AzureProjectGrade" project. When you build the "AzureProjectGraderFunctionApp", it will generate "AzureProjectGrader.exe". You can share this executable to your students and they can run NUnit test locally for quick grade check.

 

AzureProjectGrader.exe c:/azureauth.json c:testreport/ abc@stu.vtc.edu

 

 

The "AzureGraderFunction" actually is running that executable for each request and that is the magic to parallel run NUnit Tests inside Azure Function by process isolation.

 

Support Environment

For grading, students we need to process that the grader has read only access credentials to the students Azure subscription, each student has to create a service principal permission for the app. 

For the engine, it is just a simple Azure Function app with consumption plan, This process has been developed so that it can even run inside Azure student subscription plan with very low cost. We have carried out a number of tests using this with Azure for Student subscription.

 

Reference Project Assignment

Reference Assignments for the Azure Automatic Grading solution contains example assessment/references for the Azure Auto Grading Engine, these samples are provided to allow educators to quickly adopt and implement the Azure Automatic Grading Engine service/solution into their classroom teaching.

This example is set for a 10 hour set of classes, some of my students are able to fully complete all tasks in a week.

Tasks include:

1. Create 2 Virtual Networks in 2 regions.
2. Create 2 Subnets in each Virtual Network.
3. Create Route Tables & Network Security Groups.
4. Create Virtual Network Peering for 2 Virtual Networks.
5. Create 2 Storage Accounts - one for an Azure Function and another for a Azure Static Website.
   1. Azure Function Storage Account contains 1 Storage Container, 1 Storage Queue, and 1 StorageTable.
   2. Static website Storage Account contains index page index.html and error page error.html.
6. Create 1 Application Insights with Log Analytics Workspace.
7. Create 1 Azure Function App with 1 Azure Function.

The assignment is taking the Test-driven development (TDD) approach. Students have to read through the NUnit test codes and create the Microsoft Azure infrastructure that can pass all test requirements.

How to deploy and use it?

For the engine,

1. Login your Azure Account and click on the Deploy to Azure button.
2. Go Project GitHub Repo https://github.com/microsoft/AzureAutomaticGradingEngine
3. Click on the Deploy to Azure button.
4. Fill in a unique name, and SMTP settings.

Remark:

1. You can change or set SMTP by changing the App Settings Key during deployment or in the Azure Portal.
2. If you want to use Gmail, you need to allow Less Secure Apps for your Gmail.

For the reference assignment grader, 

1. Go Project GitHub Repo https://github.com/microsoft/AzureAutomaticGradingEngine_Assignments 
2. Click on the Deploy to Azure button and fill in the unique name.

If you need to use Visual Studio 2019 for customisation please follow the following video. How to deploy Azure Automatic Grading Engine 

 

Conclusion

You can full control Microsoft Azure programmatically and educators can now spend time on preparing a better course instead of wasting on grading Azure assignment manually.

 

We have been using the Azure Automatic Grading Engine for a semester. Students definitely benefit a lot both in technical and soft skills. Firstly, students must work honestly, seriously, and quickly. As a result, their technique is in industry standard. Also, using auto grading solutions is more fun for most students and you can treat this like a game. We have seen many students trying to get mark as fast as possible. Part of our process is that we do share their mark to employers and employers loves to refer that mark than the students GPA. 

 

Also, it is possible to use the engine for Azure Skills Competition. For upcoming features we all add a game like dashboard and allow students to gamify the journey of leaning Azure. We also plan to add Chaos component and we can have trouble shooting assignment, but every student will get different bugs within certain scope.

Project collaborators include, Chan Yiu Leung (Hades), So Ka Chun, Lo Chun Hei, Ling Po Chu, Cheung Ho Shing and Pearly Law from the IT114115 Higher Diploma in Cloud and Data Centre Administration. 

 

About the Author

Cyrus Wong is the senior lecturer of Department of Information Technology (IT) of the Hong Kong Institute of Vocational Education (Lee Wai Lee) and he focuses on teaching public Cloud technologies. He is one of the Microsoft Learn for Educators Ambassador.

Posted at https://sl.advdat.com/3G0uLye