2. Analysis of existing solutions¶

The aim of this chapter is to examine what DevOps teaching looks like both from a scholarly and from an institutional standpoint, and what tools the teacher has available today. I do so from two perspectives. The first is the general view of DevOps as a discipline as described in the literature, including the competencies a graduate is expected to master. The second is the specific situation at the Department of Computers and Informatics, where these competencies are actually taught. At the end of the chapter I summarise what is missing in existing solutions; these gaps are the very motivation for the design described later.

2.1 DevOps as a discipline and its key competencies¶

The term DevOps emerged in 2009, and since its inception it has evolved significantly from an informal movement into a structured discipline with well-defined principles and practices. According to Leite et al. ¹, DevOps can be characterised by five key dimensions: collaboration, automation, continuous integration, continuous delivery and monitoring. Luz, Pinto and Bonifácio ², based on a real-world case study, confirm that successful DevOps adoption requires a combination of technical tools and organisational culture. Ebert et al. ³ add that DevOps is not merely a set of tools, but above all a cultural shift that removes barriers between development and operations.

From a practical perspective the following key competencies should be mastered by a student of DevOps fundamentals:

• Version control and branch-based workflows. The student must master Git, that is, create branches according to conventions and merge changes via merge requests ⁴.
• Continuous integration and continuous delivery. The student must understand the concept of a CI/CD pipeline, the automated process of building, testing and deploying software triggered on every change in the source code ^{4 19}.
• Automated testing. Every change should include unit tests verifying correctness. Christensen ⁵ argues that systematic testing should not be taught in isolation, but as an integral part of the development process.
• Code review. Each change should be reviewed by another team member, and the review should be meaningful and constructive ⁶.
• Issue management and linking changes to issues. Project work should be organised through an issue tracker, and every change in the code should be linked to a specific issue ⁷.

Bass, Weber and Zhu ⁷ stress that effective acquisition of DevOps competencies requires hands-on experience on real projects, since theoretical knowledge without practical application does not change working habits. That is precisely why DevOps teaching needs to be supported by tools that allow monitoring of how these competencies are practically applied.

Empirical studies from industry provide useful context for what these competencies look like in real practice. Bacchelli and Bird ⁸, observing code review at Microsoft, found that although the motivation of review is mainly to find defects, in practice the dominant value-adding activity is knowledge sharing about the code and improving its readability - which presumes meaningful, not generic, comments. Sadowski et al. ⁹, analysing review processes at Google, showed that the average time-to-first-response from a reviewer is less than one hour, and that this very speed has a key impact on the smoothness of delivery. McIntosh et al. ¹⁰ in turn showed on the Qt, VTK and ITK projects that the rate of post-release defects can be statistically significantly predicted from the extent of reviewer involvement, supporting the requirement of two reviewers in SEF. Beller et al. ¹¹, in a large-scale study of reviews in open-source projects, found that only a minority of comments concern functionality and most relate to style, maintainability and readability - suggesting that students learn primarily team-work conventions through review. For testing the findings of Erdogmus, Morisio and Torchiano ¹² are relevant: in a controlled experiment they showed that the test-first approach leads to higher productivity and test quality compared with after-the-fact testing, reinforcing the requirement that every merge request should contain unit tests.

2.2 Current state of DevOps teaching at universities¶

The systematic mapping by López-Fernández et al. ¹³ identified 35 primary studies on DevOps teaching in academia published between 2015 and 2021. The following findings emerge from their analysis.

First, most courses incorporating DevOps focus only on selected aspects, most often continuous integration and automated testing, while a comprehensive workflow including issue management, code review, branch-naming conventions and systematic delivery is taught only by a small number of courses.

Moreover, in most courses adherence to DevOps practices is assessed manually by the teacher at the end of the semester, which makes ongoing feedback impossible. Bobrov et al. ¹⁴ confirm this conclusion and report that an average teacher spends 3 to 5 hours per week on manual inspection of GitLab repositories, which is unsustainable as the number of teams grows.

Finally, dedicated tools for automated assessment of DevOps practices are rare. While there is a multitude of tools for code-quality assessment, automated testing and plagiarism detection, no tool for comprehensive assessment of DevOps workflow adherence on the GitLab platform existed at the time of writing.

The view from industry points to the same gap. Smeds, Nybom and Porres ¹⁵, surveying engineers, identified that the most common obstacles when adopting DevOps are not knowledge of individual tools but mastery of branching practices and code review at the team level. Senapathi, Buchan and Osman ¹⁶, in a case study of one large organisation, pointed out that a successful DevOps transition took more than three years and was held back primarily by human, not technical, factors. Wiedemann, Wiesche, Gewald and Krcmar ¹⁷, from a longitudinal analysis of 30 case studies, conclude that DevOps is currently more an evolving set of practices than a coherent methodology, which complicates its university teaching. Fitzgerald and Stol ¹⁸, in their framework overview of continuous software engineering, emphasise that continuous integration, delivery and operations form one continuous flow of activities that cannot be assessed by isolated static indicators. This idea directly motivates the choice of a dynamic, event-driven mechanism for the tool designed in this thesis.

According to the Google State of DevOps²⁰ report, four key software-delivery performance metrics have been identified: deployment frequency, lead time from change approval to deployment, time-to-restore service, and change-failure rate. These metrics are systematically measurable and provide an objective basis for evaluating the level of DevOps adoption. However, in academia these metrics have not yet been systematically applied when assessing student teams.

A concrete example of this situation is the teaching at the Department of Computers and Informatics of the Technical University of Košice, where DevOps practices touch two courses.

The course Software Engineering Fundamentals²¹ (henceforth SEF) aims to teach students to work in a team, apply software-engineering practices and manage smaller software projects. Assessment consists of lab exercises (20 points) and a final exam (80 points). The exercises comprise three components: an analytical assignment (9 points), a team project (9 points) and class activity (2 points). The team project is run in lab groups on GitLab, and students are required to follow a defined workflow.

The workflow for solving a single task within a team project is as follows. The student picks an existing issue, creates a working branch and implements the solution including unit tests. Once finished, they create a merge request with a description of the change and a link to the corresponding issue. Each merge request must be reviewed by two other team members: one as assignee and one as reviewer. Both assess the quality of the code, the meaningfulness of the tests and adherence to the agreed conventions. If they identify deficiencies, the author of the change must fix them and re-submit. Once approved by both reviewers, the change is merged into the main branch and the issue is closed. Each team member must, during the semester, resolve at least one task as author and review at least two merge requests, one as assignee and one as reviewer.

The course DevOps Fundamentals follows on from SEF²² and is structured into two thematic areas: the first focuses on tools (Git, Docker, CI/CD) and the second on processes and communication (issue management, testing, code review). Assessment consists of a mid-term test in week 7 (10 points), a team project (10 points) and a final exam (80 points). The team project requires solving an assigned task including unit tests, reviewing the code of other team members and merging their solutions into the main branch. Compared to SEF, this course adds requirements for CI/CD pipeline configuration and automated deployment.

When assessing team collaboration, the teacher today manually verifies more than ten criteria for each student. These include, for example, whether the student picked up an issue, whether they created a branch and a merge request with a meaningful description, and whether they linked the merge request to the corresponding issue. It is further checked whether the implementation is of adequate quality, whether the student wrote tests, and whether the commit messages are meaningful (not generic messages such as "a"). It is also assessed whether the student carried out a constructive code review, whether they revised their own code in response to peer feedback, and whether they merged the corresponding request. This check has to be done individually for every member of every group, which - for the 9 lab groups of SEF with 125 students and the 8 project groups of DevOps Fundamentals with 382 students - confirms the unsustainability of the manual approach described above.

2.3 Existing tools for automated assessment¶

In this section I analyse existing tools and approaches that partially cover the problem of automated assessment in the context of software engineering. I have identified three categories of relevant tools: automated programming-assessment systems, repository-analytics tools, and rule-compliance tools.

2.3.1 Automated programming-assessment systems¶

Automated assessment systems such as Artemis²³ and CodeGrade²⁴ focus primarily on the automatic evaluation of student-code correctness through a test suite. These systems verify whether the submitted code produces expected outputs for defined inputs.

Krusche and Bruegge ⁶ extended the Artemis platform with a code-review model in which students mutually assess code quality according to defined criteria. This approach is beneficial for developing code-review competencies, but it has several limitations in the DevOps-teaching context:

• Assessment focuses exclusively on code and review quality and does not take into account the wider DevOps workflow (branch management, linking to issues, pipeline status).
• The system requires its own infrastructure and is not directly connected to industry-used platforms such as GitLab.
• Feedback is given after submission, not in real time during development.

2.3.2 Repository-analytics systems¶

The second category comprises tools for analysing Git repositories that provide statistics on developer activity. These include, for example, GitLab's built-in analytics²⁵, GitInspector²⁶ and GrimoireLab²⁷.

GitLab's built-in analytics provides basic metrics such as commit count per member, a pipeline overview and a list of merge requests. These data are useful for a quick overview but have significant limitations:

• They do not provide a compliance assessment against defined rules - they do not, for example, verify whether branches follow naming conventions or whether a merge request links to a specific issue.
• They do not allow custom rules tailored to a specific course.
• Statistics are available only at the level of a single project; they do not aggregate across all teams of a course.
• They do not provide cheating detection - for example identifying generic reviews, instant approvals or activity concentrated before deadlines.

GitInspector is a console application that analyses a Git repository and produces a report on individual authors' contributions. GrimoireLab is a comprehensive platform for analysing community software projects. Both tools are primarily designed for analysing open-source projects rather than academic assessment, and neither provides assessment against defined DevOps rules.

2.3.3 Rule-compliance tools¶

The third category consists of tools for automated rule-compliance checking in industrial environments. These include Danger²⁸, which automatically comments on merge requests based on defined rules (for example missing description, oversized changes, missing tests). Another example is static-analysis tools integrated directly into CI/CD pipelines, such as SonarQube²⁹.

These tools are valuable for automating quality control but are not designed for an educational environment. Specifically, they have the following shortcomings:

• They do not provide a summary view across multiple teams and students; they are designed for a single project and a single team.
• They do not have a mechanism for grading and presenting results to a teacher.
• They do not support rule configuration tailored to the academic context (for example weekly progression during the semester).
• They cannot detect cheating patterns specific to an academic environment.

2.4 Summary of the analysis and gap identification¶

Based on the analysis of existing approaches and tools, no tool today comprehensively addresses the problem of automated support for DevOps teaching in a university environment with the following properties:

1. Integration with an industrial platform. The solution should be directly integrated with the GitLab platform that students use for their repositories, eliminating the need for an additional tool and enabling real-environment monitoring.
2. Automated compliance assessment. The solution should automatically verify adherence to defined DevOps rules - from branch-naming conventions, through code review, to CI/CD pipeline status.
3. Real-time feedback. Assessment results should be available immediately after the activity, not days or weeks later.
4. Multi-team management. The tool must give the teacher a summary view across all teams of a course with the ability to drill down to individual students.
5. Cheating detection. The solution should identify patterns suggesting surface-level compliance without genuine understanding of DevOps practices.
6. Configurability. Assessment rules should be adjustable to the requirements of a specific course and its progression during the semester.

The comparison table below summarises the analysed tools against the identified requirements.

✓ - fully meets, ~ - partially meets, - - does not meet.

It is clear from the table that none of the existing tools meets all the identified requirements. Artemis and CodeGrade are strong on automated programming assessment, but they are not integrated with GitLab and do not assess DevOps workflow. GitLab Analytics and Danger provide GitLab integration, but they do not support comprehensive compliance assessment across multiple teams. None of the analysed tools provides cheating detection in the DevOps-practice context.

My analysis confirms that the gap identified in existing solutions justifies the proposed solution: a set of supporting tools integrated with the GitLab platform that monitor the activity of student teams in real time and automatically assess compliance with DevOps practices. The design of this solution is the subject of the next part of the thesis.