Operating System Cyber Security
Overview
This unit is designed to introduce the protection, manipulation and analysis of the inner working of operating systems. Students will learn how the Windows and Linux kernels work and study classical computer science topics such as concurrency, scheduling, Windows and Linux memory management. A variety of operating system attacks which use buffer overflows, stack smashing, heap sprays, format strings, race conditions, return to LibC (Return-Oriented Programming), integer overflows, privilege escalation, code injection, sandbox bypass, resource exhaustion and hypervisor bypasses will be studied. Defence techniques will include cryptography, authentication, code signing, containerisation and anti-virus software.
Requisites
Rule
CYB80001 System Security Project
OR
COS60011 Technology Design Project (concurrent)
02-November-2025
Learning outcomes
Students who successfully complete this unit will be able to:
- Describe the inner workings of modern operating systems, architectures and strategies used to improve hardware performance
- Describe communications and signalling systems used in operating systems
- Describe and implement common data structures and software design patterns in single and multi-threaded applications
- Explain operating system attacks including memory corruption attacks, privacy violations and unauthorised access
- Describe and demonstrate a variety of software flaws which allow privilege escalation, unauthorised access and unauthorised code execution
- Analyse and modify operating system source code and utilities, and detect and correct software flaws
- Design and implement tools and procedures which can be used to protect operating systems from a variety of attacks
Teaching methods
Hawthorn
| Type | Hours per week | Number of weeks | Total (number of hours) |
|---|---|---|---|
| Face to Face Contact (Phasing out) Lecture | 2.00 | 12 weeks | 24 |
| Face to Face Contact (Phasing out) Tutorial Labs | 2.00 | 12 weeks | 24 |
| Unspecified Learning Activities (Phasing out) Independent Learning | 8.50 | 12 weeks | 102 |
| TOTAL | 150 |
Assessment
| Type | Task | Weighting | ULO's |
|---|---|---|---|
| Assignment | Individual | 50% | 1,2,3,4,5,6,7 |
| Examination | Individual | 50% | 1,2,3,4,5,6,7 |
Content
- The Windows kernel
- The Linux kernel
- Concurrency, scheduling, race conditions, deadlock
- Windows and Linux memory management
- Inodes, file systems, RAID)
- Signals, long jumps, page mapping
- Hardware architectures (ARM and Intel), Von Neuman, caching, memory
- Threading, stacks, queues, active objects
- Memory corruption attacks: buffer overflows, stack smashing, heap sprays, format strings
- Execution path attacks: race conditions, return to LibC (Return-Oriented Programming), integer overflows, privilege escalation
- Security control bypass attacks: sandbox bypass, hypervisor bypasses
- Defence techniques: authentication, code signing, containerisation and anti-virus software
- Cryptography
Study resources
Reading materials
A list of reading materials and/or required textbooks will be available in the Unit Outline on Canvas.