feat(sched): add CPU core affinity via ClusteredEDF scheduler#701
feat(sched): add CPU core affinity via ClusteredEDF scheduler#701ytakano wants to merge 6 commits into
Conversation
Generalize the PartitionedEDF scheduler, which pinned each task to a single core, into ClusteredEDF where each task carries a CpuSet of cores it may run on. Run queues are managed by the affinity_btree_queue crate: a single affinity-aware priority queue replaces the per-core BinaryHeap array, and get_next pops the earliest-deadline task runnable on the calling CPU via pop_for_cpu. - awkernel_lib: add const-friendly CpuSet ([u64; NUM_MAX_CPU/64]) with const constructors so it can live in the const-evaluated PRIORITY_LIST. - SchedulerType::PartitionedEDF(u64, u16) -> ClusteredEDF(u64, CpuSet); Task.partitioned_core -> cpu_set: Option<CpuSet>. spawn masks out CPU 0 and out-of-range bits, falling back to all worker cores when empty. - invoke_preemption picks the core running the lowest-priority task among the set; enqueues without preemption when any core in the set is idle. - Rename partitioned symbols to clustered (PartitionedTask -> ClusteredTask, NUM_PARTITIONED_TASKS_IN_QUEUE -> NUM_CLUSTERED_TASKS_IN_QUEUE, test crate test_partitioned_edf -> test_clustered_edf). - Fix a pre-existing lost-wakeup bug in wake_workers: break -> continue so higher-numbered CPUs with queued clustered tasks are not skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Yuuki Takano <ytakanoster@gmail.com>
Signed-off-by: Yuuki Takano <ytakanoster@gmail.com>
Signed-off-by: Yuuki Takano <ytakanoster@gmail.com>
There was a problem hiding this comment.
Pull request overview
Introduces a new ClusteredEDF real-time scheduler that generalizes the prior PartitionedEDF model by allowing tasks to run on a set of worker cores (CpuSet affinity), and updates the surrounding API, tests, and documentation accordingly.
Changes:
- Add
CpuSet(const-friendly CPU affinity bitmask) and migrate task/scheduler APIs from single-core pinning to affinity sets. - Replace per-core EDF runqueues with a single affinity-aware
AffinityBTreeQueue, and add a newclustered_edfscheduler implementation. - Rename and update the userland/kernel test feature and test crate from
test_partitioned_edftotest_clustered_edf, plus doc updates/regeneration.
Reviewed changes
Copilot reviewed 18 out of 19 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| userland/src/lib.rs | Switch test entrypoint feature from test_partitioned_edf to test_clustered_edf. |
| userland/Cargo.toml | Rename optional dependency + feature flag to test_clustered_edf. |
| kernel/Cargo.toml | Rename kernel feature passthrough to test_clustered_edf. |
| awkernel_lib/src/cpu.rs | Add CpuSet type and related helpers/constants. |
| awkernel_async_lib/src/task.rs | Replace partitioned_core with cpu_set, update counters and wake logic. |
| awkernel_async_lib/src/scheduler.rs | Register ClusteredEDF, update scheduler type API, add clustered counter RAII wrapper. |
| awkernel_async_lib/src/scheduler/partitioned_edf.rs | Remove the old PartitionedEDF scheduler implementation. |
| awkernel_async_lib/src/scheduler/clustered_edf.rs | Add new ClusteredEDF scheduler implementation using AffinityBTreeQueue. |
| awkernel_async_lib/Cargo.toml | Add affinity_btree_queue dependency. |
| applications/tests/test_clustered_edf/src/lib.rs | Update tests to exercise core pinning and multi-core affinity via CpuSet. |
| applications/tests/test_clustered_edf/Cargo.toml | Rename test crate to test_clustered_edf. |
| mdbook/src/internal/scheduler.md | Update scheduler docs from PartitionedEDF to ClusteredEDF. |
| docs/print.html | Regenerated docs including new ClusteredEDF content (and additional RISC-V sections). |
| docs/internal/scheduler.html | Regenerated scheduler page reflecting ClusteredEDF. |
| docs/internal/page_table.html | Regenerated docs content (adds RISC-V sections). |
| docs/internal/memory_allocator.html | Regenerated docs content (adds RISC-V sections). |
| docs/internal/interrupt_controller.html | Regenerated docs content (adds RISC-V sections). |
| docs/internal/arch/mapper.html | Regenerated docs content (adds RISC-V sections). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Yuuki Takano <ytakanoster@gmail.com>
Signed-off-by: Yuuki Takano <ytakanoster@gmail.com>
| } | ||
|
|
||
| /// Iterate over the CPUs contained in the set, in ascending order. | ||
| pub fn iter(&self) -> impl Iterator<Item = usize> + '_ { |
There was a problem hiding this comment.
CpuSet::iter always evaluates the inner 0..64 filter for every word, so each traversal costs a fixed 512 branch evaluations regardless of how many bits are set or how many words are zero. This runs on the hot path (ClusteredTask::new/take/Drop and invoke_preemption), so even a single-core pinned task pays 512 evaluations to yield one CPU index. Consider skipping zero words and using word.trailing_zeros() with word &= word - 1 to make the cost O(popcount).
| pub(crate) fn new(inner: T, cpu_id: usize) -> Self { | ||
| crate::task::NUM_PARTITIONED_TASKS_IN_QUEUE[cpu_id].fetch_add(1, Ordering::Relaxed); | ||
| impl<T> ClusteredTask<T> { | ||
| pub(crate) fn new(inner: T, cpu_set: CpuSet) -> Self { |
There was a problem hiding this comment.
ClusteredTask::new/take/Drop each loop over the whole cpu_set doing one atomic RMW per member CPU on a distinct cache line. A wide-affinity task (e.g. CpuSet::any(), up to 511 workers) does up to 511 atomics per wake, per dequeue, and per drop, versus O(1) in the old partitioned design. Consider whether a single global clustered-task counter plus the queue's affinity-aware pop_for_cpu could replace the per-CPU array, or at least ensure the loop only touches set bits.
|
|
||
| /// Return a new set keeping only CPUs in `1..num_cpus` | ||
| /// (CPU 0 is always excluded). | ||
| pub const fn masked_workers(mut self, num_cpus: usize) -> Self { |
There was a problem hiding this comment.
The new CpuSet bit logic (masked_workers per-word masking, CPU 0 exclusion, all_workers, cross-word iter) has no unit tests, even though awkernel_lib already unit-tests a comparable bitmap in net/port_bitmap.rs. Consider adding a #[cfg(test)] mod tests covering masked_workers at and around word boundaries (num_cpus of 64/65/128), CPU 0 always excluded, all_workers for several sizes including 1, and iter ordering across words.
| ); | ||
| scheduler_type = SchedulerType::PartitionedEDF(deadline, 1); | ||
| Some(1u16) | ||
| CpuSet::all_workers(num_cpu()) |
There was a problem hiding this comment.
The empty-set fallback here (mask to empty, warn, fall back to all worker cores) is a distinct code path that no test exercises; the test crate only spawns valid single-core and {1,2} sets. Consider adding a test that spawns a ClusteredEDF task with an invalid set (e.g. CpuSet::empty(), or a set whose only cores are >= num_cpu()) and verifies it still runs on a worker core.
| let normalized = if masked.is_empty() { | ||
| log::warn!( | ||
| "Partitioned core should be between 1 and {}. Falling back to core 1. Given core: {}", | ||
| "The CPU set must contain at least one core between 1 and {}. Falling back to all worker cores. Given set: {:?}", |
There was a problem hiding this comment.
This fallback makes an invalid or empty affinity set runnable on every worker core, a much larger blast radius than the previous PartitionedEDF fallback to a single core 1. The only trace of this policy is this runtime warning; it is not mentioned in the cpu_set() doc or the mdbook scheduler section. Consider documenting the fallback so callers know an out-of-range set is normalized to all workers rather than rejected.
| if num_clustered_tasks > 0 { | ||
| let task = PRIORITY_LIST[..get_num_clustered_schedulers()] | ||
| .iter() | ||
| .find_map(|&scheduler_type| get_scheduler(scheduler_type).get_next(execution_ensured)); |
There was a problem hiding this comment.
get_scheduler (and get_priority) take SchedulerType by value, and this find_map copies the full ~72-byte value (now that CpuSet is embedded) out of the reference on every iteration, even though only the discriminant is matched. Consider changing these to take &SchedulerType and binding a reference in the closures to avoid the per-iteration payload copy.
|
|
||
| /// Create a set from a bitmask of CPUs 0..64. | ||
| /// Bit `i` of `bits` corresponds to CPU `i`. | ||
| pub const fn from_bits(bits: u64) -> Self { |
There was a problem hiding this comment.
from_bits only writes words[0], so it cannot express any CPU >= 64 even though CPU_SET_WORDS supports up to 512. It is also currently unused. Consider accepting [u64; CPU_SET_WORDS], or documenting explicitly that it is limited to the first 64 CPUs (and removing it / any() if they are not part of the intended public API).
| .iter() | ||
| .find_map(|&scheduler_type| get_scheduler(scheduler_type).get_next(execution_ensured)); | ||
|
|
||
| // Decrement is handled by ClusteredTask::Drop inside get_next(). |
There was a problem hiding this comment.
This comment says the decrement is handled by ClusteredTask::Drop, but the decrement actually happens in the explicit take() call inside get_next (Drop is a no-op once take() has run). Consider rewording to attribute the decrement to take().
Description
Generalizes the
PartitionedEDFscheduler — which pinned each task to a single core — into aClusteredEDFscheduler where each task carries aCpuSetdescribing the set of cores it may run on. Pinning to a single core is now the special case of a one-bit
CpuSet.Key changes:
awkernel_lib: adds a const-friendlyCpuSettype ([u64; NUM_MAX_CPU / 64], i.e. 512 CPUs) withconst fnconstructors (empty,from_bits,insert,contains,is_empty,masked_workers,all_workers) so it can be embedded in the const-evaluatedPRIORITY_LIST.[Mutex<EDFData>; NUM_MAX_CPU]BinaryHeaparray with a single affinity-aware priority queue from theaffinity_btree_queuecrate.get_nextpops the earliest-deadline task runnable on the calling CPU viapop_for_cpu. Priority is(absolute_deadline, wake_time), matching the previous EDF ordering, with FIFO tie-breaking on fully-equal keys.SchedulerType::PartitionedEDF(u64, u16)→ClusteredEDF(u64, CpuSet);Task.partitioned_core: Option<u16>→Task.cpu_set: Option<CpuSet>.spawnmasks out CPU 0 (primary core) and out-of-range bits, and falls back to all worker cores (1..num_cpu()) with a warning when the resulting set is empty.invoke_preemptionnow selects the core running the lowest-priority task among the set, and skips preemption entirely (enqueue only) when any core in the set is idle.PartitionedTask→ClusteredTask(now holding aCpuSetand updating the per-CPU counter for every core in the set),NUM_PARTITIONED_TASKS_IN_QUEUE→NUM_CLUSTERED_TASKS_IN_QUEUE,get_num_partitioned_schedulers→get_num_clustered_schedulers, test cratetest_partitioned_edf→test_clustered_edf.wake_workers—break→continueso higher-numbered CPUs with queued clustered tasks are no longer skipped when the global task count reaches zero.Related links
How was this PR tested?
cargo check_std,cargo check_x86(no_std target),cargo clippy_std— all pass with no warnings.cargo test_awkernel_async_lib— 42 passed.make x86_64 RELEASE=1— builds successfully.-smp 4) with thetest_clustered_edffeature:CpuSettasks ran only on their assigned core (cores 1–3) — all[OK].CpuSet {1, 2}only ever ran on cores 1 or 2 — all[OK].light(earlier deadline) correctly preemptedheavyon the same core.Notes for reviewers
SchedulerTypegrows from 16 bytes to ~72 bytes (stillCopy); it is only passed by value on thespawnpath.get_next_task(false)path (RR preemption tick, primary CPU only) can never dequeue a clustered task because
spawnalways removes CPU 0 from every set — same invariant as before. This is noted in a comment inclustered_edf::get_next.Mutexis only ever acquired whileGLOBAL_WAKE_GET_MUTEXis held (both inwake_taskand via the
get_next_taskwrapper), so no new lock cycle is introduced.invoke_preemptionstill runs before the queue lock is taken.
num_cpu()on first use (AffinityBTreeQueue::newis notconst), following the existing pattern in
prioritized_rr.rs.set_num_cpuruns before the firstspawn, sonum_cp u()is final by then.wake_workersbreak→continuefix addresses a latent lost-wakeup that becomes more reachable withclusters; it is included here rather than split out because the counter semantics changed in the same area.
mdbook/src/internal/scheduler.mdis updated to describe ClusteredEDF and the affinity-aware B-tree queue.