Add A/B experiment for synchronous PI (verbal vs. text chat)#1241
Open
sethbern wants to merge 9 commits into
Open
Add A/B experiment for synchronous PI (verbal vs. text chat)#1241sethbern wants to merge 9 commits into
sethbern wants to merge 9 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an opt-in A/B experiment mode for synchronous Peer Instruction (verbal discussion vs. switching to text chat after Vote 1), plus a few related UX/data fixes (exclude instructor responses, improved chat readiness messaging, prevent selecting self in verbal-group dropdown).
Changes:
- Adds instructor “Run A/B Experiment” flow (course-gated) and server-side assignment of verbal clusters to conditions (balanced split, never splitting a verbal cluster).
- Excludes the instructor’s dashboard-selected answer from response counts/percent-correct/chart data.
- Improves student UI/behavior: “Text Chat Ready” message, prevents selecting/logging self as a partner, and re-initializes select2 when the verbal group panel becomes visible.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| components/rsptx/templates/staticAssets/js/peer.js | Updates instructor step flow for A/B button, improves student chat banner, select2 behavior, and peer-group dropdown options. |
| components/rsptx/templates/assignment/student/peer_question.html | Logs verbal peer-group selections only when the panel is shown; avoids recording self as partner. |
| components/rsptx/templates/assignment/instructor/peer_dashboard.html | Adds conditional “Run A/B Experiment” button in the Discuss step when enable_ab is set. |
| components/rsptx/db/crud/question.py | Adds CRUD helper to delete experiment assignments so the A/B experiment can be re-run. |
| components/rsptx/db/crud/peer.py | Adds exclude_sid support to answer-fetch/count helpers to omit instructor activity. |
| components/rsptx/db/crud/init.py | Exports the new delete_user_experiment_entries CRUD function. |
| bases/rsptx/book_server_api/routers/discuss.py | Allows enableFaceChat control messages even when the sender isn’t in partnerdb. |
| bases/rsptx/assignment_server_api/routers/peer.py | Implements A/B cluster assignment + experiment persistence; excludes instructor from chart/count/pct-correct queries. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
2
to
6
| const STEP_CONFIG = { | ||
| vote1: { | ||
| next: ['makep', 'facechat', 'makeabgroups'], | ||
| next: ['makep', 'facechat'], | ||
| status: 'Vote 1 Stopped' | ||
| }, |
Comment on lines
+753
to
+757
| # Re-running the experiment will randomize it again so that it can clear any previous assignments to keep one unambiguous group per student. | ||
| experiment_id = f"{div_id}_ab" | ||
| await delete_user_experiment_entries(experiment_id) | ||
| for sid in peeps_in_person: | ||
| await create_user_experiment_entry(sid=sid, ab=experiment_id, group=0) |
Comment on lines
314
to
318
| case "enableChat": | ||
| console.log(`got enableChat message with ${mess.answer}`); | ||
| messarea = document.getElementById("imessage"); | ||
| messarea.innerHTML = `<h3>Text Chat Ready</h3><p>Please discuss with your partner(s) in the chat panel to the right.</p>`; | ||
| let discPanel = document.getElementById("discussion_panel"); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds in an A/B experiment for synchronous Peer Instruction. After vote 1, verbal discussion groups get split into two conditions: half keep discussing verbally (condition 0) and half switch to the text chat (condition 1).
A few things about how the splitting works:
peergroupevents (the "who did you talk to?" submissions after vote 1).This is off by default and has to be turned on per course, so it won't touch any existing courses. It can be enabled by noting it in the db using the
enable_abcourse attribute:If possible could the course test_py4e-int_api have the A/B test feature turned on once this is merged?
Once that's set, a "Run A/B Experiment" button shows up in the Discuss step on the instructor dashboard. Courses without it see the normal PI flow.
Also fixed a couple things from testing/feedback while I was in here: