Skip to content

Add A/B experiment for synchronous PI (verbal vs. text chat)#1241

Open
sethbern wants to merge 9 commits into
RunestoneInteractive:mainfrom
sethbern:pi-ab-testing
Open

Add A/B experiment for synchronous PI (verbal vs. text chat)#1241
sethbern wants to merge 9 commits into
RunestoneInteractive:mainfrom
sethbern:pi-ab-testing

Conversation

@sethbern

Copy link
Copy Markdown
Contributor

This adds in an A/B experiment for synchronous Peer Instruction. After vote 1, verbal discussion groups get split into two conditions: half keep discussing verbally (condition 0) and half switch to the text chat (condition 1).

A few things about how the splitting works:

  • The source of truth is the verbal peergroup events (the "who did you talk to?" submissions after vote 1).
  • Verbal groups are never split across conditions since they're sitting together physically, the whole group goes to one condition or the other.
  • Clusters get shuffled and assigned with greedy least-loaded placement so the split stays close to a 50/50 split.

This is off by default and has to be turned on per course, so it won't touch any existing courses. It can be enabled by noting it in the db using the enable_ab course attribute:

INSERT INTO course_attributes (course_id, attr, value)
VALUES (<course_id>, 'enable_ab', 'true');

If possible could the course test_py4e-int_api have the A/B test feature turned on once this is merged?

Once that's set, a "Run A/B Experiment" button shows up in the Discuss step on the instructor dashboard. Courses without it see the normal PI flow.

Also fixed a couple things from testing/feedback while I was in here:

  • Instructor's own answer was getting counted as a student response
  • Text chat now shows a "Text Chat Ready" message instead of the stale "wait for your instructor" banner when it starts.
  • You can no longer select yourself in the verbal group dropdown.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an opt-in A/B experiment mode for synchronous Peer Instruction (verbal discussion vs. switching to text chat after Vote 1), plus a few related UX/data fixes (exclude instructor responses, improved chat readiness messaging, prevent selecting self in verbal-group dropdown).

Changes:

  • Adds instructor “Run A/B Experiment” flow (course-gated) and server-side assignment of verbal clusters to conditions (balanced split, never splitting a verbal cluster).
  • Excludes the instructor’s dashboard-selected answer from response counts/percent-correct/chart data.
  • Improves student UI/behavior: “Text Chat Ready” message, prevents selecting/logging self as a partner, and re-initializes select2 when the verbal group panel becomes visible.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
components/rsptx/templates/staticAssets/js/peer.js Updates instructor step flow for A/B button, improves student chat banner, select2 behavior, and peer-group dropdown options.
components/rsptx/templates/assignment/student/peer_question.html Logs verbal peer-group selections only when the panel is shown; avoids recording self as partner.
components/rsptx/templates/assignment/instructor/peer_dashboard.html Adds conditional “Run A/B Experiment” button in the Discuss step when enable_ab is set.
components/rsptx/db/crud/question.py Adds CRUD helper to delete experiment assignments so the A/B experiment can be re-run.
components/rsptx/db/crud/peer.py Adds exclude_sid support to answer-fetch/count helpers to omit instructor activity.
components/rsptx/db/crud/init.py Exports the new delete_user_experiment_entries CRUD function.
bases/rsptx/book_server_api/routers/discuss.py Allows enableFaceChat control messages even when the sender isn’t in partnerdb.
bases/rsptx/assignment_server_api/routers/peer.py Implements A/B cluster assignment + experiment persistence; excludes instructor from chart/count/pct-correct queries.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 2 to 6
const STEP_CONFIG = {
vote1: {
next: ['makep', 'facechat', 'makeabgroups'],
next: ['makep', 'facechat'],
status: 'Vote 1 Stopped'
},
Comment on lines +753 to +757
# Re-running the experiment will randomize it again so that it can clear any previous assignments to keep one unambiguous group per student.
experiment_id = f"{div_id}_ab"
await delete_user_experiment_entries(experiment_id)
for sid in peeps_in_person:
await create_user_experiment_entry(sid=sid, ab=experiment_id, group=0)
Comment on lines 314 to 318
case "enableChat":
console.log(`got enableChat message with ${mess.answer}`);
messarea = document.getElementById("imessage");
messarea.innerHTML = `<h3>Text Chat Ready</h3><p>Please discuss with your partner(s) in the chat panel to the right.</p>`;
let discPanel = document.getElementById("discussion_panel");
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants