fix(gate): resolve issue #100

[SparshGarg999]

Root Cause

SkillOpt uses score-based rollouts to accept or reject edits to the prompt skill text. Currently, the validation gate score does not account for the semantic steerability or conciseness of the prompt content itself. This makes it harder for the optimizer to favor terminology carrying strong, well-defined associations for the target model or to prune unnecessary verbosity.

Solution

We introduced a "semantic density" heuristic into the validation gate's score projection mechanism.

1. Semantic Density Heuristic: Implemented compute_semantic_density in skillopt/evaluation/gate.py, which computes the density of high-influence prompt words ("leading words") relative to the total active words in a skill document, stripping metadata comment regions.
2. Score Adjustment: Updated select_gate_score and evaluate_gate to accept configuration parameters (use_semantic_density, semantic_density_weight, and leading_words). When enabled, the heuristic bonus weight * density is added to the metric score.
3. Configuration mapping: Added mappings in skillopt/config.py and parsed config parameters in skillopt/engine/trainer.py to seamlessly pass these parameters during training run evaluations.

Files Changed

• skillopt/config.py
• skillopt/engine/trainer.py
• skillopt/evaluation/gate.py
• tests/test_sleep_engine.py
• tests/test_semantic_density.py (NEW)

Testing/Benchmark Performed

1. Created new unit tests tests/test_semantic_density.py to verify case-insensitive word count density calculation, bonus score projection, and gating decisions favoring more concise/steerable skills when accuracy is tied.
2. Ran all test suites (including the new semantic density tests and existing engine/types/json/mcp tests):

   python -m pytest tests/test_sleep_engine.py tests/test_json_utils.py tests/test_mcp_schema.py tests/test_plugin_sync.py tests/test_types.py tests/test_semantic_density.py

   (All 133 tests passed, 4 skipped)
3. Ran the sleep experiment successfully:

   python -m skillopt_sleep.experiments.run_experiment --persona researcher --assert-improves

Fixes #100