Choose a scenario
Reference agreement on this example.
FlashStep
Decision Intelligence
Playground
Choose a gameConnecting…
Model info
Loading model…
Model input
PLAY
Decision details
Source & input
Leaderboard
Loading results…
GAMEJEVARENA
Ladder
Loading schedule…
Games & baseline roster
Games
Baselines
Baseline results
Loading baseline results…
Ready
Overall ranking
| # | Participant | Elo | Coverage / W–D–L | Agreement | Probability stability | Variance ↓ |
|---|---|---|---|---|---|---|
| Loading ranking… | ||||||
Ranking & stability methods
- Matches use the same game, seed and rules. Competitive games swap sides; single-player games compare final scores. Equal scores draw. A decision-budget limit preserves the last actual score.
- Per-game Elo starts at 1500 with K = 32. A win, draw or loss scores 1, 0.5 or 0; errors are unscored. Overall Elo averages all games equally and requires full coverage. The rule engine enters supported game rankings only.
- Adding a model schedules its new pairings only. Completed baseline matches and matching single-player episodes are reused. Ratings with few games are provisional.
- Each decision has three independent requests with shuffled choices. Votes are aligned by action; mean probabilities break ties when all distributions are complete. Agreement is the largest vote share.
- Probability metrics require three complete distributions. Variance uses sample variance. Jensen–Shannon distance uses base 2 and averages all three pairs; probability coverage remains available in the record details.
- Probability stability is defined here as 1 − mean Jensen–Shannon distance. Stability does not measure correctness or affect Elo. Rule-engine decisions have no model stability metrics.
Jensen–Shannon distance: SciPy documentation ↗
Expected score Eₐ = 1 / (1 + 10^((Rᵦ − Rₐ) / 400)); R′ₐ = Rₐ + 32 × (Sₐ − Eₐ).
Match records
| # | Game / Seed | Format | Participants / Final scores | Result | Replay |
|---|
Blog
Notes on decisions, environments, and learning.
ENGINEERINGOne engine. From play to policy.
How a shared environment connects the playground, evaluation, and RL.
One engine. From play to policy.
How a shared environment connects the playground, evaluation, and RL.
A common contract
Games are registered through jev_arena.engine. reset creates a seeded state; observe produces the policy record; step applies an action; outcome returns the result. public supplies the rendering snapshot.
Observe → choose → execute
The policy receives the public observation and chooses from its recorded candidates. Native state may contain hidden information, so it is not a substitute for observe. The trajectory retains the actual input, selected action, execution feedback, and outcome.
Time keeps moving
For real-time environments, capture freezes the observation sent to the model. commit advances elapsed physics and validates the arriving action. Expired or rejected actions keep their execution status. Rollout uses the registered runner and replay verifier.
Implementation notes · FlashStep
EVALUATIONReading the leaderboards
Scores, coverage, and the difference between missing and zero.
Reading the leaderboards
Scores, coverage, and the difference between missing and zero.
Keep metrics in context
JevBench, DecisionIndex, HarnessJevBench, and GameJevArena retain their own metrics. DecisionIndex scores are not raw accuracy; Arena Elo is not a win rate. Hover over table headings for the scoring definition.
Visible evidence
Radar charts leave missing dimensions disconnected. Harness Soft distribution agreement is shown separately from the Hard total. Tables retain coverage and errors; source records keep the scoring protocol. The displayed update time follows each benchmark JSON file.
Implementation notes · FlashStep