Dimensions
The same 16 tests for every assistant. Open one for the task and the ranking.
16 dimensions
1Carrying out an online taskTest: Book a hotel stay. Completes a real browser or web workflow end to end, not just advice.4 tested2Travel bookingTest: Book a flight and handle the trip. Finds, books and manages flights and hotels, including check-in and changes.2 tested3Recommendation qualityTest: Pick a restaurant with constraints. Relevance, taste and constraint-following when it suggests options.4 tested4Purchasing a productTest: Reorder a product on Amazon. Finds, compares and completes (or correctly stages) a purchase.3 tested5Responding to emailsTest: Reply to a scheduling email. Drafts or sends replies in your voice, with the right context and recipients.3 tested6Proactive behaviorTest: Flight day, unprompted. Acts or nudges usefully without being asked every step.3 tested7Running a routineTest: Daily digest for a week. Reliable scheduled and recurring work.3 tested8Third-party integrationsTest: Three tools, one request. Uses connected tools like calendar, inbox and Notion correctly.5 tested9Permissions & privacyTest: Scoped access and a hard rule. Lets you scope what it can see and do, honors the rules you set, and revokes cleanly.Untested10MemoryTest: Remember a preference a week later. Remembers your preferences, people and past requests across sessions.2 tested11PersonalityHas a voice worth talking to; reads like a contact, not a form. Subjective, so it is read from public quotes rather than scored.Public opinion only12Phone callsTest: Call a business and get an answer. Makes real phone calls on your behalf and reports back.Untested13Multiplayer / groupsTest: Plan dinner in a group chat. Works in group chats with several people at once.1 tested14Chained tasksTest: Flight check-in chain. Strings several steps across tools into one job, like a flight check-in.2 tested15Proactive restraintTest: Know when not to. Judgment on unprompted action: handles the small stuff, waits on the consequential.Untested16Content creation / gamesTest: Make something for the group. Makes images, video or games on request.1 tested
How scoring works
One published task per dimension, scored 1–10 against written anchors after real use. No score without a logged run.
A tested score1 – 10
Not tested yet—
Doesn't apply to this productN/A
Benchmark versionv0.2