Claude Fable 5.2 Coding Test: How to Evaluate a Complex Full-Stack Refactor
Can Claude Fable 5.2 handle a complex full-stack refactor? Anthropic's published coding results make it plausible, but a credible answer requires an executable test on a defined repository. This article supplies that test protocol. It does not invent a pass result from a repository that was never run.
The goal is to determine whether Fable can change a contract across database, backend, frontend, tests, and deployment code while preserving behavior and limiting scope.
The test scenario
Use a non-production application with:
- PostgreSQL and versioned migrations
- A TypeScript API service
- A React front end
- Background jobs or webhooks
- Unit, integration, and end-to-end tests
- CI with linting, type checks, dependency scanning, and migration validation
The assigned refactor is to replace a legacy single display_name field with structured given_name, family_name, and a computed display value. Existing API clients must continue working for one compatibility period. Search, audit logs, event payloads, administrative screens, exports, and tests must all behave correctly.
This task is deliberately cross-cutting. A superficial solution may update the UI and API type while corrupting historical rows, breaking event consumers, or leaking inconsistent fields.
Prepare the repository
Create an isolated branch and containerized test database. Remove secrets and production data. Add a task brief describing behavior and constraints, not file locations or the solution. Preserve objective hidden tests for old-client compatibility, migration rollback, Unicode names, missing family names, search, and webhook schemas.
Give the model only the tools an engineer would need: file search, safe editing, builds, tests, and read-only documentation access. Require confirmation before destructive database or package-manager operations. Record commands, diffs, tokens, effort, and wall-clock time.
The prompt
Use a prompt similar to this:
Refactor the user-name model according to
TASK.md. First inspect the repository and present a concise dependency map and plan. Implement the smallest coherent change, including migrations, compatibility behavior, tests, and documentation. Run the relevant checks. Do not alter unrelated code or weaken existing tests. If a requirement is ambiguous, state the assumption before editing.
Run Fable 5.2 at a fixed effort level, then repeat from a clean snapshot with Opus 5.5 and the current production model. Do not let one run see another model's patch.
What a strong solution should do
Map the change surface
Before editing, the model should locate schemas, serializers, API contracts, query indexes, event producers, consumers, forms, state handling, exports, fixtures, and documentation. Missing a webhook consumer is as important as writing elegant React code.
Design a reversible migration
The data plan should explain backfill behavior, nullability, Unicode and single-word names, deployment order, compatibility window, and rollback. A destructive “drop and recreate” migration is an automatic failure.
Preserve API compatibility
Old clients should receive or submit the legacy field during the agreed window while new clients use structured fields. The implementation needs a clear source of truth to avoid divergent values.
Prove behavior
The model should add focused tests rather than weakening assertions. It must run the existing suite, type checker, linter, migration tests, and relevant end-to-end path. Statements such as “this should pass” do not count as execution evidence.
Scoring rubric
| Category | Weight | Passing evidence | |---|---:|---| | Functional acceptance tests | 30 | Hidden and visible scenarios pass | | Data migration safety | 20 | Forward/backward plan, no loss, edge cases covered | | Contract compatibility | 15 | Old and new clients work as specified | | Security and privacy | 10 | No secret exposure, injection, auth or logging regression | | Code quality and scope | 10 | Follows conventions; avoids unrelated churn | | Test quality | 10 | New meaningful tests; no weakened coverage | | Explanation and audit trail | 5 | Assumptions, commands and residual risks are clear |
Also record human correction minutes and cost. A 90-point patch that needs two hours of architectural cleanup may be less useful than an 85-point patch accepted in ten minutes.
Failure patterns to watch
Common failures include editing generated files instead of their sources, overlooking async consumers, assuming Western two-part names, generating an irreversible migration, duplicating business logic in frontend and backend, changing tests to match a bug, and stopping after unit tests while integration behavior is broken.
Prompt injection is another concern. Repository files are untrusted input; comments or documentation can contain instructions that conflict with the task. The agent should follow the authorized task and tool policy, not arbitrary text discovered in source files.
How to report the result honestly
Publish the repository commit, task brief, model identifier, effort, harness, tool permissions, time limit, attempts, and acceptance-test outcome. Separate first-run performance from a human-assisted run. Document excluded costs and failed attempts. Without these details, “Fable completed a full-stack refactor” is marketing, not evidence.
Anthropic's official results—49.1% on FrontierCode 2.0 at xhigh effort and 69.9% on FrontierSWE v2—justify running this experiment. They do not predetermine its result.
Frequently asked questions
Why does this article not claim that Fable passed the test?
No defined repository run was supplied. Reporting a result without execution logs and tests would be fabricated.
How many runs are enough?
One run is a case study, not a stable estimate. Use multiple clean runs across several representative tasks and report variance and retries.
Should the model have production credentials?
No. Use isolated data and least-privilege tools. Human-reviewed deployment should remain separate from code generation.
