evalship

evalship reveals the flaws in your LLM evals, then keeps watch on every pull request. Setup is one line for your coding agent: "Set up https://evalship.com/setup.md". The agent reviews every scorer, judge and prompt in your evals and finds their flaws, such as a judge that scores 1.0 when its call fails, or a privacy score too small to ever fail a reply. Each flaw comes with fix instructions. The agent then wires your evals into GitHub Actions and opens a setup pull request. On every pull request, evalship comments with the evals the change really broke and the likely cause, sets flaky evals aside by checking them against main's history, and flags prompt and LLM-call changes that no eval exercises. The evalship check can block a merge that breaks an eval. Your evals keep running in your own GitHub Actions with your own keys. evalship reads the results file from pytest, Vitest, Jest, promptfoo, Langfuse, Braintrust or LangSmith (JUnit XML, promptfoo JSON or evalship JSON), so there's no SDK to add, and it works next to the eval platform you already use. A dashboard per repo shows what needs attention and every eval's history, and a weekly report each Monday sums up the regressions caught and what your evals cost. Public repositories are free.