Guardrails eval run, 27 Oct 2026
— of — validators ran against — test cases and got — of them right. The best was — (F1 —), the weakest — (F1 —). Missed harmful cases outnumber wrongly flagged safe ones, and the keyword and classifier filters do worst on Hindi in Devanagari script.
Scorecard
A case should FAIL when the validator ought to block, flag or rewrite it. Precision: of the cases it flagged, the share it should have. Recall: of the cases it should flag, the share it caught. Click a row to open that validator below.
| Validator | Data | Cases | Outcome mix · mistakes | Precision | Recall | F1 | Median latency |
|---|
Where they fail
Over-blocking or letting through?
A low precision dot means safe text gets flagged. A low recall dot means harmful text gets through.
Accuracy by language
Share of cases right, with the number of cases beneath. hi is Hindi in Devanagari; hi-Latn is romanized Hindi or Hinglish. Click a row to open that validator.
One validator in detail
Mistakes by category
Most mistakes first; bars share one scale. Benign look-alike categories can only be wrongly flagged, harmful ones can only be missed.
| Category | Cases | Mistakes | Wrongly flagged | Missed |
|---|
What it should do vs what it did
Cases by expected behaviour (rows) and actual behaviour (columns). Green cells are correct, amber is safe text that was changed or blocked, red is harmful text let through.
Responses
One row per test case: the input, what the validator should have returned and what it did return. Italic text describes an outcome with no text of its own, such as a block. Rewrite check compares the validator's rewrite with the expected one where the dataset has one.
| Case | Outcome | Input | Should | Expected output | Did | Validator output | Rewrite check | Category | Lang |
|---|
Run log
Every row recorded in the workbook's Run_Log. gender_assumption_bias appears twice: the first run, and the re-run after its fix. Everything above uses the re-run.
| Validator | Run | Model or list | Rows | Precision | Recall | F1 | p95 latency | Notes |
|---|
Caveats
- Only pii_remover, gender_assumption_bias, uli_slur_match and ban_list have a known expected rewrite. For the other validators, expected output describes the behaviour: pass unchanged, block, or flag.
- pii_remover's expected rewrites use the dataset's placeholders ([FIRSTNAME]) while the validator writes its own (<PERSON>), so its rewrite check does not apply. Its precision counts whole messages; inside a flagged message it also masks ordinary words such as "assessment" as a PAN number.
- uli_slur_match's expected rewrites were generated by the validator itself, so its matches say nothing about redaction quality.
- The LLM-based validators gave slightly different results on two runs the same day, so treat small differences between runs as noise.
- llamaguard_7b could not run: the Guardrails Hub it depends on was shut down and there is no self-hosted replacement.
- Synthetic sets are small (66 to 79 cases), so one case moves a score by more than a point. Their labels were written by an LLM and spot-checked.
- Validators were called directly, so the on-fail setting (fix, exception, rephrase) was not applied. Texts longer than 600 characters are cut here.