Results

How often is IsItLegit right?

On our latest test of 160 labelled emails and documents, IsItLegit caught 11.1% of attacks and wrongly flagged 0.0% of genuine mail. We publish the numbers whether they are good or not.

Every genuine email in this test is synthetic, written for the test, and the attacks are a mix of public samples and written ones. So these figures cannot say exactly how your own mail will fare: real use is counted separately below.

Eval run f91e2ec2-c171-494c-b1d5-fab69a8510c1 · 2 Nov 2026, 00:19 SAST · commit f917944 · sets synthetic-sa · set version 958883b4d88e · dataset described in eval/DATASETS.md

Attacks caught

11.1%

Target missed

Target 95% or more

False alarms on genuine mail

0.0%

Target met

Target 2% or less

Median time to analyse

7.5 s

Not measured

Forward-to-reply (target under 60 s): not measured here, only by the live mail run.

Median model cost per check

US$0.0000

Target met

Target under US$0.05

By kind of attack

Attacks caught, by kind of attack
Category	Samples	Caught
Business email compromise	15	0.0%
Credential phishing	20	0.0%
Impersonation	15	0.0%
Invoice bank change	15	0.0%
Malware attachments	10	100.0%
Voice note lures	15	0.0%
Genuine mail, flagged in error	70	0.0%

Speed and cost

Analysis time and model cost against their targets
Measure	Result	Target
Time to analyse, median	7.5 s	Forward-to-reply under 60 s: not measured
Time to analyse, 95th percentile	10.0 s	Forward-to-reply under 3 min: not measured
Model cost, median check	US$0.0000	under US$0.05
Model cost, with a walkthrough	Not measured	under US$0.25

What does each Meta model add?

We run the same test again with one model switched off at a time. The difference is what that model is worth. A model whose removal costs nothing gets cut or reworked.

No model has been switched off in a published run yet, so there is nothing to compare.

Each model switched off against the full pipeline
Switched off	Replaced by	Attacks caught	False alarms	Median time	Median cost
Nothing (full pipeline)	None	11.1%	0.0%	7.5 s	US$0.0000

What happened in real use?

Counts from members' real mail since launch (27 October 2026). Real mail is only ever counted, never added to the test set or shown. The eval and demo organisations are left out.

0 cases carried an invoice finding. The median time from arrival to a verdict was 7.0 s.

Cases checked

Malicious or suspicious

Bank changes caught

Not counted yet

Verdicts reported wrong