AI Culling Accuracy: How to Judge Claims and Measure It

Why vendor accuracy numbers are not comparable

Every accuracy or speed figure found in the sources reviewed comes from a vendor or from a rival, with no published method found, so none can be set beside another. The ones on record:

Four separate problems sit under those rows.

1. Different definitions. "Accuracy" could mean the share of true keepers the tool kept, the share of its keeps that were true keepers, the share of rejects that were true rejects, or agreement on a per-frame label. A tool that keeps everything has perfect keeper recall.
2. Different ground truth. Accuracy is measured against some human's cull. Without saying whose picks, how many people agreed, and how close the keepers were, the number describes one photographer's taste.
3. Different galleries. One wedding with a lot of low-light reception frames can score differently from one shot in daylight. Single numbers hide the mix of scene types.
4. Different targets. Some tools reject by threshold, some rank within a group, some cull to a target count. A fixed target for one and a threshold for another is not a like-for-like run.

A rival's number about another vendor is also a competitor claim, and this site never uses it as the sole basis for a statement about that vendor. The glossary of terms is on what photo culling is.

One run, four different "accuracy" figures

The definition alone can move the number by more than 30 points. Example, using invented round numbers to show the arithmetic, not a test result: a 1,000-frame gallery where a person keeps 200, and a tool keeps 300, including 190 of the person's 200.

All four describe the same output. When you read an accuracy claim, the first question is which of these it means. The second is what the 10 false rejects were, because those are the frames you would never see again unless you reviewed the reject pile.

Which measures to record

These measures can be reported by any photographer, and together they say more than a single percentage.

Keepers found (recall)

of the frames your cull kept, how many the tool also kept or ranked as selects.

False rejects

frames you kept that the tool rejected. These matter most, because a rejected keeper is lost unless someone reviews the rejects.

False keeps

frames the tool kept that you rejected, which sets the size of your review pass.

Agreement on near-duplicates

whether the tool's pick within a burst matches yours, because picking the best of a sequence is where much of the work is.

Minutes per 1,000 frames

wall-clock time for the tool's pass, and separately your review time on its output, noted with your hardware.

Upload volume and network activity

for tools that may use a cloud step, the data sent during a culling pass.

Cost per 1,000 frames

from dated prices, using the cost per 1,000 images calculator.

Run each tool once at its recommended default and once cull-to-target where that is available, and record the results per scene type (getting ready, ceremony, formals, reception, low light) as well as overall.

What makes a comparison fair

A comparison is only as good as a few decisions, and they should be fixed before any run.

• Unit and sample size. The unit is the frame, but frames in a burst are not independent, so also count per moment, with a burst as one decision. One wedding is one day and one photographer's taste, so more galleries give a sounder answer.
• Human reference. If you can, have two people cull the same gallery without seeing each other's picks, and measure their agreement first. A chance-corrected statistic such as Cohen's kappa is a common choice. A tool cannot be expected to agree with a reference more than people agree with each other.
• Threshold sweep. For tools with a threshold, try several settings and compare false rejects against review workload, not a single point.
• Settings parity. Use default settings plus a matched cull-to-target count, and note the learning state. A tool that has learned from your past galleries is not comparable with one that has not, so switch personalisation off where possible or write down that it was on.
• Contamination. The test gallery should not be one a tool has already learned from. Follow each vendor's data terms, and turn learning off where a setting exists.
• Uncertainty. Small differences on one gallery may mean nothing. The culling scorecard shows how wide the range can be, so treat a gap inside it as no difference.
• Scene mix. Keep daylight, indoor, low-light and flash, group and candid frames apart when you can.
• Weighting. Look at false rejects before false keeps. The first is invisible without review, and the second only adds work.

How to run it on your own gallery

A gallery you have already culled gives you a ready-made reference, so you do not need a new shoot.

1. Choose your set. Take about 200 frames you have already culled, spread across the day. Check first that processing them is consistent with your client agreement and each vendor's data terms, especially for tools that upload.
2. Freeze your reference. Save your original picks before any tool sees the frames, so you cannot be swayed by its choices. A second person's cull is better still.
3. Set up each trial the same way. Use the same machine and files, with learning off where a setting exists. Record the tool and app versions, operating system, hardware, settings and plan.
4. Run it twice. Do one pass at the default and one cull-to-target where available.
5. Score it. Enter the results in the culling scorecard, and count the keepers the tool rejected first.
6. Time both halves. Note the tool's pass and your review of its output separately, since a fast pass that needs a long review saves little.
7. Repeat once. Run it again to see whether results change, and score without knowing which tool made which list if you can arrange that.

That tells you more about your fit than any published percentage. Adobe's Assisted Culling tutorial lists useful things to count: subject sharpness, eye sharpness and eyes open, with optional rejects for documents and receipts, misfires and exposure issues. For what each tool documents, see the Aftershoot review, Narrative Select review, Imagen culling review, Excire Foto review and Lightroom vs paid cullers.

To try a tool on your own gallery: or .

Original test pending: WeddingWorkflowLab has not run this method; we would apply it to a licensed RAW wedding gallery with releases and a trial of each tool, as set out in our methodology and test gallery policy, and report false rejects per moment instead of a single percentage.