How automatic screenshot categories work

Apps that auto categorize screenshots first read the text in each image, then look for clues that point to a type: a total and a currency suggest a receipt, a flight number and gate suggest travel, and words like "promo code" suggest a coupon. The screenshot gets a label, and it appears in the matching group. It is pattern spotting, not mind reading, so it works well on clear, typical screenshots and less well on unusual ones.

Step one: reading the text

Everything starts with OCR, short for optical character recognition. OCR is the process of turning the letters and numbers in a picture into text a computer can search. If you want the full background, our explainer on what OCR is covers it in plain terms.

A screenshot is a good subject for OCR. The text is sharp, straight and usually on a plain background. That is why screenshot apps can read most of them accurately, while a crumpled paper receipt photographed in dim light is harder.

The output of this step is a block of text for each screenshot, often with a rough idea of where each line sat on screen.

Step two: looking for clues

Once the text is available, the app looks for signals. Different apps use different methods, but they tend to fall into a few types.

• Keywords: words such as "order", "total", "subtotal", "boarding", "check-in", "invoice" or "discount".
• Patterns: things that look like a price, a date, an order number, a card number with most digits hidden, a flight code or a booking reference.
• Combinations: one clue on its own is weak. "Total" plus a price plus "order number" is strong evidence of a receipt.
• Layout and source: some systems also consider how the screen is laid out, such as a list of items with prices on the right.
• Learned models: some apps use a small machine learning model trained to recognise types of screens, rather than fixed rules.

What each category usually looks for

Here is how typical categories tend to be spotted.

• Receipts: prices, a total, tax lines, an order or receipt number, a store name.
• Travel and bookings: flight or train numbers, departure and arrival times, airport or station names, seat numbers, "booking confirmed", hotel check-in dates.
• Payments: "paid", "sent", "transfer successful", a transaction ID, a recipient name.
• Coupons: "code", "voucher", "% off", "valid until", a short block of capital letters and numbers.
• Shopping: product names with prices, "add to basket", sizes and colours, delivery dates.

When enough clues line up, the screenshot gets the label. When clues point in two directions, such as a hotel receipt, the app may choose one or use both.

Why apps that auto categorize screenshots sometimes get it wrong

Knowing the weak spots helps you trust the system for the right things.

1. Little or no text. A photo of a product with no words gives the app nothing to read.
2. Unclear text. Small fonts, text over images or very low contrast can produce reading errors.
3. Other languages and scripts. If the reader or the rules do not cover a language well, clues are missed.
4. Mixed screens. A chat where someone pastes an order number looks a bit like a receipt.
5. Unusual layouts. A small shop's receipt may not use the words the app expects.

None of this is a reason to avoid automatic sorting. It just means you should treat the labels as a head start and correct the odd one.

Where should this processing happen?

Reading your screenshots means reading bank confirmations, chats and tickets. Some services do that work on remote servers, which means your images are uploaded. Others do it on the phone itself. On-device processing keeps the images where they already are. Our comparison of on-device OCR and cloud OCR explains the trade-offs.

How to get the most out of automatic categories

1. If your app can auto categorize screenshots, let it finish reading your existing ones before judging the results.
2. Check each category once and move anything that landed in the wrong place.
3. Add your own tags for things the app cannot know, such as who a screenshot is for.
4. Use search alongside categories, because search finds a single word even when the label is wrong.

Our guide to tagging and renaming screenshots shows how your own labels and automatic ones work together, and using screenshots to keep track of online receipts shows the receipts category in everyday use.

How Screenshot Brain sorts screenshots

Screenshot Brain reads the text in your screenshots on your phone and adds automatic labels for receipts, bookings, payments, coupon codes and travel. Receipts, Travel, Shopping and Work collections sort themselves from those labels, and you can add tags, rename items and build your own collections when the automatic choice is not quite right.

Frequently asked questions

Do apps need the internet to auto categorize screenshots?

Not always. If text reading and sorting happen on the phone, no upload is needed. Cloud-based services do need a connection, because the images are processed on their servers.

Why did a chat screenshot end up in my receipts?

It probably contained a price, an order number or a word like "total". Move it to the right place and, if your app allows, add a tag so you can find it later.

Can automatic categories read handwriting?

Clear handwriting is sometimes read, but results are much less reliable than with printed or on-screen text. Add a tag to handwritten notes you want to find.

Will categories get better over time?

That depends on the app. Some update their rules or models in new versions. Your own corrections and tags will always improve what you can find.