Where scripts get stuck is usually that they cannot read
Anyone who has automated something with a script has probably hit this: the button is clearly there, and the script still will not press it.
That time, the search went on for a long while. The problem was the judgement. What the button looks like, when it should be pressed, and which path to take after the tap all have to be written into the code in advance.
A concrete case. An order list in some back end, and you want the orders whose status shows an anomaly.
A fixed script enumerates the wording of every abnormal status, then writes a branch for each one. Writing that is heavy work. As soon as a new abnormal status appears, you go back and change the code.
A person scans the list and picks out the rows that look wrong. They can read the wording, so they do not start by listing every variant.
Below is which jobs are the kind you understand at a glance, and how to hand that kind of job to a machine.
To tell whether a job is look-then-decide, check three traits.
It has to read content. What happens next depends on what the screen says. Numbers, status, and text all count, wherever the button sits.
It branches. After the content is read, the job goes down different paths.
It identifies. You are looking for the row that says abnormal. A fixed point such as row 320, column 65 does not solve it.
A job with all three is awkward for a fixed script. A job with two of them is already worth a different approach.
The three fits, and what to do when a reading is wrong
Conditional triggers act only when a condition is met.
Order when the price drops below a value. Screenshot and keep the evidence when stock shows 0. Handle messages together once the count passes ten. Most of the time this kind of job does nothing. It acts only when the condition is met.
A fixed script has a hard time with this. Either it watches constantly, which wastes the run, or it comes back on a timer and may miss the moment. Look first, then decide: skip when it does not match, run when it does.
Content filtering picks the items that match out of a pile.
Pull the comments that contain a keyword out of dozens of comments. Pull a particular status out of an order list. Pull the messages that have not been replied to. The list is different every time. You cannot write a branch for each row. You look through it, then you pick.
Status checks act only when something is abnormal.
An account status showing abnormal, a device showing offline, a task showing failed. This is close to a conditional trigger. You have to understand what the abnormality looks like first. A conditional trigger compares a number.
Back to the job of picking out abnormal orders. Done as a description, it is roughly this:
Open the order list and read the status of each row from top to bottom. If any row’s status is neither “completed” nor “awaiting shipment”, save a screenshot of that row and note the order number. When you have read all of them, put the screenshots and order numbers into one table.
There is no “tap here” in that paragraph. You said what to look at, what to pick, and how to store it.
It can run because it reads content. If the back end later uses a new status wording, you do not go back and edit the description. It sees wording it does not recognise and still picks the row out.
A fixed script only knows the wordings you told it about in advance. One more wording and it may miss the row. The one it misses is often the one that actually needed handling.
A wrong reading on its own is not fatal. Continuing to act after a wrong reading is what causes the damage. Reading 1299 as 129, then placing an order at that price, is the case.
So add a brake: when the value read is far from what you expected, stop.
Write the boundary into the description:
If the price you see differs from the last recorded value by more than half, do not act. Save a screenshot first and wait for me to confirm.
The cost of that extra sentence is low. After a wrong reading, it becomes a prompt waiting for you to look, and it does not turn straight into a loss.
What should not be handed over
Judgement with no clear standard should stay with you.
Whether this comment counts as negative, whether this customer is worth following up first, whether this photo came out well. Ask ten people and you may get seven answers. What the machine gives you is the answer that looks reasonable. When it is wrong, you may not be able to tell.
If you can state the rule in one sentence, you can hand it over. If you cannot state it, leave it for now.
The software is free and installs on your own computer. Start with the shortest flow: look once, then do one thing.
Open a page, see whether the price has changed, and screenshot if it has. Three steps.
The shorter the flow, the easier it is to tell whether it reads accurately. A wrong reading in a three-step flow is obvious at a glance. A wrong reading in a twenty-step flow gets blamed on a later step.
Add branches after it runs. The order is: first, skip when the condition is not met; then, stop when something is abnormal; and only at the end, add the actions that change something.
A related approach is in turning phone screenshots into a spreadsheet. For the overall limits of AI phone control, see what AI phone control is.
Frequently asked questions
- What counts as a look-then-decide job?
- The screen shows some content, you have to read it before you know where to tap next. Order when the price drops below a value, screenshot when a status looks wrong, continue when a list has new entries. The difficulty is reading, not locating.
- How is that different from an ordinary script?
- A script follows a fixed sequence: tap here, wait three seconds, tap there. It does not read, so every branch has to be written in advance. Looking first means deciding at that moment based on what is on screen.
- What makes it possible?
- Reading the current screen, either structured page text or text inside an image, and then judging. That is also why it survives interface changes: the basis for the decision is content, not position.
- Which jobs fit best?
- Three: conditional triggers where something only happens past a threshold, content filtering where you pick qualifying items out of a list, and status checks where you act only on anomalies. All three are common and all three are awkward to script.
- Which jobs do not fit?
- Anything needing subjective judgement: whether a comment counts as negative, whether a lead is worth pursuing. There is no clear rule, and a machine will produce a plausible wrong answer.
- What if it reads something wrong?
- A misread is not the danger; acting on it is. So add a brake: when the value read is wildly different from expectation, stop rather than pressing on.
- Is it slower than a fixed script?
- Slightly, because of the extra read and judge. What that buys is not rewriting when the interface changes, and being able to handle branches. In most business work that trade is worth it.
- How do I start?
- Pick the shortest look-then-do-one-thing flow. Open a page, check whether the price changed, screenshot if it did. The shorter the flow, the easier it is to tell whether it reads accurately.
Got work like this?
Let it look first, then decide
Software is completely free and runs on your own computer. Start with one small job.