What to Do When You Cannot Read the English Interface

Last year a friend in cross-border trade talked with me about overseas apps. He does small appliances in Shenzhen and sells into Europe. The business is all right. Nobody on the team dared touch the back office. He had also been looking for a way to automate English apps.

The interface is all in English. The words on the buttons look familiar, and he still is not sure of them, so he does not dare tap. Once an operator changed a setting by mistake and set the shipping origin for a whole batch of products to a domestic warehouse. They noticed two days later, and support problems piled up. After that there was an unspoken agreement: leave operations on an English screen alone if you can work around them.

His fix was to hire someone with good English. He found someone, and it worked. At the end of last year that person left, and that piece broke again.

What is stuck is the cost of a wrong tap

The younger people on his team can guess their way to about seven or eight out of ten on an English screen. What is stuck is that a wrong tap has to be undone, and the path back is unfamiliar, so they do not dare tap a second time.

One path is to train people. That takes time and chances to practise. The other is that the side doing the tapping does not have to read English. It reads the screen itself. That second path is what AI phone control does, and it is where English app automation is useful.

Element targeting, and reading the screen

These two are easy to mix up when you are choosing an approach.

The old way is element targeting. The script says: find the control whose id is publish_button and tap it, or find the element whose text equals Publish and tap it. It is precise, on the condition that what is on the screen is identical to when the script was written.

The other way looks at the current screen first: what text is there, which controls, roughly what the layout is, then combines that with what you said and decides where to tap.

Take the same app. On the Chinese version that button says “publish”. On the English version it says “Publish”. A script that targets the element cannot find the target on the English version, and it stalls on the first step. The approach that reads the screen sees the button at the bottom that submits content. The language is different, and it still treats that button as publish.

Reading the screen uses two things together. One recognizes the text and the controls, and reads what is on the screen. The other is a vision model. It looks at the screenshot and judges which part of the picture is what you are looking for.

You can set the second one yourself. On the capabilities and models page, choose which vision model to use, and fill in your own API key and address.

Text recognition alone cannot handle an icon with no text. The vision model covers that part. It looks at the picture itself.

Write the action in the description, not the words on the button

In the description, write what to do. Do not write “tap the button that says such and such”.

Write it like this:

Open this app, tap the plus in the middle of the bottom bar, pick the first image from the photo library, set the title to the text I give you, and pause before submitting so I can confirm.

This makes trouble for yourself:

Tap the button that says Create.

The first still runs if the interface language changes or the button moves. The second breaks as soon as that word changes.

On the first run, switch to refinement and let it ask about the details you did not make clear. Stop before the last step and have a person confirm. That habit is useful for the first few weeks: you are not familiar with this app’s interface either, and watching one run is how you learn where it tends to go wrong.

After the flow finishes, look at Execution History. The records are in Chinese. Which step failed, roughly why, and which device are all written there. Troubleshooting does not turn into reading an English error just because the interface is in English.

Three kinds cover most of it:

• The target is not found. Usually a button that is only an icon. Add an image template, screenshot that button, and save it.
• You tapped and nothing happened. The wait was set too short. The next step was tapped before the page finished loading. Loosen it by a second or two and try again.
• The device is offline. That has nothing to do with the interface language. Handle it on the device page.

What you need to be able to do is state the business action clearly. That has nothing to do with your English level.

This approach does not translate, and it does not judge whether the English content is well written. It reads what is on the screen and carries out the actions you described. Whether the cross-border copy sounds natural is still for a person to check. What to say to a customer is still for a person to decide. It handles the operation.

The friend from the start now writes the copy himself. His English is decent, and he hands the execution to the computer. He said he used to hire someone who could read the interface. Now the tool reads the interface, and the person only writes the content.

If you are unsure, follow the install selection page, connect one phone, and try the least risky flow you can think of, such as opening the app to see how many orders there are today, without changing anything. One successful run tells you how far it can go.