Sikuli 0.10 has added tons of cool new features. Now it is easier to write smarter scripts to do more interesting and useful tasks than you can ever imagine. Here is one of my favorites. The original idea of this script was contributed by RaiMan, one of the most active Sikuli users. I simplified it a little bit and made a video to show how it runs in a real environment.
The goal of this script is to download all ZIP files whose file name starts with “Sikuli-IDE-osx” in a web page. To find all ZIP files is easy, because they have an identifiable icon. However, we can’t simply click on the icon. We need to click on the hyperlink next to the icon, instead. The really tricky thing is, how to know a hyperlink that starts with “Sikuli-IDE-osx”?
The simplest solution is capturing a screenshot of the text “Sikuli-IDE-osx”, and putting the screenshot as the parameter of click(). Unfortunately, this way is not really robust since the images of text only have tiny differences. Sikuli’s fuzzy image matching engine may treat “S1kul1-1DE-esx” as the same as what we want. So, a right way to do this is – extracting the text from the web page.
You may recall that there was an ancient technology named OCR (Optical Character Recognition), which recognizes text from scanned books. However, current OCR engines do not really work well. They are slow and inaccurate.
In fact, even without OCR, we still can extract text from the screen using Sikuli.
Use the clipboard!
The following script tells you how to do that. At first, we use Sikuli to find all locations of the ZIP icon (line 7). On line 8, we sort the list of the matched locations by their y coordinate, it’s not necessary though. The key steps are line 10 and 11. We click on the space between the icon and the hyperlink next to it using the new target offset attribute of a pattern. (denoted by a small red cross in the ZIP icon on line 7. See the figure below for its setting window.)
And then we hold the mouse button and drag to the location 600-pixel right to the center of the icon. This step (line 2) actually simulates using a mouse to select a line of text with a dragDrop(). After the selection, we simply press Command+C (Ctrl-C on Windows) to copy the selected text into the system clipboard.
Aha! We got the precise text in the clipboard.
The rest of the script are trivial. Sikuli 0.10 provides a new API Env.getClipboard() to get the content of the system clipboard. We can use this function to get the text of the hyperlinks and then use Python’s string function (startswith) to filter out the files we need.
I also made a tiny video that shows how this script looks like when it is running.