Sikuli Java Tutorial And Example

Aside from Selenium, we can also use Sikuli in automation. While Selenium mostly automates the browser, Sikuli can work anywhere as long as there is a screen. Therefore, it can work both in browser or in your desktop. With this Sikuli Java Tutorial, we will show you how you can get started with Sikuli. We will create a simple application that opens the MS Word application and save it in our drive.

Sikuli works by using images as your pattern. You need to get a screenshot of a button, textfield or anything that you want to send actions like clicking, typing, dragging, etc.

Sikuli Java Example

This example will show how you can use Sikuli in Java to automate process. We are using Windows 10 as our OS and MS Office 365. It starts by opening the Start menu in Windows. Then, it will open the MS Word application, type something in the document and Save.

To start with, create a New Maven Project in IntelliJ. You can view this tutorial on how to create one.

In your pom.xml, add the dependency for Sikuli:

        <dependency>
            <groupId>com.sikulix</groupId>
            <artifactId>sikulixapi</artifactId>
            <version>1.1.2</version>
        </dependency>

Then, create a class on where we will put our logic. In this case, we created SikuliAutomation.java. First, we need to define the Screen class that will be used.

private Screen screen;
//inside constructor, we create the instance.
screen = new Screen();

We also define the base folder that contains our images that will be used in order to find the elements. We have created our images folder inside resources folder as shown in the project structure below:

And here are our images that Sikuli needs to find in order for us to send actions.

After defining the Screen class earlier, saved the images that we need to used, we can now write our code. We need Sikuli to click the Start menu in Windows and Open MS Word application. We use “windows-start.png” image to locate the Start menu and use Key.ENTER to hit ENTER key in our keyboard:

private void clickWindowsStartAndOpenMSWord() throws FindFailed {
        screen.click(basePath + "windows-start.png");
        screen.wait(1.0); //need delay to allow animation to bring start menu
        screen.type("word");
        screen.wait(1.0); //wait for 1 second to show results
        screen.type(Key.ENTER);
    }

Then we will create a new blank document page in Word, type something and save. We will be using “blank-document.png” to locate the New Blank Document Icon.

private void typeTextInWordAndSave() throws FindFailed {
        screen.click(basePath + "blank-document.png");
        screen.type("This is a test in Sikuli Automation by JavaPointers.");
        screen.type("s", KeyModifier.CTRL);
        screen.click(basePath + "browse-save.png");
        screen.type("sikuli-test-document");
        screen.click(basePath + "btn-save.png");
    }

In order for us to save, we then hit CTRL+S in our keyboard as a shortcut to save, then use the “browse-save.png” and “btn-save.png” to save the document that we have created.

Below is the complete code of our SikuliAutomation.java.

package com.javapointers.sikuli;

import org.sikuli.script.FindFailed;
import org.sikuli.script.Key;
import org.sikuli.script.KeyModifier;
import org.sikuli.script.Screen;

import java.net.URISyntaxException;
import java.net.URL;

public class SikuliAutomation {

    private Screen screen;
    private String basePath;

    public SikuliAutomation() throws URISyntaxException {
        screen = new Screen();
        URL resourceFolderURL = this.getClass().getClassLoader().getResource("images");
        basePath = resourceFolderURL.toURI().getPath() + "/";
    }

    private void startTest() throws FindFailed {
        clickWindowsStartAndOpenMSWord();
        typeTextInWordAndSave();
    }

    private void clickWindowsStartAndOpenMSWord() throws FindFailed {
        screen.click(basePath + "windows-start.png");
        screen.wait(1.0); //need delay to allow animation to bring start menu
        screen.type("word");
        screen.wait(1.0); //wait for 1 second to show results
        screen.type(Key.ENTER);
    }

    private void typeTextInWordAndSave() throws FindFailed {
        screen.click(basePath + "blank-document.png");
        screen.type("This is a test in Sikuli Automation by JavaPointers.");
        screen.type("s", KeyModifier.CTRL);
        screen.click(basePath + "browse-save.png");
        screen.type("sikuli-test-document");
        screen.click(basePath + "btn-save.png");
    }

    public static void main(String... args) throws FindFailed, URISyntaxException {
        SikuliAutomation sikuliAutomation = new SikuliAutomation();
        sikuliAutomation.startTest();
    }
}

Just do right click and Run in the class to run our Sikuli Java example. You can download the source code using this link. Let us know in the comments if you have questions or it helps you to get started Sikuli in Java.