And programmers, and hackers, and security people sometimes need ways to automate user actions – that is, write scripts that will poke into the buttons of other people’s software so that they take it in clear coin. Who said “secretly automate the work with a mobile bank”? 🙂 Shame on you, comrade! We are for peaceful automation. Today we will teach Accessibility Service to do the job on the device for us!

Push these buttons for me

Back in the XIX century Hegel said: “Machine-like work should be given to machines.” And then with one of the creators of German classical philosophy is hard to argue: it’s unlikely that Georg Wilhelm Friedrich would refuse automation of actions such as mute sound and clicking the “Skip advertisement” button when watching a video on YouTube or receiving daily bonuses for visiting an application. Moreover, for all this we already have a ready-made toolkit!

Accessibility Service can receive events that occur on the screen, but it can also call them. For example, find the desired items in the application and click on them.

Putting someone else’s application with this kind of functionality is dangerous – we all know the reputation of Google Play and pretty much imagine what this application can do with your bank client on the phone. Therefore, there are two ways out: either decompile someone else’s software and look where it presses or to file its own, strictly under the tasks.

For research purposes, I created an application that will itself click from its own service :).

Preparation for service work

In order for our service to start working, it must be granted rights in a special section of the settings. To do this, it is desirable to know how to redirect the user there. In addition, we can double-check whether these are the rights of the application.

protected boolean checkAccess() {
  String string = getString(R.string.accessibilityservice_id);
  for (AccessibilityServiceInfo id : ((AccessibilityManager) getSystemService(Context.ACCESSIBILITY_SERVICE)).getEnabledAccessibilityServiceList(AccessibilityEvent.TYPES_ALL_MASK)) {
    if (string.equals(id.getId())) {
      return true;
    }
  }
  return false;
}

Here, accessibilityservice_id is a string of the form “package name/.service”, we have it ru.androidtools.selfclicker/.ClickService.

Here is the description of the service from the manifest:

<service
  android:name="ru.androidtools.selfcliker.ClickService"
  android:label="@string/accessibility_service_label"
  android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE">
  <intent-filter>
    <action android:name="android.accessibilityservice.AccessibilityService" />
  </intent-filter>
  <meta-data
    android:name="android.accessibilityservice"
    android:resource="@xml/serviceconfig" />
</service>

The label parameter is responsible for the name of the application in the settings of the special capabilities service. In the meta-data section, you specify the description of the required functions for the service. Here is the file serviceconfig:

<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
android:accessibilityEventTypes="typeWindowStateChanged|typeWindowContentChanged"
android:accessibilityFlags="flagDefault|flagIncludeNotImportantViews|flagReportViewIds"
android:canRetrieveWindowContent="true"
android:settingsActivity="ru.androidtools.selfcliker.MainActivity" />

In it, we describe the powers of the service, the types of events that it can handle, and the activities that will be launched to configure the service.

A full description of these parameters, as always, is in the documentation.

The life cycle of the service is controlled by the system. We can not stop the service ourselves. The OS itself will unload unnecessary services – for example, why twist the service for an application that is not running?

We can bind the service to a strictly needed application. Once we have given permission to work, the service calls the onServiceConnected method. We call it the setServiceInfo() method with the AccessibilityServiceInfo parameter. For filtering the applications with which the service operates, the string array packageNames answers.

@Override
protected void onServiceConnected() {
  super.onServiceConnected();
  Log.v(TAG, "onServiceConnected");
  AccessibilityServiceInfo info = new AccessibilityServiceInfo();
  info.flags = AccessibilityServiceInfo.DEFAULT |
    AccessibilityServiceInfo.FLAG_INCLUDE_NOT_IMPORTANT_VIEWS |
    AccessibilityServiceInfo.FLAG_REPORT_VIEW_IDS;

  info.eventTypes = AccessibilityEvent.TYPES_ALL_MASK;
  info.feedbackType = AccessibilityServiceInfo.FEEDBACK_GENERIC;
  info.packageNames = new String[]{"ru.androidtools.selfcliсker"};
  setServiceInfo(info);
}

Working with AccessibilityEvent Events

After distributing all the permissions, we need to launch the required application. If we know its package name, it’s not difficult:

private void startApp() {
  Intent launchIntent = getPackageManager().getLaunchIntentForPackage("ru.androidtools.selfclicker");
  // Run from the right place without the history of the application
  launchIntent.setFlags(Intent.FLAG_ACTIVITY_CLEAR_TOP);
  startActivity(launchIntent);
}

To start the application from scratch, use the Intent.FLAG_ACTIVITY_CLEAR_TOP flag. Otherwise, the application can return to the screen with an old state, very far from the start screen.

Now you need to handle the events in the onAccessibilityEvent method. The event has a type, it will help to determine what happened (for example, the window was changed, the element was clicked on, the element received focus). To get the source of the AccessibilityNodeInfo event, you must call the getSource() method on the event object.

The source has many useful properties that help in work: text, ID, class name. It can have parent and child elements.

It can be clickable isClickable(), and to click on it as a normal user, you need to call the performAction method(AccessibilityNodeInfo.ACTION_CLICK).

If we want more global actions, for example, to press the “Back” button on the device, then we call the performGlobalAction () method with the required parameter.

To find the required AccessibilityNodeInfo on the screen, we can call one of the methods: search by ID (findAccessibilityNodeInfosByViewId) and search by text (findAccessibilityNodeInfosByText). Be prepared for the fact that he will return us an array of elements or none at all.

We will practice on cats, more precisely – on the windows

Here is the layout of our experimental screen:

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
  android:id="@+id/activity_main"
  android:layout_width="match_parent"
  android:layout_height="match_parent"
  android:orientation="vertical"
  android:paddingBottom="@dimen/activity_vertical_margin"
  android:paddingLeft="@dimen/activity_horizontal_margin"
  android:paddingRight="@dimen/activity_horizontal_margin"
  android:paddingTop="@dimen/activity_vertical_margin">

  <Button
    android:id="@+id/buttonTest"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:layout_gravity="center_horizontal"
    android:onClick="testButtonClick"
    android:text="id/buttonTest" />

  <Button
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:layout_gravity="center_horizontal"
    android:text="Without ID"
    android:onClick="noIdClick" />

  <LinearLayout
    android:layout_width="match_parent"
    android:layout_height="wrap_content"
    android:background="@color/colorAccent"
    android:onClick="linearLayoutClick"
    android:orientation="vertical">

    <TextView
      android:id="@+id/textView"
      android:layout_width="wrap_content"
      android:layout_height="wrap_content"
      android:layout_gravity="center_horizontal"
      android:text="Clickable LinearLayout"
      android:textAppearance="@android:style/TextAppearance.DeviceDefault.Large" />

    <Button
      android:id="@+id/button3"
      android:layout_width="wrap_content"
      android:layout_height="wrap_content"
      android:layout_gravity="center_horizontal"
      android:text="Non-working button" />
  </LinearLayout>
</LinearLayout>

Some elements have ID and text, others only text, some are not clickable.

Sometimes, click handlers are placed on areas that exceed the size of an element with text or a picture.

Let’s study this task using the debugClick method.

private void debugClick(AccessibilityEvent event) {
  if (event.getEventType() == AccessibilityEvent.TYPE_VIEW_CLICKED) {
    AccessibilityNodeInfo nodeInfo = event.getSource();
    if (nodeInfo == null) {
      return;
    }
    nodeInfo.refresh();
    Log.d(TAG, "ClassName:" + nodeInfo.getClassName() +
      " Text:" + nodeInfo.getText() +
      " ViewIdResourceName:" + nodeInfo.getViewIdResourceName() +
      " isClickable:" + nodeInfo.isClickable());
    }
}

Here’s what happened in the log:

03-03 16:23:15.220 24461-24461/ru.androidtools.selfclicker D/ClickService: ClassName:android.widget.Button Text:ID/BUTTONTEST ViewIdResourceName:ru.androidtools.selfclicker:id/buttonTest isClickable:true
03-03 16:23:26.356 24461-24461/ru.androidtools.selfclicker D/ClickService: ClassName:android.widget.Button Text:БЕЗ ID ViewIdResourceName:null isClickable:true
03-03 16:23:36.697 24461-24461/ru.androidtools.selfclicker D/ClickService: ClassName:android.widget.LinearLayout Text:null ViewIdResourceName:null isClickable:true
03-03 16:23:44.320 24461-24461/ru.androidtools.selfclicker D/ClickService: ClassName:android.widget.Button Text:НЕРАБОЧАЯ КНОПКА ViewIdResourceName:ru.androidtools.selfclicker:id/button3 isClickable:true

To reproduce a sequence of clicks, you must first examine the elements that will be clicked. But sometimes it is also important to follow their clicks.

For the first two buttons, you can use findAccessibilityNodeInfosByText and findAccessibilityNodeInfosByViewId. If the text of the elements is repeated, you can additionally check for ClassName or the parent.

To click on our LinearLayout, you need to get its AccessibilityNodeInfo, it does not have an ID, but there are children of TextView and Button that have text.

First, we need to get one of them, and then click on its parent.

private boolean linearClick(AccessibilityNodeInfo nodeInfo) {
  List<AccessibilityNodeInfo> list = nodeInfo.findAccessibilityNodeInfosByText("Нерабочая кнопка");
  if (list.size() > 0) {
    for (AccessibilityNodeInfo node : list) {
      AccessibilityNodeInfo parent = node.getParent();
      parent.performAction(AccessibilityNodeInfo.ACTION_CLICK);
    }
  return true;
  } else
    return false;
}

There are also reverse situations when there is a parent, and we click into the children. To do this, use nodeInfo.getChildCount() and refer to the element in the loop by ID nodeInfo.getChild(id) (if not mistaken, the ID numbering comes from zero).

It is better to start the service operation from the window change event:

event.getEventType() == AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED

If the whole algorithm of actions is already ready, then you can start the service automatically through AlarmManager, for example, once a day.

private void setRepeatTask() {
  Intent alarmIntent = new Intent(this, ClickService.class);
  PendingIntent pendingIntent = PendingIntent.getService(
    this, 0, alarmIntent, PendingIntent.FLAG_UPDATE_CURRENT);
  AlarmManager manager = (AlarmManager) getSystemService(Context.ALARM_SERVICE);

  // We start at 10:00
  Calendar calendar = Calendar.getInstance();
  calendar.setTimeInMillis(System.currentTimeMillis());
  calendar.set(Calendar.HOUR_OF_DAY, 10);
  calendar.set(Calendar.MINUTE, 0);
  calendar.set(Calendar.SECOND, 0);

  manager.setInexactRepeating(AlarmManager.RTC_WAKEUP,
    calendar.getTimeInMillis(), AlarmManager.INTERVAL_DAY, // Repeat every 24 hours
    pendingIntent);
}

To cancel start it is possible so:

public void cancelRepeat() {
  Intent intent = new Intent(this, ClickService.class);
  final PendingIntent pIntent = PendingIntent.getService(this, 0,
    intent, PendingIntent.FLAG_UPDATE_CURRENT);
  AlarmManager alarm = (AlarmManager) this.getSystemService(Context.ALARM_SERVICE);
  alarm.cancel(pIntent);
}

Conclusion

The AccessibilityService class will get rid of routine operations on your Android device. Its capabilities are enough to realize almost any task, the main thing is to give permission and find a clickable element on the screen.