Android 辅助功能识别屏幕文字并点击 android辅助功能模拟点击

转载

mob64ca13fa6a3c 2024-06-21 05:40:53

文章标签 安卓 android HTML 辅助功能 文章分类 Android 移动开发

一、构建无障碍服务

新建MyAccessibilityService.java类继承AccessibilityService

import android.accessibilityservice.AccessibilityService;
import android.view.accessibility.AccessibilityEvent;

public class MyAccessibilityService extends AccessibilityService {
    @Override
    public void onAccessibilityEvent(AccessibilityEvent event) {

    }

    @Override
    public void onInterrupt() {

    }
}

在/res/xml/目录下新建服务配置文件accessibility_config.xml，在里面添加相关的配置信息

<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
    android:description="@string/accessibility_description"
    android:packageNames="com.example.android.six"
    android:accessibilityEventTypes="typeAllMask"
    android:accessibilityFlags="flagDefault"
    android:accessibilityFeedbackType="feedbackSpoken"
    android:notificationTimeout="100"
    android:canRetrieveWindowContent="true"
/>

description 要显示在设备辅助功能设置中的本应用辅助功能相关的说明文字
packageNames 要监控的应用包名，多个应用用“,”隔开
accessibilityEventTypes 此服务希望接收的事件类型
accessibilityFlags 辅助功能附加的标志，多个使用“|”分隔，如flagRequestFilterKeyEvents 能够监听到系统的物理按键
accessibilityFeedbackType 反馈类型，有语音、视觉、触觉等
notificationTimeout 同一类型的两个辅助功能事件发送到服务的最短间隔（毫秒，两个辅助功能事件之间的最小周期）
canRetrieveWindowContent 辅助功能服务是否能够取回活动窗口内容的属性
settingsActivity 允许用户修改辅助功能的activity组件名称

在AndroidManifest.xml中进行声明,并引用accessibility_config.xml配置文件

<service
    android:name=".MyAccessibilityService" android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE">
    <intent-filter>
        <action android:name="android.accessibilityservice.AccessibilityService" />
    </intent-filter>
    <meta-data
        android:name="android.accessibilityservice"
        android:resource="@xml/accessibility_config" />
</service>

在辅助功能设置界面开启自身应用的辅助功能开关

Intent intent =  new Intent(Settings.ACTION_ACCESSIBILITY_SETTINGS);  
startActivity(intent);

二、辅助功能流程解析

AccessibilityService绑定流程
APP的事件分发流程
AccessibilityService执行操作流程

三、可自动执行的操作类型

控件操作
通过AccessibilityServiceManager通信到对应APP的控件中，直接操作控件内部逻辑，如performClick、setText

/**
 * 使用方式
 */
AccessibilityNodeInfo.performAction(int action)
AccessibilityNodeInfo.performAction(int action, Bundle arguments)

/**
 * Action that gives input focus to the node.
 * 获取焦点
 */
public static final int ACTION_FOCUS =  0x00000001;

/**
 * Action that clears input focus of the node.
 * 清除焦点
 */
public static final int ACTION_CLEAR_FOCUS = 0x00000002;

/**
 * Action that selects the node.
 * 选中
 */
public static final int ACTION_SELECT = 0x00000004;

/**
 * Action that deselects the node.
 * 清除选中状态
 */
public static final int ACTION_CLEAR_SELECTION = 0x00000008;

/**
 * Action that clicks on the node info.
 *
 * See {@link AccessibilityAction#ACTION_CLICK}
 * 点击
 */
public static final int ACTION_CLICK = 0x00000010;

/**
 * Action that long clicks on the node.
 * 长按
 */
public static final int ACTION_LONG_CLICK = 0x00000020;

/**
 * Action that gives accessibility focus to the node.
 * 获取无障碍功能焦点
 */
public static final int ACTION_ACCESSIBILITY_FOCUS = 0x00000040;

/**
 * Action that clears accessibility focus of the node.
 * 清除无障碍功能焦点
 */
public static final int ACTION_CLEAR_ACCESSIBILITY_FOCUS = 0x00000080;

/**
 * Action that requests to go to the next entity in this node's text
 * at a given movement granularity. For example, move to the next character,
 * word, etc.
 * <p>
 * <strong>Arguments:</strong> {@link #ACTION_ARGUMENT_MOVEMENT_GRANULARITY_INT}<,
 * {@link #ACTION_ARGUMENT_EXTEND_SELECTION_BOOLEAN}<br>
 * <strong>Example:</strong> Move to the previous character and do not extend selection.
 * <code><pre><p>
 *   Bundle arguments = new Bundle();
 *   arguments.putInt(AccessibilityNodeInfo.ACTION_ARGUMENT_MOVEMENT_GRANULARITY_INT,
 *           AccessibilityNodeInfo.MOVEMENT_GRANULARITY_CHARACTER);
 *   arguments.putBoolean(AccessibilityNodeInfo.ACTION_ARGUMENT_EXTEND_SELECTION_BOOLEAN,
 *           false);
 *   info.performAction(AccessibilityNodeInfo.ACTION_NEXT_AT_MOVEMENT_GRANULARITY, arguments);
 * </code></pre></p>
 * </p>
 *
 * @see #ACTION_ARGUMENT_MOVEMENT_GRANULARITY_INT
 * @see #ACTION_ARGUMENT_EXTEND_SELECTION_BOOLEAN
 *
 * @see #setMovementGranularities(int)
 * @see #getMovementGranularities()
 *
 * @see #MOVEMENT_GRANULARITY_CHARACTER
 * @see #MOVEMENT_GRANULARITY_WORD
 * @see #MOVEMENT_GRANULARITY_LINE
 * @see #MOVEMENT_GRANULARITY_PARAGRAPH
 * @see #MOVEMENT_GRANULARITY_PAGE
 *
 * 移动到下一个实体，如：移动到下一个字符、移动到下一行、移动到下一页等
 */
public static final int ACTION_NEXT_AT_MOVEMENT_GRANULARITY = 0x00000100;

/**
 * Action that requests to go to the previous entity in this node's text
 * at a given movement granularity. For example, move to the next character,
 * word, etc.
 * <p>
 * <strong>Arguments:</strong> {@link #ACTION_ARGUMENT_MOVEMENT_GRANULARITY_INT}<,
 * {@link #ACTION_ARGUMENT_EXTEND_SELECTION_BOOLEAN}<br>
 * <strong>Example:</strong> Move to the next character and do not extend selection.
 * <code><pre><p>
 *   Bundle arguments = new Bundle();
 *   arguments.putInt(AccessibilityNodeInfo.ACTION_ARGUMENT_MOVEMENT_GRANULARITY_INT,
 *           AccessibilityNodeInfo.MOVEMENT_GRANULARITY_CHARACTER);
 *   arguments.putBoolean(AccessibilityNodeInfo.ACTION_ARGUMENT_EXTEND_SELECTION_BOOLEAN,
 *           false);
 *   info.performAction(AccessibilityNodeInfo.ACTION_PREVIOUS_AT_MOVEMENT_GRANULARITY,
 *           arguments);
 * </code></pre></p>
 * </p>
 *
 * @see #ACTION_ARGUMENT_MOVEMENT_GRANULARITY_INT
 * @see #ACTION_ARGUMENT_EXTEND_SELECTION_BOOLEAN
 *
 * @see #setMovementGranularities(int)
 * @see #getMovementGranularities()
 *
 * @see #MOVEMENT_GRANULARITY_CHARACTER
 * @see #MOVEMENT_GRANULARITY_WORD
 * @see #MOVEMENT_GRANULARITY_LINE
 * @see #MOVEMENT_GRANULARITY_PARAGRAPH
 * @see #MOVEMENT_GRANULARITY_PAGE
 *
 * 移动到上一个实体，如：移动到上一个字符、移动到上一行、移动到上一页等
 */
public static final int ACTION_PREVIOUS_AT_MOVEMENT_GRANULARITY = 0x00000200;

/**
 * Action to move to the next HTML element of a given type. For example, move
 * to the BUTTON, INPUT, TABLE, etc.
 * <p>
 * <strong>Arguments:</strong> {@link #ACTION_ARGUMENT_HTML_ELEMENT_STRING}<br>
 * <strong>Example:</strong>
 * <code><pre><p>
 *   Bundle arguments = new Bundle();
 *   arguments.putString(AccessibilityNodeInfo.ACTION_ARGUMENT_HTML_ELEMENT_STRING, "BUTTON");
 *   info.performAction(AccessibilityNodeInfo.ACTION_NEXT_HTML_ELEMENT, arguments);
 * </code></pre></p>
 * </p>
 *
 * 移动到下一个HTML标签
 */
public static final int ACTION_NEXT_HTML_ELEMENT = 0x00000400;

/**
 * Action to move to the previous HTML element of a given type. For example, move
 * to the BUTTON, INPUT, TABLE, etc.
 * <p>
 * <strong>Arguments:</strong> {@link #ACTION_ARGUMENT_HTML_ELEMENT_STRING}<br>
 * <strong>Example:</strong>
 * <code><pre><p>
 *   Bundle arguments = new Bundle();
 *   arguments.putString(AccessibilityNodeInfo.ACTION_ARGUMENT_HTML_ELEMENT_STRING, "BUTTON");
 *   info.performAction(AccessibilityNodeInfo.ACTION_PREVIOUS_HTML_ELEMENT, arguments);
 * </code></pre></p>
 * </p>
 *
 * 移动到上一个HTML标签
 */
public static final int ACTION_PREVIOUS_HTML_ELEMENT = 0x00000800;

/**
 * Action to scroll the node content forward.
 * 向前滚动控件
 */
public static final int ACTION_SCROLL_FORWARD = 0x00001000;

/**
 * Action to scroll the node content backward.
 * 向后滚动控件
 */
public static final int ACTION_SCROLL_BACKWARD = 0x00002000;

/**
 * Action to copy the current selection to the clipboard.
 * 将当前控件内容复制到剪切板
 */
public static final int ACTION_COPY = 0x00004000;

/**
 * Action to paste the current clipboard content.
 * 从剪切板粘贴内容到当前控件
 */
public static final int ACTION_PASTE = 0x00008000;

/**
 * Action to cut the current selection and place it to the clipboard.
 * 将当前控件内容剪切到剪切板
 */
public static final int ACTION_CUT = 0x00010000;

/**
 * Action to set the selection. Performing this action with no arguments
 * clears the selection.
 * <p>
 * <strong>Arguments:</strong>
 * {@link #ACTION_ARGUMENT_SELECTION_START_INT},
 * {@link #ACTION_ARGUMENT_SELECTION_END_INT}<br>
 * <strong>Example:</strong>
 * <code><pre><p>
 *   Bundle arguments = new Bundle();
 *   arguments.putInt(AccessibilityNodeInfo.ACTION_ARGUMENT_SELECTION_START_INT, 1);
 *   arguments.putInt(AccessibilityNodeInfo.ACTION_ARGUMENT_SELECTION_END_INT, 2);
 *   info.performAction(AccessibilityNodeInfo.ACTION_SET_SELECTION, arguments);
 * </code></pre></p>
 * </p>
 *
 * @see #ACTION_ARGUMENT_SELECTION_START_INT
 * @see #ACTION_ARGUMENT_SELECTION_END_INT
 *
 * 文本选中操作，可指定选中的开始位置和结束位置，不指定表示清除当前选中
 */
public static final int ACTION_SET_SELECTION = 0x00020000;

/**
 * Action to expand an expandable node.
 * 展开
 */
public static final int ACTION_EXPAND = 0x00040000;

/**
 * Action to collapse an expandable node.
 * 折叠
 */
public static final int ACTION_COLLAPSE = 0x00080000;

/**
 * Action to dismiss a dismissable node.
 * 隐藏，如通知VIEW
 */
public static final int ACTION_DISMISS = 0x00100000;

/**
 * Action that sets the text of the node. Performing the action without argument, using <code>
 * null</code> or empty {@link CharSequence} will clear the text. This action will also put the
 * cursor at the end of text.
 * <p>
 * <strong>Arguments:</strong>
 * {@link #ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE}<br>
 * <strong>Example:</strong>
 * <code><pre><p>
 *   Bundle arguments = new Bundle();
 *   arguments.putCharSequence(AccessibilityNodeInfo.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE,
 *       "android");
 *   info.performAction(AccessibilityNodeInfo.ACTION_SET_TEXT, arguments);
 * </code></pre></p>
 *
 * 设置控件文本内容
 */
public static final int ACTION_SET_TEXT = 0x00200000;

全局操作
设备的全局操作

/**
 * 使用方式
 */
AccessibilityService.performGlobalAction(int action)

/**
 * Action to go back.
 * 返回键
 */
public static final int GLOBAL_ACTION_BACK = 1;

/**
 * Action to go home.
 * HOME键
 */
public static final int GLOBAL_ACTION_HOME = 2;

/**
 * Action to toggle showing the overview of recent apps. Will fail on platforms that don't
 * show recent apps.
 * MENU键
 */
public static final int GLOBAL_ACTION_RECENTS = 3;

/**
 * Action to open the notifications.
 * 呼出通知栏
 */
public static final int GLOBAL_ACTION_NOTIFICATIONS = 4;

/**
 * Action to open the quick settings.
 * 呼出控制栏
 */
public static final int GLOBAL_ACTION_QUICK_SETTINGS = 5;

/**
 * Action to open the power long-press dialog.
 * 呼出长按电源菜单
 */
public static final int GLOBAL_ACTION_POWER_DIALOG = 6;

/**
 * Action to toggle docking the current app's window
 * 开启分屏模式
 */
public static final int GLOBAL_ACTION_TOGGLE_SPLIT_SCREEN = 7;

/**
 * Action to lock the screen
 * 锁定屏幕
 */
public static final int GLOBAL_ACTION_LOCK_SCREEN = 8;

/**
 * Action to take a screenshot
 * 截屏
 */
public static final int GLOBAL_ACTION_TAKE_SCREENSHOT = 9;

模拟操作
通过事件注入方式，模拟人为操作，类似ADB操作设备

/**
 * 模拟手势操作点击事件
 *
 * @param service AccessibilityService
 * @param node    节点信息
 * @return 执行结果
 */
public static boolean performClickAction(AccessibilityService service, AccessibilityNodeInfo node) {
    if (node == null) {
        return false;
    }
    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.N) {
        Rect rect = new Rect();
        node.getBoundsInScreen(rect);
        Path path = new Path();
        //指定要点击的位置
        path.moveTo(rect.centerX(), rect.centerY());
        GestureDescription.Builder builder = new GestureDescription.Builder();
        //
        builder.addStroke(new GestureDescription.StrokeDescription(path, 0L, 100L));
        GestureDescription gesture = builder.build();

        final CountDownLatch cdl = new CountDownLatch(1);
        final AtomicBoolean res = new AtomicBoolean();
        boolean ret = service.dispatchGesture(gesture, new AccessibilityService.GestureResultCallback() {
            @Override
            public void onCompleted(GestureDescription gestureDescription) {
                super.onCompleted(gestureDescription);
                Log.i(TAG, "gesture performClickAction end Completed");
                res.set(true);
                cdl.countDown();
            }

            @Override
            public void onCancelled(GestureDescription gestureDescription) {
                super.onCancelled(gestureDescription);
                Log.i(TAG, "gesture performClickAction end Cancelled");
                res.set(false);
                cdl.countDown();
            }
        }, null);
        Log.i(TAG, "gesture performClickAction start :" + rect + " | " + ret);
        if (!ret) {
            return false;
        }
        try {
            cdl.await(1, TimeUnit.SECONDS);
        } catch (InterruptedException ignored) {
        }
        return res.get();
    } else {
        return false;
    }
}


/**
 * 模拟屏幕滑动
 * @param service AccessibilityService
 * @param startX 起始X坐标
 * @param startY 起始Y坐标
 * @param endX 结束X坐标
 * @param endY 结束Y坐标
 * @param slideTime 滑动持续时间
 * @return true：成功，false：失败
 */
public static boolean slideScreenAction(AccessibilityService service, int startX, int startY, int endX, int endY, long slideTime) {
    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.N) {
        Path path = new Path();
        //指定滑动起始位置
        path.moveTo(startX, startY);
        //指定滑动结束位置
        path.lineTo(endX, endY);
        GestureDescription.Builder builder = new GestureDescription.Builder();
        GestureDescription description = builder.addStroke(
                new GestureDescription.StrokeDescription(path, 20L, slideTime)).build();
        final CountDownLatch cdl = new CountDownLatch(1);
        final AtomicBoolean res = new AtomicBoolean();
        AccessibilityService.GestureResultCallback callback = new AccessibilityService.GestureResultCallback() {
            @Override
            public void onCompleted(GestureDescription gestureDescription) {
                super.onCompleted(gestureDescription);
                Log.i(TAG, "gesture slide completed");
                res.set(true);
                cdl.countDown();
            }

            @Override
            public void onCancelled(GestureDescription gestureDescription) {
                super.onCancelled(gestureDescription);
                Log.i(TAG, "gesture slide cancelled");
                res.set(false);
                cdl.countDown();
            }
        };
        boolean ret = service.dispatchGesture(description, callback, null);
        ZLog.i(TAG, "gesture slideScreenAction start: " + ret);
        if (!ret) {
            return false;
        }
        try {
            cdl.await(3, TimeUnit.SECONDS);
        } catch (InterruptedException ignored) {
        }
        return res.get();
    }
    return false;
}

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。