AutoItとUI Automation UDFを使って自動化する

2024年6月24日

新しい gui を備えたアプリの場合にボタンなどが認識できない場合があります。しかしUI Automation を使うと認識できる場合があります。今回はこれを使ってみます

UI Automation UDF（User Defined Function）は、AutoItのユーザーが作成した関数の一部で、UI Automation（UIA）を使用してWindowsプログラムを自動化するためのものです. UIAは、ユーザーインターフェースと対話するための自動テストスクリプトを作成する際に役立ちます。具体的な指示やステップについては、UIAutomation UDFを使って要素を取得し、アクションを実行することができます。例えば、ボタンをクリックしたり、入力を行ったりすることができます

ダウンロードとインストール

UI Automation UDFs – AutoIt Example Scripts – AutoIt Forums (autoitscript.com)

UIAIncludes.7z

UIASpy – UI Automation Spy Tool – AutoIt Example Scripts – AutoIt Forums (autoitscript.com)

UIASpy.7z

Using UI Automation Code in AutoIt – AutoIt Example Scripts – AutoIt Forums (autoitscript.com)

UIAExamples.7z

"C:\app\autoit3\UIAIncludes\Includes"
"C:\app\autoit3\UIAIncludes\UIA_Examples"
"C:\app\autoit3\UIAIncludes\UIASpy.au3"
"C:\app\autoit3\UIAIncludes\UIASpy.ini"
"C:\app\autoit3\UIAIncludes\Includes\GuiIncludes"

UIASpy

Place mouse cursor in the control and press F1:

the UI Automation framework is able to detect your control.  Inspect.exe  is a Microsoft tool. Inspect.exe in the bin-folder of the Windows 10 SDK 

“C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\x64\inspect.exe"

日本語のメモ帳でサンプルを動かす

HelloWorldをメモ帳に入力し保存するサンプルです

“C:\app\autoit3\UIAIncludes\UIA_Examples\1) Automation\Notepad\Windows 7, 10\1) Create code\NotepadAll.au3"

サンプルはそのままでは動かないので修正します。加えて2つの命令を追加してみます。$oSave.SetFocus() $oValue.CurrentValue($myText)

#include "..\..\..\..\..\Includes\CUIAutomation2.au3"

Example()

Func Example()
  ; Open Notepad
  Run( "Notepad" )
  Sleep( 1000 )

  ; Create UI Automation object
  Local $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
  If Not IsObj( $oUIAutomation ) Then Return ConsoleWrite( "$oUIAutomation ERR" & @CRLF )
  ConsoleWrite( "$oUIAutomation OK" & @CRLF )

  ; Get Desktop element
  Local $pDesktop, $oDesktop
  $oUIAutomation.GetRootElement( $pDesktop )
  $oDesktop = ObjCreateInterface( $pDesktop, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  If Not IsObj( $oDesktop ) Then Return ConsoleWrite( "$oDesktop ERR" & @CRLF )
  ConsoleWrite( "$oDesktop OK" & @CRLF )

  ; --- Notepad window ---

  ConsoleWrite( "--- Notepad window ---" & @CRLF )

  Local $pCondition ; Note that $UIA_ClassNamePropertyId maybe ia a CASE SENSITIVE condition
  $oUIAutomation.CreatePropertyCondition( $UIA_ClassNamePropertyId, "Notepad", $pCondition )
  If Not $pCondition Then Return ConsoleWrite( "$pCondition ERR" & @CRLF )
  ConsoleWrite( "$pCondition OK" & @CRLF )

  Local $pNotepad, $oNotepad
  $oDesktop.FindFirst( $TreeScope_Descendants, $pCondition, $pNotepad )
  $oNotepad = ObjCreateInterface( $pNotepad, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  If Not IsObj( $oNotepad ) Then Return ConsoleWrite( "$oNotepad ERR" & @CRLF )
  ConsoleWrite( "$oNotepad OK" & @CRLF )

  ; --- Fill Edit element ---

  ConsoleWrite( "--- Fill Edit element ---" & @CRLF )

  ; Note that $UIA_AutomationIdPropertyId is a STRING and maybe a CASE SENSITIVE condition
  ;   ;   $oUIAutomation.CreatePropertyCondition( $UIA_AutomationIdPropertyId, "15", $pCondition )
  $oUIAutomation.CreatePropertyCondition( $UIA_ClassNamePropertyId, "RichEditD2DPT", $pCondition )
  If Not $pCondition Then Return ConsoleWrite( "$pCondition ERR" & @CRLF )
  ConsoleWrite( "$pCondition OK" & @CRLF )

  Local $pEdit, $oEdit
  $oNotepad.FindFirst( $TreeScope_Descendants, $pCondition, $pEdit )
  $oEdit = ObjCreateInterface( $pEdit, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  If Not IsObj( $oEdit ) Then Return ConsoleWrite( "$oEdit ERR" & @CRLF )
  ConsoleWrite( "$oEdit OK" & @CRLF )

  Local $pValue, $oValue
  $oEdit.GetCurrentPattern( $UIA_ValuePatternId, $pValue )
  $oValue = ObjCreateInterface( $pValue, $sIID_IUIAutomationValuePattern, $dtagIUIAutomationValuePattern )
  If Not IsObj( $oValue ) Then Return ConsoleWrite( "$oValue ERR" & @CRLF )
  ConsoleWrite( "$oValue OK" & @CRLF )
  $oValue.SetValue( "HelloWorld" )

local $myText=""
$oValue.CurrentValue($myText)
consolewrite("Text: " & $myText & @CRLF)


  ; --- Open File menu ---

  ConsoleWrite( "--- Open File menu ---" & @CRLF )

  Local $pCondition1
  $oUIAutomation.CreatePropertyCondition( $UIA_ControlTypePropertyId, $UIA_MenuItemControlTypeId, $pCondition1 )
  If Not $pCondition1 Then Return ConsoleWrite( "$pCondition1 ERR" & @CRLF )
  ConsoleWrite( "$pCondition1 OK" & @CRLF )

  Local $pCondition2 ; $UIA_NamePropertyId is LOCALIZED and maybe CASE SENSITIVE
 ; $oUIAutomation.CreatePropertyCondition( $UIA_NamePropertyId, "File", $pCondition2 ) ; File <<<<<<<<<<<<<<<<<<<<
  $oUIAutomation.CreatePropertyCondition( $UIA_NamePropertyId, "ファイル", $pCondition2 ) ; File <<<<<<<<<<<<<<<<<<<<
  If Not $pCondition2 Then Return ConsoleWrite( "$pCondition2 ERR" & @CRLF )
  ConsoleWrite( "$pCondition2 OK" & @CRLF )

  ; And condition
  $oUIAutomation.CreateAndCondition( $pCondition1, $pCondition2, $pCondition )
  If Not $pCondition Then Return ConsoleWrite( "$pCondition ERR" & @CRLF )
  ConsoleWrite( "$pCondition OK" & @CRLF )

  Local $pFile, $oFile
  $oNotepad.FindFirst( $TreeScope_Descendants, $pCondition, $pFile )
  $oFile = ObjCreateInterface( $pFile, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  If Not IsObj( $oFile ) Then Return ConsoleWrite( "$oFile ERR" & @CRLF )
  ConsoleWrite( "$oFile OK" & @CRLF )

  Local $pInvoke, $oInvoke
  $oFile.GetCurrentPattern( $UIA_InvokePatternId, $pInvoke )
  $oInvoke = ObjCreateInterface( $pInvoke, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern )
  If Not IsObj( $oInvoke ) Then Return ConsoleWrite( "$oInvoke ERR" & @CRLF )
  ConsoleWrite( "$oInvoke OK" & @CRLF )
  $oInvoke.Invoke()
  Sleep( 100 )

  ; --- Click "Save As..." menu ---

  ConsoleWrite( "--- Click ""Save As..."" menu ---" & @CRLF )

  ; Reuse $pCondition1 above

  ; Note that $UIA_AutomationIdPropertyId is a STRING and maybe a CASE SENSITIVE condition
 ; $oUIAutomation.CreatePropertyCondition( $UIA_AutomationIdPropertyId, "4", $pCondition2 )
  $oUIAutomation.CreatePropertyCondition( $UIA_NamePropertyId, "名前を付けて保存", $pCondition2 )

  If Not $pCondition2 Then Return ConsoleWrite( "$pCondition2 ERR" & @CRLF )
  ConsoleWrite( "$pCondition2 OK" & @CRLF )

  ; And condition
  $oUIAutomation.CreateAndCondition( $pCondition1, $pCondition2, $pCondition )
  If Not $pCondition Then Return ConsoleWrite( "$pCondition ERR" & @CRLF )
  ConsoleWrite( "$pCondition OK" & @CRLF )

  Local $pSaveAs, $oSaveAs
  $oNotepad.FindFirst( $TreeScope_Descendants, $pCondition, $pSaveAs )
  $oSaveAs = ObjCreateInterface( $pSaveAs, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  If Not IsObj( $oSaveAs ) Then Return ConsoleWrite( "$oSaveAs ERR" & @CRLF )
  ConsoleWrite( "$oSaveAs OK" & @CRLF )

  $oSaveAs.GetCurrentPattern( $UIA_InvokePatternId, $pInvoke )
  $oInvoke = ObjCreateInterface( $pInvoke, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern )
  If Not IsObj( $oInvoke ) Then Return ConsoleWrite( "$oInvoke ERR" & @CRLF )
  ConsoleWrite( "$oInvoke OK" & @CRLF )
  $oInvoke.Invoke()
  Sleep( 1000 )

  ; --- Save As window ---

  ConsoleWrite( "--- Save As window ---" & @CRLF )

  ; Note that $UIA_ClassNamePropertyId maybe ia a CASE SENSITIVE condition
  $oUIAutomation.CreatePropertyCondition( $UIA_ClassNamePropertyId, "#32770", $pCondition )
  If Not $pCondition Then Return ConsoleWrite( "$pCondition ERR" & @CRLF )
  ConsoleWrite( "$pCondition OK" & @CRLF )

  Local $pSaveAsWin, $oSaveAsWin
  $oNotepad.FindFirst( $TreeScope_Descendants, $pCondition, $pSaveAsWin )
  $oSaveAsWin = ObjCreateInterface( $pSaveAsWin, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  If Not IsObj( $oSaveAsWin ) Then Return ConsoleWrite( "$oSaveAsWin ERR" & @CRLF )
  ConsoleWrite( "$oSaveAsWin OK" & @CRLF )

  ; --- Set File name ---

  ConsoleWrite( "--- Set File name ---" & @CRLF )

  ; Note that $UIA_AutomationIdPropertyId is a STRING and maybe a CASE SENSITIVE condition
  $oUIAutomation.CreatePropertyCondition( $UIA_AutomationIdPropertyId, "1001", $pCondition )
  If Not $pCondition Then Return ConsoleWrite( "$pCondition ERR" & @CRLF )
  ConsoleWrite( "$pCondition OK" & @CRLF )

  $oSaveAsWin.FindFirst( $TreeScope_Descendants, $pCondition, $pEdit )
  $oEdit = ObjCreateInterface( $pEdit, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  If Not IsObj( $oEdit ) Then Return ConsoleWrite( "$oEdit ERR" & @CRLF )
  ConsoleWrite( "$oEdit OK" & @CRLF )

  $oEdit.GetCurrentPattern( $UIA_ValuePatternId, $pValue )
  $oValue = ObjCreateInterface( $pValue, $sIID_IUIAutomationValuePattern, $dtagIUIAutomationValuePattern )
  If Not IsObj( $oValue ) Then Return ConsoleWrite( "$oValue ERR" & @CRLF )
  ConsoleWrite( "$oValue OK" & @CRLF )
  $oValue.SetValue( "HelloWorld.txt" )

  ; --- Click Save button ---

  ConsoleWrite( "--- Click Save button ---" & @CRLF )

  $oUIAutomation.CreatePropertyCondition( $UIA_ControlTypePropertyId, $UIA_ButtonControlTypeId, $pCondition1 )
  If Not $pCondition1 Then Return ConsoleWrite( "$pCondition1 ERR" & @CRLF )
  ConsoleWrite( "$pCondition1 OK" & @CRLF )

  ; Note that $UIA_AutomationIdPropertyId is a STRING and maybe a CASE SENSITIVE condition
  $oUIAutomation.CreatePropertyCondition( $UIA_AutomationIdPropertyId, "1", $pCondition2 )
  If Not $pCondition2 Then Return ConsoleWrite( "$pCondition2 ERR" & @CRLF )
  ConsoleWrite( "$pCondition2 OK" & @CRLF )

  ; And condition
  $oUIAutomation.CreateAndCondition( $pCondition1, $pCondition2, $pCondition )
  If Not $pCondition Then Return ConsoleWrite( "$pCondition ERR" & @CRLF )
  ConsoleWrite( "$pCondition OK" & @CRLF )

  Local $pSave, $oSave
  $oSaveAsWin.FindFirst( $TreeScope_Descendants, $pCondition, $pSave )
  $oSave = ObjCreateInterface( $pSave, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  If Not IsObj( $oSave ) Then Return ConsoleWrite( "$oSave ERR" & @CRLF )
  ConsoleWrite( "$oSave OK" & @CRLF )

$oSave.SetFocus()
Sleep(8000)

  $oSave.GetCurrentPattern( $UIA_InvokePatternId, $pInvoke )
  $oInvoke = ObjCreateInterface( $pInvoke, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern )
  If Not IsObj( $oInvoke ) Then Return ConsoleWrite( "$oInvoke ERR" & @CRLF )
  ConsoleWrite( "$oInvoke OK" & @CRLF )
  $oInvoke.Invoke()
EndFunc

“C:\app\autoit3\UIAIncludes\UIA_Examples\5) Code snippets\1) Element array\ElementArray.au3"

#AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7

;#AutoIt3Wrapper_UseX64=n ; If target application is running as 32 bit code
;#AutoIt3Wrapper_UseX64=y ; If target application is running as 64 bit code

#include "..\..\..\Includes\CUIAutomation2.au3" ; Get proper version in UIASpy Includes folder
;#include "UIA_Functions.au3" ; Can be copied from UIASpy Includes folder
;#include "UIA_SafeArray.au3" ; Can be copied from UIASpy Includes folder
;#include "UIA_Variant.au3" ; Can be copied from UIASpy Includes folder

Opt( "MustDeclareVars", 1 )

Example()

Func Example()
  ; Create UI Automation object
  Local $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
  If Not IsObj( $oUIAutomation ) Then Return ConsoleWrite( "$oUIAutomation ERR" & @CRLF )
  ConsoleWrite( "$oUIAutomation OK" & @CRLF )

  ; Get Desktop element
  Local $pDesktop, $oDesktop
  $oUIAutomation.GetRootElement( $pDesktop )
  $oDesktop = ObjCreateInterface( $pDesktop, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  If Not IsObj( $oDesktop ) Then Return ConsoleWrite( "$oDesktop ERR" & @CRLF )
  ConsoleWrite( "$oDesktop OK" & @CRLF )

  ; --- Find window/control ---

  ConsoleWrite( "--- Find window/control ---" & @CRLF )

  Local $pCondition0
  $oUIAutomation.CreatePropertyCondition( $UIA_ClassNamePropertyId, "Notepad", $pCondition0 )
  If Not $pCondition0 Then Return ConsoleWrite( "$pCondition0 ERR" & @CRLF )
  ConsoleWrite( "$pCondition0 OK" & @CRLF )

  Local $pWindow1, $oWindow1
  $oDesktop.FindFirst( $TreeScope_Descendants, $pCondition0, $pWindow1 )
  $oWindow1 = ObjCreateInterface( $pWindow1, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  If Not IsObj( $oWindow1 ) Then Return ConsoleWrite( "$oWindow1 ERR" & @CRLF )
  ConsoleWrite( "$oWindow1 OK" & @CRLF )

  ; --- Find window/control ---

  ConsoleWrite( "--- Find window/control ---" & @CRLF )

  Local $pCondition1, $pCondition2, $pAndCondition2
  $oUIAutomation.CreatePropertyCondition( $UIA_ControlTypePropertyId, $UIA_MenuBarControlTypeId, $pCondition1 )
 ; $oUIAutomation.CreatePropertyCondition( $UIA_NamePropertyId, "Application", $pCondition2 ) ; <<<<<<<<<<<<<<<<
  $oUIAutomation.CreatePropertyCondition( $UIA_AutomationIdPropertyId, "MenuBar", $pCondition2 ) ; <<<<<<<<<<<<<<<<
  $oUIAutomation.CreateAndCondition( $pCondition1, $pCondition2, $pAndCondition2 )
  If Not $pAndCondition2 Then Return ConsoleWrite( "$pAndCondition2 ERR" & @CRLF )
  ConsoleWrite( "$pAndCondition2 OK" & @CRLF )

  Local $pMenuBar1, $oMenuBar1
  $oWindow1.FindFirst( $TreeScope_Descendants, $pAndCondition2, $pMenuBar1 )
  $oMenuBar1 = ObjCreateInterface( $pMenuBar1, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  If Not IsObj( $oMenuBar1 ) Then Return ConsoleWrite( "$oMenuBar1 ERR" & @CRLF )
  ConsoleWrite( "$oMenuBar1 OK" & @CRLF )

  ; --- $oUIAutomation methods ---

  ConsoleWrite( "--- $oUIAutomation methods ---" & @CRLF )

  Local $pCondition
  $oUIAutomation.CreateTrueCondition( $pCondition )
  ConsoleWrite( "$oUIAutomation.CreateTrueCondition()" & @CRLF )

  ; --- $oUIElement methods ---

  ConsoleWrite( "--- $oUIElement methods ---" & @CRLF )

  Local $pElements
  $oMenuBar1.FindAll( $TreeScope_Descendants, $pCondition, $pElements )
  ConsoleWrite( "$oMenuBar1.FindAll()" & @CRLF )

  ; --- Code Snippets ---

  ConsoleWrite( "--- Code Snippets ---" & @CRLF )

  Local $oUIElementArray1, $iLength1 ; $pElements is a pointer to an UI Automation element array
  $oUIElementArray1 = ObjCreateInterFace( $pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray )
  $oUIElementArray1.Length( $iLength1 )
  If Not $iLength1 Then Return ConsoleWrite( "$iLength1 = 0 ERR" & @CRLF )
  ConsoleWrite( "$iLength1 = " & $iLength1 & @CRLF )

  ; --- Code Snippets ---

  ConsoleWrite( "--- Code Snippets ---" & @CRLF )

  Local $pElement1, $oElement1, $sValue1
  For $i = 0 To $iLength1 - 1
    $oUIElementArray1.GetElement( $i, $pElement1 )
    $oElement1 = ObjCreateInterface( $pElement1, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
    ;$oElement1.GetCurrentPropertyValue( $UIA_ClassNamePropertyId, $sValue1 ) ; $UIA_ClassNamePropertyId is used as example
    ;ConsoleWrite( "$sValue1 = " & $sValue1 & @CRLF )
    $oElement1.GetCurrentPropertyValue( $UIA_NamePropertyId, $sValue1 )
    ConsoleWrite( "$sValue1 = " & $sValue1 & @CRLF )
  Next

  ; --- Copy element info ---

  ; $UIA_NamePropertyId                                 File
EndFunc

その他

IUIAutomation MS framework automate chrome, FF, IE, …. – AutoIt Example Scripts – AutoIt Forums (autoitscript.com)

EXAMPLES_V0_70.zip UIA_V0_70.zip

AutoItとUI Automation UDFを使ってポップアップウィンドウのボタンにマウスカーソルを移動させる

AutoIt

Posted by eightban