robotic process automation Download - robotic process automation Source code download

Robotic Process Automation

Robotic process automation (RPA), like human workers, automates tasks through software or hardware systems that run across a variety of applications. Software or bots can learn workflows with multiple steps and applications, such as getting received forms, sending acknowledgment messages, checking form integrity, filing forms into folders, and updating spreadsheets with form name, submission date wait. RPA software is designed to relieve employees of the burden of completing repetitive, simple tasks.

Usage

Introduction to client automation

Please ensure that the corresponding client has been installed on the local computer, and both rpa-client and rpa-server have been started.

The current system supports the following clients:

AppId	Name
wechat	WeChat
wecom	Enterprise WeChat
qq	Tencent QQ
tim	TIM
dingtalk	DingTalk
lark	Feishu

Since tasks must be executed by a specified user, you need to ensure that the user exists before executing the task.

1. Create user

Create user via HTTP interface

Interface: http://<host>:<port>/users
Method: POST
Format: JSON
Parameters: Body

Property	Type	Required	Description
users	User[]	Required	Array of user objects.
└id	String	Optional	User ID. When empty, the server will automatically generate an ID.
└appId	String	Required	Associated AppID.
└account	String	Required	User account, used to match clients.
└ Nickname	String	Optional	User nickname, used for display.
└ realname	String	Optional	The user's real name is used for display.
└company	String	Optional	The name of the company to which the user belongs is used for display.

Note that企业微信client cannot directly obtain account of the current logged-in person. Currently, the client is matched through the combination of ${realname}_${company} .

Interface example:

curl -X POST --location " http://localhost:8080/users " 
    -H " Content-Type: application/json " 
    -d " {
          " users " : [
            {
              " id " : " uid " ,
              " appId " : " wechat " ,
              " account " : " account " ,
              " nickname " : " nickname "
            }
          ]
        } "

Create user via SQL script

 insert into user (id, app_id, account, nickname, realname, company, status, created_time, updated_time)
values ( ' uid ' , ' wechat ' , ' account ' , ' nickname ' , ' realname ' , ' company ' , 1 , now(), null );

2. Create tasks

Create tasks via HTTP interface

Interface: http://<host>:<port>/tasks
Method: POST
Format: JSON
Parameters: Body

Property	Type	Required	Description
tasks	Task[]	Required	Array of task objects.
└id	String	Optional	Task ID. When empty, the server will automatically generate an ID.
└ userId	String	Required	Associated user ID.
└ type	String	Required	Task type, refer to the task type dictionary table.
└ priority	Integer	Optional	Task priority, the smaller the value, the higher the priority. When empty, the default configured priority is used.
└data	String	Optional	Task data, in the format of `JSON` string.
└ scheduleTime	DateTime	Optional	Task execution time, for example: `2022-01-01 10:00:00` . If empty, it will be executed immediately.

Example:

curl -X PATCH --location " http://localhost:8080/tasks " 
    -H " Content-Type: application/json " 
    -d " {
          " tasks " : [
            {
              " id " : " tid " ,
              " userId " : " uid " ,
              " type " : " login " ,
              " priority " : " 0 " ,
              " data " : "" ,
              " scheduleTime " : " 2022-01-01 10:00:00 " ,
            }
          ]
        } "

Create tasks through SQL scripts

 insert into task (id, user_id, app_id, type, priority, data, status, created_time, updated_time, schedule_time)
values ( ' tid ' , ' uid ' , ' wechat ' , ' login ' , 0 , null , 0 , now(), null , ' 2022-01-01 10:00:00 ' );

WeChat client automation

The server provides a runtime test page that developers can use to test simple tasks locally. Open the browser and visit http://<host>:<port>/index.html , and then select the client to be tested.

WeChat client login task

No parameters are required to log in to the client

task_wechat_login

WeChat client logout task

No parameters are required to log out of the client

task_wechat_logout

WeChat client sends personal message task

Parameter format:

Property	Type	Required	Description
target	String	Required	Send object.
messages	Message[]	Required	Array of message objects.
└ type	String	Required	Message type.
└ content	String	Required	Message content, text content or file address.

Message type:

Code	Name	Description
text	TEXT	text
image	IMAGE	picture
video	VIDEO	video
file	FILE	document

Parameter example:

{
  "target" : " friend " ,
  "messages" : [
    {
      "type" : " text " ,
      "content" : " message "
    },
    {
      "type" : " image " ,
      "content" : " https://rpa.leego.io/image.png "
    },
    {
      "type" : " video " ,
      "content" : " https://rpa.leego.io/video.mp4 "
    },
    {
      "type" : " file " ,
      "content" : " https://rpa.leego.io/file.zip "
    }
  ]
}

task_wechat_send_private_messages

WeChat client sends group message task

Parameter format:

Property	Type	Required	Description
target	String	Required	Group name.
messages	Message[]	Required	Array of message objects.
└ type	String	Required	Message type.
└ content	String	Required	Message content, text content or file address.

Message type:

Code	Name	Description
text	TEXT	text
image	IMAGE	picture
video	VIDEO	video
file	FILE	document
mention	MENTION	remind

Parameter example:

{
  "target" : " group " ,
  "messages" : [
    {
      "type" : " text " ,
      "content" : " message "
    },
    {
      "type" : " image " ,
      "content" : " https://rpa.leego.io/image.png "
    },
    {
      "type" : " video " ,
      "content" : " https://rpa.leego.io/video.mp4 "
    },
    {
      "type" : " file " ,
      "content" : " https://rpa.leego.io/file.zip "
    },
    {
      "type" : " mention " ,
      "content" : " member "
    }
  ]
}

task_wechat_send_group_messages

Enterprise WeChat client automation

Enterprise WeChat client login task

No parameters are required to log in to the client

task_wecom_login

Enterprise WeChat client logout task

No parameters are required to log out of the client

task_wecom_logout

Enterprise WeChat client sends personal message task

Parameter format:

Property	Type	Required	Description
target	String	Required	Send object.
messages	Message[]	Required	Array of message objects.
└ type	String	Required	Message type.
└ content	String	Required	Message content, text content or file address.

Message type:

Code	Name	Description
text	TEXT	text
image	IMAGE	picture
video	VIDEO	video
file	FILE	document

Parameter example:

{
  "target" : " friend " ,
  "messages" : [
    {
      "type" : " text " ,
      "content" : " message "
    },
    {
      "type" : " image " ,
      "content" : " https://rpa.leego.io/image.png "
    },
    {
      "type" : " video " ,
      "content" : " https://rpa.leego.io/video.mp4 "
    },
    {
      "type" : " file " ,
      "content" : " https://rpa.leego.io/file.zip "
    }
  ]
}

task_wecom_send_private_messages

Enterprise WeChat client sends group message task

Parameter format:

Property	Type	Required	Description
target	String	Required	Group name.
messages	Message[]	Required	Array of message objects.
└ type	String	Required	Message type.
└ content	String	Required	Message content, text content or file address.

Message type:

Code	Name	Description
text	TEXT	text
image	IMAGE	picture
video	VIDEO	video
file	FILE	document
mention	MENTION	remind

Parameter example:

{
  "target" : " group " ,
  "messages" : [
    {
      "type" : " text " ,
      "content" : " message "
    },
    {
      "type" : " image " ,
      "content" : " https://rpa.leego.io/image.png "
    },
    {
      "type" : " video " ,
      "content" : " https://rpa.leego.io/video.mp4 "
    },
    {
      "type" : " file " ,
      "content" : " https://rpa.leego.io/file.zip "
    },
    {
      "type" : " mention " ,
      "content" : " member "
    }
  ]
}

task_wecom_send_group_messages

WeChat client adding contact task

Parameter format:

Property	Type	Required	Description
target	String	Required	Group name.
contacts	Contact[]	Required	Array of contact objects.
└ target	String	Required	User’s mobile phone number or email address.
└ reason	String	Optional	Add contact notes.

Parameter example:

{
  "contacts" : [
    { "target" : " phone " },
    { "target" : " email " , "reason" : " reason " }
  ]
}

Too advanced for display.

Documentation

Multi-client operation by closing mutex

In order to prevent the client from being started multiple times, developers usually create mutex objects. A mutex lock is a mechanism used in multi-threaded programming to protect shared resources from being accessed by multiple threads or processes at the same time.

Create or open a mutex object in Windows

CreateMutexA function (synchapi.h)

HANDLE CreateMutexA (
  [in, optional] LPSECURITY_ATTRIBUTES lpMutexAttributes,
  [in]           BOOL                  bInitialOwner,
  [in, optional] LPCSTR                lpName
);

parameter

[in, optional] lpMutexAttributes

Pointer to a SECURITY_ATTRIBUTES structure. If this parameter is NULL , the handle cannot be inherited by child processes.

The lpSecurityDescriptor member of the structure specifies the security descriptor for the new mutex. If lpMutexAttributes is NULL , the mutex gets a default security descriptor. ACL in the mutex's default security descriptor comes from the creator's primary or impersonation token.

[in] bInitialOwner

If this value is TRUE and the caller creates the mutex, the calling thread takes initial ownership of the mutex object. Otherwise, the calling thread does not take ownership of the mutex lock. To determine whether the caller created the mutex, see the Return Values section.

[in, optional] lpName

The name of the mutex object. The name is limited to MAX_PATH characters. Name comparisons are case-sensitive.

If lpName matches the name of an existing named mutex, this function requests MUTEX_ALL_ACCESS access. In this case, the bInitialOwner parameter is ignored because it is already set by the creation process. If the lpMutexAttributes parameter is not NULL, it determines whether the handle can be inherited, but its security descriptor members are ignored.

If lpName is NULL , the mutex is created without a name.

If lpName matches the name of an existing event, semaphore, waitable timer, job, or file mapping object, the function fails and the GetLastError function returns ERROR_INVALID_HANDLE . This is because these objects share the same namespace.

The name can have a "global" or "local" prefix to explicitly create the object in the global or session namespace. The remainder of the name can contain any characters except the backslash character (). For more information, see Kernel Object Namespace. Use Terminal Services sessions for fast user switching. Kernel object names must follow the guidelines outlined for Terminal Services so that applications can support multiple users.

The object can be created in a private namespace. For more information, see Object Namespaces.

return value

If the function succeeds, the return value is the handle (Handle) of the newly created mutex object.

If the function fails, the return value is NULL . To obtain extended error information, call the GetLastError function.

If the mutex is a named mutex and the object existed before this function was called, the return value is a handle to the existing object and the GetLastError function returns ERROR_ALREADY_EXISTS .

Close mutex via handle

Using Process Explorer

Process Explorer is a tool officially provided by Microsoft to find out Handle and DLL information that a process has opened or loaded.

Process Explorer official page: https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer

Example:

Taking WeChat as an example, first start WeChat, find the process named WeChat.exe in the main interface of Process Explorer and select it.
Then find Handle with Type Mutant and Name Sessions1BaseNamedObjects_WeChat_App_Instance_Identity_Mutex_Name in the Lower Pane interface.
After you right-click Close Handle to close the handle, you can start a new process.

process_explorer_wechat_1

process_explorer_wechat_2

Python code implementation

 handles = handler . find_handles ( process_ids = [ 10000 ], handle_names = [ r'Sessions1BaseNamedObjects_WeChat_App_Instance_Identity_Mutex_Name' ])
handler . close_handles ( handles )

Reference source code: client/handler/handler.py

How to keep Windows Remote Desktop interactive after it is closed through `tscon`

When you use Remote Desktop to connect to a remote computer, closing Remote Desktop locks the computer and displays the login screen. In lockdown mode, the computer has no GUI , so any currently running or scheduled GUI tests will fail.

To avoid problems with GUI testing, you can use the tscon utility to disconnect from the remote desktop. tscon returns control to the original local session on the remote computer, bypassing the login screen. All programs on the remote computer continue to run normally, including GUI testing.

What is `tscon` ?

tscon is a tool provided by Windows systems that can be used to connect to another session on the Remote Desktop Session Host server.

How to use `tscon` ?

tscon { < sessionID > | < sessionname > } [/dest: < sessionname > ] [/password: < pw > | /password: * ] [/v]

parameter	describe
`<sessionID>`	Specifies the ID of the session to connect to. If the optional `/dest:<sessionname>` parameter is used, the name of the current session can also be specified.
`<sessionname>`	Specify the name of the session to connect to.
/dest: `<sessionname>`	Specifies the name of the current session. When you connect to a new session, this session will be disconnected. You can also use this parameter to connect another user's session to a different session.
/password: `<pw>`	Specify the password of the user who owns the session to connect to. This password is required when the connecting user does not own the session.
/password: `*`	Prompt for the password of the user who owns the session you want to connect to.
/v	Display information about the operation being performed.
/?	Display help at the command prompt.

Manually disconnect the remote

To disconnect from a remote desktop, run the following command on the remote computer (in the Remote Desktop Connection window) as an administrator, for example, from the command line:

%windir% S ystem32 t scon.exe RDP-Tcp# # # NNN /dest:console

Where RDP-Tcp### NNN is ID of the current remote desktop session, such as RDP-Tcp#5 . You can see it in the Session column on the Users tab of Windows Task Manager .

Windows Task Manager

You will see the message Your Remote Desktop Services session has ended and the Remote Desktop client will close. However, all programs and tests on the remote computer will continue to run normally.

Tip: The session column is hidden by default. To display it, right-click somewhere in the row showing CPU, Memory, etc. and select Session in the context menu that opens.

Disconnect via batch file

You can use a batch file to automate the disconnection process. On the remote computer, do the following:

Create a batch file using the following code:

 for /f " skip=1 tokens=3 " %%s in ( ' query user %USERNAME% ' ) do (
  %windir% S ystem32 t scon.exe %%s /dest:console
)

Create a desktop shortcut to this file. To do this, right-click on the batch file and select Send to > Desktop (create shortcut).
In the shortcut properties, click Advanced and select Run as administrator.
Double-click this shortcut on the remote computer (in the Remote Desktop Connection window) or call this batch file at the beginning of the test (provided the test is run as administrator).

Things to note

tscon keeps the remote computer unlocked, which reduces system security. After the test run is complete, you can lock the machine using the following command:
```
Rundll32.exe user32.dll, LockWorkStation
```
If the rdpclip.exe process is running on the remote computer and the clipboard is not empty when you disconnect from the remote session, the rdpclip.exe process may fail.
To avoid this problem, you can kill the rdpclip.exe process before disconnecting the session.

Reference link

https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/tscon
https://support.smartbear.com/testcomplete/docs/testing-with/running/via-rdp/keeping-computer-unlocked.html

Function expansion

If you need to expand the automation function or be compatible with different versions of clients, you can add or edit task scripts in the rpa-client/app module.

The following two methods are mainly used in the project:

pywinauto
Is a python module for Microsoft Windows GUI automation. In the simplest case, it allows you to send mouse and keyboard operations to Windows dialog boxes and controls, but it supports more complex operations, such as getting text data.
Airtest
It is a cross-platform, image recognition-based UI automation testing framework launched by NetEase Games. It is suitable for games and apps. The supported platforms are Windows, Android and iOS.

Deployment

RPA client deployment

Please ensure that the operating system you are using is Windows 7 and above, and the Python version is 3.7.0 and above.

Download and install the client

It should be noted that the current version of airtest depends on pywinauto==0.6.3 , and the current project requires pywinauto==0.6.8 . Please add the --no-deps parameter when installing the dependencies, or manually execute pip install pywinauto==0.6.8 after installing the dependencies. pip install pywinauto==0.6.8 .

git clone https://github.com/yihleego/robotic-process-automation.git
cd robotic-process-automation/rpa-client
pip install --no-deps -r requirements.txt

Configure client

The client configuration file is located in rpa-client/config.yml, and developers can modify the configuration according to actual scenarios.

Property	Description	Default
server.host	Server host	localhost
server.port	Server port	18888
server.path	Server path	/rpa
server.ssl	Whether to enable SSL	False
app.size	Maximum number of programs that can be run	32
app.path. `<appid>`	Custom program path	Get from registry
airtest.cvstrategy	Image recognition algorithm	[tpl,sift,brisk]
airtest.timeout	Image recognition algorithm	20 seconds
airtest.timeout-tmp	Image recognition algorithm	3 seconds
logging.level	Log level	DEBUG
logging.format	Log format	Default format
logging.filename	Log file name	./logs/rpa-client.log

Start client

Just run rpa-client/main.py.

RPA server deployment

Please make sure that the version of Java you are using is 17 or above. The service depends on MySQL and Redis when running. Please be sure to install and start them before deploying the service.

Download and install the server

git clone https://github.com/yihleego/robotic-process-automation.git
cd robotic-process-automation/rpa-server
mvn clean install

Configure server

Required configuration

Property	Description	Default
spring.datasource.driver-class-name	Data source driven	com.mysql.cj.jdbc.Driver
spring.datasource.url	Data source URL	jdbc:mysql://localhost:3306/rpa
spring.datasource.username	Data source username
spring.datasource.password	Data source password
spring.data.redis.host	Redis host	localhost
spring.data.redis.port	Redis port	6379
spring.data.redis.password	Redis password
spring.data.redis.database	Redis database	0

The above configuration can be modified in the application.properties file.

Optional configuration

Property	Description	Default
rpa.websocket.port	WebSocket service port	18888
rpa.websocket.path	WebSocket service path	/rpa
rpa.websocket.idle-timeout	WebSocket service idle timeout	5m
rpa.converter.date-time-pattern	Global date time format	yyyy-MM-dd HH:mm:ss
rpa.converter.date-pattern	global date format	yyyy-MM-dd
rpa.converter.time-pattern	global time format	HH:mm:ss
rpa.client.cache-key	Client cache key format	rpa:client: `<appid>` : `<account>`
rpa.client.cache-timeout	Client cache timeout	5m

See RpaProperties for details.

Start the server

Before starting the service, please execute the following script in the MySQL instance

rpa_ddl.sql
rpa_dml.sql

Just run RpaApplication.java.

Q&A

The difference between UiaApp and AirApp

UiaApp: A program implemented based on UIA. It locates the corresponding elements through the ID, Name, Class, etc. of the UI component, and then operates on them. Similar to the DOM tree of HTML, elements can be located and operated through selectors, for example: $(".btn").click(); .
AirApp: It is based on image recognition.

To determine whether an application supports UIA, you can use the software Inspect officially provided by Microsoft, which can be downloaded from the official website or this warehouse:

In the example, WeChat uses the UiaApp mode because it is implemented based on UIA; while Enterprise WeChat uses the AirApp mode, which is similar to the fact that there is only one Canvas in a web page. All elements are drawn and rendered through code, so they can only be positioned through image recognition. .

Could not find module 'libiconv.dll'

Download Visual C++ Redistributable Packages for Visual Studio 2013 https://www.microsoft.com/zh-cn/download/confirmation.aspx?id=40784

RPA Server issues

Please refer to: 2#issue

Contact

Submit a question

Attention

This project is for learning reference only, please do not use it in production environment.

License

This project is under the MIT license. See the LICENSE file for details.

Expand

robotic process automation