Wrench: control your Android phone from your Desktop

Here’s a video of Wrench: playing monument valley (with phone screen mirrored), using Lua to script your phone, and ordering coffee (kinda).

Here’s another youtube video which introduced Wrench’s most useful functions: chatting with IM and Social Media Apps, finding contacts in IM and Social Media Apps, writing emails, and so on (inputing text is best done with the PC, then sent with the phone).

Install

There are prebuilt versions for Windows, Mac and Linux (ubuntu 14.04). They are released at https://github.com/SmartisanTech/Wrench-releases/releases .

Compilation

To compilen Wrench, you need make sure SnoreNotify is compiled and installed first. It’s in the SnoreNotify sub-directory. Please refer to the SnoreNotify/build-*.sh for your platform.

Compiling Wrench

• Under Linux, compile Wrench with the build-linux.sh script. You should be able to run this script directly.

  Because Wrench used QHotkey, you need to install libqt5x11extras5-dev first if you are under Debian/Ubuntu or something similar under other Linux distributions.

• For Windows version, I compile with Wine under Linux, with the Qt for msys installed as compiler. Please refer to the ./build-wine.sh script.
• For Mac version, please refer to the ./build-mac.sh script.

You need to adapt the ./build-wine.sh and ./build-mac.sh scripts, there are some hardcoded parameters (such as the PATH environment and the hostname of the Mac PC that I use for compilation).

How does it work

Android has provided quite some debugging abilities through adb, for example, you can

1. Inject keyboard and touch events with input
2. Start Activities and Services with am start|startservice
3. Take a screen shot with screencap
4. Record your screen with screenrecord
5. Dump window infomation with dumpsys window (where you can also get infomation about the Input Method)

Wrench just makes use of all the above, with a great deal of optimization.

Optimazations

better and faster adb shell input

Android’s builtin input command allows you to inject touch, keyboard, text events, but there’re some limitations.

1. You can use adb shell input text <string> to input a string, but the string is limited: it can’t contain spaces or any other special characters, such as quote and double quote, or Chinese characters.

   I solved this problem by using the Android clipboard and my own Input Method WrenchIme adapted from Google Pinyin. When Wrench need to input any string, it:

   • Put the string into the phone’s clipboard.
   • Trigger a special key (SCROLL_LOCK) with adb shell input
   • WrenchIme interpret the SCROLL_LOCK key to send the clipboard text to the current App, as if the text is an IME candidate and has been selected.
2. adb shell input is very slow, because every time input is run, a framework/input.jar is loaded and a new java virtual machine is created by running app_process.

   To solve this problem, Wrench comes with a Setclip.apk, which whileas being an Android Apk, it also provides the functionality of framework/input.jar (i.e., it can also be loaded and run by app_process), but instead of loading every time, it loads once and then runs as a daemon.

faster adb shell am

For the same reason as adb shell input, am is also very slow because an am.jar is loaded with a new java virtual machine each time. Wrench improved it using the same trick as input, in fact, Setclip.apk will load am.jar in turn when it is daemonized.

(Wrench’s am.jar is from AOSP, with userdebug build to make sure the .dex java class files are not optimized away and thus dalvik loadable.)

faster phone screen mirroring

At first, Wrench used screencap to make a screenshot of the phone screen, and displayed on the PC. This is very slow.

Now, with some phones (according to the Android SDK version and CPU arch), Wrench can use the VNC protocol to mirror your phone screen, which is a lot smoother as you can see from the video. In doing this, Wrench used source code from 3 projects: it combined the AOSP screenrecord project and the droidVncServer project to run a VNC server on the phone; and used the Qt VNC Viewer to talk to the phone’s VNC server.

How to use Wrench

Please refer to this youtube video.

There is also a blog explaining all the details, but it’s in Chinese.

Here’s the simple steps:

1. Start the Wrench program
2. Enable the phone’s USB debugging.
3. Connect the phone to Desktop with USB.

(Wrench is packaged with the AOSP adb program, so you need to make sure you are not running a different version of adb, as it may cause connection trouble.)

How to compile droidVncServer

Note: if your phone is deeply customized, it might not work if you compile the droidVncServer in AOSP source code. I found this out on Smartisan (the company that I’m working at)’s U1 phone, where I compiled droidVncServer with aosp-lollipop-mr1-dev, which has the same SDK version as U1, but sadly the program crashes when run. I also compiled the AOSP screenrecord program with aosp-lollipop-mr1-dev, then run it on U1, and found that it can do .mp4 file recording, but also crashes when run for streaming (with --output-format frames). You will see logcat error like this:

01-07 21:03:11.879  4960  4960 F DEBUG   : signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
01-07 21:03:11.879  4960  4960 F DEBUG   : Abort message: 'Vector<> have different types (this=0x7fc3d671b0 size=80, rhs=0x7fc3d67300, size=72)'
01-07 21:03:11.879  4960  4960 F DEBUG   :     x0  0000000000000000  x1  0000000000001359  x2  0000000000000006  x3  0000000000000008
01-07 21:03:11.879  4960  4960 F DEBUG   :     x4  fefeff761f685ca3  x5  fefeff761f685ca3  x6  fefeff761f685ca3  x7  7f7f7f7f7f7f7fff
01-07 21:03:11.879  4960  4960 F DEBUG   :     x8  0000000000000083  x9  b832f112249117ca  x10 0000000000000000  x11 fffffffc7fffffdf
01-07 21:03:11.879  4960  4960 F DEBUG   :     x12 0000000000000001  x13 ffffffffffffffff  x14 ffffffffff000000  x15 ffffffffffffffff
01-07 21:03:11.879  4960  4960 F DEBUG   :     x16 00000077207192b0  x17 0000007720638c24  x18 0000000000000010  x19 0000000000001359
01-07 21:03:11.879  4960  4960 F DEBUG   :     x20 0000000000001359  x21 0000000000000000  x22 000000772177f5e8  x23 0000007720ea7150
01-07 21:03:11.879  4960  4960 F DEBUG   :     x24 0000007720ea71a0  x25 000000772071b5b8  x26 0000005be4616d5c  x27 0000000000000003
01-07 21:03:11.879  4960  4960 F DEBUG   :     x28 0000005be466e000  x29 0000007fc3d66c60
01-07 21:03:11.879  4960  4960 F DEBUG   :     sp  0000007fc3d66c20  lr  000000772062c354  pc  000000772062c37c
01-07 21:03:11.882  4960  4960 I unwind  : Malformed section header found, ignoring...
01-07 21:03:11.882  4960  4960 F DEBUG   :
01-07 21:03:11.882  4960  4960 F DEBUG   : backtrace:
01-07 21:03:11.882  4960  4960 F DEBUG   :     #00 pc 000000000002237c  /system/lib64/libc.so (abort+116)
01-07 21:03:11.882  4960  4960 F DEBUG   :     #01 pc 0000000000008644  /system/lib64/liblog.so (__android_log_assert+296)
01-07 21:03:11.882  4960  4960 F DEBUG   :     #02 pc 0000000000011b98  /system/lib64/libutils.so (android::VectorImpl::operator=(android::VectorImpl const&)+2
20)
01-07 21:03:11.882  4960  4960 F DEBUG   :     #03 pc 0000000000097d50  /system/lib64/libgui.so (android::SurfaceComposerClient::Transaction::apply(bool)+540)
01-07 21:03:11.882  4960  4960 F DEBUG   :     #04 pc 000000000001f6e0  /data/data/com.android.shell/androidvncserver
01-07 21:03:11.882  4960  4960 F DEBUG   :     #05 pc 0000000000049c88  /data/data/com.android.shell/androidvncserver
01-07 21:03:11.882  4960  4960 F DEBUG   :     #06 pc 00000000000c905c  /system/lib64/libc.so (__libc_init+88)
01-07 21:03:12.140  1914  2840 E bt_brm  : bt_brm_chkdbglevelprop lbdebug: property not change
01-07 21:03:12.346  1307  1968 W NativeCrashListener: Couldn't find ProcessRecord for pid 4953

In this case, the only way to make it work is to compile under the same source code where your phone’s system image is compiled, which is unlikely unless you are a developper of this phone hardware company. Sorry.

If your phone’s Android SDK version/CPU arch is not supported, you can compile it yourself.

First you need get AOSP source code. SDK/NDK is not supported to compile droidVncServer because some system APIs are used which is not available in SDK/NDK.

Next copy the droidVncServer directory into AOSP’s external/droidVncServer, set up AOSP’s build appropriately, and run mma in external/droidVncServer.

Note that the source code for droidVncServer may need some porting if your phone’s SDK version is not 22 or 23. The changes may be small or big or even impossible depending on how different your phone’s SDK version is.

When build is finished, the binary is in external/droidVncServer, you need copy it back into your Wrench executable directory, with a rename (for e.g., androidvncserver-arm64.sdk23 should be renamed to androidvncserver-aarch64.sdk23, where aarch64 is the output from your phone’s adb shell uname -m ), and generate a .md5 file for it (Please refer to release/androidvncserver-aarch64.sdk23.md5).

Security Risks

Wrench need to use adb to connect to your phone. So it has all the security risks that comes with leaving adb open on your phone.

Wrench input text by putting it into the phone’s clipboard, so if there are clipboard sniffers running, you may accidentally leak sensitive infomation.

Contribute

Bug reports, patches, ideas are welcome.

Or, if you like Wrench, you may also donate some money to me with Paypal:

https://pledgie.com/campaigns/32699

or with alipay: