wxmp tensorflow

wxmp-tensorflow

The latest TensorFlow engineering solution running under the WeChat applet has the following characteristics

0. Run balzeface Android √, IOS √
1. Run face-landmarks Android √, IOS √
2. Integrated tfjsPlugin, easy to use, no need for secondary reference√
3. Support wasm backend √ The PR submitted to tfjs has been merged
4. Run posenet (example is mobilenet 0.50 stride16) √
5. custom tfjs reduce package size blazeface example√
6. Run handpose √
7. Run movenet √
8. Run blazepose-tfjs √

Note: Paddle.js has been adapted to Baidu mini-programs and WeChat mini-programs, https://github.com/PaddlePaddle/Paddle.js

Performance data

machine	Model	FPS	Backend
iPhone 7	BlazeFace	12	WebGL
iPhone 7	BlazeFace	1	WASM
MI 8	BlazeFace	20	WebGL
MI 8	BlazeFace	17	WASM
MI 8	FaceLandMarks	20	WebGL
MI 8	FaceLandMarks	10	WASM
MI 8	PoseNet	20	WebGL
MI 8	PoseNet	5	WASM
MI 8	HandPose	14	WebGL
MI 8	HandPose	1	WASM
MI 8	MoveNet	14	WebGL
MI 8	MoveNet	2.5	WASM

Note: WASM is a non-SIMD version. The Android WeChat applet runs SIMD WASM and reports an error. Although it can run, there are many problems with WeChat and IOS. It is recommended to use webview or webview. The performance is good, gl at least meets the standards and there are no memory problems.

question

0. The ios async version will be stuck with async await, so it is changed to the sync version, and other uses of async await may also occur (Promise under IOS is simulated by setTimeout, see the applet JavaScript support situation). Currently, it is easier to get initialization stuck under ios
1. wasm is better than webgl only on Huawei mobile phones (2.7 test results)

Optimization experience

0. Use FrameAdapter to dynamically skip frames
1. Before formally processing the data, preheat a blank frame and compile the shader used.
2. Use profile to generate custom tfjs to optimize package size
3. The ArrayBuffer of WeChat applet cameraFrame is reused. Maybe after the inference is completed, the content is updated, so the background is drawn first, and then the inference result is drawn. Another solution is to make a copy when getting the camera frame (about 0~5ms for iPhone7 , mostly 1ms)

TODO

0. Result: UI beautification done shows normal display
1. face-api.js
2. warm up is generalized, warm up is done based on the registered registerKernel

run

 > pnpm i / npm i / yarn
> npm run dev
# 使用小程序开发工具打开，替换appid为测试appid，不校验域名

# 使用custom tfjs编译，例子是仅仅blazeface可用tf + webgl backend 小程序包从 985KB下降到534KB (js大概下降234KB)
> npm run make-custom
> npm run build-custom

Use custom_tfjs to further optimize package size

Please refer to

0. [962K -> 347K] TensorflowJS TreeShaking based on Runtime results
1. https://github.com/mattsoulanille/tfjs_custom_module_demo
2. https://github.com/deepkolos/tfjs-treeshaking-test

tfjs-models async to sync method

You only need to change the async method of reading data, such as Tensor.array() to Tensor.arraySync() , or Tensor.buffer() to Tensor.bufferSync() , and then change async and await , Promise.all and other keywords Just remove it

sponsor

If the project is helpful to you or has adaptation needs, please feel free to tip.