WeChat MiniProgram AR AI

Change log

date	content
2022-12-12	Newly added: AI human detection, AI hand detection, AR image tracking, etc., AR subway schematic diagram. Delete: package_image_classify directory of AI target classification. The package_face_detect directory of the old version of AI face detection.
2022-10-27	Newly added: AI face detection (new version), AI text detection, etc.
2022-09-20	New: single sample detection function. Update: webglBusiness.js has been upgraded to the latest official version, and the fix code for the Android deep conflict phenomenon has also been retained. Fixes: 1. When previewing the developer tools, it prompts that the size exceeds 2MB. 2. The texture maps of Three.PlaneGeometry, Three.BoxGeometry, etc. turn black, and the screen mirrors and flickers at the same time.
2022-01-05	Newly added: 1. AR spatial audio 2. AR+AI image classification 3. AR toy robot adds video recording function.
2022-01-01	Repair: The third modification method for screen depth conflict on Android phones.
2021-12-24	Fix: Use the top navigation bar to block the depth conflict phenomenon on Android mobile phones.
2021-12-23 night	Update: 1. The default size of the 3D model for plane tracking can be customized. 2. The 3D model changes from matrix update (matrixAutoUpdate=false) to attribute update (matrixAutoUpdate=true), so that the posture of the 3D model can be adjusted without a matrix.
2021-12-23pm	Fix: Snowflakes are displayed in the upper left corner of the screen on Android phones
2021-11-30	Newly added: 1. AR try-on glasses 2. AR measuring ruler 3. AR toy robot fixed on a flat surface

introduce

This project contains the following AR and AI examples.

1. AR+ content: Use AR plane tracking to display toy robots. Directory package_world_track.

2. AR+efficiency: Use AR to measure the length of objects on a plane. Directory package_measure.

3. AR+games: Use AR to follow spatial audio changes in the user's position. Directory package_spatial_audio.

4. AI face detection (new version): Wear virtual glasses on the face based on the face posture detected by AI. Catalog package_face_detect_v2.

5. AI text detection: detect text in pictures and camera screens. Catalog package_text_detect.

6. AR image tracking: Use an image to detect the posture of a flat image. Directory package_image_tracker.

7. AI single sample detection: Use only one picture to detect the position of the target object in the real environment. Directory package_oneshot_tracker.

8. AI human body key point detection: Detect human body key points and postures. Catalog package_body_detect.

9. AI hand key point detection: detect hand key points and gestures. Directory package_hand_gesture.

10. Shanghai subway map: View the map in a natural and interactive way. Directory package_plane_track.

Deleted example

AI face detection (old version): Wear virtual glasses on the person's face based on the face posture detected by AI. Catalog package_face_detect. (Deleted because the old version of the API is no longer officially maintained.)

AR+AI: Use AI to detect objects in images and display names on the objects. Directory package_image_classify. (The applet often exits due to insufficient memory, so it is deleted.)

Reference documentation

planar tracking

https://developers.weixin.qq.com/miniprogram/dev/framework/open-ability/visionkit/plane.html

Face key point detection (new version)

https://developers.weixin.qq.com/miniprogram/dev/framework/open-ability/visionkit/face.html

Text detection

https://developers.weixin.qq.com/miniprogram/dev/framework/open-ability/visionkit/ocr.html

Image tracking

https://developers.weixin.qq.com/miniprogram/dev/framework/open-ability/visionkit/marker.html

Single sample detection

https://developers.weixin.qq.com/miniprogram/dev/framework/open-ability/visionkit/osd.html

Human body key point detection

https://developers.weixin.qq.com/miniprogram/dev/framework/open-ability/visionkit/body.html

Hand keypoint and gesture detection

https://developers.weixin.qq.com/miniprogram/dev/framework/open-ability/visionkit/hand.html

Get RGBA camera image

https://developers.weixin.qq.com/miniprogram/dev/api/ai/visionkit/VKFrame.getCameraBuffer.html

front page

1. AR+ content

The toy robot stands on the floor of the room.

2. AR+efficiency

Tap the screen to start measuring. Please align the cursor position with both ends of the object being measured.

Tap the screen again to end the measurement.

The V1 version measurement results are inaccurate. For accurate measurements, please use version V2.

3. AR+games

When the user moves around with the phone, the volume of the player will change according to the distance of the user. Behind the red plane, the player's sound disappears.

4. AI face detection

The locations of 106 feature points. This project uses two feature points with index values ​​78 (left eye) and 79 (right eye).

Detecting rotated faces

5. AI text detection

Detect Chinese, English, numbers, etc. in the camera screen in real time.

6. AR image tracking

Identification map

Show toy robot

7. AI single sample detection

Show box

8. AI human body key point detection

19 key points

Using cosine distance, determine the movements of the eighth set of broadcast gymnastics.

stretching exercise

chest enlargement exercise

side movement

9. AI hand key point detection

21 key points

Using Euclidean distance to judge the rock-paper-scissors gesture.

cloth gesture

scissors gesture

stone gesture

10. Subway diagram

Pin the subway map to your desktop and read the map like a newspaper.

How to use

1. Use WeChat developer tools to open the project source code and preview it on your mobile phone.

2. If you encounter situations such as the model not loading or the image not being displayed, please turn on the debugging mode of the mini program.

If you change the 3D model

Modify the constant robotUrl in the source code.

  // 机器人模型
  const robotUrl = 'https://m.sanyue.red/demo/gltf/robot.glb' ;