wechat spider

WeChat crawler

The following is the deployment document

For technical documentation, please view: https://t.zsxq.com/7ubmqNJ

For the reverse crawling plan, please see: https://wx.zsxq.com/dweb2/index/topic_detail/215584212588541

Function:

• Detect new articles posted daily by public accounts
• Capture public account information
• Fetch article list
• Fetch article information
• Capture readings, likes, and comments
• Fetch comment information
• Convert temporary link to permanent link

Packaged executable file download address

Link: https://pan.baidu.com/s/1hyhj6YnV-L9w8LPx42FFzQ Password: qnk6

feature:

1. No installation required : supports mac and window, double-click the software to execute it
2. Automation : You only need to configure the list of public accounts to be monitored, and after starting the software, you can automatically capture public accounts, articles and other information every day
3. Good docking : the captured data is stored using mysql, which facilitates data processing
4. No missing items : Use task status marking to prevent every public account and every article from being missed
5. Distributed : supports simultaneous collection of multiple WeChat signals, and the WeChat client supports all platforms of Android, iPhone, Mac, and Window

Data example

1. Official account data

2. Article list data

3. Article data

4. Read likes and comments data

5. Comment data

Required environment

1. mysql: used to store captured data and task tables
2. redis: task cache, reducing the number of mysql operations

Installation configuration

The following installation instructions are for reference only. Because everyone's environment is different, the installation may be slightly different. You can refer to the information online.

1. Install mysql

1.1 window

1.2 mac

2. Install redis

2.1 windows

2.2 mac

3. Install the certificate

You can visit mitm.it with a browser and download it, or how to install the mitmproxy certificate from Baidu

3.1 iPhone

1. After downloading and installing, don’t forget the last step
2. Open Settings-General-About This Mac-Certificate Trust Settings
3. Turn on the mitmproxy option.

3.2 android

1. Check after installation
2. Open Settings-Security-Trusted Credentials
3. Check whether the installed certificate exists

3.3 window

2. Double click to run
3. Install to local computer
4. Skip when key is required
5. Select "Place all certificates in the following store" and then select "Trusted Root Certification Authorities"
6. Finally, a warning window pops up, just click "Yes"

3.4 mac

2. After downloading, double-click to install
3. Open Keychain Access.app
4. Select login(Keychains) and Certificates(Category) to find mitmproxy
5. Click mitmproxy and select Always Trust in Trust

4. Configure proxy

If you use a mobile phone, you need to ensure that the mobile phone and the computer running wechat-spider are connected to the same router.

3.1 iPhone

Open Settings-Wireless LAN-Connected Wifi-Configure Agent-Manually fill in the IP and port 8080 of the installation server

3.2 android

Open settings-WLAN-long press the connected network-modify network-advanced options-manually fill in the IP and port 8080 of the installation server

3.3 window

Open chrome settings->advanced

3.4 mac

Open System Configuration (System Preferences.app) - Network - Advanced - Proxies - Secure Web Proxy (HTTPS) Fill in the IP and port 8080 of the installation server

Instructions for use

1. Install the certificate and configure the agent as described above.

2. Correctly configure config.yaml

The main thing is to configure the link information of mysql and redis to ensure that they can be connected correctly.

3. Create database wechat

4. Start wechat-spider

In this step, if the auto_create_tables value in config is true, the mysql data table will be automatically created. It is recommended to set it to true when starting for the first time and to false after creating the table.

5. Issue public account tasks

Enter data into wechat_account_task, such as: Just fill in __biz, such as: MzIxNzg1ODQ0MQ==

6. Click on any public account to view historical messages

When the prompt message in the red box above appears, it means you are done. After a while, you can go to the database to verify the data.

Technical exchange

If you have any questions or suggestions, you can join the QQ group to discuss the issues together. Please note微信爬虫学习交流

FAQ

1. mysql link problem

Problem: Printing object supporting the buffer api required exception when linking Solution: If the key is an integer, such as 123456, you need to add double quotes in the configuration file, as follows:

 mysqldb:
  ip: localhost
  port: 3306
  db: wechat
  user: root
  passwd: "123456"
  auto_create_tables: true # 是否自动建表 建议当表不存在是设置为true，表存在是设置为false，加快软件启动速度

2. After correctly configuring the proxy, a certificate or security question is prompted.

The reason is that my certificate has expired. Please refer to https://www.cnblogs.com/yunlongaimeng/p/9617708.html for installation data.

3. Prompt that there is no task

Check whether __biz is delivered in the wechat_account_task table. You can issue more tests

4. Exception:DISCARD without MULTI

5. Unable to capture package after normal startup

1. Check whether proxy is set
2. Check whether the port is occupied

WeChat rewards

Open source projects are not easy. Maintaining the code and solving everyone's problems often take up most of the time. In order to ensure continuous output of content, and this project happens to be helpful to you , I hope you will support me a lot (*￣︶￣).

Can provide deployment support and answer questions (only for tipping users and developers who raise PR).

WeChat: boris_tm