wechat_articles_spider

WeChat public account article crawler (obtaining likes for reading WeChat articles)

Install

pip install wechatarticles

Display address:

Daily update, get the latest article link of the public account, support daily update, read likes and comment the text

Note: This project is only for learning and communication, and is strictly prohibited for commercial use (this project cannot be used directly) and cannot reach the out-of-the-box level. To use this project, you need to read the documentation + source code + hands-on practice, and refer to the sample code (under the test folder) for rewriting.

Tip: In addition, many friends (big guys) have already based on this set of projects or rewritten them to meet their own needs by directly looking at the source code.

Implementation idea one:

1. Obtain the URLs of all WeChat public articles from the WeChat public account platform
2. Log in to WeChat on PC or mobile to get the number of reads, likes, and comments of articles.

For complete ideas, you can refer to my blog: Remembering an experience of crawling WeChat public accounts (obtaining likes for reading WeChat articles)

Implementation idea two:

1. Log in to the PC or mobile version of WeChat to get the URLs of all articles on the public account. The number of URLs obtained is greater than 500, and the specific number is not completely consistent for each WeChat account (currently it can only be obtained once and cannot be obtained a second time!!! Please Use test_GetUrls.py with caution, preferably without .)
2. Use the same method as above to obtain the number of article reads, likes, and comments.

The permanent links to the crawled historical articles of public accounts are disclosed. The dates are as of the commit time. They are only for testing and learning. You are welcome to pay attention to these high-quality public accounts.

List of public accounts

Technological aesthetics

Communist Youth League Central Committee

southern weekend

AppSo

## Notes

The project started in 2017 and is currently updated in March 2023

To adjust the project code, please use pip install wechatarticles to call the previous interface.

1. When crawling fails, there may be the following reasons:
   1. When running, you need to turn off the network proxy (packet capture software), or add relevant parameters
   2. Whether the parameters are the latest. When obtaining WeChat related parameters (cookie, token), be sure to ensure that it is any article corresponding to the official account.
   3. Check code
   4. Need to pay attention to the corresponding public account (Maybe)
2. Idea 1: When obtaining the URL, the interval between each page can be set longer, such as 3 minutes, and the duration is several hours (from netizen testing)
3. When obtaining article reading likes, each article can be set to about 5-10 seconds, and the expiration time is 4 hours; if it is blocked, the crawling can continue in about 5-10 minutes.
4. Idea 2: When obtaining the URL, if it is blocked, it will take 24 hours before it can be crawled again (this article is invalid and cannot be unblocked temporarily)

Please see README for parameter file description.

python version

• python : 3.6.2, 3.7.3

Function realization

Function

Official account related

Official account information

Public account biz. How to obtain: Qingbo, official account webpage

Number of articles published by the public account (not entirely accurate)

Article related

The URL of a certain public account article. How to obtain: Official account webpage, PC WeChat, mobile WeChat, WeChat reading

All article information of a public account (including number of likes, number of reads, and comment information) needs to be manually changed.

Information about designated articles of a certain public account

Supports downloading WeChat articles locally and converting them to HTML (optional whether to save images)

API examples

Use the official account web version to obtain the WeChat article URL

There is a limit on the number of times here, so you cannot get too many URLs at one time. Solution: Multiple accounts crawl test_WechatUrls.py at the same time

Log in to WeChat PC to get article information (read and like)

test_WechatInfo.py

Quickly obtain a large number of article URLs (use historical articles to obtain links)

test_GetUrls.py

WeChat articles are downloaded as offline HTML (including pictures)

test_Url2Html.py

Learn/run process

You can read this issue, thank you very much for the concise explanation.

Related documents

See blog and documentation below

Please see this document for how to manually obtain official_cookie and token.

For an introduction to manual acquisition of wechat_cookie and appmsg_token, please refer to this document

Contact notes :

1. Cannot (cannot) automatically log in to WeChat official accounts and WeChat

2. Can’t (can’t) do real-time (get parameters, read likes, get articles)

3. Changing an official account or expiring parameters require manual updating. How to obtain parameters is mentioned in the article, please read carefully.

4. You cannot do keyword searches (i.e. WeChat search function), such as searching for all articles containing the word "technology".

Q&A

1. Can the project run normally?

   Answer: The project can run normally.

2. How to run/start xxx and what parameters need to be obtained?

   Answer: Please look at the source code and run it manually to see the output error.

3. How to obtain xxx parameters?

   Answer: The documentation and blog are very clear, please read it carefully.

4. I want xxxx, what should I do?

   Answer: Read the documentation and source code

5. How to update the web page daily?

   Answer: It is not a universal key. The solution is very simple, which is to simulate clicks + proxy software (Fiddler or Mitmproxy) to intercept packets and capture them once a day. If you have a better solution, please let us know. This part is not open source (if you see the relevant complete open source, you can raise an issue to learn more), purely because it is too troublesome to configure the environment + customization, and there are certain problems. If you see what can be achieved here, you can achieve it. If you ask me, I can’t answer you. It’s too time-consuming and energy-consuming.

6. Are WeChat and packet capture software Fiddler required on PC?

   Answer: No. This is just what I understand (think) that these two are relatively easiest to complete the entire process. Alternatives: You can capture WeChat on mobile phones (both Android and IOS are available, Android requires root to capture reading and likes); there are many alternatives to the packet capture software Fiddler, as long as it can capture HTTPS packets and view the data.

7. How to capture articles from a large number of public accounts?

   Answer: This project cannot be realized. There is no good solution, please refer to 5. The time cost of switching an official account is about 3-5 minutes, depending on the level of proficiency.

advertising space

appendix

Normal way to ask questions:

1. Describe clearly the system environment and Python environment you are running in... (this step can be optionally ignored)
2. What code was run (which part was changed), and what error was reported (please take a complete screenshot)?
3. What have you tried based on the error report? (Is there a description in the document? What are the solutions searched online)

Programming comes from practice. You can directly test whether the operation is correct or not. There is no need to waste the time of two people. If there is a problem during operation, please follow the above process to ask questions, but only if you have run it yourself . Please tell me your questions or needs directly without waiting for my reply. Thanks! Most of the questions can be exchanged. If you ask any detailed questions, we will also accept paid teaching.

WeChat Appreciation Code