Python 100 Days
1.0.0
Author : Luo Hao
Note : From the launch of the project to the acquisition of 8w+ stars, we have received feedback that the basic part (the content of the first 15 days) is difficult for novices. It is recommended that there be supporting videos to explain it. Recently, the basic part of the content has been remade into a project called "Python-Core-50-Courses". This part of the content has been rewritten in a simpler and more popular way and comes with video explanations. Beginners can take a look at this new 's warehouse. If domestic users are slow to access GitHub, they can follow my Zhihu account Python-Jack . The above column "Learn Python from Scratch" is more suitable for beginners. Other columns such as "Data Thinking and Statistical Thinking", "Based on Python" "Data Analysis" and so on are also being continuously created and updated. Everyone is welcome to pay attention, like and comment.
Friends who want to get learning videos can scan the QR code below to enter the WeChat mini program to see if there is content suitable for you. The machine learning content that everyone is thinking about can be found in the mini program, which was recorded by my colleagues and me for everyone.
If you encounter some difficult problems or need relevant learning resources during the learning process, you can join the following QQ communication group. You can join one of the three groups. Please do not join the group repeatedly, and do not post in the group. Advertising and other pornographic, vulgar or sensitive content. If you lack self-discipline and need to pay for learning, you can add my WeChat (jackfrued) private chat and note your name and needs. I will provide you with some guidance on study plans and career planning .
The supporting videos are being continuously updated on Douyin and Bilibili. Interested friends can follow my Douyin or Bilibili account. I have just opened my account recently. I hope you can support me a lot. Thank you very much!
The "Machine Learning and Deep Learning" that everyone has been urging to update has been at a standstill due to personal and company reasons. Related content will be updated in the near future. Thank you for your continued support and understanding.
Simply put, Python is an "elegant", "clear" and "simple" programming language.
Low learning curve, even non-professionals can get started
Open source system with a strong ecosystem
Interpreted language, perfect platform portability
Dynamically typed language that supports object-oriented and functional programming
The code is highly standardized and readable
Python is useful in the following fields.
Backend Development - Python/Java/Go/PHP
DevOps - Python/Shell/Ruby
Data Acquisition - Python/C++/Java
Quantitative Trading - Python/C++/R
Data Science - Python/R/Julia/Matlab
Machine Learning - Python/R/C++/Julia
Automated testing - Python/Shell
As a Python developer, there are many employment fields to choose from based on personal preferences and career plans.
Python back-end development engineer (server, cloud platform, data interface)
Python operation and maintenance engineer (automated operation and maintenance, SRE, DevOps)
Python data analyst (data analysis, business intelligence, digital operations)
Python data mining engineer (machine learning, deep learning, algorithm expert)
Python crawler engineer
Python test engineer (automated testing, test development)
Note : Currently, data analysis and data mining are very popular directions , because both the Internet industry and traditional industries have accumulated a large amount of data, and all walks of life need data analysts to discover more business insights from existing data. Value, thereby providing data support for corporate decision-making, which is the so-called data-driven decision-making.
A few suggestions for beginners:
Make English as your working language.
Practice makes perfect.
All experience comes from mistakes.
Don't be one of the leeches.
Either outstanding or out. (either outstanding or out)
Introduction to Python - History of Python/Advantages and Disadvantages of Python/Application Fields of Python
Build a programming environment - Windows environment/Linux environment/MacOS environment
Run Python program from terminal - Hello, world / print function / run program
Using IDLE - Interactive Environment (REPL) / Write multiple lines of code / Run program / Exit IDLE
Comments - the role of comments/single-line comments/multi-line comments
Programs and Base - Instructions and Programs/Von Neumann Machine/Binary and Decimal/Octal and Hexadecimal
Variables and types - naming of variables/use of variables/ input functions/checking variable types/type conversion
Numbers and strings - integers/floating point numbers/complex numbers/strings/basic string operations/character encoding
Operators - mathematical operators/assignment operators/comparison operators/logical operators/identity operators/operator precedence
Application case - Convert Fahrenheit temperature to Celsius temperature / Enter the radius of the circle to calculate the circumference and area / Enter the year to determine whether it is a leap year
Application scenarios of branch structure - conditions/indentation/code blocks/flow charts
if statement - simple if / if - else structure/ if - elif - else structure/nested if
Application cases - User authentication/Interchange of imperial units and metric units/Rolling dice to decide what to do/Converting percentile scores to grade system/Evaluating piecewise functions/Inputting the lengths of three sides and calculating the perimeter and area if a triangle can be formed
Application scenarios of loop structures - conditions/indentation/code blocks/flow charts
while loop - basic structure/ break statement/ continue statement
for loop - basic structure/ range type/branch structure in the loop/nested loop/end the program early
Application cases - Summing 1~100 / Determining prime numbers / Guessing number games / Printing ninety-nine tables / Printing triangle patterns / Monkey eating peaches / Hundreds of coins and hundreds of chickens
Classic cases: Narcissus Number/Hundred Money and Hundred Chickens/Craps Gambling Game
Practice questions: Fibonacci Sequence/Perfect Numbers/Prime Numbers
The role of functions - the bad smell of code/encapsulating functional modules with functions
Define function - def keyword/function name/parameter list/ return statement/call custom function
Calling functions - Python built-in functions/importing modules and functions
Function parameters - default parameters/variable parameters/keyword parameters/named keyword parameters
Return value of function - no return value / return single value / return multiple values
Scope issues - local scope/nested scope/global scope/built-in scope/scope-related keywords
Using module management functions - the concept of modules/using custom module management functions/what happens when there is a naming conflict (the same module and different modules)
Use of strings - length calculation/subscript operation/slicing/common methods
Basic usage of lists - define lists/use the following table to access elements/subscript out of bounds/add elements/delete elements/modify elements/slice/loop traverse
Common operations on lists - connection/copy (copy elements and copy arrays)/length/sort/reverse/search
Generate lists - use range to create lists of numbers/generate expressions/generators
Use of tuples - define tuples/use values in tuples/modify tuple variables/tuple and list conversions
Basic usage of sets - the difference between sets and lists/create a set/add elements/delete elements/clear
Common operations on sets - intersection/union/difference/symmetric difference/subset/superset
Basic usage of dictionary - characteristics of dictionary/create dictionary/add element/delete element/get value/clear
Common dictionary operations - keys method/ values method/ items method/ setdefault method
Basic exercises - Marquee effect/Finding the largest element in a list/The average score of statistical test scores/Fibonacci sequence/Yang Hui triangle
Comprehensive Case - Double Color Ball Number Selection/Tic-Tac-Toe
Classes and Objects - What is a class/What is an object/Other related concepts of object-oriented
Define class - basic structure/properties and methods/constructor/destructor/ __str__ method
Using Objects - Create Objects/Send Messages to Objects
The four pillars of object-oriented - abstraction/encapsulation/inheritance/polymorphism
Basic exercises - define student class/define clock class/define graphics class/define car class
Properties - class properties/instance properties/property accessors/property modifiers/property deleters/use __slots__
Methods in classes - instance methods/class methods/static methods
Operator overloading - __add__ / __sub__ / __or__ / __getitem__ / __setitem__ / __len__ / __gt__ / __lt__ / __le__ / __ge__ / __ge__ / __eq__ / __ne__ __contains__ __repr__
Relationship between classes (objects) - association/inheritance/dependence
Inheritance and polymorphism - what is inheritance/syntax of inheritance/calling parent class method/method overriding/type determination/multiple inheritance/diamond inheritance (diamond inheritance) and C3 algorithm
Comprehensive case-salary settlement system/automatic book discount system/custom score category
Develop GUI programs using tkinter
Develop game applications using pygame third-party library
"Big ball eats small ball" game
Read file - read the entire file / read line by line / file path
Write file - overwrite/append/text file/binary file
Exception handling - the importance of exception mechanism / try - except code block / else code block / finally code block / built-in exception type / exception stack / raise statement
Data persistence - CSV file overview/Application of csv module/JSON data format/Application of json module
Advanced operations on strings - Escape characters/original strings/multiline strings/ in and not in operators/ is_xxx methods/ join and split methods/ strip related methods/ pyperclip module/immutable strings and variable strings/ Use of StringIO
Introduction to regular expressions - the role of regular expressions/metacharacters/escaping/quantifiers/grouping/zero-width assertions/greedy matching and lazy matching lazy/use the re module to implement regular expression operations (matching, searching, replacing, capturing)
Using regular expressions - re module/ compile function/ group and groups method/ match method/ search method/ findall and finditer method/ sub and subn method/ split method
Application case - Use regular expressions to validate input strings
Concepts of processes and threads - What is a process / What is a thread / Multi-threading application scenarios
Using processes - fork function/ multiprocessing module/process pool/inter-process communication
Using threads - threading module/ Thread class/ RLock class/ Condition class/Thread pool
Computer Network Basics - History of Computer Network Development/"TCP-IP" Model/IP Address/Port/Protocol/Other Related Concepts
Network application mode - "Client-Server" mode / "Browser-Server" mode
Access network resources based on HTTP protocol - Network API overview/Access URL/ requests third-party library/Parse JSON format data
Python network programming - the concept of sockets/ socket module/ socket function/create TCP server/create TCP client/create UDP server/create UDP client
Email - SMTP/POP3/IMAP/ smtplib module/ poplib module/ imaplib module
SMS Service - Call SMS Service Gateway
Use Pillow to process pictures - picture reading and writing/picture synthesis/geometric transformation/color conversion/filter effects
Reading and writing Word documents - processing of text content/paragraphs/headers and footers/style processing
Read and write Excel files - xlrd / xlwt / openpyxl
Common data structures
Advanced usage of functions - "First-class citizens" / Higher-order functions / Lambda functions / Scope and closures / Decorators
Advanced knowledge of object-oriented - "Three Pillars" / Relationship between classes / Garbage collection / Magic properties and methods / Mixing / Metaclass / Object-oriented design principles / GoF design pattern
Iterators and Generators - Related Magic Methods/Two Ways to Create a Generator/
Concurrent and asynchronous programming - multi-threading/multi-process/asynchronous IO/ async and awai
Use HTML tags to carry page content
Render the page with CSS
Handling interactive behavior with JavaScript
Getting started with jQuery and improving it
Getting started with Vue.js
Use of Element
Use of Bootstrap
Operating system development history and Linux overview
Linux basic commands
Utilities in Linux
Linux file system
Vim editor application
Environment variables and shell programming
Software installation and service configuration
Network access and management
Other related content
Overview of relational databases
Introduction to MySQL
InstallMySQL
MySQL basic commands
Build database and table
Delete and modify tables
insert operation
delete operation
update operation
Projections and aliases
Filter data
Null value handling
Remove duplicates
sort
aggregate function
Nested queries
Group
table join
Cartesian product
inner join
natural connection
outer join
window function
Define window
ranking function
access function
Create user
grant permissions
Recall permissions
JSON type
window function
common table expression
view
Usage scenarios
Create view
Usage restrictions
function
built-in functions
User-defined function (UDF)
process
Creation process
Calling process
execution plan
The principle of indexing
Create index
Ordinary index
unique index
prefix index
composite index
Things to note
Install third-party libraries
Create connection
Get cursor
Execute SQL statement
Fetch data via cursor
Transaction commit and rollback
Release connection
Write ETL scripts
Hadoop ecosystem
Hive overview
Preparation
data type
DDL operations
DML operations
Data query
Web application working mechanism
HTTP requests and responses
Django framework overview
Get started quickly in 5 minutes
Relational database configuration
Use ORM to complete CRUD operations on the model
Use of management background
Django model best practices
Model definition reference
Load static resources
Ajax overview
Implement voting function using Ajax
Implement user tracking
The relationship between cookies and sessions
Django framework’s support for sessions
Cookie reading and writing operations in view functions
Modify response headers through HttpResponse
Use StreamingHttpResponse to handle large files
Use xlwt to generate Excel reports
Use reportlab to generate PDF reports
Use ECharts to generate front-end charts
Configuration log
Configure Django-Debug-Toolbar
Optimize ORM code
What is middleware
Django framework built-in middleware
Custom middleware and its application scenarios
Return data in JSON format
Render the page with Vue.js
REST overview
Getting started with the DRF library
Separate development of front-end and back-end
Application of JWT
Use CBV
Data paging
Data filtering
The first law of website optimization
Using Redis to provide caching services in Django projects
Read and write cache in view function
Using decorators to implement page caching
Provide caching services for data interfaces
File upload form control and image file preview
How to handle uploaded files on the server side
The second law of website optimization
Configure the message queue service
Use Celery to implement task asynchronousization in the project
Use Celery to implement scheduled tasks in the project
Unit testing in Python
Django framework’s support for unit testing
Use a version control system
Configuring and using uWSGI
Dynamic and static separation and Nginx configuration
Configure HTTPS
Configure domain name resolution
The concept of web crawler and its application areas
Discussion on the legality of web crawlers
Related tools for developing web crawlers
The composition of a crawler program
Use requests third-party library to implement data capture
Three ways to parse pages
Regular expression parsing
XPath parsing
CSS selector parsing
multithreading
multi-process
Asynchronous I/O
Install Selenium
Load page
Find elements and simulate user behavior
Implicit wait and explicit wait
Execute JavaScript code
Selenium anti-crawling crack
Set up a headless browser
Scrapy core components
Scrapy workflow
Install Scrapy and create the project
Write a spider program
Write middleware and pipeline programs
Scrapy configuration file
Data Analyst Responsibilities
Data Analyst Skill Stack
Data analysis related libraries
Install and use anaconda
conda related commands
Install and use jupyter-lab
Install and start
Use tips
Create array object
Array object properties
Index operations on array objects
Ordinary index
fancy index
boolean index
slice index
Case: Using arrays to process images
Related methods of array objects
Get descriptive statistics
Other related methods
Array operations
Array and scalar operations
Arrays and array operations
Universal unary function
general binary function
broadcast mechanism
Numpy common functions
vector
Determinant
matrix
polynomial
Create Series object
Operations on Series objects
Properties and methods of Series objects
Create DataFrame object
DataFrame object properties and methods
Read and write data in DataFrame
Data reshaping
Data splicing
Data merge
Data cleaning
Missing values
Duplicate values
Outliers
preprocessing
Pivot data
Get descriptive statistics
Sorting and header values
Group aggregation
Pivot tables and crosstabs
Data presentation
Calculate year-on-year comparison
window calculation
Relevance determination
Use of index
range index
Category index
multilevel index
interval index
datetime index
Install and import matplotlib
Create canvas
Create a coordinate system
Draw charts
Line chart
Scatter plot
bar chart
pie chart
Histogram
boxplot
Show and save charts
Advanced charts
bubble chart
area chart
radar chart
rose diagram
3D charts
Seaborn
Pyecharts
software process model
Product Backlog (user stories, product prototypes).
Planning meetings (evaluation and budgeting).
Daily development (stand-up meetings, Pomodoro technique, pair programming, test first, code refactoring...).
Fix bugs (problem description, reproduction steps, testers, assignees).
Release version.
Review meeting (Showcase, users need to participate).
Retrospective meeting (make a summary of the current iteration cycle).
Individuals and interactions over processes and tools
Working software over thorough documentation
Customer cooperation takes precedence over contract negotiation
Responding to change is better than following a plan
Feasibility analysis (whether to conduct research or not) and output a "Feasibility Analysis Report".
Requirements analysis (research on what to do), output "Requirements Specification" and product interface prototype diagram.
Outline design and detailed design, output conceptual model diagrams (ER diagrams), physical model diagrams, class diagrams, sequence diagrams, etc.
Coding/Testing.
Go online/maintenance.
Classic process model (waterfall model)
The biggest disadvantage of the waterfall model is that it cannot embrace changes in demand. The product cannot be seen until the entire process is completed, which leads to low team morale.
Agile development (Scrum) - product owner, Scrum Master, developers - Sprint
Supplement: Manifesto for Agile Software Development
Role: Product owner (the person who decides what to do and can make decisions on requirements), team leader (solves various problems, focuses on how to work better, and shields external influence on the development team), development team (project executive, Specifically developers and testers).
Preparation: business case and funding, contracts, vision, initial product requirements, initial release plan, taking equity stakes, building a team.
Agile teams usually have 8-10 people.
Workload estimation: Quantify development tasks, including prototypes, logo design, UI design, front-end development, etc., and try to decompose each work into the minimum task amount. The minimum task amount standard is that the working time cannot exceed two days, and then estimate the overall project time . Post each task on the Kanban board, which is divided into three parts: to do (to be completed), in progress (in progress) and done (completed).
Project team formation
Code comments are too few or no comments
Code breaks language best practices
Anti-pattern programming (spaghetti code, copy-paste programming, ego programming,…)
Team composition and roles
Note: Thanks to Ms. Fu Xiangying for helping me draw the following beautiful company organizational chart.
Programming specifications and code review ( flake8 , pylint )
Some "conventions" in Python (please refer to "Python Conventions-How to Write Pythonic Code")
Reasons that affect code readability:
Introduction to team development tools
Please refer to "Problems and Solutions in Team Project Development".
Version control: Git, Mercury
Defect management: Gitlab, Redmine
Agile closed-loop tools: ZenTao, JIRA
Continuous integration: Jenkins, Travis-CI
Topic scope setting
CMS (client): news aggregation website, Q&A/sharing community, movie review/book review website, etc.
MIS (user side + management side): KMS, KPI assessment system, HRS, CRM system, supply chain system, warehouse management system, etc.
App backend (management terminal + data interface): second-hand transactions, newspapers and magazines, niche e-commerce, news and information, travel, social networking, reading, etc.
Other types: Own industry background and work experience, business is easy to understand and control.
Requirement understanding, module division and task allocation
Requirements understanding: brainstorming and competitive product analysis.
Module division: Draw a mind map (XMind). Each module is a branch node, and each specific function is a leaf node (expressed with verbs). It is necessary to ensure that each leaf node cannot regenerate new nodes and determine each leaf. Importance, priority and workload of nodes.
Task allocation: The project leader assigns tasks to each team member based on the above indicators.
Develop project schedule (updated daily)
| module | Function | personnel | state | Finish | working hours | Plan begins | Actual start | Plan ends | actual end | Remark |
|---|---|---|---|---|---|---|---|---|---|---|
| Comment | Add comment | Wang Dachui | ongoing | 50% | 4 | 2018/8/7 | 2018/8/7 | |||
| Delete comment | Wang Dachui | wait | 0% | 2 | 2018/8/7 | 2018/8/7 | ||||
| View comments | Bai Yuanfang | ongoing | 20% | 4 | 2018/8/7 | 2018/8/7 | Code review required | |||
| Comment vote | Bai Yuanfang | wait | 0% | 4 | 2018/8/8 | 2018/8/8 |
OOAD and database design
Class diagram for UML (Unified Modeling Language)
Create a table through the model (forward engineering). For example, in a Django project, you can create a two-dimensional table through the following command.
python manage.py makemigrations app python manage.py migrate
Use PowerDesigner to draw physical model diagrams.
Create a model (reverse engineering) through a data table. For example, in a Django project, you can generate a model through the following command.
python manage.py inspectdb > app/models.py
Introduction to Docker
Install Docker
Use Docker to create containers (Nginx, MySQL, Redis, Gitlab, Jenkins)
Build a Docker image (Dockerfile writing and related instructions)
Container orchestration (Docker-compose)
Cluster management (Kubernetes)
basic principles
InnoDB engine
Index usage and precautions
data partition
SQL optimization
Configuration optimization
Architecture optimization
design principles
key questions
Other questions
Document writing
Database configuration (multiple databases, master-slave replication, database routing)
Cache configuration (partition cache, key settings, timeout settings, master-slave replication, failure recovery (Sentinel))
Log configuration
Profiling and Debugging (Django-Debug-ToolBar)
Useful Python modules (date calculation, image processing, data encryption, third-party API)
RESTful architecture
Understand RESTful architecture
RESTful API Design Guidelines
RESTful API best practices
Writing API interface documentation
RAP2
YAPI
Application of django-REST-framework
Use caching to relieve database pressure - Redis
Use message queues for decoupling and peak clipping - Celery + RabbitMQ
Type of test
Write unit tests ( unittest , pytest , nose2 , tox , ddt ,...)
Test coverage ( coverage )
Preparation before deployment
Key settings (SECRET_KEY / DEBUG / ALLOWED_HOSTS / cache / database)
HTTPS/CSRF_COOKIE_SECUR/SESSION_COOKIE_SECURE
Log related configuration
Review of common Linux commands
Installation and configuration of common Linux services
Usage of uWSGI/Gunicorn and Nginx
For simple applications that don't require a lot of customization, Gunicorn is a good choice. The learning curve of uWSGI is much steeper than Gunicorn, and Gunicorn's default parameters can already be adapted to most applications.
uWSGI supports heterogeneous deployment.
Since Nginx itself supports uWSGI, Nginx and uWSGI are generally deployed together online, and uWSGI is a fully functional and highly customized WSGI middleware.
In terms of performance, Gunicorn and uWSGI actually perform equally well.
Comparison of Gunicorn and uWSGI
Deploy test environment and production environment using virtualization technology (Docker)
Use of AB
Use of SQLslap
Use of sysbench
Automated testing using Shell and Python
Automated testing using Selenium
Selenium IDE
Selenium WebDriver
Selenium Remote Control
Introduction to testing tool Robot Framework
Business model and requirements points
Physical model design
Third party login
Cache warm-up and query caching
Shopping cart implementation
Payment function integration
Flash sales and oversold issues
Static resource management
Full text search solution
MySQL database tuning
Web server performance optimization
Nginx load balancing configuration
Keepalived achieves high availability
Code performance tuning
multithreading
Asynchronization
Static resource access optimization
cloud storage
CDN
computer basics
Python basics
Web framework related
Reptile related issues
data analysis
Project related
