HouseAnalysisWeb
1.0.0
Web-side data visualization platform based on joint-spider crawler data, online access
The data source comes from the project Joint-spider
Backend: django-restframework
Frontend: Vue
# 环境配置需要 python3.7 redis mysql vue vue-cli3.x
#
# 项目启动
# 1. 后端服务启动,进入项目HouseAnalysis
python manage.py runserver
# 2. 前端项目启动,进入项目目录HouseAnalysisWeb
npm run dev
# 3. 前端项目打包
npm run build-prod
# 4. 访问地址:http://localhost:9527/
# 5. 后端接口地址: http://localhost:8000/ Community housing list details
Search property page
Specific property details page
Around the house
# 项目依赖
Django==2.1.5
django-cors-headers==2.4.0
django-crispy-forms==1.7.2
django-filter==2.0.0
django-formtools==2.1
django-guardian==1.4.9
django-import-export==1.1.0
django-redis==4.10.0
django-reversion==3.0.2
djangorestframework==3.9.0
djangorestframework-jwt==1.11.0
drf-extensions==0.4.0
PyMySQL==0.9.3
redis==3.0.1
SQLAlchemy==1.3.2Basic process
Django life cycle:
The front end sends a request-->Django's wsgi-->middleware-->routing system-->view-->ORM database operation-->template-->return data to the user
rest framework life cycle:
The front end sends a request-->Django's wsgi-->Middleware-->Routing system_Execute CBV's as_view(), which is to execute the internal dispath method-->Before executing dispath, there is version analysis and renderer--> In dispath, encapsulate the request-->Version-->Authentication-->Permissions-->Limit current-->View-->If the view uses caching (request.data or request.query_params ) uses the parser --> the view processes the data and uses serialization (serializing or verifying the data) --> the view returns the data using paging.
Project file tree:
# 1. 初始化项目
django-admin startproject HouseAnalysis
# 2. 创建相关应用
cd HouseAnalysis
django-admin startapp house # 项目根目录下创建extra_app 文件夹，存放xadmin压缩文件，运行
pip3 install xadmin # 1. 配置settings.py 文件 位于\HouseAnalysisHouseAnalysissettings.py
# 找到INSTALLED_APPS,加入配置项
' corsheaders ' ,
' xadmin ' ,
' crispy_forms ' ,
' rest_framework ' ,
' django_filters ' ,
' house '
# 上述配置向是用于将app注册到项目中
# 2. 增加跨域支持
CORS_ALLOW_CREDENTIALS = True
CORS_ORIGIN_ALLOW_ALL = True
CORS_ORIGIN_WHITELIST = (
' * '
)
CORS_ALLOW_METHODS = (
' DELETE ' ,
' GET ' ,
' OPTIONS ' ,
' PATCH ' ,
' POST ' ,
' PUT ' ,
' VIEW ' ,
)
CORS_ALLOW_HEADERS = (
' XMLHttpRequest ' ,
' X_FILENAME ' ,
' accept-encoding ' ,
' authorization ' ,
' content-type ' ,
' dnt ' ,
' origin ' ,
' user-agent ' ,
' x-csrftoken ' ,
' x-requested-with ' ,
' Pragma ' ,
)
# 3. 在DATABASES中配置MySQL数据库信息
DATABASES = {
' default ' : {
' ENGINE ' : ' django.db.backends.mysql ' ,
' NAME ' : ' house ' ,
' USER ' : ' root ' ,
' PASSWORD ' : ' 123456 ' ,
' HOST ' : ' 127.0.0.1 ' ,
' PORT ' : ' 3306 ' ,
' OPTIONS ' :{
' init_command ' : " SET sql_mode='STRICT_TRANS_TABLES' "
},
}
}
# 4. 配置drf缓存
REST_FRAMEWORK_EXTENSIONS = {
# 10s后失效
' DEFAULT_CACHE_RESPONSE_TIMEOUT ' : 10
}
# 5. 配置redis缓存
CACHES = {
" default " : {
" BACKEND " : " django_redis.cache.RedisCache " ,
" LOCATION " : " redis://127.0.0.1:6379/3 " ,
" OPTIONS " : {
" CLIENT_CLASS " : " django_redis.client.DefaultClient " ,
}
}
}Note: This project is an interface development and does not involve template templates and static resources, so there is no need to configure these two items.
Enter HouseAnalysisHouseAnalysisurls.py and configure routing information:
Add path('xadmin/', xadmin.site.urls) , re_path(r'docs/', include_docs_urls(title="HA")) to the urlpatterns list, xadmin is the management page routing,
Docs is the interface document routing, which can be accessed through http://localhost:8000/docs/, as shown below:
Django Model design is one of the five basic core designs of Django (Model design, URL configuration, View writing, Template design, From use), and it is also an important link in the MVC (MVT) model.
Model is a data model, not a database. It describes the structure of data and the logical relationship between them. In Django, Model design is essentially a class. Each model maps to a database table.
Enter HouseAnalysishousemodels.py and add Api，Elevator，Floor，Layout，Region，Decortion，Purposes，Orientation，Constructure，Community，CommunityRange models to provide data for serialization.
After the model is created, perform data migration:
# 进入项目根目录
python manage.py makemigrations
# 同步至数据库
python manage.py migrateCreate a new file serializer.py file in the house folder to serialize data.
Define a serializer (essentially a class), which generally includes fields of the model class and has its own field type rules. After implementing the serializer, you can create serialized objects and query sets for serialization operations, and obtain data through serialized object.data (you do not need to construct a dictionary yourself and then return Json data)
When used for serialization, pass the model class object into the instance parameter
When used for deserialization, pass the data to be deserialized into the data parameter.
In addition to the instance and data parameters, when constructing the Serializer object, you can also add additional data through the context parameter.
Here we use the model serializer (ModelSerializer), which is the same as the regular Serializer, but provides:
For example: House content can be serialized using simple code
class HouseSerializer ( serializers . ModelSerializer ):
class Meta :
# 需要序列化的model
model = Api
# 需要序列化的字段
fields = "__all__" In the same way, serialize the remaining models, including ElevatorSerializer，FloorSerializer，LayoutSerializer，RegionSerializer，RegionS，CommunitySerializer，CommunityRangeSerializer，DecortionSerializer，PurposesSerializer，ConstructureSerializer，OrientationSerializer , among which RegionS serializer serves as the superior serializer of CommunitySerializer and CommunityRangeSerializer .
The function of the view: To put it bluntly, it is to receive front-end requests and perform data processing.
(The processing here includes: if the front-end is a GET request, construct a query set and return the results. This process is serialization ; if the front-end is a POST request, if you want to make changes to the database, you need to get the data sent by the front-end. Verify and write the data to the database. This process is called deserialization )
The most original view can implement such logical processing, but for different requests, multiple methods need to be defined in the class view to implement their own processing. This can solve the problem, but there is a flaw, that is, the general method in each function The logic is similar: read request, get data from the database, write things to the database, and return the results to the front end. This will produce a lot of duplicate code.
In the views of developing REST API, although the specific data operations of each view are different, the implementation process of adding, deleting, modifying and checking is basically routine, so this part of the code can also be reused and simplified to write:
Added : Verify request data -> Execute deserialization process -> Save database -> Serialize and return the saved object
Delete : Determine whether the data to be deleted exists -> perform database deletion
Modification : Determine whether the data to be modified exists -> verify the requested data -> perform the deserialization process -> save the database -> serialize and return the saved object
Query : Query the database -> Serialize and return the data
Views are written in the views.py file, and here we use class recognition diagrams for development.
http://127.0.0.1/api/v1/allhouse , we need to return and expose the serialized results of all house information.http://127.0.0.1/api/v1/allhouse /1, we need to change the house id (the id at this time can be configured as The serialization result that specifies the requirement parameter) is 1 is returned and exposed.{id:xxx,values:xxx...} , we need to check the current information and add the current data to the database if it is legal.example:
# CacheResponseMixin: 缓存，内存级别
# ListModelMixin处理GET（all）请求
# RetrieveModelMixin处理（id）请求
# 基础类GenericViewSet，提供分页，过滤，排序等功能
class HouseListViewSet ( CacheResponseMixin , mixins . ListModelMixin , mixins . RetrieveModelMixin , GenericViewSet ):
queryset = Api . objects . all ()
# 房源序列化
serializer_class = HouseSerializer
# 房源分页
pagination_class = HousePagination
# 自定义排序，过滤类
filter_backends = ( DjangoFilterBackend , filters . SearchFilter , filters . OrderingFilter )
# 过滤信息字段
filter_fields = ( 'price' ,)
# 搜搜字段
search_fields = ( 'title' , 'region' , 'community_name' )
# 排序字段
ordering_fields = ( 'price' , 'unit_price' , 'construction_area' )
# 重写retrieve方法，实现获取单个序列化对象信息
def retrieve ( self , request , * args , ** kwargs ):
instance = self . get_object ()
serializer = self . get_serializer ( instance )
return Response ( serializer . data )
# 分页配置项
class HousePagination ( PageNumberPagination ):
page_size = 30
page_size_query_param = 'page_size'
page_query_param = "page"
max_page_size = 100 Other views are similar, including: ElevaorViewSet，FloorViewSet，LayoutViewSet，RegionViewSet，CommunityViewSet，CommunityPagination，AllCommunityViewSet，CommunityRangeViewSet，DecortionViewSet，PurposesViewSet，ConstructureViewSet，OrientationViewSet
For the CommunityViewSet class, we need to return all its information based on the specific community_id. At this time, we cannot use the retrieve method of the RetrieveModelMixin class. We need to override the get_queryset method to return a specific model.
def get_queryset ( self ):
if self . request . GET [ 'rid' ] == '0' :
# 获取所有
return Community . objects . all ()
else :
return Community . objects . filter ( region = self . request . GET [ 'rid' ])Other parameters
Use rest_framework 's DefaultRouter for route registration:
from rest_framework . routers import DefaultRouter
router = DefaultRouter ()
router . register ( r'v1/api/all_house' , HouseListViewSet , base_name = "house" )
......Add router to urlpatterns:
path ( '' , include ( router . urls )),At this time, when a request accesses the existing route, the route will forward the request to the view class. After the request is processed in the view class, the serialized result is returned, which is the interface data exposed by the backend.
[The external link image transfer failed. The source site may have an anti-leeching mechanism. It is recommended to save the image and upload it directly (img-JmH9EULX-1588257851195) (imgs/interface data.png)]
Data description
HTTP status code:
# 将所有房源数，小区数，总体单价价均价，总体总价均价，存入redis
def insertIndexInfo ():
all_number = house . count ()
com_number = Api . objects . values ( 'community_name' ). distinct (). count ()
mean_price = format ( house . aggregate ( Avg ( 'price' ))[ 'price__avg' ], '.3f' )
mean_unit_price = format ( house . aggregate ( Avg ( 'unit_price' ))[ 'unit_price__avg' ], '.3f' )
data = {
"all_number" : all_number ,
"com_number" : com_number ,
"mean_price" : mean_price ,
"mean_unit_price" : mean_unit_price
}
redis_conn . mset ( data ) # 1. 定义专门数据库操作类InsertIntoMysql
class InsertIntoMysql ():
# 属性中定义sql语句获取数据
def __init__ ( self ):
self . elevator_sql = 'select unit_price, price, elevator from house_api'
# ......
# 定义数据插入方法
def insertElevator ( self ):
index , values , price_mean , unit_price_mean = multiple ( self . elevator_sql , engine , 'elevator' )
for i in range ( len ( index )):
elevator = Elevator ()
elevator . version = 'v1'
elevator . title = '电梯分布情况'
elevator . has_ele = index [ i ]
elevator . el_num = values [ i ]
elevator . mean_price = price_mean [ i ]
elevator . mean_unit_price = unit_price_mean [ i ]
elevator . save ()
# 其他剩余数据插入方法
# ......
# 2. 数据处理模块，包括单因子和对因子处理
# 利用数据处理模块pandas
def single ( sql , engine , feature ):
"""
:param sql: sql语句
:param engine: mysql引擎
:param feature: 数据特征
:return: 特征，数量
"""
df_sql = sql
df = pd . read_sql ( df_sql , engine )
df [ feature ] = df [ feature ]. astype ( str )
if feature == 'floor' :
df [ df [ feature ] == '暂无数据' ] = '18'
df [ feature ] = df [ feature ]. apply ( lambda x : re . findall ( 'd+' , x )[ 0 ])
feature_res = df [ feature ]. value_counts ()
index = feature_res . index
values = feature_res . values
return index , values
def multiple ( sql , engine , feature ):
"""
:param sql: sql语句
:param engine: mysql引擎
:param feature: 数据特征
:return: 特征，数量，总价均价，单价均价
"""
df_sql = sql
df = pd . read_sql ( df_sql , engine )
df [ feature ] = df [ feature ]. astype ( str )
if feature == 'region' :
df [ feature ] = df [ feature ]. apply ( lambda x : re . sub ( r"[|]|'" , '' , x ). split ( ',' )[ 0 ])
feature_res = df [ feature ]. value_counts ()
index = feature_res . index
values = feature_res . values
price_mean = []
unit_price_mean = []
max_unit_price = []
min_unit_price = []
for inx , val in zip ( index , values ):
price_mean . append ( format ( df [ df [ feature ] == inx ][ 'price' ]. astype ( float ). mean (), '.3f' ))
unit_price_mean . append ( format ( df [ df [ feature ] == inx ][ 'unit_price' ]. astype ( float ). mean (), '.3f' ))
max_unit_price . append ( format ( df [ df [ feature ] == inx ][ 'unit_price' ]. astype ( float ). max (), '.3f' ))
min_unit_price . append ( format ( df [ df [ feature ] == inx ][ 'unit_price' ]. astype ( float ). min (), '.3f' ))
return index , values , price_mean , unit_price_mean , max_unit_price , min_unit_price The front-end is built using the basic framework Vue and developed based on vue-admin-template. The main front-end business is used for chart display and map information. It is mainly used to visualize Chengdu second-hand house price data and related information, including overall information display, various characteristics - such as house decoration status, chart information visualization, housing search, basic housing Source information visualization, map display and other functions.
Project description
Front-end project file tree:
Specific process
# 安装到项目
npm install -S element-ui
# 全局加载: 在main.js中
import ElementUI from ' element-ui '
import ' element-ui/lib/theme-chalk/index.css '
import locale from ' element-ui/lib/locale/lang/en ' // i18n
Vue.use(ElementUI, { locale })
# 这样我们据可以在vue文件中使用UI组件，如: 按钮组件<Button /> # 安装到项目
npm install -S vue-baidu-map
# 局部加载: 在任意vue文件中
import BaiduMap from ' vue-baidu-map/components/map/Map.vue '
# 地图标注组件
import { BmMarker } from ' vue-baidu-map '
# 使用，例如:
< baidu-map ref= " baidumap "
class= " bm-view "
style= " width:100%;height:93vh "
ak= " YOUR AK "
:center= " {lng: 104.07, lat: 30.67} "
:scroll-wheel-zoom= " true "
:zoom= " 16 " >
< bm-navigation anchor= " BMAP_ANCHOR_TOP_LEFT " ></bm-navigation >
< bm-marker v-for= " spoi in searchResults " -- >
< ! --:position= " {lng: spoi.lng, lat: spoi.lat} " -- >
< ! --:title= " spoi.community_name " > -- >
< /bm-marker >
< /baidu-map >Create a new folder utils in the src directory to store our customized tool files, create a new requests.js, and re-encapsulate the axios method to meet the operations when we request the interface. This file will expose an object used for asynchronous requests as a more advanced interface access method. This method will be used in the interface file design.
# 新建一个servie对象
const service = axios . create ( {
// 项目基本配置文件中定义的VUE_APP_BASE_API，开发和发布环境各部相同，在.env.devolopment和.env.product文件中进行配置
baseURL : process . env . VUE_APP_BASE_API ,
// 直接写死的url，无论环境如何都不会改变
// baseURL: 'http://127.0.0.1:8000/v1/api',
timeout : 8000 // 最大请求时间
} ) service . interceptors . request . use ( config => {
// 请求方法为POST时，配置请求头
config . method === 'post'
? config . data = qs . stringify ( { ... config . data } )
: config . params = { ... config . params }
config . headers [ 'Content-Type' ] = 'application/x-www-form-urlencoded'
return config
} , error => {
// 错误时提示信息
Message ( {
message : error ,
type : 'error' ,
duration : 3 * 1000
} )
Promise . reject ( error )
} ) service . interceptors . response . use (
response => {
if ( response . statusText === 'OK' ) {
// 正确响应时，直接返回数据
return response . data
} else {
Notification ( {
title : '错误' ,
message : '访问api错误' , // TODO 后端拦截错误请求
type : 'error' ,
duration : 3 * 1000
} )
}
} ,
// 错误处理
error => {
let dtext = JSON . parse ( JSON . stringify ( error ) )
try {
if ( dtext . response . status === 404 ) {
Notification ( {
type : 'error' ,
title : '出问题404' ,
message : '访问api错误或服务器忙' ,
duration : 3 * 1000
} )
}
} catch ( err ) {
Notification ( {
title : '错误' ,
message : '请求超时,请稍后再试或刷新页面' ,
type : 'error' ,
duration : 3 * 1000
} )
}
return Promise . reject ( error )
}
) export default serviceUse vue-router to implement the routing function, create a new router.js, and configure routing items.
// 定义路由列表
export const constantRoutes = [
{
path : '/404' ,
component : ( ) => import ( '@/views/404' ) ,
hidden : true
} ,
{
path : '/' ,
component : Layout ,
redirect : '/dashboard' ,
children : [ {
path : 'dashboard' ,
name : 'Dashboard' ,
// 采用路由懒加载的方式
component : ( ) => import ( '@/views/dashboard/index' ) ,
meta : { title : '房源信息' , icon : 'house' }
} ]
} ,
// ......
]
// 构造路由对象
const createRouter = ( ) => new Router ( {
// mode: 'history', // require service support
scrollBehavior : ( ) => ( { y : 0 } ) ,
routes : constantRoutes
} )
// 在main.js 中引入并挂载
import router from './router'
new Vue ( {
el : '#app' ,
router ,
render : h => h ( App )
} )The interface api.js file defines all interface access requests related to the project, and is encapsulated into functions and exposed for use by components.
// 例如获取区划信息接口
export function getRegionInfo ( id , params ) {
return request ( {
url : '/region/' + id ,
method : 'get' ,
params
} )
}The request is the interface access object encapsulated above, which needs to be imported before use. When using the interface in the component, import it on demand, for example:
// charts.js是接口函数文件
import { getRegionInfo } from '@/api/charts'
// 使用
// 第一种方式，由于它本来就是一个promise对象，所以直接在then中获取响应
getRegionInfo ( 1 ) . then ( ( res , err ) => {
// 相关操作
} )
// 第二种方式，使用async await方式,包含await的方法中需要加上async 作为方法前缀
let res = await getRegionInfo ( 1 )
