fuzi nlu

FUZI NLU

?️ Chatbot -夫子’s “natural language understanding” module.

1 Introduction

夫子chatbot consists of 7 modules:

1. fuzi-bot: Chat interface module to interact with users.
2. fuzi-bot-api: Chat interface module, communicates with other back-end modules.
3. fuzi-nlu: Natural language processing module to understand user questions.
4. fuzi: Dialogue management module to infer user intentions.
5. fuzi-admin: Backend management module to manage robot settings.
6. fuzi-admin-api: Backend management interface to communicate with other backend modules.
7. fuzi-mark: Data annotation module, marking user questions.

There are two common types of chatbots:

1. Chat type open domain
2. task oriented

This project belongs to the second type, which is a task-oriented chatbot. A common application of this type of robot is intelligent customer service, which aims to solve the clear needs of users .

The picture above shows the general process of a task-oriented chat robot. The project currently implements the first part of the NLU function, including Slot Filling and Intent Prediction .

2 preview

>>Click me to try now<<

3 data

3.1 Dataset

1. guotie : The main content of this data set is about my family’s锅贴, and only the intent recognition function is used.
2. weather : A Chinese public data set about weather found on Github.
3. fewjoint : SMP2020.

3.2 Data annotation

The RASA open source annotation tool RASA-NLU-Trainer is used here for annotation.

I have deployed a copy online and can use it directly.

After the annotation is completed, the format needs to be converted before it can be used. Here, take /back/data/guotie.json as an example:

pip install rasa==2.6.3

cd fuzi-nlu/data
mkdir guotie

# rasa 暂时不支持从 json 直接转成 yaml，因此需要先转 md，再转 yaml
rasa data convert nlu -f md --data guotie.json --out ./guotie/nlu.md
rasa data convert nlu -f yaml --data ./guotie/nlu.md --out ./guotie/

rm ./guotie/nlu.md
mv ./guotie/nlu_converted.yml ./guotie/nlu.yml

# 生成 domain
python -m run.generate_domain_from_nlu --nlu ./data/guotie/nlu.yml --domain ./data/guotie/domain.yml

4 Get started quickly

git clone https://github.com/Ailln/fuzi-nlu.git

cd fuzi-nlu
# 安装依赖
pip install -r requirements.txt

python -m run.server

5 Deployment

5.1 Docker

git clone https://github.com/Ailln/fuzi-nlu.git

# in amd64
cd fuzi-nlu
# 打包
docker build -t fuzi-nlu:1.0.0 .
# 运行
docker run -d --restart=always --name fuzi-nlu -p 8081:8081 fuzi-nlu:1.0.0

# in arm64
cd fuzi-nlu
# 打包
docker build -t fuzi-nlu:1.0.0 -f deploy/arm64.Dockerfile .
# 运行
docker run -d --restart=always --name fuzi-nlu -p 8081:8081 fuzi-nlu:1.0.0

5.2 Kubernetes

 cd fuzi-nlu
# 准备好镜像
kubectl apply -f deploy/deployment.yaml

6 others

6.1 Retrain the model

 cd fuzi-nlu
# 训练
python -m run.train

# 测试
python -m run.test

6.2 QPS Test

pip install locust -U

locust -f test/qps_test.py -u 10 -r 2
# 打开 http://127.0.0.1:8089 

7 References

• Tensorflow dynamic seq2seq usage summary (r1.3)
• Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling
• BERT for Joint Intent Classification and Slot Filling
• From “connection” to “interaction”—Alibaba’s intelligent dialogue interaction practice and reflections
• A Frustratingly Easy Approach for Joint Entity and Relation Extraction
• FewJoint: A Few-shot Learning Benchmark for Joint Language Understanding

8 licenses

9 communication

Please add WeChat ID: Ailln_ , remark "fuzi", I invite you to join the communication group.