nebula siwi
1.0.0
Siwi (/ˈsɪwi/) is a PoC of Dialog System With Graph Database Backed Knowledge Graph.
For now, it's a demo for task-driven(not general purpose) dialog bots with KG(Knowledge Graph) leveraging Nebula Graph with the minimal/sample dataset from Nebula Graph Manual/ NG中文手册.
Tips: Now you can play with the graph online without installing yourself!
Nebula Playground | Nebula Playground - China Mainland
Supported queries:
relation:
serving:
friendship:
You can try with it from scratch here: https://siwei.io/learn/nebula-101-siwi-kgqa/
This is one of the most naive pipeline for a specific domain/ single purpose chat bot built on a Knowledge Graph.
The Backend(Siwi API) is a Flask based API server:
Flask API server takes questions in HTTP POST, and calls the bot API.
In bot API part there are classfier(Symentic Parsing, Intent Matching, Slot Filling), and question actors(Call corresponding actions to query Knowledge Graph with intents and slots).
Knowledge Graph is built on an Open-Source Graph Database: Nebula Graph
The Frontend is a VueJS Single Page Applicaiton(SPA):
┌────────────────┬──────────────────────────────────────┐
│ │ │
│ │ Speech │
│ ┌──────────▼──────────┐ │
│ │ Frontend │ Siwi, /ˈsɪwi/ │
│ │ Web_Speech_API │ A PoC of │
│ │ │ Dialog System │
│ │ Vue.JS │ With Graph Database │
│ │ │ Backed Knowledge Graph │
│ └──────────┬──────────┘ │
│ │ Sentence │
│ │ │
│ ┌────────────┼──────────────────────────────┐ │
│ │ │ │ │
│ │ │ Backend │ │
│ │ ┌──────────▼──────────┐ │ │
│ │ │ Web API, Flask │ ./app/ │ │
│ │ └──────────┬──────────┘ │ │
│ │ │ Sentence ./bot/ │ │
│ │ ┌──────────▼──────────┐ │ │
│ │ │ │ │ │
│ │ │ Intent matching, │ ./bot/classifier│ │
│ │ │ Symentic Processing │ │ │
│ │ │ │ │ │
│ │ └──────────┬──────────┘ │ │
│ │ │ Intent, Entities │ │
│ │ ┌──────────▼──────────┐ │ │
│ │ │ │ │ │
│ │ │ Intent Actor │ ./bot/actions │ │
│ │ │ │ │ │
│ └─┴──────────┬──────────┴───────────────────┘ │
│ │ Graph Query │
│ ┌──────────▼──────────┐ │
│ │ │ │
│ │ Graph Database │ Nebula Graph │
│ │ │ │
│ └─────────────────────┘ │
│ │
│ │
│ │
└───────────────────────────────────────────────────────┘
.
├── README.md
├── src
│ ├── siwi # Siwi-API Backend
│ │ ├── app # Web Server, take HTTP requests and calls Bot API
│ │ └── bot # Bot API
│ │ ├── actions # Take Intent, Slots, Query Knowledge Graph here
│ │ ├── bot # Entrypoint of the Bot API
│ │ ├── classifier # Symentic Parsing, Intent Matching, Slot Filling
│ │ └── test # Example Data Source as equivalent/mocked module
│ └── siwi_frontend # Browser End
│ ├── README.md
│ ├── package.json
│ └── src
│ ├── App.vue # Listening to user and pass Questions to Siwi-API
│ └── main.js
└── wsgi.pyThe backend relis on the Nebula Graph, an Open Source Distributed Graph Database.
Install Nebula Graph in oneliner:
curl -fsSL nebula-up.siwei.io/install.sh | bashLoad the basketballplayer dataset.
~/.nebula-up/console.sh
nebula-console -addr graphd -port 9669 -user root -p nebula -e ":play basketballplayer"Install and run.
# Install siwi backend
python3 -m build
# Configure Nebula Graph Endpoint
export NG_ENDPOINTS=127.0.0.1:9669
# Run Backend API server
gunicorn --bind :5000 wsgi --workers 1 --threads 1 --timeout 60For OpenFunction/ KNative
docker build -t weygu/siwi-api .
docker run --rm --name siwi-api
--env=PORT=5000
--env=NG_ENDPOINTS=127.0.0.1:9669
--net=host
weygu/siwi-apiTry it out Web API:
$ curl -s --header "Content-Type: application/json"
--request POST
--data '{"question": "What is the relationship between Yao Ming and Lakers?"}'
http://192.168.8.128:5000/query | jq
{
"answer": "There are at least 23 relations between Yao Ming and Lakers, one relation path is: Yao Ming follows Shaquille O'Neal serves Lakers."
}Call Bot Python API:
from nebula3.gclient.net import ConnectionPool
from nebula3.Config import Config
# define a config
config = Config()
config.max_connection_pool_size = 10
# init connection pool
connection_pool = ConnectionPool()
# if the given servers are ok, return true, else return false
ok = connection_pool.init([('127.0.0.1', 9669)], config)
# import siwi bot
from siwi.bot import bot
# instantiate a bot
b = bot.SiwiBot(connection_pool)
# make the question query
b.query("Which team had Jonathon Simmons served?")Then a response will be like this:
In [4]: b.query("Which team had Jonathon Simmons serv
...: ed?")
[DEBUG] ServeAction intent: {'entities': {'Jonathon Simmons': 'player'}, 'intents': ('serve',)}
[DEBUG] query for RelationshipAction:
USE basketballplayer;
MATCH p=(v)-[e:serve*1]->(v1) WHERE id(v) == "player112"
RETURN p LIMIT 100;
[2021-07-02 02:59:36,392]:Get connection to ('127.0.0.1', 9669)
Out[4]: 'Jonathon Simmons had served 3 teams. Spurs from 2015 to 2015; 76ers from 2019 to 2019; Magic from 2017 to 2017; 'Referring to siwi_frontend
┌─────────────────────────────┐
│ kind: Ingress │ ┌───────────────────┐
│ path: / │ │ Pod │
│ -> siwi-frontend ────┼─────┤ siwi-frontend │
│ │ │ │
│ │ └───────────────────┘
│ │
│ path: /query │ ┌───────────────────────────────────┐
│ -> siwi-api ────┼─────┤ KNative Service │
│ KNative Serving │ │ serving-xxxx │
│ │ │ │
│ │ │ apiVersion: serving.knative.dev/v1│
│ │ │ kind: Service │
└─────────────────────────────┘ └─────────┬─────────────────────────┘
│
└────────────┐
│
┌───────────────────────────────────────────────────────┐ │
│apiVersion: core.openfunction.io/v1alpha1 │ │
│kind: Function │ │
│spec: │ │
│ version: "v1.0.0" │ │
│ image: "weygu/siwi-api:latest" │ │
│ imageCredentials: │ │
│ name: push-secret │ │
│ port: 8080 │ │
│ build: │ │
│ builder: openfunction/builder:v1 │ │
│ env: │ │
│ FUNC_NAME: "siwi_api" │ │
│ FUNC_TYPE: "http" │ │
│ FUNC_SRC: "main.py" │ │
│ srcRepo: │ │
│ url: "https://github.com/wey-gu/nebula-siwi.git" │ │
│ sourceSubPath: "src" │ │
│ serving: │ │
│ runtime: Knative ─────────────────────────────────┼──┘
│ params: │
│ NG_ENDPOINTS: "NEBULA_GRAPH_ENDPOINT" │
│ template: │ │
│ containers: │ │
│ - name: function │ │
│ imagePullPolicy: Always │ │
└───────────────────────────────────────┼───────────────┘
│
┌──────────┘
│
┌────────────────────────────┴───────────────────────────┐
│apiVersion:lapps.nebula-graph.io/v1alpha1 │
│kind: NebulaCluster │
│spec: │
│ graphd: │
│ config: │
│ system_memory_high_watermark_ratio: "1.0" │
│ image: vesoft/nebula-graphd │
│ replicas: 1 │
│... │
└────────────────────────────────────────────────────────┘
Assumed we have a k8s with OpenFunctions installed
Install a Nebula Graph with kubesphere-all-in-one nebula installer on KubeSphere:
curl -sL nebula-kind.siwei.io/install-ks-1.sh | bashGet Nebula Graph NodePort:
NEBULA_GRAPH_ENDPOINT=$(kubectl get svc nebula-graphd-svc-nodeport -o yaml -o jsonpath='{.spec.clusterIP}:{.spec.ports[0].port}')
echo $NEBULA_GRAPH_ENDPOINTLoad Dataset into the nebula cluster:
wget https://docs.nebula-graph.io/2.0/basketballplayer-2.X.ngql
~/.nebula-kind/bin/console -u root -p password --address=<nebula-graphd-svc-nodeport> --port=32669 -f basketballplayer-2.X.ngqlCreate the siwi-api powered by Openfunction:
cat siwi-api-function.yaml | sed "s/NEBULA_GRAPH_ENDPOINT/$NEBULA_GRAPH_ENDPOINT/g" | kubectl apply -f -Get the function nebula-siwi and the KNative Service:
kubectl get functions nebula-siwi
FUNCTION=$(kubectl get functions nebula-siwi -o go-template='{{.status.serving.resourceRef}}')
kubectl get ksvc -l openfunction.io/serving=$FUNCTION
KSVC=$(kubectl get ksvc -l openfunction.io/serving=$FUNCTION -o=jsonpath='{.items[0].metadata.name}')
kubectl get revision -l serving.knative.dev/service=$KSVC
REVISION=$(kubectl get revision -l serving.knative.dev/service=$KSVC -o=jsonpath='{.items[0].metadata.name}')
echo $REVISIONVerify the function worked fine:
curl -s --header "Content-Type: application/json"
--request POST
--data '{"question": "What is the relationship between Yao Ming and Lakers ?"}'
$(kubectl get ksvc -l openfunction.io/serving=$FUNCTION -o=jsonpath='{.items[0].status.url}')/queryCreate the siwi-app resources on K8s:
cat siwi-app.yaml | sed "s/REVISION/$REVISION/g" | kubectl apply -f -Verify the function worked fine through the ingress:
Here nodeport with http port 31059 was used as ingress controller endpoint.
curl -s --header "Content-Type: application/json"
--request POST
--data '{"question": "how does Tim Duncan and Lakers connected?"}'
demo-siwi.local:31059/queryVerify the frontend:
curl $(kubectl get svc -l app=siwi -o=jsonpath='{.items[0].spec.clusterIP}')Verify the frontend beind the ingress:
curl demo-siwi.local:31059Get all resources in siwi-app:
kubectl get service,pod,ingress,function -l app=siwiAnd it should be something like this:
[root@wey nebula-siwi]# kubectl get service,pod,ingress,function -l app=siwi
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/siwi-frontend-file ClusterIP 10.233.60.81 <none> 80/TCP 64m
NAME READY STATUS RESTARTS AGE
pod/siwi-frontend-file 1/1 Running 0 64m
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/siwi-service <none> demo-siwi.local 80 59m
NAME BUILDSTATE SERVINGSTATE BUILDER SERVING AGE
function.core.openfunction.io/nebula-siwi Succeeded Running builder-sbfz6 serving-vvjvl 26h
[root@wey nebula-siwi]# kubectl get service,pod,ingress,function -l app=siwi
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/siwi-frontend-file ClusterIP 10.233.60.81 <none> 80/TCP 65m
NAME READY STATUS RESTARTS AGE
pod/siwi-frontend-file 1/1 Running 0 65m
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/siwi-service <none> demo-siwi.local 80 59m
NAME BUILDSTATE SERVINGSTATE BUILDER SERVING AGE
function.core.openfunction.io/nebula-siwi Succeeded Running builder-sbfz6 serving-vvjvl 26hdocker build -t weygu/siwi-frontend . -f Dockerfile.froentend
docker push weygu/siwi-frontend
