AI Gateway

AI Native API Gateway

Build Status

English | Chinese | Japanese

Higress is a cloud-native API gateway. The core is based on Istio and Envoy. Wasm plug-ins can be written with Go/Rust/JS, etc., and provides dozens of ready-made general plug-ins, as well as an out-of-the-box console (demo click here)

Higress was born within Alibaba to solve the problem of Tengine reload's damage to long-connection services and the lack of gRPC/Dubbo load balancing capabilities.

Alibaba Cloud has built a cloud-native API gateway product based on Higress, providing a large number of enterprise customers with 99.99% gateway high-availability guaranteed service capabilities.

Higress is based on AI gateway capabilities and supports AI businesses such as Tongyi Qianwen APP, Bailian large model API, and machine learning PAI platform. At the same time, we serve domestic leading AIGC companies (such as Zero One Thousand Things), as well as AI products (such as FastGPT)

Summary

quick start
Function display
Usage scenarios
Core advantages
Community

quick start

Higress only requires Docker to start, which is convenient for individual developers to build and learn locally, or to build simple sites:

 # 创建一个工作目录
mkdir higress ; cd higress
# 启动 higress，配置文件会写到工作目录下
docker run -d --rm --name higress-ai -v ${PWD} :/data 
        -p 8001:8001 -p 8080:8080 -p 8443:8443  
        higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:latest

The listening port description is as follows:

Port 8001: Higress UI console entrance
Port 8080: Gateway HTTP protocol entry
Port 8443: Gateway HTTPS protocol entry

All Docker images of Higress have always used their own exclusive warehouse and are not affected by the restricted domestic access of Docker Hub.

For other installation methods such as Helm deployment under K8s, please refer to the official Quick Start documentation.

If you are deploying on the cloud, it is recommended to use the enterprise version for the production environment. For development and testing, you can use the following one-click deployment of the community version:

Usage scenarios

AI Gateway :
Higress can use a unified protocol to connect to all domestic and foreign LLM model manufacturers, and has rich AI observability, multi-model load balancing/fallback, AI token flow control, AI caching and other capabilities:
Kubernetes Ingress gateway :
Higress can be used as the Ingress entry gateway for K8s clusters, and is compatible with a large number of K8s Nginx Ingress annotations, allowing for quick and smooth migration from K8s Nginx Ingress to Higress.
Supports the Gateway API standard and supports smooth migration of users from the Ingress API to the Gateway API.
Compared with ingress-nginx, resource overhead is significantly reduced, and routing changes take effect ten times faster:
Microservice gateway :
Higress can serve as a microservice gateway and can connect to multiple types of registration centers to discover service configuration routes, such as Nacos, ZooKeeper, Consul, Eureka, etc.
It is deeply integrated with microservice technology stacks such as Dubbo, Nacos, and Sentinel, and is based on the excellent performance of the Envoy C++ gateway core. Compared with traditional Java-based microservice gateways, it can significantly reduce resource usage and reduce costs.
Security protection gateway :
Higress can serve as a security protection gateway, provide WAF capabilities, and support multiple authentication strategies, such as key-auth, hmac-auth, jwt-auth, basic-auth, oidc, etc.

Core advantages

production grade
Born out of Alibaba's internal product that has been produced and verified for more than two years, it supports large-scale scenarios with hundreds of thousands of requests per second.
Completely get rid of traffic jitter caused by Nginx reload, configuration changes take effect in milliseconds and the business is not affected. It is particularly friendly to long-term connection scenarios such as AI business.
streaming
Supporting true full streaming request/response body, the Wasm plug-in can easily customize the processing of messages of streaming protocols such as SSE (Server-Sent Events).
In large-bandwidth scenarios such as AI services, memory overhead can be significantly reduced.
Easy to expand
Provides a rich official plug-in library, covering common functions such as AI, traffic management, security protection, etc., meeting the needs of more than 90% of business scenarios.
It focuses on Wasm plug-in extensions, ensures memory security through sandbox isolation, supports multiple programming languages, allows plug-in versions to be upgraded independently, and achieves traffic-free hot update of gateway logic.
Safe and easy to use
Based on the Ingress API and Gateway API standards, it provides an out-of-box UI console, WAF protection plug-in, and IP/Cookie CC protection plug-in.
It supports docking with Let's Encrypt to automatically issue and renew free certificates, and can be deployed without K8s. It can be started with one line of Docker command, making it convenient for individual developers to use.

Function display

AI Gateway Demo Display

From OpenAI to other large models, migration is completed in 30 seconds

Higress UI console

Rich observables
Provides out-of-the-box observability. Grafana & Prometheus can use built-in ones or connect to self-built ones.
Plug-in extension mechanism
The official provides a variety of plug-ins. Users can also develop their own plug-ins, build them into docker/oci images and configure them on the console. The plug-in logic can be changed in real time without any damage to the traffic.
Multiple service discovery
K8s Service service discovery is provided by default. Through configuration, it can be connected to registration centers such as Nacos/ZooKeeper to realize service discovery, or it can be discovered based on static IP or DNS.
Domain name and certificate
You can create and manage TLS certificates and configure the HTTP/HTTPS behavior of the domain name. The domain name policy supports plugins that take effect on specific domain names.
Rich routing capabilities
Through the service discovery mechanism defined above, the discovered services will appear in the service list; when creating a route, select the domain name, define the route matching mechanism, and then select the target service for routing; the routing policy supports plug-ins that take effect on specific routes.

Community

grateful

Higress would not be possible without the open source work of Envoy and Istio, and I would like to express my sincerest respect to these two projects.