kafka proxy
v0.3.12
El proxy Kafka se basa en la idea del proxy SQL de la nube. Permite que un servicio se conecte a los corredores de Kafka sin tener que lidiar con la autenticación SASL/simple y los certificados SSL.
Funciona abriendo enchufes TCP en la máquina local y las conexiones de proxy a los corredores Kafka asociados cuando se usan los enchufes. El host y el puerto en las respuestas de metadatos y findcoordinator recibidas de los corredores son reemplazados por homólogos locales. Para los corredores descubiertos (no configurados como los servidores Boostrap), los oyentes locales se inician en puertos aleatorios. La función dinámica de oyentes locales se puede deshabilitar y se puede proporcionar una lista adicional de asignaciones de servidores externas.
El proxy puede finalizar el tráfico TLS y autenticar a los usuarios utilizando SASL/PLAIN. El método de verificación de credenciales es configurable y utiliza el sistema de complementos Golang a través de RPC.
Los proxies también pueden autenticarse entre sí utilizando un método enchufable que es transparente a otros servidores y clientes de Kafka. Actualmente, el token de ID de Google para cuentas de servicio se implementa, es decir, solicitud de cliente proxy y envía una cuenta de servicio JWT y Proxy Server recibe y lo valida con Google JWKS.
Las llamadas de la API de Kafka se pueden restringir para evitar algunas operaciones, por ejemplo, la eliminación del tema o las solicitudes de producir.
Ver:
Proxy de Kafka con Amazon MSK
Una guía del protocolo Kafka
Guía del protocolo Kafka
La siguiente tabla proporciona una descripción general de las versiones Kafka compatibles (especificadas una y todas las versiones de Kafka anteriores). Como no todos los lanzamientos de Kafka agregan nuevos mensajes/versiones que son relevantes para el proxy de Kafka, las versiones más nuevas de Kafka también pueden funcionar.
| Versión de Kafka Proxy | Versión de Kafka |
|---|---|
| de 0.11.0 | |
| 0.2.9 | a 2.8.0 |
| 0.3.1 | a 3.4.0 |
| 0.3.11 | a 3.7.0 |
| 0.3.12 | a 3.9.0 |
Descargue el último lanzamiento
Linux
curl -Ls https://github.com/grepplabs/kafka-proxy/releases/download/v0.3.12/kafka-proxy-v0.3.12-linux-amd64.tar.gz | tar xz
macosa
curl -Ls https://github.com/grepplabs/kafka-proxy/releases/download/v0.3.12/kafka-proxy-v0.3.12-darwin-amd64.tar.gz | tar xz
Mueva el binario a su camino.
sudo mv ./kafka-proxy /usr/local/bin/kafka-proxy
make clean build
Las imágenes de Docker están disponibles en Docker Hub.
Puede iniciar un contenedor Kafka-Proxy para probarlo con
docker run --rm -p 30001-30003:30001-30003 grepplabs/kafka-proxy:0.3.12 server --bootstrap-server-mapping "localhost:19092,0.0.0.0:30001" --bootstrap-server-mapping "localhost:29092,0.0.0.0:30002" --bootstrap-server-mapping "localhost:39092,0.0.0.0:30003" --dial-address-mapping "localhost:19092,172.17.0.1:19092" --dial-address-mapping "localhost:29092,172.17.0.1:29092" --dial-address-mapping "localhost:39092,172.17.0.1:39092" --debug-enable
Kafka-Proxy ahora será accesible en localhost:30001 , localhost:30002 y localhost:30003 , conectándose a los corredores de Kafka que se ejecutan en Docker (Network Bridge Gateway 172.17.0.1 ) Publicidad de oyentes de texto sin formato en localhost:19092 , localhost:29092 y localhost:39092 .
Las imágenes de Docker con complementos precompilados ubicados en /opt/kafka-proxy/bin/ se etiquetan con <release>-all .
Puede iniciar un contenedor Kafka-Proxy con un complemento Auth-LDAP para probarlo con
docker run --rm -p 30001-30003:30001-30003 grepplabs/kafka-proxy:0.3.12-all server --bootstrap-server-mapping "localhost:19092,0.0.0.0:30001" --bootstrap-server-mapping "localhost:29092,0.0.0.0:30002" --bootstrap-server-mapping "localhost:39092,0.0.0.0:30003" --dial-address-mapping "localhost:19092,172.17.0.1:19092" --dial-address-mapping "localhost:29092,172.17.0.1:29092" --dial-address-mapping "localhost:39092,172.17.0.1:39092" --debug-enable --auth-local-enable --auth-local-command=/opt/kafka-proxy/bin/auth-ldap --auth-local-param=--url=ldap://172.17.0.1:389 --auth-local-param=--start-tls=false --auth-local-param=--bind-dn=cn=admin,dc=example,dc=org --auth-local-param=--bind-passwd=admin --auth-local-param=--user-search-base=ou=people,dc=example,dc=org --auth-local-param=--user-filter="(&(objectClass=person)(uid=%u)(memberOf=cn=kafka-users,ou=realm-roles,dc=example,dc=org))"
Run the kafka-proxy server
Usage:
kafka-proxy server [flags]
Flags:
--auth-gateway-client-command string Path to authentication plugin binary
--auth-gateway-client-enable Enable gateway client authentication
--auth-gateway-client-log-level string Log level of the auth plugin (default "trace")
--auth-gateway-client-magic uint Magic bytes sent in the handshake
--auth-gateway-client-method string Authentication method
--auth-gateway-client-param stringArray Authentication plugin parameter
--auth-gateway-client-timeout duration Authentication timeout (default 10s)
--auth-gateway-server-command string Path to authentication plugin binary
--auth-gateway-server-enable Enable proxy server authentication
--auth-gateway-server-log-level string Log level of the auth plugin (default "trace")
--auth-gateway-server-magic uint Magic bytes sent in the handshake
--auth-gateway-server-method string Authentication method
--auth-gateway-server-param stringArray Authentication plugin parameter
--auth-gateway-server-timeout duration Authentication timeout (default 10s)
--auth-local-command string Path to authentication plugin binary
--auth-local-enable Enable local SASL/PLAIN authentication performed by listener - SASL handshake will not be passed to kafka brokers
--auth-local-log-level string Log level of the auth plugin (default "trace")
--auth-local-mechanism string SASL mechanism used for local authentication: PLAIN or OAUTHBEARER (default "PLAIN")
--auth-local-param stringArray Authentication plugin parameter
--auth-local-timeout duration Authentication timeout (default 10s)
--bootstrap-server-mapping stringArray Mapping of Kafka bootstrap server address to local address (host:port,host:port(,advhost:advport))
--debug-enable Enable Debug endpoint
--debug-listen-address string Debug listen address (default "0.0.0.0:6060")
--default-listener-ip string Default listener IP (default "0.0.0.0")
--dial-address-mapping stringArray Mapping of target broker address to new one (host:port,host:port). The mapping is performed during connection establishment
--dynamic-advertised-listener string Advertised address for dynamic listeners. If empty, default-listener-ip is used
--dynamic-listeners-disable Disable dynamic listeners.
--dynamic-sequential-min-port int If set to non-zero, makes the dynamic listener use a sequential port starting with this value rather than a random port every time.
--external-server-mapping stringArray Mapping of Kafka server address to external address (host:port,host:port). A listener for the external address is not started
--forbidden-api-keys ints Forbidden Kafka request types. The restriction should prevent some Kafka operations e.g. 20 - DeleteTopics
--forward-proxy string URL of the forward proxy. Supported schemas are socks5 and http
--gssapi-auth-type string GSSAPI auth type: KEYTAB or USER (default "KEYTAB")
--gssapi-disable-pa-fx-fast Used to configure the client to not use PA_FX_FAST.
--gssapi-keytab string krb5.keytab file location
--gssapi-krb5 string krb5.conf file path, default: /etc/krb5.conf (default "/etc/krb5.conf")
--gssapi-password string Password for auth type USER
--gssapi-realm string Realm
--gssapi-servicename string ServiceName (default "kafka")
--gssapi-spn-host-mapping stringToString Mapping of Kafka servers address to SPN hosts (default [])
--gssapi-username string Username (default "kafka")
-h, --help help for server
--http-disable Disable HTTP endpoints
--http-health-path string Path on which to health endpoint (default "/health")
--http-listen-address string Address that kafka-proxy is listening on (default "0.0.0.0:9080")
--http-metrics-path string Path on which to expose metrics (default "/metrics")
--kafka-client-id string An optional identifier to track the source of requests (default "kafka-proxy")
--kafka-connection-read-buffer-size int Size of the operating system's receive buffer associated with the connection. If zero, system default is used
--kafka-connection-write-buffer-size int Sets the size of the operating system's transmit buffer associated with the connection. If zero, system default is used
--kafka-dial-timeout duration How long to wait for the initial connection (default 15s)
--kafka-keep-alive duration Keep alive period for an active network connection. If zero, keep-alives are disabled (default 1m0s)
--kafka-max-open-requests int Maximal number of open requests pro tcp connection before sending on it blocks (default 256)
--kafka-read-timeout duration How long to wait for a response (default 30s)
--kafka-write-timeout duration How long to wait for a transmit (default 30s)
--log-format string Log format text or json (default "text")
--log-level string Log level debug, info, warning, error, fatal or panic (default "info")
--log-level-fieldname string Log level fieldname for json format (default "@level")
--log-msg-fieldname string Message fieldname for json format (default "@message")
--log-time-fieldname string Time fieldname for json format (default "@timestamp")
--producer-acks-0-disabled Assume fire-and-forget is never sent by the producer. Enabling this parameter will increase performance
--proxy-listener-ca-chain-cert-file string PEM encoded CA's certificate file. If provided, client certificate is required and verified
--proxy-listener-cert-file string PEM encoded file with server certificate
--proxy-listener-cipher-suites strings List of supported cipher suites
--proxy-listener-curve-preferences strings List of curve preferences
--proxy-listener-keep-alive duration Keep alive period for an active network connection. If zero, keep-alives are disabled (default 1m0s)
--proxy-listener-key-file string PEM encoded file with private key for the server certificate
--proxy-listener-key-password string Password to decrypt rsa private key
--proxy-listener-read-buffer-size int Size of the operating system's receive buffer associated with the connection. If zero, system default is used
--proxy-listener-tls-enable Whether or not to use TLS listener
--proxy-listener-tls-required-client-subject strings Required client certificate subject common name; example; s:/CN=[value]/C=[state]/C=[DE,PL] or r:/CN=[^val.{2}$]/C=[state]/C=[DE,PL]; check manual for more details
--proxy-listener-write-buffer-size int Sets the size of the operating system's transmit buffer associated with the connection. If zero, system default is used
--proxy-request-buffer-size int Request buffer size pro tcp connection (default 4096)
--proxy-response-buffer-size int Response buffer size pro tcp connection (default 4096)
--sasl-aws-profile string AWS profile
--sasl-aws-region string Region for AWS IAM Auth
--sasl-enable Connect using SASL
--sasl-jaas-config-file string Location of JAAS config file with SASL username and password
--sasl-method string SASL method to use (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, GSSAPI, AWS_MSK_IAM (default "PLAIN")
--sasl-password string SASL user password
--sasl-plugin-command string Path to authentication plugin binary
--sasl-plugin-enable Use plugin for SASL authentication
--sasl-plugin-log-level string Log level of the auth plugin (default "trace")
--sasl-plugin-mechanism string SASL mechanism used for proxy authentication: PLAIN or OAUTHBEARER (default "OAUTHBEARER")
--sasl-plugin-param stringArray Authentication plugin parameter
--sasl-plugin-timeout duration Authentication timeout (default 10s)
--sasl-username string SASL user name
--tls-ca-chain-cert-file string PEM encoded CA's certificate file
--tls-client-cert-file string PEM encoded file with client certificate
--tls-client-key-file string PEM encoded file with private key for the client certificate
--tls-client-key-password string Password to decrypt rsa private key
--tls-enable Whether or not to use TLS when connecting to the broker
--tls-insecure-skip-verify It controls whether a client verifies the server's certificate chain and host name
--tls-same-client-cert-enable Use only when mutual TLS is enabled on proxy and broker. It controls whether a proxy validates if proxy client certificate exactly matches brokers client cert (tls-client-cert-file)kafka-proxy server --bootstrap-server-mapping "192.168.99.100:32400,0.0.0.0:32399" kafka-proxy server --bootstrap-server-mapping "192.168.99.100:32400,127.0.0.1:32400" --bootstrap-server-mapping "192.168.99.100:32401,127.0.0.1:32401" --bootstrap-server-mapping "192.168.99.100:32402,127.0.0.1:32402" --dynamic-listeners-disable kafka-proxy server --bootstrap-server-mapping "kafka-0.example.com:9092,0.0.0.0:32401,kafka-0.grepplabs.com:9092" --bootstrap-server-mapping "kafka-1.example.com:9092,0.0.0.0:32402,kafka-1.grepplabs.com:9092" --bootstrap-server-mapping "kafka-2.example.com:9092,0.0.0.0:32403,kafka-2.grepplabs.com:9092" --dynamic-listeners-disable kafka-proxy server --bootstrap-server-mapping "192.168.99.100:32400,127.0.0.1:32400" --external-server-mapping "192.168.99.100:32401,127.0.0.1:32402" --external-server-mapping "192.168.99.100:32402,127.0.0.1:32403" --forbidden-api-keys 20 export BOOTSTRAP_SERVER_MAPPING="192.168.99.100:32401,0.0.0.0:32402 192.168.99.100:32402,0.0.0.0:32403" && kafka-proxy server
kafka-proxy server --bootstrap-server-mapping "localhost:19092,0.0.0.0:30001,localhost:30001" --bootstrap-server-mapping "localhost:29092,0.0.0.0:30002,localhost:30002" --bootstrap-server-mapping "localhost:39092,0.0.0.0:30003,localhost:30003" --proxy-listener-cert-file "tls/ca-cert.pem" --proxy-listener-key-file "tls/ca-key.pem" --proxy-listener-tls-enable --proxy-listener-cipher-suites TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256
La autenticación SASL es iniciada por el proxy. La autenticación SASL está deshabilitada en los clientes y está habilitada en los corredores Kafka.
kafka-proxy server --bootstrap-server-mapping "kafka-0.grepplabs.com:9093,0.0.0.0:32399" --tls-enable --tls-insecure-skip-verify --sasl-enable --sasl-username myuser --sasl-password mysecret kafka-proxy server --bootstrap-server-mapping "kafka-0.example.com:9092,0.0.0.0:30001" --bootstrap-server-mapping "kafka-1.example.com:9092,0.0.0.0:30002" --bootstrap-server-mapping "kafka-1.example.com:9093,0.0.0.0:30003" --sasl-enable --sasl-username "alice" --sasl-password "alice-secret" --sasl-method "SCRAM-SHA-512" --log-level debug make clean build plugin.unsecured-jwt-provider && build/kafka-proxy server --sasl-enable --sasl-plugin-enable --sasl-plugin-mechanism "OAUTHBEARER" --sasl-plugin-command build/unsecured-jwt-provider --sasl-plugin-param "--claim-sub=alice" --bootstrap-server-mapping "192.168.99.100:32400,127.0.0.1:32400"
Autenticación de GSSAPI / KERBEROS
kafka-proxy server --bootstrap-server-mapping "kafka-0.grepplabs.com:9092,127.0.0.1:32500" --bootstrap-server-mapping "kafka-1.grepplabs.com:9092,127.0.0.1:32501" --bootstrap-server-mapping "kafka-2.grepplabs.com:9092,127.0.0.1:32502" --sasl-enable --sasl-method "GSSAPI" --gssapi-servicename kafka --gssapi-username kafkaclient1 --gssapi-realm EXAMPLE.COM --gssapi-krb5 /etc/krb5.conf --gssapi-keytab /etc/security/keytabs/kafka.keytab
AWS MSK IAM
kafka-proxy server --bootstrap-server-mapping "b-1-public.kafkaproxycluster.uls9ao.c4.kafka.eu-central-1.amazonaws.com:9198,0.0.0.0:30001" --bootstrap-server-mapping "b-2-public.kafkaproxycluster.uls9ao.c4.kafka.eu-central-1.amazonaws.com:9198,0.0.0.0:30002" --bootstrap-server-mapping "b-3-public.kafkaproxycluster.uls9ao.c4.kafka.eu-central-1.amazonaws.com:9198,0.0.0.0:30003" --tls-enable --tls-insecure-skip-verify --sasl-enable --sasl-method "AWS_MSK_IAM" --sasl-aws-region "eu-central-1" --log-level debug
La autenticación SASL es realizada por el proxy. La autenticación SASL está habilitada en los clientes y se deshabilita en los corredores Kafka.
make clean build plugin.auth-user && build/kafka-proxy server --proxy-listener-key-file "server-key.pem" --proxy-listener-cert-file "server-cert.pem" --proxy-listener-ca-chain-cert-file "ca.pem" --proxy-listener-tls-enable --auth-local-enable --auth-local-command build/auth-user --auth-local-param "--username=my-test-user" --auth-local-param "--password=my-test-password" make clean build plugin.auth-ldap && build/kafka-proxy server --auth-local-enable --auth-local-command build/auth-ldap --auth-local-param "--url=ldaps://ldap.example.com:636" --auth-local-param "--user-dn=cn=users,dc=exemple,dc=com" --auth-local-param "--user-attr=uid" --bootstrap-server-mapping "192.168.99.100:32400,127.0.0.1:32400" make clean build plugin.unsecured-jwt-info && build/kafka-proxy server --auth-local-enable --auth-local-command build/unsecured-jwt-info --auth-local-mechanism "OAUTHBEARER" --auth-local-param "--claim-sub=alice" --auth-local-param "--claim-sub=bob" --bootstrap-server-mapping "192.168.99.100:32400,127.0.0.1:32400"
Validar que el certificado del cliente utilizado por Proxy Client es exactamente el mismo que el Certificado del Cliente en autenticación iniciado por Proxy
kafka-proxy server --bootstrap-server-mapping "kafka-0.grepplabs.com:9093,0.0.0.0:32399" --tls-enable --tls-client-cert-file client.crt --tls-client-key-file client.pem --tls-client-key-password changeit --proxy-listener-tls-enable --proxy-listener-key-file server.pem --proxy-listener-cert-file server.crt --proxy-listener-key-password changeit --proxy-listener-ca-chain-cert-file ca.crt --tls-same-client-cert-enable
Autenticación entre Kafka Proxy Client y Kafka Proxy Server con Google-ID (cuenta de servicio JWT)
kafka-proxy server --bootstrap-server-mapping "kafka-0.grepplabs.com:9092,127.0.0.1:32500" --bootstrap-server-mapping "kafka-1.grepplabs.com:9092,127.0.0.1:32501" --bootstrap-server-mapping "kafka-2.grepplabs.com:9092,127.0.0.1:32502" --dynamic-listeners-disable --http-disable --proxy-listener-tls-enable --proxy-listener-cert-file=/var/run/secret/server.cert.pem --proxy-listener-key-file=/var/run/secret/server.key.pem --auth-gateway-server-enable --auth-gateway-server-method google-id --auth-gateway-server-magic 3285573610483682037 --auth-gateway-server-command google-id-info --auth-gateway-server-param "--timeout=10" --auth-gateway-server-param "--audience=tcp://kafka-gateway.grepplabs.com" --auth-gateway-server-param "--email-regex=^[email protected]$" kafka-proxy server --bootstrap-server-mapping "127.0.0.1:32500,127.0.0.1:32400" --bootstrap-server-mapping "127.0.0.1:32501,127.0.0.1:32401" --bootstrap-server-mapping "127.0.0.1:32502,127.0.0.1:32402" --dynamic-listeners-disable --http-disable --tls-enable --tls-ca-chain-cert-file /var/run/secret/client/ca-chain.cert.pem --auth-gateway-client-enable --auth-gateway-client-method google-id --auth-gateway-client-magic 3285573610483682037 --auth-gateway-client-command google-id-provider --auth-gateway-client-param "--credentials-file=/var/run/secret/client/service-account.json" --auth-gateway-client-param "--target-audience=tcp://kafka-gateway.grepplabs.com" --auth-gateway-client-param "--timeout=10"
Conecte a través del servidor de proxy de prueba de Test Socks5
kafka-proxy tools socks5-proxy --addr localhost:1080 kafka-proxy server --bootstrap-server-mapping "kafka-0.grepplabs.com:9092,127.0.0.1:32500" --bootstrap-server-mapping "kafka-1.grepplabs.com:9092,127.0.0.1:32501" --bootstrap-server-mapping "kafka-2.grepplabs.com:9092,127.0.0.1:32502" --forward-proxy socks5://localhost:1080
kafka-proxy tools socks5-proxy --addr localhost:1080 --username my-proxy-user --password my-proxy-password kafka-proxy server --bootstrap-server-mapping "kafka-0.grepplabs.com:9092,127.0.0.1:32500" --bootstrap-server-mapping "kafka-1.grepplabs.com:9092,127.0.0.1:32501" --bootstrap-server-mapping "kafka-2.grepplabs.com:9092,127.0.0.1:32502" --forward-proxy socks5://my-proxy-user:my-proxy-password@localhost:1080
Conecte a través de la prueba del servidor proxy HTTP utilizando el método de conexión
kafka-proxy tools http-proxy --addr localhost:3128 kafka-proxy server --bootstrap-server-mapping "kafka-0.grepplabs.com:9092,127.0.0.1:32500" --bootstrap-server-mapping "kafka-1.grepplabs.com:9092,127.0.0.1:32501" --bootstrap-server-mapping "kafka-2.grepplabs.com:9092,127.0.0.1:32502" --forward-proxy http://localhost:3128
kafka-proxy tools http-proxy --addr localhost:3128 --username my-proxy-user --password my-proxy-password kafka-proxy server --bootstrap-server-mapping "kafka-0.grepplabs.com:9092,127.0.0.1:32500" --bootstrap-server-mapping "kafka-1.grepplabs.com:9092,127.0.0.1:32501" --bootstrap-server-mapping "kafka-2.grepplabs.com:9092,127.0.0.1:32502" --forward-proxy http://my-proxy-user:my-proxy-password@localhost:3128
A veces puede ser necesario no solo validar que el certificado del cliente sea válido, sino también que el Certificado del Cliente DN se emite para un caso de uso concreto. Esto se puede lograr utilizando el siguiente conjunto de argumentos:
--proxy-listener-tls-client-cert-validate-subject bool Whether to validate client certificate subject (default false) --proxy-listener-tls-required-client-subject-common-name string Required client certificate subject common name --proxy-listener-tls-required-client-subject-country stringArray Required client certificate subject country --proxy-listener-tls-required-client-subject-province stringArray Required client certificate subject province --proxy-listener-tls-required-client-subject-locality stringArray Required client certificate subject locality --proxy-listener-tls-required-client-subject-organization stringArray Required client certificate subject organization --proxy-listener-tls-required-client-subject-organizational-unit stringArray Required client certificate subject organizational unit
Al establecer --proxy-listener-tls-client-cert-validate-subject true , Kafka proxy inspeccionará los campos de DN del certificado del cliente para los valores esperados establecidos con los argumentos de --proxy-listener-tls-required-client-* . Las coincidencias siempre son exactas y usadas juntas, para todos los valores no vacíos. Por ejemplo, para permitir un certificado válido para country=DE y organization=grepplabs , configure Kafka Proxy de la siguiente manera:
kafka-proxy server --proxy-listener-tls-client-cert-validate-subject true --proxy-listener-tls-required-client-subject-country DE --proxy-listener-tls-required-client-subject-organization grepplabs
--- APiversión: APPS/V1KIND: ImplementmentMetadata: Nombre: MyAppSpec: Replicas: 1
Selector: MatchLabels: App: MyApp
plantilla: metadatos: etiquetas: aplicación: myapp anotaciones: prometheus.io/scrape: 'true'pec: contenedores:
-Nombre: Kafka-Proxy Imagen: Grepplabs/Kafka-Proxy: Últimos args:
-'server'-'--log-format = json'- '--bootstrap-server-mapping = kafka-0: 9093,127.0.0.1: 32400'-' --bootstrap-server-mapping = kafka-1: 9093,127.0.0.1: 32401'- '--bootstrap-server-mapping = kafka-2: 9093,127.0.0.1: 32402'-'--tls-enable'- '--tls-c-chain-cert- file =/var/run/secret/kafka-ca-chain-certificate/ca-chain.cert.pem'- '--tls-client-cert-file =/var/run/secret/kafka-client-certificate/ client.cert.pem'- '--tls-client-keykil =/var/run/secret/kafka-client-key/client.key.pem'-'--tls-client-key-password = $ (Tls_client_key_password) '-' --sasl-enable'- '--sasl-jaas-config-file =/var/run/secret/kafka-client-jaas/jaas.config' env:
-Nombre: tls_client_key_passwordValueFrom: SecretKeyRef: Nombre: TLS-Client-Key-PasswordKey: Contraseña VolumeMounts:
-Nombre: "Sasl-Jaas-Config-File" MountPath: "/var/run/secret/kafka-client-jaas"-Nombre: "TLS-CA-CHAIN-Certificate" MountPath: "/var/run/secret/ Kafka-Ca-Chain-Certificate "-Nombre:" TLS-Client-Cert-File "MountPath:"/var/run/secret/kafka-client-certificate "-nombre:" TLS-client-key-file "MountPath: "/var/run/secret/kafka-client-key":
- Nombre: MetricsContainerPort: 9080 LivityProbe: Httpget: Path: /Health Port: 9080InitialDelaysEseconds: 5PeriodSeconds: 3 Preparación: Httpget: : MyApp Image: MyApp: Últimos puertos:
- Containerport: 8080 Name: Métricos Env:
- Nombre: Bootstrap_ServersValue: "127.0.0.1:32400,127.0.0.1:32401,127.0.0.1:32402" Volúmenes:
-Nombre: Sasl-Jaas-Config-FilesCret: SecretName: Sasl-Jaas-Config-File-Nombre: TLS-CA-CHAIN-CertificatesCret: SecretName: TLS-CA-CHAIN-Certificate-Nombre: TLS-Cliente-Cert-FilleSecretretterette : SecretName: TLS-Client-Cert-File-Nombre: TLS-Client-Key-FilesCret: SecretName: TLS-Client-Key-File --- APiversión: APPS/V1KIND: StatefulSetMetadata: Nombre: Kafka-ProxySpec: Selector: MatchLabels: App: Kafka-Proxy
Replicas: 1
nombre de servicio: kafka-proxy
plantilla: metadatos: etiquetas: aplicación: kafka-proxyspec: contenedores:
-Nombre: Kafka-Proxy Imagen: Grepplabs/Kafka-Proxy: Últimos args:
-'server'-'--log-format = json'- '--bootstrap-server-mapping = kafka-0: 9093,127.0.0.1: 32400'-' --bootstrap-server-mapping = kafka-1: 9093,127.0.0.1: 32401'- '--bootstrap-server-mapping = kafka-2: 9093,127.0.0.1: 32402'-'--tls-enable'- '--tls-c-chain-cert- file =/var/run/secret/kafka-ca-chain-certificate/ca-chain.cert.pem'- '--tls-client-cert-file =/var/run/secret/kafka-client-certificate/ client.cert.pem'- '--tls-client-keykil =/var/run/secret/kafka-client-key/client.key.pem'-'--tls-client-key-password = $ (Tls_client_key_password) '-' --sasl-enable'- '--sasl-jaas-config-file =/var/run/secret/kafka-client-jaas/jaas.config'-' --proxy-request-buffer -size = 32768'- '--Proxy-Resesponse-Buffer-size = 32768'-' --Proxy-Listener-read-buffer-size = 32768'- '--Proxy-Listener-write-buffer-size = 1310772 '-' --Kafka-Connection-read-buffer-size = 131072'- '--Kafka-Connection-Write-Buffer-size = 32768' Env:
-Nombre: tls_client_key_passwordValueFrom: SecretKeyRef: Nombre: TLS-Client-Key-PasswordKey: Contraseña VolumeMounts:
-Nombre: "Sasl-Jaas-Config-File" MountPath: "/var/run/secret/kafka-client-jaas"-Nombre: "TLS-CA-CHAIN-Certificate" MountPath: "/var/run/secret/ Kafka-Ca-Chain-Certificate "-Nombre:" TLS-Client-Cert-File "MountPath:"/var/run/secret/kafka-client-certificate "-nombre:" TLS-client-key-file "MountPath: "/var/run/secret/kafka-client-key":
- Nombre: MetricsContainerPort: 9080 - Nombre: Kafka -0ContainerPort: 32400 - Nombre: Kafka -1ContainerPort: 32401 - Nombre: Kafka -2ContainerPort: 32402 LivityProbe: Httpget: Path: /Health: 9080InitiaLaysEdsEds : /Puerto de salud: 9080initialDelayseconds: 5 Periodseconds: 10TimeOutSeconds: 5SuccessThreshold: 2failurethreshold: 5 Recursos: Solicitudes: Memoria: 128MI CPU: 1000M Reiniciario: Siempre volúmenes:
-Nombre: Sasl-Jaas-Config-FilesCret: SecretName: Sasl-Jaas-Config-File-Nombre: TLS-CA-CHAIN-CertificatesCret: SecretName: TLS-CA-CHAIN-Certificate-Nombre: TLS-Cliente-Cert-FilleSecretretterette : SecretName: TLS-Client-Cert-File-Nombre: TLS-Client-Key-FilesCret: SecretName: TLS-Client-Key-FileKubectl Port-Forward Kafka-Proxy-0 32400: 32400 32401: 32401 32402: 32402
Use localhost: 32400, localhost: 32401 y localhost: 32402 como servidores de arranque
kafka.properties
broker.id=0 advertised.listeners=PLAINTEXT://kafka-0.kafka-headless.kafka:9092 ...
Kubectl Port-Forward -N Kafka Kafka-0 9092: 9092
Kafka-Proxy Server --Bootstrap-server-mapping "127.0.0.1:9092,0.0.0.0:19092"--dial-adress-mapping "Kafka-0.kafka-headless.kafka: 9092,0.0.0.0: 9092"
Use Localhost: 19092 como servidores de bootstrap
strimzi 0.13.0 CRD
Apiversión: kafka.strimzi.io/v1beta1kind: kafkametadata: nombre: clúster de prueba
espacio de nombres: kafkaspec: kafka: versión: 2.3.0replicas: 3listeners: Plain: {} tls: {} config: offsets.topic.replication.factor: 3 transacciones.state.log.replication.factor: 3 transacciones.state.log. Min.Isr: 2 num.Partitions: 60 predeterminado.
- ID: 0 Tipo: Persistente Tamaño de la reclamación: 20GI Deleteclamar: Verdadero
Zookeeper: REPLICAS: 3Storaz: Tipo: Persistente Tamaño de reclamos: 5GI DeleteClaim: Verdadero
EntityOperator: TopicOperator: {} UserOperator: {}Kubectl Port-Forward -N Kafka Test-Cluster-Kafka-0 9092: 9092 Kubectl Port-Forward -N Kafka Test-Cluster-Kafka-1 9093: 9092 Kubectl Port-Forward -N Kafka Test-Cluster-Kafka-2 9094: 9092 Kafka-Proxy Server-Debug a nivel de log --Bootstrap-server-mapping "127.0.0.1:9092,0.0.0.0:19092" --bootstrap-server mapping "127.0.0.1:9093,0.0.0.0:19093" --Bootstrap-server-mapping "127.0.0.1:9094,0.0.0.0:19094" --dial-address-mapping "test-cluster-kafka-0.test cluster-kafka-brokers.kafka.svc.cluster.local: 9092,0.0.0.0: 9092" --Dial-Address-Mapping "Test-cluster-kafka-1.test-cluster-kafka-brokers.kafka.svc.cluster.local: 9092,0.0.0.0: 9093" --Dial-Address-Mapping "Test-cluster-kafka-2.test cluster-kafka-brokers.kafka.svc.cluster.local: 9092,0.0.0.0: 9094"
Use Localhost: 19092 como servidores de bootstrap
Proxy de nube SQL
Sarama
