kafka proxy
v0.3.12
Le proxy Kafka est basé sur l'idée du proxy Cloud SQL. Il permet à un service de se connecter aux courtiers Kafka sans avoir à traiter avec des certificats SASL / PLAIN Authentification et SSL.
Il fonctionne en ouvrant des sockets TCP sur la machine locale et en proxyant les connexions aux courtiers Kafka associés lorsque les prises sont utilisées. L'hôte et le port dans les métadonnées et les réponses FindCoordinator reçus des courtiers sont remplacés par des homologues locaux. Pour les courtiers découverts (non configurés comme serveurs Boostrap), les écouteurs locaux sont démarrés sur des ports aléatoires. La fonctionnalité dynamique des auditeurs locaux peut être désactivée et une liste supplémentaire de mappages de serveurs externes peut être fourni.
Le proxy peut résilier le trafic TLS et authentifier les utilisateurs à l'aide de SASL / PLAIN. La méthode de vérification des informations d'identification est configurable et utilise le système de plugin Golang sur RPC.
Les proxys peuvent également s'authentifier mutuellement en utilisant une méthode enfichable qui est transparente pour d'autres serveurs et clients Kafka. Actuellement, le jeton Google ID pour les comptes de service est implémenté les demandes du client IE IE et envoie le compte de service JWT et Proxy Server reçoit et le valide contre Google JWKS.
Les appels de l'API Kafka peuvent être limités pour empêcher certaines opérations, par exemple, la suppression ou les demandes de produits.
Voir:
Kafka Proxy avec Amazon MSK
Un guide du protocole Kafka
Guide du protocole Kafka
Le tableau suivant donne une vue d'ensemble des versions Kafka prises en charge (spécifiées une et toutes les versions Kafka précédentes). Comme toutes les versions de Kafka n'ajoutent pas de nouveaux messages / versions qui sont pertinents pour le proxy Kafka, les versions Kafka plus récentes peuvent également fonctionner.
| Version proxy kafka | Version kafka |
|---|---|
| à partir de 0,11,0 | |
| 0.2.9 | à 2.8.0 |
| 0.3.1 | à 3.4.0 |
| 0.3.11 | à 3.7.0 |
| 0.3.12 | à 3.9.0 |
Télécharger la dernière version
Linux
curl -Ls https://github.com/grepplabs/kafka-proxy/releases/download/v0.3.12/kafka-proxy-v0.3.12-linux-amd64.tar.gz | tar xz
macos
curl -Ls https://github.com/grepplabs/kafka-proxy/releases/download/v0.3.12/kafka-proxy-v0.3.12-darwin-amd64.tar.gz | tar xz
Déplacez le binaire sur votre chemin.
sudo mv ./kafka-proxy /usr/local/bin/kafka-proxy
make clean build
Les images Docker sont disponibles sur Docker Hub.
Vous pouvez lancer un conteneur kafka-proxy pour l'essayer avec
docker run --rm -p 30001-30003:30001-30003 grepplabs/kafka-proxy:0.3.12 server --bootstrap-server-mapping "localhost:19092,0.0.0.0:30001" --bootstrap-server-mapping "localhost:29092,0.0.0.0:30002" --bootstrap-server-mapping "localhost:39092,0.0.0.0:30003" --dial-address-mapping "localhost:19092,172.17.0.1:19092" --dial-address-mapping "localhost:29092,172.17.0.1:29092" --dial-address-mapping "localhost:39092,172.17.0.1:39092" --debug-enable
Kafka-Proxy sera désormais accessible sur localhost:30001 , localhost:30002 et localhost:30003 , se connectant aux courtiers de Kafka qui exécutent Docker (réseau Bridge Network 172.17.0.1 ) Publicité des auditeurs en clair sur localhost:19092 , localhost:29092 et localhost:39092 .
Les images Docker avec des plugins précompilés situés dans /opt/kafka-proxy/bin/ sont tagués avec <release>-all .
Vous pouvez lancer un conteneur Kafka-Proxy avec un plugin Auth-LDAP pour l'essayer avec
docker run --rm -p 30001-30003:30001-30003 grepplabs/kafka-proxy:0.3.12-all server --bootstrap-server-mapping "localhost:19092,0.0.0.0:30001" --bootstrap-server-mapping "localhost:29092,0.0.0.0:30002" --bootstrap-server-mapping "localhost:39092,0.0.0.0:30003" --dial-address-mapping "localhost:19092,172.17.0.1:19092" --dial-address-mapping "localhost:29092,172.17.0.1:29092" --dial-address-mapping "localhost:39092,172.17.0.1:39092" --debug-enable --auth-local-enable --auth-local-command=/opt/kafka-proxy/bin/auth-ldap --auth-local-param=--url=ldap://172.17.0.1:389 --auth-local-param=--start-tls=false --auth-local-param=--bind-dn=cn=admin,dc=example,dc=org --auth-local-param=--bind-passwd=admin --auth-local-param=--user-search-base=ou=people,dc=example,dc=org --auth-local-param=--user-filter="(&(objectClass=person)(uid=%u)(memberOf=cn=kafka-users,ou=realm-roles,dc=example,dc=org))"
Run the kafka-proxy server
Usage:
kafka-proxy server [flags]
Flags:
--auth-gateway-client-command string Path to authentication plugin binary
--auth-gateway-client-enable Enable gateway client authentication
--auth-gateway-client-log-level string Log level of the auth plugin (default "trace")
--auth-gateway-client-magic uint Magic bytes sent in the handshake
--auth-gateway-client-method string Authentication method
--auth-gateway-client-param stringArray Authentication plugin parameter
--auth-gateway-client-timeout duration Authentication timeout (default 10s)
--auth-gateway-server-command string Path to authentication plugin binary
--auth-gateway-server-enable Enable proxy server authentication
--auth-gateway-server-log-level string Log level of the auth plugin (default "trace")
--auth-gateway-server-magic uint Magic bytes sent in the handshake
--auth-gateway-server-method string Authentication method
--auth-gateway-server-param stringArray Authentication plugin parameter
--auth-gateway-server-timeout duration Authentication timeout (default 10s)
--auth-local-command string Path to authentication plugin binary
--auth-local-enable Enable local SASL/PLAIN authentication performed by listener - SASL handshake will not be passed to kafka brokers
--auth-local-log-level string Log level of the auth plugin (default "trace")
--auth-local-mechanism string SASL mechanism used for local authentication: PLAIN or OAUTHBEARER (default "PLAIN")
--auth-local-param stringArray Authentication plugin parameter
--auth-local-timeout duration Authentication timeout (default 10s)
--bootstrap-server-mapping stringArray Mapping of Kafka bootstrap server address to local address (host:port,host:port(,advhost:advport))
--debug-enable Enable Debug endpoint
--debug-listen-address string Debug listen address (default "0.0.0.0:6060")
--default-listener-ip string Default listener IP (default "0.0.0.0")
--dial-address-mapping stringArray Mapping of target broker address to new one (host:port,host:port). The mapping is performed during connection establishment
--dynamic-advertised-listener string Advertised address for dynamic listeners. If empty, default-listener-ip is used
--dynamic-listeners-disable Disable dynamic listeners.
--dynamic-sequential-min-port int If set to non-zero, makes the dynamic listener use a sequential port starting with this value rather than a random port every time.
--external-server-mapping stringArray Mapping of Kafka server address to external address (host:port,host:port). A listener for the external address is not started
--forbidden-api-keys ints Forbidden Kafka request types. The restriction should prevent some Kafka operations e.g. 20 - DeleteTopics
--forward-proxy string URL of the forward proxy. Supported schemas are socks5 and http
--gssapi-auth-type string GSSAPI auth type: KEYTAB or USER (default "KEYTAB")
--gssapi-disable-pa-fx-fast Used to configure the client to not use PA_FX_FAST.
--gssapi-keytab string krb5.keytab file location
--gssapi-krb5 string krb5.conf file path, default: /etc/krb5.conf (default "/etc/krb5.conf")
--gssapi-password string Password for auth type USER
--gssapi-realm string Realm
--gssapi-servicename string ServiceName (default "kafka")
--gssapi-spn-host-mapping stringToString Mapping of Kafka servers address to SPN hosts (default [])
--gssapi-username string Username (default "kafka")
-h, --help help for server
--http-disable Disable HTTP endpoints
--http-health-path string Path on which to health endpoint (default "/health")
--http-listen-address string Address that kafka-proxy is listening on (default "0.0.0.0:9080")
--http-metrics-path string Path on which to expose metrics (default "/metrics")
--kafka-client-id string An optional identifier to track the source of requests (default "kafka-proxy")
--kafka-connection-read-buffer-size int Size of the operating system's receive buffer associated with the connection. If zero, system default is used
--kafka-connection-write-buffer-size int Sets the size of the operating system's transmit buffer associated with the connection. If zero, system default is used
--kafka-dial-timeout duration How long to wait for the initial connection (default 15s)
--kafka-keep-alive duration Keep alive period for an active network connection. If zero, keep-alives are disabled (default 1m0s)
--kafka-max-open-requests int Maximal number of open requests pro tcp connection before sending on it blocks (default 256)
--kafka-read-timeout duration How long to wait for a response (default 30s)
--kafka-write-timeout duration How long to wait for a transmit (default 30s)
--log-format string Log format text or json (default "text")
--log-level string Log level debug, info, warning, error, fatal or panic (default "info")
--log-level-fieldname string Log level fieldname for json format (default "@level")
--log-msg-fieldname string Message fieldname for json format (default "@message")
--log-time-fieldname string Time fieldname for json format (default "@timestamp")
--producer-acks-0-disabled Assume fire-and-forget is never sent by the producer. Enabling this parameter will increase performance
--proxy-listener-ca-chain-cert-file string PEM encoded CA's certificate file. If provided, client certificate is required and verified
--proxy-listener-cert-file string PEM encoded file with server certificate
--proxy-listener-cipher-suites strings List of supported cipher suites
--proxy-listener-curve-preferences strings List of curve preferences
--proxy-listener-keep-alive duration Keep alive period for an active network connection. If zero, keep-alives are disabled (default 1m0s)
--proxy-listener-key-file string PEM encoded file with private key for the server certificate
--proxy-listener-key-password string Password to decrypt rsa private key
--proxy-listener-read-buffer-size int Size of the operating system's receive buffer associated with the connection. If zero, system default is used
--proxy-listener-tls-enable Whether or not to use TLS listener
--proxy-listener-tls-required-client-subject strings Required client certificate subject common name; example; s:/CN=[value]/C=[state]/C=[DE,PL] or r:/CN=[^val.{2}$]/C=[state]/C=[DE,PL]; check manual for more details
--proxy-listener-write-buffer-size int Sets the size of the operating system's transmit buffer associated with the connection. If zero, system default is used
--proxy-request-buffer-size int Request buffer size pro tcp connection (default 4096)
--proxy-response-buffer-size int Response buffer size pro tcp connection (default 4096)
--sasl-aws-profile string AWS profile
--sasl-aws-region string Region for AWS IAM Auth
--sasl-enable Connect using SASL
--sasl-jaas-config-file string Location of JAAS config file with SASL username and password
--sasl-method string SASL method to use (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, GSSAPI, AWS_MSK_IAM (default "PLAIN")
--sasl-password string SASL user password
--sasl-plugin-command string Path to authentication plugin binary
--sasl-plugin-enable Use plugin for SASL authentication
--sasl-plugin-log-level string Log level of the auth plugin (default "trace")
--sasl-plugin-mechanism string SASL mechanism used for proxy authentication: PLAIN or OAUTHBEARER (default "OAUTHBEARER")
--sasl-plugin-param stringArray Authentication plugin parameter
--sasl-plugin-timeout duration Authentication timeout (default 10s)
--sasl-username string SASL user name
--tls-ca-chain-cert-file string PEM encoded CA's certificate file
--tls-client-cert-file string PEM encoded file with client certificate
--tls-client-key-file string PEM encoded file with private key for the client certificate
--tls-client-key-password string Password to decrypt rsa private key
--tls-enable Whether or not to use TLS when connecting to the broker
--tls-insecure-skip-verify It controls whether a client verifies the server's certificate chain and host name
--tls-same-client-cert-enable Use only when mutual TLS is enabled on proxy and broker. It controls whether a proxy validates if proxy client certificate exactly matches brokers client cert (tls-client-cert-file)kafka-proxy server --bootstrap-server-mapping "192.168.99.100:32400,0.0.0.0:32399" kafka-proxy server --bootstrap-server-mapping "192.168.99.100:32400,127.0.0.1:32400" --bootstrap-server-mapping "192.168.99.100:32401,127.0.0.1:32401" --bootstrap-server-mapping "192.168.99.100:32402,127.0.0.1:32402" --dynamic-listeners-disable kafka-proxy server --bootstrap-server-mapping "kafka-0.example.com:9092,0.0.0.0:32401,kafka-0.grepplabs.com:9092" --bootstrap-server-mapping "kafka-1.example.com:9092,0.0.0.0:32402,kafka-1.grepplabs.com:9092" --bootstrap-server-mapping "kafka-2.example.com:9092,0.0.0.0:32403,kafka-2.grepplabs.com:9092" --dynamic-listeners-disable kafka-proxy server --bootstrap-server-mapping "192.168.99.100:32400,127.0.0.1:32400" --external-server-mapping "192.168.99.100:32401,127.0.0.1:32402" --external-server-mapping "192.168.99.100:32402,127.0.0.1:32403" --forbidden-api-keys 20 export BOOTSTRAP_SERVER_MAPPING="192.168.99.100:32401,0.0.0.0:32402 192.168.99.100:32402,0.0.0.0:32403" && kafka-proxy server
kafka-proxy server --bootstrap-server-mapping "localhost:19092,0.0.0.0:30001,localhost:30001" --bootstrap-server-mapping "localhost:29092,0.0.0.0:30002,localhost:30002" --bootstrap-server-mapping "localhost:39092,0.0.0.0:30003,localhost:30003" --proxy-listener-cert-file "tls/ca-cert.pem" --proxy-listener-key-file "tls/ca-key.pem" --proxy-listener-tls-enable --proxy-listener-cipher-suites TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256
L'authentification SASL est initiée par le proxy. L'authentification SASL est désactivée sur les clients et activée sur les courtiers Kafka.
kafka-proxy server --bootstrap-server-mapping "kafka-0.grepplabs.com:9093,0.0.0.0:32399" --tls-enable --tls-insecure-skip-verify --sasl-enable --sasl-username myuser --sasl-password mysecret kafka-proxy server --bootstrap-server-mapping "kafka-0.example.com:9092,0.0.0.0:30001" --bootstrap-server-mapping "kafka-1.example.com:9092,0.0.0.0:30002" --bootstrap-server-mapping "kafka-1.example.com:9093,0.0.0.0:30003" --sasl-enable --sasl-username "alice" --sasl-password "alice-secret" --sasl-method "SCRAM-SHA-512" --log-level debug make clean build plugin.unsecured-jwt-provider && build/kafka-proxy server --sasl-enable --sasl-plugin-enable --sasl-plugin-mechanism "OAUTHBEARER" --sasl-plugin-command build/unsecured-jwt-provider --sasl-plugin-param "--claim-sub=alice" --bootstrap-server-mapping "192.168.99.100:32400,127.0.0.1:32400"
Authentification GSSAPI / KERBEROS
kafka-proxy server --bootstrap-server-mapping "kafka-0.grepplabs.com:9092,127.0.0.1:32500" --bootstrap-server-mapping "kafka-1.grepplabs.com:9092,127.0.0.1:32501" --bootstrap-server-mapping "kafka-2.grepplabs.com:9092,127.0.0.1:32502" --sasl-enable --sasl-method "GSSAPI" --gssapi-servicename kafka --gssapi-username kafkaclient1 --gssapi-realm EXAMPLE.COM --gssapi-krb5 /etc/krb5.conf --gssapi-keytab /etc/security/keytabs/kafka.keytab
AWS MSK IAM
kafka-proxy server --bootstrap-server-mapping "b-1-public.kafkaproxycluster.uls9ao.c4.kafka.eu-central-1.amazonaws.com:9198,0.0.0.0:30001" --bootstrap-server-mapping "b-2-public.kafkaproxycluster.uls9ao.c4.kafka.eu-central-1.amazonaws.com:9198,0.0.0.0:30002" --bootstrap-server-mapping "b-3-public.kafkaproxycluster.uls9ao.c4.kafka.eu-central-1.amazonaws.com:9198,0.0.0.0:30003" --tls-enable --tls-insecure-skip-verify --sasl-enable --sasl-method "AWS_MSK_IAM" --sasl-aws-region "eu-central-1" --log-level debug
L'authentification SASL est réalisée par le proxy. L'authentification SASL est activée sur les clients et désactivée sur les courtiers Kafka.
make clean build plugin.auth-user && build/kafka-proxy server --proxy-listener-key-file "server-key.pem" --proxy-listener-cert-file "server-cert.pem" --proxy-listener-ca-chain-cert-file "ca.pem" --proxy-listener-tls-enable --auth-local-enable --auth-local-command build/auth-user --auth-local-param "--username=my-test-user" --auth-local-param "--password=my-test-password" make clean build plugin.auth-ldap && build/kafka-proxy server --auth-local-enable --auth-local-command build/auth-ldap --auth-local-param "--url=ldaps://ldap.example.com:636" --auth-local-param "--user-dn=cn=users,dc=exemple,dc=com" --auth-local-param "--user-attr=uid" --bootstrap-server-mapping "192.168.99.100:32400,127.0.0.1:32400" make clean build plugin.unsecured-jwt-info && build/kafka-proxy server --auth-local-enable --auth-local-command build/unsecured-jwt-info --auth-local-mechanism "OAUTHBEARER" --auth-local-param "--claim-sub=alice" --auth-local-param "--claim-sub=bob" --bootstrap-server-mapping "192.168.99.100:32400,127.0.0.1:32400"
Valider que le certificat client utilisé par le client proxy est exactement le même que le certificat client dans l'authentification initiée par proxy
kafka-proxy server --bootstrap-server-mapping "kafka-0.grepplabs.com:9093,0.0.0.0:32399" --tls-enable --tls-client-cert-file client.crt --tls-client-key-file client.pem --tls-client-key-password changeit --proxy-listener-tls-enable --proxy-listener-key-file server.pem --proxy-listener-cert-file server.crt --proxy-listener-key-password changeit --proxy-listener-ca-chain-cert-file ca.crt --tls-same-client-cert-enable
Authentification entre Kafka Proxy Client et Kafka Proxy Server avec Google-ID (Service Compte JWT)
kafka-proxy server --bootstrap-server-mapping "kafka-0.grepplabs.com:9092,127.0.0.1:32500" --bootstrap-server-mapping "kafka-1.grepplabs.com:9092,127.0.0.1:32501" --bootstrap-server-mapping "kafka-2.grepplabs.com:9092,127.0.0.1:32502" --dynamic-listeners-disable --http-disable --proxy-listener-tls-enable --proxy-listener-cert-file=/var/run/secret/server.cert.pem --proxy-listener-key-file=/var/run/secret/server.key.pem --auth-gateway-server-enable --auth-gateway-server-method google-id --auth-gateway-server-magic 3285573610483682037 --auth-gateway-server-command google-id-info --auth-gateway-server-param "--timeout=10" --auth-gateway-server-param "--audience=tcp://kafka-gateway.grepplabs.com" --auth-gateway-server-param "--email-regex=^[email protected]$" kafka-proxy server --bootstrap-server-mapping "127.0.0.1:32500,127.0.0.1:32400" --bootstrap-server-mapping "127.0.0.1:32501,127.0.0.1:32401" --bootstrap-server-mapping "127.0.0.1:32502,127.0.0.1:32402" --dynamic-listeners-disable --http-disable --tls-enable --tls-ca-chain-cert-file /var/run/secret/client/ca-chain.cert.pem --auth-gateway-client-enable --auth-gateway-client-method google-id --auth-gateway-client-magic 3285573610483682037 --auth-gateway-client-command google-id-provider --auth-gateway-client-param "--credentials-file=/var/run/secret/client/service-account.json" --auth-gateway-client-param "--target-audience=tcp://kafka-gateway.grepplabs.com" --auth-gateway-client-param "--timeout=10"
Connectez-vous via Test Socks5 Proxy Server
kafka-proxy tools socks5-proxy --addr localhost:1080 kafka-proxy server --bootstrap-server-mapping "kafka-0.grepplabs.com:9092,127.0.0.1:32500" --bootstrap-server-mapping "kafka-1.grepplabs.com:9092,127.0.0.1:32501" --bootstrap-server-mapping "kafka-2.grepplabs.com:9092,127.0.0.1:32502" --forward-proxy socks5://localhost:1080
kafka-proxy tools socks5-proxy --addr localhost:1080 --username my-proxy-user --password my-proxy-password kafka-proxy server --bootstrap-server-mapping "kafka-0.grepplabs.com:9092,127.0.0.1:32500" --bootstrap-server-mapping "kafka-1.grepplabs.com:9092,127.0.0.1:32501" --bootstrap-server-mapping "kafka-2.grepplabs.com:9092,127.0.0.1:32502" --forward-proxy socks5://my-proxy-user:my-proxy-password@localhost:1080
Connexion via le serveur proxy HTTP Test à l'aide de la méthode Connect
kafka-proxy tools http-proxy --addr localhost:3128 kafka-proxy server --bootstrap-server-mapping "kafka-0.grepplabs.com:9092,127.0.0.1:32500" --bootstrap-server-mapping "kafka-1.grepplabs.com:9092,127.0.0.1:32501" --bootstrap-server-mapping "kafka-2.grepplabs.com:9092,127.0.0.1:32502" --forward-proxy http://localhost:3128
kafka-proxy tools http-proxy --addr localhost:3128 --username my-proxy-user --password my-proxy-password kafka-proxy server --bootstrap-server-mapping "kafka-0.grepplabs.com:9092,127.0.0.1:32500" --bootstrap-server-mapping "kafka-1.grepplabs.com:9092,127.0.0.1:32501" --bootstrap-server-mapping "kafka-2.grepplabs.com:9092,127.0.0.1:32502" --forward-proxy http://my-proxy-user:my-proxy-password@localhost:3128
Parfois, il peut être nécessaire non seulement de valider que le certificat client est valide, mais aussi que le certificat client DN est émis pour un cas d'utilisation en béton. Cela peut être réalisé en utilisant l'ensemble d'arguments suivants:
--proxy-listener-tls-client-cert-validate-subject bool Whether to validate client certificate subject (default false) --proxy-listener-tls-required-client-subject-common-name string Required client certificate subject common name --proxy-listener-tls-required-client-subject-country stringArray Required client certificate subject country --proxy-listener-tls-required-client-subject-province stringArray Required client certificate subject province --proxy-listener-tls-required-client-subject-locality stringArray Required client certificate subject locality --proxy-listener-tls-required-client-subject-organization stringArray Required client certificate subject organization --proxy-listener-tls-required-client-subject-organizational-unit stringArray Required client certificate subject organizational unit
En définissant --proxy-listener-tls-client-cert-validate-subject true , Kafka Proxy inspectera les champs DN de certificat client pour les valeurs attendues définies avec les arguments --proxy-listener-tls-required-client-* . Les correspondances sont toujours exactes et utilisées ensemble, pour toutes les valeurs non vides. Par exemple, pour permettre un certificat valide pour country=DE et organization=grepplabs , configurez le proxy Kafka de la manière suivante:
kafka-proxy server --proxy-listener-tls-client-cert-validate-subject true --proxy-listener-tls-required-client-subject-country DE --proxy-listener-tls-required-client-subject-organization grepplabs
--- Apiversion: apps / v1kind: DeploymentMetAdata: Nom: MyAppSpec: répliques: 1
Sélecteur: MatchLabels: App: MyApp
Modèle: Metadata: Étiquettes: App: MyApp Annotations: prometheus.io/scrapes: 'true'spec: conteneurs:
- Nom: Kafka-Proxy Image: GreppLabs / Kafka-Proxy: Dernières Args:
- 'server'-' --log-format = json'- '--bootstrap-server-mapping = kafka-0: 9093,127.0.0.1: 32400'-' --bootstrap-server-mapping = kafka-1: 9093,127.0.0.1: 32401'- '--bootstrap-server-mapping = kafka-2: 9093,127.0.0.1: 32402'-' --tls-enable'- '--tls-ca-chain-cerd- file = / var / run / secret / kafka-ca-chain-certificate / ca-chain.cert.pem'- '--tls-Client-cert-file = / var / run / secret / kafka-client-certificate / client.cert.pem'- '--tls-Client-Key-file = / var / run / secret / kafka-client-key / client.key.pem'-' --tls-Client-Key-Password = $ (Tls_client_key_password) '-' --sal-enable'- '--sal-jaas-config-file = / var / run / secret / kafka-client-jaas / jaas.config' env:
- Nom: tls_client_key_passwordValueFrom: SecretKeyRef: Nom: TLS-Client-Key-Passwordkey: Mot de passe volumeMounts:
- Nom: "SASL-JAAS-CONFIG-FILE" MountPath: "/ var / Run / Secret / Kafka-Client-Jaas" - Nom: "Tls-Ca-Chain-Certificate" MountPath: "/ var / Run / Secret / Kafka-Ca-Chain-Certificate "- Nom:" Tls-Client-CERT-FILE "MountPath:" / var / Run / Secret / Kafka-Client-Certificate "- Nom:" Tls-Client-Key-File "MountPath: Ports "/ var / run / secret / kafka-client-key":
- Nom: MetricsContainerport: 9080 LivelyProbe: Httpget: Path: / Health Port: 9080 Nom: MyApp Image: MyApp: Derniers ports:
- Containerport: 8080Name: Metrics Env:
- Nom: bootstrap_serversvalue: "127.0.0.1:32400,127.0.0.1:32401,127.0.1:32402" Volumes:
- Nom: SASL-JAAS-CONFIG-FILESECRET: Secretname: SASL-JAAS-CONFIG-FILE - NOM: TLS-CA-CA-CERTIFATSECRET: SecretName: Tls-Ca-Chain-certificate - Nom: TLS-Client-CERT-FILECRETRE : SecretName: TLS-Client-CERT-FILE - Nom: TLS-Client-Key-FileSecret: SecretName: TLS-Client-Key-File --- Apversion: Apps / V1Kind: StatefulSetMetadata: Nom: Kafka-ProxySpec: Sélecteur: MatchLabels: App: Kafka-Proxy
répliques: 1
ServiceName: kafka-proxy
Modèle: métadonnées: Étiquettes: App: Kafka-ProxySpec: Conteneurs:
- Nom: Kafka-Proxy Image: GreppLabs / Kafka-Proxy: Dernières Args:
- 'server'-' --log-format = json'- '--bootstrap-server-mapping = kafka-0: 9093,127.0.0.1: 32400'-' --bootstrap-server-mapping = kafka-1: 9093,127.0.0.1: 32401'- '--bootstrap-server-mapping = kafka-2: 9093,127.0.0.1: 32402'-' --tls-enable'- '--tls-ca-chain-cerd- file = / var / run / secret / kafka-ca-chain-certificate / ca-chain.cert.pem'- '--tls-Client-cert-file = / var / run / secret / kafka-client-certificate / client.cert.pem'- '--tls-Client-Key-file = / var / run / secret / kafka-client-key / client.key.pem'-' --tls-Client-Key-Password = $ (Tls_client_key_password) '-' --sal-enable'- '--sal-jaas-config-file = / var / run / secré -size = 32768'- '--proxy-réponse-buffer-size = 32768'-' --proxy-listener-read-buffer-size = 32768'- '--proxy-listener-write-buffer-size = 131072 '-' --kafka-connection-read-buffer-size = 131072'- '--kafka-connection-write-buffer-size = 32768' Env:
- Nom: tls_client_key_passwordValueFrom: SecretKeyRef: Nom: TLS-Client-Key-Passwordkey: Mot de passe volumeMounts:
- Nom: "SASL-JAAS-CONFIG-FILE" MountPath: "/ var / Run / Secret / Kafka-Client-Jaas" - Nom: "Tls-Ca-Chain-Certificate" MountPath: "/ var / Run / Secret / Kafka-Ca-Chain-Certificate "- Nom:" Tls-Client-CERT-FILE "MountPath:" / var / Run / Secret / Kafka-Client-Certificate "- Nom:" Tls-Client-Key-File "MountPath: Ports "/ var / run / secret / kafka-client-key":
- name: metricscontainerPort: 9080 - name: kafka-0containerPort: 32400 - name: kafka-1containerPort: 32401 - name: kafka-2containerPort: 32402 livenessProbe:httpGet: path: /health port: 9080initialDelaySeconds: 5periodSeconds: 3 readinessProbe:httpGet: path : / Port de santé: 9080InitialDelaysEcondes: 5PeriodSeconds: 10TimeoutsEcondes: 5SucCesStHreshold: 2failulurethreshold: 5 Ressources: Demandes: Mémoire: 128Mi CPU: 1000m RestartPolicy: toujours volumes:
- Nom: SASL-JAAS-CONFIG-FILESECRET: Secretname: SASL-JAAS-CONFIG-FILE - NOM: TLS-CA-CA-CERTIFATSECRET: SecretName: Tls-Ca-Chain-certificate - Nom: TLS-Client-CERT-FILECRETRE : SecretName: TLS-Client-CERT-FILE - Nom: TLS-Client-Key-FileSecret: SecretName: TLS-Client-Key-FileKubectl Port-Forward Kafka-Proxy-0 32400: 32400 32401: 32401 32402: 32402
Utilisez localhost: 32400, localhost: 32401 et localhost: 32402 comme serveurs bootstrap
kafka.properties
broker.id=0 advertised.listeners=PLAINTEXT://kafka-0.kafka-headless.kafka:9092 ...
Kubectl Port-Forward -N Kafka Kafka-0 9092: 9092
Kafka-Proxy Server --bootstrap-server-mapping "127.0.0.1:9092,0.0.0.0:19092" --dial-address-mapping "kafka-0.kafka-headless.kafka: 9092,0.0.0.0.0: 9092"
Utilisez LocalHost: 19092 comme serveurs bootstrap
Strimzi 0,13,0 CRD
Apiversion: kafka.strimzi.io/v1beta1kind: Kafkametadata: Nom: Test-Cluster
Espace de noms: Kafkaspec: Kafka: Version: 2.3.0Replicas: 3Listeners: Plain: {} tls: {} config: offsets.topic.replication.factor: 3 transaction.state.log.replication.factor: 3 transaction.state.log. min.isr: 2 num.Partitions: 60 default.replication.factor: 3Storage: Type: JBOD Volumes:
- ID: 0 Type: Persistant-Claim Taille: 20gi DeleteClaim: true
Zookeeper: répliques: 3Storage: Type: Taille de la réclamation persistante: 5gi DeleteClaim: true
EntityOperator: TopicOperator: {} UserOperator: {}KUBECTL PORT-FARWER -N KAFKA TEST-CLUSTER-KAFKA-0 9092: 9092 Kubectl Port-Forward -N Kafka Test-Cluster-Kafka-1 9093: 9092 Kubectl Port-Forward -N Kafka Test-Cluster-Kafka-2 9094: 9092 serveur kafka-proxy - débogage de niveau-logarithme - Bootstrap-Server-Mapping "127.0.0.1:9092,0.0.0:19092" - Bootstrap-Server-Mapping "127.0.0.1:9093,0.0.0:19093" - Bootstrap-Server-Mapping "127.0.0.1:9094,0.0.0:19094" --dial-address-mapping "test-cluster-kafka-0.test-cluster-kafka-brokers.kafka.svc.cluster.local: 9092,0.0.0.0: 9092" --dial-address-mapping "test-cluster-kafka-1.test-cluster-kafka-brokers.kafka.svc.cluster.local: 9092,0.0.0.0: 9093" --dial-address-mapping "test-cluster-kafka-2.test-cluster-kafka-brokers.kafka.svc.cluster.local: 9092,0.0.0.0: 9094"
Utilisez LocalHost: 19092 comme serveurs bootstrap
Cloud SQL Proxy
Sarama
