carbonate
1.1.10
"Pecahkan botol." -- Manusia Burung
Cluster grafit cukup keren. Berikut beberapa alat primitif untuk membantu Anda mengelola kluster grafit.
Semua alat mendukung dua argumen umum; jalur ke file konfigurasi, dan nama cluster. Dengan menggunakan alat ini bersama dengan file konfigurasi yang menjelaskan kluster grafit, Anda dapat membuat skrip untuk mengelola metrik Anda. Beberapa alat dapat dengan mudah diganti dengan satu baris di shell, tetapi ada di sini untuk kenyamanan dan keterbacaan. Tujuannya adalah untuk menyediakan utilitas yang cepat dan dapat diprediksi yang dapat dengan mudah disusun menjadi peralatan yang lebih canggih.
Carbonate tersedia dari repositori resmi pihak ketiga Python (alias PyPi) dan dengan demikian dapat diinstal melalui manajer paket Python biasa. Perhatikan bahwa Anda mungkin harus menginstal manajer paket python (misalnya apt-get install python-setuptools pada host ubuntu)
pip install carbonate
Carbonate mengharapkan file konfigurasi yang mendefinisikan cluster di lingkungan Anda. File konfigurasi default terletak di /opt/graphite/conf/carbonate.conf atau dapat disediakan di baris perintah. Cluster default diberi nama 'utama'. Kedua default dapat diganti dengan mengatur masing-masing lingkungan CARBONATE_CONFIG dan CARBONATE_CLUSTER .
[main]
DESTINATIONS = 192.168.9.13:2004:carbon01, 192.168.9.15:2004:carbon02, 192.168.6.20:2004:carbon03
REPLICATION_FACTOR = 2
SSH_USER = carbon
[agg]
DESTINATIONS = 192.168.9.13:2004:carbon01, 192.168.9.15:2004:carbon02, 192.168.6.20:2004:carbon03
RELAY_METHOD = aggregated-consistent-hashing
REPLICATION_FACTOR = 2
SSH_USER = carbon
[fnv]
DESTINATIONS = 192.168.9.13:2004:ba603c36342304ed77953f84ac4d357b, 192.168.9.15:2004:5dd63865534f84899c6e5594dba6749a, 192.168.6.20:2004:866a18b81f2dc4649517a1df13e26f28
REPLICATION_FACTOR = 2
SSH_USER = carbonate
HASHING_TYPE = fnv1a_ch
Anda harus berhati-hati dalam mencocokkan daftar IP tujuan atau nama host dengan node di klaster Anda (yaitu, harus cocok dengan konfigurasi perutean relai karbon Anda). Urutan penting karena cara pembuatan cincin hash yang konsisten.
Anda dapat mengonfigurasi metode relai menjadi salah satu "hashing-konsisten" atau "hashing-konsisten-agregat". Jika dihilangkan, "hashing konsisten" digunakan secara default. Penggunaan "hashing-konsisten-agregat" biasanya memerlukan file aturan untuk disediakan ke perintah yang relevan.
Faktor replikasi harus sesuai dengan faktor replikasi klaster.
Selain itu, Anda dapat memilih untuk menyediakan pengguna SSH yang akan digunakan ketika karbonat memerlukan koneksi ke node lain di klaster untuk melakukan suatu operasi. Jika ini tidak disediakan, maka pengguna saat ini yang menjalankan perintah akan dipilih.
Terakhir, Anda dapat memberikan HASHING_TYPE cluster Anda. Defaultnya adalah carbon_ch , fnv1a_ch juga didukung. Harap dicatat bahwa untuk menggunakan hashing fnv1a_ch Anda perlu menginstal carbon 1.0.2 atau yang lebih baru (atau Anda perlu menggunakan relay carbon-c-relay sebagai gantinya).
usage: carbon-hosts [-h] [-c CONFIG_FILE] [-C CLUSTER]
Return the addresses for all nodes in a cluster
optional arguments:
-h, --help show this help message and exit
-c CONFIG_FILE, --config-file CONFIG_FILE
Config file to use (default:
/opt/graphite/conf/carbonate.conf)
-C CLUSTER, --cluster CLUSTER
Cluster name (default: main)
usage: carbon-lookup [-h] [-c CONFIG_FILE] [-C CLUSTER] [-s] METRIC
Lookup where a metric lives in a carbon cluster
positional arguments:
METRIC Full metric name to search for
optional arguments:
-h, --help show this help message and exit
-c CONFIG_FILE, --config-file CONFIG_FILE
Config file to use (default:
/opt/graphite/conf/carbonate.conf)
-C CLUSTER, --cluster CLUSTER
Cluster name (default: main)
-a AGGREGATION_RULES, --aggregation-rules AGGREGATION_RULES
File containing rules used in conjunction with the
"aggregated-consistent-hashing" relay method (default:
/opt/graphite/conf/aggregation-rules.conf)
-s, --short Only display the address, without port and cluster
name (default: False)
usage: carbon-list [-h] [-c CONFIG_FILE] [-C CLUSTER] [-d STORAGE_DIR]
List the metrics this carbon node contains
optional arguments:
-h, --help show this help message and exit
-c CONFIG_FILE, --config-file CONFIG_FILE
Config file to use (default:
/opt/graphite/conf/carbonate.conf)
-C CLUSTER, --cluster CLUSTER
Cluster name (default: main)
-d STORAGE_DIR, --storage-dir STORAGE_DIR
Storage dir (default: /opt/graphite/storage/whisper)
usage: carbon-sieve [-h] [-c CONFIG_FILE] [-C CLUSTER] [-f METRICS_FILE]
[-n NODE] [-I]
Given a list of metrics, output those that belong to a node
optional arguments:
-h, --help show this help message and exit
-c CONFIG_FILE, --config-file CONFIG_FILE
Config file to use (default:
/opt/graphite/conf/carbonate.conf)
-C CLUSTER, --cluster CLUSTER
Cluster name (default: main)
-a AGGREGATION_RULES, --aggregation-rules AGGREGATION_RULES
File containing rules used in conjunction with the
"aggregated-consistent-hashing" relay method (default:
/opt/graphite/conf/aggregation-rules.conf)
-f METRICS_FILE, --metrics-file METRICS_FILE
File containing metric names to filter, or '-' to read
from STDIN (default: -)
-n NODE, --node NODE Filter for metrics belonging to this node (default:
self)
-I, --invert Invert the sieve, match metrics that do NOT belong to
a node (default: False)
usage: carbon-sync [-h] [-c CONFIG_FILE] [-C CLUSTER] [-f METRICS_FILE] -s
SOURCE_NODE [-d STORAGE_DIR] [-b BATCH_SIZE]
[--source-storage-dir SOURCE_STORAGE_DIR]
[--rsync-options RSYNC_OPTIONS] [--rsync-disable-copy-dest]
[--tmpdir TMP_STAGING_DIR] [--rsync-max-retries MAX_RETRIES]
[--rsync-retries-interval SECONDS] [--dirty] [-l] [-o]
Sync local metrics using remote nodes in the cluster
optional arguments:
-h, --help show this help message and exit
-c CONFIG_FILE, --config-file CONFIG_FILE
Config file to use (env: CARBONATE_CONFIG) (default:
/opt/graphite/conf/carbonate.conf)
-C CLUSTER, --cluster CLUSTER
Cluster name (env: CARBONATE_CLUSTER) (default: main)
-f METRICS_FILE, --metrics-file METRICS_FILE
File containing metric names to filter, or '-' to read
from STDIN (default: -)
-s SOURCE_NODE, --source-node SOURCE_NODE
Override the source for metrics data (default: None)
-d STORAGE_DIR, --storage-dir STORAGE_DIR
Storage dir (default: /opt/graphite/storage/whisper)
-b BATCH_SIZE, --batch-size BATCH_SIZE
Batch size for the rsync job (default: 1000)
--source-storage-dir SOURCE_STORAGE_DIR
Source storage dir (default:
/opt/graphite/storage/whisper)
--rsync-options RSYNC_OPTIONS
Pass option(s) to rsync. Make sure to use "--rsync-
options=" if option starts with '-' (default: -azpS)
--rsync-disable-copy-dest
Avoid --copy-dest, transfer all whisper data between
nodes. (default: False)
--rsync-max-retries RETRIES
Number of times rsync will attempt to copy each batch
of metrics before moving on. If all retry attempts are
unsuccessful, carbon-sync will write a file containing
the name of each metric in the failed batch so they can
be easily retried at a later time. (Default: 3)
--rsync-retries-interval SECONDS
How long to wait in between each rsync retry attempt
(see --rsync-max-retries). (default: 5)
-t TMP_STAGING_DIR, --tmpdir TMP_STAGING_DIR
Specify an alternate location in which the temporary
rsync staging dirs will be created. This can be useful
for large syncs where the default location (as chosen
by mkdtemp) resides on a filesystem that's too small
to store all the metrics being copied from the remote
host.
--dirty If set, don't clean temporary rsync directory
(default: False)
-l, --lock Lock whisper files during filling (default: False)
-o, --overwrite Write all non nullpoints from src to dst (default:
False)
usage: carbon-path [-h] [-c CONFIG_FILE] [-C CLUSTER] [-f METRICS_FILE] [-r]
[-p] [-d STORAGE_DIR]
Transform metric paths to (or from) filesystem paths
optional arguments:
-h, --help show this help message and exit
-c CONFIG_FILE, --config-file CONFIG_FILE
Config file to use (default:
/opt/graphite/conf/carbonate.conf)
-C CLUSTER, --cluster CLUSTER
Cluster name (default: main)
-f METRICS_FILE, --metrics-file METRICS_FILE
File containing metric names to transform to file
paths, or '-' to read from STDIN (default: -)
-r, --reverse Transform from file paths to metric paths (default:
False)
-p, --prepend Prepend storage dir to file paths (default: False)
-d STORAGE_DIR, --storage-dir STORAGE_DIR
Whisper storage directory to prepend when -p given
(default: /opt/graphite/storage/whisper)
usage: carbon-stale [-h] [-c CONFIG_FILE] [-C CLUSTER] [-f METRICS_FILE] [-r]
[-d STORAGE_DIR] [-l HOURS] [-o HOURS] [-w] [-p]
Find and list potentially stale metrics.
optional arguments:
-h, --help show this help message and exit
-c CONFIG_FILE, --config-file CONFIG_FILE
Config file to use (default:
/opt/graphite/conf/carbonate.conf)
-C CLUSTER, --cluster CLUSTER
Cluster name (default: main)
-f METRICS_FILE, --metrics-file METRICS_FILE
File containing metric names to transform to file
paths, or '-' to read from STDIN (default: -)
-r, --reverse Output metrics which are not stale instead (default:
False)
-d STORAGE_DIR, --storage-dir STORAGE_DIR
Whisper storage directory to prepend when -p given
(default: /opt/graphite/storage/whisper)
-l HOURS, --limit HOURS
Definition of staleness, in hours (default: 24)
-o HOURS, --offset HOURS
Use a whisper data window ending HOURS ago (implies
-w) (default: 0)
-w, --whisper Use whisper data instead of filesystem stat() call
(default: False)
-p, --paths Print filesystem paths instead of metric names
(default: False)
usage: whisper-aggregate [-h] [-f METRICS_FILE] [-d STORAGE_DIR]
Set aggregation for whisper-backed metrics this carbon instance contains
optional arguments:
-h, --help show this help message and exit
-f METRICS_FILE, --metrics-file METRICS_FILE
File containing metric names and aggregation modes, or
'-' to read from STDIN (default: -)
-d STORAGE_DIR, --storage-dir STORAGE_DIR
Whisper storage directory (default:
/opt/graphite/storage/whisper)
usage: whisper-fill [-h] [-l] [-o] SRC DST
Backfill datapoints from one whisper file into another
positional arguments:
SRC Whisper source file
DST Whisper destination file
optional arguments:
-h, --help show this help message and exit
-l, --lock Lock whisper files during filling (default: False)
-o, --overwrite Write all non nullpoints from src to dst (default: False)
#!/bin/sh
#
# Resync a node from other nodes in the cluster
#
LOCAL_IP="$1"
for h in $(carbon-hosts) ; do
(
ssh $h -- carbon-list |
carbon-sieve -n $LOCAL_IP |
carbon-sync -s $h
) &
done
#!/bin/sh
#
# Rebalance a cluster from one size to another. Remember to cleanup metrics
# that no longer belong when all nodes are rebalanced!
#
LOCAL_IP="$1"
OLD_CLUSTER="old"
NEW_CLUSTER="main"
for h in $(carbon-hosts -C "$OLD_CLUSTER") ; do
ssh $h -- carbon-list |
carbon-sieve -C "$NEW_CLUSTER" -n $LOCAL_IP |
carbon-sync -s $h
done
#!/bin/sh
#
# List metrics from disk that don't belong
#
LOCAL_IP="$1"
carbon-list | carbon-sieve -I -n $LOCAL_IP
Metrik dengan data bisikan yang sepenuhnya kosong selama 2 jam terakhir (mungkin berguna jika Anda mencurigai adanya masalah dengan stempel waktu fs atau klien karbon yang menulis di 'masa depan'):
carbon-list | carbon-stale --whisper --limit=2
Metrik yang file metriknya tampak tidak tersentuh selama 48 jam atau lebih (secara fungsional identik dengan find /your/data/dir -type f -mtime +2 ):
carbon-list | carbon-stale --limit=48
Yang lebih menarik adalah jika Anda menggunakan carbon-stale , lalu saring untuk mengidentifikasi metrik basi yang tidak termasuk di sini (vs metrik tidak basi yang memang termasuk di sini tetapi salah dilaporkan dalam saringan karbon karena hal-hal seperti periode yang digandakan dalam jalur metrik karena kolektor rusak.
carbon-list | carbon-stale --limit=48 | carbon-sieve -I -n $LOCAL_IP
Untuk mencetak jalur file untuk digunakan misalnya xargs rm atau yang lainnya, gunakan -p :
carbon-list | carbon-stale -p | xargs -n 100 rm
Kode ini tersedia di bawah lisensi MIT.
