How to quickly get started with VUE3.0: Let’s learn
about using the redis cluster function in nodejs . I didn’t find a relatively complete and easy-to-understand article, so I also took a lot of detours during the development and debugging process.
This article will introduce in detail how to build a redis cluster locally, how to use the cluster on the client, and summarize and explain the problems and errors encountered during the construction process to avoid detours next time and improve development and work efficiency.
The background of using the cluster is: in the Redis singleton mode, with the increase in the number of users and visits, the qps value rises sharply??, a large number of io operations cause cpu(100%) at a certain time, and there may be downtime at any time. It is dangerous. At the same time, batch processing redis and other methods only treat the symptoms but not the root cause, and cannot break through the bottleneck of server performance . Therefore, it is imperative to use a cluster solution or add redis instances.
cluster generally refers to a server cluster, which is different from a distributed system. It gathers many servers together to perform the same service. From the client's perspective, it looks like there is only one server. A cluster can use multiple computers for parallel computing to achieve high computing speed. It can also use multiple computers for backup, so that if any machine breaks down, the entire system can still run normally. ( Before redis3.0, the sentinel mode was generally used, but the configuration of the sentinel is slightly complicated, and the performance and high availability are average )
that due to the voting fault tolerance mechanism, more than half of the nodes believe that a certain node has failed. The node is down, so 2 nodes cannot form a cluster, so the Redis cluster requires at least 3 nodes.
To ensure the high availability of the cluster, each node needs to have a slave node (that is, a backup node), so Redis cluster requires at least 6 servers. (Three masters and three slaves, three deposits and three retrievals, high availability, and backup)
Of course, it is impossible for us to use so many servers when debugging locally, so we can simulate running 6 redis instances locally. In fact, the Redis cluster in the production environment The build is basically the same as here.
. You can choose to install it on the official website, or you can install it with the named line
#install brew install redis #Start redis-server #Enter the redis client redis-cli
you must first find the location of the redis configuration file
brew list redis # Check the location of the redis installationcd /opt/homebrew/Cellar/redis/6.2.4 # Enter the version according to the location The folder where the number is locatedopen . # Open the folderXcode.app to open homebrew.mxcl.redis.plist to find the location of redis.conf , as shown below:
Create six service configuration files
cd /opt/homebrew/etc/ (the configuration file directory found in the previous step)
# Need to be in the /opt/homebrew/etc/ path mkdir -p redis/cluster/7000 mkdir -p redis/cluster/7001 mkdir -p redis/cluster/7002 mkdir -p redis/cluster/7003 mkdir -p redis/cluster/7004 mkdir -p redis/cluster/7005
Modify the configuration file
The configuration file under the path /opt/homebrew/etc/redis.conf does not need to be modified. Just copy it to the redis/cluster/7000 directory created above and then modify it. The steps are as follows:
cd /opt/homebrew/etc/ # Enter the configuration file directory cp redis.conf redis/cluster/7000/7000.conf code redis/cluster/7000/7000.conf # Open the configuration file with an editor or vim to modify it.
7000.conf , modify the following properties# Redis port number (each configuration file 7000-7005 must be modified) port 7000 # Enable cluster mode and run cluster-enabled yes # Cluster internal configuration file configuration file path, default nodes-6379.conf (each configuration file 7000-7005 must be modified) cluster-config-file nodes-7000.conf # Inter-node communication timeout cluster-node-timeout 5000 # Data persistence appendonly yes
cd /opt/homebrew/etc/redis/cluster # Enter the configuration file directory cp 7000/7000.conf 7001/7001.conf cp 7000/7000.conf 7002/7002.conf cp 7000/7000.conf 7003/7003.conf cp 7000/7000.conf 7004/7004.conf cp 7000/7000.conf 7005/7005.conf
7001.conf-7005.confNote: Each configuration file must configure different port和cluster-config-file value (otherwise the cluster will not take effect), which is distinguished by port.
The directory of the configuration file can be found through the find /opt/homebrew -name nodes-7000.conf command.
Since we have configured 6 services, it is impossible to start or stop them one by one. You need to use a shell script to
enter the /opt/homebrew/etc/redis/cluster directory and create start.sh and stop. .sh file
# start.sh file#!/bin/sh redis-server /opt/homebrew/etc/redis/cluster/7000/7000.conf & redis-server /opt/homebrew/etc/redis/cluster/7001/7001.conf & redis-server /opt/homebrew/etc/redis/cluster/7002/7002.conf & redis-server /opt/homebrew/etc/redis/cluster/7003/7003.conf & redis-server /opt/homebrew/etc/redis/cluster/7004/7004.conf & redis-server /opt/homebrew/etc/redis/cluster/7005/7005.conf & # stop.sh file #!/bin/sh redis-cli -p 7000 shutdown & redis-cli -p 7001 shutdown & redis-cli -p 7002 shutdown & redis-cli -p 7003 shutdown & redis-cli -p 7004 shutdown & redis-cli -p 7005 shutdown &
execute ./start.sh or ./stop.sh to start and stop the service.
Execute ps -ef |grep redis to view the started redis service.
Note: It is required to execute ./start.sh for the first time. Authorize execution permissions through sudo chmod +x start.sh
redis-cli -p 7000 # Start a single client redis-server 7000/7000.conf # Start a single server redis-cli -p 7000 shutdown # Shut down the server sudo chmod +x start.sh # Enable script execution permission # Set redis master-slave relationship (three masters and three slaves) redis-cli --cluster create --cluster-replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 cluster nodes #View cluster node status (enter a client to execute) cluster info #View cluster information (enter a client to execute) View all key values: keys * Delete the value at the specified index: del key Clear the data of the entire Redis server: flushall Clear all keys in the current library: the flushdb
Redis.Cluster provides the function of automatic sharding on multiple Redis nodes. Use the six redis servers built previously, and then start node redis.js locally. node redis.js , you can test the effect of the cluster. ioredis
// redis.js
const Redis = require("ioredis");
const cluster = new Redis.Cluster([
{
port: 7000,
host: "127.0.0.1",
},
{
port: 7001,
host: "127.0.0.1",
},
]);
cluster.set("foo", "bar");
cluster.get("foo", (err, res) => {
// res === 'bar'
}); import Queue from 'bull'
// Create a redis queue instance const instance = new Queue('custom', {
prefix : '{myprefix}',
createClient(type) {
// cluster cluster instance is the same as above return cluster
}
})
//Add data to redis queue (producer)
instance.add(
'request',
{
...params
},
{
removeOnComplete: false
}
).catch(e => {
console.error(e)
})
// Consumer callback instance.process('request', 5, async (job, done) => {
console.log('Get currently consumed data:', job.data)
//Perform asynchronous operation await new Promise((resolve)=>resolve())
done()
}) There is a problem when using the bull framework to connect to the ioredis cluster: every time data push to the redis queue, the corresponding callback function may be triggered multiple times. It is currently impossible to determine whether it is a problem with the use or a problem with the framework itself (if you know more, welcome everyone Leave a message to inform).
Alternative to clustering: If data synchronization and data migration are not required, multiple redis instances can be used on the client, combined with Math.random() to divide the data equally into one of redis , thus solving the problem of single instance hardware ( cpu etc.) bottleneck problem.
1. An error occurs when connecting to redis under Mac system?
Console error message: Could not connect to Redis at 127.0.0.1:6379: Connection refused
Reason: The server is not started or fails to start
Solution: You need to start the redis server first redis-server
reference link
https://blog.csdn. net/qq_23347459/article/details/104257529
2. When starting the client, does it report an error when reading or writing?
Error message: ClusterAllFailedError: Failed to refresh slots cache.
Reason: The cluster-config-file attributes in the configuration files under each service are consistent.
Processing: Modify to unique attribute value
Reference link 1
https://stackoverflow.com/questions/57350961/ioredis-clusterallfailederror-failed-to-refresh-slots-cache
Reference 2
https://github.com/luin/ioredis/ issues/711
3. Failed to execute the create master-slave redis statement?
Execute statement: redis-cli --cluster create --cluster-replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
Prompt error: [ERR] Node 127.0.0.1:7000 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0
: When executing the create statement, the data is not cleared and the cluster is not reset.
Processing: Clearing the data And reset the cluster, clear rdb and aof files,
refer to clearing redis data
https://stackoverflow.com/questions/37206993/redis-server-cluster-not-working
# Take the service of port 7000 as an example, repeat the following operations for 7001-7005 $redis-cli -p 7000 127.0.0.1:7000> flushall 127.0.0.1:7000> cluster reset 127.0.0.1:7000>exit # Use find to find the rdb and aof files (also in the rdb directory) find /opt/homebrew -name dump.rdb # Re-execute the create statement successfully redis-cli --cluster create --cluster-replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
The use of redis cluster on the client is very simple, in comparison, the configuration on the server will be more cumbersome.
The specific use of the client is only briefly explained. During use, attention should be paid to issues such as the synchronization and migration of redis data.
Although using a cluster can improve service capabilities, support master-slave replication, sentinel mode, read-write separation, and evenly distribute the pressure on the server, etc. However, it does not have automatic fault tolerance and recovery functions. If a downtime occurs, some read and write requests will fail, reducing the availability of the system. When using, analyze and choose different solutions based on business conditions.
This article is reproduced from: https://juejin.cn/post/7079928963107127327
Author: tager