Memcached is a distributed memory object caching system developed by danga.com (the technical team that operates LiveJournal) to reduce database load and improve performance in dynamic systems. Regarding this thing, I believe many people have used it. The purpose of this article is to gain a deeper understanding of this excellent open source software through the implementation and code analysis of memcached, and to further optimize it according to our needs. Finally, through the analysis of the BSM_Memcache extension, we will deepen our understanding of the use of memcached.
Some of the content in this article may require a better mathematical foundation as assistance.
◎What is Memcached?
Before elaborating on this issue, we must first understand what it "is not". Many people use it as a storage carrier like SharedMemory. Although memcached uses the same "Key=>Value" method to organize data, it is very different from local caches such as shared memory and APC. Memcached is distributed, which means it is not local. It completes the service based on network connection (of course it can also use localhost). It is an application-independent program or daemon process (Daemon mode).
Memcached uses the libevent library to implement network connection services and can theoretically handle an unlimited number of connections. However, unlike Apache, it is more often oriented towards stable continuous connections, so its actual concurrency capabilities are limited. Under conservative circumstances, the maximum number of simultaneous connections for memcached is 200, which is related to the Linux thread capability. This value can be adjusted. For information about libevent, please refer to relevant documentation. Memcached memory usage is also different from APC. APC is based on shared memory and MMAP. Memcachd has its own memory allocation algorithm and management method. It has nothing to do with shared memory and has no restrictions on shared memory. Normally, each memcached process can manage 2GB of memory space. If If more space is needed, the number of processes can be increased.
◎What occasions is Memcached suitable for?
In many cases, memcached has been abused, which of course inevitably leads to complaints about it. I often see people posting on forums, similar to "how to improve efficiency", and the reply is "use memcached". As for how to use it, where to use it, and what it is used for, there is no sentence. Memcached is not a panacea, nor is it suitable for all situations.
Memcached is a "distributed" memory object caching system. That is to say, for those applications that do not need to be "distributed", do not need to be shared, or are simply small enough to have only one server, memcached will not bring any benefits. On the contrary It also slows down system efficiency because network connections also require resources, even UNIX local connections. My previous test data showed that memcached local read and write speeds are dozens of times slower than direct PHP memory arrays, while APC and shared memory methods are similar to direct arrays. It can be seen that if it is only a local-level cache, using memcached is very uneconomical.
Memcached is often used as a database front-end cache. Because it has a lot less SQL parsing, disk operations and other overhead than a database, and it uses memory to manage data, it can provide better performance than directly reading the database. In large systems, it is very difficult to access the same data. Frequently, memcached can greatly reduce database pressure and improve system execution efficiency. In addition, memcached is often used as a storage medium for data sharing between servers. For example, data that saves the system's single sign-on status in an SSO system can be saved in memcached and shared by multiple applications.
It should be noted that memcached uses memory to manage data, so it is volatile. When the server is restarted or the memcached process is terminated, the data will be lost, so memcached cannot be used to persist data. Many people misunderstand that the performance of memcached is very good, as good as the comparison between memory and hard disk. In fact, memcached will not get hundreds or thousands of reading and writing speed improvements by using memory. Its actual bottleneck lies in the network connection, which is related to the use of memory. Compared with the disk database system, the advantage is that it is very "light". Because there is no excessive overhead and direct reading and writing methods, it can easily handle a very large amount of data exchange, so there are often two gigabit network bandwidths. They are all fully loaded, and the memcached process itself does not occupy much CPU resources.
◎How Memcached works
In the following sections, it is best for readers to prepare a copy of the source code of memcached.
Memcached is a traditional network service program. If the -d parameter is used when starting, it will be executed as a daemon process. Creating a daemon process is completed by daemon.c. This program has only one daemon function, which is very simple (if no special instructions are given, the code shall be subject to 1.2.1):
CODE:[Copy to clipboard]#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
int
daemon(nochdir, noclose)
int nochdir, noclose;
{
int fd;
switch (fork()) {
case-1:
return (-1);
case 0:
break;
default:
_exit(0);
}
if (setsid() == -1)
return (-1);
if (!nochdir)
(void)chdir("/");
if (!noclose && (fd = open("/dev/null", O_RDWR, 0)) != -1) {
(void)dup2(fd, STDIN_FILENO);
(void)dup2(fd, STDOUT_FILENO);
(void)dup2(fd, STDERR_FILENO);
if (fd > STDERR_FILENO)
(void)close(fd);
}
return (0);
}
After this function forks the entire process, the parent process exits, and then relocates STDIN, STDOUT, and STDERR to empty devices, and the daemon is successfully established.
The startup process of Memcached itself is as follows in the main function of memcached.c:
1. Call settings_init() to set the initialization parameters.
2. Read parameters from the startup command to set the setting value
3. Set LIMIT parameter
4. Start network socket monitoring (if non-socketpath exists) (UDP mode is supported after 1.2)
5. Check user identity (Memcached does not allow root identity to be started)
6. If socketpath exists, open the UNIX local connection (Sock pipe)
7. If started in -d mode, create a daemon process (call the daemon function as above)
8. Initialize item, event, status information, hash, connection, slab
9. If managed takes effect in the settings, create a bucket array
10. Check whether the memory page needs to be locked
11. Initialize signal, connection, delete queue
12. If in daemon mode, process process ID
13. The event starts, the startup process ends, and the main function enters the loop.
In daemon mode, because stderr has been directed to the black hole, no visible error messages during execution will be fed back.
The main loop function of memcached.c is drive_machine. The incoming parameter is a structure pointer pointing to the current connection, and the action is determined based on the status of the state member.
Memcached uses a set of custom protocols to complete data exchange. Its protocol document can be referred to: http://code.sixapart.com/svn/memcached/trunk/server/doc/protocol.txt
In the API, the newline symbols are unified as rn
◎Memcached's memory management method
Memcached has a very unique memory management method. In order to improve efficiency, it uses pre-application and grouping methods to manage memory space, instead of malloc every time it needs to write data. , free a pointer when deleting data. Memcached uses the slab->chunk organization method to manage memory.
There are some differences in the slab space division algorithms in slabs.c in 1.1 and 1.2, which will be introduced separately later.
Slab can be understood as a memory block. A slab is the smallest unit for memcached to apply for memory at one time. In memcached, the default size of a slab is 1048576 bytes (1MB), so memcached uses the entire MB of memory. Each slab is divided into several chunks, and each chunk stores an item. Each item also contains the item structure, key and value (note that the value in memcached is only a string). Slabs form linked lists according to their own IDs, and these linked lists are hung on a slabclass array according to their IDs. The entire structure looks a bit like a two-dimensional array. The length of slabclass is 21 in 1.1 and 200 in 1.2.
slab has an initial chunk size, which is 1 byte in 1.1 and 80 bytes in 1.2. There is a factor value in 1.2, which defaults to 1.25.
In 1.1, the chunk size is expressed as initial size * 2^n, n is classid, That is: the slab with id 0 has a chunk size of 1 byte, the slab with id 1 has a chunk size of 2 bytes, the slab with id 2 has a chunk size of 4 bytes... the slab with id 20 has a chunk size of 4 bytes. The size is 1MB, which means that there is only one chunk in the slab with ID 20:
CODE:[Copy to clipboard]void slabs_init(size_t limit) {
int i;
int size=1;
mem_limit = limit;
for(i=0; i<=POWER_LARGEST; i++, size*=2) {
slabclass[i].size = size;
slabclass[i].perslab = POWER_BLOCK / size;
slabclass[i].slots = 0;
slabclass[i].sl_curr = slabclass[i].sl_total = slabclass[i].slabs = 0;
slabclass[i].end_page_ptr = 0;
slabclass[i].end_page_free = 0;
slabclass[i].slab_list = 0;
slabclass[i].list_size = 0;
slabclass[i].killing = 0;
}
/* for the test suite: faking of how much we've already malloc'd */
{
char *t_initial_malloc = getenv("T_MEMD_INITIAL_MALLOC");
if (t_initial_malloc) {
mem_malloced = atol(getenv("T_MEMD_INITIAL_MALLOC"));
}
}
/* pre-allocate slabs by default, unless the environment variable
for testing is set to something non-zero */
{
char *pre_alloc = getenv("T_MEMD_SLABS_ALLOC");
if (!pre_alloc || atoi(pre_alloc)) {
slabs_preallocate(limit / POWER_BLOCK);
}
}
}
In 1.2, the chunk size is expressed as the initial size * f^n, f is factor, which is defined in memcached.c, and n is classid. At the same time, not all 201 heads need to be initialized because the factor is variable and the initialization only loops to The calculated size reaches half of the slab size, and it starts from id1, that is: slab with id 1, each chunk size is 80 bytes, slab with id 2, each chunk size is 80*f, id is 3 slab, each chunk size is 80*f^2, and the initialization size has a correction value CHUNK_ALIGN_BYTES to ensure n-byte alignment (guaranteing that the result is an integral multiple of CHUNK_ALIGN_BYTES). In this way, under standard circumstances, memcached1.2 will be initialized to id40. The size of each chunk in this slab is 504692, and there are two chunks in each slab. Finally, the slab_init function will add an id41 at the end, which is a whole block, that is, there is only one 1MB chunk in this slab:
CODE:[Copy to clipboard]void slabs_init(size_t limit, double factor) {
int i = POWER_SMALLEST - 1;
unsigned int size = sizeof(item) + settings.chunk_size;
/* Factor of 2.0 means use the default memcached behavior */
if (factor == 2.0 && size < 128)
size = 128;
mem_limit = limit;
memset(slabclass, 0, sizeof(slabclass));
while (++i < POWER_LARGEST && size <= POWER_BLOCK / 2) {
/* Make sure items are always n-byte aligned */
if (size % CHUNK_ALIGN_BYTES)
size += CHUNK_ALIGN_BYTES - (size % CHUNK_ALIGN_BYTES);
slabclass[i].size = size;
slabclass[i].perslab = POWER_BLOCK / slabclass[i].size;
size *= factor;
if (settings.verbose > 1) {
fprintf(stderr, "slab class %3d: chunk size %6d perslab %5dn",
i, slabclass[i].size, slabclass[i].perslab);
}
}
power_largest = i;
slabclass[power_largest].size = POWER_BLOCK;
slabclass[power_largest].perslab = 1;
/* for the test suite: faking of how much we've already malloc'd */
{
char *t_initial_malloc = getenv("T_MEMD_INITIAL_MALLOC");
if (t_initial_malloc) {
mem_malloced = atol(getenv("T_MEMD_INITIAL_MALLOC"));
}
}
#ifndef DONT_PREALLOC_SLABS
{
char *pre_alloc = getenv("T_MEMD_SLABS_ALLOC");
if (!pre_alloc || atoi(pre_alloc)) {
slabs_preallocate(limit / POWER_BLOCK);
}
}
#endif
}
As can be seen from the above, the memory allocation of memcached is redundant. When a slab is not divisible by the chunk size it owns, the remaining space at the end of the slab is discarded. For example, in id40, two chunks occupy 1009384 Bytes, this slab has a total of 1MB, so 39192 bytes are wasted.
Memcached uses this method to allocate memory in order to quickly locate the slab's classid through the item length. It is a bit similar to hash, because the item length can be calculated. For example, the length of an item is 300 bytes. In 1.2 You can get that it should be stored in the slab of id7, because according to the above calculation method, the chunk size of id6 is 252 bytes, the chunk size of id7 is 316 bytes, and the chunk size of id8 is 396 bytes, which means all 252 to All 316-byte items should be stored in id7. Similarly, in 1.1, it can also be calculated that it is between 256 and 512, and should be placed in id9 with a chunk_size of 512 (32-bit system).
When Memcached is initialized, slabs will be initialized (as you can see earlier, slabs_init() is called in the main function). It will check a constant DONT_PREALLOC_SLABS in slabs_init(). If this is not defined, it means that the slab is initialized using pre-allocated memory, so that a slab is created for each ID among all the slabclasses that have been defined. This means that 1.2 will allocate 41MB of slab space after starting the process in the default environment. During this process, the second memory redundancy of memcached occurs, because it is possible that an id has not been used at all, but it is also A slab is applied for by default, and each slab will use 1MB of memory.
When a slab is used up, and a new item needs to be inserted with this ID, it will re-apply for a new slab. When applying for a new slab, the corresponding ID The slab linked list will grow. This linked list will grow exponentially. In the function grow_slab_list function, the length of this chain changes from 1 to 2, from 2 to 4, from 4 to 8...:
CODE:[Copy to clipboard]static int grow_slab_list (unsigned int id) {
slabclass_t *p = &slabclass[id];
if (p->slabs == p->list_size) {
size_t new_size = p->list_size ? p->list_size * 2 : 16;
void *new_list = realloc(p->slab_list, new_size*sizeof(void*));
if (new_list == 0) return 0;
p->list_size = new_size;
p->slab_list = new_list;
}
return 1;
}
When locating the item, the slabs_clsid function is used. The incoming parameter is the item size and the return value is the classid. From this process, it can be seen that the third memory redundancy of memcached occurs in the process of saving the item. The item is always smaller than Or equal to the chunk size. When the item is smaller than the chunk size, space is wasted again.
◎Memcached's NewHash algorithm
Memcached's item storage is based on a large hash table. Its actual address is the chunk offset in the slab, but its positioning relies on the result of hashing the key, which is found in the primary_hashtable. All hash and item operations are defined in assoc.c and items.c.
Memcached uses an algorithm called NewHash, which is very effective and efficient. There are some differences between NewHash in 1.1 and 1.2. The main implementation method is still the same. The hash function of 1.2 has been organized and optimized, and its adaptability is better.
NewHash prototype reference: http://burtleburtle.net/bob/hash/evahash.html . Mathematicians are always a little strange, haha~
In order to facilitate conversion, two data types, u4 and u1, are defined. u4 is an unsigned long integer, and u1 is an unsigned char (0-255).
For specific codes, please refer to the 1.1 and 1.2 source code packages.
Pay attention to the hashtable length here. There is also a difference between 1.1 and 1.2. In 1.1, the HASHPOWER constant is defined as 20, and the hashtable table length is hashsize (HASHPOWER), which is 4MB (hashsize is a macro, indicating that 1 is shifted to the right by n bits). In 1.2 It is variable 16, that is, the hashtable table length is 65536:
CODE:[Copy to clipboard]typedef unsigned long int ub4; /* unsigned 4-byte quantities */
typedef unsigned char ub1; /* unsigned 1-byte quantities */
#define hashsize(n) ((ub4)1<<(n))
#define hashmask(n) (hashsize(n)-1)
In assoc_init(), the primary_hashtable will be initialized. The corresponding hash operations include: assoc_find(), assoc_expand(), assoc_move_next_bucket(), assoc_insert(), assoc_delete(), corresponding to the read and write operations of the item. Among them, assoc_find() is a function that finds the corresponding item address based on the key and key length (note that in C, many times the string and string length are directly passed in at the same time instead of doing strlen inside the function), and what is returned is Item structure pointer, its data address is on a chunk in the slab.
items.c is the operation program for data items. Each complete item includes several parts, which are defined in item_make_header() as:
key: key
nkey: key length
flags: user-defined flag (in fact, this flag is not enabled in memcached)
nbytes: value length (including newline symbol rn)
suffix: suffix Buffer
nsuffix: The suffix
length of a complete item is the key length + value length + suffix length + item structure size (32 bytes). The item operation is based on this length to calculate the slab's classid.
Each bucket in the hashtable is hung with a double linked list. During item_init(), the three arrays of heads, tails, and sizes have been initialized to 0. The sizes of these three arrays are the constant LARGEST_ID (the default is 255, this value requires modified with factor), each time item_assoc() is called, it will first try to obtain a free chunk from the slab. If there is no available chunk, it will scan the linked list 50 times to get a chunk that was kicked off by LRU. item, unlink it, and then insert the item to be inserted into the linked list.
Pay attention to the refcount member of item. After the item is unlinked, it is only removed from the linked list. It is not freed immediately. It is just placed in the deletion queue (item_unlink_q() function).
Item corresponds to some read and write operations, including remove, update, and replace. Of course, the most important one is the alloc operation.
Another feature of item is that it has an expiration time, which is a very useful feature of memcached. Many applications rely on memcached's item expiration, such as session storage, operation locks, etc. The item_flush_expired() function scans the items in the table and performs an unlink operation on expired items. Of course, this is just a recycling action. In fact, time judgment is also required when getting:
CODE:[Copy to clipboard]/* expires items that are more recent than the oldest_live setting. */
void item_flush_expired() {
int i;
item *iter, *next;
if (! settings.oldest_live)
return;
for (i = 0; i < LARGEST_ID; i++) {
/* The LRU is sorted in decreasing time order, and an item's timestamp
* is never newer than its last access time, so we only need to walk
* back until we hit an item older than the oldest_live time.
* The oldest_live checking will auto-expire the remaining items.
*/
for (iter = heads[i]; iter != NULL; iter = next) {
if (iter->time >= settings.oldest_live) {
next = iter->next;
if ((iter->it_flags & ITEM_SLABBED) == 0) {
item_unlink(iter);
}
} else {
/* We've hit the first old item. Continue to the next queue. */
break;
}
}
}
}
CODE:[Copy to clipboard]/* wrapper around assoc_find which does the lazy expiration/deletion logic */
item *get_item_notedeleted(char *key, size_t nkey, int *delete_locked) {
item *it = assoc_find(key, nkey);
if (delete_locked) *delete_locked = 0;
if (it && (it->it_flags & ITEM_DELETED)) {
/* it's flagged as delete-locked. let's see if that condition
is past due, and the 5-second delete_timer just hasn't
gotten to it yet... */
if (!item_delete_lock_over(it)) {
if (delete_locked) *delete_locked = 1;
it = 0;
}
}
if (it && settings.oldest_live && settings.oldest_live <= current_time &&
it->time <= settings.oldest_live) {
item_unlink(it);
it = 0;
}
if (it && it->exptime && it->exptime <= current_time) {
item_unlink(it);
it = 0;
}
return it;
}
Memcached's memory management method is very sophisticated and efficient. It greatly reduces the number of direct allocs of system memory, reduces function overhead and the probability of memory fragmentation. Although this method will cause some redundant waste, this waste is It is trivial in large system applications.
◎The theoretical parameter calculation method of Memcached
has several parameters that affect the work of memcached:
constant REALTIME_MAXDELTA 60*60*24*30
Maximum expiration time of 30 days
freetotal (=200) in conn_init()
Maximum number of simultaneous connections
constant KEY_MAX_LENGTH 250
Maximum key length
settings.factor (=1.25)
factor will affect the step size of chunk
settings.maxconns (=1024)
Maximum soft connection
settings.chunk_size (=48)
A conservatively estimated key+value length, used to generate the chunk length (1.2) in id1. The chunk length of id1 is equal to this value plus the length of the item structure (32), which is the default 80 bytes.
Constant POWER_SMALLEST 1
Minimum classid (1.2)
constant POWER_LARGEST 200
Maximum classid (1.2)
constant POWER_BLOCK 1048576
Default slab size
constant CHUNK_ALIGN_BYTES (sizeof(void *))
Ensure that the chunk size is an integer multiple of this value to prevent out-of-bounds (the length of void * is different on different systems, it is 4 on standard 32-bit systems)
constant ITEM_UPDATE_INTERVAL 60
Queue refresh interval
constant LARGEST_ID 255
Maximum number of items in the linked list (this value cannot be smaller than the largest classid)
variable hashpower (constant HASHPOWER in 1.1)
Determining the size of the hashtable.
Based on the content and parameter settings introduced above, some results can be calculated:
1. There is no software upper limit for the number of items that can be saved in memcached. My previous statement of 1 million was wrong.
2. Assuming that the NewHash algorithm has uniform collisions, the number of cycles to find an item is the total number of items divided by the hashtable size (determined by hashpower), which is linear.
3. Memcached limits the maximum acceptable item to 1MB, and data larger than 1MB will be ignored.
4. Memcached’s space utilization has a great relationship with data characteristics, and is also related to the DONT_PREALLOC_SLABS constant. In the worst case, 198 slabs will be wasted (all items are concentrated in one slab, and all 199 IDs are fully allocated).
◎Memcached’s fixed-length optimization
Based on the descriptions in the above sections, I have a more in-depth understanding of memcached. It can only be optimized based on in-depth understanding.
Memcached itself is designed for variable-length data. According to the data characteristics, it can be said to be a "public-oriented" design. However, many times, our data is not so "universal". In typical cases, one is non-uniform distribution. , that is, the data length is concentrated in several areas (such as saving user sessions); the other more extreme state is equal-length data (such as fixed-length key values, fixed-length data, mostly seen in access, online statistics or execution locks).
Here we mainly study the optimization solution for fixed-length data (1.2). The centralized distributed variable-length data is for reference only and is easy to implement.
To solve fixed-length data, the first thing that needs to be solved is the slab allocation problem. The first thing that needs to be confirmed is that we don’t need so many slabs with different chunk lengths. In order to maximize the use of resources, it is best for chunks and items to be of equal length, so first To calculate the item length.
There has been an algorithm for calculating item length before. It should be noted that in addition to the string length, the length of the item structure of 32 bytes must be added.
Suppose we have calculated that we need to save 200 bytes of equal length data.
The next step is to modify the relationship between slab's classid and chunk length. In the original version, there is a corresponding relationship between chunk length and classid. Now if all chunks are set to 200 bytes, then this relationship does not exist. We need to re-determine the relationship between the two. One method is to use only a fixed ID for the entire storage structure, that is, only use 1 of the 199 slots. Under this condition, DONT_PREALLOC_SLABS must be defined to avoid additional pre-allocation waste. Another method is to establish a hash relationship to determine the classid from the item. You cannot use the length as the key. You can use variable data such as the NewHash result of the key, or directly do the hash based on the key (the keys of fixed-length data must also be of the same length). ). For the sake of simplicity here, we choose the first method. The disadvantage of this method is that only one ID is used. When the amount of data is very large, the slab chain will be very long (because all the data is crowded on one chain). The cost of traversing is relatively high.
The three types of space redundancy were introduced earlier. Setting the chunk length equal to the item length solves the first space waste problem. Not applying for space in advance solves the second space waste problem. So what about the first problem (remaining in the slab)? To solve this, you need to modify the POWER_BLOCK constant so that the size of each slab is exactly equal to an integer multiple of the chunk length, so that a slab can be divided into n chunks. This value should be closer to 1MB. If it is too large, it will also cause redundancy. If it is too small, it will cause too many allocs. According to the chunk length of 200, choose 1000000 as the value of POWER_BLOCK. In this way, a slab will be 1 million bytes, not 1048576. All three redundancy issues are solved, and space utilization will be greatly improved.
Modify the slabs_clsid function so that it directly returns a fixed value (such as 1):
CODE:[Copy to clipboard]unsigned int slabs_clsid(size_t size) {
return 1;
}
Modify the slabs_init function, remove the part that loops to create all classid attributes, and directly add slabclass[1]:
CODE:[Copy to clipboard]slabclass[1].size = 200; //200 bytes per chunk
slabclass[1].perslab = 5000; //1000000/200
◎Memcached client
Memcached is a service program. When using it, you can connect to the memcached server according to its protocol, send commands to the service process, and then operate the above data. For ease of use, memcached has many client programs available, corresponding to various languages, and there are clients in various languages. Those based on C language include libmemcache and APR_Memcache; those based on Perl include Cache::Memcached; there are also support for Python, Ruby, Java, C# and other languages. PHP has the most clients, not only the two extensions mcache and PECL memcache, but also a large number of encapsulation classes written by PHP. Here is an introduction to how to use memcached in PHP:
The mcache extension is re-encapsulated based on libmemcache. libmemcache has not released a stable version. The current version is 1.4.0-rc2, which can be found here. A very bad feature of libmemcache is that it writes a lot of error messages to stderr. Generally, when used as a lib, stderr is usually directed to other places, such as Apache's error log, and libmemcache will commit suicide, which may cause Abnormal, but its performance is still very good.
The mcache extension was last updated to 1.2.0-beta10. The author probably resigned. Not only did he stop updating, but he also couldn't open the website (~_~). He had to go elsewhere to get this irresponsible extension. After decompression, the installation method is as usual: phpize & configure & make & make install. Be sure to install libmemcache first. Using this extension is simple:
CODE:[Copy to clipboard]<?php
$mc = memcache(); // Create a memcache connection object. Note that new is not used here!
$mc->add_server('localhost', 11211); // Add a service process
$mc->add_server('localhost', 11212); // Add a second service process
$mc->set('key1', 'Hello'); // Write key1 => Hello
$mc->set('key2', 'World', 10); // Write key2 => World, expires in 10 seconds
$mc->set('arr1', array('Hello', 'World')); // Write an array
$key1 = $mc->get('key1'); // Get the value of 'key1' and assign it to $key1
$key2 = $mc->get('key2'); // Get the value of 'key2' and assign it to $key2. If it exceeds 10 seconds, it will not be available.
$arr1 = $mc->get('arr1'); // Get the 'arr1' array
$mc->delete('arr1'); // Delete 'arr1'
$mc->flush_all(); // Delete all data
$stats = $mc->stats(); // Get server information
var_dump($stats); // Server information is an array
?>
The advantage of this extension is that it can easily implement distributed storage and load balancing, because it can add multiple service addresses. When saving data, it will be located on a certain server based on the hash result. This is also a feature of libmemcache. libmemcache supports centralized hashing methods, including CRC32, ELF and Perl hash.
PECL memcache is an extension released by PECL. The latest version is 2.1.0, which can be obtained on the pecl website. The usage of memcache extension can be found in some newer PHP manuals. It is very similar to mcache, really similar:
CODE:[Copy to clipboard]<?php
$memcache = new Memcache;
$memcache->connect('localhost', 11211) or die ("Could not connect");
$version = $memcache->getVersion();
echo "Server's version: ".$version."n";
$tmp_object = new stdClass;
$tmp_object->str_attr = 'test';
$tmp_object->int_attr = 123;
$memcache->set('key', $tmp_object, false, 10) or die ("Failed to save data at the server");
echo "Store data in the cache (data will expire in 10 seconds)n";
$get_result = $memcache->get('key');
echo "Data from the cache:n";
var_dump($get_result);
?>
This extension uses PHP's stream to directly connect to the memcached server and send commands through the socket. It is not as complete as libmemcache, nor does it support distributed operations such as add_server, but because it does not rely on other external programs, it has better compatibility and is relatively stable. As for efficiency, the difference is not huge.
In addition, there are many PHP classes available, such as MemcacheClient.inc.php, and many can be found on phpclasses.org. They are generally re-encapsulation of the perl client API and are used in similar ways.
◎BSM_Memcache
From a C client perspective, APR_Memcache is a very mature and stable client program that supports thread locks and atomic-level operations to ensure operational stability. However, it is based on APR (APR will be introduced in the last section) and does not have as wide a range of applications as libmemcache. Currently, there are not many programs developed based on it. Most of the existing ones are Apache Modules because it cannot run outside the APR environment. However, APR can be installed separately from Apache. APR and APR-util can be downloaded from the APR website. Apache is not required and can be installed directly, and it is cross-platform.
BSM_Memcache is a PHP extension based on APR_Memcache that I developed in the BS.Magic project. It is a bit hard to say, but at least it incorporates APR into the PHP extension. This program is very simple and does not do too many functions. It is just a formal attempt to support server grouping.
Different from the mcache extension that supports multi-server distributed storage, BSM_Memcache supports multiple groups of servers. The servers in each group still distribute and save data according to the hash method, but the data saved in the two groups is the same, that is, hot backup is implemented. It will not cause data to be unavailable due to a single point of failure on one server, unless all server groups are damaged (such as a power outage in the computer room). Of course, the cost of implementing this function is the sacrifice of performance. Every time you add or delete data, all groups must be scanned. When getting data, a group of servers will be randomly selected to start polling until the data is found. Normally You can get it next time.
BSM_Memcache only supports these functions:
CODE:[Copy to clipboard]zend_function_entry bsm_memcache_functions[] =
{
PHP_FE(mc_get, NULL)
PHP_FE(mc_set, NULL)
PHP_FE(mc_del, NULL)
PHP_FE(mc_add_group, NULL)
PHP_FE(mc_add_server, NULL)
PHP_FE(mc_shutdown, NULL)
{NULL, NULL, NULL}
};
The mc_add_group function returns an integer (actually it should be an object, I was lazy~_~) as the group ID. When mc_add_server is used, two parameters must be provided, one is the group ID and the other is the server address (ADDRORT).
CODE:[Copy to clipboard]/**
*Add a server group
*/
PHP_FUNCTION(mc_add_group)
{
apr_int32_t group_id;
apr_status_t rv;
if (0 != ZEND_NUM_ARGS())
{
WRONG_PARAM_COUNT;
RETURN_NULL();
}
group_id = free_group_id();
if (-1 == group_id)
{
RETURN_FALSE;
}
apr_memcache_t *mc;
rv = apr_memcache_create(p, MAX_G_SERVER, 0, &mc);
add_group(group_id, mc);
RETURN_DOUBLE(group_id);
}
CODE:[Copy to clipboard]/**
* Add a server into group
*/
PHP_FUNCTION(mc_add_server)
{
apr_status_t rv;
apr_int32_t group_id;
double g;
char *srv_str;
int srv_str_l;
if (2 != ZEND_NUM_ARGS())
{
WRONG_PARAM_COUNT;
}
if (zend_parse_parameters(ZEND_NUM_ARGS() TSRMLS_CC, "ds", &g, &srv_str, &srv_str_l) == FAILURE)
{
RETURN_FALSE;
}
group_id = (apr_int32_t) g;
if (-1 == is_validate_group(group_id))
{
RETURN_FALSE;
}
char *host, *scope;
apr_port_t port;
rv = apr_parse_addr_port(&host, &scope, &port, srv_str, p);
if (APR_SUCCESS == rv)
{
//Create this server object
apr_memcache_server_t *st;
rv = apr_memcache_server_create(p, host, port, 0, 64, 1024, 600, &st);
if (APR_SUCCESS == rv)
{
if (NULL == mc_groups[group_id])
{
RETURN_FALSE;
}
//Add server
rv = apr_memcache_add_server(mc_groups[group_id], st);
if (APR_SUCCESS == rv)
{
RETURN_TRUE;
}
}
}
RETURN_FALSE;
}
When setting and delating data, loop through all groups:
CODE:[Copy to clipboard]/**
* Store item into all groups
*/
PHP_FUNCTION(mc_set)
{
char *key, *value;
int key_l, value_l;
double ttl = 0;
double set_ct = 0;
if (2 != ZEND_NUM_ARGS())
{
WRONG_PARAM_COUNT;
}
if (zend_parse_parameters(ZEND_NUM_ARGS() TSRMLS_CC, "ss|d", &key, &key_l, &value, &value_l, ttl) == FAILURE)
{
RETURN_FALSE;
}
// Write data into every object
apr_int32_t i = 0;
if (ttl < 0)
{
ttl = 0;
}
apr_status_t rv;
for (i = 0; i < MAX_GROUP; i++)
{
if (0 == is_validate_group(i))
{
// Write it!
rv = apr_memcache_add(mc_groups[i], key, value, value_l, (apr_uint32_t) ttl, 0);
if (APR_SUCCESS == rv)
{
set_ct++;
}
}
}
RETURN_DOUBLE(set_ct);
}
In mc_get, you first randomly select a group and then start polling from this group:
CODE:[Copy to clipboard]/**
* Fetch an item from a random group
*/
PHP_FUNCTION(mc_get)
{
char *key, *value = NULL;
int key_l;
apr_size_t value_l;
if (1 != ZEND_NUM_ARGS())
{
WRONG_PARAM_COUNT;
}
if (zend_parse_parameters(ZEND_NUM_ARGS() TSRMLS_CC, "s", &key, &key_l) == FAILURE)
{
RETURN_MULL();
}
// I will try...
// Random read
apr_int32_t curr_group_id = random_group();
apr_int32_t i = 0;
Apr_int32_t Try = 0;
apr_uint32_t Flag;
apr_memcache_t *Oper;
apr_status_t RV;
for (I = 0; I <max_group; i ++)
{
try = i + curr_group_id;
try = try % max_group;
if (0 == is_validate_group (try))
{
// get a value
ope = mc_groups [try];
RV = APR_MEMCACACHE_GETP (mc_groups [try], p, (const char *) key, & value, & value_l, 0);
If_success == RV)
{
Return_string (value, 1);
}
}
}
Return_false;
}
CODE: [Copy to Clipboard]/**
* Random Group ID
* For mc_get ()
*/
apr_int32_t random_group ()
{
Struct Timeval TV;
Struct Timezone TZ;
int usec;
gettimeofday (& TV, & tz);
usec = tv.tv_usec;
int Curr = usec % count_group ();
resturn (Apr_int32_t) curr;
}
The use of BSM_Memcache is similar to other clients:
CODE: [Copy to Clipboard] <? PHP
$ g1 = mc_add_group (); // Add the first group
$ g2 = mc_add_group (); // Add the second group
mc_add_server ($ g1, 'localhost: 11211'); // Add the first server to the first group
mc_add_server ($ g1, 'localhost: 11212'); // Add the second server to the first group
mc_add_server ($ g2, '10.0.0.16: 11211 '); // Add the first server to the second group
MC_SET (' key ',' hello ')
in the second group
; //
$ key = mc_get ('key'); // Read the data
mc_del ('key'); // Delete data
mc_shutdown (); // Close all groups
?>
The relevant information of Apr_Memcache can be found here, and BSM_Memcache can be downloaded on this site.
◎ The full name of APR environment introduction
APR: Apache Portable Runtime. It is a cross -platform C language library created and maintained by the Apache Software Foundation. It extracted from Apache httpd1.x and was independent of HTTPD, and Apache HTTPD2.X was built on APR. APR provides a lot of convenient API interfaces to use, including practical functions such as memory pools, string operations, networks, arrays, and hash tables. The development of Apache2 Module must contact many APR functions. Of course, APR can be used independently and used independently. It can be used to write its own application. It is not necessarily related to the Apache HTTPD.
◎ Postscript
This is my last article in the Lunar New Year (my fate). Because of the many connotations of Memcached, there must be a lot of omissions and errors in hastily organizing. Thank you for the research opportunities provided by Sina.com, and thank you for your help.
NP Dr. 02-13-2007