skraps at hushmail
Apr 8, 2012, 12:38 AM
Post #1 of 2
(179 views)
Permalink
|
|
[dbmail-memcache] - messages seperated into keys
|
|
Alright we got the messages seperated into chunks for the keys
tonight. Using the below it will separate the message into chunks
suitable for the keys.
while(buf_pos < msg_size+1){
if(buf_pos % KVCHUNK_SIZE == 0&&buf_pos !=0){
msg_buff[key_num][cbuf_pos]='';
key_num++;
cbuf_pos=0;
msg_buff[key_num][cbuf_pos]=buf[buf_pos];
}else{
msg_buff[key_num][cbuf_pos]=buf[buf_pos];
}
cbuf_pos++;
buf_pos++;
}
I'm trying to get bzip compression to work but it keeps returning a
-8(output buffer is full). Because I have a good feeling that 25-50%
of the mail can be compressed into one key. Unless its a 20MB chunk or
compressed base64 mime data then it turns into 11651 keys. Then I
don't think its feasible to use memcache, but to lay to load back to
the sql server for processing
I am trying to use the bzip2 library like so to compress the buffer.
char *cbuf=malloc(sizeof(*buf));
size_t cbuf_size=0;
int
ret=BZ2_bzBuffToBuffCompress(cbuf,&cbuf_size,buf,(size_t)strlen(buf),9,4,30);
printf("COMPRESSED MSG:nRETURN VALUE: %dnORIGINAL
SIZE:%dnDATA:%snCOMPRESS
SIZE:%dnDATA:%sm",ret,strlen(buf),buf,cbuf_size,cbuf);
free(cbuf);
It always returns a -8 anyone know what I am doing wrong here?
I know I am adding alot of processing into dbmail and that it takes
the load and places more on dbmail and memcache instead of mysql, I
think this might be the ticket though in a multi server cloud farm
where there is one mysql, instead of adding replicating mysql servers
you would only need to add memcache servers then clones of the mail
servers.
Shared web hosting cloud diagram layout
Load balancer
|
-------------------------------------------------------------------------------------------
| | |
| |
server 1 server 2 server 3 server 4
server 5
| | |
| |
--------------------------------------------------------------------------------------------
| |
|
| |
|
Mysql server |
memcache servers
| |
| |
----------------------------
|
|
NFS server(unlimited redundant
storage) for mail and user data
|
