Gossamer Forum
Home : Products : Gossamer Mail : Development, Plugins and Globals :

Problem with downloading messages

Quote Reply
Problem with downloading messages
Hi All,

I'm working on a Plugin for downloading messages either in the message list of while viewing a message. I ran into a problem with some of the downloaded messages though. HTML messages seem to have these ='s and =20's and some of the HTML source show once I have it downloaded. The plugin basically uses some borrowed code from GMail::Messages and GT::FileMan::Commands.pm to read in the raw message (or messages) from disk and send them/it to the browser (tarred and ziped if multiple messages).

Here is some of the output of a message:
Code:
Click on Folders to see a list of your current fo= lders. You now have one click trash buttons!
Try adding some addresses to our integrated addre= ss book. If you give a user a nickname,
then when you compose a message, you= can just enter in the nickname.
Create yourself a signature, this will get append= ed to the bottom of all your outgoing mail.
From the options screen you can set up a wide var= iety of features to get
Gossamer Mail working just the way you want.
Be sure to try adding a Remote POP account! This = lets you pull in mail from other accounts
into Gossamer Mail. Click on Check E= mail to get the email once you have setup a POP account.
If you have any questions at all about the demo, ple= ase visit the support= forum, or
contact = us for more information.

Cheers,

Alex Krohn
Gossamer Threads Inc.

</HTML

And here is a snippet of the code I use to make up the messages:

Code:
sub download {
#----------------------------------------------------------------
# download a message or messages

my $self = shift;

my @id = $IN->param('msgtrack_id') or return {message => 'DWNLDMSG_NOIDS'};
my $compression = $USER->{opts}->{download_messages}->{compression_type} || 'tar';

my $mode = 'auto';

if ($#id == 0 && !$USER->{opts}->{download_messages}->{compress_single}) {
my $file = $self->_file($id[0]);
return $self->_send_to_browser($file->{path}, $mode, "$file->{subject}-$file->{id}.eml");
}

my $tempfile = new GT::TempFile;

if ($compression eq 'zip' and $HAVE_AZIP) {
my $error = $self->_zip_process("$$tempfile.zip", \@id);
die "Failed to process zip" if ($error);
} else {
$self->_tar_process("$$tempfile.$compression", \@id);
}

$self->_send_to_browser("$$tempfile.$compression", $mode, "Messages.$compression");

return {message => 'DWNLDMSG_SUCCESS'}
}

sub _zip_process {
#--------------------------------------------------------------
# Create a .zip file

my ($self, $to, $ids) = @_;

require Archive::Zip::Tree;
my $zip = Archive::Zip->new();
my $member;
foreach my $id (@$ids) {
my $file = $self->_file($id);
next unless -f $file->{path};
$member = $zip->addFile($file->{path}, "Messages/$file->{subject}-$file->{id}.eml") or die "$!";
}

return $zip->writeToFileNamed($to) ? "$!" : 0;
}


sub _tar_process {
#--------------------------------------------------------------
# Create tar file

my ($self, $to, $ids) = @_;

# Make sure tar file goes out of scope and cleans up temp files
{
require GT::Tar;
my $tar = new GT::Tar($to) or die "Failed to create GT::Tar file";

foreach my $id (@$ids) {
my $file = $self->_file($id);
$tar->add_file($file->{path});
my $f = $tar->get_file($file->{path});
$f->{name} = "Messages/$file->{subject}-$file->{id}.eml";
}
$tar->write("$to");
}
}

sub _send_to_browser {
#----------------------------------------------------------------
# send the contents of a file to browser for downloading

my $self = shift;
my $send_file = shift;
my $mode = shift;
my $name = shift;

if ($mode eq 'auto') { $mode = (-T $send_file) ? 'ascii' : 'binary'; }

open SENDFILE, $send_file or die "Failed to open sendfile for output to browser: $!";
my $file_size = -s $send_file;
if (!$name) { ($name) = $send_file =~ m,/([^/]+)$,; }

print $IN->header( '-type' => 'application/download',
'-Content-Length' => $file_size,
'-Content-Transfer-Encoding' => $mode,
'-Content-Disposition' => \"attachment; filename=\"$name\"");

binmode STDOUT if ($^O eq 'MSWin32');
binmode SENDFILE;
my $buffer;
while (read(SENDFILE, $buffer, 500000)){
if ($mode eq 'ascii') { $buffer =~ s,\r\n,\n,g; }
print $buffer;
}
close SENDFILE;

return 1;
}

I'm not sure where to start looking at this point. Is it 1) A problem with how OutlookExpress is decoding the messages? 2) A problem with my code to grab the message or 3) A problem with how the message was generated to begin with?

Any help would be greatly appreciated. I've included the tar with all the code and files if you need it. The Plugin will install but it's still missing some parts/pieces to make it work off the shelf so it's not a good idea to install it as is.

Thanks,
Charlie
Quote Reply
Re: [Chaz] Problem with downloading messages In reply to
Hey Charlie.

This is what i was looking for in following Threads:

http://www.gossamer-threads.com/...Save%20Email;#265719

http://www.gossamer-threads.com/...Save%20Email;#265022

tgz compression mode eliminates the = shown though i am not sure what it could mean. I have it running and wow, it downloads messages many times faster that any Desktop Client would do (saving on Bandwidth too :) ). Perhaps GT would be able to comment of the Finer aspects of '=' in downloaded messages.

Thanks as it works fine for me after selecting the compression mode to tgz :) which i have set in prefs.cfg template so that all new signups get that by default.

Looking forward to Attachments Scanning Plugin as you had posted y'day that it would be easy to implement...

[EDIT]
It's much more than what's there on Horde-IMP where only the text part is Saved. Here Full HTML Messages could be downloaded and also, from the expanded archive, the Messages could be moved to Outlook Express .... Gr8
[/EDIT]


Anup

Last edited by:

anup123: Jul 12, 2004, 2:39 AM
Quote Reply
Re: [anup123] Problem with downloading messages In reply to
Hey Anup,

Glad it's working for you. I forgot to mention that the ='s and =20's go away in tgz mode. That makes me think it has something to do with the way I'm packing the file. I'll have to poke around some more. I'll hopefully have all the rough edges taken care of soon for this one and release a viable Plugin.

~Charlie
Quote Reply
Re: [Chaz] Problem with downloading messages In reply to
Looks like you're losing the newlines in the email. The ='s are part of the email itself (the emails are wrapped like that). Look at the original source of the email.

Adrian
Quote Reply
Re: [Chaz] Problem with downloading messages In reply to
Hi Charlie.

Maybe i keep asking for more but yet another HORDE-IMP Feature on GM with a difference though. This is wrt "Mailbox Maintenance" feature on HORDE-IMP.

HORDE-IMP:
Monthly Archiving Of Emails Folder Wise (Just restricted to Inbox And Sent)

Extension:
  • Monthly Compressed Archiving Of Emails In All But Trash Folder.
  • User Option To Set It On/Off (Archive Automatically)
  • Compressed Archives Available For Download (Through Your Already In Place DownLoad Plugin)
  • Auto Archiving (even if user has set Archiving to off) in case Mailbox is 90% full, so that compressed archive (oldest 90 days mails, followed by increments of 30 days if not much mails in earliest 90 days) and Mails Deleted After Archiving. This would at least Free Up Space and Coupled With Point Below would Theoretically mean no mails and present GM functionality from users point of view affected.
  • Restore Archive Functionality So that if a user wants to Restore the Mails to Pre-Archive State it can be done.

I Can Guess That This Can't Be a *Free Plugin*. The Compressed Archives could be in /data/users/domain/.../username/archive directory or some other location consistent with GM.

A *Big Jump* over Horde's feature And A Big Enhancement To GM :)

Thanks
Anup
Quote Reply
Re: [brewt] Problem with downloading messages In reply to
Hey Adrian,

Below is what the message looks like on disk. I have tried ASCII, binary and auto mode for download but nothing seems to make a difference. Is there something additional or different that I should be doing in my _send_to_browser routine maybe?

Thanks,
Charlie

[message]
From: gmail@gmail_domain.com
Subject: Welcome to Gossamer Mail user
To: me@myisp.com
Content-Type: multipart/related; type="multipart/alternative";
boundary="---------=_985209202-12592-88589201"
Content-Transfer-Encoding: binary
Mime-Version: 1.0

This is a multi-part message in MIME format.

-----------=_985209202-12592-88589201
Content-Type: multipart/alternative; boundary="---------=_985209202-12592-439281353";
boundary="---------=_985209202-12592-439281353"

This is a multi-part message in MIME format.
-----------=_985209202-12592-439281353
Content-Type: text/html; charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable

<HTML>
<HEAD>
<META NAME=3D"GENERATOR" Content=3D"Gossamer Threads HTML Editor - http://g=
=
ossamer-threads.com">
<TITLE></TITLE>
</HEAD>
<BODY>
<P><A href=3D"http://www.gossamer-threads.com/scripts/webmail/"><IMG=20
alt=3D"Powered By Gossamer Mail!"=20
src=3D"cid:3678ef537253dcb0a0ba5f2d1a93393d"=20
align=3Dright border=3D0></A><FONT face=3DTahoma,Arial,Helvetica=20
size=3D4><STRONG>Welcome to Gossamer Mail 2!</STRONG></FONT></P>
<P><FONT face=3DTahoma,Arial,Helvetica>Please be sure to try out all the fe=
=
atures=20
in your email account like:</FONT></P>
<UL>
<LI><FONT face=3DTahoma>When viewing the list of messages, you can easily=
=
change=20
the sort orders, as well as mark messages read, or mass move and delete=
=
=20
mail.</FONT>=20
<LI><FONT face=3DTahoma>Composing new mail. If you are on IE, you can try=
=
to=20
send HTML mail! This entire email was generated using our HTML editor (in=
=
line=20
images and all).</FONT>=20
<LI><FONT face=3DTahoma>Click on Folders to see a list of your current fo=
=
lders.=20
You now have one click trash buttons!</FONT>=20
<LI><FONT face=3DTahoma>Try adding some addresses to our integrated addre=
=
ss=20
book. If you give a user a nickname, then when you compose a message, you=
=
can=20
just enter in the nickname.</FONT>=20
<LI><FONT face=3DTahoma>Create yourself a signature, this will get append=
=
ed to=20
the bottom of all your outgoing mail.</FONT>=20
<LI><FONT face=3DTahoma>From the options screen you can set up a wide var=
=
iety of=20
features to get Gossamer Mail working just the way you want. </FONT>
<LI><FONT face=3DTahoma>Be sure to try adding a Remote POP account! This =
=
lets=20
you pull in mail from other accounts into Gossamer Mail. Click on Check E=
=
mail=20
to get the email once you have setup a POP account.</FONT></LI></UL>
<P><FONT face=3DTahoma>If you have any questions at all about the demo, ple=
=
ase=20
visit the <A href=3D"http://www.gossamer-threads.com/perl/forum/">support=
=
=20
forum</A>, or <A href=3D"http://www.gossamer-threads.com/contact/">contact =
=
us</A>=20
for more information.</FONT></P>
<P><FONT face=3DTahoma>Cheers,</FONT></P>
<P><FONT face=3DTahoma>Alex Krohn<BR>Gossamer Threads=20
Inc.</FONT></P>
</BODY>
</HTML>
-----------=_985209202-12592-439281353
Content-Type: image/gif; name="gmail.gif"
Content-Transfer-Encoding: Base64
Content-Id: <3678ef537253dcb0a0ba5f2d1a93393d>

R0lGODlhXAApAOcAAP39/f39yv39l/39ZP39Mf39AP3K/f3Kyv3Kl/3KZP3K
Mf3KAP2X/f2Xyv2Xl/2XZP2XMf2XAP1k/f1kyv1kl/1kZP1kMf1kAP0x/f0x
yv0xl/0xZP0xMf0xAP0A/f0Ayv0Al/0AZP0AMf0AAMr9/cr9ysr9l8r9ZMr9
Mcr9AMrK/crKysrKl8rKZMrKMcrKAMqX/cqXysqXl8qXZMqXMcqXAMpk/cpk
yspkl8pkZMpkMcpkAMox/coxysoxl8oxZMoxMcoxAMoA/coAysoAl8oAZMoA
McoAAJf9/Zf9ypf9l5f9ZJf9MZf9AJfK/ZfKypfKl5fKZJfKMZfKAJeX/ZeX
ypeXl5eXZJeXMZeXAJdk/Zdkypdkl5dkZJdkMZdkAJcx/Zcxypcxl5cxZJcx
MZcxAJcA/ZcAypcAl5cAZJcAMZcAAGT9/WT9ymT9l2T9ZGT9MWT9AGTK/WTK
ymTKl2TKZGTKMWTKAGSX/WSXymSXl2SXZGSXMWSXAGRk/WRkymRkl2RkZGRk
MWRkAGQx/WQxymQxl2QxZGQxMWQxAGQA/WQAymQAl2QAZGQAMWQAADH9/TH9
yjH9lzH9ZDH9MTH9ADHK/THKyjHKlzHKZDHKMTHKADGX/TGXyjGXlzGXZDGX
MTGXADFk/TFkyjFklzFkZDFkMTFkADEx/TExyjExlzExZDExMTExADEA/TEA
yjEAlzEAZDEAMTEAAAD9/QD9ygD9lwD9ZAD9MQD9AADK/QDKygDKlwDKZADK
MQDKAACX/QCXygCXlwCXZACXMQCXAABk/QBkygBklwBkZABkMQBkAAAx/QAx
ygAxlwAxZAAxMQAxAAAA/QAAygAAlwAAZAAAMQAAAP//////////////////
////////////////////////////////////////////////////////////
////////////////////////////////////////////////////////////
/////////////////////ywAAAAAXAApAAAI/gABCBxIsKDBgwgTKlzIsKHD
hxAjSpxIsaLFixgzatzIsSPDFR5DXsxTheAoUiJTPqzirGSVUQP/OHOmQqVN
gy9VAcjjDCYAkACezGx50+aTP3lUDFWxogoJgiuGOvtTVCTPoQBIkTyodObV
qiFlOlOVVCEprwBmlgTLUagzrUAPjppJFYDMumw3Sp05qkrcnTNJPQXA0lle
jmL38q1SmCbBmU8Oa3RLKrFiZ5EJnsUr+SJfgSpenh3KWWDhzhlHG1RxtLTA
q5lRVyy8duGTKlenyrbY1bVBVZDfxm64Im5xgUBBKj9ePDnz41CTI0+o+uCK
ygDmqlJRMyKrQKwA/lj5Hj6QlRWBAIS/Fii9FfAg2buHP3DFdysAAq2wgh/h
1doAqJBHUDMRNpx39uUnkH7oBXINeuqdl997+LEioX7q1XeNhQJ9p1BXA9Zn
WFohWnSNhOHlBxIrFnLIClDtpfjigsWlh9x5Npq3UHUDYbZRiuLFuGB613TY
nnj7VXjkexwi952NEioEW0p/QQcVcsxN19xPf2FZH0O97XYYWYOJaeaZaFp0
wF9rHjDQAXC+ySZBBxggkJt3rgBnnW0S1F2Afg7EVIB/ToSDGmoYoYZAPySq
Bg4AxICoGjIYgKgRP9ypBp4AyJApAD/I8JOjlMrgKBB2KmUaSgKR4CNh/oaR
wupEDqiRwwENiGqqDAfkYISnPwTQwAGmwgnUoTkM1CgAB6iRabO8wnkosZTC
WtJZA7FUF0/Z+TRRDosSBMSnADwKrqidGpHsQGQoOhAQlMZgBBk//apnp4se
YASkPP3RFVCjPcVSVrNK9MOzB5tKrrMHwPuDm+BWK4MRkorarK3gGmFAs4rq
qqgaZLg5Gm0EslQSt6P45lAO9Bqwq7oD2SqQqes2vOgPIWNab8bg6rnpQIfW
uu5UfDkzoExKoTSwrBXh8Cuzv04LwKEN3HvwT1NvSuqmh5a76b4rGBGXqVNT
HNUTZz1xUldDPYGytxM1Oimv8CKarK+IymCqT6I/SB2ArVfnkOwYD1/66MQC
AUHGwIy39OpUS0tFkQwO8CqQyzI0cLneeFZuMafGuhnn6G0GYABQGw9KWIAr
qM76oEzFnubstNdu++1nBgQAOw==

-----------=_985209202-12592-439281353--

-----------=_985209202-12592-88589201--

[/message]
Quote Reply
Re: [brewt] Problem with downloading messages In reply to
I think I've got it. It's not losing the new lines it's just that they are not Windows new lines :/ If I change the \n's to \r\n's it opens fine. I suppose I will need to put in another user option in to set unix or windows new lines and adjust accordingly.

Thanks for pointing me in the right direction.

~Charlie