Mailing List Archive: DBMail: users

Pipe to DBMAIL-SMTP

Index | Next | Previous | View Threaded

nabble.jm at themitchells

Sep 20, 2009, 10:13 PM

Post #1 of 4 (289 views)
Permalink

Pipe to DBMAIL-SMTP

Hello all,
I have researched a fair amount via the dbmail wiki, googling and searching
the mailing list archives and have yet to come across clear direction and am
seeking your collecive wisdom. I am considering the use of dbmail for a
project where I have email that needs to be processed and inserted into
mysql. I am thinking it would be very helpful to incorporate dbmail into
the mix to properly parse and insert the email into mysql. Ultimately what
I am trying to do seems very similar to what I am certain others have done
to port over existing mail from a remote IMAP server into a local dbmail
instance.

At present, I am accessing/processing multiple email boxes, one at a time,
via IMAP in a custom built script. As such, I have access to the raw email
message. I was hoping to be able to fetch the email via IMAP and pipe the
email directly to dbmail-smtp but I am uncertain what format dbmail-smtp is
expecting. Can I simply open a pipe to it and start shoving email across
the pipe? Though I wish it was that simple, I am guessing that is not the
case. Do you know what communications format dbmail-smtp is expecting when
it is being communicated with via a un*x pipe? Is is expecting a fully
formed SMTP interchange over the pipe connection?

Alternately, if I can not pipe directly to dbmail, do you have a suggested
path of least resistance? I have thought about writing the email out from
my custom script in maildir or similar format and using some middleman
software to read the emails from the filesystem and send them on to dbmail.
Presumably mailbox2dbmail would serve this purpose. This method is less
desirable to me because of the the intermediary involvement of the
filesystem but I am willing to consider it. I suppose it does have some
upside as I could use dbmail-lmtp as a daemon for receiving the files.

Of greatest importance, I prefer the email to be imported as-is. Meaning, I
do not want the headers disturbed or appended to by the intervening
processes. Where the process looks like the following: original IMAP
account --> Custom Script accessing via IMAP --> (intervening steps) -->
dbmail. Maybe, I should just be downloading the email from the remote IMAP
server and saving/inserting the messages via IMAP to the dbmail users
mailbox. Experience with any method suggested above or informed discussion
(pro or con) of other solutions is greatly appreciated.

Thank you in advance for your time and consideration.
--
View this message in context: http://www.nabble.com/Pipe-to-DBMAIL-SMTP-tp25530183p25530183.html
Sent from the dbmail users mailing list archive at Nabble.com.

_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

josh at worldhosting

Sep 20, 2009, 10:23 PM

Post #2 of 4 (274 views)
Permalink

Re: Pipe to DBMAIL-SMTP [In reply to]

I use and recommend:

http://home.arcor.de/armin.diehl/imapcopy/imapcopy.html

For copying emails from one IMAP server to another, regardless of what
the server it is. Note that it's best to do this at a time when you
don't expect emails to be accessed, as it's not a two-way sync software.
It just copies from one spot to another. But it does the job well.

Regards,
Josh.

_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

jesse at kci

Sep 21, 2009, 8:39 AM

Post #3 of 4 (272 views)
Permalink

Re: Pipe to DBMAIL-SMTP [In reply to]

Hello,

A few more comments/responses to your questions:

There are numerous options for copying, which you should find in the
documentation you've been searching. Imap->imap is a common way, though
you do have to know (or temporarily change) the account passwords. You
could do imap->dbmail-smtp (eg. I think fetchmail can do that, and
probably other tools). If you have a large amount of mail to move, it
would probably be faster to do so directly from the filesystem level of
the remote mail server, if that's accessible (using something like
mailbox2dbmail you mentioned).

The poorly named dbmail-smtp (more recenctly renamed dbmail-inject) does
not talk smtp, it takes a single message as stdin and stores it in the
database. Check the manpage for the format, I think it's just mbox (if
not, you just remove the "From " line 1 from mbox format to send it a
raw message).

If you wanted something you can talk smtp to, use dbmail-lmtp (it of
course talks lmtp, but does what you need for delivery). Performance is
better than spawning lots of processes/connections with dbmail-smtp, but
I don't think you can specify a mailbox to store to, everything goes to
INBOX (though I may be mistaken there). You could probably have a
custom migration script fiddle with sieve scripts on the backend to work
around that, if it's worth the effort (imap->imap would be less work).

--
Jesse Norell
Kentec Communications, Inc.
jesse[at]kci.net
_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

paul at nfg

Sep 21, 2009, 1:23 PM

Post #4 of 4 (269 views)
Permalink

Re: Pipe to DBMAIL-SMTP [In reply to]

brainjuice wrote:
> I was hoping to be able to fetch the email via IMAP and pipe the
> email directly to dbmail-smtp but I am uncertain what format dbmail-smtp is
> expecting. Can I simply open a pipe to it and start shoving email across
> the pipe? Though I wish it was that simple, I am guessing that is not the
> case. Do you know what communications format dbmail-smtp is expecting when
> it is being communicated with via a un*x pipe? Is is expecting a fully
> formed SMTP interchange over the pipe connection?

dbmail-smtp expects just a single raw rfc822 or mime-compliant message.
It is not designed for pipelining more messages in a single session. You
want to use dbmail-lmtpd for that, which expects to hear and talks LMTP
on the wire. You'd have to use a network socket, though in 2.3.x you
could also use a pipe (which I use to run functional tests against the
daemons) to hook things up.

> Alternately, if I can not pipe directly to dbmail, do you have a suggested
> path of least resistance? I have thought about writing the email out from
> my custom script in maildir or similar format and using some middleman
> software to read the emails from the filesystem and send them on to dbmail.
> Presumably mailbox2dbmail would serve this purpose. This method is less
> desirable to me because of the the intermediary involvement of the
> filesystem but I am willing to consider it. I suppose it does have some
> upside as I could use dbmail-lmtp as a daemon for receiving the files.

If you want to connect an imap store to dbmail insertion, fetchmail
could do the honors of dispatching imap retrieved mail. Using the pipe
interface (dbmail-smtp, aka dbmail-deliver) or is the easy way.
Insertion can be done unconditional but will spawn a delivery process
for each message.

You could also have fetchmail deliver into dbmail over lmtp. That would
most likely be faster, but insertion then becomes conditional.
Messages may bounce if there's no valid recipient for the message at
hand. So it depends.

> Of greatest importance, I prefer the email to be imported as-is. Meaning, I
> do not want the headers disturbed or appended to by the intervening
> processes.

Do your own tests. Dbmail doesn't append or delete any headers but for
one exception: Lmtp delivery adds the RFC required From_ envelope
header. But gmime has been observed to change ordering and
tab-formatting, and possibly even encoding. Nothing to break message
integrity though.

> Where the process looks like the following: original IMAP
> account --> Custom Script accessing via IMAP --> (intervening steps) -->
> dbmail. Maybe, I should just be downloading the email from the remote IMAP
> server and saving/inserting the messages via IMAP to the dbmail users
> mailbox. Experience with any method suggested above or informed discussion
> (pro or con) of other solutions is greatly appreciated.
>
> Thank you in advance for your time and consideration.

--
________________________________________________________________
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com