Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: ripMIME: general

ripmime making tons of files??

 

 

ripMIME general RSS feed   Index | Next | Previous | View Threaded


kitchen at scriptkitchen

Dec 8, 2004, 3:10 PM

Post #1 of 7 (5571 views)
Permalink
ripmime making tons of files??

I'm having an issue with ripmime and a relatively large email.. it's creating
26 text files and 15 of them are almost the same size as the original email,
and including most of the content of the original email. This is causing
major problems as this is being used in conjunction with clamav for scanning
inbound emails for viruses.

Here's an example:
[root [at] mai tmp2]# ls -l msg.1102542794.736261.1027
-rw-r--r-- 1 simscan root 10228587 Dec 8 16:55 msg.1102542794.736261.1027

[root [at] mai tmp2]# ripmime -i msg.1102542794.736261.1027

[root [at] mai tmp2]# ls -l
total 163324
-rw-r--r-- 1 root root 1944183 Dec 8 17:57 00537_RJ.JPG
-rw-r--r-- 2 root root 53208 Dec 8 17:57 applefile
-rw-r--r-- 1 root root 1221858 Dec 8 17:57 DSC_0509 copy_1.JPG
-rw-r--r-- 2 root root 53208 Dec 8 17:57 DSC_0509 copy.JPG
-rw-r--r-- 1 simscan root 10228587 Dec 8 16:55 msg.1102542794.736261.1027
-rw-r--r-- 1 root root 1 Dec 8 17:57 textfile0
-rw-r--r-- 1 root root 198 Dec 8 17:57 textfile1
-rw-r--r-- 1 root root 235 Dec 8 17:57 textfile10
-rw-r--r-- 1 root root 10224479 Dec 8 17:57 textfile11
-rw-r--r-- 1 root root 10224397 Dec 8 17:57 textfile12
-rw-r--r-- 1 root root 10224262 Dec 8 17:57 textfile13
-rw-r--r-- 1 root root 10224105 Dec 8 17:57 textfile14
-rw-r--r-- 1 root root 10224007 Dec 8 17:57 textfile15
-rw-r--r-- 1 root root 10223826 Dec 8 17:57 textfile16
-rw-r--r-- 1 root root 10223699 Dec 8 17:57 textfile17
-rw-r--r-- 1 root root 10223668 Dec 8 17:57 textfile18
-rw-r--r-- 1 root root 10223660 Dec 8 17:57 textfile19
-rw-r--r-- 1 root root 235 Dec 8 17:57 textfile2
-rw-r--r-- 1 root root 10223602 Dec 8 17:57 textfile20
-rw-r--r-- 1 root root 1 Dec 8 17:57 textfile21
-rw-r--r-- 1 root root 198 Dec 8 17:57 textfile22
-rw-r--r-- 1 root root 235 Dec 8 17:57 textfile23
-rw-r--r-- 1 root root 139 Dec 8 17:57 textfile24
-rw-r--r-- 1 root root 1041 Dec 8 17:57 textfile25
-rw-r--r-- 1 root root 0 Dec 8 17:57 textfile26
-rw-r--r-- 1 root root 10226899 Dec 8 17:57 textfile3
-rw-r--r-- 1 root root 10226817 Dec 8 17:57 textfile4
-rw-r--r-- 1 root root 10226653 Dec 8 17:57 textfile5
-rw-r--r-- 1 root root 10226367 Dec 8 17:57 textfile6
-rw-r--r-- 1 root root 10226308 Dec 8 17:57 textfile7
-rw-r--r-- 1 root root 1 Dec 8 17:57 textfile8
-rw-r--r-- 1 root root 198 Dec 8 17:57 textfile9

Is this normal? This is ending up using 160MB of storage space, as well as
hogging up a lot of cpu, and clamav is also eating up lots of cpu trying to
scan these files...

If you need the email causing the problem, I will have to contact the customer
to see if I can release it (but it would only be privately) for debugging
purposes, just let me know.

Thanks in advance.

-Jeremy

--
Jeremy Kitchen ++ Systems Administrator ++ Inter7 Internet Technologies, Inc.
jeremy [at] inter7 ++ www.inter7.com ++ 866.528.3530 ++ 815.776.9465 int'l
kitchen @ #qmail #gentoo on EFnet IRC ++ scriptkitchen.com/qmail
GnuPG Key ID: 481BF7E2 ++ jabber:kitchen [at] scriptkitchen


dallase at nmgi

Dec 8, 2004, 3:19 PM

Post #2 of 7 (5327 views)
Permalink
RE: ripmime making tons of files?? [In reply to]

>
> I'm having an issue with ripmime and a relatively large
> email.. it's creating
> 26 text files and 15 of them are almost the same size as the
> original email, and including most of the content of the
> original email. This is causing major problems as this is
> being used in conjunction with clamav for scanning inbound
> emails for viruses.
>

I had the exact same problem with an almost identical message. Although
it only expanded to like 15 files and took up about 50MB, but it damn
near kills the system when that message comes through... Ripmime didn't
seem to have any problems decoding them, but clamav runs forever... The
120 second alarm is the only thing that saves it.

If you decode that same message with reformime or something similar you
get the same results right? I believe I did. I cant remember if it was
something with the JPG compression or the mime structure, but I
diffinitely remember it was picture files from a camera, just like you
are seeing here.

And the other thing I remember is I could never get a resolution as to
why clamav took so damn long when uvscan scanned it in about 13 seconds.
I finally put a block on that users email addres until they called in :)
haha

Dallas

_______________________________________________
Ripmime-general mailing list
Ripmime-general [at] pldaniels
http://www.pldaniels.com/mailman/listinfo/ripmime-general


pldaniels at pldaniels

Dec 8, 2004, 5:40 PM

Post #3 of 7 (5353 views)
Permalink
Re: ripmime making tons of files?? [In reply to]

Hello all,

The problem lies with the oinion-shelling nature of ripMIME.

If you turn off things like --disable-qmail-bounce you'll notice that a lot of the repeated files go away.

I'm trying to find an effective way eliminate these repeats.

Regards.


--
PLDaniels - Software - Xamime
Unix systems Internet Development A.B.N. 19 500 721 806
PGP Public Key at http://www.pldaniels.com/gpg-keys.pld


dallase at nmgi

Dec 9, 2004, 6:16 AM

Post #4 of 7 (5361 views)
Permalink
RE: ripmime making tons of files?? [In reply to]

>
> The problem lies with the oinion-shelling nature of ripMIME.
>
> If you turn off things like --disable-qmail-bounce
> you'll notice that a lot of the repeated files go away.
>

I don't believe that was the case in my situation. I have been running
like this for over a year now,

-q --no-ole --disable-qmail-bounce --unique_names -i - -d
$ENV{'TMPDIR'}/

and this many large files issue was only a fews months ago. Jeremy,
are you running with --disable-qmail-bounce? If not, when you run that
file through with that option does the output change?

Thanks,

Dallas Engelken



_______________________________________________
Ripmime-general mailing list
Ripmime-general [at] pldaniels
http://www.pldaniels.com/mailman/listinfo/ripmime-general


pldaniels at pldaniels

Dec 9, 2004, 6:59 AM

Post #5 of 7 (5338 views)
Permalink
Re: ripmime making tons of files?? [In reply to]

Dallas,


> I don't believe that was the case in my situation. I have been running
> like this for over a year now,

Okay, would you like to forward to me a mailpack that exhibits these large/long files lists?

Ultimately it still tends to come down to the onion-peel nature of how ripMIME decodes the files. It's currently
"sometimes" an issue which can be controlled by the --recursion-max parameter (set it to 5 perhaps), however, because
MIME is a format capable of being nested these files do have a bad habit of multiplying.

Paul.




--
PLDaniels - Software - Xamime
Unix systems Internet Development A.B.N. 19 500 721 806
PGP Public Key at http://www.pldaniels.com/gpg-keys.pld


pldaniels at pldaniels

Dec 10, 2004, 9:55 PM

Post #6 of 7 (5357 views)
Permalink
Re: ripmime making tons of files?? [In reply to]

Have you ever noticed how you can forget even the simplest of things about a program that you've written yourself?

Try using the --no-nameless flag for removing the 'textfile*' files - while they'll still be generated during the
process, they will be culled at the end leaving you with just the derived attachments.

Paul.

--
PLDaniels - Software - Xamime
Unix systems Internet Development A.B.N. 19 500 721 806
PGP Public Key at http://www.pldaniels.com/gpg-keys.pld


pldaniels at pldaniels

Dec 16, 2004, 11:50 PM

Post #7 of 7 (5335 views)
Permalink
Re: ripmime making tons of files?? [In reply to]

Okay everybody,

I've not yet solved the problems that you guys are having, with the masses of files all different slightly (I'm working
on it).

The solution I'm working on is basically one where an output file will be flagged according to how it was generated.
In short, if it required no special translation (ie, not B64/QP/UU) it won't be specifically preserved, rather it'll
just leave the parent file to be the holder of the information. The trick is, I have to work out how to implement this
;-)

Paul.




--
PLDaniels - Software - Xamime
Unix systems Internet Development A.B.N. 19 500 721 806
PGP Public Key at http://www.pldaniels.com/gpg-keys.pld

ripMIME general RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.