Gossamer Forum
Home : Products : Gossamer Mail : Discussion :

consistency.pl and purge.pl inconsistencies

(Page 2 of 2)
> >
Quote Reply
Re: [brewt] consistency.pl and purge.pl inconsistencies In reply to
Hi Adrian,

Would you want me to send the msgs.pm and consistency.pl files of my install to you. In case any other files (which you think could _possibly_ be causing this) from my installation is needed pl do let me know.

Thnx

Anup
Quote Reply
Re: [anup123] consistency.pl and purge.pl inconsistencies In reply to
where can i find consistancy.pl file on my gmail
Quote Reply
Re: [aghajunaid1] consistency.pl and purge.pl inconsistencies In reply to
Hi,

In case u have shell access (which i presume u must be having in order to execute), then:

path_to_batch/tools is where the consistency.pl is.

Anup
Quote Reply
Re: [anup123] consistency.pl and purge.pl inconsistencies In reply to
Hi!

Just to draw your attention to what I vaguely remember on one thing I found years ago.

Could you check the user + group of the directories >> After the second level <<< of the messages? like

a/2/s >>> s >>> user = ?
Quote Reply
Re: [dearnet] consistency.pl and purge.pl inconsistencies In reply to
Hi,

At the end of the day, it would be more refreshing if some solution could be found.

I am waiting sitting on the edge of my seat for quite some time now. Sometimes do some iterations (free time) to see if any pattern could be found but of no avail. There's always a ray of hope.

Anup
Quote Reply
Re: [anup123] consistency.pl and purge.pl inconsistencies In reply to
Hi!

Probably I need to explain.

The reason why I asked to figure out, is because the script created user = nobody in the second or third level. The first level was created as user = userid under telnet. I do not know how and why. During installation I used telnet and therefore all should be under userid.

I just had a thought that the messages were getting deleted from the database and the script could not delete from the disk because the script user = userid and the messages user = nobody. So the purge function got stopped due to the ownership conflict problems.

However, I have written this thought on this forums years ago and GT has never reacted to even say if this has been read or if that was a wrong idea, which could be as I am not an expert. Having no answer, I assumed that the thought did not lead to any further analysis and did not have any need for further discussion. One also cannot expect discussions or answers to all the threads from GT.

Thats why I had an idea of asking you. Thats also the reason why there was a reaction that GT Staff could not see what you see or me have seen on the problem. However, I understand your feeling very much. The difference in the system environments is really hard to figure out where the hell a problem is.
Quote Reply
Re: [dearnet] consistency.pl and purge.pl inconsistencies In reply to
Hi,

If messages can be written to disk, then the same should be deleted. At least that's how a layman would put it. If it can be written but cannot be deleted (that too not having any fixed pattern), then the only thing that could cause this is the permission on the file itself.

To my surprise, i just checked the /data/msgs tree and found that some directory are owned by nobody and some by userid. The messages themselves are owned by userid.

/data/msgs level:

drwxrwxrwx 18 nobody domain10 4096 Jun 18 2002 .
drwxr-xr-x 7 scorpioi domain10 4096 Jun 18 2002 ..
drwxrwxrwx 18 nobody domain10 4096 Sep 17 2002 0
drwxrwxrwx 18 nobody domain10 4096 Jul 29 2002 1
drwxrwxrwx 18 nobody domain10 4096 Jul 26 2002 2
drwxrwxrwx 18 nobody domain10 4096 Aug 13 2002 3
drwxrwxrwx 18 nobody domain10 4096 Aug 18 2002 4
drwxrwxrwx 18 nobody domain10 4096 Aug 3 2002 5
drwxrwxrwx 18 nobody domain10 4096 Jul 21 2002 6
drwxrwxrwx 18 nobody domain10 4096 Aug 3 2002 7
drwxrwxrwx 18 nobody domain10 4096 Jul 30 2002 8
drwxrwxrwx 18 nobody domain10 4096 Aug 9 2002 9
drwxrwxrwx 18 scorpioi domain10 4096 Jul 27 2002 a
drwxrwxrwx 18 nobody domain10 4096 Jul 30 2002 b
drwxrwxrwx 18 nobody domain10 4096 Jul 29 2002 c
drwxrwxrwx 18 nobody domain10 4096 Jul 24 2002 d
drwxrwxrwx 18 scorpioi domain10 4096 Sep 7 2002 e
drwxrwxrwx 18 nobody domain10 4096 Jul 21 2002 f

/data/msgs/0 level:

drwxrwxrwx 18 nobody domain10 4096 Sep 17 2002 .
drwxrwxrwx 18 nobody domain10 4096 Jun 18 2002 ..
drwxrwxrwx 18 nobody domain10 4096 Jan 13 20:20 0
drwxrwxrwx 18 nobody domain10 4096 Jan 25 10:30 1
drwxrwxrwx 18 nobody domain10 4096 Dec 23 21:35 2
drwxrwxrwx 18 nobody domain10 4096 Jan 3 04:35 3
drwxrwxrwx 18 nobody domain10 4096 Jan 5 04:45 4
drwxrwxrwx 18 scorpioi domain10 4096 Nov 19 2002 5
drwxrwxrwx 18 nobody domain10 4096 Nov 11 2002 6
drwxrwxrwx 18 nobody domain10 4096 Dec 18 15:30 7
drwxrwxrwx 18 scorpioi domain10 4096 Dec 18 15:50 8
drwxrwxrwx 18 nobody domain10 4096 Dec 3 2002 9
drwxrwxrwx 18 nobody domain10 4096 Jan 16 20:25 a
drwxrwxrwx 18 scorpioi domain10 4096 Apr 26 13:10 b
drwxrwxrwx 18 nobody domain10 4096 Nov 30 2002 c
drwxrwxrwx 18 nobody domain10 4096 Feb 3 10:50 d
drwxrwxrwx 18 scorpioi domain10 4096 Dec 9 09:20 e
drwxrwxrwx 18 nobody domain10 4096 Feb 24 05:30 f

/data/msgs/0/0 level:

drwxrwxrwx 18 nobody domain10 4096 Jan 13 20:20 .
drwxrwxrwx 18 nobody domain10 4096 Sep 17 2002 ..
drwxrwxrwx 2 scorpioi domain10 4096 May 22 04:45 0
drwxrwxrwx 2 scorpioi domain10 4096 Jun 2 01:16 1
drwxrwxrwx 2 scorpioi domain10 4096 May 26 05:45 2
drwxrwxrwx 2 scorpioi domain10 4096 Feb 21 04:30 3
drwxrwxrwx 2 scorpioi domain10 4096 May 17 03:15 4
drwxrwxrwx 2 scorpioi domain10 4096 May 27 23:37 5
drwxrwxrwx 2 scorpioi domain10 4096 May 27 00:12 6
drwxrwxrwx 2 nobody domain10 4096 Jun 3 05:20 7
drwxrwxrwx 2 scorpioi domain10 4096 Jun 5 09:08 8
drwxrwxrwx 2 nobody domain10 4096 Jun 2 01:16 9
drwxrwxrwx 2 scorpioi domain10 4096 Jun 6 02:45 a
drwxrwxrwx 2 scorpioi domain10 4096 Jun 2 01:16 b
drwxrwxrwx 2 scorpioi domain10 4096 May 12 07:10 c
drwxrwxrwx 2 scorpioi domain10 4096 Jun 5 18:05 d
drwxrwxrwx 2 scorpioi domain10 4096 Jun 1 14:25 e
drwxrwxrwx 2 scorpioi domain10 4096 Jun 2 18:54 f

/data/msgs/0/0/8

drwxrwxrwx 2 scorpioi domain10 4096 Jun 5 09:08 .
drwxrwxrwx 18 nobody domain10 4096 Jan 13 20:20 ..
-rw-rw-rw- 1 scorpioi domain10 5094 Jan 25 05:35 0085b15bbcbaeacb3a4b08f2c111ee3a
-rw-rw-rw- 1 scorpioi domain10 3602 Jan 25 05:35 0085b15bbcbaeacb3a4b08f2c111ee3a.1
-rw-rw-rw- 1 scorpioi domain10 4048 Mar 28 02:55 008b68906c3c29eecb1fb365a39826b0
-rw-rw-rw- 1 scorpioi domain10 3107 Mar 28 02:55 008b68906c3c29eecb1fb365a39826b0.1

If this could provide some clue. Probably running consistency.pl from shell is changing the ownership of directories where the corrections are being done. Also i had posted a data set about the entries in tables to which there has been no comment.

BTW: The incoming.pl working copy (Installed By GT) is ownership scorpioi and the backup copy is under nobody (left by GT on system). Should the ownership of working copy be changed to nobody? What happens to the directories above which are not owned by nobody?

Could GT throw some light as to the details posted above are as per expectation or not. If not then why are directories taking different ownership. Could this be a reason why there is no pattern of error when the messages are purged?

Anup

Last edited by:

anup123: Jun 6, 2003, 3:27 AM
Quote Reply
Re: [brewt] consistency.pl and purge.pl inconsistencies In reply to
Hello Adrian!

Earlier I could not explain you when we talked about the permission issue.

But this time, Anup has, to my surprise an exactly the same pattern. What I do not know is if this is the case also in yout test installation. Is this Normal? Could it be something in there that you in your server do not and cannot see things that we have of user permissions?

Then the best is to change the permissions the way how Anup published on your server and see if it could be there.

Just a try to get rid of this irritating thing of purge.

Could it be some Module that is on our server as a default installation that has a different version as a part of an ENV variable different to yours, any speculation?

Is there a way to print a message AFTER the consistency.pl checks and find out which EXACTLY are the messages that are on the disk.

My problem was also that I could not delete the messages owned by the user = nobody and I did not dare to tell my provider sending a long list of those thousands of message data still hanging on the disk out of an amazing phenomina.
Quote Reply
Re: [anup123] consistency.pl and purge.pl inconsistencies In reply to
Hi Anup!

Pl. do not get upset if what I am asking in terms of tracing "Is for nothing". I am simply trying to eastabilish some consistent points between our inconsistency of consistency to be handled by inconsistency.pl! Wink
  • Could you see and publish user permission + ownership of the following two directories:
  1. /data/msgs/0/0/7
  2. /data/msgs/0/0/9
  • Try to see which are the messages listed in there.
  • Figure out if they are in the database tables.


Figuring them in the database tables is a big pain... But mostlikey you have inconsistency that your script inconsistency.pl cannot handle and delete those messages of the users which are having different ownerships. But thats only my wild guess. I could be wrong. I reported a long time ago as seen in the following threads:

Jul 17, 2001, 11:30 AM
http://www.gossamer-threads.com/...nsistency.pl;#151524

Aug 22, 2001, 8:09 PM
http://www.gossamer-threads.com/...nsistency.pl;#155729

Jul 8, 2002, 12:25 PM
http://www.gossamer-threads.com/...nsistency.pl;#204521

Jul 23, 2002, 8:51 AM
http://www.gossamer-threads.com/...nsistency.pl;#207022

Jul 26, 2002, 11:31 AM
http://www.gossamer-threads.com/...nsistency.pl;#207579

Dec 7, 2002, 11:23 AM
http://www.gossamer-threads.com/...nsistency.pl;#225093

Last edited by:

dearnet: Jun 6, 2003, 3:33 AM
Quote Reply
Re: [dearnet] consistency.pl and purge.pl inconsistencies In reply to
Hi,

Why should i get annoyed unless it is a useless pro vs anti duel which have mostly been started in response to my posts. In fact this thread itself has a session where a user with barely 100 links in 5 years in LSQL and who has not used GM for two years had enough of time to enter into a lengthy duel.

Here you go for the details needed:

/data/msgs/0/0/7

drwxrwxrwx 2 nobody domain10 4096 Jun 3 05:20 .
drwxrwxrwx 18 nobody domain10 4096 Jan 13 20:20 ..
-rw-rw-rw- 1 scorpioi domain10 1876 Jun 3 05:20 0073214abd879c534bb26529b3e6a836
-rw-rw-rw- 1 scorpioi domain10 1293 Jun 3 05:20 0073214abd879c534bb26529b3e6a836.1
-rw-rw-rw- 1 scorpioi domain10 1506 May 25 04:40 0075ce3d835132172d0826edea331c67
-rw-rw-rw- 1 scorpioi domain10 693 May 25 04:40 0075ce3d835132172d0826edea331c67.1
-rw-rw-rw- 1 scorpioi domain10 1259 May 20 02:20 00792e4f133fcadbd82cc0db37b4f2ad
-rw-rw-rw- 1 scorpioi domain10 467 May 20 02:20 00792e4f133fcadbd82cc0db37b4f2ad.1
-rw-rw-rw- 1 scorpioi domain10 4757 Oct 14 2002 0079467c7d80225eb10282a03b6a662e
-rw-rw-rw- 1 scorpioi domain10 3389 Oct 14 2002 0079467c7d80225eb10282a03b6a662e.1
-rw-rw-rw- 1 scorpioi domain10 2740 May 17 09:40 007c5e6acfb0c432590aa1e2dc566134
-rw-rw-rw- 1 scorpioi domain10 952 May 17 09:40 007c5e6acfb0c432590aa1e2dc566134.1
-rw-rw-rw- 1 scorpioi domain10 603 May 17 09:40 007c5e6acfb0c432590aa1e2dc566134.2

/data/msgs/0/0/9

drwxrwxrwx 2 nobody domain10 4096 Jun 2 01:16 .
drwxrwxrwx 18 nobody domain10 4096 Jan 13 20:20 ..
-rw-rw-rw- 1 scorpioi domain10 5013 Jan 23 15:35 009a3fa7ffca17185c5424648ae23279
-rw-rw-rw- 1 scorpioi domain10 3544 Jan 23 15:35 009a3fa7ffca17185c5424648ae23279.1

I know checking if these are the database messages also is a big pain. But since consistency.pl is not reporting any inconsistency (which only happens when there is a admin purge) i safely assume that these must be there though i had posted some details about data in tables and have had no reply so far on that.

My incoming.pl is under userid (which by default install should have been under nobody) but seems to be there as originally i was told I had wrong version of incoming.pl when GT stepped in to see why mesages were not coming in....to GM accounts and replaced the original one with the correct one. There are few .pm files also which are under userid ownership. God knows if this could be the whole cause of this NIGHTMARE......And We are expected to push towards a solution giving these findings (only during free time which is scarce commodity).

Hope there is an answer from GT on this....and a solution perhaps...

Anup
Quote Reply
Re: [anup123] consistency.pl and purge.pl inconsistencies In reply to
Hi Anup!

Thanks for a positive approach.

The third level in your case of 7 and 9 directory has user pattern. The best is to have a FTP client which shows the ownership. I advise all following this thread to STRNGLY USE this FTP client from http://www.winscp.vse which gives a true SSH + FTP + ownerships! I have a reference of all possible patterns. I advise all users + GT to see this pattern. Its like:

user/user/user

user/user/nobody

user/nobody/nobody

user/nobody/user

The last one is the message file. What interesting would be to identify if the messages deleted from the database i.e. tables first and then later on the files on the disk is deleted. Which is the pattern that cannot delete the files. I very much suspect that the pattern user/nobody/user would not be able to delete those messages and gets killed or goes further.

But this Adrian or Alex can throw light on this speculation. Anathor thing would be also to actually show which messages are being deleted --verbose style. Or to delete the messages from the disk first and delete the users last would be the change in the design.
Quote Reply
Re: [dearnet] consistency.pl and purge.pl inconsistencies In reply to
Ack, the SSH + FTP client + Ownerships is available from the following URL:

http://winscp.vse.cz

Its great. WinSCP is a freeware SCP (Secure CoPy) client for Windows 95/98/ME/NT/2000/XP using SSH (Secure SHell).

Last edited by:

dearnet: Jun 6, 2003, 4:21 AM
Quote Reply
Re: [dearnet] consistency.pl and purge.pl inconsistencies In reply to
Hi Rajani,

Many of your posts have gone unanswered or unattended. But i have not touched the "Purge User" feature as i do not want to be wiped out of market with valid paid members being kicked out from the system for no fault of theirs or administrators.

I hope some Serious Thought Is put into this "Purge Issue" instead of treating it a trivial matter. The problem does not get over if the same cannot be duplicated. This is the reason why so much pain is being taken to publish the findings which gets no answer except from users who have burnt their fingers exploring certain features....or uninvited remarks which take the thread to star wars type of a scenario.

Check The Admin/Validate/Delete option .... the user gets deleted from the database and never from the disk. I figured this out pretty early and spent a day cleaning the system manually as consistency.pl never reports that. Imagine taking an unlimited domain license and having more than 100 domains and then one day it boomerangs on to you.....

In my case, installation was from web, no suEXEC/no cgi-wrapper etc etc and here i have some directory owned by nobody and some by userid that too within the msgs tree where the only directory which the user creates is data and the rest is all written by GM.

I keep coming back to this issue whenever i have free time. Adrian said that there is some free time to work on GM and hope that this issue is attended to in this session....

Anup
Quote Reply
Re: [dearnet] consistency.pl and purge.pl inconsistencies In reply to
Hi,

have reported all the details thru ssh. Using secure crt from vandyke.com

thnx

Anup
Quote Reply
Re: [dearnet] consistency.pl and purge.pl inconsistencies In reply to
Take another inconsistency:

Ran consistency.pl (though no purge was done since last one ab two months back). the start shows:

Messags in database: 6102

The stats of admin has a different story:

Message Counts (does not include deleted messages): Total 6211 Incoming 5786 Outgoing 425

The webmail_msgs table shows 6102

Why is Admin interface not consistent with the data?
Anup

Last edited by:

anup123: Jun 6, 2003, 5:18 AM
Quote Reply
Re: [anup123] consistency.pl and purge.pl inconsistencies In reply to
Hi!

In my case, it is the

The webmail_msgtrack table shows 6102

would show the statistics of Total and not the webmail_msgs table. FUNNY

It is the MSGS table that actually is the main table that gives birth to an ID for a message. MSGTRACK table is the property of that message like replied, seen, etc and helps the searches.

Users do exists but do not have messages, hence they are not kicked out due to the famous BUG. But the messages are still not getting deleted. Users do NOT exists but their messages do! I have used the new Gmail after the BUG reported by me was removed. So I do not really know whats going on.

What is common in between us is that there are inconsistency in the database that should not have happened. Lets wait untill GT comes out with some clever ideas, or may be a new version is in planning.

Just tried to run consistency.pl --fix to figure out what the case now. It gives the following:

Could not unlink /usr/.../checksum Reason Permission denied at line 272 of consistency.pl!

without fix, it showed:

43 messages were on disk and not in the database.

1886 messages were in the database but not on disk

msgs table inconsistencies: 1996

those figures do not match to any table in the database figures EXCEPT Messages in database!

Last edited by:

dearnet: Jun 6, 2003, 7:01 AM
Quote Reply
Re: [dearnet] consistency.pl and purge.pl inconsistencies In reply to
Hi,

Quote:
1886 messages were in the database but not on disk

So what does a user with one of those missing messages on disk do. Doesn't it cause "Professional Embarassment"

I am surprised to find you and me as lone participants in this thread and find no comment from GT on the ownership issues data published.


Anup
Quote Reply
Re: [dearnet] consistency.pl and purge.pl inconsistencies In reply to
Quote:
The last one is the message file. What interesting would be to identify if the messages deleted from the database i.e. tables first and then later on the files on the disk is deleted. Which is the pattern that cannot delete the files. I very much suspect that the pattern user/nobody/user would not be able to delete those messages and gets killed or goes further.

If that be the case then when u run empty folder then also the messages must be there. In case emty folder works fine from admin, then "Purge" is definitely causing the inconsistency.

In my case, every action from admin (except purge) which calls for deleting messages works fine. Emty Folder, delete messages etc does not cause any inconsistency.

With this in view, I am not ready to believe that it could be the server setup which is casuing this funny behaviour with "Purge" Also i am living with this mysterious hybrid ownership issues.

I_INSTALLED_FROM_WEB_SO_LOGICALLY_IT_SHOULD_BE_NOBODY but i have a mixed bag....

Any thoughts on this???

Anup
Quote Reply
Re: [brewt] consistency.pl and purge.pl inconsistencies In reply to
Hi Adrian

Is there a way to purge trash from command prompt in verbose mode so that the details can be logged to an out file to see where the messages are not deleted from the disk.

Anup
Quote Reply
Re: [dearnet] consistency.pl and purge.pl inconsistencies In reply to
Hi,

At least the count part is getting slightly clear to me now. The way consistency.pl is counting as number of messages on disk ie.

Just check

http://www.gossamer-threads.com/...;;page=unread#unread

This is on a test install.

So it is how consisency.pl is counting the messages (which are not removed from the disk for sure in my case at least).

Anup
> >