Gossamer Forum
Home : Products : Gossamer Mail : Discussion :

consistency.pl and purge.pl inconsistencies

(Page 1 of 2)
> >
Quote Reply
consistency.pl and purge.pl inconsistencies
Hi,

This is just an observation:

purged messages for a user and the count was somewhere around 2500 or so.

regularly purged trash folder.

before doing all this consistency.pl --fix was run to ensure that all is well.

However, ran consistency.pl and found following:

2734 messages were on disk and not in the database

0 messages were in the database but not on disk
validate table inconsistancies: 0
msgtrack table inconsistancies: 0
msgs table inconsistancies: 1
msgs_threads table inconsistancies: 0
msgdata table inconsistancies: 0
msgsearch table inconsistancies: 0
msgs_remote table inconsistancies: 0
pop_accounts table inconsistancies: 0
sigs table inconsistancies: 0
address table inconsistancies: 0
filter table inconsistancies: 0
users => dgraph table inconsistancies: 0
dgraph => users table inconsistancies: 0
sessions table inconsistancies: 0
folders table inconsistancies: 0

Does sit mean that purge from admin is not removing the messages from the disk?

Have run consistency.pl again to synchro everything.

Obviously all the while consistency.pl was being run the site was disabled. Again with everything synchronized, again did a purge messages (Trash 1 Messages Deleted):

consistency.pl gave following result:

2 messages were on disk and not in the database

0 messages were in the database but not on disk
validate table inconsistancies: 0
msgtrack table inconsistancies: 0
msgs table inconsistancies: 0
msgs_threads table inconsistancies: 0
msgdata table inconsistancies: 0
msgsearch table inconsistancies: 0
msgs_remote table inconsistancies: 0
pop_accounts table inconsistancies: 0
sigs table inconsistancies: 0
address table inconsistancies: 0
filter table inconsistancies: 0
users => dgraph table inconsistancies: 0
dgraph => users table inconsistancies: 0
sessions table inconsistancies: 0
folders table inconsistancies: 0

Again did purge messages for a user: 265 messages purged.

Ran consistency.pl and got following:

681 messages were on disk and not in the database

0 messages were in the database but not on disk
validate table inconsistancies: 0
msgtrack table inconsistancies: 0
msgs table inconsistancies: 0
msgs_threads table inconsistancies: 0
msgdata table inconsistancies: 0
msgsearch table inconsistancies: 0
msgs_remote table inconsistancies: 0
pop_accounts table inconsistancies: 0
sigs table inconsistancies: 0
address table inconsistancies: 0
filter table inconsistancies: 0
users => dgraph table inconsistancies: 0
dgraph => users table inconsistancies: 0
sessions table inconsistancies: 0
folders table inconsistancies: 0

So there is some bit of inconsistency between purge and consistency.pl

What could be the reasons?

However, when admin does all the deleting (empty folder) by going to the user account from admin then there are no inconsistencies. So i believe, Purge.pm or something else needs a closer look. It is definitely leading to inconsistencies wrt the mesages on disk and database. For example for a given user when the folder is emptied (85 messages) and also the trash after that, the consistency.pl gives:

0 messages were on disk and not in the database

0 messages were in the database but not on disk
validate table inconsistancies: 0
msgtrack table inconsistancies: 0
msgs table inconsistancies: 0
msgs_threads table inconsistancies: 0
msgdata table inconsistancies: 0
msgsearch table inconsistancies: 0
msgs_remote table inconsistancies: 0
pop_accounts table inconsistancies: 0
sigs table inconsistancies: 0
address table inconsistancies: 0
filter table inconsistancies: 0
users => dgraph table inconsistancies: 0
dgraph => users table inconsistancies: 0
sessions table inconsistancies: 0
folders table inconsistancies: 0

Which is perfectly OK.

It is only the "Purge" operation which is leading to all sorts of inconsistencies.

Any comments.....

Thnx

Anup

Last edited by:

anup123: Mar 14, 2003, 11:10 PM
Quote Reply
Re: [anup123] consistency.pl and purge.pl inconsistencies In reply to
Hi,

Just in case if any comment is avilable on the subject please do let me know as the problem is reproducible as far as inconsistencies wrt synchro of messages on disk and database is concerned.

I think that the same definitely needs attention....

Thnx

Anup
Quote Reply
Re: [anup123] consistency.pl and purge.pl inconsistencies In reply to
Hi everyone at GT,

I had looked around on forum on this issue and it seems that Purge from admin has frequently been touched upon. I had been in touch with Jack on the subject (when i had found no response to the post from author(s) of the scripts) once and was promised a feedback soon.

My case is that I have not yet touched the Purge Users utility and fearing a case of losing all users and I don't think I'll do that in the near future till there is a full confidence expressed in the same by GT. More so becasue purge messages has been leading to gross inconsistencies.

And kindly do not misinterpret my helplessness, but just like one of the users of GT products, at this stage of time even i cannot allow access to my server primarily on advise from my service provider who had provided the courtsey of putting up a standby server when they had to take this server offline in January for two days due to severe hacking attempt on 9/10/ jan 2003. The system is under monitoring by them 24/7 and they have a list of all the web based admin interfaces also. So I hope my situation of not being able to provide access to server is not interpreted as being rude or stubborn by anyone. I believe that this should have been tested and taken care of during the course of diffrent releases of GM that has come up. I am sure that i would be able to see some light at the end of the tunnel..... and not light of an oncoming train surely.

Again for reproducing the same please see the consistency.pl outputs:

----cosistency.pl before purge messages from Admin Section-----

This was run after the last run and fixing on 15th march (my first post on the subject)

0 messages were on disk and not in the database

0 messages were in the database but not on disk
validate table inconsistancies: 0
msgtrack table inconsistancies: 0
msgs table inconsistancies: 2
msgs_threads table inconsistancies: 0
msgdata table inconsistancies: 0
msgsearch table inconsistancies: 0
msgs_remote table inconsistancies: 0
pop_accounts table inconsistancies: 0
sigs table inconsistancies: 0
address table inconsistancies: 0
filter table inconsistancies: 0
users => dgraph table inconsistancies: 0
dgraph => users table inconsistancies: 0
sessions table inconsistancies: 0
folders table inconsistancies: 0

-------consistency.pl after purging 132 messages-----------

303 messages were on disk and not in the database

0 messages were in the database but not on disk
validate table inconsistancies: 0
msgtrack table inconsistancies: 0
msgs table inconsistancies: 2
msgs_threads table inconsistancies: 0
msgdata table inconsistancies: 0
msgsearch table inconsistancies: 0
msgs_remote table inconsistancies: 0
pop_accounts table inconsistancies: 0
sigs table inconsistancies: 0
address table inconsistancies: 0
filter table inconsistancies: 0
users => dgraph table inconsistancies: 0
dgraph => users table inconsistancies: 0
sessions table inconsistancies: 0
folders table inconsistancies: 0

So the errors are reproducible. I think that this has not been used by many users or used and not seen the impact possibly so this has gone unreported so far. As of now it seems definitely a "Bug"

I am sure that I have confined my post to correct forum, abided by all the rules and have "stuck to the basics" in order to avoid any invitation of unwanted replies to the same as it had happened many a times in the past whenever a Bug was reported.....

A solution is expected....

Anup
Quote Reply
Re: [anup123] consistency.pl and purge.pl inconsistencies In reply to
Hi!

For your information, I have reported this problem since the very first versions!
Quote Reply
Re: [dearnet] consistency.pl and purge.pl inconsistencies In reply to
Hi,

I was aware of the fact but i also presumed that it must have been fixed.

Do u also have that error while using the Multiple Validate/Delete option of Admin. In my case as reported, when i do the Validate/Delete then the deletion from databse is fine but to my horror i found that the user exists on disk. I had to go about manually syncro the same with database as i never kept record of how many times i have used...and for which users.

Anup
Quote Reply
Re: [anup123] consistency.pl and purge.pl inconsistencies In reply to
Hello!

To be honest, I found the problem that the inconsistencies grow with time. There was no logical explaination other than bugs concept.

I did delete users - not though the Validate/delete - through search >> delete and mostlikely due to this all those inconsistencies were growing. So may be its the general delete routine that does not clean things everywhere. However I did not track in this direction of tracing users like you did, which I think is interesting or a logical explaination.

It is not only this but what I found at that time was that after the second hirarchy of the directory system, the scripts were creating user nobody directories which it should not have created in that way.

So there were other problems also reported by myself and do not actually know its status, if they are already addressed.
Quote Reply
Re: [dearnet] consistency.pl and purge.pl inconsistencies In reply to
Hi,

Its certain components which have not been addressed to and last i got the message was that the problems reported by me are not being able to be duplicated at GT so they wanted to have a look at the system. However, due to reasons of advice from my service providers, i have not been able to provide access to system.

For example: Admin gose about deleted messages in a folder of any user and there are no inconsistencies.... but when purge messages is selected for then inconsistencies come up and the difference in the numberrs is randon.

Same with delete users. The moment Validate/Delete option is taken, the users are not deleted from the disk. I only wonder that in such a case if the same username is taken by a new registrant, what would happen to the data on the disk because the directory already existsed before the username was there in the database.

I believe GT must be definitely pursuing the same with due diligence and urgency so that the problems are ironed out...


=_= =_= =_= =_= =_= =_=


You can chain me, you can torture me, you can even destroy this body, but you will never imprison my mind.
-Mahatma Gandhi
Quote Reply
Re: [anup123] consistency.pl and purge.pl inconsistencies In reply to
Hello!

What about the report shown from the admin side for each mailbox which says 0 messages in Total out of which 26 messages read! How can that happen?

This has been also reported by myself. All this leads to something not correct. At that time also there was a feeling of GT not being able to reproduce this part of the problem, I beleive.

However, I did not check the areas you are talking about carefully. I may do so within a weeks time and report my opinion, if I was able to reproduce. Worth giving it a try.

Further, deleting users does not delete the directories for messages even if in there there are no users or messages!!! Anathor issue. Permissions are user nobody, since thats the ownership script created on my server!
Quote Reply
Re: [dearnet] consistency.pl and purge.pl inconsistencies In reply to
I do not have that 0/26 case as yet though the inconsistencies wrt ,essages there and the unread messages (as viewed from admin) are there with count difference being 1 to 2
Quote Reply
Re: [anup123] consistency.pl and purge.pl inconsistencies In reply to
Hi!

As I said earlier, I would work on it and report. No one from GT has shown any interest and therefore, I would not want to continue this topic.

The most interesting part of the story is that the purge messages could only delete messages from the database and thereafter not their linkages on the disk.

Hence inconsistency increases over time. Have no idea why this could happen?
Quote Reply
Re: [dearnet] consistency.pl and purge.pl inconsistencies In reply to
Quote:
The most interesting part of the story .....

Oh i always believed that it was a fact, is a fact and will be a fact till it is resolved (not on case case case basis though) by GT. I was never reporting a story. At this moment i am thinking of DEACTIVATING THAT PURGE LINK AS IT IS DANGEROUS TO HAVE IT THERE.
Frown

Anup
Quote Reply
Re: [anup123] consistency.pl and purge.pl inconsistencies In reply to
Hi Anup!

I meant a fact too.

Further, I have a feeling that it is a server specific fact that you and me are facing.

My problem is what the hell needs to be changed. There is no list of modules that the scripts may be needing or calling.

Atleast if GT could post a list of ALL modules on their (testing) server, one could compare whats different and where could be a pontential glitch in the system! This would help everyone.
Quote Reply
Re: [dearnet] consistency.pl and purge.pl inconsistencies In reply to
Hi Dearnet.

Quote:
Further, I have a feeling that it is a server specific fact that you and me are facing

Or possibly that the same has been noticed by us. Without offending anyone, I think not many would have tested /checked the same.

I think that it would be a good thing to include the consistency check on 'users' on database and disk in the consistency.pl. It was nightmarish for me to keep going to individual cases of deleting the users on disk but not on database.

And for cases where server access could not be given (like in your and mine cases) at least the user could be instructed with a "Checklist" sort of a thing. At least i would agree to dish out all the scripts/ components of my install which may be required by GT to test.

Hoping against hope, but i do hope that the waiting period is not indefinitely long enough.

Hope the case is being looked into by GT...

Anup

Last edited by:

anup123: May 3, 2003, 5:46 PM
Quote Reply
Re: [dearnet] consistency.pl and purge.pl inconsistencies In reply to
Well on Inbox issue from Admin, I always find a consistency of 1 message more being unread than it actually is.

So if all the messages are read the inbox would say 1 message unread (from Admin side)...

So the consistency of this error should imply that it should be easier to fix....

Anup
Quote Reply
Re: [anup123] consistency.pl and purge.pl inconsistencies In reply to
Hi!

This particular error I thought could be due to scripts not being able to delete the data from the disk besause (stupid enough) the user is nobody.

Do you have something similar? Did you check owner of the directories in the second and the third level of the breaking of alphabet tree? You know what I mean? The first two level will be OK. The problem is in the third level directories that scripts creates during the insertion. Thereafter things goes wrong!!!
Quote Reply
Re: [dearnet] consistency.pl and purge.pl inconsistencies In reply to
Hello,

It's all a case of 'nobody' except when it comes to the 'ownership of pain' of talking to nobody, dealing with nobody and being listened to by nobody. that pain is owned by owner of the license.

Problem is a problem and if you and me are reporting a problem of similar nature it cannot be an isolated case. It's a case of improper documentation on bug fixes, etc, not knowing who has been supplied with what and often the case of having preference of going on live server and trying to eliminate the problem and then forgetting down the line. I still have two copies of login.cgi (one functional and one non functional obviously) on my server. So probably it would be some admin.cgi or some other component whch was of the wrong version packaged into my install. I do not like the idea of giving access to server. The day i can afford a development server, i would definitely give access to that with the CC of the installation which is there on my server. Not that i distrust anyone but it's a matter of serving certain clients who do not want that I give access to server to anyone. In addition, my service providers have also advised me against that and have changed the default ssh port for that reason so that they would know for sure that if it's shell access then it was provided by me and they are closely watching the same due to hacking in Jan 2003. I would gladly give out piece by piece of the components which may be desired (if t all) by GT to look into the problem ..some wrong version of some component(s) of GM there on my server (like it happened with login.cgi). I had a wrong versionof user.pm though the same was resolved two months before i downloaded.

Basically I am frustrated and fed up. GM still has to reach it's maturity level in terms of consistency across the board and i am sure that no one would quote MS in defense of dishing out products full of bugs. Perhaps some day my son can pay for an upgrade to a decently functioning version of GM. As of now it is apparently 'As Is Where Is' basis product. this may sound discomforting to many but the fact remains that success is quanitified by the amount of patting that end users do rather than doing an insignificant excercise of "patting your own back'

Personally I feel GM has never been a priority product. Few may call these as 'rantings' but do i /should i care when i am made to bleed?



Anup
Quote Reply
Re: [anup123] consistency.pl and purge.pl inconsistencies In reply to
I have read this thread, and have been using GT products for close to 5 years now.

The bottom line here, is, if you cannot, or will not, give GT access to your server, to _see_ the problem, and they cannot duplicate it on their servers, there is _NO_ way for them to fix it.

In fact, there is _no_ way for any programmer, anywhere, to do that. What you are asking, (or really complaining about) is something you have apparantly contributed to, by taking the "advice from [your] service providers." Maybe you should get your service providers to fix the problem?

In all the time I have been using GT products, they have _NEVER_ ignored a bug. They have been slow to release new features, or updates, while they fix bugs, but they _NEVER_ ignore them. But to fix them, they have to _see_ them.

I just reinstalled Gmail after 2 years of not running it, and came here for support. The first thing I run into is this thread, and it really blows my mind.

How can you expect GT to fix something they can't verify, duplicate, or even "see" exists?

FWIW -- People in the Links area, with serious problems (even not so serious) give GT access to their servers admin, or telent, all the time, and I don't think _any_ person has ever reported a problem because of that. I really didn't want, or feel I really needed, to say this, but I guess it goes to reenforce the concept that by not giving GT access to the problem, you are contributing to it still being here.


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.
Quote Reply
Re: [pugdog] consistency.pl and purge.pl inconsistencies In reply to
Wrong version of login.cgi -- was it a bug or a problem of management of affairs.

Wrong version of user.pm -- was it a bug or a problem of management of affairs. There are many issues which i would not like to delve into as that does not concern this thread.

Quote:
People in the Links area, with serious problems (even not so serious) give GT access to their servers admin, or telent, all the time, and I don't think _any_ person has ever reported a problem because of that. I really didn't want, or feel I really needed, to say this, but I guess it goes to reenforce the concept that by not giving GT access to the problem, you are contributing to it still being here.

First of all i cannot comment on LSQL as I am not using it. Maybe if i get to use it, i would be in a better position to comment (without in a haste). I try to use the six inch space between my eyes rather than believing my ears. Please read the thread carefully. My inability to provide access is not implying distrust.
Quote:
I just reinstalled Gmail after 2 years of not running it, and came here for support. The first thing I run into is this thread, and it really blows my mind

If you have not been running GM then i think it was too premature to blow your mind on this. If non goodies cannot be talked of then please get that included in the forum "Blue book" and i would sign of permanently from this forum as I am sick of this for vs against stuff. I am fed up not because of bugs being reportedly fixed but the wrong versions of the files being replaced by the correct ones after providing access. Is this accidental Always?

Quote:
How can you expect GT to fix something they can't verify, duplicate, or even "see" exists?

That's like saying that I am a liar. Your passion for GT is well appreciated undoubtedly, and i don't mind your interpretations and conclusions. And BTW, the problems have never been duplicated and one of the problem is continuing since first version if notings of dearnet are to be taken into consideration.

Quote:
In fact, there is _no_ way for any programmer, anywhere, to do that. What you are asking, (or really complaining about) is something you have apparantly contributed to, by taking the "advice from [your] service providers." Maybe you should get your service providers to fix the problem?

On the first part a slight rephrasing would be more appropriate "there is no way a support dept would really know whether the versions of files are proper and compatible with the release or not" (cf first 2 points in this message) ...

On the second part --That's like passing the buck. As regards my service provider fixing it... well though it was not under agreement yet they put up a backup server (not in contract though) for two days when they had to take my server offline. With that level of support I don't think any sane person would ever challenge their preconditions imposed post hacking or even request for a permission.

I am sick of this for vs against battle. GT has the version that was offered to me. And if the problems cannot be duplicated with that version then if i were to conclude so hastily like you i would say certain things like you which may not be liked so i shall let that pass......

Anup
Quote Reply
Re: [anup123] consistency.pl and purge.pl inconsistencies In reply to
Wondered if this problem could be version related. So downloaded the 2.1.1 and on a test install with individual pop access only and then "Purged" one message in trash of a acount from Admin.

After that ran consistency.pl and found:

2 Messages on disk but not on database.

Continued this iteration Purging One Message At A Time from Admin and then runnignconsistency.pl and the results including the first run are as follows:

Iteration Purge Trash Message consistency.pl discrepancy

1 1 2

2 1 2

3 1 5

4 1 7

5 1 9

6 1 11

7 1 13

8 1 16

9 1 18

10 1 21

Tomorrow I would try to repeat the same iteration, purging 2 messages at a time and report the same. This is with the latest version 2.1.1 test install. I fail to figure out any pattern of error as such. This is just for information......

Anup

Last edited by:

anup123: May 21, 2003, 4:24 PM
Quote Reply
Re: [anup123] consistency.pl and purge.pl inconsistencies In reply to
Hi Anup,

Thanks for helping out testing. In a few days, I'll hopefully get some time to work on Gossamer Mail. Hopefully, we'll be able to figure out what exactly is going on. One thing that comes to mind that may be one cause of the consistencies is the welcome email.

Adrian
Quote Reply
Re: [brewt] consistency.pl and purge.pl inconsistencies In reply to
Hi Adrian,

The Welcome Email in this test install has been turned off. And i forgot to mention that I was doing all this test on a single account deleting one mail at a time followed by Purging Trash for each iteration.

Continuing with the iteration above with the same account but taking 2 messages at a time the results are:

Iter# Purged consistency

11 2 25

12 2 27

13 2 29

14 2 31

15 2 35

I am sure there would be a solution to this. Would continue iteration from here later

Thnx

Anup
Quote Reply
Re: [brewt] consistency.pl and purge.pl inconsistencies In reply to
Iteration with 3 messages at a time deleted and then Admin Purged from previous stage of 2 messages at a time (consistency.pl --fix not yet run):

iter# Purged consistency.pl

16 3 41

17 3 48

18 3 63

19 3 71

From what i see is (as a pattern) that at every 9th or 10th message purged there is abnormal jump in the consistency.pl reported value (by abnormal i mean is that it deviates from the pattern). I have run out of messages so I have to stop at this stage.

Thnx

Anup
Quote Reply
Re: [anup123] consistency.pl and purge.pl inconsistencies In reply to
I've been trying to make my test install of Gossamer Mail inconsistent, but so far haven't gotten it to do.

How many emails are you testing it with? And you're only doing it with a single user right? How are you purging one email at a time?

Adrian
Quote Reply
Re: [brewt] consistency.pl and purge.pl inconsistencies In reply to
Hi,

For one user, i login to his account and delete one message from inbox which goes to trash. I do not logout.

From admin i purge trash folder.

Similarly i did for two and three mails at a time deleted from inbox from login account and then purged by admin.

I did this with the latest version and since i found the same inconsistency which grows with number of messages, i verified that it was not a problem with version. (original one working is 2.1.0 which was orginally installed by GT).

There is one account on the regualt install which has lot many mails in the inbox and if you want i could repeat the test there to report the findings in line with what i had done on test install.

---EDIT---

Please test under RAID1 because that's how my server is. Software RAID1 setup. I wonder (just a wild guess) if that has anything to do with the counts being shown wrong....if at all. Though other stuff (non GM) have yet not shown any abnormalities.

Thnx

Anup

Last edited by:

anup123: May 28, 2003, 1:16 AM
Quote Reply
Re: [anup123] consistency.pl and purge.pl inconsistencies In reply to
It shouldn't have anything to do with your raid setup. I'll try doing the purging tomorrow.

Adrian
> >