Mailing List Archive: DRBD: Users

Re: Zero time file saving problem from nfs clients on latest rhel 5.4 and drbd-8.3.2

Index | Next | Previous | View Threaded

lars.ellenberg at linbit

Sep 10, 2009, 9:25 AM

Post #1 of 5 (928 views)
Permalink

Re: Zero time file saving problem from nfs clients on latest rhel 5.4 and drbd-8.3.2

On Thu, Sep 10, 2009 at 10:45:24AM -0400, diego.remolina [at] physics wrote:
> Hi,
>
> I have discovered an issue which is related to the following configuration.
>
> Server and Client running RHEL5 Update 4.
> kernel-2.6.18-164.el5
>
> RPMS on file servers
> drbd-8.3.2-3
> drbd-km-2.6.18_128.7.1.el5-8.3.2-3
> drbd-km-2.6.18_164.el5-8.3.2-3
> pacemaker-libs-1.0.5-4.1
> pacemaker-1.0.5-4.1
> openais-0.80.6-8.el5
>
> Whenever the servers are booted into the 2.6.18-164 kernel the problem manifests. The problem is pretty much that any file that is created from graphical applications have wrong permissions and a timestamp which is either close to epoch or the wrap around unix time.
> So the timestamps are in years 1970 or 2038. Some files have no permissions set, e.g. --------- or some bad combination of permissions.
>
> Booting the servers back into the 2.6.18_128.7.1 kernel fixes the problems.
>
> Has anybody else tried drbd 8.3.2 on rhel 5.4 with the latest kernel and nfs? Have you seen the same problem?

I think it is "impossible" for this to have anything to do with block
device drivers such as DRBD, as on the block level, we neither know nore
care for the _content_, which is submitted.

This will be a generic problem with that kernel and nfs,
or that kernel and nfs on certain file systems.

Nothing to do with DRBD.
But keep us posted, nevertheless.

--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBDŽ and LINBITŽ are registered trademarks of LINBIT, Austria.
__
please don't Cc me, but send to list -- I'm subscribed
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

diego.remolina at physics

Sep 10, 2009, 9:33 AM

Post #2 of 5 (872 views)
Permalink

Re: Zero time file saving problem from nfs clients on latest rhel 5.4 and drbd-8.3.2 [In reply to]

>
> I think it is "impossible" for this to have anything to do with block
> device drivers such as DRBD, as on the block level, we neither know nore
> care for the _content_, which is submitted.
>
> This will be a generic problem with that kernel and nfs,
> or that kernel and nfs on certain file systems.

I ruled this out by creating an nfs 4 share on a non-drbd file system on
the machine that was currently the drbd master. I shared that file
system in the same way I have the drbd ones via nfs, then launched open
office, created an odt file. When saving to the nfs share on a non-drbd
device, it saved correctly (correct date and time). When saving to the
nfs share on the drbd device, saving failed, an empty file was created
there with the bad time stamp and no permissions, ---------.

I do not know much about the internals of how drbd works, but based on
my observation that it does not work on the drbd partition with the new
kernel, isn't it possible that some patch applied to the new Redhat
kernel in 5.4 breaks drbd when it is compiled against the new kernel
sources?

Diego

>
> Nothing to do with DRBD.
> But keep us posted, nevertheless.
>
>

--
Diego Julian Remolina
System Administrator - Systems Support Specialist IV
School of Physics
Georgia Institute of Technology
Phone: (404) 385-3499
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

lars.ellenberg at linbit

Sep 10, 2009, 9:50 AM

Post #3 of 5 (873 views)
Permalink

Re: Zero time file saving problem from nfs clients on latest rhel 5.4 and drbd-8.3.2 [In reply to]

On Thu, Sep 10, 2009 at 12:33:53PM -0400, Diego Remolina wrote:
>>
>> I think it is "impossible" for this to have anything to do with block
>> device drivers such as DRBD, as on the block level, we neither know nore
>> care for the _content_, which is submitted.
>>
>> This will be a generic problem with that kernel and nfs,
>> or that kernel and nfs on certain file systems.
>
> I ruled this out by creating an nfs 4 share on a non-drbd file system on
> the machine that was currently the drbd master. I shared that file
> system in the same way I have the drbd ones via nfs, then launched open
> office, created an odt file. When saving to the nfs share on a non-drbd
> device, it saved correctly (correct date and time). When saving to the
> nfs share on the drbd device, saving failed, an empty file was created
> there with the bad time stamp and no permissions, ---------.
>
> I do not know much about the internals of how drbd works, but based on
> my observation that it does not work on the drbd partition with the new
> kernel, isn't it possible that some patch applied to the new Redhat
> kernel in 5.4 breaks drbd when it is compiled against the new kernel
> sources?

That certainly would be possible, but it would break in totally
different was. No, these symptoms I'd say are still impossible
to be caused by a block driver.

Did you use the same file system _type_?

i.e. if your "non-working on drbd" file system is XFS,
and your "working on non-drbd" file system is ext3

>
> Diego
>
>>
>> Nothing to do with DRBD.
>> But keep us posted, nevertheless.

--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBDŽ and LINBITŽ are registered trademarks of LINBIT, Austria.
__
please don't Cc me, but send to list -- I'm subscribed
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

lars.ellenberg at linbit

Sep 10, 2009, 10:44 AM

Post #4 of 5 (878 views)
Permalink

Re: Zero time file saving problem from nfs clients on latest rhel 5.4 and drbd-8.3.2 [In reply to]

On Thu, Sep 10, 2009 at 01:01:42PM -0400, Diego Remolina wrote:
>> Did you use the same file system _type_?
>
> Yep, the file system was ext3 in both cases. I was originally using ext4
> and thought the original problem was related to ext4 and redhat having
> the very early version, so I backed up the data, reformatted ext3,
> restored the data and the problem was still there. I then backed down to
> the older kernel and have no problems.
>
> [root [at] phys-file0 ~]# mount | grep drbd
> /dev/drbd0 on /export/data type ext3 (rw,user_xattr,acl,usrquota,grpquota)
> /dev/drbd1 on /export/scratch type ext3
> (rw,user_xattr,acl,usrquota,grpquota)
>
> [root [at] phys-file0 ~]# mount | grep slash
> /dev/mapper/VolGroup01-slash on / type ext3 (rw)
>
> [root [at] phys-file0 export]# ls -l /export/test/
> total 10264
> -rw------- 1 dr126 2626 7179 Sep 10 09:36 Diego-File.odt
> -rw------- 1 dr126 2626 10485760 Sep 10 09:35 testfile
>
> So as you can see, the Diego-File.odt was written correctly to
> /export/test/ from a client using nfs4.
>
> [root [at] phys-file0 export]# grep test /etc/exports
> /export/test gss/krb5(rw,sync,root_squash,nohide)
>
> A sample of a file with the broken permissions and time stamp:
>
> [root [at] phys-file0 dr126]# ls -l KDE.startkde.bC4799
> --w-rwxrwt 1 dr126 2626 0 Jan 18 2038 KDE.startkde.bC4799

nice one.

Ok, so then add an other fresh and new DRBD running on that kernel,
and do a fresh mkfs.ext3 on it, and export it.
just to prove that it is NOT drbd.

then compare tune2fs and mount options, and clients,
maybe update clients as well, and see if you find a client
that works...

Really. It just cannot be the block layer.

--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBDŽ and LINBITŽ are registered trademarks of LINBIT, Austria.
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

diego.remolina at physics

Sep 11, 2009, 9:21 AM

Post #5 of 5 (858 views)
Permalink

Re: Zero time file saving problem from nfs clients on latest rhel 5.4 and drbd-8.3.2 [In reply to]

> Really. It just cannot be the block layer.

You were right, I reinstalled another set of machines, with and without
drbd and was actually able to figure out that the problem occurred both
on the drbd and non-drbd nfs4 shared file systems.

I guess one thing to mention is that if you write files as root, then
files are written correctly, however, writing files as a regular user
exhibits the problem.

https://bugzilla.redhat.com/show_bug.cgi?id=522163

Thanks for screwing it for us RH.

Diego
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads