Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: DRBD: Users

Unable to perform initial sync

 

 

DRBD users RSS feed   Index | Next | Previous | View Threaded


vyacheslav at karpukhin

Apr 10, 2012, 9:22 AM

Post #1 of 7 (1160 views)
Permalink
Unable to perform initial sync

Hi. I just installed drbd and now trying to perform initial sync. It works fine in one direction, but if I'm trying to perform it in the opposite direction, I get this:

> Apr 10 11:13:54 web kernel: block drbd0: Becoming sync source due to disk states.
> Apr 10 11:13:54 web kernel: block drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS )
> Apr 10 11:13:54 web kernel: block drbd0: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 25(1), total 25; compression: 100.0%
> Apr 10 11:13:54 web kernel: block drbd0: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 25(1), total 25; compression: 100.0%
> Apr 10 11:13:54 web kernel: block drbd0: helper command: /sbin/drbdadm before-resync-source minor-0
> Apr 10 11:13:54 web kernel: block drbd0: helper command: /sbin/drbdadm before-resync-source minor-0 exit code 0 (0x0)
> Apr 10 11:13:54 web kernel: block drbd0: conn( WFBitMapS -> SyncSource )
> Apr 10 11:13:54 web kernel: block drbd0: Began resync as SyncSource (will sync 15519040 KB [3879760 bits set]).
> Apr 10 11:13:54 web kernel: block drbd0: updated sync UUID AF78EBCA7F218B01:C237DF3A275A375B:C236DF3A275A375B:C235DF3A275A375B
> 275A375B:C235DF3A275A375B
> Apr 10 11:13:54 web kernel: block drbd0: /dev/shm/drbd-8.4.1/drbd/drbd_receiver.c:2541: sector: 0s, size: 4194304
> Apr 10 11:13:54 web kernel: d-con r0: error receiving RSDataRequest, e: -22 l: 0!
> Apr 10 11:13:54 web kernel: d-con r0: peer( Secondary -> Unknown ) conn( SyncSource -> ProtocolError )
> Apr 10 11:13:54 web kernel: d-con r0: asender terminated
> Apr 10 11:13:54 web kernel: d-con r0: Terminating asender thread

What's wrong?

Thank you.
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


dbarker at visioncomm

Apr 10, 2012, 11:36 AM

Post #2 of 7 (1079 views)
Permalink
Re: Unable to perform initial sync [In reply to]

I do not understand what you did or are trying to do. In Sync has no
"direction". If you are in sync from Primary to Secondary, you are in sync
period. There is no reason to think about a direction.

To do a recovery, when one of the resources is to be used as the sync source
due to having more current data, there are commands to do that,
discard-my-data, overwrite-data-of-peer, etc. They have no inherent
direction, but have complimentary meanings depending on which node the
command is run.

Please explain more what you are trying to accomplish, which node is
primary, secondary, which device is considered "current" and the commands
you are issuing and on which host.

Dan

-----Original Message-----
From: drbd-user-bounces [at] lists
[mailto:drbd-user-bounces [at] lists] On Behalf Of Vyacheslav
Karpukhin
Sent: Tuesday, April 10, 2012 12:23 PM
To: drbd-user [at] lists
Subject: [DRBD-user] Unable to perform initial sync

Hi. I just installed drbd and now trying to perform initial sync. It works
fine in one direction, but if I'm trying to perform it in the opposite
direction, I get this:

> Apr 10 11:13:54 web kernel: block drbd0: Becoming sync source due to disk
states.
> Apr 10 11:13:54 web kernel: block drbd0: peer( Unknown -> Secondary )
conn( WFReportParams -> WFBitMapS )
> Apr 10 11:13:54 web kernel: block drbd0: send bitmap stats
[Bytes(packets)]: plain 0(0), RLE 25(1), total 25; compression: 100.0%
> Apr 10 11:13:54 web kernel: block drbd0: receive bitmap stats
[Bytes(packets)]: plain 0(0), RLE 25(1), total 25; compression: 100.0%
> Apr 10 11:13:54 web kernel: block drbd0: helper command: /sbin/drbdadm
before-resync-source minor-0
> Apr 10 11:13:54 web kernel: block drbd0: helper command: /sbin/drbdadm
before-resync-source minor-0 exit code 0 (0x0)
> Apr 10 11:13:54 web kernel: block drbd0: conn( WFBitMapS -> SyncSource )
> Apr 10 11:13:54 web kernel: block drbd0: Began resync as SyncSource (will
sync 15519040 KB [3879760 bits set]).
> Apr 10 11:13:54 web kernel: block drbd0: updated sync UUID
AF78EBCA7F218B01:C237DF3A275A375B:C236DF3A275A375B:C235DF3A275A375B
> 275A375B:C235DF3A275A375B
> Apr 10 11:13:54 web kernel: block drbd0:
/dev/shm/drbd-8.4.1/drbd/drbd_receiver.c:2541: sector: 0s, size: 4194304
> Apr 10 11:13:54 web kernel: d-con r0: error receiving RSDataRequest, e:
-22 l: 0!
> Apr 10 11:13:54 web kernel: d-con r0: peer( Secondary -> Unknown ) conn(
SyncSource -> ProtocolError )
> Apr 10 11:13:54 web kernel: d-con r0: asender terminated
> Apr 10 11:13:54 web kernel: d-con r0: Terminating asender thread

What's wrong?

Thank you.
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


vyacheslav at karpukhin

Apr 10, 2012, 11:48 AM

Post #3 of 7 (1084 views)
Permalink
Re: Unable to perform initial sync [In reply to]

On 11.04.2012, at 0:36, Dan Barker wrote:

> I do not understand what you did or are trying to do. In Sync has no
> "direction". If you are in sync from Primary to Secondary, you are in sync
> period. There is no reason to think about a direction.


I'm talking about direction because in my case sync from server B to server A works fine, but from server A to server B -- fails.

Since it's initial sync of newly created volume, it doesn't matter for me which of the hosts to make Primary, and which -- Secondary. So, that initial sync may be performed in two directions -- from server A to server B, or from server B to server A.

In my case when I mark server B as primary, everything is fine, drbd synchronizes from B to A. But if instead I mark server A as primary, synchronization won't perform -- there are protocol errors in the log.

I tried to use different versions of drbd, and found out that this issue starts with 8.3.11. Right now drbd performs synchronization from A to B with 8.3.10, but I couldn't make it do that with 8.3.11, 8.3.12 and 8.4.1.

In my experiments each time I do the following:
1) "create-md" on both servers
2) starting the resource on both servers
3) marking one of the servers as primary
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


dbarker at visioncomm

Apr 10, 2012, 12:16 PM

Post #4 of 7 (1078 views)
Permalink
Re: Unable to perform initial sync [In reply to]

That is part of the story. You most likely have some protocol issues (thus
the log entries).

Why would you experiment with cross-version drbd? You should have the
easiest results with both servers at the same, recent level.

You would need to verify the kernel module and userland program versions on
both servers, the commands run, and the relevant dmesg logs from both sides
for folks to help you on this problem.

... BTW ...

Are you aware that if you do not care about the contents of the disks, you
don't have to sync all the zeros? You can create up-to-date disks instantly,
and then put a file system on it. Everything will be in sync. The first
verify will find a bunch of out-of-sync blocks, but they are in the
filesystem's free space and are synced by simply doing a
disconnect/reconnect on the secondary node. It really speeds up initial
setup, expecially with multi-terabyte resources. (See new-current-uuid in
http://www.drbd.org/users-guide-8.3/re-drbdsetup.html; syntax may be
different on 8.3 versions).

Dan

-----Original Message-----
From: drbd-user-bounces [at] lists
[mailto:drbd-user-bounces [at] lists] On Behalf Of Vyacheslav
Karpukhin
Sent: Tuesday, April 10, 2012 2:49 PM
To: drbd-user [at] lists
Subject: Re: [DRBD-user] Unable to perform initial sync


On 11.04.2012, at 0:36, Dan Barker wrote:

> I do not understand what you did or are trying to do. In Sync has no
> "direction". If you are in sync from Primary to Secondary, you are in
> sync period. There is no reason to think about a direction.


I'm talking about direction because in my case sync from server B to server
A works fine, but from server A to server B -- fails.

Since it's initial sync of newly created volume, it doesn't matter for me
which of the hosts to make Primary, and which -- Secondary. So, that initial
sync may be performed in two directions -- from server A to server B, or
from server B to server A.

In my case when I mark server B as primary, everything is fine, drbd
synchronizes from B to A. But if instead I mark server A as primary,
synchronization won't perform -- there are protocol errors in the log.

I tried to use different versions of drbd, and found out that this issue
starts with 8.3.11. Right now drbd performs synchronization from A to B with
8.3.10, but I couldn't make it do that with 8.3.11, 8.3.12 and 8.4.1.

In my experiments each time I do the following:
1) "create-md" on both servers
2) starting the resource on both servers
3) marking one of the servers as primary
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


vyacheslav at karpukhin

Apr 10, 2012, 12:36 PM

Post #5 of 7 (1078 views)
Permalink
Re: Unable to perform initial sync [In reply to]

On 11.04.2012, at 1:16, Dan Barker wrote:

> That is part of the story. You most likely have some protocol issues (thus
> the log entries).
>
> Why would you experiment with cross-version drbd? You should have the
> easiest results with both servers at the same, recent level.

I'm not. I'm using exactly the same drbd version on both servers. Furthermore, I'm using exactly the same linux distro and kernel version (CentOS 6, 2.6.32-220.7.1).

> You would need to verify the kernel module and userland program versions on
> both servers, the commands run, and the relevant dmesg logs from both sides
> for folks to help you on this problem.

As I mentioned previously, both server had drbd 8.4.1. 8.3.11 and 8.3.12 are affected too.
I posted log from Primary before, log on Secondary doesn't have anything interesting:


>> Apr 10 11:13:54 web_backup kernel: block drbd0: Becoming sync target due to disk states.
>> Apr 10 11:13:54 web_backup kernel: block drbd0: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate )
>> Apr 10 11:13:54 web_backup kernel: block drbd0: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 25(1), total 25; compression: 100.0%
>> Apr 10 11:13:54 web_backup kernel: block drbd0: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 25(1), total 25; compression: 100.0%
>> Apr 10 11:13:54 web_backup kernel: block drbd0: conn( WFBitMapT -> WFSyncUUID )
>> Apr 10 11:13:54 web_backup kernel: block drbd0: updated sync uuid C237DF3A275A375A:0000000000000000:0000000000000000:0000000000000000
>> Apr 10 11:13:54 web_backup kernel: block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0
>> Apr 10 11:13:54 web_backup kernel: block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)
>> Apr 10 11:13:54 web_backup kernel: block drbd0: conn( WFSyncUUID -> SyncTarget )
>> Apr 10 11:13:54 web_backup kernel: block drbd0: Began resync as SyncTarget (will sync 15519040 KB [3879760 bits set]).
>> Apr 10 11:13:54 web_backup kernel: d-con r0: sock was shut down by peer
>> Apr 10 11:13:54 web_backup kernel: d-con r0: peer( Primary -> Unknown ) conn( SyncTarget -> BrokenPipe ) pdsk( UpToDate -> DUnknown )
>> Apr 10 11:13:54 web_backup kernel: d-con r0: short read (expected size 16)
>> Apr 10 11:13:54 web_backup kernel: d-con r0: asender terminated
>> Apr 10 11:13:54 web_backup kernel: d-con r0: Terminating asender thread




> Are you aware that if you do not care about the contents of the disks, you
> don't have to sync all the zeros?

I didn't know that, thanks. But still, there is clearly something wrong, and I'm sure this problem will appear not only during initial sync, but also during normal operation.

_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


lars.ellenberg at linbit

Dec 12, 2012, 3:01 AM

Post #6 of 7 (795 views)
Permalink
Re: Unable to perform initial sync [In reply to]

On Mon, Dec 10, 2012 at 02:43:51PM -0500, Carmen Devito wrote:
> Hi all,
>
> I am currently working on setting up a new DRBD resource inside one of
> our enivironments, but whenever I try and sync the secondary to the
> primary I get this:

Config?

Did you follow any guide?
If so, which one?

Something in the kernel logs?

> version: 8.3.13 (api:88/proto:86-96)
> GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by dag [at] Build64R, 2012-09-04 12:06:10
> 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent B r-----
> ns:2087156 nr:0 dw:15297896 dr:16399 al:5116 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:209714176
> [>....................] sync'ed: 0.1% (204796/204796)M
> finish: 624919:07:03 speed: 0 (0) K/sec
>
> I've tried to drbdadm disconnect r0 from the secondary, as well as
> force the primary to overwrite its data to the secondary, but no luck.
> Any ideas.
>
>
>
> Carmen De Vito
> Tata Consultancy Services
> Mailto: carmen.devito [at] tcs
> Website: http://www.tcs.com
> ____________________________________________
> Experience certainty. IT Services
> Business Solutions
> Outsourcing


--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
__
please don't Cc me, but send to list -- I'm subscribed
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


carmen.devito at tcs

Dec 13, 2012, 1:43 PM

Post #7 of 7 (775 views)
Permalink
Re: Unable to perform initial sync [In reply to]

Actually, I figured it out. Although I had set the syncer rate up initially within my drbd conf, it didn't seem to pick it up when I brought up the resource initially. running "drbdadm DEVICE syncer -r 110M" straightened it out for me.

Carmen De Vito
Tata Consultancy Services
Mailto: carmen.devito [at] tcs
Website: http://www.tcs.com
____________________________________________
Experience certainty. IT Services
Business Solutions
Outsourcing
____________________________________________

-----drbd-user-bounces [at] lists wrote: -----
To: drbd-user [at] lists
From: Lars Ellenberg
Sent by: drbd-user-bounces [at] lists
Date: 12/12/2012 06:01AM
Subject: Re: [DRBD-user] Unable to perform initial sync

On Mon, Dec 10, 2012 at 02:43:51PM -0500, Carmen Devito wrote: > Hi all, > > I am currently working on setting up a new DRBD resource inside one of > our enivironments, but whenever I try and sync the secondary to the > primary I get this: Config? Did you follow any guide? If so, which one? Something in the kernel logs? > version: 8.3.13 (api:88/proto:86-96) > GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by dag [at] Build64R, 2012-09-04 12:06:10 > 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent B r----- > ns:2087156 nr:0 dw:15297896 dr:16399 al:5116 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:209714176 > [>....................] sync'ed: 0.1% (204796/204796)M > finish: 624919:07:03 speed: 0 (0) K/sec > > I've tried to drbdadm disconnect r0 from the secondary, as well as > force the primary to overwrite its data to the secondary, but no luck. > Any ideas. > > > > Carmen De Vito > Tata Consultancy Services > Mailto: carmen.devito [at] tcs > Website: http://www.tcs.com/"]http://www.tcs.com > ____________________________________________ > Experience certainty. IT Services > Business Solutions > Outsourcing -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com/"]http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. __ please don't Cc me, but send to list -- I'm subscribed _______________________________________________ drbd-user mailing list drbd-user [at] lists http://lists.linbit.com/mailman/listinfo/drbd-user"]http://lists.linbit.com/mailman/listinfo/drbd-user


=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you

DRBD users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.