Mailing List Archive: DRBD: Users

upgrade drbd 8.3.7 -> 8.3.11

Index | Next | Previous | View Threaded

y at yas

Jun 11, 2012, 10:15 PM

Post #1 of 8 (799 views)
Permalink

upgrade drbd 8.3.7 -> 8.3.11

Hi all

I want to upgrade a two node cluster, without downtime of the complete drbd node.
The two different versions should only run a short time.

One node is already upgraded to ubuntu 12.04 - drbd 8.3.11 - kernel 3.2.0
The other node is running ubuntu 10.04 - drbd 8.3.7, as primary.

When I try to connect the node running drbd 8.3.11 to sync to the node running 8.3.7.
I get some errors.
The sync gets started, but stops.

root [at] hos /var/log # cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: 71955441799F513ACA6DA60

1: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----
ns:0 nr:32 dw:32 dr:0 al:0 bm:0 lo:0 pe:120 ua:0 ap:0 ep:1 wo:f oos:524281120
[>....................] sync'ed: 0.1% (511992/511992)Mfinish: 364:05:02 speed: 0 (0) want: 76,800 K/sec

version: 8.3.11 (api:88/proto:86-96)
srcversion: 71955441799F513ACA6DA60

1: cs:BrokenPipe ro:Secondary/Unknown ds:Inconsistent/DUnknown C r-----
ns:0 nr:32 dw:32 dr:0 al:0 bm:0 lo:2 pe:120 ua:0 ap:0 ep:1 wo:f oos:524281120

conn( WFBitMapT -> WFSyncUUID )
updated sync uuid 4521C444BE77EC04:0000000000000000:0000000000000000:0000000000000000
helper command: /sbin/drbdadm before-resync-target minor-1
helper command: /sbin/drbdadm before-resync-target minor-1 exit code 0 (0x0)
conn( WFSyncUUID -> SyncTarget )
Began resync as SyncTarget (will sync 524281096 KB [131070274 bits set]).
sock was shut down by peer
meta connection shut down by peer.
peer( Primary -> Unknown ) conn( SyncTarget -> BrokenPipe ) pdsk( UpToDate -> DUnknown )
short read expecting header on sock: r=0
asender terminated
Terminating drbd1_asender
bitmap WRITE of 4000 pages took 53 jiffies
conn( BrokenPipe -> Disconnecting )

Somebody already run into this issue?

yves

marcel at kraan

Jun 11, 2012, 10:35 PM

Post #2 of 8 (769 views)
Permalink

Re: upgrade drbd 8.3.7 -> 8.3.11 [In reply to]

Is this not a split brain?

# http://www.drbd.org/users-guide-8.3/s-resolve-split-brain.html

# make a secondary node (it's there already)
# Do this on the secondary only
drbdadm secondary main
drbdadm -- --discard-my-data connect main

# do this on the primary
drbdadm connect main

Sometimes you need to disconnect first..
disconnect main

On 12 jun. 2012, at 07:15, y [at] yas wrote:

> Hi all
>
> I want to upgrade a two node cluster, without downtime of the complete drbd node.
> The two different versions should only run a short time.
>
> One node is already upgraded to ubuntu 12.04 - drbd 8.3.11 - kernel 3.2.0
> The other node is running ubuntu 10.04 - drbd 8.3.7, as primary.
>
> When I try to connect the node running drbd 8.3.11 to sync to the node running 8.3.7.
> I get some errors.
> The sync gets started, but stops.
>
> root [at] hos /var/log # cat /proc/drbd
> version: 8.3.11 (api:88/proto:86-96)
> srcversion: 71955441799F513ACA6DA60
>
> 1: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----
> ns:0 nr:32 dw:32 dr:0 al:0 bm:0 lo:0 pe:120 ua:0 ap:0 ep:1 wo:f oos:524281120
> [>....................] sync'ed: 0.1% (511992/511992)Mfinish: 364:05:02 speed: 0 (0) want: 76,800 K/sec
>
> version: 8.3.11 (api:88/proto:86-96)
> srcversion: 71955441799F513ACA6DA60
>
> 1: cs:BrokenPipe ro:Secondary/Unknown ds:Inconsistent/DUnknown C r-----
> ns:0 nr:32 dw:32 dr:0 al:0 bm:0 lo:2 pe:120 ua:0 ap:0 ep:1 wo:f oos:524281120
>
> conn( WFBitMapT -> WFSyncUUID )
> updated sync uuid 4521C444BE77EC04:0000000000000000:0000000000000000:0000000000000000
> helper command: /sbin/drbdadm before-resync-target minor-1
> helper command: /sbin/drbdadm before-resync-target minor-1 exit code 0 (0x0)
> conn( WFSyncUUID -> SyncTarget )
> Began resync as SyncTarget (will sync 524281096 KB [131070274 bits set]).
> sock was shut down by peer
> meta connection shut down by peer.
> peer( Primary -> Unknown ) conn( SyncTarget -> BrokenPipe ) pdsk( UpToDate -> DUnknown )
> short read expecting header on sock: r=0
> asender terminated
> Terminating drbd1_asender
> bitmap WRITE of 4000 pages took 53 jiffies
> conn( BrokenPipe -> Disconnecting )
>
> Somebody already run into this issue?
>
> yves
>
> _______________________________________________
> drbd-user mailing list
> drbd-user [at] lists
> http://lists.linbit.com/mailman/listinfo/drbd-user

y at yas

Jun 11, 2012, 11:36 PM

Post #3 of 8 (788 views)
Permalink

Re: upgrade drbd 8.3.7 -> 8.3.11 [In reply to]

I tried your suggestions, thanks.
But unfortunately it doesn't worked.

I don't have a split brain entry in my log.
I also get this error on the secondary I try to reconnect.

root [at] secondar ~ # cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: 71955441799F513ACA6DA60

1: cs:NetworkFailure ro:Secondary/Unknown ds:Inconsistent/DUnknown C r-----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:1 pe:0 ua:0 ap:0 ep:1 wo:f oos:524280916

on the primary I see this

root [at] primar / # cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root [at] primar, 2012-06-07 18:53:37

1: cs:ProtocolError ro:Primary/Unknown ds:UpToDate/Inconsistent C r----
ns:0 nr:0 dw:3944607 dr:39099983 al:1450 bm:1193 lo:1 pe:0 ua:0 ap:0 ep:1 wo:b oos:524280916

regards yves

On 12.06.2012, at 07:35, Marcel Kraan wrote:

> Is this not a split brain?
>
> # http://www.drbd.org/users-guide-8.3/s-resolve-split-brain.html
>
> # make a secondary node (it's there already)
> # Do this on the secondary only
> drbdadm secondary main
> drbdadm -- --discard-my-data connect main
>
> # do this on the primary
> drbdadm connect main
>
> Sometimes you need to disconnect first..
> disconnect main
>
>
>
>
> On 12 jun. 2012, at 07:15, y [at] yas wrote:
>
>> Hi all
>>
>> I want to upgrade a two node cluster, without downtime of the complete drbd node.
>> The two different versions should only run a short time.
>>
>> One node is already upgraded to ubuntu 12.04 - drbd 8.3.11 - kernel 3.2.0
>> The other node is running ubuntu 10.04 - drbd 8.3.7, as primary.
>>
>> When I try to connect the node running drbd 8.3.11 to sync to the node running 8.3.7.
>> I get some errors.
>> The sync gets started, but stops.
>>
>> root [at] hos /var/log # cat /proc/drbd
>> version: 8.3.11 (api:88/proto:86-96)
>> srcversion: 71955441799F513ACA6DA60
>>
>> 1: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----
>> ns:0 nr:32 dw:32 dr:0 al:0 bm:0 lo:0 pe:120 ua:0 ap:0 ep:1 wo:f oos:524281120
>> [>....................] sync'ed: 0.1% (511992/511992)Mfinish: 364:05:02 speed: 0 (0) want: 76,800 K/sec
>>
>> version: 8.3.11 (api:88/proto:86-96)
>> srcversion: 71955441799F513ACA6DA60
>>
>> 1: cs:BrokenPipe ro:Secondary/Unknown ds:Inconsistent/DUnknown C r-----
>> ns:0 nr:32 dw:32 dr:0 al:0 bm:0 lo:2 pe:120 ua:0 ap:0 ep:1 wo:f oos:524281120
>>
>> conn( WFBitMapT -> WFSyncUUID )
>> updated sync uuid 4521C444BE77EC04:0000000000000000:0000000000000000:0000000000000000
>> helper command: /sbin/drbdadm before-resync-target minor-1
>> helper command: /sbin/drbdadm before-resync-target minor-1 exit code 0 (0x0)
>> conn( WFSyncUUID -> SyncTarget )
>> Began resync as SyncTarget (will sync 524281096 KB [131070274 bits set]).
>> sock was shut down by peer
>> meta connection shut down by peer.
>> peer( Primary -> Unknown ) conn( SyncTarget -> BrokenPipe ) pdsk( UpToDate -> DUnknown )
>> short read expecting header on sock: r=0
>> asender terminated
>> Terminating drbd1_asender
>> bitmap WRITE of 4000 pages took 53 jiffies
>> conn( BrokenPipe -> Disconnecting )
>>
>> Somebody already run into this issue?
>>
>> yves
>>
>> _______________________________________________
>> drbd-user mailing list
>> drbd-user [at] lists
>> http://lists.linbit.com/mailman/listinfo/drbd-user
>

ff at mpexnet

Jun 12, 2012, 12:05 AM

Post #4 of 8 (764 views)
Permalink

Re: upgrade drbd 8.3.7 -> 8.3.11 [In reply to]

On 06/12/2012 07:15 AM, y [at] yas wrote:
> sock was shut down by peer
> meta connection shut down by peer.
> peer( Primary -> Unknown ) conn( SyncTarget -> BrokenPipe ) pdsk(
> UpToDate -> DUnknown )

Disconcerting.

What does the log on the peer say? Hopefully it presents a reason for
shutting down the connection.

Maybe your new kernel has a new NIC driver that bugs out on your
hardware? Is replacing components an option? Can you use a different NIC
for the initial sync, perhaps?

HTH,
Felix
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

florian at hastexo

Jun 12, 2012, 2:11 AM

Post #5 of 8 (765 views)
Permalink

Re: upgrade drbd 8.3.7 -> 8.3.11 [In reply to]

On 06/12/12 07:35, Marcel Kraan wrote:
> Is this not a split brain?

No.

Hence, the rest of your suggestions are moot, sadly.

Cheers,
Florian

--
Need help with High Availability?
http://www.hastexo.com/now
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

y at yas

Jun 12, 2012, 8:47 AM

Post #6 of 8 (805 views)
Permalink

Re: upgrade drbd 8.3.7 -> 8.3.11 [In reply to]

This is the log output of the primary (8.3.7) which is the sync source

[426929.295891] block drbd1: Handshake successful: Agreed network protocol version 91
[426929.296233] block drbd1: Peer authenticated using 20 bytes of 'sha1' HMAC
[426929.296249] block drbd1: conn( WFConnection -> WFReportParams )
[426929.296275] block drbd1: Starting asender thread (from drbd1_receiver [14699])
[426929.296572] block drbd1: data-integrity-alg: <not-used>
[426929.296580] block drbd1: max_segment_size ( = BIO size ) = 65536
[426929.296590] block drbd1: drbd_sync_handshake:
[426929.296593] block drbd1: self B2E75D9367A1C71B:41B5C01D5EDD0915:6F31928B8C7C4278:7E2AA5F059F1D9D5 bits:131070228 flags:0
[426929.296595] block drbd1: peer 41B5C01D5EDD0914:0000000000000000:0000000000000000:0000000000000000 bits:131070228 flags:0
[426929.296598] block drbd1: uuid_compare()=1 by rule 70
[426929.296599] block drbd1: Becoming sync source due to disk states.
[426929.296602] block drbd1: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS )
[426929.646334] block drbd1: conn( WFBitMapS -> SyncSource )
[426929.646347] block drbd1: Began resync as SyncSource (will sync 524280912 KB [131070228 bits set]).
[426929.703228] block drbd1: /var/lib/dkms/drbd8/8.3.7/build/drbd/drbd_receiver.c:1995: sector: 128s, size: 65536
[426929.719913] block drbd1: error receiving RSDataRequest, l: 24!
[426929.728258] block drbd1: peer( Secondary -> Unknown ) conn( SyncSource -> ProtocolError )
[426929.728373] block drbd1: asender terminated
[426929.728379] block drbd1: Terminating asender thread
[426929.736571] block drbd1: Connection closed
[426929.736577] block drbd1: conn( ProtocolError -> Unconnected )
[426929.736582] block drbd1: receiver terminated
[426929.736584] block drbd1: Restarting receiver thread
[426929.736586] block drbd1: receiver (re)started

the nodes are connect via a cross private ip gigabit connection. without lvm or software raid

yves

On 12.06.2012, at 09:05, Felix Frank wrote:

> On 06/12/2012 07:15 AM, y [at] yas wrote:
>> sock was shut down by peer
>> meta connection shut down by peer.
>> peer( Primary -> Unknown ) conn( SyncTarget -> BrokenPipe ) pdsk(
>> UpToDate -> DUnknown )
>
> Disconcerting.
>
> What does the log on the peer say? Hopefully it presents a reason for
> shutting down the connection.
>
> Maybe your new kernel has a new NIC driver that bugs out on your
> hardware? Is replacing components an option? Can you use a different NIC
> for the initial sync, perhaps?
>
> HTH,
> Felix

_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

ff at mpexnet

Jun 12, 2012, 8:49 AM

Post #7 of 8 (761 views)
Permalink

Re: upgrade drbd 8.3.7 -> 8.3.11 [In reply to]

Hi,

On 06/12/2012 05:47 PM, Yves Serrano wrote:
> [426929.703228] block drbd1: /var/lib/dkms/drbd8/8.3.7/build/drbd/drbd_receiver.c:1995: sector: 128s, size: 65536
> [426929.719913] block drbd1: error receiving RSDataRequest, l: 24!

this looks potentially enlightening, but not to me, sadly.

Any idea, anyone? Are the versions not so compatible after all?

Regards,
Felix
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

lars.ellenberg at linbit

Jun 12, 2012, 10:44 AM

Post #8 of 8 (781 views)
Permalink

Re: upgrade drbd 8.3.7 -> 8.3.11 [In reply to]

On Tue, Jun 12, 2012 at 05:49:53PM +0200, Felix Frank wrote:
> Hi,
>
> On 06/12/2012 05:47 PM, Yves Serrano wrote:
> > [426929.703228] block drbd1: /var/lib/dkms/drbd8/8.3.7/build/drbd/drbd_receiver.c:1995: sector: 128s, size: 65536
> > [426929.719913] block drbd1: error receiving RSDataRequest, l: 24!
>
> this looks potentially enlightening, but not to me, sadly.
>
> Any idea, anyone? Are the versions not so compatible after all?

From the ChangeLog:

8.3.12 (api:88/proto:86-96)
--------
* Fixed a rare compatibility issue with DRBD's older than 8.3.7
when negotiating the bio_size

Looks like that one.

iirc, to trigger it you'd need a certain setup, AND a certain sequence
of drbd connection/disconnection events.
Not remembering the details right now.

--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD� and LINBIT� are registered trademarks of LINBIT, Austria.
__
please don't Cc me, but send to list -- I'm subscribed
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads