zweiss at scout
May 23, 2012, 2:34 PM
Post #10 of 13
(846 views)
Permalink
|
On May 23, 2012, at 4:18 PM, Lars Ellenberg wrote:
> On Wed, May 23, 2012 at 04:12:23PM -0500, Zev Weiss wrote:
>>
>> On May 23, 2012, at 3:45 PM, Lars Ellenberg wrote:
>>
>>> On Wed, May 23, 2012 at 03:34:27PM -0500, Zev Weiss wrote:
>>>>
>>>> On May 23, 2012, at 3:22 PM, Florian Haas wrote:
>>>>
>>>>> On Wed, May 23, 2012 at 10:14 PM, Zev Weiss <zweiss [at] scout> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I'm running DRBD 8.3.12, and recently hit what looks to me like a bug that was listed as fixed in 8.3.13 -- getting into a state where both nodes are in SyncSource (it's just stuck like that, going nowhere). Luckily this happened on a test resource and not a live one, so it's not a big problem, but I was wondering if there were any known ways of recovering it without doing anything disruptive to the other resources (e.g. rebooting or unloading the kernel module).
>>>>>>
>>>>>> I've tried 'drbdadm down', but it just hangs -- anyone have any other suggestions? It doesn't really matter to me if it wipes the resource or anything, I'd just like to have my test device back in a working state without disturbing anything else.
>>>>>
>>>>> Can you post /proc/drbd contents from both nodes here?
>>>>>
>>>>
>>>> Sure -- here's one node:
>>>>
>>>> version: 8.3.12 (api:88/proto:86-96)
>>>> GIT-hash: e2a8ef4656be026bbae540305fcb998a5991090f build by zweiss [at] mydomai, 2012-03-14 19:52:38
>>>>
>>>> <snip other resources>
>>>> 9: cs:SyncSource ro:Secondary/Primary ds:UpToDate/Inconsistent C r-----
>>>> ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:65536
>>>> [>...................] sync'ed: 5.9% (65536/65536)K
>>>> finish: 19046:04:53 speed: 0 (0 -- 0) K/sec (stalled)
>>>> 0% sector pos: 0/10698352
>>>> resync: used:0/61 hits:0 misses:0 starving:0 dirty:0 changed:0
>>>> act_log: used:0/3389 hits:0 misses:0 starving:0 dirty:0 changed:0
>>>
>>> drbdsetup 9 disconnect --force
>>> may work,
>>> if you did not try a non-forced disconnect or similar before,
>>> that is to say, if the drbd worker thread is not blocked yet.
>>>
>>
>> I think I had tried a non-forced disconnect previously (and perhaps
>> also implicitly as part of a 'down' attempt, though I'm not sure
>> whether it would have gotten to that step if the disconnect operation
>> didn't complete), but 'drbdsetup 9 disconnect --force' also just
>> hangs.
>>
>>> You can always cut the tcp connection using iptables,
>>> which should at least get the worker into a responsive state again.
>>>
>>
>> As mentioned in another message in response to Florian, blocking the replication port via iptables doesn't seem to have had any effect.
>
> You would not need to "block" as in DROP, but to REJECT with tcp reset.
>
Sorry, should have worded that more carefully -- it wasn't strictly a DROP, but a REJECT, with icmp-port-unreachable. I've since tweaked it to reject with tcp-reset instead. No changes in DRBD state on either side as far as I can see, though both nodes still seem to have (or at least think they have) a tcp connection or two involving that port, despite my efforts with iptables:
[root [at] node ~]# netstat -tn | fgrep 7789
tcp 156 0 192.168.1.2:37324 192.168.1.1:7789 ESTABLISHED
tcp 0 0 192.168.1.2:7789 192.168.1.1:47548 ESTABLISHED
[root [at] node ~]# netstat -tn | fgrep 7789
tcp 0 0 192.168.1.1:47548 192.168.1.2:7789 ESTABLISHED
For what it's worth, node1 is the one that thinks the resource is Secondary/Secondary, node2 is the one that shows it as Secondary/Primary.
> You could also try "ifdown" sleep a while and ifup again.
> (which obviously will impact the other resources, and everything going
> via that interface).
>
Right, but I don't really want to disrupt replication on all the other (production) resources, so just leaving it as-is until my next scheduled maintenance reboot (at which point I plan on would be preferable.
Thanks,
Zev
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user
|
