Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Wikipedia: Wikitech

Database replication

 

 

Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded


niklas.laxstrom at gmail

Jun 16, 2012, 1:34 AM

Post #1 of 4 (216 views)
Permalink
Database replication

I have now had multiple issues in Translate extension that are related
to database replication. After doing write to the master, some
maintenance tasks kick in and use slave to read the (now stale) data.

I've heard that there is some automatic handling for this, so that
editors for example see the latest version of Article after saving it.
But apparently there is no such thing, or at least it doesn't kick in
automatically.

What should I do? I don't fancy adding parameters to many methods to
define whether to use master or slave for data retrieval. Is there
method that waits for the used slave to catch up with the state after
last write?
-Niklas

--
Niklas Laxström

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


hashar+wmf at free

Jun 16, 2012, 5:26 AM

Post #2 of 4 (197 views)
Permalink
Re: Database replication [In reply to]

Niklas Laxström wrote:
> I have now had multiple issues in Translate extension that are related
> to database replication. After doing write to the master, some
> maintenance tasks kick in and use slave to read the (now stale) data.

The maintenance script need to wait for the slave to catchup the master
position. The wfWaitForSlaves() global function is exactly intended for
that.
A typical use cases is when you make tons of writes to master and dont
want to break replication or have the slaves lagging too much. You would
split the writes in batches and then wait until slaves catch up.

Have a look at maintenance/runBatchedQuery.php for an easy to read example.


> I've heard that there is some automatic handling for this, so that
> editors for example see the latest version of Article after saving it.
> But apparently there is no such thing, or at least it doesn't kick in
> automatically.

We had that issue at one point, I am not sure how it get fixed though.
Maybe we fetch the article from the master database to make sure it is
up to date after the user hit the save button.

> What should I do? I don't fancy adding parameters to many methods to
> define whether to use master or slave for data retrieval. Is there
> method that waits for the used slave to catch up with the state after
> last write?
> -Niklas

What is your code? :-)

(note I am out for the weekend so catch up with that Monday).

--
Antoine "hashar" Musso


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


Platonides at gmail

Jun 16, 2012, 8:56 AM

Post #3 of 4 (194 views)
Permalink
Re: Database replication [In reply to]

On 16/06/12 14:26, Antoine Musso wrote:
>> I've heard that there is some automatic handling for this, so that
>> editors for example see the latest version of Article after saving it.
>> But apparently there is no such thing, or at least it doesn't kick in
>> automatically.
>
> We had that issue at one point, I am not sure how it get fixed though.
> Maybe we fetch the article from the master database to make sure it is
> up to date after the user hit the save button.

You mean the recent bug 37225 ? (stale data problems have happened
several times in the past). It should have fixed with
https://gerrit.wikimedia.org/r/#/c/10722/1


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


niklas.laxstrom at gmail

Jun 16, 2012, 12:13 PM

Post #4 of 4 (198 views)
Permalink
Re: Database replication [In reply to]

On 16 June 2012 15:26, Antoine Musso <hashar+wmf [at] free> wrote:
> Niklas Laxström wrote:
>> I have now had multiple issues in Translate extension that are related
>> to database replication. After doing write to the master, some
>> maintenance tasks kick in and use slave to read the (now stale) data.
>
> The maintenance script need to wait for the slave to catchup the master
> position.  The wfWaitForSlaves() global function is exactly intended for
> that.
> A typical use cases is when you make tons of writes to master and dont
> want to break replication or have the slaves lagging too much. You would
> split the writes in batches and then wait until slaves catch up.
>
> Have a look at maintenance/runBatchedQuery.php for an easy to read example.

The latest case is https://bugzilla.wikimedia.org/37647 - it is not a
maintenance script and the problem is not lot of writes, but getting
fresh data after write.
-Niklas

--
Niklas Laxström

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.