Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Wikipedia: Wikitech

Wikidata update percolation

 

 

Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded


daniel at brightbyte

Oct 15, 2012, 1:31 AM

Post #1 of 1 (360 views)
Permalink
Wikidata update percolation

Hi!

I have published a draft of how changes on the wikidata repository are going to
percolate to the client wikis:

https://meta.wikimedia.org/wiki/Wikidata/Notes/Percolation

Any feedback would be appreciated!

Of course, we are not starting this from scratch. We are currently implementing
a stripped down, naive version of the draft. Basically, it works like this:

* Each change on the repository is recorded in the changes table.
* On each client wiki, a poll script periodically checks the changes table.
* The polling script maintains a local copy of the latest version of each data
entity on each cluster used by client wikis.
* If any page on the wiki is affected by the change, an entry representing that
change is injected into the client's recentchanges table.
* Wiki pages that are affected by a change are invalidated.

I think this will work for now, that is, for a small number of client wikis. The
new draft is an attempt to make this architecture scale up to several hundred
client wikis on multiple database clusters.

-- daniel


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.