Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Wikipedia: Wikitech

Evaluating Google Summer of Code

 

 

Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded


z at mzmcbride

Aug 27, 2012, 4:32 PM

Post #1 of 10 (2304 views)
Permalink
Evaluating Google Summer of Code

(Splitting this off from John's critique of ConventionExtension.)

Hi.

MediaWiki has participated in several (Google) Summer of Code iterations now
(<https://www.mediawiki.org/wiki/Summer_of_Code>) and I'm wondering how this
partnership program is evaluated.

Whenever this program wraps up at the end of the (Northern Hemisphere's)
summer, I always sense a worrying amount of frustration and annoyance from
all parties involved. The projects are usually overly large and complex and
from what I understand, nearly all of the projects from Google Summer of
Code don't end up in production environments. If the projects are lucky,
they end up in a MediaWiki extension; if they're unlucky, they rot away in a
code repo branch somewhere or behind a configuration variable set to false
by default. The end result being that:

* the people who worked on these projects are frustrated and annoyed because
they didn't get their code deployed [.to Wikimedia wikis, a wide audience, or
anyone at all in some cases];

* the people who mentored these students are frustrated and annoyed for
similar reasons; and

* the people (end-users) who wanted to see these projects successfully
completed are frustrated and annoyed that these features still don't exist.

So I'm left wondering how the cost v. benefit equation works out for this
program. How do you evaluate the program and whether MediaWiki ought to
remain a continued participant?

And, of course, should MediaWiki decide not to participate in Google Summer
of Code in 2013, are there other [better] ideas for getting people involved
in MediaWiki development?

MZMcBride



_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


z at mzmcbride

Aug 27, 2012, 4:41 PM

Post #2 of 10 (2221 views)
Permalink
Re: Evaluating Google Summer of Code [In reply to]

MZMcBride wrote:
> MediaWiki has participated in several (Google) Summer of Code iterations now
> (<https://www.mediawiki.org/wiki/Summer_of_Code>) and I'm wondering how this
> partnership program is evaluated.

After I posted this, Sumana pointed out that MaxSem has done a great
evaluation here: <https://www.mediawiki.org/wiki/User:MaxSem/GSoC_analysis>.

There's also a good overview of past projects at
<https://www.mediawiki.org/wiki/Summer_of_Code_Past_Projects>.

> And, of course, should MediaWiki decide not to participate in Google Summer
> of Code in 2013, are there other [better] ideas for getting people involved
> in MediaWiki development?

Perhaps a better model: <https://www.mediawiki.org/wiki/UCOSP_Spring_2012>?

MZMcBride



_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


cmcmahon at wikimedia

Aug 27, 2012, 4:58 PM

Post #3 of 10 (2221 views)
Permalink
Re: Evaluating Google Summer of Code [In reply to]

Most software projects fail (for some definition of "fail"). Even for
highly skilled and highly experienced companies and shops, most software
projects fail. I'm not going to look up the Gartner and Forrester and
Chaos reports this late on a Monday night, but google away.

GSoC is an investment that is not intended to have a short-term payoff.
The fact that ANY GSoC code makes to production is fantastic.

GSoC is an investment in the long term. It is intended to provide real
concrete experience to promising students in real environments, including
all the frustrations and annoyances that everyone on a software team
experiences in the real world all the time. Schools simply do not provide
that experience. Some fraction of those participants will take those
experiences into the future of software development, to make real
improvements, both to code and to process.

Furthermore, considering GSoC solely in terms of benefit to
Mediawiki/Wikipedia is short-sighted. Take a look at the organizations
participating:
http://www.google-melange.com/gsoc/projects/list/google/gsoc2012 . What
would your opinion be if WMF were not on that list?




On Mon, Aug 27, 2012 at 5:32 PM, MZMcBride <z [at] mzmcbride> wrote:

> (Splitting this off from John's critique of ConventionExtension.)
>
> Hi.
>
> MediaWiki has participated in several (Google) Summer of Code iterations
> now
> (<https://www.mediawiki.org/wiki/Summer_of_Code>) and I'm wondering how
> this
> partnership program is evaluated.
>
> Whenever this program wraps up at the end of the (Northern Hemisphere's)
> summer, I always sense a worrying amount of frustration and annoyance from
> all parties involved. The projects are usually overly large and complex and
> from what I understand, nearly all of the projects from Google Summer of
> Code don't end up in production environments. If the projects are lucky,
> they end up in a MediaWiki extension; if they're unlucky, they rot away in
> a
> code repo branch somewhere or behind a configuration variable set to false
> by default. The end result being that:
>
> * the people who worked on these projects are frustrated and annoyed
> because
> they didn't get their code deployed [.to Wikimedia wikis, a wide audience,
> or
> anyone at all in some cases];
>
> * the people who mentored these students are frustrated and annoyed for
> similar reasons; and
>
> * the people (end-users) who wanted to see these projects successfully
> completed are frustrated and annoyed that these features still don't exist.
>
> So I'm left wondering how the cost v. benefit equation works out for this
> program. How do you evaluate the program and whether MediaWiki ought to
> remain a continued participant?
>
> And, of course, should MediaWiki decide not to participate in Google Summer
> of Code in 2013, are there other [better] ideas for getting people involved
> in MediaWiki development?
>
> MZMcBride
>
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


z at mzmcbride

Aug 27, 2012, 6:07 PM

Post #4 of 10 (2220 views)
Permalink
Re: Evaluating Google Summer of Code [In reply to]

Chris McMahon wrote:
> Most software projects fail (for some definition of "fail"). Even for
> highly skilled and highly experienced companies and shops, most software
> projects fail. I'm not going to look up the Gartner and Forrester and
> Chaos reports this late on a Monday night, but google away.
>
> GSoC is an investment that is not intended to have a short-term payoff.
> The fact that ANY GSoC code makes to production is fantastic.
>
> GSoC is an investment in the long term. It is intended to provide real
> concrete experience to promising students in real environments, including
> all the frustrations and annoyances that everyone on a software team
> experiences in the real world all the time. Schools simply do not provide
> that experience. Some fraction of those participants will take those
> experiences into the future of software development, to make real
> improvements, both to code and to process.

Thank you for this. You make some good points.

So I guess it might just be a matter of better managing expectations of all
parties involved? The disappointment factor is serious and dangerous, in my
opinion. It can be mitigated by setting appropriate expectations for the
students involved, the mentors involved, and the end-users involved, many of
whom desperately want to see some of these features live.

> Furthermore, considering GSoC solely in terms of benefit to
> Mediawiki/Wikipedia is short-sighted. Take a look at the organizations
> participating:
> http://www.google-melange.com/gsoc/projects/list/google/gsoc2012 . What
> would your opinion be if WMF were not on that list?

Personally, I don't care very much about being a participant for the sake of
being a participant (and I imagine many others feel similarly). I think for
a lot of people who watch these Summer of Code projects, it _is_ about
benefit to MediaWiki/Wikimedia, particularly as getting involved in these
projects can detract from already painfully finite mentoring and reviewing
resources.

MZMcBride



_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


lydia.pintscher at wikimedia

Aug 27, 2012, 11:27 PM

Post #5 of 10 (2218 views)
Permalink
Re: Evaluating Google Summer of Code [In reply to]

On Tue, Aug 28, 2012 at 3:07 AM, MZMcBride <z [at] mzmcbride> wrote:
>> Furthermore, considering GSoC solely in terms of benefit to
>> Mediawiki/Wikipedia is short-sighted. Take a look at the organizations
>> participating:
>> http://www.google-melange.com/gsoc/projects/list/google/gsoc2012 . What
>> would your opinion be if WMF were not on that list?
>
> Personally, I don't care very much about being a participant for the sake of
> being a participant (and I imagine many others feel similarly). I think for
> a lot of people who watch these Summer of Code projects, it _is_ about
> benefit to MediaWiki/Wikimedia, particularly as getting involved in these
> projects can detract from already painfully finite mentoring and reviewing
> resources.

I think you touch the most important point here. I'm one of the main
people who manage GSoC for KDE. KDE is the biggest org taking part in
GSoC this year in terms of number of students mentored. In addition we
are running our own program (Season of KDE) next to it. So in total
I'd say we've mentored about 100 students through these programs this
year alone. We didn't get there by accident. You know what the main
benefit of GSoC is in my opinion? Getting an organisation into the
mindset of mentoring - into the mindset of "yes we need to train new
people because they can do awesome things even if they screw up
sometimes on the way there".


Cheers
Lydia

--
Lydia Pintscher - http://about.me/lydia.pintscher
Community Communications for Wikidata

Wikimedia Deutschland e.V.
Obentrautstr. 72
10963 Berlin
www.wikimedia.de

Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.

Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg
unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das
Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


rlane32 at gmail

Aug 28, 2012, 12:41 AM

Post #6 of 10 (2230 views)
Permalink
Re: Evaluating Google Summer of Code [In reply to]

> I think you touch the most important point here. I'm one of the main
> people who manage GSoC for KDE. KDE is the biggest org taking part in
> GSoC this year in terms of number of students mentored. In addition we
> are running our own program (Season of KDE) next to it. So in total
> I'd say we've mentored about 100 students through these programs this
> year alone. We didn't get there by accident. You know what the main
> benefit of GSoC is in my opinion? Getting an organisation into the
> mindset of mentoring - into the mindset of "yes we need to train new
> people because they can do awesome things even if they screw up
> sometimes on the way there".
>

+1000. I <3 that way of thinking and I'd very much like our community
to have the same attitude towards GSoC. We're a community working
towards education of the world. Participating in GSoC as a mentoring
organization is a right fit.

- Ryan

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


hennar at gmail

Aug 28, 2012, 1:44 AM

Post #7 of 10 (2213 views)
Permalink
Re: Evaluating Google Summer of Code [In reply to]

On Tue, Aug 28, 2012 at 9:41 AM, Ryan Lane <rlane32 [at] gmail> wrote:
>> I think you touch the most important point here. I'm one of the main
>> people who manage GSoC for KDE. KDE is the biggest org taking part in
>> GSoC this year in terms of number of students mentored. In addition we
>> are running our own program (Season of KDE) next to it. So in total
>> I'd say we've mentored about 100 students through these programs this
>> year alone. We didn't get there by accident. You know what the main
>> benefit of GSoC is in my opinion? Getting an organisation into the
>> mindset of mentoring - into the mindset of "yes we need to train new
>> people because they can do awesome things even if they screw up
>> sometimes on the way there".
>>
>
> +1000. I <3 that way of thinking and I'd very much like our community
> to have the same attitude towards GSoC. We're a community working
> towards education of the world. Participating in GSoC as a mentoring
> organization is a right fit.

Agreed.

GSoC imho is not about getting code written but getting people
involved in our community. As such we should look and see how many
people remain involved after GSoC.

Finne

--
"Maybe you knew early on that your track went from point A to B, but
unlike you I wasn't given a map at birth!" Alyssa, "Chasing Amy"

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


yaron at wikiworks

Aug 28, 2012, 10:13 AM

Post #8 of 10 (2212 views)
Permalink
Re: Evaluating Google Summer of Code [In reply to]

Hi,

It seems like some people are saying that getting workable code out of the
Google Summer of Code is a relatively minor aspect of the program - maybe
the third or fourth most important outcome of it, behind things like
finding new developers, getting better at mentoring, teaching software
development, etc. I disagree with this view - I think creating useful
projects is the most important part of GSoC, and that this aspect of it
should be taken much more seriously than it is. That's for a few reasons:

1) It's Google's money that's paying for all of this, and in their listing
of goals for GSoC, they do list most of these things in some form, but the
#1 goal they list is "Create and release open source code for the benefit
of all". [1]

2) It's a real waste to not use this resource to get actual improvements to
the software, given the (relatively) limited resources that the Wikimedia
Foundation and MediaWiki have. GSoC offers free money, and a framework, for
getting development done - it's a tremendous resource that should be taken
full advantage of.

3) As MZ notes, even from the vantage point of recruiting developers,
having unsuccessful projects is a problem - putting all that effort in,
with nothing in the end to show for it, can be demoralizing. And that sort
of thing can accumulate: if talented young open-source developers are
considering doing a GSoC project for Wikimedia, and they look at the track
record for the WMF's previous GSoC project, they may draw negative
conclusions about it, and maybe about doing MediaWiki development in
general.

So how can this be improved? Having mentored three successful GSoC projects
for the WMF, I have some perspective on this. As MZ also notes, WMF
projects so far have tended to be too large in scope for the length of time
allotted. But that's not something that's easy to correct - estimating
development time is always a challenge, even when the developers involved
are experienced and not newbies. And smaller projects tend not to inspire
anyone, mentors or students.

So here is my proposal: there should be in place some plan of action,
ideally for every project, in case it goes overlong and doesn't get
completed in time. That can take several forms: a commitment by the mentor
or another experienced developer to finish up the project; a commitment by
the WMF to fund the student to finish it, if the student turns out to be
serious and it's just a simple lack of time that was the issue; a
commitment by the WMF to fund someone else to finish it or just a
commitment by the student to finish it themselves, on a volunteer basis.
The last of those is tricky, because that tends to be the conclusion at the
end of uncompleted projects anyway - the student keeps working on it, but
that usually only lasts for a few months before the student's school work
and/or regular work get in the way, and then the project often falls by the
wayside. A commitment on the WMF and/or mentor side would be the ideal
thing - and if there's no such guarantee for a specific project, then that
should be taken into consideration when deciding on whether to accept it.

Of the three projects I mentored, none of them produced something that was
fully usable on the final day of GSoC. In all three cases, more work was
put in - the first time, by the student, the second two times by me. The
post-GSoC work was significant in all cases, but it was still a lot less
than it would have been to try to do the project from scratch. It was a
comparatively small amount of effort, to get the code to at least
"beta"-level or higher, that ended up making a huge difference. Something
like that, from what I've seen, is almost always needed - so it should be
factored into the planning.

[1]
http://www.google-melange.com/document/show/gsoc_program/google/gsoc2012/faqs#goals

-Yaron

--
WikiWorks MediaWiki Consulting http://wikiworks.com
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


sumanah at wikimedia

Nov 18, 2012, 5:17 AM

Post #9 of 10 (1833 views)
Permalink
Re: Evaluating Google Summer of Code [In reply to]

TL;DR: we're improving at student retention but need to get better at
producing something useful at the end of a mentorship period; some
suggestions follow.

On 08/28/2012 04:44 AM, Finne Boonen wrote:
> On Tue, Aug 28, 2012 at 9:41 AM, Ryan Lane <rlane32 [at] gmail> wrote:
>>> I think you touch the most important point here. I'm one of the main
>>> people who manage GSoC for KDE. KDE is the biggest org taking part in
>>> GSoC this year in terms of number of students mentored. In addition we
>>> are running our own program (Season of KDE) next to it. So in total
>>> I'd say we've mentored about 100 students through these programs this
>>> year alone. We didn't get there by accident. You know what the main
>>> benefit of GSoC is in my opinion? Getting an organisation into the
>>> mindset of mentoring - into the mindset of "yes we need to train new
>>> people because they can do awesome things even if they screw up
>>> sometimes on the way there".
>>>
>>
>> +1000. I <3 that way of thinking and I'd very much like our community
>> to have the same attitude towards GSoC. We're a community working
>> towards education of the world. Participating in GSoC as a mentoring
>> organization is a right fit.
>
> Agreed.
>
> GSoC imho is not about getting code written but getting people
> involved in our community. As such we should look and see how many
> people remain involved after GSoC.
>
> Finne

It has now been three months since the "pencils down" late in late
August when GSoC 2012 officially ended. Therefore it's a good time to
look at the nine students we accepted to see how many are still involved
with MediaWiki, how many of the GSoC projects themselves are anywhere
near what their original targets were, and whether we would prefer to
try this again next year or go for a different approach to outreach and
mentoring.

The answers:
* seven of the nine students are still actively working on MediaWiki
development. This is better than last year -- at this time last year,
only about 2-3 of the eight accepted students remained involved.
* one project hit its target (including merge into trunk) and about four
others are partially merged into trunk and still have momentum towards
completion. This is a little better than last year; about 2 projects
from last year are fully merged into trunk and about 1 more is partially
merged but there's no momentum towards the finish.

https://www.mediawiki.org/wiki/Summer_of_Code_2012/management reflects
how I wanted to run this year's GSoC. I think we did well at "People are
more important than code" selection, which is one reason we're doing
well at retention. But I failed at ensuring that students were scoping
their proposals at about 6 weeks of coding work and dedicating the
entire last month of the summer to bugfixing and code review. The
proposals usually allotted either no time or about 2 weeks for merging
with trunk, pre-deploy code review, and integration. I think smaller
scoping would have helped, so in the future, we should simply not accept
proposals that do not allot at least a third of the summer to code
review and response to code review.

I also failed at staying hands-on enough in ensuring frequent
mentor-student communication. Having multiple hands-on mentors who were
available to do code review REALLY helps, as we see from Nischay
Nahata's success (he is our only student this year who got all his
project code merged), and I think that in future years we should mandate
that all students have two mentors, each of whom commit to reviewing new
patchsets within 2 business days. Students can't afford delays in code
review. The WMF is improving how it makes time for its engineers to
mentor newer contributors (more on that this week) and that will help
with this; we're also growing experienced volunteer developers who are
interested in mentoring, so I think a 2-mentors-per-student mandate is
doable. We should also mandate twice-weekly voice or videocall meetings
and do spot checks to ensure they're happening; multiple students didn't
have enough communication with their mentors and were shy about pushing
for it.

Now, to discuss future mentoring approaches:

I tried to get us into UCOSP again for fall 2012 and was told they were
out of slots, so I'm pushing to get us into UCOSP for spring 2013.

We couldn't muster enough mentors to do Google Code-In well so we aren't
doing it this year.

Quim Gil is the administrator for our participation in the Outreach
Program for Women, so I'll let him discuss how he wants to manage that
and how he'll be setting expectations appropriately. :-) I think
mandating two mentors per participant is a very good idea, Quim, what do
you think? Also: instead of making newbies write fixed proposals with
schedules, these OPW internships can follow broader charters and flex to
accommodate students' interests and abilities, and it's easier to
rescope iteratively to ensure that something useful gets produced by the
end of the three months. The fact that some of the internships will be
in marketing or communications will help with this, as it's easier to
get fast critique of those deliverables. (Also, participants don't have
to be students, so some of them will be more mature!)

And regarding future Google Summer of Code management, Yaron Koren wrote:

> So here is my proposal: there should be in place some plan of action,
> ideally for every project, in case it goes overlong and doesn't get
> completed in time. That can take several forms: a commitment by the mentor
> or another experienced developer to finish up the project; a commitment by
> the WMF to fund the student to finish it, if the student turns out to be
> serious and it's just a simple lack of time that was the issue; a
> commitment by the WMF to fund someone else to finish it or just a
> commitment by the student to finish it themselves, on a volunteer basis.
> The last of those is tricky, because that tends to be the conclusion at the
> end of uncompleted projects anyway - the student keeps working on it, but
> that usually only lasts for a few months before the student's school work
> and/or regular work get in the way, and then the project often falls by the
> wayside. A commitment on the WMF and/or mentor side would be the ideal
> thing - and if there's no such guarantee for a specific project, then that
> should be taken into consideration when deciding on whether to accept it.

This is worth discussing when we decide whether to apply for Google
Summer of Code 2013 (if Google decides to run it again).

Does anyone particularly agree or disagree?

--
Sumana Harihareswara
Engineering Community Manager
Wikimedia Foundation

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


qgil at wikimedia

Nov 20, 2012, 3:02 PM

Post #10 of 10 (1823 views)
Permalink
Re: Evaluating Google Summer of Code [In reply to]

Thank you for this summary. It is very useful.

Would you mind commenting more on lessons learned about the selection of
candidates? We need the right now for

https://www.mediawiki.org/wiki/Outreach_Program_for_Women


On 11/18/2012 05:17 AM, Sumana Harihareswara wrote:
> Quim Gil is the administrator for our participation in the Outreach
> Program for Women, so I'll let him discuss how he wants to manage that
> and how he'll be setting expectations appropriately. :-)

As commented in another thread, I have just drafted

https://www.mediawiki.org/wiki/Talk:Mentorship_programs

> I think
> mandating two mentors per participant is a very good idea, Quim, what do
> you think?

Defaulting to two is ok, but we can be flexible here - in both ways.
Sometimes a single mentor can be trustworthy, sometimes not even two
will convince us. ;)

Just like with candidates, there is no magic formula for mentors
(neither for program admins) :)

> Also: instead of making newbies write fixed proposals with
> schedules, these OPW internships can follow broader charters and flex to
> accommodate students' interests and abilities, and it's easier to
> rescope iteratively to ensure that something useful gets produced by the
> end of the three months.

I like a lot your "People are more important than code" meme but we need
to find a balance here.

I agree that the candidate shouldn't spend much time with formalities
like defining whether task X will take exactly 2 or 3 weeks. Still, it
is good (for everybody) to go through that exercise before presenting a
proposal. It filters those excessively lazy or unskilled and it helps
the candidate and the mentor to see whether the generic idea survives a
first reality check.

Still, your concerns are valid, and I have tried to address them in the
list of selection criteria.

https://www.mediawiki.org/wiki/Talk:Mentorship_programs#Criteria

>> So here is my proposal: there should be in place some plan of action,
>> ideally for every project, in case it goes overlong and doesn't get
>> completed in time.

Good idea. Also included in the selection criteria.

--
Quim Gil
Technical Contributor Coordinator
Wikimedia Foundation

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.