Mailing List Archive: MythTV: Users

imdp.pl oddities?

Index | Next | Previous | View Threaded

sarah at sarahhayes

Feb 17, 2008, 6:10 PM

Post #1 of 15 (3738 views)
Permalink

imdp.pl oddities?

Is anyone else getting odd behavior in the imdb lookup part of
MythVideo? It was working fine yesterday, but suddenly now removes any
edits to the metadata I've made and doesn't return anything.

Tried it on two seperate machines and two seperate files both running
20.1/ one Ubuntu and one Gentoo, same behavior.

DB up and ok, no software change made in the last 48 hours to either
machine.

Sarah

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

pebender at san

Feb 17, 2008, 8:00 PM

Post #2 of 15 (3650 views)
Permalink

Re: imdp.pl oddities? [In reply to]

Sarah Hayes wrote:
> Is anyone else getting odd behavior in the imdb lookup part of
> MythVideo? It was working fine yesterday, but suddenly now removes any
> edits to the metadata I've made and doesn't return anything.
>
> Tried it on two seperate machines and two seperate files both running
> 20.1/ one Ubuntu and one Gentoo, same behavior.
>
> DB up and ok, no software change made in the last 48 hours to either
> machine.
>
> Sarah

Yes. It stopped working for me. It worked this morning, but it does not
work this evening.
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

awithers at anduin

Feb 17, 2008, 8:12 PM

Post #3 of 15 (3650 views)
Permalink

Re: imdp.pl oddities? [In reply to]

> Is anyone else getting odd behavior in the imdb lookup part of
> MythVideo? It was working fine yesterday, but suddenly now removes any
> edits to the metadata I've made and doesn't return anything.

They appear to have started blocked user agents matching "lwp-trivial/1.41".
So anyone know of a site where we can comply with the Terms of Use in the
next week or so?

Yes changing the user agent works, simply switching away from LWP::Simple
works. I'm not really willing to deliberately circumvent their blocking even
if it is simple and untargeted.

--
Anduin Withers

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

sarah at sarahhayes

Feb 18, 2008, 4:37 AM

Post #4 of 15 (3646 views)
Permalink

Re: imdp.pl oddities? [In reply to]

Anduin Withers wrote:
>> Is anyone else getting odd behavior in the imdb lookup part of
>> MythVideo? It was working fine yesterday, but suddenly now removes any
>> edits to the metadata I've made and doesn't return anything.
>>
>
> They appear to have started blocked user agents matching "lwp-trivial/1.41".
> So anyone know of a site where we can comply with the Terms of Use in the
> next week or so?
>
> Yes changing the user agent works, simply switching away from LWP::Simple
> works. I'm not really willing to deliberately circumvent their blocking even
> if it is simple and untargeted.
>
>
I wonder if that means the number of Myth machines scraping them has
reached "annoyance" level? Perhaps they should just provide us the XML
data as a raw feed and stick adverts for themselves in it? :) Actually
that'd be nice piccie, blurb and "Provided by IMDB" along with watermarks.

I also echo the question of anyone know another imdb-esque place we can
point ourselves at?

Sarah
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

awithers at anduin

Feb 18, 2008, 9:59 AM

Post #5 of 15 (3616 views)
Permalink

Re: imdp.pl oddities? [In reply to]

> I wonder if that means the number of Myth machines scraping them has
> reached "annoyance" level? Perhaps they should just provide us the XML
> data as a raw feed and stick adverts for themselves in it? :) Actually
> that'd be nice piccie, blurb and "Provided by IMDB" along with watermarks.

I've sent an e-mail to them asking if we can get official permission (I can
find no evidence we ever had it). I have some hope as it appears Freevo was
granted permission. No idea if I'll have an answer or solution before
release.

That said, imdbpy.py could be made to use the DB they make available, this
would be the worst solution (the DB is large, with few mirrors, and some
information unavailable). Not to mention it would force everyone to install
imdbpy. Currently imdbpy.py works but requires the imdbpy Python module to
be installed (why it isn't just replacing imdb.pl for the release).

--
Anduin Withers

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

gary.dawes at gmail

Feb 18, 2008, 11:48 AM

Post #6 of 15 (3622 views)
Permalink

Re: imdp.pl oddities? [In reply to]

What about changing the script so it identified itself as a mythtv device?
Then the script works for now, we identify ourselves correctly and
accurately, and the guys at IMDB can actually see what is coming from Mythtv
users rather than a general mass of people, and gain an accurate
understanding of what is hitting their DB.

I for one spend much less time accessing IMDB from the mythbox, via scripts
or otherwise than I do manually.

Gary

On 18/02/2008, Anduin Withers <awithers [at] anduin> wrote:
>
> > I wonder if that means the number of Myth machines scraping them has
> > reached "annoyance" level? Perhaps they should just provide us the XML
> > data as a raw feed and stick adverts for themselves in it? :) Actually
> > that'd be nice piccie, blurb and "Provided by IMDB" along with
> watermarks.
>
> I've sent an e-mail to them asking if we can get official permission (I
> can
> find no evidence we ever had it). I have some hope as it appears Freevo
> was
> granted permission. No idea if I'll have an answer or solution before
> release.
>
> That said, imdbpy.py could be made to use the DB they make available, this
> would be the worst solution (the DB is large, with few mirrors, and some
> information unavailable). Not to mention it would force everyone to
> install
> imdbpy. Currently imdbpy.py works but requires the imdbpy Python module to
> be installed (why it isn't just replacing imdb.pl for the release).
>
> --
> Anduin Withers
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users [at] mythtv
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
>

awithers at anduin

Feb 18, 2008, 12:37 PM

Post #7 of 15 (3599 views)
Permalink

Re: imdp.pl oddities? [In reply to]

> What about changing the script so it identified itself as a mythtv device?
> Then the script works for now, we identify ourselves correctly and
accurately,
> and the guys at IMDB can actually see what is coming from Mythtv users
rather
> than a general mass of people, and gain an accurate understanding of what
is
> hitting their DB.

If we get permission I'll do that. Until recently I had assumed we had
permission (I removed Amazon.com support some time ago because it was
clear we couldn't comply with their AUP).

Just changing the user agent isn't enough, their current "Conditions of Use"
are unambiguous.

http://www.imdb.com/help/show_article?conditions

--
Anduin Withers

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

mikep at randomtraveller

Feb 19, 2008, 7:12 AM

Post #8 of 15 (3591 views)
Permalink

Re: imdp.pl oddities? [In reply to]

Anduin Withers wrote:
>> What about changing the script so it identified itself as a mythtv device?
>> Then the script works for now, we identify ourselves correctly and
> accurately,
>> and the guys at IMDB can actually see what is coming from Mythtv users
> rather
>> than a general mass of people, and gain an accurate understanding of what
> is
>> hitting their DB.
>
> If we get permission I'll do that. Until recently I had assumed we had
> permission (I removed Amazon.com support some time ago because it was
> clear we couldn't comply with their AUP).
>
> Just changing the user agent isn't enough, their current "Conditions of Use"
> are unambiguous.
>
> http://www.imdb.com/help/show_article?conditions
>
Is this something that could perhaps be tied in with SchedulesDirect? After all,
my finger in the air guess says that the volumes of data might be comparable,
and Schedules Direct was set up to solve a roughly similar situation, ie hordes
of Myth machines randomly hammering thier database.

Any way we can leverage that? Perhaps point IMDB to Schedules Direct and suggest
that as a model method of sloving the problem.

Mike Perkins
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

gislio at gmail

Feb 20, 2008, 9:00 AM

Post #9 of 15 (3562 views)
Permalink

Re: imdp.pl oddities? [In reply to]

Anything to report?

On Mon, Feb 18, 2008 at 5:59 PM, Anduin Withers <awithers [at] anduin> wrote:

> > I wonder if that means the number of Myth machines scraping them has
> > reached "annoyance" level? Perhaps they should just provide us the XML
> > data as a raw feed and stick adverts for themselves in it? :) Actually
> > that'd be nice piccie, blurb and "Provided by IMDB" along with
> watermarks.
>
> I've sent an e-mail to them asking if we can get official permission (I
> can
> find no evidence we ever had it). I have some hope as it appears Freevo
> was
> granted permission. No idea if I'll have an answer or solution before
> release.
>
> That said, imdbpy.py could be made to use the DB they make available, this
> would be the worst solution (the DB is large, with few mirrors, and some
> information unavailable). Not to mention it would force everyone to
> install
> imdbpy. Currently imdbpy.py works but requires the imdbpy Python module to
> be installed (why it isn't just replacing imdb.pl for the release).
>
> --
> Anduin Withers
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users [at] mythtv
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
>

awithers at anduin

Feb 20, 2008, 9:58 AM

Post #10 of 15 (3585 views)
Permalink

Re: imdp.pl oddities? [In reply to]

>> [IMDB broken]
> Anything to report?

No, I'm still waiting on a response to the e-mail I sent them (well, their
web form).

If someone has a better contact that would be useful. If I don't hear
anything soon I'll try contacting the Freevo project and see if they have a
contact they are willing to share.

--
Anduin Withers

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

mythtv at frouse3

Feb 20, 2008, 4:44 PM

Post #11 of 15 (3575 views)
Permalink

Re: [SPAM] Re: imdp.pl oddities? [In reply to]

>>> [IMDB broken]
>> Anything to report?
>
> No, I'm still waiting on a response to the e-mail I sent them (well, their
> web form).
>
> If someone has a better contact that would be useful. If I don't hear
> anything soon I'll try contacting the Freevo project and see if they have
> a
> contact they are willing to share.

In what circumstance is imdb.pl not working? I saw this thread yesterday
morning and attempted a few manual queries with imdb.pl and all were
sucessful.

I then noticed there were 3 entries for imdb.com in nscd and I tried 3
concurrent lookups (functionally should have cycled through all 3
entries), and again all were successful.

On Sunday, when I was testing the 1.8 version of my bulk updater I did
notice some blank entries coming back. However, I had chalked it up to
being a network anomaly at the time with my ISP.

In what circumstances is this reproducable?

Ryan
>
> --
> Anduin Withers
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users [at] mythtv
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
>

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

arbrandes at gmail

Feb 20, 2008, 5:15 PM

Post #12 of 15 (3567 views)
Permalink

Re: [SPAM] Re: imdp.pl oddities? [In reply to]

On Wed, Feb 20, 2008 at 9:44 PM, Ryan Pisani
<mythtv [at] frouse3> wrote:
> On Sunday, when I was testing the 1.8 version of my bulk updater I did
> notice some blank entries coming back.

I can confirm this behavior (blank titles), but only for TV shows.
Maybe the newer imdb.pl doesn't like quotes (present in all TV shows)?

Adolfo
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

awithers at anduin

Feb 20, 2008, 5:20 PM

Post #13 of 15 (3559 views)
Permalink

Re: [SPAM] Re: imdp.pl oddities? [In reply to]

> In what circumstance is imdb.pl not working? I saw this thread yesterday
> morning and attempted a few manual queries with imdb.pl and all were
> sucessful.

It does appear they are allowing LWP::Simple back in, when it was broken the
message on the page returned was clear about them blocking the user agent.

--
Anduin Withers

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

gislio at gmail

Feb 20, 2008, 10:57 PM

Post #14 of 15 (3560 views)
Permalink

Re: [SPAM] Re: imdp.pl oddities? [In reply to]

Yes, imdb.pl seems to be working correctly again. Does anyone know if this
is a temporary reprieve and whether we need to lobby IMDB on behalf of
MythVideo for a permanent solution?

On Thu, Feb 21, 2008 at 1:20 AM, Anduin Withers <awithers [at] anduin> wrote:

> > In what circumstance is imdb.pl not working? I saw this thread
> yesterday
> > morning and attempted a few manual queries with imdb.pl and all were
> > sucessful.
>
> It does appear they are allowing LWP::Simple back in, when it was broken
> the
> message on the page returned was clear about them blocking the user agent.
>
> --
> Anduin Withers
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users [at] mythtv
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
>

awithers at anduin

Feb 21, 2008, 8:45 AM

Post #15 of 15 (3545 views)
Permalink

Re: [SPAM] Re: imdp.pl oddities? [In reply to]

> Yes, imdb.pl seems to be working correctly again. Does anyone know
> if this is a temporary reprieve and whether we need to lobby IMDB
> on behalf of MythVideo for a permanent solution?

I have two e-mails out to them requesting the same permission Freevo
obtained.

I'm not taking them blocking the user agent string for many hours as
a good sign (also not taking it as directly targeting us as we use
the generic LWP::Simple one).

I am considering a MythTV specific user agent for the release. Easier
for us to be blocked but also doesn't lump us in with every other
simple perl script.

--
Anduin Withers

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads