Mailing List Archive: MythTV: Mythtvnz

Python errors in tv_grab_nz-py so mythfilldatabase fails

Index | Next | Previous | View Threaded

g8ecj at gilks

Jul 3, 2011, 3:44 AM

Post #1 of 7 (773 views)
Permalink

Python errors in tv_grab_nz-py so mythfilldatabase fails

Greetings

Just noticed I've got only 2 days of listings left and when I run
mythfilldatabase I get the following from tv_grab_nz-py.

2011-07-03 22:29:22.272 Using runtime prefix = /usr
2011-07-03 22:29:22.272 Using configuration directory = /home/mythtv/.mythtv
2011-07-03 22:29:22.273 Empty LocalHostName.
2011-07-03 22:29:22.273 Using localhost value of jupiter
2011-07-03 22:29:22.273 Testing network connectivity to '192.168.1.10'
2011-07-03 22:29:24.393 New DB connection, total: 1
2011-07-03 22:29:24.682 Connected to database 'mythconverg' at host:
192.168.1.10
2011-07-03 22:29:24.684 Closing DB connection named 'DBManager0'
2011-07-03 22:29:24.788 Connected to database 'mythconverg' at host:
192.168.1.10
2011-07-03 22:29:24.912 Current locale en_NZ
2011-07-03 22:29:24.937 No locale defaults file for en_NZ, skipping
2011-07-03 22:29:25.052 Loading en_gb translation for module mythfrontend
2011-07-03 22:29:25.214 Current MythTV Schema Version (DBSchemaVer): 1264
2011-07-03 22:29:25.438 New DB connection, total: 2
2011-07-03 22:29:25.495 Connected to database 'mythconverg' at host:
192.168.1.10
2011-07-03 22:29:25.567 mythfilldatabase: Listings Download Started
2011-07-03 22:29:25.700 Updating source #2 (S-Video) with grabber
tv_grab_nz-py
2011-07-03 22:29:25.756 Found 69 channels for source 2 which use grabber
2011-07-03 22:29:36.204 Grabber has capabilities: baseline manualconfig
preferredmethod cache
2011-07-03 22:29:44.731 Grabber prefers method: allatonce
2011-07-03 22:29:44.821 New DB connection, total: 3
2011-07-03 22:29:44.880 Connected to database 'mythconverg' at host:
192.168.1.10
2011-07-03 22:29:44.881 XMLTV config file is:
/home/mythtv/.mythtv/S-Video.xmltv
2011-07-03 22:29:44.881 New DB connection, total: 4
2011-07-03 22:29:44.970 Connected to database 'mythconverg' at host:
192.168.1.10
Traceback (most recent call last):
File "/usr/local/bin/tv_grab_nz-py", line 257, in <module>
text = getListings(cache=options.cache)
File "/usr/local/bin/tv_grab_nz-py", line 87, in getListings
res = doDownload(source)
File "/usr/local/bin/tv_grab_nz-py", line 64, in doDownload
h.request('GET', url, headers=headers)
File "/usr/lib64/python2.7/httplib.py", line 941, in request
self._send_request(method, url, body, headers)
File "/usr/lib64/python2.7/httplib.py", line 975, in _send_request
self.endheaders(body)
File "/usr/lib64/python2.7/httplib.py", line 937, in endheaders
self._send_output(message_body)
File "/usr/lib64/python2.7/httplib.py", line 797, in _send_output
self.send(msg)
File "/usr/lib64/python2.7/httplib.py", line 759, in send
self.connect()
File "/usr/lib64/python2.7/httplib.py", line 740, in connect
self.timeout, self.source_address)
File "/usr/lib64/python2.7/socket.py", line 553, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno -2] Name or service not known
2011-07-03 22:29:51.952 FillData, Error: xmltv returned error code 256
2011-07-03 22:29:51.981 Error in 1:1: unexpected end of file
2011-07-03 22:29:52.032 IconData: Updating icons for sourceid: 2

I'm running a Gentoo based system and having been through the emerge log I
can't find any major changes to python in the last week apart from the
following and I can't see that these are relevant to the grabber :(

1308772619: === (15 of 37) Compiling/Merging
(dev-python/numpy-1.6.0::/usr/portage/dev-python/numpy/numpy-1.6.0.ebuild)
1308772999: === (16 of 37) Compiling/Merging
(dev-python/argparse-1.2.1::/usr/portage/dev-python/argparse/argparse-1.2.1.ebuild)
1308773400: === (23 of 37) Compiling/Merging
(app-admin/python-updater-0.9::/usr/portage/app-admin/python-updater/python-updater-0.9.ebuild)
1309648157: === (18 of 68) Compiling/Merging
(dev-python/sip-4.12.3-r2::/usr/portage/dev-python/sip/sip-4.12.3-r2.ebuild)
1309648828: === (31 of 68) Compiling/Merging
(dev-python/pygobject-2.28.6::/usr/portage/dev-python/pygobject/pygobject-2.28.6.ebuild)
1309650622: === (50 of 68) Compiling/Merging
(dev-python/pygtk-2.24.0-r1::/usr/portage/dev-python/pygtk/pygtk-2.24.0-r1.ebuild)

Hope someone has some ideas...

--
Robin Gilks

_______________________________________________
mythtvnz mailing list
mythtvnz [at] lists
http://lists.ourshack.com/mailman/listinfo/mythtvnz
Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/

stephen_agent at jsw

Jul 3, 2011, 7:08 AM

Post #2 of 7 (747 views)
Permalink

Re: Python errors in tv_grab_nz-py so mythfilldatabase fails [In reply to]

On Sun, 3 Jul 2011 22:44:26 +1200, you wrote:

>Greetings
>
>Just noticed I've got only 2 days of listings left and when I run
>mythfilldatabase I get the following from tv_grab_nz-py.

[snip]
> File "/usr/lib64/python2.7/socket.py", line 553, in create_connection
> for res in getaddrinfo(host, port, 0, SOCK_STREAM):
>socket.gaierror: [Errno -2] Name or service not known
[snip]
>Hope someone has some ideas...

That looks like a DNS lookup failure. What does an nslookup of
nzepg.org show?

_______________________________________________
mythtvnz mailing list
mythtvnz [at] lists
http://lists.ourshack.com/mailman/listinfo/mythtvnz
Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/

g8ecj at gilks

Jul 3, 2011, 2:11 PM

Post #3 of 7 (747 views)
Permalink

Re: Python errors in tv_grab_nz-py so mythfilldatabase fails [In reply to]

> On Sun, 3 Jul 2011 22:44:26 +1200, you wrote:
>
>>Greetings
>>
>>Just noticed I've got only 2 days of listings left and when I run
>>mythfilldatabase I get the following from tv_grab_nz-py.
>
> [snip]
>> File "/usr/lib64/python2.7/socket.py", line 553, in create_connection
>> for res in getaddrinfo(host, port, 0, SOCK_STREAM):
>>socket.gaierror: [Errno -2] Name or service not known
> [snip]
>>Hope someone has some ideas...
>
> That looks like a DNS lookup failure. What does an nslookup of
> nzepg.org show?
>

You are quite right - I generally reference my own system but reverted to
the original epg source for testing after the failures. Unfortunately, the
version of tv_grab_nz-py I have been using was old enough to have the old
'geek' URL, hence the failure. I've updated to 0.2.6 and checked with the
online epg freeview source and that is fine.

Having pointed back at my own epgsnoop generated data it appears that the
problem is extended character set usage from 'the other' channels such
that U omlaut, pound sign, e acute accent all cause the problem below.

2011-07-04 08:46:13.298 XMLTV config file is: /home/mythtv/.mythtv/DVB.xmltv
Traceback (most recent call last):
File "/usr/local/bin/tv_grab_nz-py", line 264, in <module>
doc = ElementTree.parse(StringIO(text)).getroot()
File "<string>", line 56, in parse
File "<string>", line 35, in parse
cElementTree.ParseError: not well-formed (invalid token): line 16602,
column 78
2011-07-04 08:46:13.484 FillData, Error: xmltv returned error code 256
2011-07-04 08:46:13.485 Error in 1:1: unexpected end of file

Interesting that mythfilldatabase in '--file' mode is OK with the data so
I can at least continue to schedule stuff.

So, it appears that:
* epgsnoop is OK generating the data,
* tv_cat (xlmtv utility) is OK with the data,
* tv_grab_nz-py barfs on the data,
* mythfilldatabase is OK with the data

Sure beats me at the moment...

--
Robin Gilks

_______________________________________________
mythtvnz mailing list
mythtvnz [at] lists
http://lists.ourshack.com/mailman/listinfo/mythtvnz
Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/

ross at inertia

Jul 3, 2011, 3:26 PM

Post #4 of 7 (737 views)
Permalink

Re: Python errors in tv_grab_nz-py so mythfilldatabase fails [In reply to]

> Having pointed back at my own epgsnoop generated data it appears that the
> problem is extended character set usage from 'the other' channels such
> that U omlaut, pound sign, e acute accent all cause the problem below.

Which version of tv_grab_nz-py are you running? I seem to remember there
being an issue with this on the earlier versions (or was that epgsnoop,
I can't remember). Latest is 0.2.6 as far as I'm aware.

FWIW my epgsnoop+tv_grab_nz-py setup is working just fine. Using
epgsnoop 0.84 and tv_grab_nz-py 0.2.4 (time I upgraded, heh!) so the
system isn't broken for everyone, at least.

Ross

_______________________________________________
mythtvnz mailing list
mythtvnz [at] lists
http://lists.ourshack.com/mailman/listinfo/mythtvnz
Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/

g8ecj at gilks

Jul 4, 2011, 6:13 PM

Post #5 of 7 (726 views)
Permalink

Re: Python errors in tv_grab_nz-py so mythfilldatabase fails [In reply to]

>> Having pointed back at my own epgsnoop generated data it appears that
>> the
>> problem is extended character set usage from 'the other' channels such
>> that U omlaut, pound sign, e acute accent all cause the problem below.
>
> Which version of tv_grab_nz-py are you running? I seem to remember there
> being an issue with this on the earlier versions (or was that epgsnoop,
> I can't remember). Latest is 0.2.6 as far as I'm aware.
>
> FWIW my epgsnoop+tv_grab_nz-py setup is working just fine. Using
> epgsnoop 0.84 and tv_grab_nz-py 0.2.4 (time I upgraded, heh!) so the
> system isn't broken for everyone, at least.
>
> Ross
>

I'm running the same - tv_grab_nz-py-0.2.6 and epgsnoop-0.84

The problem appears to be an update to something in my python subsystem
which has occurred in the last couple of weeks. It may be an environment
issue that the latest and greatest python-2.7 now requires that has always
been missing - character set or something. It certainly seems to be
accented characters and symbols that it barfs on.

--
Robin Gilks

_______________________________________________
mythtvnz mailing list
mythtvnz [at] lists
http://lists.ourshack.com/mailman/listinfo/mythtvnz
Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/

g8ecj at gilks

Jul 4, 2011, 6:36 PM

Post #6 of 7 (728 views)
Permalink

Re: Python errors in tv_grab_nz-py so mythfilldatabase fails [In reply to]

> I'm running the same - tv_grab_nz-py-0.2.6 and epgsnoop-0.84
>
> The problem appears to be an update to something in my python subsystem
> which has occurred in the last couple of weeks. It may be an environment
> issue that the latest and greatest python-2.7 now requires that has always
> been missing - character set or something. It certainly seems to be
> accented characters and symbols that it barfs on.

Oh what a hack....

I've solved it as follows:

--- tv_grab_nz-py.orig 2011-07-04 07:29:17.000000000 +1200
+++ tv_grab_nz-py 2011-07-05 13:32:53.000000000 +1200
@@ -258,7 +258,8 @@
sys.exit(2)

log.info('Parsing listings...')
-doc = ElementTree.parse(StringIO(text)).getroot()
+parser = ElementTree.XMLParser(encoding="ISO-8859-1")
+doc = ElementTree.parse(StringIO(text), parser=parser).getroot()

newdoc = ElementTree.Element('tv', doc.attrib)

since it appears that the data coming from the 'other channels' is encoded
in ISO-8859-1 and one of the changes to ElementTree in python2.7 is that a
parser can be defined by the user. Looks like the default one doesn't cut
the mustard any more.

I may have the wrong encoding standard but its now working for me :)

--
Robin Gilks

_______________________________________________
mythtvnz mailing list
mythtvnz [at] lists
http://lists.ourshack.com/mailman/listinfo/mythtvnz
Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/

ross at inertia

Aug 8, 2011, 6:30 PM

Post #7 of 7 (587 views)
Permalink

Re: Python errors in tv_grab_nz-py so mythfilldatabase fails [In reply to]

>> The problem appears to be an update to something in my python subsystem
>> which has occurred in the last couple of weeks. It may be an environment
>> issue that the latest and greatest python-2.7 now requires that has always
>> been missing - character set or something. It certainly seems to be
>> accented characters and symbols that it barfs on.
>
> Oh what a hack....

For what it's worth, I have just encountered the same issue.

This weekend I moved epgsnoop to a new machine as the old machine was being retired. The new machine is running Ubuntu 11.04 and defaults to Python 2.7. The listings that the epgsnoop+xmltv-proc-nz combo were outputting had characters in there that tv_grab_nz-py was barfing on.

I fixed the issue, not in tv_grab_nz-py, but by forcing epgsnoop+xmltv-proc-nz to use Python 2.6 (changed shebang line). There are a number of people using my listings so I didn't want to have to update tv_grab_nz-py on all of them. This procedure has worked (although obviously a short term solution) so no modifications to clients were necessary.

If it helps anyone, it would appear that xmltv-proc-nz was outputting this while running under Python 2.7:

Traceback (most recent call last):
File "/usr/local/bin/xmltv-proc-nz", line 572, in <module>
ElementTree.dump(tree)
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1159, in dump
elem.write(sys.stdout)
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 815, in write
serialize(write, self._root, encoding, qnames, namespaces)
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 934, in _serialize_xml
_serialize_xml(write, e, encoding, qnames, None)
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 934, in _serialize_xml
_serialize_xml(write, e, encoding, qnames, None)
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 932, in _serialize_xml
write(_escape_cdata(text, encoding))
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1068, in _escape_cdata
return text.encode(encoding, "xmlcharrefreplace")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 8: ordinal not in range(128)

This does not appear when running under Python 2.6. I did not try only running xmltv-proc-nz under 2.6 (ie without epgsnoop) so maybe that is all that is necessary but I don't really have time to check this scenario out right now.

Ross
_______________________________________________
mythtvnz mailing list
mythtvnz [at] lists
http://lists.ourshack.com/mailman/listinfo/mythtvnz
Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads