Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: MythTV: Dev

GetMythDownloadManager -> download returning stale data?

 

 

MythTV dev RSS feed   Index | Next | Previous | View Threaded


rkulagow at gmail

Sep 5, 2012, 1:15 PM

Post #1 of 6 (437 views)
Permalink
GetMythDownloadManager -> download returning stale data?

I'm using mythdownloadmanager to retrieve files, but it appears that
in some circumstances something is going terribly wrong; subsequent
calls to the download function return a file which has already been
downloaded?

I may just be misunderstanding something though.

Here's the log from the point of view of the server.
204.79.172.4 - - [05/Sep/2012:19:39:34 +0000] "GET
/proc.php?command=get&p1=lineup&p2=IL57303 HTTP/1.1" 200 23112 "-"
"MythTV v0.26.20120822-1 MythDownloadManager"
204.79.172.4 - - [05/Sep/2012:19:39:34 +0000] "GET
/proc.php?command=get&p1=schedule&p2=11848&rand=cff83fc70c106c4e4cfa689e2cd23e84
HTTP/1.1" 200 18015 "-" "MythTV v0.26.20120822-1 MythDownloadManager"
204.79.172.4 - - [05/Sep/2012:19:39:42 +0000] "GET
/proc.php?command=get&p1=schedule&p2=12475&rand=cff83fc70c106c4e4cfa689e2cd23e84
HTTP/1.1" 200 17859 "-" "MythTV v0.26.20120822-1 MythDownloadManager"
204.79.172.4 - - [05/Sep/2012:19:39:46 +0000] "GET
/proc.php?command=get&p1=schedule&p2=14789&rand=cff83fc70c106c4e4cfa689e2cd23e84
HTTP/1.1" 200 1089 "-" "MythTV v0.26.20120822-1 MythDownloadManager"
204.79.172.4 - - [05/Sep/2012:19:39:51 +0000] "GET
/proc.php?command=get&p1=schedule&p2=27549&rand=cff83fc70c106c4e4cfa689e2cd23e84
HTTP/1.1" 200 10840 "-" "MythTV v0.26.20120822-1 MythDownloadManager"
204.79.172.4 - - [05/Sep/2012:19:39:55 +0000] "GET
/proc.php?command=get&p1=schedule&p2=60165&rand=cff83fc70c106c4e4cfa689e2cd23e84
HTTP/1.1" 200 21392 "-" "MythTV v0.26.20120822-1 MythDownloadManager"
204.79.172.4 - - [05/Sep/2012:19:40:06 +0000] "GET
/proc.php?command=get&p1=schedule&p2=74348&rand=cff83fc70c106c4e4cfa689e2cd23e84
HTTP/1.1" 200 1143 "-" "MythTV v0.26.20120822-1 MythDownloadManager"

Here's what I'm seeing in mythfilldatabase:
2012-09-05 14:38:07.996585 I Updated headend. New version: 6 New last
modified: 2012-09-05
2012-09-05 14:38:07.997413 I Headend has been updated. Refreshing
channel table.
2012-09-05 14:38:08.501326 I Downloaded file IL57303.txt
2012-09-05 14:38:08.620840 I 7 unique XMLIDs to download
2012-09-05 14:38:11.457863 I Starting mythlogserver
destfile is "/tmp/11848_sched.txt"
size of downloaded file is 18015
2012-09-05 14:38:16.427312 I Downloaded file /tmp/11848_sched.txt
destfile is "/tmp/12475_sched.txt"
size of downloaded file is 17859
2012-09-05 14:38:20.833061 I Downloaded file /tmp/12475_sched.txt
destfile is "/tmp/14789_sched.txt"
size of downloaded file is 1089
2012-09-05 14:38:25.236396 I Downloaded file /tmp/14789_sched.txt
destfile is "/tmp/27549_sched.txt"
size of downloaded file is 10840
2012-09-05 14:38:29.441170 I Downloaded file /tmp/27549_sched.txt

This is where it starts to break down:

destfile is "/tmp/60165_sched.txt"
size of downloaded file is 10840
2012-09-05 14:38:39.447517 I Downloaded file /tmp/60165_sched.txt

The server logs indicate that the sent file size is 21392 bytes, but
mythdownloadmanager returned 10840 bytes.

2012-09-05 14:38:41.492280 I Starting mythlogserver
destfile is "/tmp/74348_sched.txt"
size of downloaded file is 10840
2012-09-05 14:38:49.454275 I Downloaded file /tmp/74348_sched.txt
destfile is "/tmp/76943_sched.txt"
size of downloaded file is 10840
2012-09-05 14:38:59.460241 I Downloaded file /tmp/76943_sched.txt
2012-09-05 14:38:59.462265 I Loading schedule files into database.

The code is:

QByteArray dl_file;

LOG(VB_GENERAL, LOG_INFO, QString("%1 unique XMLIDs to
download").arg(query.size()));

while (query.next())
// We're going to update all chanid's in the database that use this
// particular XMLID no matter where they are.
{
xmltvid = query.value(0).toString();
urlquery.bindValue(":XMLTVID", xmltvid);

urlquery.exec();
urlquery.next();
url = urlquery.value(0).toString();

url = url + "&rand=" + randhash;

destfile = "/tmp/" + xmltvid + "_sched.txt";

GetMythDownloadManager()->download(url, &dl_file, false);
QFile file(destfile);
qDebug() << "url is " << url;
qDebug() << "destfile is " << destfile;
qDebug() << "size of downloaded file is " << dl_file.size();

if (file.open(QIODevice::WriteOnly))
{
file.write(gUncompress(dl_file));
file.close();
LOG(VB_GENERAL, LOG_INFO, QString("Downloaded file
%1").arg(destfile));
}
else
{
LOG(VB_GENERAL, LOG_ERR, LOC + QString("Could not
create file %1").arg(destfile));
return false;
}
}

From looking at the code for GetMythDownloadManager, I thought that
the download method wasn't queued, and that each invocation was
considered separate. Since the call to gUncompress(dl_file) is
succeeding (even though dl_file seems to have the wrong information in
it), this doesn't appear to be a case of truncated downloads; for some
reason dl_file is returning stale information. I've also tried moving
the define of QByteArray dl_file inside the scope of the loop so that
it gets reset each time, but no help there either; I'll sometimes
download sizes of 0 bytes, even though the server logs showed accurate
bytes sent.

Myth code is synced from master.

Thoughts / assistance?
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


cpinkham at bc2va

Sep 5, 2012, 2:19 PM

Post #2 of 6 (419 views)
Permalink
Re: GetMythDownloadManager -> download returning stale data? [In reply to]

* On Wed Sep 05, 2012 at 03:15:36PM -0500, Robert Kulagowski wrote:
> 2012-09-05 14:38:16.427312 I Downloaded file /tmp/11848_sched.txt
...
> 2012-09-05 14:38:20.833061 I Downloaded file /tmp/12475_sched.txt
...
> 2012-09-05 14:38:25.236396 I Downloaded file /tmp/14789_sched.txt
...
> 2012-09-05 14:38:29.441170 I Downloaded file /tmp/27549_sched.txt

> This is where it starts to break down:

> 2012-09-05 14:38:39.447517 I Downloaded file /tmp/60165_sched.txt
...
> 2012-09-05 14:38:49.454275 I Downloaded file /tmp/74348_sched.txt
...
> 2012-09-05 14:38:59.460241 I Downloaded file /tmp/76943_sched.txt
...

> GetMythDownloadManager()->download(url, &dl_file, false);

You need to check the status of ::download() here. It returns true or
false for success/fail. Your later downloads are hitting the 10-second
timeout in MythDownloadManager::downloadNow(). If we don't receive a
status update from QNetworkAccessManager within 10 seconds of the
last status, the code times out and returns false.

This is why you're seeing the same data if you have the QByteArray
outside the loop but 0 bytes of data if you create the QByteArray
inside the loop.

Also, have you tried just letting the webserver compress and
QNetworkAccessManager/client decompress the data instead of
pre-compressing and having to gUncompress it after you download it?
That might eliminate some of the delay. We could bump up the timeout
if necessary, but don't want it to be too high by default. If you want
to do this for testing, edit libs/libmythbase/mythdownloadmanager.cpp
around line 804, change the 10 to 30 or 60. Because of the way we
abort the download, the actual download is completed in the background
by QNetworkAccessManager, so the server sees a successful download.

--
Chris
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


rkulagow at gmail

Sep 5, 2012, 7:43 PM

Post #3 of 6 (410 views)
Permalink
Re: GetMythDownloadManager -> download returning stale data? [In reply to]

On Wed, Sep 5, 2012 at 4:19 PM, Chris Pinkham <cpinkham [at] bc2va> wrote:
> Also, have you tried just letting the webserver compress and
> QNetworkAccessManager/client decompress the data instead of
> pre-compressing and having to gUncompress it after you download it?
> That might eliminate some of the delay.

The files are already pre-compressed on the server, so there's no
delay there. I'm looking at how the themechooser does things as
possible inspiration. I have no idea if users are on dial-up anymore,
but for any files that are significantly large, 10 seconds may be too
short a time to complete the download process if it's done in blocking
mode.

If I instead fire off multiple queueDownload's, then it doesn't appear
that I'll run into the 10 second timer. Since queueDownload doesn't
block, would it just be easier set up a loop that checks the download
directory for the correct number of files, sleep for 30 seconds if
they're not all downloaded, repeat? Then, if all the files aren't
downloaded within some reasonable time period, exit the loop, process
the files that we did download and just move on...

A quick glance at mythdownloadmanager.cpp doesn't show any sort of
timeout for the queueDownload; is there something that will clean up a
zombie queued download?
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


cpinkham at bc2va

Sep 10, 2012, 8:27 AM

Post #4 of 6 (391 views)
Permalink
Re: GetMythDownloadManager -> download returning stale data? [In reply to]

* On Wed Sep 05, 2012 at 09:43:15PM -0500, Robert Kulagowski wrote:
> The files are already pre-compressed on the server, so there's no
> delay there. I'm looking at how the themechooser does things as

If this is the case, then I wonder why you're seeing a 10-second delay
in downloading some of them.

> possible inspiration. I have no idea if users are on dial-up anymore,
> but for any files that are significantly large, 10 seconds may be too
> short a time to complete the download process if it's done in blocking
> mode.

The 10 seconds is for status updates, not completion. Status updates are
sent out every so often as the download progresses. You can watch these
with "-v file --loglevel debug" debugging turned on. QNetworkAccessManager
send out updates and MythDownloadManager receives them and sends out
MythEvents for clients using the non-blocking mode of MythDownloadManager.

> block, would it just be easier set up a loop that checks the download
> directory for the correct number of files, sleep for 30 seconds if
> they're not all downloaded, repeat? Then, if all the files aren't
> downloaded within some reasonable time period, exit the loop, process
> the files that we did download and just move on...

You could, but that's a bit of a hack. If you use queueDownload(), a
QMap to track queued files, and a customEvent() handler, the you could
either process files as they are downloaded or just delete their entry
from the QMap as they are downloaded so you know if anything was left
or if it was safe to proceed. The ThemeChooser does the former, it
performs actions when the finished message comes in. In the second
method, the queue-er could wait on a QWaitCondition after queueing and
the customEvent handler could wake up the queue-er whenever a file was
downloaded and the QMap was updated. When the queue-er is woken up or
times out, it checks the size of the QMap. If it is empty, then all
downloads were completed. If the QMap has anything still in it, then
the wait must have timed out waiting for a download and you can either
process the files you did download or abort.

> A quick glance at mythdownloadmanager.cpp doesn't show any sort of
> timeout for the queueDownload; is there something that will clean up a
> zombie queued download?

There's no cleanup when running in non-blocking mode, you normally just
watch for status updates and will get back the updates for current position
and a finished message when the download completes or fails.

--
Chris
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


rkulagow at gmail

Sep 10, 2012, 10:45 AM

Post #5 of 6 (389 views)
Permalink
Re: GetMythDownloadManager -> download returning stale data? [In reply to]

On Mon, Sep 10, 2012 at 10:27 AM, Chris Pinkham <cpinkham [at] bc2va> wrote:
> * On Wed Sep 05, 2012 at 09:43:15PM -0500, Robert Kulagowski wrote:
>> The files are already pre-compressed on the server, so there's no
>> delay there. I'm looking at how the themechooser does things as
>
> If this is the case, then I wonder why you're seeing a 10-second delay
> in downloading some of them.

Turns out it was on the server side, which I fixed. The downloads are
now speeding along quite nicely.

>> block, would it just be easier set up a loop that checks the download
>> directory for the correct number of files, sleep for 30 seconds if
>> they're not all downloaded, repeat? Then, if all the files aren't
>> downloaded within some reasonable time period, exit the loop, process
>> the files that we did download and just move on...
>
> You could, but that's a bit of a hack. If you use queueDownload(), a
> QMap to track queued files, and a customEvent() handler, the you could
> either process files as they are downloaded or just delete their entry
> from the QMap as they are downloaded so you know if anything was left
> or if it was safe to proceed. The ThemeChooser does the former, it
> performs actions when the finished message comes in. In the second
> method, the queue-er could wait on a QWaitCondition after queueing and
> the customEvent handler could wake up the queue-er whenever a file was
> downloaded and the QMap was updated. When the queue-er is woken up or
> times out, it checks the size of the QMap. If it is empty, then all
> downloads were completed. If the QMap has anything still in it, then
> the wait must have timed out waiting for a download and you can either
> process the files you did download or abort.

I've switched to the queueDownload method to get the files rather than
blocking (which was what actually allowed me to find the issue on the
server side)

Right now I've implemented the "hacky" way, because I'm still learning...

QDir dir;
dir = QDir(tempDLDirectory);
QStringList files;
QString searchfor = "*_sched.txt.gz";

int LoopCount = 0;

while (LoopCount < 60) // Determine the best loop count /
sleep combination to download files.
{
qDebug() << "Sleeping";
sleep(5);
LoopCount++;

files = dir.entryList(QStringList(searchfor), QDir::Files
| QDir::NoSymLinks);

if (files.size() == total_XMLID)
{
qDebug() << "Done sleeping";
LoopCount = 99; // Magic value to let routine down the
line know that we got all the files?
break;
}
qDebug() << "Downloaded " << files.size() << "of " << total_XMLID;
}

Work in progress...
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev


cpinkham at bc2va

Sep 10, 2012, 2:38 PM

Post #6 of 6 (387 views)
Permalink
Re: GetMythDownloadManager -> download returning stale data? [In reply to]

* On Mon Sep 10, 2012 at 12:45:48PM -0500, Robert Kulagowski wrote:
> > If this is the case, then I wonder why you're seeing a 10-second delay
> > in downloading some of them.
>
> Turns out it was on the server side, which I fixed. The downloads are
> now speeding along quite nicely.

Great. Sorry for the delay in responding, I started that reply Thursday night,
got interrupted, and never got around to sending it before I went out of town
Friday morning.

--
Chris
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://www.mythtv.org/mailman/listinfo/mythtv-dev

MythTV dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.