Mailing List Archive: Request Tracker: Users

TicketSQL Date parse weird..

Index | Next | Previous | View Threaded

stuart.browne at ausregistry

Jun 5, 2012, 1:06 AM

Post #1 of 8 (592 views)
Permalink

TicketSQL Date parse weird..

Hi,

Using RHEL5, RT 3.8.8, Time::DateParse 2006.0814.

When doing a TicketSQL search similar to the following:

Status = 'open' AND Due <= '10am tomorrow'

The time portion of the string isn't taken into account.

On the command line, a quick test of Time::DateParse shows it works:

]# perl -e 'use Time::ParseDate; print parsedate("2012-06-06 10:00") . "\n"; print parsedate("10am tomorrow") . "\n";'
1338940800
1338940800

So it appears as if RT is parsing the date string of 'Due' and messing with it. Using the time portion before or after, using 10am vs 10:00, none appear to work. Using any time component has the same result (thinking it may be a timezone issue. It wasn't).

I can't see in the RT code where this might be happening (been looking in SQL.pm). Can someone point me in the right direction?

Thanks

Stuart J. Browne
Senior Unix Administrator, Network Administrator, Database Administrator
AusRegistry Pty Ltd
Level 8, 10 Queens Road
Melbourne. Victoria. Australia. 3004.
Ph:� +61 3 9866 3710
Fax: +61 3 9866 1970
Email: stuart.browne [at] ausregistry
Web: www.ausregistry.com.au

The information contained in this communication is intended for the named recipients only. It is subject to copyright and may contain privileged and/or confidential information. If you are not an intended recipient you must not use, copy, distribute or take any action in reliance on it. If you have received this communication in error, please delete all copies from your system and notify us immediately.

falcone at bestpractical

Jun 6, 2012, 8:00 PM

Post #2 of 8 (545 views)
Permalink

Re: TicketSQL Date parse weird.. [In reply to]

On Tue, Jun 05, 2012 at 06:06:09PM +1000, Stuart Browne wrote:
> When doing a TicketSQL search similar to the following:
>
> Status = 'open' AND Due <= '10am tomorrow'
>
> The time portion of the string isn't taken into account.
>
> On the command line, a quick test of Time::DateParse shows it works:
>
> ]# perl -e 'use Time::ParseDate; print parsedate("2012-06-06 10:00") . "\n"; print parsedate("10am tomorrow") . "\n";'
> 1338940800
> 1338940800
>
> So it appears as if RT is parsing the date string of 'Due' and messing with it. Using the time portion before or after, using 10am vs 10:00, none appear to work. Using any time component has the same result (thinking it may be a timezone issue. It wasn't).
>
> I can't see in the RT code where this might be happening (been looking in SQL.pm). Can someone point me in the right direction?

RT actually passes a number of arguments to parsedate:

grep can find where it's called (not in SQL.pm)

$ grep -r Time::ParseDate::parsedate lib/
lib/RT/Articles.pm: my $seconds = Time::ParseDate::parsedate( $args{$date}, FUZZY => 1, PREFER_PAST => 1 );
lib/RT/Date.pm: my $date = Time::ParseDate::parsedate(

-kevin

stuart.browne at ausregistry

Jun 7, 2012, 12:51 AM

Post #3 of 8 (549 views)
Permalink

Re: TicketSQL Date parse weird.. [In reply to]

> -----Original Message-----
> From: rt-users-bounces [at] lists [mailto:rt-users-
> bounces [at] lists] On Behalf Of Kevin Falcone
> Sent: Thursday, 7 June 2012 1:00 PM
> To: rt-users [at] lists
> Subject: Re: [rt-users] TicketSQL Date parse weird..
>
> On Tue, Jun 05, 2012 at 06:06:09PM +1000, Stuart Browne wrote:
> > When doing a TicketSQL search similar to the following:
<snip>
>
> RT actually passes a number of arguments to parsedate:
>
> grep can find where it's called (not in SQL.pm)
>
> $ grep -r Time::ParseDate::parsedate lib/
> lib/RT/Articles.pm: my $seconds = Time::ParseDate::parsedate(
> $args{$date}, FUZZY => 1, PREFER_PAST => 1 );
> lib/RT/Date.pm: my $date = Time::ParseDate::parsedate(
>
> -kevin

Ok, further test (see attached perl routine):

[root [at] psg-app tmp]# ./test.pl.txt
current time() output: 1339054410

fixed-date Time::ParseDate: 1339113600
tomorrow 10am Time::ParseDate: 1339113600

fixed-date RT::Date output: 1339149600
tomorrow 10am RT::Date output: 1339200000
timezone according to RT::Date: 36000

Using either 'user' or 'server' as the timezone I get the same result but I'm probably missing something here for the fixed-date (I can accept that the date might be being modified elsewhere I'm yet to find), but the 'tomrorow 10am' shouldn't be a full day off. So the adding of the timezone offset before Time::ParsEdate::parsedate then removing it again appears to be causing the issue here; pushing today past the end of the day boundary.

I'll try this routine tomorrow morning (before 10am) to see if it shows the right thing, then try to figure out a way to get it to return the same result for the two RT::Date outputs.

Stuart

Attachments:

test.pl.txt (0.78 KB)

ruz at bestpractical

Jun 7, 2012, 3:25 AM

Post #4 of 8 (537 views)
Permalink

Re: TicketSQL Date parse weird.. [In reply to]

On Thu, Jun 7, 2012 at 11:51 AM, Stuart Browne
<stuart.browne [at] ausregistry> wrote:
>> -----Original Message-----
>> From: rt-users-bounces [at] lists [mailto:rt-users-
>> bounces [at] lists] On Behalf Of Kevin Falcone
>> Sent: Thursday, 7 June 2012 1:00 PM
>> To: rt-users [at] lists
>> Subject: Re: [rt-users] TicketSQL Date parse weird..
>>
>> On Tue, Jun 05, 2012 at 06:06:09PM +1000, Stuart Browne wrote:
>> > When doing a TicketSQL search similar to the following:
> <snip>
>>
>> RT actually passes a number of arguments to parsedate:
>>
>> grep can find where it's called (not in SQL.pm)
>>
>> $ grep -r Time::ParseDate::parsedate lib/
>> lib/RT/Articles.pm: my $seconds = Time::ParseDate::parsedate(
>> $args{$date}, FUZZY => 1, PREFER_PAST => 1 );
>> lib/RT/Date.pm: my $date = Time::ParseDate::parsedate(
>>
>> -kevin
>
> Ok, further test (see attached perl routine):
>
> [root [at] psg-app tmp]# ./test.pl.txt
> current time() output: 1339054410
>
> fixed-date Time::ParseDate: 1339113600
> tomorrow 10am Time::ParseDate: 1339113600
>
> fixed-date RT::Date output: 1339149600
> tomorrow 10am RT::Date output: 1339200000
> timezone according to RT::Date: 36000
>
>
> Using either 'user' or 'server' as the timezone I get the same result but I'm probably missing something here for the fixed-date (I can accept that the date might be being modified elsewhere I'm yet to find), but the 'tomrorow 10am' shouldn't be a full day off. So the adding of the timezone offset before Time::ParsEdate::parsedate then removing it again appears to be causing the issue here; pushing today past the end of the day boundary.
>

I can believe it's wrong. Time::ParseDate can do timezones, but
timezones support is very limited, so we do our timezones after
parsing in UTC. Understandably 'tomorrow 10am' in london while you're
somewhere else is completely different thing.

We do some compensation in NOW argument and in theory it should work.
Are you on the latest version of the module?

Never noticed before that ZONE argument can take whatever TZ
environment variable can take. May be attached patch will just work.
Have not tested it. Applies cleanly to 3.8 and 4.0.

> I'll try this routine tomorrow morning (before 10am) to see if it shows the right thing, then try to figure out a way to get it to return the same result for the two RT::Date outputs.
>
> Stuart

--
Best regards, Ruslan.

Attachments:

RT-4.x-use-parsedate-timezones-support.patch (1.16 KB)

stuart.browne at ausregistry

Jun 8, 2012, 12:12 AM

Post #5 of 8 (538 views)
Permalink

Re: TicketSQL Date parse weird.. [In reply to]

> > <snip>
> >
> > Using either 'user' or 'server' as the timezone I get the same result
> but I'm probably missing something here for the fixed-date (I can accept
> that the date might be being modified elsewhere I'm yet to find), but the
> 'tomrorow 10am' shouldn't be a full day off. So the adding of the
> timezone offset before Time::ParsEdate::parsedate then removing it again
> appears to be causing the issue here; pushing today past the end of the
> day boundary.
> >
>
> I can believe it's wrong. Time::ParseDate can do timezones, but
> timezones support is very limited, so we do our timezones after
> parsing in UTC. Understandably 'tomorrow 10am' in london while you're
> somewhere else is completely different thing.
>
> We do some compensation in NOW argument and in theory it should work.
> Are you on the latest version of the module?
>
> Never noticed before that ZONE argument can take whatever TZ
> environment variable can take. May be attached patch will just work.
> Have not tested it. Applies cleanly to 3.8 and 4.0.
>
> > I'll try this routine tomorrow morning (before 10am) to see if it shows
> the right thing, then try to figure out a way to get it to return the same
> result for the two RT::Date outputs.

Ruslan,

I’m using RT 3.8.8 at the moment. did some tests this morning (before the 10-hour's from UTC) and all the times aligned. With your patch, the worded time was off by TZ:

#
# BEFORE 10AM
#

#
# Before patch
#
[bekar [at] psg-app tmp]$ ./test.pl
current time() output: 1339111474

fixed-date Time::ParseDate: 1339200000
tomorrow 10am Time::ParseDate: 1339200000

fixed-date RT::Date output: 1339200000
tomorrow 10am RT::Date output: 1339200000
timezone according to RT::Date: 36000
timezone according to RT::Date: 36000

#
# After patch
#
[bekar [at] psg-app tmp]$ ./test.pl
current time() output: 1339111434

fixed-date Time::ParseDate: 1339200000
tomorrow 10am Time::ParseDate: 1339200000

fixed-date RT::Date output: 1339200000
tomorrow 10am RT::Date output: 1339236000
timezone according to RT::Date: 36000
timezone according to RT::Date: 36000

This was fine all the way until there was TZ seconds until the end of the day (after 14:00 for me). Results from that period:

#
# After 14:00
#

#
# Before Patch
#
[bekar [at] psg-app tmp]$ ./test.pl
current time() output: 1339128542

fixed-date Time::ParseDate: 1339200000
tomorrow 10am Time::ParseDate: 1339200000

fixed-date RT::Date output: 1339200000
tomorrow 10am RT::Date output: 1339286400
timezone according to RT::Date: 36000
timezone according to RT::Date: 36000

#
# After Patch
#
[bekar [at] psg-app tmp]$ ./test.pl
current time() output: 1339128507

fixed-date Time::ParseDate: 1339200000
tomorrow 10am Time::ParseDate: 1339200000

fixed-date RT::Date output: 1339200000
tomorrow 10am RT::Date output: 1339236000
timezone according to RT::Date: 36000
timezone according to RT::Date: 36000

So the un-patched is definitely wrong. The patched is more consistent but sadly still wrong. Perhaps putting the removal of TZ back in to get the expected result more consistently:

#
# Additional line
#
[bekar [at] psg-app tmp]$ ./test.pl
current time() output: 1339139429

fixed-date Time::ParseDate: 1339200000
tomorrow 10am Time::ParseDate: 1339200000

fixed-date RT::Date output: 1339200000
tomorrow 10am RT::Date output: 1339200000
timezone according to RT::Date: 36000
timezone according to RT::Date: 36000

Coming up to a public holiday here so I won't be in a position to do any testing until Tue/Wed next week. I'll try it again before 10-am to see if I get a consistent result.

Stuart

Attachments:

alternate-date.patch (1.06 KB)

ruz at bestpractical

Jun 12, 2012, 7:30 AM

Post #6 of 8 (489 views)
Permalink

Re: TicketSQL Date parse weird.. [In reply to]

Hello Stuart,

I've made a typo in the patch. It should be "ZONE" instead of "GMT".
You can change code by hands after patching, it's close to line 208 in
lib/RT/Date.pm.

Also, make sure ParseDate module is the latest from the CPAN.

On Fri, Jun 8, 2012 at 11:12 AM, Stuart Browne
<stuart.browne [at] ausregistry> wrote:
>> > <snip>
>> >
>> > Using either 'user' or 'server' as the timezone I get the same result
>> but I'm probably missing something here for the fixed-date (I can accept
>> that the date might be being modified elsewhere I'm yet to find), but the
>> 'tomrorow 10am' shouldn't be a full day off. So the adding of the
>> timezone offset before Time::ParsEdate::parsedate then removing it again
>> appears to be causing the issue here; pushing today past the end of the
>> day boundary.
>> >
>>
>> I can believe it's wrong. Time::ParseDate can do timezones, but
>> timezones support is very limited, so we do our timezones after
>> parsing in UTC. Understandably 'tomorrow 10am' in london while you're
>> somewhere else is completely different thing.
>>
>> We do some compensation in NOW argument and in theory it should work.
>> Are you on the latest version of the module?
>>
>> Never noticed before that ZONE argument can take whatever TZ
>> environment variable can take. May be attached patch will just work.
>> Have not tested it. Applies cleanly to 3.8 and 4.0.
>>
>> > I'll try this routine tomorrow morning (before 10am) to see if it shows
>> the right thing, then try to figure out a way to get it to return the same
>> result for the two RT::Date outputs.
>
> Ruslan,
>
> I’m using RT 3.8.8 at the moment. did some tests this morning (before the 10-hour's from UTC) and all the times aligned. With your patch, the worded time was off by TZ:
>
> #
> # BEFORE 10AM
> #
>
> #
> # Before patch
> #
> [bekar [at] psg-app tmp]$ ./test.pl
> current time() output: 1339111474
>
> fixed-date Time::ParseDate: 1339200000
> tomorrow 10am Time::ParseDate: 1339200000
>
> fixed-date RT::Date output: 1339200000
> tomorrow 10am RT::Date output: 1339200000
> timezone according to RT::Date: 36000
> timezone according to RT::Date: 36000
>
> #
> # After patch
> #
> [bekar [at] psg-app tmp]$ ./test.pl
> current time() output: 1339111434
>
> fixed-date Time::ParseDate: 1339200000
> tomorrow 10am Time::ParseDate: 1339200000
>
> fixed-date RT::Date output: 1339200000
> tomorrow 10am RT::Date output: 1339236000
> timezone according to RT::Date: 36000
> timezone according to RT::Date: 36000
>
> This was fine all the way until there was TZ seconds until the end of the day (after 14:00 for me). Results from that period:
>
> #
> # After 14:00
> #
>
> #
> # Before Patch
> #
> [bekar [at] psg-app tmp]$ ./test.pl
> current time() output: 1339128542
>
> fixed-date Time::ParseDate: 1339200000
> tomorrow 10am Time::ParseDate: 1339200000
>
> fixed-date RT::Date output: 1339200000
> tomorrow 10am RT::Date output: 1339286400
> timezone according to RT::Date: 36000
> timezone according to RT::Date: 36000
>
> #
> # After Patch
> #
> [bekar [at] psg-app tmp]$ ./test.pl
> current time() output: 1339128507
>
> fixed-date Time::ParseDate: 1339200000
> tomorrow 10am Time::ParseDate: 1339200000
>
> fixed-date RT::Date output: 1339200000
> tomorrow 10am RT::Date output: 1339236000
> timezone according to RT::Date: 36000
> timezone according to RT::Date: 36000
>
> So the un-patched is definitely wrong. The patched is more consistent but sadly still wrong. Perhaps putting the removal of TZ back in to get the expected result more consistently:
>
> #
> # Additional line
> #
> [bekar [at] psg-app tmp]$ ./test.pl
> current time() output: 1339139429
>
> fixed-date Time::ParseDate: 1339200000
> tomorrow 10am Time::ParseDate: 1339200000
>
> fixed-date RT::Date output: 1339200000
> tomorrow 10am RT::Date output: 1339200000
> timezone according to RT::Date: 36000
> timezone according to RT::Date: 36000
>
> Coming up to a public holiday here so I won't be in a position to do any testing until Tue/Wed next week. I'll try it again before 10-am to see if I get a consistent result.
>
> Stuart

--
Best regards, Ruslan.

stuart.browne at ausregistry

Jun 12, 2012, 4:20 PM

Post #7 of 8 (482 views)
Permalink

Re: TicketSQL Date parse weird.. [In reply to]

> -----Original Message-----
> From: ruslan.zakirov [at] gmail [mailto:ruslan.zakirov [at] gmail]
> Sent: Wednesday, 13 June 2012 12:31 AM
> Subject: Re: [rt-users] TicketSQL Date parse weird..
>
> Hello Stuart,
>
> I've made a typo in the patch. It should be "ZONE" instead of "GMT".
> You can change code by hands after patching, it's close to line 208 in
> lib/RT/Date.pm.
>
> Also, make sure ParseDate module is the latest from the CPAN.
>

<snip>

Ensured I'm running the latest ParseDate - 2011.0517. Yes, I was running an older version. The output didn't appear different before changing GMT to ZONE (for before-10am).

After changing GMT to ZONE, this happened:

#
# BEFORE 10AM
#

#
# Before patch
#
[bekar [at] psg-app tmp]$ ./test.pl
current time() output: 1339542313

fixed-date Time::ParseDate: 1339632000
tomorrow 10am Time::ParseDate: 1339632000

fixed-date RT::Date output: 1339668000
tomorrow 10am RT::Date output: 1339632000
timezone according to RT::Date: 36000
timezone according to RT::Date: 36000

#
# After patch
#
[bekar [at] psg-app tmp]$ ./test.pl
current time() output: 1339542211

fixed-date Time::ParseDate: 1339632000
tomorrow 10am Time::ParseDate: 1339632000

fixed-date RT::Date output: 1339668000
tomorrow 10am RT::Date output: 0
timezone according to RT::Date: 36000
timezone according to RT::Date: 36000

So it doesn't like ZONE there it seems:

- GMT => $self->Timezone($args{'Timezone'}),
+ ZONE => $self->Timezone($args{'Timezone'}),

I'll try again after 14:00 (with GMT and the new ParseDate), although why RT::Date is returning 0 there is just odd.

Stuart

ruz at bestpractical

Jun 15, 2012, 9:51 AM

Post #8 of 8 (441 views)
Permalink

Re: TicketSQL Date parse weird.. [In reply to]

Hi,

I took another look at this. ParseDate can not deal with all timezones
zoneinfo database has. The module just knows
about a few zones and time differences for them, so we will have to
keep our magic.

Wrote a few tests and couldn't find any problem with how we parse
"tomorrow 10am". You can see new tests in
4.2/relative-time-parsing-tests
branch.

I don't see any changes between 3.8 and master that can affect
outcome, so I conclude that your test is wrong or you're misreading
Unix time values. If you still think that I'm missing something then
please change your script to use ->ISO or any other date formatter
instead of comparing Unix values.

On Wed, Jun 13, 2012 at 3:20 AM, Stuart Browne
<stuart.browne [at] ausregistry> wrote:
>> -----Original Message-----
>> From: ruslan.zakirov [at] gmail [mailto:ruslan.zakirov [at] gmail]
>> Sent: Wednesday, 13 June 2012 12:31 AM
>> Subject: Re: [rt-users] TicketSQL Date parse weird..
>>
>> Hello Stuart,
>>
>> I've made a typo in the patch. It should be "ZONE" instead of "GMT".
>> You can change code by hands after patching, it's close to line 208 in
>> lib/RT/Date.pm.
>>
>> Also, make sure ParseDate module is the latest from the CPAN.
>>
>
> <snip>
>
> Ensured I'm running the latest ParseDate - 2011.0517. Yes, I was running an older version. The output didn't appear different before changing GMT to ZONE (for before-10am).
>
> After changing GMT to ZONE, this happened:
>
> #
> # BEFORE 10AM
> #
>
> #
> # Before patch
> #
> [bekar [at] psg-app tmp]$ ./test.pl
> current time() output: 1339542313
>
> fixed-date Time::ParseDate: 1339632000
> tomorrow 10am Time::ParseDate: 1339632000
>
> fixed-date RT::Date output: 1339668000
> tomorrow 10am RT::Date output: 1339632000
> timezone according to RT::Date: 36000
> timezone according to RT::Date: 36000
>
> #
> # After patch
> #
> [bekar [at] psg-app tmp]$ ./test.pl
> current time() output: 1339542211
>
> fixed-date Time::ParseDate: 1339632000
> tomorrow 10am Time::ParseDate: 1339632000
>
> fixed-date RT::Date output: 1339668000
> tomorrow 10am RT::Date output: 0
> timezone according to RT::Date: 36000
> timezone according to RT::Date: 36000
>
> So it doesn't like ZONE there it seems:
>
> - GMT => $self->Timezone($args{'Timezone'}),
> + ZONE => $self->Timezone($args{'Timezone'}),
>
> I'll try again after 14:00 (with GMT and the new ParseDate), although why RT::Date is returning 0 there is just odd.
>
> Stuart

--
Best regards, Ruslan.

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads