Gossamer Forum
Home : General : Internet Technologies :

fetch and save a text file

Quote Reply
fetch and save a text file
Hi,

I am looking for a script that would:

run once a day using cron
fetch a text file
save the text file on my server
append the date to the filename so multiple days may be saved in the same directory without overwritting previous files

I currently have two files that I would like to save, but don't want to be at the computer to manually save them. The files are:

http://water.gp.usbr.gov/...ydromet_archives?CCR
http://water.gp.usbr.gov/...ydromet_dayfiles?ccr

I need to fetch and save the first file between 6:00-7:15 AM and the second file between 11:30-11:59 PM on a daily basis. It would be OK if there were variables in the script that could set the URL and filename and run under cron as something like fetch1 and fetch2.

This is not a fetch and display content operation, so the files don't need to be publicly viewable. Likewise, it would be OK if they were and it would make reviewing them with a browser simple.

I have looked around hotscripts.com and haven't found what I was looking for. Might be searching for the wrong term. The scripts I found so far allow users to upload files to my site or take content from another site and display it on my site.

Any thoughts or suggestions would be welcome.
--
Rob

SW Montana's Online Community
Modular Model Railroading
Quote Reply
Re: [BeaverheadRiver] fetch and save a text file In reply to
Something like this should work;

Code:
#!/usr/bin/perl

use CGI::Carp qw(fatalsToBrowser);
print "Content-type: text/html \n\n";

$file = "http://www.site.com/file.txt";

# change directory, and grab the file in question....
chdir("/path/to/folder/") or die "Cant chdir to new folder. Reason: $!";
get($file) or die "Cant grab file. Reason: $!";

# rename the file
rename("file.txt","newfilename.txt") or die "Cant rename file. Reason: $!";

# print our message.
print "Grabbed $file";

I don't have my cron script to hand, so unfortunatly cannot provide you with a script to setup the cron with Frown

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] fetch and save a text file In reply to
Sooo, where is get() coming from? Tongue
Quote Reply
Re: [Andy] fetch and save a text file In reply to
Thanks Andy! I will give it a try.

If I wanted to rename the file to a unique, time related name, would something like this work?

$seconds = time;

# rename the file
rename("file.txt","newname$seconds.txt") or die "Cant rename file. Reason: $!";
--
Rob

SW Montana's Online Community
Modular Model Railroading
Quote Reply
Re: [BeaverheadRiver] fetch and save a text file In reply to
As Paul mentioned, because I use get(), then you need to have this at the top of the script too;

Code:
use LWP::Simple;

Regarding the name/date thing. I would use something like;

Code:
$seconds = localtime;
$seconds =~ s/ //g;
$seconds =~ s/://g;

# rename the file
rename("file.txt","newname$seconds.txt") or die "Cant rename file. Reason: $!";


Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] fetch and save a text file In reply to
Code:
my $seconds = (localtime)[0];

The above will work just as good.

- wil
Quote Reply
Re: [Wil] fetch and save a text file In reply to
No it won't, it will give a syntax error =)

Code:
my $seconds = ((localtime)[0]);

The above will work better.
Quote Reply
Re: [Paul] fetch and save a text file In reply to
No it won't. My example is valid and works fine.

- wil
Quote Reply
Re: [Paul] fetch and save a text file In reply to
Code:
#!/usr/bin/perl

my $seconds = (localtime)[0];

print "$seconds\n";

Code:
soy% perl seconds.pl
50
soy%

- wil
Quote Reply
Re: [Wil] fetch and save a text file In reply to
Yep, I forgot to declare the variable in my test and just printed it, which requires the extra parenthesis.

Last edited by:

Paul: Jun 2, 2003, 3:53 AM
Quote Reply
Re: [Andy] fetch and save a text file In reply to
It's still not right Andy, you just call get() without actually doing anything with the returned value.
Quote Reply
Re: [Paul] fetch and save a text file In reply to
Mmm. Good point Frown

This works though (just tested it);

Code:
#!/usr/bin/perl

use CGI::Carp qw(fatalsToBrowser);
use LWP::Simple;

print "Content-type: text/html \n\n";

$file = "http://water.gp.usbr.gov/htbin/hydromet_dayfiles?ccr";

# change directory, and grab the file in question....
chdir("/path/to/folder/save/in/") or die "Cant chdir to new folder. Reason: $!";
$got = get($file) or die "Cant grab file. Reason: $!";

$file =~ m/^http:\/\/.+\/(.+?)$/ and $save_as = $1;

open(FILE,">>$save_as") || die "Cant write $save_as file. Reason: $!";
print FILE $got;
close(FILE);

# rename the file
rename($save_as,"newfilename.txt") or die "Cant rename file. Reason: $!";

# print our message.
print "Grabbed $file";

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Wil] fetch and save a text file In reply to
Quote:
my $seconds = (localtime)[0]

Not really the same as Andy was generating a string like: MonJun21048352003, whereas you are justing getting the seconds which isn't going to be unique (the variable naming probably threw you off).

Cheers,

Alex
--
Gossamer Threads Inc.
Quote Reply
Re: [Andy] fetch and save a text file In reply to
Quote:
$file =~ m/^http:\/\/.+\/(.+?)$/ and $save_as = $1

Argh =)

Try:

Code:
my ($save_as) = $file =~ m|([^/]+)$|;
Quote Reply
Re: [Paul] fetch and save a text file In reply to
In Reply To:
Argh =)

You know as well as me, that my regexes are no where near perfect Tongue

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] fetch and save a text file In reply to
Everyone,

Thanks for the help. Installed the scripts and it seems to be working fine as a cron job.

I was wondering what should the permissions be set for the directory where I am saving the files? I have it set to 744 right now.
--
Rob

SW Montana's Online Community
Modular Model Railroading
Quote Reply
Re: [BeaverheadRiver] fetch and save a text file In reply to
If 744 works ok, then leave it at that. I wouldn't recommend going any lower than that...

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!