Gossamer Forum
Home : General : Perl Programming :

Changing From Flat File to MYSQL

(Page 1 of 2)
> >
Quote Reply
Changing From Flat File to MYSQL
I am writing a little script to change my flat file database of members to mysql. The problem I am having is because each member has it's own text file (All in one folder), I can't get it to read all the files and transfer all of them, I can get it to transfer individual files though. I am trying to use grep to get all the file names. And that works if I print it on the page but not when I try to open them... I have for the code:

Code:

#!/usr/bin/perl

use CGI qw(:standard);
print "Content-type: text/html\n\n";
require "shopconfig.cgi";

use DBI();


# Connect to the database.
my $dbh = DBI->connect("DBI:mysql:database=crashint_test;host=localhost",
"crashint", "PASSWORD",
{'RaiseError' => 1});



my @count = ();
opendir (DR,"$datapath/members/");
@count = grep { /\..*/ } readdir(DR);
closedir (DR);

foreach $line (sort @count) {

print "$line<br>";



open(LIST, "<$datapath/members/$line");
while(<LIST>) {
my ($Username, $Password, $lname, $fname, $country, $email, $add1, $add2, $city, $state, $zip, $phone) = split/\|/;



$dbh->do("INSERT INTO foo VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", undef, "$Username", "$Password", "$fname", "$lname", "$country", "$email", "$add1", "$add2", "$city", "$state", "$zip", "$phone");


}

close(LIST);

}

# Disconnect from the database.
$dbh->disconnect();

Thanks in advance..
-------------
Jeremy
http://lc.crashinto.com - Crashinto Learning Central
Quote Reply
Re: [nolimit] Changing From Flat File to MYSQL In reply to
Are you getting a 500 IS Error when you add in the 'while' to add the info to the database? also, not sure if it matters, but shouldnt the syntax for the split be;

my ($Username, $Password, $lname, $fname, $country, $email, $add1, $add2, $city, $state, $zip, $phone) = split(/\|/, $_);

??? May just be another way of doing it, but thats how I prefer to do it Tongue



Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [AndyNewby] Changing From Flat File to MYSQL In reply to
I guess you like doing things the long way andy :) ... split /\|/; ...is perfectly valid.

Jeremy, you can change: database=crashint_test;host=localhost
....to just crashint_test:localhost

Also grep is only looking for files beginning with . - is that correct?

Last edited by:

PaulW: Nov 18, 2001, 7:12 AM
Quote Reply
Re: [PaulW] Changing From Flat File to MYSQL In reply to
Yes only files with . but the only files in the directory are .txt.. When I printed $line it showed the list of files correctly.

Would $datapath/members/$line be right? Or should I need something else then grep?
-------------
Jeremy
http://lc.crashinto.com - Crashinto Learning Central
Quote Reply
Re: [nolimit] Changing From Flat File to MYSQL In reply to
Quote:
Yes only files with . but the only files in the directory are .txt.. When I printed $line it showed the list of files correctly.
When you say "files with . ", do you mean files without a suffix like

filename.

or files with the dot as a prefex, like:

.filename.txt


--
Rob

SW Montana's Online Community
Modular Model Railroading
Quote Reply
Re: [vanderen] Changing From Flat File to MYSQL In reply to
$line prints out:

Username1.txt
Username2.txt
Username3.txt
.....

It's just not opening each one.
-------------
Jeremy
http://lc.crashinto.com - Crashinto Learning Central
Quote Reply
Re: [nolimit] Changing From Flat File to MYSQL In reply to
I think you mean it to be:

@count = grep { !/^\.+/ } readdir(DR);

Last edited by:

PaulW: Nov 18, 2001, 9:24 AM
Quote Reply
Re: [PaulW] Changing From Flat File to MYSQL In reply to
That didn't do anything that time.. It only listed one file... And it still didn't get the data from it.
-------------
Jeremy
http://lc.crashinto.com - Crashinto Learning Central
Quote Reply
Re: [nolimit] Changing From Flat File to MYSQL In reply to
You still should use it though because the way you had it would also match:

.
..
.htaccess
.htpasswd
etc...

Also you may be better using $sth = $dbh->prepare( ... ); and $sth->execute

You should use $DBI::errstr to find the possible problem as it could be a syntax error and you'd never know.

Last edited by:

PaulW: Nov 18, 2001, 9:56 AM
Quote Reply
Re: [PaulW] Changing From Flat File to MYSQL In reply to
The problem is open(LIST, "<$datapath/members/$line"); Just doing one file like

open(LIST, "<$datapath/members/username.txt");

Will work fine but when I put $line it does not
-------------
Jeremy
http://lc.crashinto.com - Crashinto Learning Central
Quote Reply
Re: [nolimit] Changing From Flat File to MYSQL In reply to
How do you know?

For all you know it could something in one of the files causing a syntax error with mysql
Quote Reply
Re: [PaulW] Changing From Flat File to MYSQL In reply to
Because if I change $line with one of the filenames it works
-------------
Jeremy
http://lc.crashinto.com - Crashinto Learning Central
Quote Reply
Re: [nolimit] Changing From Flat File to MYSQL In reply to
Yes but it's inside a loop - just because you replace the file name and it works doesn't mean anything.

It could be another file causing the problem.
Quote Reply
Re: [nolimit] Changing From Flat File to MYSQL In reply to
Why are you using a while block? Wouldn't it be better if you just split an array into your given fields? You don't need that while statement.

Having two variables on the line

> open(LIST, "<$datapath/members/$line");

is a bad idea. It would be better for you to define the data path first, and then just use the one variable $line to read individual files. Maybe something like the following would be a better bet.

open(LIST,"</path/to/files/members/$line");

Note: I wouldn't use the filehandle LIST just incase it conflicted with a reserved perl word.

> foreach $line (sort @count) {

Why are you sorting there? Wouldn't it be faster if you just left that out? You don't really need to sort there, do you?

Wil

- wil

Last edited by:

Wil: Nov 18, 2001, 1:24 PM
Quote Reply
Re: [Wil] Changing From Flat File to MYSQL In reply to
1. 2 scalars on one line is irrelevant
2. while() or for() is needed
3. LIST is fine too.
Quote Reply
Re: [PaulW] Changing From Flat File to MYSQL In reply to
No. I reckon he's having problems with those two varialbes in his file path. I wonder what would happen if he tried cutting that down to just one variable at the end of the line.

- wil
Quote Reply
Re: [Wil] Changing From Flat File to MYSQL In reply to
thanks but I tried one variable but that still hasn't worked...
-------------
Jeremy
http://lc.crashinto.com - Crashinto Learning Central
Quote Reply
Re: [nolimit] Changing From Flat File to MYSQL In reply to
Please just try using $DBI::errstr
Quote Reply
Re: [PaulW] Changing From Flat File to MYSQL In reply to
Just used it and no errors came up.
-------------
Jeremy
http://lc.crashinto.com - Crashinto Learning Central
Quote Reply
Re: [nolimit] Changing From Flat File to MYSQL In reply to
File permissions on your directory or files?

- wil
Quote Reply
Re: [nolimit] Changing From Flat File to MYSQL In reply to
The following is untested code, but I think it should work better and more efficently for you. Let me know how it goes.

Code:
#!/usr/bin/perl -w

$datapath = '/your/data/path';

use CGI;
$query = CGI::new();

print $query->header();

use DBI;

$dbh = DBI->connect('dbi:mysql:crashint_test:localhost','crashint','PASSWORD');

opendir (DIR,"$datapath/members"); # Note NO trailing slash.
@files = grep (!/^\.\.?$/, readdir (DIR));
closedir (DIR);

foreach $file (@files) {

print "$file <BR>";

open (DATA,"$datapath/members/$file");
@filedata = <DATA>;
close (DATA);

$filefields = join('',@filedata);

@filefields = split(/\|/,$filefields);

$Username = $filefields[0];
$Password = $filefields[1];
$lname = $filefields[2];
$fname = $filefields[3];
$country = $filefields[4];

...

$sth = $dbh->prepare("INSERT INTO foo VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", undef, "$Username", "$Password", "$fname", "$lname", "$country", "$email", "$add1", "$add2", "$city", "$state", "$zip", "$phone");
$sth->execute;
$sth->finish;

}

$dbh->disconnect;

- wil

Last edited by:

Wil: Nov 19, 2001, 1:35 AM
Quote Reply
Re: [Wil] Changing From Flat File to MYSQL In reply to
What on earth are you using join for Tongue

That is a very slow method.

Last edited by:

PaulW: Nov 19, 2001, 4:36 AM
Quote Reply
Re: [PaulW] Changing From Flat File to MYSQL In reply to
oops. No need for that join in there. Nothing wrong with that regex, it gets all the filenames bar . and .. in directory listings, and stores each filename in a variable so you can play with them later.

- wil
Quote Reply
Re: [Wil] Changing From Flat File to MYSQL In reply to
It is very slow. Jeremy's original code was better.

All he needs to do is add some error checking otherwise he'll be searching forever.
Quote Reply
Re: [PaulW] Changing From Flat File to MYSQL In reply to
Well, I was just offering an alternative. I think my code is cleaner and it's easier to spot where you're going wrong. You shouldn't go too wrong with the code I provided.

- wil
> >