Gossamer Forum
Home : Products : Gossamer Links : Version 1.x :

Quick import prolems

Quote Reply
Quick import prolems
Hi,

I have a problem with the quick import. When importing
with | as delimiter editer.cgi gives the following error:

Error importing data: [Mon May 29 12:21:13 2000] editor.cgi: DBSQL (11317): Fatal Error: Unable to execute query: [Mon May 29 12:21:13 2000] editor.cgi: LOAD DATA INFILE 'tester.txt' [Mon May 29 12:21:13 2000] editor.cgi: INTO TABLE Links [Mon May 29 12:21:13 2000] editor.cgi: FIELDS TERMINATED BY '|' [Mon May 29 12:21:13 2000] editor.cgi: . Reason: at /kunder/iguidedk/htdocs/login/editor.cgi line 316 .

In line 316 or just before we have: ($delim eq '\t') and ($delim = "\t");

That means tab right? But tabs doesn't work either.

Can somebody please tell what I should use to delimit
my file. I works without quick import, but then no more
than 170 links can be added at the time before the
process cracks and I importing a lot.

Hope somebody can help me.

Martin

Quote Reply
Re: Quick import prolems In reply to
I'll have to look at the code to see. I had a lot of trouble with imports, but I always used the "|" character without a problem. If the format of your import file is correct, you can use an SQL command directly to import it -- that often works (but it's stupid, so things have to line up just right).

As for the line:

($delim eq '\t') and ($delim = "\t");

This is a bit of perl trickery to get passed using an if-then type loop. What this one says is "If the left hand side is true, look at the right hand side, other wise, ignore the right hand side completely" (In this case this reads: If $delim is equal to the TAB character set $delim equal to the TAB character, otherwise ignore this line.)

So, if ($delim eq '\t') evaluates to "TRUE" then perl will look at ($delim = "\t") and see if it evaluates to true. But, the trick is you put something you want perl to do there -- in this case: set $delim = "\t" . Perl will only do (or see) the right hand side _if_ the left hand one is true -- pretty convoluted, but it has _limited_ usefullness in some situations.

In this case, setting $delim='\t' if $delim already == '\t' seems pretty useless.

So, that line shouldn't give you any problems, because it either evalutes to true, and sets itself back to '\t' or it evaluates to false, and is ignored.

If you are passing in a '|' you should see this assigned somewhere -- taking the $in-> delim (or something) and assiging it to $delim

-------

If you want to try to blow your brains out ... look at what happens if you do this:

($delim eq '\t') or ($delim = "\t");

This reads: If $delim is eqaul to the TAB character evaluates to TRUE go on to the next line, otherwise set $delim equal to the TAB character. Why? The right side of an "or" is only ever evaluated if the left side evaluates to False. If the left side evaluates to TRUE, the whole OR is True, so the right side is never lookd at.

Don't use these constructs unless you have a _really_ good reason to. They make debugging a real B*T*H, and logic problems -- especially nested ones (such as this inside an IF or a loop) become impossible. You almost need to write it down on a logic flow card and keep track of what is true where.

If you put a group of them together to set default values, or test for a value you need to screen out, you are cued as to what is going on. Otherwise, use at your own risk. I spent 3-4 hours trying to debug one of these 2 weeks ago.

As for the original line posted, without looking at the code (which I should do If I could) it seems a useless line, because it doesn't change anything wether it's true or false.



Quote Reply
Re: Quick import prolems In reply to
Hi Pugdog,

if I can avoid it I wont use SQL commands (I don't know any).

The line that delivers the error is
eval { $res = $db->load_data ($file, { quick => 1, delim => $delim } ); };

As far as I can see is if Quick is checked then it has
to do something. I don't know what.

The format of the links is right because I can import
some of them if I don't use Quick import, but as a said
that means I have to run import about 138 times. And I'm
not really up to that. I'm trying to import about 22,000
links and without Quick import it allows me to import about
170.

How can it work with you and not with me. There must be
something I'm doing wrong. Do you alter your editor.cgi.

Martin

Quote Reply
Re: Quick import prolems In reply to
Can you send me a copy of your Links.def and a couple of lines from the top of the file you are trying to import?

I can give you the SQL command to put into the SQL Monitor to import the links from a file. As long as the fields line up ok, you can use the LOAD command to import a file, and bypass some of the problems.

Otherwise, you can import the database to a new temporary table, then insert all the records from that old table into your current Links table, and MySQL will line up the fields for you.

Quote Reply
Re: Quick import prolems In reply to
Hi pugdog,
I have just the same problem, we are trying to import 17000 lines and there is a blank page after the regular import and no more than 9000 lines imported and with quick import I get the same error message from line 316 in editor.cgi.
So if you could solve the problem it would be nice if you could provide some help.
Cheers

Niko

http://www.master-productions.com
Quote Reply
Re: Quick import prolems In reply to
This got lost in the problems I was having connecting. I'll take a look at it again this weekend and see. Still, using an import command is the easiest.

The problem you are probably having with the 9000 links, is your http: session is timing out. It's something you might have to do via telnet.

Do you have a sample of the top of the file file you are trying to import, and a copy of the .def file (for the database fields in the table)? It might actually be easier to write a quick updated import routine. It's been talked about for awhile, and I can see the need. I think some of the functionality for that is built into MySQLMan and phpMyAdmin as well.

But, since importing a delimited file into a database is basic to the mysql program, it should not be hard to make a wrapper for it in perl.

I know I've posted the "load" commands here before, and they do work for databases of any size. I've used them for very large imports...





http://www.postcards.com
FAQ: http://www.postcards.com/FAQ/LinkSQL/

Quote Reply
Re: Quick import prolems In reply to
Thanks pugdog, the mysqladmin does work great. I sent you an eMail with the information though.

Cheers

Niko

http://www.master-productions.com
Quote Reply
Re: Quick import prolems In reply to
Does that mean you got the problem solved?



http://www.postcards.com
FAQ: http://www.postcards.com/FAQ/LinkSQL/

Quote Reply
Re: Quick import prolems In reply to
Not really but it did work out with sqldbman fast and without any problems. I still donīt know why it did not work.

Cheers

Niko


http://www.master-productions.com