Hello !

As the message says, I have horror since many months with this script. Very very sorry to say all this and come here again and again.

There is no explaination of how to import categories or links. No help. ABSOLUTELY RIDICULOUS!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!1

HOW THE HELL DO I IMPORT LINKS AND CATEGORIES???????? [editor: took out excessive question marks so it formats properly]
------------------
rajani













[This message has been edited by Alex (edited October 14, 1999).]

Hello!

In this problematic situation I found out that the path " require"category.def" seems to be absolutely unromantic to the Perl darling. Alex do change this or pl. give a solution.

I have no possibility to import anything from my database. I cannot work. Nothing is working. Thats not nice.

------------------
rajani

I have been further banging my head against the wall because of the following :


eval { require "links.def"; }; $@ and die "import: can't open 'categories.def' ($!)";

So in the telnet I read and hunted for something that was in a wrong direction.

Pay attention to also the following :

links.def + categories.def >>>
Links.def + Category.def

------------------
rajani

Further, It always gives "Cannot be left blank" error for Name and Has_Changed" fields.

------------------
rajani

The Import.pl script is for import data from Links 2.0 only! It looks for links.def and category.def because those are files used in Links 2.0! You do not want to rename it to capitals.

If you want to import data from other programs, you should use the import from the editor.

Cheers,

Alex

Why not import the data into a generic MySQL table, using MySQL's import features. ALTER that table to the format you want for LinkSQL (rename fields, change types or formats, etc) and export it. Then import or copy it to the Links table.

Hello Alex!

You are giving me more and more misunderstanding. You know that I am not a Links v2.0 user. I have my database in Access, and this I always have said.

So I repeat the description of the problem again.

I have my database in Access.

I have exported in txt format with | delimiter.

The Category is a real name.

The date is exported as >>

15.Aug.99 Short date if I convert it to a text format.

otherwise
15-Aug-99 SS:MM:HH Medium date will be the format.

I asked you earlier. You told me not to use the editor but to use Import.pl as the CategoryIDs + Date needs to be processed. I had lots of problems before. Now also I had too many problems. Now you are telling me not to use Import.pl but use Editor.cgi.


[This message has been edited by rajani (edited October 15, 1999).]

Hello Alex!

So you answered the other less important question but not this one.

Very upseting

------------------
rajani

Rajani,

The date routine I gave you before can be made to work with this.

Right now the import.pl assumes a links 2.0 format for date. If you change the format for the input date, change the routine to match. Save it as AccessImport.pl

I've been able to do that successfully with other databases by writing out my own .def file for the database I've created. I then change it to the links2.0 format and put it in the setup database.

It's a little extra work, yes, but it sure beats importing 16,000 records by hand.

What I don't understand is that there are so many different work-arounds, that by now I would have thought you'd have hit one just by pure chance!

One of the helpful things is in the .def files _REMOVE_ all 'REQUIRED' and 'VALIDATION' flags from the records during an import -- they are more trouble than they are worth. I'd rather have a broken record, than no records at all-- and -- FWIW -- I've successfully imported data with no errors even when links choked, spit and refused to enter the data because the 'required' or 'validation' fields were set. I don't know why, and I manually validated and checked the records, and they import individually, but not as a group. Turning off those checks during this kind of import helps.

Also, if you have a large database, breaking it into chunks of 50, 100 or 150 records can help too. Keep the data size of each chunk relatively small.

Remember, this kind of import only happens ONCE. So the goal is to just get it done. Once the data is in the SQL database, it can be exported/imported with no problems.

One thing I do know, is that if this was so important to you, if you had paid a consultant to do it for you, it would have been done and over with weeks or months ago. To paraphrase Gov. Juliani, if I could do it, anyone could it <G>

Hello Alex!


www.gossamer-threads.com/scripts/forum/resources/Forum9/HTML/000093.html

There was therefore no reason to use Import.pl if I did not have Links data. So this was a wrong direction and further a wrong advice which led the confusion. It would be nice to write in the editor.cgi "one line saying that - Import.pl is for Links v2.0 users! To import your data from the Links v2.0 format, click here <a href=...">"

The concept of import.pl is clearly to get the field names in an array and get into the respective table.

I have not really understood the code functions of the sub_routines of the &import_data and &export_data. I would like to know. Will it get all the categories and links in the correct tables + Index + Alternates + Hierarchy + the necessary things which is done after Validation table routine?

Can I simply export Validate table, compress it, work offline, zip it get it on the server and dump into the database via import data? Will then all be done via &import_data routine the same as Validate routine?

------------------
rajani

Hello Mr. Pataki!

Date sent: Tue, 15 Jun 1999 16:49:42 -0700
From: Alex Krohn <alex@gossamer-threads.com>
To: gossamer@rajani.com
Subject: Re: Links SQL

rajani wrote :
> > My anathor question, I have also asked in the forum, will you also give a
> > text converter to the new database format? I have some links in Access
> > collect and may be I can also use them as my first batch of
> > submissions.'
>
Alex wrote :
> Yes, the installation includes inserting any existing links. Also if
> they are already in Access, importing them into Links SQL will be very
> easy.
>
rajani wrote :
> > I really like your programming a lot, although Links v2.0 I thought
> > needed to be developed by you much more, in terms of basic logic of
> > programming.
> >
> > I beleive that your new creation will be much better.[/quote]
I was somehow given a feeling that Alex would help me in importing my database or there is an import script that would work also with other database exports other than links. That should, actually. He has said many many times, from Index finger we will give you an import utility etc. What about me? All this is said good but where is the solution?

For each and every problem I have to write so many times here. This is nonsense. Pleople get frustrated and upset and there are reasons for that. There is a provocation due to a situation. Why do I need to come here and say that i have a problem?

If multiple editors scripts comes as an upgrade does not interest me now but a good import/export utility. This should have been there and this is basic.

No, daily. I cannot afford to pay German telephone bills and sit down in front of the computer and watch at the nice long pages being created for every link in the admin area. All this was fine for Links v2.0 but not for "Large directories". The manner of Links Validation going through each and every Validation/Modification, I cannot wait to see and go through all those tables created.

The entire admin webpages creation is for me so slow. The admin I can imagine to use it only for basic functions like build or editor from it. Rest I would do export and import and then >>> OFFLINE.

Mind you, i do not enjoy 640K DSL like you do.

Getting 300 links submission per day, I would have to get 30 pages via admin Validate function and connect to the telephone line 30 times if each time it would create 10 Links to be validated each time. OR if it makes all at once, then I will have to download a Kilometer long webpage production from Validation each having a Kilometer long popup listing produced twice (With 15,000 categories). Further the Alternate categories field would be in square kilometers for every link!!! I havnt had the luck to see more then five links in my database till today, since the last three months I was not able to import any link from my database.

The biggest problem is Category Names <> Category IDs. Names are for people to understand and IDs are for MySQL. So as a designer of the internet script, not forgetting how internet work with numbers and names, one would have had the import export in such a way that if the category goes inside the database it would have got a CatID and when it comes out then it would get the real names back. So we can work with it and handle them. Or have options.

The one of the most problematic area is imports and exports. This is a daily necessicity for a webmaster.


------------------
rajani













[This message has been edited by rajani (edited October 15, 1999).]

Hello Alex!

I have done the following :

In editor.cgi,
- I added all the "use=...", @ISA/EXPORTER from DB_Utils.pm.
- I also added my (%...) from line 27-41

I copied the entire sub &get_category_id and pasted in there.

I copied the entire sub of import_data in editor.cgi pasted below it with a new name called

sub import_links.

Then I tried to add the following code (Bold marked) in the sub import_links before the line

$record{'CategoryID'} = $record{'Category'} = &get_category_id ($record{'Category'};
if (!$db->add_record (\%record)) {

What I am trying to do is get my links from my access database to Links table. The categories are names and not IDs. Therefore I need to change them by &get_category_id <> &get_category_name in the import and export routines so that I can handle them. Also after the imßort is done, I would want to see what IS NOT imported so that I can create the related categories or change them.

I also need to convert the date from Long format to date_to_unix format and that should be also inserted in the import routine somewhere.

This does not work. Is there a way? Import.pl does not work.

I have now simply imported the Main categories with editor.cgi. But when I import categories.txt which also has the subcategories then they do not get into the Subcategories table where it registers the Hierarchy & weight.

My problem is that currently Links SQL database works only in a way within MYSQL. The export - import exchange is not there which I can use it and work offline.

[This message has been edited by rajani (edited October 17, 1999).]

Hello!

I forgot to mention that CASE is defined as = &import_links.

Here in editor.cgi it does not select $database. How can I have this variable shown?

------------------
rajani

Hello Alex!

Read your reply with pleasure but ..........

Execution of /usr/www/users/rajani/.../cgi-bin/../admin/editor.cgi aborted due to compilation errors.

So it does not execute at all. The moment I put a # before all those new lines, it gets back into its best action as designed.

Further I tried with the $record'{....}' = $record'{....}' or even $record'{....}' = '{....}' but it simply does not wants to have record variable in there!

This code what you mentioned was added to a clean and original distribution copy of editor.cgi in the line you mentioned.

What I need is the following :

1 - Category Names >>>> CategoryID Where

The Name is the name of a Category/Subcategory in Category table.

2 - DateTime (Hatched by the shity and optional less access export format) changed to Date format for Add_date/Mod_date colums.

3 - After insertion of rows into the database, the rows should be deleted. This helps in finding out what went in and what is remaining in the file import.txt. This may be due to no categories in the table or anything missing or whatever.

4 - Export data with Category names and not the CategoryIDs. This two way system helps in offline work as explained above.



------------------
rajani

In editor.cgi, right after line 330:

%record = map { $header[$i] => $data[$i++] } @data;

add:


Then, when importing, use CategoryID and it will fix it up for you. Let me know if this works for you..

Cheers,

Alex


[This message has been edited by Alex (edited October 18, 1999).]

do you have more detailed error logs? even when links sql says compilation error i am allowed to access my error log and find out the exact reason(s) why.. or you can go into telnet and type "perl editor.cgi"

the usual problem is that you didn't use MY $variable.. cause Strict makes you.. (it's annoying..) some guy that can script really good can start crying the first time he uses strict.. i kept wondering why i could never make a program the first time i ever encountered strict.. but it's all good now

jerry

Hello!

What I thought of doing is as follows:

After the table is exported into a file name import-export/$table-export.txt, it opens as a FILE makes
if ($file) {
open (FILE, "<$file") or return "Unable to open file: '$file'. Reason: $!"; # -->
$header = <FILE>;
@lines = <FILE>;
close FILE;
}

and then substitute afterwards all the Category IDs to &get_cat_name in the same way how it did substituted during the imports.
i.e.

%record = map { $header[$i] => $data[$i++] } @data;

......!

Hello Alex!
Can wwe get the categoryID from MySQL instead of Perl? for e.g.

if ($table eq 'Links') {
foreach $record{'CategoryID'} !~ = /^\d*$/) {
$record{'CategoryID'} = qw!
SELECT * FROM ID.Category, Name.Category
WHERE Name.Category = '($record{'CategoryID'});'
return $ID.Category
!}
}

I do not know the syntax error and I have never tried the above.


[This message has been edited by rajani (edited October 19, 1999).]

Sorry Rajani,

That should be:


plus you need to change line 309 from:


to:


However, I'm not sure if that solves all your problems with importing/exporting. Is it possible to modify your Access tables to use the same format as Links SQL? i.e. have two tables, one for Category and one for Links?

We could get your data into the program once, and then you just use the export to export the Category and Links table which you manage in Access, and then import normally.

Cheers,

Alex


[This message has been edited by Alex (edited October 19, 1999).]

Hello Alex!

Thanks for your help!

This helped me in getting links in my database! However only some and that for testing. This routine converts the Category Names into the IDs.

What I need is that :

1-It should delete the lines after it has imported.

2-Export with the Categories names back instead of Ids. This is to work OFFline and develop databank in Access for many reasons.

3-DateTime in the Lines needs to be changed to Date format for Links SQL format of Add_Date/Modify date.

The problem is that I have no Category Database but only Links database in Access format. Thats no big deal to organise Links table & Access table to have same configurations in terms of cloumn names and sizes. The only problem is the Date/Datetime that is not possible to solve.

I have 30,000 Links in Access now with the latest parsing of emails now. So all I need to de is get my links data into the MySQL and prepare for Links SQL to communicate and work with.

Therefore after it imports I need to know what remains in the file so that I can constantly create new related categories. Evenotherwise, if the connection is briken, as it may happen in a dial-up, then I need to know what has been imported and what is still remaining so that there is NO DOUBLE import. It simply should delete the line after imports.

If export with cat_name, datetime format, and erase of lines after import is solved then I will be ready to launched the site with the first batch of 30,000 links in a week.

------------------
rajani

Hello widgetz!

Thats really good suggestion from you, did not think of that.

Well now it tells me more.

Before I add the codes suggested by Alex it clearly wants to go to sleep. The error log is as follows:

rajani@x% perl5 editor.cgi
(offline mode: enter name=value pairs on standard input)
^C

After I add the codes from Alex it shows the following:

rajani@x% perl5 editor.cgi
[Tue Oct 19 03:41:06 1999] editor.cgi: Global symbol "table" requires explicit p
ackage name at editor.cgi line 454.
syntax error at editor.cgi line 455, near "!~ ="
syntax error at editor.cgi line 458, near "}"
Content-type: text/html

<H1>Software error:</H1>
<CODE>Execution of editor.cgi aborted due to compilation errors.
</CODE>
<P>
For help, please send mail to this site's webmaster, giving this error message
and the time and date of the error.
;
[Tue Oct 19 03:41:06 1999] editor.cgi: Execution of editor.cgi aborted due to co
mpilation errors.



Both these line are ofcourse not the same as the original distribution but as the following:

: 454 : $error .= "<li>" . ($line_num+2) . ": Failed validation. Error: <ul>$Links: BSQL::error</ul>\n";

: 455
$err_cnt++;
& 458 : as
$good_cnt++;

I still have the ISA EXPORTER lines defined that i copied from DBSQL.pm for the sub &get_cat_id routine. i also have the sub of &get_cat_id routine inside the editor.cgi

Without the codes from Alex, it is executable and when I insert it is not executable anymore.

So it means that it needs some more dafinitions somewhere. However I can try with a clean and rthe original editor.cgi from the distribution. I think thats what I tried in the begining and the results was the same.

But widgetz thanks in advance for helping me out.



------------------
rajani

Hello!

Yes, it gives the error always. The Global package table is not defined and the two lines mentioned as syntax error.

Now for this test i have done with the original distribution editor.cgi

Alex, is it not possible for you to make the editor.cgi really stable abd properly designed for such general cases? Somethings like this will always help and cover broad number of users.

I would like to ask other links SQL users, Dont you all need some features like this for daily export-imports for Offline works or am I the only person?

------------------
rajani

Hello Alex!

I need to delete the lines after the import is done.



if (!$db->add_record (\%record)) {
$error .= "<li>" . ($line_num+2) . ": Failed validation. Error: <ul>$Links: BSQL::error</ul>\n";
$err_cnt++;
next LINE;
}

delete ($line_num + 2)

$good_cnt++;

Is there a way to to this?



------------------
rajani

Hi Rajani,

If you email me (or if it's big, send me a URL) to your access database. I'll put it in a format that you can easily import/export into Links.

Cheers,

Alex

Hello Alex!

I have a problem to understand you. I do have few more cloums which are not a part of the Default Links Database. Thats not a problem to create few new fields in a table. I can ofcourse send to you first ten lines. But thats not the solution.

Lets take a default Links Database. Even just for a test, I am not able to import and export it correctly. For e.g.

The Category are names in my database. Once it gets into MySQL, it will not come out as names back in the export. Also how do I get also the alternate categories in to the correct place. Further how do I get the Alternate category + Category information back into one table, as I cannot have two tables like in links inside MySQL. I can ofcourse have but its better to have one line that is having a unique identifier with Category + Alternate category information togather.

For e.g. lets take a default database from links v2.0, but with Category Names & Alternate Category names instead of their IDS. The Add/Mod_date is in DateTime format. Thats all. Thats my database then. It has more colums but if other columns can go through an array, the remaining will also get in and that should not be a problem. The biggest problem is that :

1-How to get Names in to IDs &get_category_id

2-Get an id into the respective Alternate Category table.

3-Change DateTime format to Time format during import.

4-After import those lines should be deleted so that one knows what went in and whats left to be deleted or what category is not there, etc

How to export for daily offline work:

5-The category IDs should be changed back to names for Category + & Alternate Cat.

6-The tables should be merged to one so that handling is easy. This means that Alternate Category table could be merged togather with the main Links Table. By doing so one does not misplace or loose the information related to it.



Therefore as you see, I have special columns, but what I need is the logic and function as to how the program will work. Honestly I can play a lot with Access and change many things and prepare for the import export. The limitations are datetimeformat and ID<>Names. If both this are programmed then anyone can do exports imports daily without any problems.

Moreover one can also export Validate table and then import it back to Validate of Links.

So a if serious thought is given on this area that will make a script really favorite of webmasters. For e.g. while exporting, one clicks - Delete after export-file of delete line from txt after import, or delete Old data from the table before import etc.

Delete from file during export/import
Delete from table data during export/import

Change to Category names during export/import

Change to Category IDs during export/import.

Change to Date do DateTime and viseversa during export/import

Enter a default value in a field number
during export/import.

Therefore when a link comes out with deletion from the old table forever, there would be no indication of that link in Hits or Votes. This means a Link is complete installed in a database and also completely de-installed from the database.

One can install a link it directly from web into the database or through import/export. This feature is missing. If this was there, I should have had no problem. This would make the life of everybody easier, atleast who are concious of their telephone bills or costs of internet provider sensitive prices.

I know Alex, once you take this in hand this area will be fully developed into its excellence that no one would have to say anything anymore.

------------------
rajani