Gossamer Forum
Home : Products : Gossamer Links : Version 1.x :

Build options : nph-build.cgi : Build step by step...

Quote Reply
Build options : nph-build.cgi : Build step by step...
Hello Alex!

In the Build All option, I have a problem. It calculated stats for 20 minitues and then killed itself for playing music too long. I then commented out the &calculate_stats option and also everything else except &build_all. Then it went further to build &category_pages and cannot build even half of the categories.

What do you think if we have a &build_select routine as a pop_up?

Then...

1-Fill in then numbers of categories to be build i.e. 1-100, 101-200, .....

OR

2-Pop-up gives all the main categories listed and one could click in the check box.

One could go for &Build_Number based option OR if one has &Build_Main_Category_Name

3- A pre-defined value. Build every 30 next category or every 50 next category.....

Further, if the entire nph-build.cgi is gives options to handle it better, for e.g. A pop-up giving &Build_only_stats

The problem I have is ofcfourse that of a shared serever.

------------------
rajani











Quote Reply
Re: Build options : nph-build.cgi : Build step by step... In reply to
rajani --

Sounds like your server is having either I/O problems or you are not getting enough CPU.

Are you on a shared (virutal) machine, or do you have a dedicated one? Alex's stats match my own. I gain in the speed of my machine, but I lose in the re-indexing since some of my descriptions are 15k long, the program chews on them for awhile.

Quote Reply
Re: Build options : nph-build.cgi : Build step by step... In reply to
20 Minutes?! Wow, that seems a bit excessive. You might want to look at increasing the power of the server. The Open Directory sample you see on our server has 45,000 links and 9,000 categories and it builds entirely in under 8 minutes.

Our server is only a Pentium 233 with 256 MB of memory, no powerhouse by any means.

I'm planning on a build_changed feature that will only update changed sections of the directory, however this is a major feature, and is probably a while away (not in the next release).

Cheers,

Alex
Quote Reply
Re: Build options : nph-build.cgi : Build step by step... In reply to
Hello Alex and pugdog!

At the moment I have 4600 categories. Sometimes it takes eighteen minites and sometimes twenty-five minites to calculate stats. After this if it crosses twonty minites limit, it gets too tired singing.

I duplicated and made many brothers of nph-build cgis. I renames them with nph-build-stats.cgi, nph-build-pages,cgi, and nph-build-staggered.cgi. This seems to help me. In all of those, only those related sub_routines are activated and others are simply commented out.

Now, I am trying to work on how to build 20,000 pages, mind you without having a SINGLE link. Alex has completely refused to help me and introduce import functions. I am absolutely unable to work. However I try to go further.

It has takes about thirty minites to build about 2,500 pages and then went to sleep forever. This pages have no links, but categories are 4600 in number.

Alex, the distribution zip file of links SQL does not include any heart attack pills for people like me. Why? Once it went crazy and build, just just like a mad horse, thousands of directories every where in my account directory. Well there was a slash too much Wink I plan to erase them during the entire Christmas time.

Hello Mr. Pataki!

My server is a virtual shared one, with 128 RAM, and pentium II at 450Mhz. This should be OK. Moreover, I tried to work at midnight EST or few before so that the server loads are minimum at mainly midnight in America.

Also there are not too many people on my server. and is certainly not crowded.

This means that the main difference is 1/2 of RAM compared to Alexes example, that makes it so slow.

[This message has been edited by rajani (edited October 25, 1999).]
Quote Reply
Re: Build options : nph-build.cgi : Build step by step... In reply to
 
Quote:
Alex has completely refused to help me and introduce import functions

That's a lie. I've offered numerous times to work with your Access file to make it an easy import/export out of Links SQL. You have either refused, or demanded that the functionality be instead built into editor.cgi.

Alex

Quote Reply
Re: Build options : nph-build.cgi : Build step by step... In reply to
Actually, the simple answer on the RAM is _YES_

The reason they stopped making 'overdrive' chips is that more performance could be obtained by adding RAM, rather than increasing the speed of the CPU. Most slow downs occur with I/O, be it to disk file input/output or with swapping memory. If you doubled the RAM you'd get more speed -- a lot more.

Also, by running virtual servers you have used up RAM for each process. In order to provide performance, the server pool has to be set with at least 25 machines, which if PHP is enabled are 3+ meg apiece, closer to 4. Right away you have lost almost all the RAM to idle 'keep alive' processes, and more to active ones. The machine is _probably_ paging memory to handle requests, so adding another 128K of RAM may very well solve many of your problems.

Apache without PHP takes up only about 1.2 meg of RAM per process in a beafy configuration, less if you run it lean.

I've seriously considered adding a machine with Apache 'lean' to handle the static requests, and re-direct requests for dynamic pages. The lean machine can be much less powerful to handle the same load -- or as powerful and can handle a much higher load.

Don't underestimate or discount the impact of hardware and software configuration into performance.

I get good performance since I'm running a dedicated machine for a dedicated purpose. I'm not running a lot of extra processes, and if I could find a tool as useful as MyPHPAdmin, I'd re-compile apache without PHP to reclaim the resources.

The performance difference is quite noticable.

Quote Reply
Re: Build options : nph-build.cgi : Build step by step... In reply to
Hello Mr. Pataki!

Thanks for your information. You always seem to be heavily informed all about server problems and how to go about! Thats a wealth of information you really have to share. Thanks very very much. I have to always read twice to digest all what you are talking about and thats not because of english.

Hello Alex!

Quote:
That's a lie. I've offered numerous times to work with your Access file to make it an easy import/export out of Links SQL. You have either refused, or demanded that the functionality be instead built into editor.cgi.
Yes ofcourse, you offered me to work on my Access database and also said that I gould give you the URL of my Access database.

My problems are not Access related specific.
Access 97 specifis problems:

There was one DATETIME during the exports which I managed to get it in by removing the Validate info from the defs, even though it is there in the MySQL database the format written. Then it imports only DATE as it is in MySQL like that to be stored due to charachter limitations.

This I have solved it myself.

Further, I can play with Access 97 in all the forms that it will result into a Links.db database. So you will achieve the same as much as I could.

Non-Access 97 specific import problems: lacking functionality in the import-export routines:

1.
A small tip "(Pugdog said) I have removed all the Validate info from the defs as it was more of the problems..." from Pugdog has helped me tremendously. I noticed that if this info is there than it is not able to recognize the import functions at all. This can be also taken care of during import sub_routine so that not every user has to go through this learning process.

2.
Importing Category.txt took four minites and had less number of tables and only 4600 lines. If I would like to work OFFLINE everyday, I will need to import and export a lot everyday. So if I have 40,000 links to export then I may need half an hour. Such an export ofcourse cannot be everyday.

After re-working the moment I import it all again completely new, everything will get a new CategoryID and LinkID. If I re-worked on the Categories OFFLINE, then their IDs are lost. However everywhere in the Links database they remain as IDs.

Therefore this database does not have a Relational functionality anymore. Hence if during every export/back-up if &get_category_name is called for instead of ID makes sense and is anyday better way of exporting the Links table.

Once I have the Category_names I can work import in any spreadsheet program. In my case it would be Access 97.

3.
Importing Links may take 20 minites, if I have about 40,000 links.

I do not have any Category database but only Links database with four columns of categories submitted by users.

So I can make a category table myself. Then it may be possible that there may be some category missing.

During imports, one can click in editor.cgi to delete the table data before insert. Now if there is one more click which will delete the line from @LINE in the import loop, will solve the problem.

To have a delete loop in the import loop makes it effective and help everyone because

a-Every user will then not have to telnet into the respective directory and erase the file.

b-Every would know exactly what is imported.

c-It secures to every users that it will not only delete the data from the table and will not produce duplicates BUT WILL ALSO inform the webmaster if in case if a line is corrupt or is not imported properly at a given time and that information will not be lost. It will also help if a category is modifies what consequence it has on the relative links thereby encouraging and offering OFFLINE work with and on links.

For this one needs to simply add few lines which will then create an additional file with a name "not-imported.$date.$table.txt"!

4.
Currently, Links SQL does not offer a real support an OFFLINE work.

This is because the import-export functions are simple dumping ones. Think about how can a webmaster work OFFLINE with the following:

-What happens to Category Names?

-What happens to Category Alternates?

-What happens to the Category IDs in their places when they are imported back after exports.

-How long it may take and how time comsuming it may be to see the Admin_HTML.pm creating the webpages of Validate. Here one could also simply export the table, work offline and correct/edit some things, import it back into links and simply build! One could ofcourse get it back to the Validate table and continue working further.

5.
All the necessary care is taken for the current Links v2.0 users. They will have absolutely no problems to import. A person which starts from scratch will also have no problems.

But a person who is in between will have some problems and all the issues resulting from my difficulties could be minimized by adding some codes to their basic routines.

7.
As mentioned above, I have Category Alternates in one table table of Access 97.

What shall I do during the imports? How can I get it in the Category Alternates.

How is the format of Links v2.0 users Links.db? The information of Category Alternates is within the same Links.db but in Links SQL it is a seperate table. How do I get them in to this seperate table?

Brent loopy gave a code - how to insert CategoryAlternates in to the table.

I tried to use Import.pl! You told me not to use it as it is for the links v2.0 users. The import codes from brent is not related to editor.cgi:. So how do I import?

So if the export will bring Category Alternates back to Links Table, which makes all the sense in relation to their IDs, Import will do the same. Import will bring it back to its respective table. If this function was there of exports-imports I would not have any problems at all.

The situation says that there needs a two way system required. Export and Import, where both are in harmony with each other and brings consistancy to the database and OFFLINE work. If this is achieved then not many people can have problems anywhere.

Today, the fact remains. This functiona are missing. They ARE NOT DEMANDED BY ME BUT BROUGHT INTO LIMELIGHT. They are the issues that have surfaced during my set of working. I simply have humbly requested and not demanded. If you develop them, fine. Then I would be the first one who would benefit. If not your product will miss this features. I will somehow manage my way, however dissatisfied and frustrated I may be.

I mention above that I assume that you do not want to help in adding some more functions. This functions that I mention as necessary and missing functions you did not react to my earlier thread and intentionally did not response. Hence I assume that there seems to be no interest from your side to think on working on such functions.

Therefore my problems, as seen from above are not really Access specific problems ONLY but that of functionality of Links SQL basic import-exports resulting out of missing functions in the editor.cgi. Helping me ONE TIME for importing my Access database will not solve many future hardships of future users including myself and that is what I have been talking all the time.

------------------
rajani











Quote Reply
Re: Build options : nph-build.cgi : Build step by step... In reply to
Hello Alex!

Quote:
That's a lie. I've offered numerous times to work with your Access file to make it an easy import/export out of Links SQL. You have either refused, or demanded that the functionality be instead built into editor.cgi.

You have clearly designed Links SQL for Links v2.0 users or it can be used as a completely new with 0 links. In both the cases it works.

I do not belong to any of those. Since three months I have hardships and have written essays and essays on problems.

You take your own database if Links.db and convert the Category id in to their Names. If other users would like to know what I am talking about, simply make in the right format one line for category and one for links table import in an ASCI file.

Please answer me the following questions :

When you have names instead of their ids, try to import?

What will happen to the Has_Num_links?

What will happen to Has_new_links?

Will you be able to build at all?

Please tell me how to do it.



[This message has been edited by rajani (edited October 27, 1999).]