Gossamer Forum
Home : Products : Links 2.0 : Discussions :

Database will not build - random crashes

Quote Reply
Database will not build - random crashes
I have been playing with Links for a while now on a UNIX system and think it is great.
I used to have a bunch of static links pages with about 1400 links on them. Recently I converted the pages to the links database format.
I have triple checked there are no errors in the links.db file and also ensured all the categories are correct etc...
When I try a build it always bombs out - just finishing early - before the entire db is created. I have added debugging info in some areas to show me which links are being created and stuff like that and it indicates to me it is a radom error. Very odd.
Is there a limit to the size of the database or could me request be timing out or soemthing - even though it displays everything ok until it just stops.
Are there any links.db "db format" verifying tools/features". I made a few sh scripts to do this for me.
It seems to crash more often in the creation of the largest of the categories and also in the whats new or cool links bit - both for build all or staggered.

Any help appreciated.
Quote Reply
Re: Database will not build - random crashes In reply to
That's very odd. It sounds like you are timing out, but I wouldn't expect that to happen during a staggered build. Do you have telnet access? If so, try building that way, as it should bypass any timing out problems.

Dan
Quote Reply
Re: Database will not build - random crashes In reply to
> ./nph-build.cgi
[truncated output]
Building What's New Page . . .
Processing category: Aviation_Fire_and_Rescue
Processing links: 449 450 451 452 453 454 455 done.
Processing category: Emergency_Management
Processing links: 392 393 394 395 396 397 398 399 400 401 402
Processing links: 403 404 405 406 407 408 409 410 411 412 413
Processing links: 414 415 416 417 418 419 420 421 422 423 424
Processing links: 425 426 427 428 429 430 431 432 433 434 435
Processing links: 436 437 438 439 440 441 442 443 444 445 446
Processing links: 447 448 done.
Processing category: Emergency_Medical_Services
Processing links: 501 502 503 504 505 506 507 508 509 510 511
Processing links: 512 513 514 515 516 517 518 519 520 521 522
Processing links: 523 524 525 526 527 528 529 530 531 532 533
Processing links: 534 535 536 537 538 539 540 541 542 543 544
Processing links: 545 546 547 548 549 550 551 552 553 554 555
Processing links: 556 557 558 559 560 561 562 563 564 565 566
Processing links: 567 568 569 570 571 572 573 574 575 576 577
Processing links: 578 579 580 581 582 583 584 585 586 587 588
Processing links: 589 590 591 592 593 594 595 596 597 598 599
Processing links: 600 601 602 603 604 605 606 607 608 609 610
Processing links: 611 612 613 614 615 616 617 618 619 620 621
Processing links: 622 623 624 625 626 627 628 629 630 631 632
Processing links: 633 634 635 636 637 638 639 640 641 642 643
Processing links: 644 645 646 647 648 649 650 651 652 653 654
Processing links: 655 656 657 658 659 660 661 662 663 664 665
Processing links: 666 667 668 669 670 671 672 673 674 675 676
Processing links: 677 678 679 680 681 682 683 684 685 686 687
Processing links: 688 689 690 691 692 693 694 695 696 697 698
Processing links: 699 700 701 702 703 704 705 706 707 708 709
Processing links: 710 711 712 713 714 715 716 717 718 719 720
Processing links: 721 722 723 done.
Processing category: Emergency_Radio_Networks
Processing links: 456 457 458 459 460 461 462 463 464 465 466
Processing links: 467 468 469 470 471 472 473 474 475 476 477
Processing links: 478 479 480 481 482 483 484 485 486 487 488
Processing links: 489 490 491 492 493 494 495 496 497 498 499
Processing links: 500 done.
Processing category: Fire_Departments
Processing links: 10 11 12 3 4 5 6 7 8 9 20
Processing links: 21 22 13 14 15 16 17 18 19 30 31
Processing links: 32 23 24 25 26 27 28 29 40 41 42
Processing links: 33 34 35 36 37 38 39 50 51 43 Terminated
>

I ran this four times and each time it stopped at a different location. The code I added was in the whats new and cool sections to output the link id as it goes - thought I may have had a dead link or something.
As it crashes at different times - I am not sure what to do...
Quote Reply
Re: Database will not build - random crashes In reply to
When in doubt, replace all scripts with the standard distribution and see what happens. It's possible that you accidentally changed something...

Dan
Quote Reply
Re: Database will not build - random crashes In reply to
I have tired the old scripts and it just says "Terminated" randomly in different spots. Mainly in the building of the whats new section. Always in the category with the most links.

Are the sizes of the URL and Name fixed to any length? As I bulkloaded it, the size restraints of the web form are not in place?
Quote Reply
Re: Database will not build - random crashes In reply to
That's possible. Links.def sets the max length for those fields. You could try raising the setting for URL to 150 or so and see if that changes anything.

How many "new" links do you have? Bulkloading a large number of new links might cause that to overwhelm the build process. If nothing else works, you might want to try making the new cutoff 1 day or bulkloading the links with an older date.

Dan
Quote Reply
Re: Database will not build - random crashes In reply to
# Definition of your database file.
%db_def = (
ID => [0, 'numer', 5, 8, 1, '', ''],
Title => [1, 'alpha', 40, 75, 1, '', ''],
URL => [2, 'alpha', 40, 75, 1, 'http://', '^http|news|mailto|ftp'],
Date => [3, 'date', 15, 15, 1, \&get_date, ''],
Category => [4, 'alpha', 0, 150, 1, '', ''],
Description => [5, 'alpha', '40x3', 500, 0, '', ''],
'Contact Name' => [6, 'alpha', 40, 75, 1, '', ''],
'Contact Email' => [7, 'alpha', 40, 75, 1, '', '.+@.+\..+'],
Hits => [8, 'numer', 10, 10, 1, '0', '\d+'],
isNew => [9, 'alpha', 0, 5, 0, 'No', ''],
isPopular => [10, 'alpha', 0, 5, 0, 'No', ''],
Rating => [11, 'numer', 10, 10, 1, 0, '^[\d\.]+$'],
Votes => [12, 'numer', 10, 10, 1, 0, '^\d+$'],
ReceiveMail => [13, 'alpha', 10, 10, 1, 'Yes', 'No|Yes']
);

Which column is the max length one? Smile
Quote Reply
Re: Database will not build - random crashes In reply to
#4 (75) is for max length.

id, type, input length, max length, required, etc.

Dan
Quote Reply
Re: Database will not build - random crashes In reply to
Thanks for that Dan. Your support service is wonderful.

It still barfs though Smile

I am going to split the category that is causing problems up into several smaller ones to see if I can narrow down any bad links or something.

Thanks for your help again.
Quote Reply
Re: Database will not build - random crashes In reply to
The problem is twofold:

1) You are adding a lot of links at one time.
2) You have way too many links in each category. Try splitting up your database into more categories/sub_categories.

Regards,

------------------
Eliot Lee
Anthro TECH,L.L.C
www.anthrotech.com
* Be sure to visit the Resource Center for FAQ's, Modifications and Extra Goodies!!
* Search Forums!
* Say NO to Duplicate Threads. :)
----------------------








Quote Reply
Re: Database will not build - random crashes In reply to
220 links in a category (based on above) probably is excessive, but I have a couple of categories with 150-190 (30 per page) links each and it hasn't posed any problem.

Dan
Quote Reply
Re: Database will not build - random crashes In reply to
Welp, again, I am contradicted. What a surprise...

Frown

Regards,

------------------
Eliot Lee
Anthro TECH,L.L.C
www.anthrotech.com
* Be sure to visit the Resource Center for FAQ's, Modifications and Extra Goodies!!
* Search Forums!
* Say NO to Duplicate Threads. :)
----------------------








Quote Reply
Re: Database will not build - random crashes In reply to
Man, would you quit taking things so personal?! I'm not out to contradict your every statement, as you make it sound. I was merely pointing out that it could be one thing, it could be another, it could be a combination of the two, or it could be neither. I don't see any reason to make it an absolute.

Dan
Quote Reply
Re: Database will not build - random crashes In reply to
And I provided two possible reasons...which is not absolute...If I was absolute, I would've only provided ONE FACTOR!

Regards,

------------------
Eliot Lee
Anthro TECH,L.L.C
www.anthrotech.com
* Be sure to visit the Resource Center for FAQ's, Modifications and Extra Goodies!!
* Search Forums!
* Say NO to Duplicate Threads. :)
----------------------








Quote Reply
Re: Database will not build - random crashes In reply to
absolute = positive, definite
singular = one

.
..

absolute != singular
Quote Reply
Re: Database will not build - random crashes In reply to
The build worked this morning when I got in. This is odd as I didnt do anything since last Friday.

I have now made a lot of other categories and started moving links into them to spread the load.

Maybe the bandwidth between my work and my ISP was good - but this is odd as I have run the script on the server explicitly and it barfed before also?

Will let you know if I find any reason for this problem.

PS: the site is http://www.techrescue.org/ and the links section is off the [links] tag.