duplicate check in links sql is weak. for example if you have http://www.mysite.com
and you added http://www.mysite.com/ and the software will never find these as duplicate
not to mention www.mysite.com and mysite.com aslo mysite.com and mysite.com/index.html
and you added http://www.mysite.com/ and the software will never find these as duplicate
not to mention www.mysite.com and mysite.com aslo mysite.com and mysite.com/index.html