uhh.. hi

anyways.. i was just fiddling around with something that steveb brought up and made a really simple script to automatically get the TITLE and DESCRIPTION of the link by having a URL...

anyways.. i found that this is a really easy mod for add.cgi and it would probably do what lots of you want.. (spider)..

anyways.. right now it's really simple.. if you guys really really want it.. i'll make an admin for it that will spider as many links as it can to get "updated" titles and descriptions.. (of course that will take along time.. can make it do like a max of 5 minutes of something.. and start at the one it left off so it will keep going again.. later)

well.. you can see what it is here:

http://www.pdamania.com/...samer-threads.com%2F

some sites still seem to be able to sneak javascript tags by me.. i have to fix that..

you can try it on your own site here:

http://www.pdamania.com/geturl.cgi

on the very bottom.. if you have a meta description tag.. it will simply use that.. (since people make it for search engines )

anyways.. feedback please

jerry

here is a good one..

http://www.pdamania.com/geturl.cgi?url=http%3A%2F%2Fwww.gossamer-threads.com%2Fscripts%2Fforum%2Fresources%2FForum3%2FHTML%2F002961.html

wooo.. long url..

remember.. this will go into the add.cgi.. so it's not really gonna be visible like this

[This message has been edited by widgetz (edited August 29, 1999).]

[This message has been edited by widgetz (edited August 29, 1999).]

[This message has been edited by widgetz (edited August 29, 1999).]

[This message has been edited by widgetz (edited August 29, 1999).]

This looks good. People would probably have to setup a new database though because alot of them have a bunch of links alread and since this gets the description, title, and url, like altavista, additions using this script and the way things are setup wouldn't mesh. I was thinking of doing something like this, and i have it going to the url, and i can get it to look in the head of the document and take description, but do you know how to also take keywords? Does this use lwp?

------------------
-----------------
Lavon Russell
LookHard! Search
www.lh.yi.org
webmaster@lh.yi.org

no...... an admin utility to update each entry is really easy to make.. just time consuming to use

jerry

Oh, i get what your saying. This would be a good mod, but like you said it will be time consuming. Maybe to the point of like Hotbot where they spider like every 3 weeks or more.

------------------
-----------------
Lavon Russell
LookHard! Search
www.lh.yi.org
webmaster@lh.yi.org

It's good. I would love to have it. It just spiders the pages and prints back the info right? I mean, there is no information that is stored.

Would be nice if you could somehow exclude all the unwanted info from tags such as images, javascripts and so on.

Sounds great!! If you don't mind, i'd love to have it, also...

Though, i had an idea for it:
If there is a description already specified, the program would go out and grab the description, otherwise, it'd leave it alone.

Thanks,
--Tom

[This message has been edited by garadon (edited August 30, 1999).]

i already got all HTML tags.. just javascript...

anyways.. the information IS stored.. in links.db as the Title and Description fields..

jerry

What you have there currently seems just like a spider that brings back info and prints, like the other spider mod that is in the resource center. Am I wrong?

Because the other script brings back just the URL's. One could use this in conjunction to show the description and all.

[This message has been edited by socrates (edited August 30, 1999).]

no.. i could add mine to add.cgi and it would get the fields.

Wow Jerry...that's pretty cool. When you say you could put it in add.cgi, do you mean a visitor could spider their site, then fill in the rest of the form...or did you mean to replace the add.cgi?

Either way, pretty nifty. I ask again...DO YOU SLEEP???

I think he means you would have all the fields minus Title, and Description, and only have url. Like altavista you'd have to tell them what your doing. Then on the results, and error page, you can put the Title and Description field so you can get what the spidering brought back. I'd only use the short and the meta description.

------------------
-----------------
Lavon Russell
LookHard! Search
www.lh.yi.org
webmaster@lh.yi.org

you could change links to only category, url, title, description.. and you get altavista..

jerry

That would work good with this thing i did for mine for when i have two instances of links one of which maybe a regular links using categories and stuff with a few mods of mine and others, but then the other with your mod and it will be like "webpage results" like Yahoo/Inktomi, with the automatic thing of going to Inktomi. So for the second one, links will totally impersonal and only be an altavista with urls, title and meta_description. The thing i did to make it search the second automatically was this

------------------
-----------------
Lavon Russell
LookHard! Search
http://lh.hypermart.net
webmaster@lh.hypermart.net

Can't wait to see this puppy! When users add a site, that would be excellent.. it would ecourage users just typing in a bunch of URLs they know since it means less typing for them.

On the admin side, it's also incredibly useful... although since I'm pretty picky I probably wouldn't use it on that end. I have my own syntaxes for that users get used to.. little things like if the link is to Yahoo's Baseball page then its
Yahoo: Baseball

which would differentiate it from
About.Com: Baseball

nothing really original.. lots of people do it, but I'd lose that. However... for the add part... let's go!

Hey Jerry
How's it going with this thing?
I just checked it out and see you got the results modified, and you have either the short description or the meta_description going in the same spot. Really cool. I like the way you have either one truncated to a specified number of characters.

What is the max output of the description field?

------------------
www.webgotcha.com
[b]admin@webgotcha.com[/b]

that is based on what you selected inside of links.def...

jerry

I tried it on a fake url: http://-Sanguinarius-no-url.com/~Sanguinarius/vampires.htm and it came back only as a hyperlink to that url. Is there any way of doing it so that if the link doesn't exist or is not valid it says so?

------------------

--Sanguinarius
The Real Vampire Directory
www.sanguinarius.org/cgi-bin/links/pages/

yea.. just say..

if (!$description) { blah blah.. is not a real link.. }

something like that...

jerry

the problem now is what do i do with a page with no title?

(Untitled) ?

Why bother indexing pages that don't have titles?

Why don't you tell them to put one on the page like i guess

[This message has been edited by Bmxer (edited September 08, 1999).]

i don't know.. maybe they can supply the title.. like a error page with a form input for the title..

anyways.. i'm not even going to use this.. so

jerry

Alright, whatever you wanna do with it. I'm sure we can enhance it if we want it to.