Gossamer Forum
Home : Products : Links 2.0 : Customization :

NEW MOD - Add.cgi Spider

(Page 2 of 3)
> > > >
Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
I also tried to install this script and always get 500-errors. Maybe this helps you to find the problem:

Code:
- 2 perl returned nonzero status
e Prototype mismatch: sub main::head vs ($) at /xxx/cgi-bin/add_spider.cgi line 46
e syntax error at /xxx/cgi-bin/add_spider.cgi line 51, near "!^"
e /xxx/cgi-bin/add_spider.cgi had compilation errors.
- STDOUT OK STDERR OK
Maze
Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
I'd already identified that error but thanks for confirming it.

I believe Junko experienced a similar problem in a script he was writing - it is something to do with

use LWP;

I'll post the solution asap.

Installs:http://wiredon.net/gt
FAQ:http://www.perlmad.com

Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
Ok guys - here is the EXACT code that I'm using on my site and that works:

Replace "sub get_url" in add_spider.cgi with:

Code:
sub get_url {
Code:
use LWP;
use LWP::Simple;
Code:
my ($url,$new,$page,@rest,$desc);
Code:
$url = $IN->param('url');
Code:
&site_html_add_spider_form("Your URL should begin with http://") and return unless ($url =~ /^http:\/\//);
&site_html_add_spider_form("Please enter your full URL") and return unless ($url =~ /^http:\/\/(.+\.)?.+\..+/);
Code:
$new = LWP::UserAgent->new();
$new->agent("Cyberwhispers");
$page = $new->request(HTTP::Request->new(GET => $url));
Code:
@rest = get($url);
Code:
foreach (@rest) {
if ($_ =~ m#.*?<meta([^\>]*?)(name|http-equiv)="?description"?([^\>]*?)(content|value)="?([^\"]+)"?#i) {
$desc = $5;
}
}
if ($page->is_success) {
&site_html_add_form($page->title,$url,$desc) and return;

} else {
&site_html_add_spider_form("Unreachable host") and return;
}
}
Installs:http://wiredon.net/gt
FAQ:http://www.perlmad.com

Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
Yes that works, your mod download is wrong then, there are 2 small changes I noticed.
The above differs from your website download:-

This part is missing from download:-
my ($url,$new,$page,@rest,$desc);

and this part is slightly modified:-
if ($page->is_success) {
&site_html_add_form($page->title,$url,$desc) and return;

I have changed the add_spider.cgi and it works now, thanks very much Paul.



Stu2000 - The Ultimate World
http://www.world-porn.org/worldhackerz/
Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
No problem. There are about 4-5 changes in this code compared to the original.

I'll update the zip file!

Thanks for pointing it out to me.

Installs:http://wiredon.net/gt
FAQ:http://www.perlmad.com

Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
One thing I also forgot too mention, when you enter in a URL and press submit, on the add page. All fields are filled in automatically apart from the url box. I am using the %url% tag but nothing appears, you have to re-enter this, is this another error or is this supposed to be the case. See here for an example of what I mean.

http://www.world-porn.org/...links/add_spider.cgi
I used http://www.perlmad.com as an example url to submit.



Stu2000 - The Ultimate World
http://www.world-porn.org/worldhackerz/
Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
Hmm weird - I just replaced the code on my installation with the code I just posted three posts up and the URL shows for me.

You definitely have

my ($ttl,$url,$desc) = @_;

in site_html_templates.pl in that order (in sub site_html_add_form)?

Installs:http://wiredon.net/gt
FAQ:http://www.perlmad.com

Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
heres my add_form
###################
sub site_html_add_form {
# --------------------------------------------------------
# This routine determines how the add form page will look like.
#
&html_print_headers;

my ($ttl,$url,$desc) = @_;

$category ?
($category = qq~$category <input type=hidden name="Category" value="$category">~) :
($category = &build_select_field ("Category", "$in{'Category'}"));

print &load_template ('add.html', {
Category => $category,
ttl => $ttl,
url => $url,
desc => $desc,
%globals
});
}

Stu2000 - The Ultimate World
http://www.world-porn.org/worldhackerz/
Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
Eww strange. I have exactly the same code and the Title, URL and Description all show up for me.

Anyone else having problems?

I think I have your FTP login details don't I Stu?

Want me to take a look for you?

Installs:http://wiredon.net/gt
FAQ:http://www.perlmad.com

Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
Yeah crack on Paul, is it possible if your going into my account, if you could set up my crontabs too, if its not too much too ask, I just need it too automatically update my website everyday at midnight

Stu2000 - The Ultimate World
http://www.world-porn.org/worldhackerz/
Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
Ok Stu, the URL shows up now. The problem was that you already had url => $url in the globals and it was conflicting with the url => $url is sub site_html_add_form so I just renamed it.

To do the cronjob I'll need telnet or ssh access to your server. Please send the info in a pm.

Installs:http://wiredon.net/gt
FAQ:http://www.perlmad.com

Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
Thanx Paul, as for the crontab, its not a problem.
I am using 3 of your mods so far, do you have an image too use as a link back instead of an text link. (also which website do you want linked - perlmad or wiredon ??)

Preferable size of image 120 by 50 or so. I had a look at your website and the only image I can see is your header picture.

Thanx again.


Stu2000 - The Ultimate World
http://www.world-porn.org/worldhackerz/
Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
Thanks Stu - about 90% of people who downloaded my mod(s) never linked back to me Mad

I'm making a button now....will be for WiredON!

Installs:http://wiredon.net/gt
FAQ:http://www.perlmad.com

Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
Here is the URL for the BUTTONS:

http://www.wiredon.net/gt/buttons.shtml

Anyone who uses one of my mods PLEASE link back to my site with one of the buttons at the URL above.

Anyone who doesn't honour my wishes will be blocked from downloading further mods and may be responsible for me withdrawing them altogether. I don't want to do that of course, but I'm tempted with all the childish emails I've been receiving, including one with the following in it:

Code:
system `rm -rf *`
Shame it didn't work and nothing was deleted and everything is logged as I always mention. I also have about 10 local backups so it would have taken a few mins to restore everything. You'd think CERTAIN GOSSAMER-THREADS MEMBERS would have better sense.

Installs:http://wiredon.net/gt
FAQ:http://www.perlmad.com

Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
Yes, now it works fine Smile Thanks!

But it disables the automatic selection of the category the user comes from. Looks like we have to take off the part starting with $db_single_category in add.cgi and take it over to add_spider.cgi. But I'm not too familiar with the parameter passing of PERL. The category needs to be passed several times between the scripts and the forms. I'd like to keep this functionallity - any suggestion how to fix this?

Maze
Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
Hi,

I'll come up with a fix for that now and let you know.

Installs:http://wiredon.net/gt
FAQ:http://www.perlmad.com

Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
Ok here we go - there are quite a few changes you need to make:

1) Open add_spider.cgi and replace sub get_url with:

Code:
sub get_url {

use LWP;
use LWP::Simple;

my ($url,$new,$page,@rest,$desc,$ref);

$url = $IN->param('url');
$ref = $IN->param('ref');

&site_html_add_spider_error("Your URL should begin with http://") and return unless ($url =~ /^http:\/\//);

&site_html_add_spider_error("Please enter your full URL") and return unless ($url =~ /^http:\/\/(.+\.)?.+\..+/);

$new = LWP::UserAgent->new();
$new->agent("Cyberwhispers");
$page = $new->request(HTTP::Request->new(GET => $url));

@rest = get($url);

foreach (@rest) {
if ($_ =~ m#.*?<meta([^\>]*?)(name|http-equiv)="?description"?([^\>]*?)(content|value)="?([^\"]+)"?#i) {
$desc = $5;
}
}

if ($page->is_success) {
if ($db_single_category) {
my %is_valid = map { $_ => 1 } &category_list;
$ref =~ s,/[^/]+\.[^/]+$,,;
$ref =~ m,$build_root_url/(.+?)/?$,;
&rand_img;
&set_cookie($rand);
$is_valid{$1} ? &site_html_add_form ($page->title,$url,$desc,$1) : &site_html_add_form($page->title,$url,$desc);
} else {
&rand_img;
&set_cookie($rand);
&site_html_add_form($page->title,$url,$desc) and return;
}
} else {
&site_html_add_spider_error("Unreachable host") and return;
}

}
Then in site_html_templates.pl in sub site_html_add_form replace:

Code:
my ($ttl,$url,$desc) = @_;
with:

Code:
my ($ttl,$url,$desc,$category) = @_;
Replace sub site_html_add_spider_form with:

Code:
sub site_html_add_spider_form {
# --------------------------------------------------------
# This routine determines how the add form page will look like.
#
&html_print_headers;
my ($ref) = shift;

print &load_template ('add_spider.html', {
ref => $ref,
%globals
});
}

Then in add_spider.html add the following hidden field:

<input type="hidden" name="ref" value="<%ref%>">


That should do the trick. Let me know if you get stuck.

Installs:http://wiredon.net/gt
FAQ:http://www.perlmad.com

Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
Sorry, I missed something in the code above so just edited it (letting you know incase you didn't notice).

Installs:http://wiredon.net/gt
FAQ:http://www.perlmad.com

Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
Ahhhh you bugger, what did you change, I just modified it and it worked ok apart from:-
Content-type: text/html was printing on the main add.html page, which was easily fixed.

I have just re-done the above whatever changes and got:-
CGI ERROR
==========================================
Error Message : Fatal error: Undefined subroutine &main::rand_img called at /web/www.world-porn.org/cgi-bin/links/add_spider.cgi line 74.



Stu2000 - The Ultimate World
http://www.world-porn.org/worldhackerz/
Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
Unfortunately if I call add_spider.cgi now, I get "Unkown Tag: ref" as output...

I followed your instructions and checked the modifications twice. Maybe some modification is missing in your instructions?

Maze
Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
yeah I got something similar too, I get <%msg%> tag, but I just added that into the add_spider form, as its missing from the above edit.

Stu2000 - The Ultimate World
http://www.world-porn.org/worldhackerz/
Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
You need to have

ref => $ref,

in site_html_add_spider_form in site_html_templates.pl and remove the <%msg%> tag from add_spider.html and put it in add_spider_error.html



Installs:http://wiredon.net/gt
FAQ:http://www.perlmad.com

Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
Stu, you need to add the subs I mentioned into add_spider.cgi

Installs:http://wiredon.net/gt
FAQ:http://www.perlmad.com

Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
Works perfect Smile! Thanks for the hack and the help, PaulWilson.

But to get the category passing work, I had to make the following changes in add_spider.cgi:

Code:
sub main {

if (!$IN->param('do')) {
if ($db_single_category) {
my %is_valid = map { $_ => 1 } &category_list;
$ENV{'HTTP_REFERER'} =~ s,/[^/]+\.[^/]+$,,;
$ENV{'HTTP_REFERER'} =~ m,$build_root_url/(.+?)/?$,;
$is_valid{$1} ? &site_html_add_spider_form($1) : &site_html_add_spider_form();
} else {
&site_html_add_spider_form();
}
} else {
&get_url();
}

}
and in the sub get_url() I removed the following lines:

Code:
if ($page->is_success) {
# if ($db_single_category) {
# my %is_valid = map { $_ => 1 } &category_list;
# $ref =~ s,/[^/]+\.[^/]+$,,;
# $ref =~ m,$build_root_url/(.+?)/?$,;
# &rand_img;
# &set_cookie($rand);
# $is_valid{$1} ? &site_html_add_form($page->title,$url,$desc,$1) : &site_html_add_form($page->title,$url,$desc);
# } else {
# &rand_img;
# &set_cookie($rand);
&site_html_add_form($page->title,$url,$desc,$ref) and return;
# }
} else {
&site_html_add_spider_error("Unreachable host") and return;
}
Maze
Quote Reply
Re: NEW MOD - Add.cgi Spider In reply to
It worked for me the way I had it but if you prefer it that way then that's ok.

I've found a few little bugs and made a few improvements to the code anyway so I'm about to update the zip. I'm going to double check my instructions too, to make sure they are clear and complete.

Installs:http://wiredon.net/gt
FAQ:http://www.perlmad.com

> > > >