Has anyone written a script that will take a text file from an outside server and place that text on their own site? Example: I have people who want to write daily articles for my site. They could just upload their text file on their web site and my site would pull it from there.

use LWP::Simple;
$file = get($url);
$file =~ s/\n//g;

This requires the LWP::Simple perl module.

Dan

Thanks Dan ... worked like a charm!

Hi..

How would I get that to work on my site?

I presume that it'd have to be written into a cgi scipt of it's own.. and would it grab the file from another server, or only from my own?

Thanks..

Simon - a tad clueless about cgi

Yes, the above code snippet is just that. It would have to be incorporated into a script or at least modified as a script depending on what you want done.

It fetches files from any server - local or remote - given an URL.

Dan

Does LWP module is a standard for perl. I mean does server like www.virtualave.net or hypermart.net (server that provide free hosting) support this module. Is there any other way to do thats without using this module.

[This message has been edited by aneip (edited February 11, 1999).]

Whether the LWP module is standard varies server to server - in my case, I had to get them to install the module. But I believe hypermart does as many of the Gossamer Threads 'Links' sites are hosted by them. If you are running 'Links' then you have it installed. If not then contact your server.

There is a "sockets method" but it is a tad more involved and I don't have an example at hand.

Dan

To Simon:

The script snippet fetches a file from a supplied URL to the $file string variable.

Step by step, what exactly do you want done?

Is the URL to be inputted by yourself or a surfer, or another script, or a data file.

What do you wish to do with the fetched file ($file)? Save it as a file or use it as input for another script? Or... ?

Dan

Hi..

I might as well be honest.. Im totally clueless about CGI.. I can install Links, and have links2b5 running, but any more than that, and I get stuck..

Can someone make

--
use LWP::Simple;
$file = get($url);
$file =~ s/\n//g;
--

into a proper cgi script that will run for me, as I don't have a clue what it'd need to do..

so far I have got

--
#!/usr/bin/perl
#geturl.cgi - grabs url from any server
#######################################
use LWP::Simple;
$file = get($url);
$file =~ s/\n//g;
--

Now what else does it need, and how do i tell it what url to grab, and what to do with it once it's been grabbed?! (told you i was clueless *grin*)

Thanks

Simon

Hi Dan..

Im extreemly greatful to you for being able to help me

>>"Is the URL to be inputted by yourself or a surfer, or another script, or a data file."

I think that the best way would be for it to be called by the script, eg..

www.a-web-host.com/cgi-bin/urlgrab.cgi?http://www.anotherhost.com/index.html

>>"What do you wish to do with the fetched file ($file)? Save it as a file or use it as input for another script? Or... ?"

id like it to be saved as a file (html or .txt) and then i will be including it via ssi, eg <!--INCLUDE VIRTUAL="saved-file.txt"-->

So i spose it'd be useful to be able to set where it's saving the file to as well..

Do you think you'd be able to help me with that??? Id be soooo please if you can

Thanks in advance..

A very happy and still cgi-clueless Simon


[This message has been edited by Simon Martin (edited February 12, 1999).]

How about the following. This script will save the fetched file to the file you define as $datafile in the script. Note that the script will overwrite the file if it already exists - the script could be altered to accept the filename via the command line like the URL. Call the script as follows:

www.a-web-host.com/cgi-bin/urlgrab.cgi?url=http://www.anotherhost.com/index.html



Dan


[This message has been edited by dan (edited February 12, 1999).]

Dan, your code for some reason gives me a "500" error. What could be a problem?

------------------
-[ It's just another night; good morning Webmasters! ]-

Are you able to access your server error logs? My guess is that the error message is "Premature end of script headers".

I should preface that I am not a professional programmer and only dabble in Perl-CGI. For some reason it seems that you cannot call scripts directly (as URL) that do not include a HTTP header. But the text file should have been created nevertheless - did in my case, you?

Tagging the following (HTTP Header) to the end of the script should solve things:


But I know there is a better way. Alex (or other pro) can you shed some light on this?

Dan

Dan,
you mean like this:


#!/usr/local/bin/perl
$datafile = "/home/find/public_html/01/test.txt";
@pairs = split(/&/, $ENV{'QUERY_STRING'});
foreach $pair (@pairs) {
($name, $value) = split(/=/, $pair);
$FORM{$name} = $value;
}
$lwp = 1;
use LWP::Simple;
$file = get($FORM{url});
open (DATA, ">$datafile") or &cgierr("Error: Unable to open $datafile. Reason: $!");
print DATA "$file";
close (DATA);
chmod (0666, "$datafile") or &cgierr("Error: Unable to chmod $datafile. Reason: $!");
print "Content-type: text/html\n\n";
print "File fetched and saved!\n";


Well, it still gave me "500".
I tried to do this:
http://find.virtualave.net/01/test.cgi?url=http://www.yahoo.com/index.html

Any suggestions?
Thanks!

Pasha

------------------
-[ It's just another night; good morning Webmasters! ]-

the following line is corrupted by the BB, so maybe you should mail eachother!

/*en (DATA, ">$datafil*/; or &cgierr("Error: Unable to open $datafile. Reason: $!");

What do you mean, corrupted?

------------------
-[ It's just another night; good morning Webmasters! ]-

don't you see it?

/*en should be open for example.

hehe!

I guess that wouldn't help avoid those damned 500 errors

------------------
Craig Antill
10 Tenths Motorsport (1.1) - www.Ten-Tenths.com/links/
THE Directory (2.0b3) - www.gardenbuildings.com/

To Pasha:

I (1) copied the pasted the code from 'your message', (2) changed the path to perl and the datafile and (3) uploaded it to my server and tested it. No problems or errors.

So now I must ask the gimme questions:

1. Did you save (in a ASCII editor-type program like Notepad or HotDog) and upload the file in ASCII?

2. Did you set permissions of the script to 0755 (_rwxr_xr_x)?

3. Are you sure your path to perl is #!/usr/local/bin/perl and not #!/usr/bin/perl?

4. Did you set the permissions of the directory that includes the script and datafile to 0777 (drwxrwxrwx)? My guess is that you did not do this.

BTW, I'm guessing you do not have access to your error logs. If you know the path (ask your server), there are a number of scripts that will extract the logs as they are handy in debugging scripts and other problems.

Dan

Woah, thank you very much for creating the script Dan

I shall have to try it out, and get back to you early next week as to wether it works or not, as Im at home over the weekend, and back at uni on Sunday evening.. - I shall upload it now, but Im in a bit or a hurry to get a stack of emails sent, so can't fully test it tonight

To: DS, the line doesn't appear corrupted on my computer.. it's definelty showing as open and not */pen

Anyway, thanks again.. I really appreciate it

Simon.

Dan,
I did all chmodding, and the path to perl is correct. And I still get "500". What I think is that it's not my fault, because I am right now getting "500" on rate.cgi from Links 2, but I didn't mess with it. So, what I think is that something else is going on here. Could it be that perl wasn't installed properly on the server???
Now, what kind of error logs are you talking about, where can I find them?
Thanks!

Pasha

----------------------------
-[Love 4 solving problems!]-

Where do you put the url of the page you want the script to grab?

------------------
elitecaraudio.com
carforums.com

Pasha: Get in touch with your server. I could add 'error handling' that might trace the problem.

elite: The URL is retrieved from the command line. For example, www.a-web-host.com/cgi-bin/urlgrab.cgi?url=http://www.anotherhost.com/index.html

But it can very easily changed to accept from <form> output

Dan

Something strange is going on on virtualave.net I didn't mess or change anything related to rate.cgi and now it works fine!
May be they upgrading some stuff, or something?

Hell knows...

Well, everything works fine, except for fetching a file from another server:
http://find.virtualave.net/01/test.cgi?url=http://www.yahoo.com/index.html
Still can't get it work. Have any one?

Thanks for your help guys!

Pasha

------------------
-[ It's just another night; good morning Webmasters! ]-

Hi Dan..

I can't get it to work either..

the error message that i get in the server logs is..
--------------
Undefined subroutine &main::cgierr called at /data1/hypermart.net/hertsinternet/cgi-bin/geturl/grab.cgi line 14.
[Sun Feb 14 17:25:09 1999] access to /data1/hypermart.net/hertsinternet/cgi-bin/geturl/grab.cgi failed for 147.197.183.165, reason: Premature end of script headers
--------------

I know that i definetly uploaded in ASCII, that the directories are chmodded 777 and that the scrit is chmodded 755, I also tried with the test.txt created and chmodded 666..

Any ideas?

Simon