Hye,

Often when I'm reading some topics in this forum, people ask about 'how much link could i have' and other questions like this.

I know that my page anytime in the future will grow up to much more than 1.000.000 links. So I thinked about a way how to make this possible.

My idea: LLinks should have an option for only building pages which must be updated after adding a link. For 1 new entry this wouldn't be much more than 3 or 4 pages (new, cool, page with the link and if you're using LLinks the mainpage). only if a link have more alternate categories we must rebuild some more pages.

but this way would save very, very much time!!

OHH! I've forgot, you MUST rebuild all pages - cause otherwise you wouldn't have the latest number of hits for each link. WRONG! not when my mod is ready. I'm adding a script which loads a grapic-counter :-)

If you're interested you should know that i don't know if this feature will be available for 'Links'. Cause I'm building for 'Language Links' and haven't the time to make it compatible with 'Links'.

Ok, if anyone is already working on a mod like this, let me know.

If you're interested on 'Language Links' take a look to the infopage
www.ford-network.at/LLinks

cu
and nice greetings from austria
it's 12:50 o'clock in austria :-)

------------------
www.ford-network.at

I am really interested in this mod, infact I have a little suggestion:
I've found that the time of building the pages depends on how many NEW links you have. So, if in the future I'll have to build my pages wil about 500 new links per day, with database of 1.000.000, it will take about an hour (build all), and about 5-10 hours with (build staggered).
The suggestion is, to make a mod to build just new links, or build them first. I don't actually know the proccess of building pages, but the fact is, it will take too much time to build NEW and the rest links together.
Bogus, let me know what you think of this.

Regards,

Pasha

------------------
webmaster@find.virtualave.net
http://find.virtualave.net

Bogus,


Why? Links updates the hits and marks things as new before it does anything else (other than backups and rebuilding url.db). You could modify sub build_update_newpop in nph-build.cgi to do what you want without any other program being needed.

That subroutine reads every record in links.db to do its job. That means, every field of every record is available to you to determine what category pages need to be updated and which ones do not need updating.

You could build an array of categories that need updating and, when done, use it, along with the paths in categories.db, to rebuild pnly those category pages that have changed, along with the Home (if you display the total number of available links on it), New, Cool, and Top Rated pages.

I don't see why you think a seperate program is necessary. If you are going to modify Links, then I recommend you do it IN the Links scripts and avoid using other programs to do what can be done in Links.

Just a few thoughts, for what they are worth.

[This message has been edited by Bobsie (edited March 07, 1999).]

This is excatly what i want to do.
Only building that pages, which changes when new links was added. This are the New-page, Cool-page, the category page with this link and maybe the pages of alternate categories.

but before it is possible to work with this feature, we need to have another way to view the hits of every link, otherwise we must build all pages again.

i've found a programm which will do this, it's a hitcounter for a homepage - with some modifications it's possible to use this for this problem. so you always will have the actual number of hits for every link without rebuilding the pages :-)

when i'm ready with this mods, i'll post a message in this forum.

cu

------------------
www.ford-network.at

Hye Bobsie,

you're wrong at all - sorry.

the problem ist, that links only show the actual number of hits for a link, if you've first re-builded all pages (!). or say all pages on which are links which has got some hits.

you know what i mean?
so we need a way to change the number of hits (for a link) every time the link get's a hit.

but you're right, link updates the links before building the pages. but this doesn't help us cause if we would use this, we must rebuild all pages.

so, i realy hope you could understand my english.

nice greetings from austria
cu



------------------
www.ford-network.at

Hello Bobsie!

Bogus is very correct. In other words, if Bogus did not get it through in english as he says, the pages that are "Already" built are static pages and there is NO WAY you are going to tell me that he can modify IN the links. Once they are built they are going to remain the same for further "Worships" by the surfers.

So what he triggers is "Not Bad"!!!

Bogus, glad that you struck that idea. Keep it up!!!












[This message has been edited by rajani (edited March 07, 1999).]

Sorry for the double posting of the same message. In a hurry!

[This message has been edited by rajani (edited March 07, 1999).]

Bogus, can I take a look at that script you've found? May be I'll be helpful in some way with this project.

------------------
webmaster@find.virtualave.net
http://find.virtualave.net

Bogus,

Okay, I think I misunderstood. I thought you wanted to find a way of updating only the pages that changed when rebuilding the pages. Now I see that you want to update the page whenever a link is hit (i.e., realtime updating).

A few thoughts on this whole project:

First, if you anticipate eventually having 1,000,000 links to update, you need something other than Links to maintain it with. Otherwise, your server is going to be so bogged down doing nothing more than updating links, everything else is going to really crawl. This is assuming that anyone would be able to display a page at all or, if they could (and it would come in very slowly), clicking on a link would still take a long time to execute jump.cgi.

Second, Links is not designed to handle anything that large which even Alex would say (and has said). Even with SQL, the overhead would be tremendous.

My comments were in reference to rebuilding only those pages that have changed since the last rebuild (which I erroneously thought was your intention). That, I think, is a better consideration than trying to do realtime updating. This would benefit a much larger number of Links administrators.

These are just my thoughts and opinions. They are not meant to discourage you from your project. I wish you good luck with it.

[This message has been edited by Bobsie (edited March 08, 1999).]

Hi guys,
I was thinking about this subject for a long time and I haven't found a 100% solution for the problem... Right now, my links db has 115.000 links. I started to build a modified version of links when my db was with 90.000 links.
My first modification was to create sql version of the search procedure. It really makes a big diference.
On my second modification, I've tried to change the build routine in Links to:
first: Create the sql table and insert the links before everything.
Second: Use the bdi (mysql or msql) to query the data and build all the pages. It was really cool, because I won 40 seconds in each page (When I have 90.000 links).

Well,
I've stopped to modify links script, because I think that when we are working with a big db, we need to develop another script, based only in sql (to add, modify and ...) and not only use a sql mod when searching.
We can improve our search time using a search routine written in C that queries on a sql table.

Regards,
Ulysses Moraes

hye,

I've found a way to display the actual number of hits for each link. take a look at www.ford-network.at

pasha, to which script would you take a look?

to umoraes: I'm very intersted on your modifications. at the moment i haven't sql access and also not in the near future. so i couldn't use this way at the moment. but i've also thinked about it (sql).
you think that peoples with big db's should use c. i also think this. but i don't be familar with c and i don't would learn it at the moment - cause now it's perl time.

once when I'll have more then 200.000 links and all the work goes realy slow, i'll think about c and may be i'll programm a c version of Language links.

---
Does anybody know some information about XBM images?

cu

------------------
www.ford-network.at

could be my bad english or you misunderstood me again :-(

i don't realy know what you mean with 'realtime updating'..??
i wouldn't update the pages everytime a link gets a hit, no! let me explain:

1. you install links
2. you add some links to the .db
3. you build all pages
4. if you get new links, you re-build only that pages which must be changed.

you understand? the only time when you need to 'build all' is when you've started with links or if you've changed some of the html-code.

a hit wouldn't rebuild that page. nothing. i only load a counter for this link :-)
this work is already done. some modifications to a counter-script, and now it only counts if the link was clicked. otherwise the script only displays the actual number of hits.

on my homepage you could take a look - this feature already works (must do some modifications to the image).

but you're right: once anyone has nearly 1.000.000 links, 'Links' wouldn't do good work. this would be the time for me, to learn C and write my completely own 'Links' in C. But for now 'Links' is enough (cause i think i'll get only - may be - 100.000 links and up within the next year.

LAST: i hope you could understand me, cause i don't know what i've written :-)

------------------
www.ford-network.at

Bogus, I mean the counter script that you talked about in your second posting in this topic. You said, that this code might help us with the problem we discuss here.

So, how far did any one get in this?

Regards,

Pasha

------------------
webmaster@find.virtualave.net
http://find.virtualave.net

the script i'm using for the counter you'll find at
http://www.GetCruising.com
it's called HITMAT.

I've done some modifications: the script only counts a hit if it's called from jump.cgi.
otherwise it only displays the actual number of hits.

also the original script have a 'bug': when you was clicking on a link, the script counts a hit, if you clicked the link again, the script also counts another hit. I now use the routine from 'Links' to check if this is the same user... (i hope you'll understand what i mean)

the only help i need for this script: someone which could help me changing the look of the numbers. (XBM image)

next time i would include the 'hitmat' functions much better to LLinks. so we don't have do had the number of hits in the links.db, and some other usefull stuff :-)

cu

------------------
www.ford-network.at

So, your script will actally access 'links.db' file? If so, then how can it be done, if your 'links.db' file is located in password protected directory, but your script is not (because users will be accessing it). And, if you move your 'links.db' file out of password protected directory, then the data in this file could be stolen or possible damaged.
If this is the case, then I have a little suggestion: To move 'links.db' file to the unaccessable (with browser) directory, for example /usr/hidden or /usr/private. The script is running on server, so it doesn't matter where your database is located; and it's safe to store your database in the directory, which is not accessable by any browser or "an outsider program" like 'Black Widow' or 'SuperBot'.

I am not a good writer, but I hope you know what I mean


Regards,

Pasha

------------------
webmaster@find.virtualave.net
http://find.virtualave.net

short: no, the script wouldn't access the links.db! links.db is always in a protected directory. only the counter-file is in another directory.

cu

------------------
www.ford-network.at

If the counter will not access 'links.db' file, then I don't understand, where the counter will get the data, which it will count/update. Second database?

Regards,

Pasha

------------------
webmaster@find.virtualave.net
http://find.virtualave.net

yes pasha,

second database.

but at the moment, if you click build all, links will also update the links.db and clear the hits directory.

i think in the future i'll use the hits field in the links.db for 'hits of the day' or something else.

------------------
www.ford-network.at

So, you actually want the build/script to update two databases at the same time, one for 'Links 2.0' and second for counter/script? If yes, then again, your second database will be open to the public, if you don't hide it or password protect it.

Ok, I'm lost here!
Why would you want to clear the hits, if you could just update them?

------------------
webmaster@find.virtualave.net
http://find.virtualave.net

Bogus,

Sorry for not replying to your post a bit earlier. For some reason, I couldn't get a response from the Gossomer Threads server for over 24 hours. Seems like I was the only one having that problem based on the number of replies since I last replied.

Anyway, 'realtime' means when a link is hit, the page would rebuild immediately afterwards to increase the hit counter that is displayed. Since this is not what you want to do, based on your latest clarification, then my original thoughts on this project apply.

If you are going to rebuild only the pages that need updating rather than all of the pages, all you would need to do is like I said originally. Build an array or file that is created in the section of nph-build.cgi that updates the hits and/or ratings. The array/file would track the category and other pages that need updating. Then, a special "build" routine could be written to use that array/file to only rebuild the pages that actually have changed instead of building all of them. You would end up with 4 options: Build All, Build Changed, Build Staggered, and Build Staggered (Auto). Again, I do not see any need for additional scripts to maintain, just add additional routines or modify current routines using the current scripts.

Again, these are just my thoughts and opinions; I am not trying to talk you out of anything.

[This message has been edited by Bobsie (edited March 09, 1999).]

Bogus,
About the passwords, forgive me, I was wrong.
I just found that the CGI scripts can access database files even in password protected directories. I didn't know this before

And, after thinking about your project I figured that you don't need to create a second database, if the script can actually access everything and anywhere on the server. You just need to point your new script to the original 'links.db' file in your /admin/data/ directory, and set the script to use/update just one field ($hits). If this is the case, then there's not much of a job to do, and a simple counter/script would be just fine. Just include the script with SSI in every page in 'Links', so it will run automatically when some one visits your site. This way hits will always be updated, and the script will not be running at the time, when it shouldn't be.

Let me know, what you think of this.

Regards,

Pasha

------------------
webmaster@find.virtualave.net
http://find.virtualave.net

hye all

hmmm... where should i beginn?

bobsie: you're right. i mustn't build another script for this routine. BUT, why don't make another script. i've thought i will do it this way: when a link was validated, it would be written to links.db and a new_links.db. and then build the pages which changed out from new_links.db.
I'll think about how to do this feature again when i've time.
You're not right with 4 build options. I write for Language Links and then there are only 3 options: Build All, Delete All, Build changed.

Pasha: you're right, i mustn't have 2 databases with the hits for a link. but note: links.db have 18398 KB and the hit.cnt has 335 KB. you'll see, using another hits database would be much faster.
About SSI: i know this would be the easiest way to do this feature. But i wouldn't use SSI for this, cause it could be, that some of the users servers don't supports SSI. And also: if i doing it this way, linkowners would be able to include there counter with an img-tag on there own homepage.

ok, must go
cu

------------------
www.ford-network.at

Agree, SSI is not a good choice, but you still have to update hits regularly. How about: when the user clicks on the link, 'jump.cgi' calls your counter/script to add another hit.

------------------
webmaster@find.virtualave.net
http://find.virtualave.net