WG vs Bots-Spyders

[Stoneslinger76]

Hi I am probably not the only one haveing web search bot's causeing some troubles. Some helpfull info for your robots.txt
This should help to keep some of the nasty asian search bots and spyders from beating the heck out of you.

Some additional information on robots.txt here
http://searchenginewatch.com/article/20 ... -Parasites

You may want to edit your domain name and add this to your robots.txt

User-agent: Baiduspider
User-agent: Baiduspider-video
User-agent: Baiduspider-image
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: moget
User-agent: ichiro
Disallow: /
User-agent: Naverbot
User-agent: Yeti
Disallow: /
User-agent: YoudaoBot
Disallow: /
User-agent: google,yahoo,*
Allow: /index.htm
Allow: /twindex.htm
Allow: /majormud.htm
Allow: /about.htm
Allow: /game.htm
Allow: /news.htm
Allow: /help.htm
Allow: /sitemap.xml
Disallow: /forums/
Disallow: /library/

Disallow: /email/
Sitemap: http://theswampbbs.net/sitemap.xml

[dspain]

Hi I am probably not the only one haveing web search bot's causeing some troubles. Some helpfull info for your robots.txt
This should help to keep some of the nasty asian search bots and spyders from beating the heck out of you.

Some additional information on robots.txt here
http://searchenginewatch.com/article/20 ... -Parasites

You may want to edit your domain name and add this to your robots.txt

User-agent: Baiduspider
User-agent: Baiduspider-video
User-agent: Baiduspider-image
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: moget
User-agent: ichiro
Disallow: /
User-agent: Naverbot
User-agent: Yeti
Disallow: /
User-agent: YoudaoBot
Disallow: /
User-agent: google,yahoo,*
Allow: /index.htm
Allow: /twindex.htm
Allow: /majormud.htm
Allow: /about.htm
Allow: /game.htm
Allow: /news.htm
Allow: /help.htm
Allow: /sitemap.xml
Disallow: /forums/
Disallow: /library/

Disallow: /email/
Sitemap: http://theswampbbs.net/sitemap.xml

yeah seems WG went backwards in their internet stuff, i dont use it really, my newuser email verifrication module uses it cause it has to but only enough to connect to my smarthost powered by synchronet and do the email, the web server is a little buggy but the ftp server is crashable, and if someone dont like ya they can make a script to keep your system offline permanently til you disable the ftp server altogether, theres a few other exploits too in the http server i sent the stuff to gcomm many years ago in hopes to fix it, just like their spr function, buggy as hell, when gcomm got taken over by whomever did it in 1998 they really dropped the ball.

[Stoneslinger76]

yeah seems WG went backwards in their internet stuff, i dont use it really, my newuser email verifrication module uses it cause it has to but only enough to connect to my smarthost powered by synchronet and do the email, the web server is a little buggy but the ftp server is crashable, and if someone dont like ya they can make a script to keep your system offline permanently til you disable the ftp server altogether, theres a few other exploits too in the http server i sent the stuff to gcomm many years ago in hopes to fix it, just like their spr function, buggy as hell, when gcomm got taken over by whomever did it in 1998 they really dropped the ball.

The robots.txt helps keep some of the more nasty search bots away, the ones from Asia can be very nasty and quickly knock wg offline. The changes to robots.txt seems to have helped greatly.

It would be nice to see some improvements to wg's web stuff, one being Authentication mode added to smtp for use with most smtp mail servers like dyndns or most isp's use, along with non standard smtp port setting.

[Ragtop]

Dspain's latest smtp server allows you to switch the sending port, but does not have authentication. If he can add that, all my email problems go away

[dspain]

Dspain's latest smtp server allows you to switch the sending port, but does not have authentication. If he can add that, all my email problems go away

yeah im looking at some other smtp code on how to pass the authentication strings and what type to use.

i am having some success, i think im gonna keep it to just standard logon authent.

[Ragtop]

Dspain's latest smtp server allows you to switch the sending port, but does not have authentication. If he can add that, all my email problems go away

yeah im looking at some other smtp code on how to pass the authentication strings and what type to use.

i am having some success, i think im gonna keep it to just standard logon authent.

Let me know if you want me to test it out. Look forward to having email working again.

[dspain]

Dspain's latest smtp server allows you to switch the sending port, but does not have authentication. If he can add that, all my email problems go away

yeah im looking at some other smtp code on how to pass the authentication strings and what type to use.

i am having some success, i think im gonna keep it to just standard logon authent.

Let me know if you want me to test it out. Look forward to having email working again.

cool. i got outgoing to work i had to make it work so i could get my newuser email verification addon to work but incoming is still sketchy due to authentication issues.
im working on that now, by looking at some other win32 smtp server sources to see how they are parsing authentication info to the server.
i'll get it.

[Ragtop]

yeah im looking at some other smtp code on how to pass the authentication strings and what type to use.

i am having some success, i think im gonna keep it to just standard logon authent.

Let me know if you want me to test it out. Look forward to having email working again.

cool. i got outgoing to work i had to make it work so i could get my newuser email verification addon to work but incoming is still sketchy due to authentication issues.
im working on that now, by looking at some other win32 smtp server sources to see how they are parsing authentication info to the server.
i'll get it.

Sounds great! I am mostly concerned with outgoing for the same reason, email account verification. But incoming would be a nice luxury too. Hope you get it working soon!

[Stoneslinger76]

Dspain's latest smtp server allows you to switch the sending port, but does not have authentication. If he can add that, all my email problems go away

yeah im looking at some other smtp code on how to pass the authentication strings and what type to use.

i am having some success, i think im gonna keep it to just standard logon authent.

Cool, some form of authentication would be great. Hopefully it will work with dyndns or Ma Bell's smtp.

[dspain]

Dspain's latest smtp server allows you to switch the sending port, but does not have authentication. If he can add that, all my email problems go away

yeah im looking at some other smtp code on how to pass the authentication strings and what type to use.

i am having some success, i think im gonna keep it to just standard logon authent.

Cool, some form of authentication would be great. Hopefully it will work with dyndns or Ma Bell's smtp.

im testing using comcast which are very lax in their smtp stuff, you can run email servers whatever.

right now im going with plain authentication which is a simple connect to your isp in for us it will be the "smarthost" sending a login name and password.

one of my test servers uses synchronet as a smarthost which connects to smtp.comcast.net using my login stuff and sends the message, for testing its a newuser email with a verification code which has been working flawlessly.

problem is i want to do away with the smarthost and have wg connect directly to the isp and do the work.
for receiving from what i see no authent needed just the ability to receive and process which the galsmtp.dll does anyhow.

its a work in process.

[Stoneslinger76]

im testing using comcast which are very lax in their smtp stuff, you can run email servers whatever.

right now im going with plain authentication which is a simple connect to your isp in for us it will be the "smarthost" sending a login name and password.

one of my test servers uses synchronet as a smarthost which connects to smtp.comcast.net using my login stuff and sends the message, for testing its a newuser email with a verification code which has been working flawlessly.

problem is i want to do away with the smarthost and have wg connect directly to the isp and do the work.
for receiving from what i see no authent needed just the ability to receive and process which the galsmtp.dll does anyhow.

its a work in process.

The recieving works without any troubles in WG from dyndns and also recieves many other unkown servers looking to relay mail (relay mail disabled). My isp's techies even call it "the smtp black hole" and relay some traffic they cant stop to theswampbbs while they hunt down the source of the spammer.
It would probably be best to have wg look after authentication rather than have smarthost use manditory. Smarthost options might allow some sysops their one and only avenue to webmail depending on thier isp or domain name relay host. A variable port for inbound and outbound would also be great. I see many dyndns type domain name relay host's incorpeating optional ports in thier smtp mail relay packages to avoid isp port restricitions.

My current mail options with dydns are nearly expired, they have a new smtp package which allows for unlimited users names and about 150 relays a day free (yes they stick you extra per message and per each relay over if you have a spammer). Dyndns.com's new smtp package is only 20 bucks a year much less than thier past offerings for smtp.
Would it be possible while your codeing to add a MAX Number of outbound sends per day and maybe max send by a single user per day to avoid additional cost's for over useage per day?

[dspain]

im testing using comcast which are very lax in their smtp stuff, you can run email servers whatever.

right now im going with plain authentication which is a simple connect to your isp in for us it will be the "smarthost" sending a login name and password.

one of my test servers uses synchronet as a smarthost which connects to smtp.comcast.net using my login stuff and sends the message, for testing its a newuser email with a verification code which has been working flawlessly.

problem is i want to do away with the smarthost and have wg connect directly to the isp and do the work.
for receiving from what i see no authent needed just the ability to receive and process which the galsmtp.dll does anyhow.

its a work in process.

The recieving works without any troubles in WG from dyndns and also recieves many other unkown servers looking to relay mail (relay mail disabled). My isp's techies even call it "the smtp black hole" and relay some traffic they cant stop to theswampbbs while they hunt down the source of the spammer.
It would probably be best to have wg look after authentication rather than have smarthost use manditory. Smarthost options might allow some sysops their one and only avenue to webmail depending on thier isp or domain name relay host. A variable port for inbound and outbound would also be great. I see many dyndns type domain name relay host's incorpeating optional ports in thier smtp mail relay packages to avoid isp port restricitions.

My current mail options with dydns are nearly expired, they have a new smtp package which allows for unlimited users names and about 150 relays a day free (yes they stick you extra per message and per each relay over if you have a spammer). Dyndns.com's new smtp package is only 20 bucks a year much less than thier past offerings for smtp.
Would it be possible while your codeing to add a MAX Number of outbound sends per day and maybe max send by a single user per day to avoid additional cost's for over useage per day?

max per day would be easy just store a variable that is reset at cleanup.

ie:

int sendmail = 0;

during send:

if(sendmail < maxsend)
{
blah.
++sendmail;
}

have it in the modules cleanup function so it wont be reset
if the server just goes down or crashes, it would have to be a system
cleanup to reset it, and store it in a small database maybe galsmtp.dat
so if the system unexpectantly goes down when t comes back up it remembers the send count.

[Stoneslinger76]

have it in the modules cleanup function so it wont be reset
if the server just goes down or crashes, it would have to be a system
cleanup to reset it, and store it in a small database maybe galsmtp.dat
so if the system unexpectantly goes down when t comes back up it remembers the send count.

Looking forward to completion as are a few other sysops.

[Ragtop]

Hey Dan,
Any progress on this? Looking forward to email validator working

[dspain]

Hey Dan,
Any progress on this? Looking forward to email validator working

i got it working enough for my email validator as far as the sending,
still having an issue with the authentication strings though without using a smarthost.

what i do is use another server as my smarthost, my synchronet ftp server was perfect cause it has an smtp server.
i have wg send the email to synchronet who then dispatches it to the internet.
tested 5 new accounts each new account got its own unique 10 digit email activation code sent to their email address when the code was entered wg switched the new accounts to the verified account class and allowed them to exit the newuser menu.

what i cannot get it to do is send the email from within wg alone (yet) and
cannot receive, but im closer on that now its just a wg error.

you probably use the hvs validator am i right?

[Ragtop]

Yes, I'm using HVS Autovalidator. It does fine generating codes for them but I have to manually log on to my email account to send them. Guess I can look at setting up synchronet as a smarthost, but would rather not have to do that.

Tried using a freeware program call Stunnel which, among other things, is supposed to be able to intercept a connection and forward it with authentication to whatever destination you define. Haven't been able to get it to work though. Config docs are a little sketchy so I'm sure it's something I'm not setting up right. This would be a good answer since it can run in the background on the same machine and just grab the WG connection and forward on to the ISP mail server. You may want to look into this instead of trying to rewrite the smtp server.

[dspain]

Yes, I'm using HVS Autovalidator. It does fine generating codes for them but I have to manually log on to my email account to send them. Guess I can look at setting up synchronet as a smarthost, but would rather not have to do that.

Tried using a freeware program call Stunnel which, among other things, is supposed to be able to intercept a connection and forward it with authentication to whatever destination you define. Haven't been able to get it to work though. Config docs are a little sketchy so I'm sure it's something I'm not setting up right. This would be a good answer since it can run in the background on the same machine and just grab the WG connection and forward on to the ISP mail server. You may want to look into this instead of trying to rewrite the smtp server.

well the thing with those programs is even if you disable smtp through wg it still uses the port, it does that so it can send a rejection message.

the easiest way to make one of those work on the same machine as wg is compile galsmtp again with port options, for testing set it to 2525
now wg will initialize on port 2525 and you can install a secondary program on the same machine using port 25.
then have your program read it using port commands:

192.168.1.100:2525 etc.....

thats how i did the workaround with synchronet door servers running on the same machine as wg.

i'm not really rewriting the smtp stuff just tracing how it communicates with the smtp server of your isp so i can send authentication strings.

like with me i use smtp.comcast.net using authentication type 'Login" then use my comcast.net login name and password.

boom sends email just fine, but the wg smtp stuff doesnt have that option
it was written when everyone though the internet was safe and you coulod do direct email delivery methods through your dns server which all isps block now.

as soon as i get tradewars II testable, and this other module i just got compiled, i will tinker with it again.

[Ragtop]

Yes, I'm using HVS Autovalidator. It does fine generating codes for them but I have to manually log on to my email account to send them. Guess I can look at setting up synchronet as a smarthost, but would rather not have to do that.

Tried using a freeware program call Stunnel which, among other things, is supposed to be able to intercept a connection and forward it with authentication to whatever destination you define. Haven't been able to get it to work though. Config docs are a little sketchy so I'm sure it's something I'm not setting up right. This would be a good answer since it can run in the background on the same machine and just grab the WG connection and forward on to the ISP mail server. You may want to look into this instead of trying to rewrite the smtp server.

well the thing with those programs is even if you disable smtp through wg it still uses the port, it does that so it can send a rejection message.

the easiest way to make one of those work on the same machine as wg is compile galsmtp again with port options, for testing set it to 2525
now wg will initialize on port 2525 and you can install a secondary program on the same machine using port 25.
then have your program read it using port commands:

192.168.1.100:2525 etc.....

thats how i did the workaround with synchronet door servers running on the same machine as wg.

i'm not really rewriting the smtp stuff just tracing how it communicates with the smtp server of your isp so i can send authentication strings.

like with me i use smtp.comcast.net using authentication type 'Login" then use my comcast.net login name and password.

boom sends email just fine, but the wg smtp stuff doesnt have that option
it was written when everyone though the internet was safe and you coulod do direct email delivery methods through your dns server which all isps block now.

as soon as i get tradewars II testable, and this other module i just got compiled, i will tinker with it again.

Not exactly what I meant. With stunnel, you wouldn't disable smtp. It is simply a utility that sets up a secure tunnel on your machine. It listens on whatever port you tell it for smtp traffic, then forwards it to whatever address:port you want it to go to. Here is a portion of the config file to better explain.
-------------------------------------------
[smtp]
protocol = smtp
accept = 25
connect = 587
protocolHost = {my isp smtp server}
protocolUsername = {my user name}
protocolPassword = {my password}
-------------------------------------------
This listens for smtp traffic on port 25, forwards it to {my isp smtp server} on port 587, using my login information for authentication. Theoretically, this should work, but there must be something else in the config that I don't have set up correctly.

[dspain]

Yes, I'm using HVS Autovalidator. It does fine generating codes for them but I have to manually log on to my email account to send them. Guess I can look at setting up synchronet as a smarthost, but would rather not have to do that.

Tried using a freeware program call Stunnel which, among other things, is supposed to be able to intercept a connection and forward it with authentication to whatever destination you define. Haven't been able to get it to work though. Config docs are a little sketchy so I'm sure it's something I'm not setting up right. This would be a good answer since it can run in the background on the same machine and just grab the WG connection and forward on to the ISP mail server. You may want to look into this instead of trying to rewrite the smtp server.

well the thing with those programs is even if you disable smtp through wg it still uses the port, it does that so it can send a rejection message.

the easiest way to make one of those work on the same machine as wg is compile galsmtp again with port options, for testing set it to 2525
now wg will initialize on port 2525 and you can install a secondary program on the same machine using port 25.
then have your program read it using port commands:

192.168.1.100:2525 etc.....

thats how i did the workaround with synchronet door servers running on the same machine as wg.

i'm not really rewriting the smtp stuff just tracing how it communicates with the smtp server of your isp so i can send authentication strings.

like with me i use smtp.comcast.net using authentication type 'Login" then use my comcast.net login name and password.

boom sends email just fine, but the wg smtp stuff doesnt have that option
it was written when everyone though the internet was safe and you coulod do direct email delivery methods through your dns server which all isps block now.

as soon as i get tradewars II testable, and this other module i just got compiled, i will tinker with it again.

Not exactly what I meant. With stunnel, you wouldn't disable smtp. It is simply a utility that sets up a secure tunnel on your machine. It listens on whatever port you tell it for smtp traffic, then forwards it to whatever address:port you want it to go to. Here is a portion of the config file to better explain.
-------------------------------------------
[smtp]
protocol = smtp
accept = 25
connect = 587
protocolHost = {my isp smtp server}
protocolUsername = {my user name}
protocolPassword = {my password}
-------------------------------------------
This listens for smtp traffic on port 25, forwards it to {my isp smtp server} on port 587, using my login information for authentication. Theoretically, this should work, but there must be something else in the config that I don't have set up correctly.

[reply]
yeah but it all comes back to we want WG to handle it, WG wouldnt know what to do with the stunnel traffic.
i want wg to send it, wg to receive it, smarthosts are ok but eliminates
the point, and not to rattle any cages here but gcomm should have known
better than to charge $9,995.00 for a piece of software that was this out dated and yes i am referring to the 1998 gcomm not the 2010.
in 1998 3.0 had this same exact smtp setup yet freeware packagesl ike synchronet were piping out industry standard internet packages.
[/reply]