Site Downtime and a Drastic Solution (discussion thread)

edited July 18 in Forum Support
Hey y'all.

You may or may not have noticed that theorangebelt.org was offline for most of the day yesterday (1/14/2013). After working with Dreamhost to correct the problem, this was the reply I received:
Hello,

Thank you for writing. Our data center team was investigating stability
issues on this shared server and determined the high load caused by the
site theorangebelt.org. Per http logs, the site appears to average 250k
hits daily.

234024 access.log
249035 access.log.0
249035 access.log.2013-01-13
321689 access.log.2013-01-12

Most of the traffic is to the /forums section of the site.

As a result your user "theorangebelt" is going over the memory limits on
the shared server. Please understand that while you may not be hitting
your disk space or bandwidth limits, these are actually entirely
different things. If we didn't limit the amount of memory our customers
were using on the server, then one customer could bring the system down
completely if they so desired.

The site has been re-enabled at this time and will be accessible shortly.
I would highly recommend that you follow the steps in the following wiki
article in order to reduce your usage:

http://wiki.dreamhost.com/Slow_site_troubleshooting

http://wiki.dreamhost.com/Finding_Causes_of_Heavy_Usage

http://wiki.dreamhost.com/Fine_Tuning_Your_WordPress_Install

http://wiki.dreamhost.com/WordPress_Optimization

If the above is not an effective option, you will want to look into
DreamHost VPS as the site has outgrown shared hosting:

http://dreamhost.com/servers/vps/
https://panel.dreamhost.com/index.cgi?tree=vserver.provision&

Please let me know if you have further questions about any of this.

Miguel R,

--
- DreamHost Abuse/Security Team

As I'm sure you're all aware, our community and daily activity on the forums have both shrunk considerably over the years, so I don't think anything we're doing could be contributing to this memory usage increase. My only guess is that our frequent scourge -- the spambot -- is to blame. Looking at our users, I can see that we haven't had a real human register an account in quite some time. Almost all our user registration is from automated spambots trying to (albeit sometimes hilariously) sell us products through forum posts and private messages. This has sadly made our site a liability on the shared Dreamhost server and puts us at risk of being disabled.

Starting today, I am completely disabling the registration of new accounts and denying all webcrawler traffic.

My hope is that this will eventually decrease the amount of spambot-related traffic we're receiving. It's unfortunate that this will also effectively block us from Google searches, but the most important thing to me is to retain this small community for whoever still participates in it.

If you would like a new account created, send me a PM with the username and email address and I'll put it together for you.

Let me know if you have any questions or suggestions on how we might address this issue in the longterm. Otherwise, keep on being awesome!

~mario

Comments

  • edited January 2013
    It's our own little corner of the internet.

    Disabling new registrations makes me a little sad, but functionally it's no different than how we've already been the last few years.
  • edited January 2013
    And if anyone does want to bring friends in (we're pretty cool, can't blame you for wanting others to be cool-adjacent), I'll be happy to create accounts for them.

    Also, holy crap where is all this spambot traffic coming from?
  • edited January 2013
    So would new accounts be handled in a "Please make this account for my friend" thread? Or would you prefer private messages instead? (Not that I have anyone to recommend to join, for now anyways)
  • edited January 2013
    Private messages would probably be best.

    On the bright side, this means I can lift our secret "you can't make threads until you've made ten posts to prevent spambots from making threads" block that I, uh, probably just mentioned right now for the first time? I wanted to make sure spambot creators wouldn't catch on, but now it doesn't matter yay!

    yayyyy
  • edited January 2013
    Nah, you've mentioned it before. Don't remember where though.
  • edited January 2013
    I thought new users could literally not post at all unless approved? I'd agree that we may as well close automatic registration right now.

    It's the most passive-aggressive robot invasion ever.
  • edited January 2013
    It's completely clear to me, no one cares about the site until you take away the free all access. Then EVERYONE wants an in.
  • edited January 2013
    "Yo dog, did you hear about that forum that shut down automatic registration?"

    "Yes, friend, I believe it was called The Orange Belt and I totes want to get on that."

    That's the conversation happening all across the internet right now.
  • edited January 2013
    Holy crap, traffic is spiking higher than ever in the forum's history. Are bots attracted to closed forums??

    I also updated robots.txt, but that's only going to keep out honorable bots. Really wish I knew how this forum became such a hot target for bots. Maybe it's the outdated version of vBulletin we're running? In any case, registration is disabled and bots are mostly blocked, so I assume traffic will eventually die down.
  • edited January 2013
    Now I'm playing a FUN GAME where I just keep finding more IP addresses to block. THERE ARE SO MANY YOU GUYS
  • edited January 2013
    Worst case scenario, you gotta find a way to block "guest" users from even accessing more than a login page. Ultimate exclusivity.
  • edited January 2013
    Block every IPv4 IP and only allow IPv6 in. Then buy everyone upgraded IPv6 capable hardware.

    I feel I should point out however that my lack of IPv6 goes down to ISP level, so it could get pricey...
  • edited January 2013
    The problem is that, even if you can't access those pages, you can still attempt to access those pages. The forum software will bring up some sort of error page, but it uses memory to acknowledge the request, and with so many IP addresses making the requests, it adds up to a lot of traffic. I've resorted to blocking IP addresses via the .htaccess file, but have no hope of keeping up. I'm going to take a break from IP blocking for now, but the traffic is getting ridiculously excessive.

    Is someone actually attacking us? It's the only conclusion I can draw; the influx of bots seems to directly correspond to these security changes.