Future of the Forum

[icerat]

Originally Posted by Belz...

Yes but I assumed that, since the JREF has a forum, it has people that take care of it and can identify problems.

Very well might have when the forum was started. That's part of the problem with volunteer driven resources - when someone with a passion for a particular area leaves for whatever reason, it's difficult to sustain what they were doing. One of the things I do is help people design projects from the ground up that will be sustainable without the passion of the founder.

[This is The End]

Originally Posted by icerat

That will mean lots of whining from people who like things the way they are.

If you fix the performance issue there is absolutely no reason for a change in software.

Anyone attempting to change the software would be almost guaranteed to give up on preserving the current degree of access to the entire thread history. They might think they can pull it off at first, and then they will likely realize they cannot, but by that time they will be married to the idea and we will lose access. The thread history will be archived only from that point on.

Being able to find a thread from 2004 that matches our current format, and even be able to bump the thread if need be, is what I like about this forum.

Please do not fix things that are not broken.

[icerat]

Originally Posted by OnlyTellsTruths

If you fix the performance issue there is absolutely no reason for a change in software.

Anyone attempting to change the software would be almost guaranteed to give up on preserving the current degree of access to the entire thread history. They might think they can pull it off at first, and then they will likely realize they cannot, but by that time they will be married to the idea and we will lose access. The thread history will be archived only from that point on.

Being able to find a thread from 2004 that matches our current format, and even be able to bump the thread if need be, is what I like about this forum.

Please do not fix things that are not broken.

While in principle I agree, the reality is that the forum is broken. There's severe performance issues that need addressing. While it can probably be temporarily addressed with hardware upgrades, the is reality things become "broken" from a user perspective anyway - more than half of internet traffic in the US is now on mobile devices, and most websites typically have around a third or more of their visitors from phones or tablets. If you don't cater to that audience, you lose them.

Then you have issues with version creep. The OS version this forum runs on is no longer being updated and in less than 3 years won't even receive security updates. You have similar issues with the forum and database software. Often upgrading one will break one of the others without also upgrading there (and more often than not something will break even then!)

Most forum software has the same general structure, it's a matter of being able to find things in the same place, having the same kind of functionality (tags for embedding youtube videos for example) etc that make something familiar. Nothing I hate more than when (for example) Facebook just changes things and they're not where you expect.

So I hear you!

Ideally we'll setup a mirror test version for people like yourself to use before the site software is upgraded. Time and funding limitations apply of course, but hopefully we can do it properly.

[SezMe]

WARNING: If you break post counts, the whining will break every whine meter ever built.

[varwoche]

Originally Posted by icerat

it's almost certainly a mySQL bottleneck

However, RAM and/or disk bottleneck could be causing db bottleneck.

[varwoche]

Originally Posted by icerat

ugh, performance has been disastrous for me the past half hour ... peak time?

Not for jref, but maybe for co-tenants.

[varwoche]

Originally Posted by icerat

My current favourite is Xenforo, which was developed by the original vBulletin lead developers. One thing I think is important is native support for different screen sizes (eg cell/tablet). All suggestions appreciated.

Fortunately...

Originally Posted by xenforo

XenForo has the ability to import data from other forum software, a process also known as converting. Version 1.2 of XenForo supports importing data from the following:

vBulletin 3.6, 3.7, 3.8, and 4.x
...

[rjh01]

I am willing to help. Though about the only job I might have experience in is testing. If you give me something untested I can break it.

[Slowvehicle]

Originally Posted by icerat

Ok,

I've got some hard details on the current server and performance and as most expected it's almost certainly a mySQL bottleneck. I'm pretty confident we can get things up on running well for half the cost JREF is currently paying, and then it's just a matter of $$$ to improve performance even more (SSD vs SATA more cache etc etc).

Put your hand up if you have experience with -

* CentOS
* mySQL
* vbulletin 3.7.7

And are willing to contribute some time.

The current goal would be to move the forum as is to new hardware to stabilise performance, and then at a later date upgrade the forum software itself. Suggestions and experiences on the latter are requested. vBulletin as a project has gone downhill so it will most likely mean migrating to other commercial or opensource forum software. That will mean lots of whining from people who like things the way they are

My current favourite is Xenforo, which was developed by the original vBulletin lead developers. One thing I think is important is native support for different screen sizes (eg cell/tablet). All suggestions appreciated.

The other issue is of course funding. I suspect JREF has little interest in continuing to financially support the forum so we'll have to look at a donor/advertising/whatever model. Again, your input appreciated.

...I will not pretend to expertise, but I do have some time, and enthusiasm. Holler if you ever need cannon fodder (or good cookie recipes).

[idoubtit]

I suppose it's OK to release details. I'm trying to be conscientious here but I'm not that worried about hackers.

There has not been dedicated staff to deal with the forum or website. OBVIOUSLY.

There WILL be change. To hope that every nook, cranny and post count remains the same might be a pie in the sky. What do you want? Improvement. Sometimes it means resetting the odometer and starting from scratch.

Life goes on.

[Christian Klippel]

Originally Posted by icerat

(SSD vs SATA more cache etc etc).

Are SSD's reliable enough nowdays, given the intrinsic limitation of the number of write-cycles for the used flash memory technology?

Or are you planning to use SSD for the OS and slow-changing stuff only, and have the fast-changing stuff on regular (RAID-)drives?

Greetings,

Chris

[marplots]

My worth as a human being is directly pinned to my post count at JREF.

[Blue Mountain]

I'm familiar with CentOS 5 and 6; my day job is the system administrator for a web services company.

[Michel H]

Originally Posted by idoubtit

I suppose it's OK to release details. I'm trying to be conscientious here but I'm not that worried about hackers.

There has not been dedicated staff to deal with the forum or website. OBVIOUSLY.

There WILL be change. To hope that every nook, cranny and post count remains the same might be a pie in the sky. What do you want? Improvement. Sometimes it means resetting the odometer and starting from scratch.

Life goes on.

Thank you Sharon. You said there has not been dedicated staff to deal with the forum or website. But then who is this mysterious "person with responsibility for the site" icerat was talking about? (I hope I am not asking too much here):

Originally Posted by icerat

Well I got a response that they're collecting names of people offering assistance and it's being passed on to the person with responsibility for the site who will be deciding how to proceed.
...

Is it not reasonable to assume that, at the JREF, there is one person who is more technically-oriented and who is in charge of the site?

[jhunter1163]

Originally Posted by AdMan

Seconded. Plus I'm happy to donate to a funding effort. I have little doubt that a crowdfunding effort will be able to keep the forum going if it's needed.

I have little doubt that we could raise $10,000 in 24 hours if it'd preserve the Forum and solve the performance issues. My $100 is still on the table.

[jhunter1163]

Originally Posted by Michel H

Thank you Sharon. You said there has not been dedicated staff to deal with the forum or website. But then who is this mysterious "person with responsibility for the site" icerat was talking about? (I hope I am not asking too much here):

Is it not reasonable to assume that, at the JREF, there is one person who is more technically-oriented and who is in charge of the site?

I assume that Darat is the person who has been in charge of the technical aspects of the forum.

[Michel H]

Originally Posted by jhunter1163

I assume that Darat is the person who has been in charge of the technical aspects of the forum.

Ah OK, thank you for the clear answer, jhunter. However, Darat said, about two months ago:

Originally Posted by Darat

...

Unfortunately server and so on are outside my responsibility.

I am looking to see if there is anything else I can do on my side to increase responsiveness, I've been checking a few things and it may be that Tapatalk is adding quite a lot of overhead to the Forum but that's a very preliminary conclusion at the moment.

Isn't there another person in charge of more technical aspects at JREF (sorry for asking so much). I hope we're not in a Kafka novel here.

[Wolfman]

Originally Posted by Michel H

Isn't there another person in charge of more technical aspects at JREF (sorry for asking so much). I hope we're not in a Kafka novel here.

Basically, no. Darat can deal with software issues, but not hardware. And until Sharon, there's been nobody at the JREF offices to specifically deal with such issues.

[Donn]

It's weird how a website can look so real to some and so abstract to others. To me, the forum is JREF. I can't visit America, I'll never shake Randi's hand (I hz a sad) but I come here almost everyday to get a speck of this culture.

Good luck icerat and friends, keep the bytes of reason flowing!

[icerat]

Originally Posted by varwoche

However, RAM and/or disk bottleneck could be causing db bottleneck.

And cpu as well (see below)

Originally Posted by varwoche

Not for jref, but maybe for co-tenants.

aren't any (see below)

Originally Posted by varwoche

Fortunately...

one reason why it's currently my first choice

[icerat]

Originally Posted by Christian Klippel

Are SSD's reliable enough nowdays, given the intrinsic limitation of the number of write-cycles for the used flash memory technology?

Yes, indeed with decent brands they may be more reliable than SATA

Quote:

Or are you planning to use SSD for the OS and slow-changing stuff only, and have the fast-changing stuff on regular (RAID-)drives?

A lot depends on funding of course. Even just upgrading the server without SSD will make a noticeable difference. Anything more is icing.

[icerat]

Originally Posted by jhunter1163

I have little doubt that we could raise $10,000 in 24 hours if it'd preserve the Forum and solve the performance issues. My $100 is still on the table.

Any preference on a crowd-funding platform to support this?

[Donn]

icerat - without meaning to teach you how to suck eggs - please don't forget to cron some kind of regular backup of the db to an external location!

[icerat]

Originally Posted by idoubtit

I suppose it's OK to release details. I'm trying to be conscientious here but I'm not that worried about hackers.

There has not been dedicated staff to deal with the forum or website. OBVIOUSLY.

There WILL be change. To hope that every nook, cranny and post count remains the same might be a pie in the sky. What do you want? Improvement. Sometimes it means resetting the odometer and starting from scratch.

Life goes on.

Thanks. I think we can maintain the site pretty much as including post counts etc. Look and feel will eventually change a bit, but we can do things incrementally.

Here's the current basic server configuration. It's a dedicated box installed 5.5 years ago -

Bare Metal Server Installed 2/19/2009 in Seattle @ Softlayer
CentOS 2.6.18-92.e15
SuperMicro X7QCE Intel Xeon HexCore QuadProc Sata [4Proc]
4x2GB Generic RAM
4x2.13GHz Intel Xeon-Tigerton (7320-Quadcore)
SuperMicro AOC-SIMSO-plus Remote Management Card
2xSuperMicro PWS-1K01-1R Power Supply
SuperMicro BPN-SAS-828TQ Backplane
Adaptec 3405 Drive Controller
Western Digital Raptor 10,000 RPM WD1500ADFD (sdb) for Database
Seagate Cheetah ST373455SS [73GB] (sda) for system

It is currently hosting both the forum and main website, there are already plans to migrate the website. The main website is filtered through a firewall, which is why it has a different IP.

Here's some network statistics

In: Average 201.11 Kbps, Max 781.84Kbps
Out: Average 2.6 Mbps, max 25.2 Mbps

I assume from that it's on a 100Mbps network.

Hard drive performance -

%iostat -d 3
Linux 2.6.18-92.el5 (jref.randi.org)    08/02/2014

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              15.23       226.15       455.27   78704786  158440916
sdb             231.65      6702.42      1794.54 2332545177  624528576

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               1.67         5.33        24.00         16         72
sdb             271.00      3770.67      6261.33      11312      18784

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               4.00         0.00       253.33          0        760
sdb             273.67      3789.33      6064.00      11368      18192

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              18.33         2.67       357.33          8       1072
sdb             265.00      3882.67      5906.67      11648      17720

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               2.33         0.00        58.67          0        176
sdb             279.33      3888.00      7200.00      11664      21600

File system usage

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5             9.7G  586M  8.7G   7% /
/dev/sda8             996M   34M  911M   4% /tmp
/dev/sda7             104G   52G   46G  54% /home
/dev/sda3             9.7G  6.3G  3.0G  68% /usr
/dev/sda2             9.7G  2.2G  7.1G  24% /var
/dev/sda1              99M   12M   83M  13% /boot
tmpfs                 3.9G     0  3.9G   0% /dev/shm
/dev/sdb1              68G   33G   32G  51% /var/lib/mysql
/home/domlogs         104G   52G   46G  54% /usr/local/apache/domlogs

I thought the MySQL db seemed overly large however vbulletin 3.7 apparently stores images in the db by default. Assuming this is the current setup then moving them to the file system would probably help performance.

I have not yet got details on current backup strategy.

System Performance

%vmstat 3

%vmstat 3
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 7  0    252  86180  55660 7092544    0    0   219    71    7    7 26  3 62  9  0
 4  0    252  80972  55740 7094984    0    0   912  1376 1625 2253 20  1 76  3  0
 3  0    252  57136  55820 7100208    0    0  1621   692 1396 3459 21  1 74  4  0
 3  1    252  57728  55824 7094740    0    0  1211   616 1423 2835 19  1 76  3  0
 4  5    252  47628  55536 7052016    0    0  3113  1072 1399 11270 21  3 65 11  0
 2  5    252  47948  55572 7050052    0    0  3379   479 1529 2090 10  1 79 10  0

I'm not an expert on interpreting vmstat but this sample doesn't appear too bad (ie it's not a time when things are lagging badly) but a cpu upgrade would provide some immediate benefits, as I suspect would moving images out of the db, if they're not already.

[icerat]

Originally Posted by Donn

icerat - without meaning to teach you how to suck eggs - please don't forget to cron some kind of regular backup of the db to an external location!

yeah, as I note above I don't yet know what the current backup strategy is. Hopefully there is one!

a batch backup of a 32gb database ain't quite so simple ....

[Sherman Bay]

This thread is moving too fast for me to keep up with it, so perhaps I'm repeating what has already been said. I've only read the first page, but (1) Abaddon is right, and (2) it seems like JREF is trying to host the hardware itself. Not a good idea. Let the cloud do it and you don't worry about CPUs, RAM, power supplies, or cooling towers -- all you need is to arrange for sufficient bandwidth and adequate storage.

There are many suppliers out there. Although I am no longer running a message board, my domain is hosted by GoDaddy, which has an entire staff dedicated to 24/7 support, and not in India, either. They have been very good to me and I've never paid a penny for additional support, even though I call them every 6 months or so with a concern.

I'm using a shared server, which is adequate for me, and GoDaddy gives me some outrageous amount of storage for video plus "unlimited" email space, and unmeasured data volume, all for about $100 per year for a business account. Moving to a dedicated server would be only a small upgrade.

Not long ago, I was hosting about 500GB of video for public access, and the shared server worked just fine. I daresay that the JREF message board's overall traffic for a few hundred users, being text-based, is not much different than a dozen users downloading video. JREF's needs are not that challenging in today's cloud world.

Last I checked, vBulletin's price was just a few hundred dollars for unlimited use, and your hosting service should assist in installation and migration for little or no cost.

Please, Sharon, look into this, and you'll be glad you did. We all will.

[jhunter1163]

Originally Posted by icerat

Any preference on a crowd-funding platform to support this?

I think Causes would be best; since JREF is a nonprofit the fee would only be 4.75% instead of the 7-9% on Kickstarter or Gofundme.

[a_unique_person]

Originally Posted by icerat

Thanks. I think we can maintain the site pretty much as including post counts etc. Look and feel will eventually change a bit, but we can do things incrementally.

Here's the current basic server configuration. It's a dedicated box installed 5.5 years ago -

Bare Metal Server Installed 2/19/2009 in Seattle @ Softlayer
CentOS 2.6.18-92.e15
SuperMicro X7QCE Intel Xeon HexCore QuadProc Sata [4Proc]
4x2GB Generic RAM
4x2.13GHz Intel Xeon-Tigerton (7320-Quadcore)
SuperMicro AOC-SIMSO-plus Remote Management Card
2xSuperMicro PWS-1K01-1R Power Supply
SuperMicro BPN-SAS-828TQ Backplane
Adaptec 3405 Drive Controller
Western Digital Raptor 10,000 RPM WD1500ADFD (sdb) for Database
Seagate Cheetah ST373455SS [73GB] (sda) for system

It is currently hosting both the forum and main website, there are already plans to migrate the website. The main website is filtered through a firewall, which is why it has a different IP.

Here's some network statistics

In: Average 201.11 Kbps, Max 781.84Kbps
Out: Average 2.6 Mbps, max 25.2 Mbps

I assume from that it's on a 100Mbps network.

Hard drive performance -

%iostat -d 3
Linux 2.6.18-92.el5 (jref.randi.org)    08/02/2014

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              15.23       226.15       455.27   78704786  158440916
sdb             231.65      6702.42      1794.54 2332545177  624528576

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               1.67         5.33        24.00         16         72
sdb             271.00      3770.67      6261.33      11312      18784

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               4.00         0.00       253.33          0        760
sdb             273.67      3789.33      6064.00      11368      18192

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              18.33         2.67       357.33          8       1072
sdb             265.00      3882.67      5906.67      11648      17720

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               2.33         0.00        58.67          0        176
sdb             279.33      3888.00      7200.00      11664      21600

File system usage

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5             9.7G  586M  8.7G   7% /
/dev/sda8             996M   34M  911M   4% /tmp
/dev/sda7             104G   52G   46G  54% /home
/dev/sda3             9.7G  6.3G  3.0G  68% /usr
/dev/sda2             9.7G  2.2G  7.1G  24% /var
/dev/sda1              99M   12M   83M  13% /boot
tmpfs                 3.9G     0  3.9G   0% /dev/shm
/dev/sdb1              68G   33G   32G  51% /var/lib/mysql
/home/domlogs         104G   52G   46G  54% /usr/local/apache/domlogs

I thought the MySQL db seemed overly large however vbulletin 3.7 apparently stores images in the db by default. Assuming this is the current setup then moving them to the file system would probably help performance.

I have not yet got details on current backup strategy.

System Performance

%vmstat 3

%vmstat 3
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 7  0    252  86180  55660 7092544    0    0   219    71    7    7 26  3 62  9  0
 4  0    252  80972  55740 7094984    0    0   912  1376 1625 2253 20  1 76  3  0
 3  0    252  57136  55820 7100208    0    0  1621   692 1396 3459 21  1 74  4  0
 3  1    252  57728  55824 7094740    0    0  1211   616 1423 2835 19  1 76  3  0
 4  5    252  47628  55536 7052016    0    0  3113  1072 1399 11270 21  3 65 11  0
 2  5    252  47948  55572 7050052    0    0  3379   479 1529 2090 10  1 79 10  0

I'm not an expert on interpreting vmstat but this sample doesn't appear too bad (ie it's not a time when things are lagging badly) but a cpu upgrade would provide some immediate benefits, as I suspect would moving images out of the db, if they're not already.

Really want some stats over time. The snapshot looks fine, but we don't know what it looks like when it is hanging. More RAM to buffer the DB wouldn't hurt.

[Klimax]

Interesting details on server. Seems not that many changes since Darat last time published them. (Last public upgrade was some years ago)

[Klimax]

Wait, is anybody else missing avatars of other posters?

[Donn]

Originally Posted by icerat

yeah, as I note above I don't yet know what the current backup strategy is. Hopefully there is one!

a batch backup of a 32gb database ain't quite so simple ....

Ouch. That takes it out of my range of experience.

I googled a bit. This link:
http://dba.stackexchange.com/questio...large-database
.. mentions a 32gb database and shows how to use mysqldump and some bash to perform a backup.

I also saw mention of this free backup tool:
http://www.percona.com/software/percona-xtrabackup
.. which looks pretty good.

Another idea is dd. Since mysql has its own partition (a wise move) and this it's own mount, you could (after pausing writes somehow):
dd if=/dev/sdb1 of=/some/place/dbbackup_{some rolling date}.bin

http://linuxpoison.blogspot.com/2009...-using-dd.html


After all is done, maybe some kind of raid would be best. I always worry about making an exact copy of what may be a corrupt drive! Not sure how to avoid that.

[Wowbagger]

Originally Posted by idoubtit

There WILL be change. To hope that every nook, cranny and post count remains the same might be a pie in the sky. What do you want? Improvement. Sometimes it means resetting the odometer and starting from scratch.

If you must "reset the odometer", I would like to see an archive of all the old posts kept somewhere. You can keep them in a static database, perhaps on a separate URL.

I agree that life moves on. But, we don't destroy the works of great artists while doing so. We shouldn't destroy the work of great posters, either. (Nor the bad ones, for context.)

[icerat]

Originally Posted by Donn

After all is done, maybe some kind of raid would be best. I always worry about making an exact copy of what may be a corrupt drive! Not sure how to avoid that.

What you want is both -a live backup of the current database in order to have a rapid (or even immediate) return to service in case of hardware failure, plus a regular offsite backup in case of some more serious catastrophe.

There are a number of solutions, it comes down to $$$$.

[blobru]

Originally Posted by Klimax

Wait, is anybody else missing avatars of other posters?

I'm seeing everybody's.

About the vmstats:

Originally Posted by icerat

System Performance

%vmstat 3

%vmstat 3
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 7  0    252  86180  55660 7092544    0    0   219    71    7    7 26  3 62  9  0
 4  0    252  80972  55740 7094984    0    0   912  1376 1625 2253 20  1 76  3  0
 3  0    252  57136  55820 7100208    0    0  1621   692 1396 3459 21  1 74  4  0
 3  1    252  57728  55824 7094740    0    0  1211   616 1423 2835 19  1 76  3  0
 4  5    252  47628  55536 7052016    0    0  3113  1072 1399 11270 21  3 65 11  0
 2  5    252  47948  55572 7050052    0    0  3379   479 1529 2090 10  1 79 10  0

- if i'm reading them right, the top line is average values since the last upgrade: so on average over that period, each second there were 7 r (ready, requesting cpu time) processes and 0 b (blocked, waiting for disk input), with 7 in (interrupts: a higher priority process interrupting another) and 7 cs (context switches: recording the state of the interrupted process). In the rest of the lines (the recent values, measured each second[?]), the average for r ready processes is down to 3.2, with 5 b blocked processes in the bottom 2 lines; while interrupts/sec has risen from 7 way up to an average of roughly 1.5K, and cs context switches from 7 to over 4K (note the over 11K context switches once the 5 blocked processes appear, and the jump to over 3k-bytes/sec (bi) disk input into memory).

Which i guess just confirms that even when it's not hanging, the system's a heckuva lot busier than normal. I have no further insight; curious if the numbers suggest anything to anyone with expertise.

[idoubtit]

Mr. Adams, on the JREF board, is able to get in and do some technical tweaks to the hardware and software.

One insight he has given me is that right now 40% of the queries are from search engine bots, a good proportion of the traffic is from China. Legitimate traffic?

[Sherman Bay]

Originally Posted by idoubtit

Mr. Adams, on the JREF board, is able to get in and do some technical tweaks to the hardware and software.

One insight he has given me is that right now 40% of the queries are from search engine bots, a good proportion of the traffic is from China. Legitimate traffic?

I think not. Your figs roughly match mine, back when I was worrying about such things.

Search engines are good to have, but you are probably lumping "good" engines like Yahoo and Google with "bad" ones who are merely trying to aggregate and repackage data and links for resale to anyone who will buy.

Some of that traffic is the bad guys probing for weakness, harvesters looking for emails, and scammers looking for victims. Unless you have an extremely good filter, it's a fact of life that host providers have to accommodate the bad traffic to some degree; to do otherwise risks filtering out the odd but good guys.

[Wolfman]

Originally Posted by idoubtit

One insight he has given me is that right now 40% of the queries are from search engine bots, a good proportion of the traffic is from China. Legitimate traffic?

Well, I'm in China, as are a few other members...but even with my typically lengthy posts, I don't think we can account for "a good proportion of the traffic". I think we can safely assume that the rest is not the kinda' traffic we want.

[Blue Mountain]

Originally Posted by icerat

yeah, as I note above I don't yet know what the current backup strategy is. Hopefully there is one!

a batch backup of a 32gb database ain't quite so simple ....

Using the Logical Volume Manager makes it a lot easier:

• Do something to have HTTP stop accepting connections briefly (e.g. shut it down, or block packets on port 80)
• Stop MySQL, or issue FLUSH TABLES WITH READ LOCK. This gives us a clean and up-to-date copy of the database.
• Create a snapshot using LVM. This is actually very quick. The snapshot works by setting up a copy-on-write volume: as blocks on the source volume change, a "before" image is written to the CoW volume. Reading from the snapshot will read from either the original volume if the block is unchanged since the snapshot was taken, or from the before image on the CoW volume.
• Restart MySQL (or issue UNLOCK TABLES) and httpd (unblock port 80). Total downtime: less than 30 seconds.
• Back up from the snapshot volume at leisure (tar, cp, or even mysqldump if you're feeling masochistic)
• Delete the snapshot volume when the backup is done, to prevent it from filling up

[Blue Mountain]

For hardware, there are nice systems like the following:

Hwelett-Packard ProLiant DL320e Gen8 v2 Server.

I'm biased toward HP because that's what we run at work. There are similar offerings from IBM and Dell. These computers are built very well and have excellent SAS RAID controllers with battery-backed write caching. Replacement discs can be bought at a premium from the vendor or at a discount from companies in eBay.

With, say, four discs set up in RAID 5 mode, a disc can fail and the system will just carry on. If there's a standby spare, the controller can immediately put the spare into service. Because the SAS discs are hot swappable, the replacement disc can be put into service without bringing down the system.

Setting up one of these systems properly also means setting up the monitoring tools so that the sysadmins receive alerts when something goes wrong. It's of no use having the RAID controller switch over to a spare disc if no-one knows it's happened, because the next disc to fail will destroy the array.

[zooterkin]

Or RAID 6, which needs another disk, but will allow 2 to fail without losing data (but you still have to monitor them, as you say).