IS Forum
Forum Index Register Members List Events Mark Forums Read Help

Go Back   International Skeptics Forum » Welcome to ISF » Other Skeptical Organizations » JREF
 


Welcome to the International Skeptics Forum, where we discuss skepticism, critical thinking, the paranormal and science in a friendly but lively way. You are currently viewing the forum as a guest, which means you are missing out on discussing matters that are of interest to you. Please consider registering so you can gain full use of the forum features and interact with other Members. Registration is simple, fast and free! Click here to register today.
Tags Future of Forum , ISF , new forum

Reply
Old 2nd August 2014, 06:02 PM   #121
icerat
Philosopher
 
icerat's Avatar
 
Join Date: Apr 2008
Location: sweden
Posts: 5,547
Originally Posted by Belz... View Post
Yes but I assumed that, since the JREF has a forum, it has people that take care of it and can identify problems.
Very well might have when the forum was started. That's part of the problem with volunteer driven resources - when someone with a passion for a particular area leaves for whatever reason, it's difficult to sustain what they were doing. One of the things I do is help people design projects from the ground up that will be sustainable without the passion of the founder.
__________________
Benford's law of controversy - Passion is inversely proportional to the amount of real information available
icerat is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 2nd August 2014, 06:02 PM   #122
This is The End
 
This is The End's Avatar
 
Join Date: Sep 2007
Posts: 10,920
Originally Posted by icerat View Post
That will mean lots of whining from people who like things the way they are.

If you fix the performance issue there is absolutely no reason for a change in software.

Anyone attempting to change the software would be almost guaranteed to give up on preserving the current degree of access to the entire thread history. They might think they can pull it off at first, and then they will likely realize they cannot, but by that time they will be married to the idea and we will lose access. The thread history will be archived only from that point on.

Being able to find a thread from 2004 that matches our current format, and even be able to bump the thread if need be, is what I like about this forum.

Please do not fix things that are not broken.
__________________
________________________

Last edited by This is The End; 2nd August 2014 at 06:06 PM.
This is The End is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 2nd August 2014, 06:27 PM   #123
icerat
Philosopher
 
icerat's Avatar
 
Join Date: Apr 2008
Location: sweden
Posts: 5,547
Originally Posted by OnlyTellsTruths View Post
If you fix the performance issue there is absolutely no reason for a change in software.

Anyone attempting to change the software would be almost guaranteed to give up on preserving the current degree of access to the entire thread history. They might think they can pull it off at first, and then they will likely realize they cannot, but by that time they will be married to the idea and we will lose access. The thread history will be archived only from that point on.

Being able to find a thread from 2004 that matches our current format, and even be able to bump the thread if need be, is what I like about this forum.

Please do not fix things that are not broken.
While in principle I agree, the reality is that the forum is broken. There's severe performance issues that need addressing. While it can probably be temporarily addressed with hardware upgrades, the is reality things become "broken" from a user perspective anyway - more than half of internet traffic in the US is now on mobile devices, and most websites typically have around a third or more of their visitors from phones or tablets. If you don't cater to that audience, you lose them.

Then you have issues with version creep. The OS version this forum runs on is no longer being updated and in less than 3 years won't even receive security updates. You have similar issues with the forum and database software. Often upgrading one will break one of the others without also upgrading there (and more often than not something will break even then!)

Most forum software has the same general structure, it's a matter of being able to find things in the same place, having the same kind of functionality (tags for embedding youtube videos for example) etc that make something familiar. Nothing I hate more than when (for example) Facebook just changes things and they're not where you expect.

So I hear you!

Ideally we'll setup a mirror test version for people like yourself to use before the site software is upgraded. Time and funding limitations apply of course, but hopefully we can do it properly.
__________________
Benford's law of controversy - Passion is inversely proportional to the amount of real information available
icerat is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 2nd August 2014, 06:45 PM   #124
SezMe
post-pre-born
 
SezMe's Avatar
 
Join Date: Dec 2003
Location: Santa Barbara, CA
Posts: 25,118
WARNING: If you break post counts, the whining will break every whine meter ever built.
SezMe is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 2nd August 2014, 07:08 PM   #125
varwoche
Penultimate Amazing
 
varwoche's Avatar
 
Join Date: Feb 2004
Location: Puget Sound
Posts: 15,465
Originally Posted by icerat View Post
it's almost certainly a mySQL bottleneck
However, RAM and/or disk bottleneck could be causing db bottleneck.
__________________
To survive election season on a skeptics forum, one must understand Hymie-the-Robot.
varwoche is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 2nd August 2014, 07:22 PM   #126
varwoche
Penultimate Amazing
 
varwoche's Avatar
 
Join Date: Feb 2004
Location: Puget Sound
Posts: 15,465
Originally Posted by icerat View Post
ugh, performance has been disastrous for me the past half hour ... peak time?
Not for jref, but maybe for co-tenants.
__________________
To survive election season on a skeptics forum, one must understand Hymie-the-Robot.
varwoche is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 2nd August 2014, 07:33 PM   #127
varwoche
Penultimate Amazing
 
varwoche's Avatar
 
Join Date: Feb 2004
Location: Puget Sound
Posts: 15,465
Originally Posted by icerat View Post
My current favourite is Xenforo, which was developed by the original vBulletin lead developers. One thing I think is important is native support for different screen sizes (eg cell/tablet). All suggestions appreciated.
Fortunately...
Originally Posted by xenforo
XenForo has the ability to import data from other forum software, a process also known as converting. Version 1.2 of XenForo supports importing data from the following:

vBulletin 3.6, 3.7, 3.8, and 4.x
...
__________________
To survive election season on a skeptics forum, one must understand Hymie-the-Robot.
varwoche is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 2nd August 2014, 07:36 PM   #128
rjh01
Gentleman of leisure
Tagger
 
rjh01's Avatar
 
Join Date: May 2005
Location: Flying around in the sky
Posts: 26,621
I am willing to help. Though about the only job I might have experience in is testing. If you give me something untested I can break it.
__________________
This signature is for rent.
rjh01 is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 2nd August 2014, 08:33 PM   #129
Slowvehicle
Membership Drive
Co-Ordinator,
Russell's Antinomy
 
Slowvehicle's Avatar
 
Join Date: Sep 2012
Location: ...1888 miles from home by the shortest route without tolls...
Posts: 17,348
Originally Posted by icerat View Post
Ok,

I've got some hard details on the current server and performance and as most expected it's almost certainly a mySQL bottleneck. I'm pretty confident we can get things up on running well for half the cost JREF is currently paying, and then it's just a matter of $$$ to improve performance even more (SSD vs SATA more cache etc etc).

Put your hand up if you have experience with -

* CentOS
* mySQL
* vbulletin 3.7.7

And are willing to contribute some time.

The current goal would be to move the forum as is to new hardware to stabilise performance, and then at a later date upgrade the forum software itself. Suggestions and experiences on the latter are requested. vBulletin as a project has gone downhill so it will most likely mean migrating to other commercial or opensource forum software. That will mean lots of whining from people who like things the way they are

My current favourite is Xenforo, which was developed by the original vBulletin lead developers. One thing I think is important is native support for different screen sizes (eg cell/tablet). All suggestions appreciated.

The other issue is of course funding. I suspect JREF has little interest in continuing to financially support the forum so we'll have to look at a donor/advertising/whatever model. Again, your input appreciated.
...I will not pretend to expertise, but I do have some time, and enthusiasm. Holler if you ever need cannon fodder (or good cookie recipes).
__________________
"They want to make their molehills equal to the mountains by cutting the mountains down." -turingtest
"The universe did not come from nothing, it came from 'We don't know'." -Dancing David
"Cry, booga, booga, booga! and let slip the Hamsters of Silly!" -JFDHintze
Slowvehicle is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 2nd August 2014, 09:05 PM   #130
idoubtit
Critical Thinker
 
idoubtit's Avatar
 
Join Date: May 2007
Posts: 365
I suppose it's OK to release details. I'm trying to be conscientious here but I'm not that worried about hackers.

There has not been dedicated staff to deal with the forum or website. OBVIOUSLY.

There WILL be change. To hope that every nook, cranny and post count remains the same might be a pie in the sky. What do you want? Improvement. Sometimes it means resetting the odometer and starting from scratch.

Life goes on.
idoubtit is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 2nd August 2014, 09:13 PM   #131
Christian Klippel
Master Poster
 
Christian Klippel's Avatar
 
Join Date: Feb 2008
Location: Ruhr Area in Germany
Posts: 2,431
Originally Posted by icerat View Post
(SSD vs SATA more cache etc etc).
Are SSD's reliable enough nowdays, given the intrinsic limitation of the number of write-cycles for the used flash memory technology?

Or are you planning to use SSD for the OS and slow-changing stuff only, and have the fast-changing stuff on regular (RAID-)drives?

Greetings,

Chris
Christian Klippel is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 2nd August 2014, 09:13 PM   #132
marplots
Penultimate Amazing
 
Join Date: Feb 2006
Posts: 29,167
My worth as a human being is directly pinned to my post count at JREF.
marplots is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 2nd August 2014, 09:19 PM   #133
Blue Mountain
Resident Skeptical Hobbit
 
Blue Mountain's Avatar
 
Join Date: Jul 2005
Location: Waging war on woo-woo in Winnipeg
Posts: 6,518
I'm familiar with CentOS 5 and 6; my day job is the system administrator for a web services company.
__________________
The social illusion reigns to-day upon all the heaped-up ruins of the past, and to it belongs the future. The masses have never thirsted after truth. They turn aside from evidence that is not to their taste, preferring to deify error, if error seduce them. Gustav Le Bon, The Crowd, 1895 (from the French)
Canadian or living in Canada? PM me if you want an entry on the list of Canadians on the forum.
Blue Mountain is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 2nd August 2014, 10:07 PM   #134
Michel H
Master Poster
 
Join Date: May 2012
Location: Belgium
Posts: 2,050
Originally Posted by idoubtit View Post
I suppose it's OK to release details. I'm trying to be conscientious here but I'm not that worried about hackers.

There has not been dedicated staff to deal with the forum or website. OBVIOUSLY.

There WILL be change. To hope that every nook, cranny and post count remains the same might be a pie in the sky. What do you want? Improvement. Sometimes it means resetting the odometer and starting from scratch.

Life goes on.
Thank you Sharon. You said there has not been dedicated staff to deal with the forum or website. But then who is this mysterious "person with responsibility for the site" icerat was talking about? (I hope I am not asking too much here):
Originally Posted by icerat View Post
Well I got a response that they're collecting names of people offering assistance and it's being passed on to the person with responsibility for the site who will be deciding how to proceed.
...
Is it not reasonable to assume that, at the JREF, there is one person who is more technically-oriented and who is in charge of the site?
Michel H is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 2nd August 2014, 10:42 PM   #135
jhunter1163
beer-swilling semiliterate
 
jhunter1163's Avatar
 
Join Date: Jul 2006
Location: Connecticut, or King Arthur's Court. Hard to tell sometimes.
Posts: 25,313
Originally Posted by AdMan View Post
Seconded. Plus I'm happy to donate to a funding effort. I have little doubt that a crowdfunding effort will be able to keep the forum going if it's needed.
I have little doubt that we could raise $10,000 in 24 hours if it'd preserve the Forum and solve the performance issues. My $100 is still on the table.
jhunter1163 is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 2nd August 2014, 10:46 PM   #136
jhunter1163
beer-swilling semiliterate
 
jhunter1163's Avatar
 
Join Date: Jul 2006
Location: Connecticut, or King Arthur's Court. Hard to tell sometimes.
Posts: 25,313
Originally Posted by Michel H View Post
Thank you Sharon. You said there has not been dedicated staff to deal with the forum or website. But then who is this mysterious "person with responsibility for the site" icerat was talking about? (I hope I am not asking too much here):

Is it not reasonable to assume that, at the JREF, there is one person who is more technically-oriented and who is in charge of the site?
I assume that Darat is the person who has been in charge of the technical aspects of the forum.
jhunter1163 is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 2nd August 2014, 11:11 PM   #137
Michel H
Master Poster
 
Join Date: May 2012
Location: Belgium
Posts: 2,050
Originally Posted by jhunter1163 View Post
I assume that Darat is the person who has been in charge of the technical aspects of the forum.
Ah OK, thank you for the clear answer, jhunter. However, Darat said, about two months ago:
Originally Posted by Darat View Post
...

Unfortunately server and so on are outside my responsibility.

I am looking to see if there is anything else I can do on my side to increase responsiveness, I've been checking a few things and it may be that Tapatalk is adding quite a lot of overhead to the Forum but that's a very preliminary conclusion at the moment.
Isn't there another person in charge of more technical aspects at JREF (sorry for asking so much). I hope we're not in a Kafka novel here.
Michel H is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 12:51 AM   #138
Wolfman
Chief Solipsistic
Autosycophant
 
Wolfman's Avatar
 
Join Date: Jan 2007
Location: Vancouver, Canada
Posts: 13,394
Originally Posted by Michel H View Post
Isn't there another person in charge of more technical aspects at JREF (sorry for asking so much). I hope we're not in a Kafka novel here.
Basically, no. Darat can deal with software issues, but not hardware. And until Sharon, there's been nobody at the JREF offices to specifically deal with such issues.
__________________
Please check out my business, The Language of Culture
Wolfman is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 02:02 AM   #139
Donn
Philosopher
 
Donn's Avatar
 
Join Date: Sep 2003
Location: In my head.
Posts: 7,758
It's weird how a website can look so real to some and so abstract to others. To me, the forum is JREF. I can't visit America, I'll never shake Randi's hand (I hz a sad) but I come here almost everyday to get a speck of this culture.

Good luck icerat and friends, keep the bytes of reason flowing!
__________________
"If I hadn't believed it with my own mind, I would never have seen it." - thanks sackett
"If you stand on a piece of paper, you are indeed closer to the moon." - MRC_Hans
"I was a believer. Until I saw it." - Magrat
Donn is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 02:29 AM   #140
icerat
Philosopher
 
icerat's Avatar
 
Join Date: Apr 2008
Location: sweden
Posts: 5,547
Originally Posted by varwoche View Post
However, RAM and/or disk bottleneck could be causing db bottleneck.
And cpu as well (see below)

Originally Posted by varwoche View Post
Not for jref, but maybe for co-tenants.
aren't any (see below)

Originally Posted by varwoche View Post
one reason why it's currently my first choice
__________________
Benford's law of controversy - Passion is inversely proportional to the amount of real information available
icerat is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 02:42 AM   #141
icerat
Philosopher
 
icerat's Avatar
 
Join Date: Apr 2008
Location: sweden
Posts: 5,547
Originally Posted by Christian Klippel View Post
Are SSD's reliable enough nowdays, given the intrinsic limitation of the number of write-cycles for the used flash memory technology?
Yes, indeed with decent brands they may be more reliable than SATA

Quote:
Or are you planning to use SSD for the OS and slow-changing stuff only, and have the fast-changing stuff on regular (RAID-)drives?
A lot depends on funding of course. Even just upgrading the server without SSD will make a noticeable difference. Anything more is icing.
__________________
Benford's law of controversy - Passion is inversely proportional to the amount of real information available
icerat is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 02:44 AM   #142
icerat
Philosopher
 
icerat's Avatar
 
Join Date: Apr 2008
Location: sweden
Posts: 5,547
Originally Posted by jhunter1163 View Post
I have little doubt that we could raise $10,000 in 24 hours if it'd preserve the Forum and solve the performance issues. My $100 is still on the table.
Any preference on a crowd-funding platform to support this?
__________________
Benford's law of controversy - Passion is inversely proportional to the amount of real information available
icerat is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 02:46 AM   #143
Donn
Philosopher
 
Donn's Avatar
 
Join Date: Sep 2003
Location: In my head.
Posts: 7,758
icerat - without meaning to teach you how to suck eggs - please don't forget to cron some kind of regular backup of the db to an external location!
__________________
"If I hadn't believed it with my own mind, I would never have seen it." - thanks sackett
"If you stand on a piece of paper, you are indeed closer to the moon." - MRC_Hans
"I was a believer. Until I saw it." - Magrat
Donn is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 03:14 AM   #144
icerat
Philosopher
 
icerat's Avatar
 
Join Date: Apr 2008
Location: sweden
Posts: 5,547
Originally Posted by idoubtit View Post
I suppose it's OK to release details. I'm trying to be conscientious here but I'm not that worried about hackers.

There has not been dedicated staff to deal with the forum or website. OBVIOUSLY.

There WILL be change. To hope that every nook, cranny and post count remains the same might be a pie in the sky. What do you want? Improvement. Sometimes it means resetting the odometer and starting from scratch.

Life goes on.
Thanks. I think we can maintain the site pretty much as including post counts etc. Look and feel will eventually change a bit, but we can do things incrementally.

Here's the current basic server configuration. It's a dedicated box installed 5.5 years ago -

Code:
Bare Metal Server Installed 2/19/2009 in Seattle @ Softlayer
CentOS 2.6.18-92.e15
SuperMicro X7QCE Intel Xeon HexCore QuadProc Sata [4Proc]
4x2GB Generic RAM
4x2.13GHz Intel Xeon-Tigerton (7320-Quadcore)
SuperMicro AOC-SIMSO-plus Remote Management Card
2xSuperMicro PWS-1K01-1R Power Supply
SuperMicro BPN-SAS-828TQ Backplane
Adaptec 3405 Drive Controller
Western Digital Raptor 10,000 RPM WD1500ADFD (sdb) for Database
Seagate Cheetah ST373455SS [73GB] (sda) for system
It is currently hosting both the forum and main website, there are already plans to migrate the website. The main website is filtered through a firewall, which is why it has a different IP.

Here's some network statistics
Code:
In: Average 201.11 Kbps, Max 781.84Kbps
Out: Average 2.6 Mbps, max 25.2 Mbps
I assume from that it's on a 100Mbps network.

Hard drive performance -

Code:
%iostat -d 3
Linux 2.6.18-92.el5 (jref.randi.org)    08/02/2014

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              15.23       226.15       455.27   78704786  158440916
sdb             231.65      6702.42      1794.54 2332545177  624528576

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               1.67         5.33        24.00         16         72
sdb             271.00      3770.67      6261.33      11312      18784

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               4.00         0.00       253.33          0        760
sdb             273.67      3789.33      6064.00      11368      18192

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              18.33         2.67       357.33          8       1072
sdb             265.00      3882.67      5906.67      11648      17720

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               2.33         0.00        58.67          0        176
sdb             279.33      3888.00      7200.00      11664      21600
File system usage

Code:
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5             9.7G  586M  8.7G   7% /
/dev/sda8             996M   34M  911M   4% /tmp
/dev/sda7             104G   52G   46G  54% /home
/dev/sda3             9.7G  6.3G  3.0G  68% /usr
/dev/sda2             9.7G  2.2G  7.1G  24% /var
/dev/sda1              99M   12M   83M  13% /boot
tmpfs                 3.9G     0  3.9G   0% /dev/shm
/dev/sdb1              68G   33G   32G  51% /var/lib/mysql
/home/domlogs         104G   52G   46G  54% /usr/local/apache/domlogs
I thought the MySQL db seemed overly large however vbulletin 3.7 apparently stores images in the db by default. Assuming this is the current setup then moving them to the file system would probably help performance.

I have not yet got details on current backup strategy.

System Performance

%vmstat 3
Code:
%vmstat 3
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 7  0    252  86180  55660 7092544    0    0   219    71    7    7 26  3 62  9  0
 4  0    252  80972  55740 7094984    0    0   912  1376 1625 2253 20  1 76  3  0
 3  0    252  57136  55820 7100208    0    0  1621   692 1396 3459 21  1 74  4  0
 3  1    252  57728  55824 7094740    0    0  1211   616 1423 2835 19  1 76  3  0
 4  5    252  47628  55536 7052016    0    0  3113  1072 1399 11270 21  3 65 11  0
 2  5    252  47948  55572 7050052    0    0  3379   479 1529 2090 10  1 79 10  0
I'm not an expert on interpreting vmstat but this sample doesn't appear too bad (ie it's not a time when things are lagging badly) but a cpu upgrade would provide some immediate benefits, as I suspect would moving images out of the db, if they're not already.
__________________
Benford's law of controversy - Passion is inversely proportional to the amount of real information available
icerat is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 03:16 AM   #145
icerat
Philosopher
 
icerat's Avatar
 
Join Date: Apr 2008
Location: sweden
Posts: 5,547
Originally Posted by Donn View Post
icerat - without meaning to teach you how to suck eggs - please don't forget to cron some kind of regular backup of the db to an external location!
yeah, as I note above I don't yet know what the current backup strategy is. Hopefully there is one!

a batch backup of a 32gb database ain't quite so simple ....
__________________
Benford's law of controversy - Passion is inversely proportional to the amount of real information available
icerat is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 04:22 AM   #146
Sherman Bay
Master Poster
 
Join Date: May 2002
Location: Wisconsin, USA
Posts: 2,334
This thread is moving too fast for me to keep up with it, so perhaps I'm repeating what has already been said. I've only read the first page, but (1) Abaddon is right, and (2) it seems like JREF is trying to host the hardware itself. Not a good idea. Let the cloud do it and you don't worry about CPUs, RAM, power supplies, or cooling towers -- all you need is to arrange for sufficient bandwidth and adequate storage.

There are many suppliers out there. Although I am no longer running a message board, my domain is hosted by GoDaddy, which has an entire staff dedicated to 24/7 support, and not in India, either. They have been very good to me and I've never paid a penny for additional support, even though I call them every 6 months or so with a concern.

I'm using a shared server, which is adequate for me, and GoDaddy gives me some outrageous amount of storage for video plus "unlimited" email space, and unmeasured data volume, all for about $100 per year for a business account. Moving to a dedicated server would be only a small upgrade.

Not long ago, I was hosting about 500GB of video for public access, and the shared server worked just fine. I daresay that the JREF message board's overall traffic for a few hundred users, being text-based, is not much different than a dozen users downloading video. JREF's needs are not that challenging in today's cloud world.

Last I checked, vBulletin's price was just a few hundred dollars for unlimited use, and your hosting service should assist in installation and migration for little or no cost.

Please, Sharon, look into this, and you'll be glad you did. We all will.
Sherman Bay is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 04:36 AM   #147
jhunter1163
beer-swilling semiliterate
 
jhunter1163's Avatar
 
Join Date: Jul 2006
Location: Connecticut, or King Arthur's Court. Hard to tell sometimes.
Posts: 25,313
Originally Posted by icerat View Post
Any preference on a crowd-funding platform to support this?
I think Causes would be best; since JREF is a nonprofit the fee would only be 4.75% instead of the 7-9% on Kickstarter or Gofundme.
jhunter1163 is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 05:31 AM   #148
a_unique_person
Director of Hatcheries and Conditioning
 
a_unique_person's Avatar
 
Join Date: Jul 2002
Location: Waiting for the pod bay door to open.
Posts: 43,078
Originally Posted by icerat View Post
Thanks. I think we can maintain the site pretty much as including post counts etc. Look and feel will eventually change a bit, but we can do things incrementally.

Here's the current basic server configuration. It's a dedicated box installed 5.5 years ago -

Code:
Bare Metal Server Installed 2/19/2009 in Seattle @ Softlayer
CentOS 2.6.18-92.e15
SuperMicro X7QCE Intel Xeon HexCore QuadProc Sata [4Proc]
4x2GB Generic RAM
4x2.13GHz Intel Xeon-Tigerton (7320-Quadcore)
SuperMicro AOC-SIMSO-plus Remote Management Card
2xSuperMicro PWS-1K01-1R Power Supply
SuperMicro BPN-SAS-828TQ Backplane
Adaptec 3405 Drive Controller
Western Digital Raptor 10,000 RPM WD1500ADFD (sdb) for Database
Seagate Cheetah ST373455SS [73GB] (sda) for system
It is currently hosting both the forum and main website, there are already plans to migrate the website. The main website is filtered through a firewall, which is why it has a different IP.

Here's some network statistics
Code:
In: Average 201.11 Kbps, Max 781.84Kbps
Out: Average 2.6 Mbps, max 25.2 Mbps
I assume from that it's on a 100Mbps network.

Hard drive performance -

Code:
%iostat -d 3
Linux 2.6.18-92.el5 (jref.randi.org)    08/02/2014

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              15.23       226.15       455.27   78704786  158440916
sdb             231.65      6702.42      1794.54 2332545177  624528576

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               1.67         5.33        24.00         16         72
sdb             271.00      3770.67      6261.33      11312      18784

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               4.00         0.00       253.33          0        760
sdb             273.67      3789.33      6064.00      11368      18192

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              18.33         2.67       357.33          8       1072
sdb             265.00      3882.67      5906.67      11648      17720

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               2.33         0.00        58.67          0        176
sdb             279.33      3888.00      7200.00      11664      21600
File system usage

Code:
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5             9.7G  586M  8.7G   7% /
/dev/sda8             996M   34M  911M   4% /tmp
/dev/sda7             104G   52G   46G  54% /home
/dev/sda3             9.7G  6.3G  3.0G  68% /usr
/dev/sda2             9.7G  2.2G  7.1G  24% /var
/dev/sda1              99M   12M   83M  13% /boot
tmpfs                 3.9G     0  3.9G   0% /dev/shm
/dev/sdb1              68G   33G   32G  51% /var/lib/mysql
/home/domlogs         104G   52G   46G  54% /usr/local/apache/domlogs
I thought the MySQL db seemed overly large however vbulletin 3.7 apparently stores images in the db by default. Assuming this is the current setup then moving them to the file system would probably help performance.

I have not yet got details on current backup strategy.

System Performance

%vmstat 3
Code:
%vmstat 3
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 7  0    252  86180  55660 7092544    0    0   219    71    7    7 26  3 62  9  0
 4  0    252  80972  55740 7094984    0    0   912  1376 1625 2253 20  1 76  3  0
 3  0    252  57136  55820 7100208    0    0  1621   692 1396 3459 21  1 74  4  0
 3  1    252  57728  55824 7094740    0    0  1211   616 1423 2835 19  1 76  3  0
 4  5    252  47628  55536 7052016    0    0  3113  1072 1399 11270 21  3 65 11  0
 2  5    252  47948  55572 7050052    0    0  3379   479 1529 2090 10  1 79 10  0
I'm not an expert on interpreting vmstat but this sample doesn't appear too bad (ie it's not a time when things are lagging badly) but a cpu upgrade would provide some immediate benefits, as I suspect would moving images out of the db, if they're not already.
Really want some stats over time. The snapshot looks fine, but we don't know what it looks like when it is hanging. More RAM to buffer the DB wouldn't hurt.
__________________
Continually pushing the boundaries of mediocrity.
Everything is possible, but not everything is probable.
“Perception is real, but the truth is not.” - Imelda Marcos
a_unique_person is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 06:03 AM   #149
Klimax
NWO Cyborg 5960x (subversion VPUNPCKHQDQ)
 
Klimax's Avatar
 
Join Date: Feb 2008
Location: Starship Wanderer - DS9
Posts: 13,372
Interesting details on server. Seems not that many changes since Darat last time published them. (Last public upgrade was some years ago)
__________________
ModBorg

Engine: Ibalgin 400
Klimax is online now   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 06:05 AM   #150
Klimax
NWO Cyborg 5960x (subversion VPUNPCKHQDQ)
 
Klimax's Avatar
 
Join Date: Feb 2008
Location: Starship Wanderer - DS9
Posts: 13,372
Wait, is anybody else missing avatars of other posters?
__________________
ModBorg

Engine: Ibalgin 400
Klimax is online now   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 06:36 AM   #151
Donn
Philosopher
 
Donn's Avatar
 
Join Date: Sep 2003
Location: In my head.
Posts: 7,758
Originally Posted by icerat View Post
yeah, as I note above I don't yet know what the current backup strategy is. Hopefully there is one!

a batch backup of a 32gb database ain't quite so simple ....
Ouch. That takes it out of my range of experience.

I googled a bit. This link:
http://dba.stackexchange.com/questio...large-database
.. mentions a 32gb database and shows how to use mysqldump and some bash to perform a backup.

I also saw mention of this free backup tool:
http://www.percona.com/software/percona-xtrabackup
.. which looks pretty good.

Another idea is dd. Since mysql has its own partition (a wise move) and this it's own mount, you could (after pausing writes somehow):
dd if=/dev/sdb1 of=/some/place/dbbackup_{some rolling date}.bin

http://linuxpoison.blogspot.com/2009...-using-dd.html


After all is done, maybe some kind of raid would be best. I always worry about making an exact copy of what may be a corrupt drive! Not sure how to avoid that.
__________________
"If I hadn't believed it with my own mind, I would never have seen it." - thanks sackett
"If you stand on a piece of paper, you are indeed closer to the moon." - MRC_Hans
"I was a believer. Until I saw it." - Magrat
Donn is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 06:44 AM   #152
Wowbagger
The Infinitely Prolonged
 
Wowbagger's Avatar
 
Join Date: Feb 2006
Location: Westchester County, NY (when not in space)
Posts: 15,452
Originally Posted by idoubtit View Post
There WILL be change. To hope that every nook, cranny and post count remains the same might be a pie in the sky. What do you want? Improvement. Sometimes it means resetting the odometer and starting from scratch.
If you must "reset the odometer", I would like to see an archive of all the old posts kept somewhere. You can keep them in a static database, perhaps on a separate URL.

I agree that life moves on. But, we don't destroy the works of great artists while doing so. We shouldn't destroy the work of great posters, either. (Nor the bad ones, for context.)
__________________
WARNING: Phrases in this post may sound meaner than they were intended to be.

SkeptiCamp NYC: http://www.skepticampnyc.org/
An open conference on science and skepticism, where you could be a presenter!

By the way, my first name is NOT Bowerick!!!!
Wowbagger is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 06:55 AM   #153
icerat
Philosopher
 
icerat's Avatar
 
Join Date: Apr 2008
Location: sweden
Posts: 5,547
Originally Posted by Donn View Post
After all is done, maybe some kind of raid would be best. I always worry about making an exact copy of what may be a corrupt drive! Not sure how to avoid that.
What you want is both -a live backup of the current database in order to have a rapid (or even immediate) return to service in case of hardware failure, plus a regular offsite backup in case of some more serious catastrophe.

There are a number of solutions, it comes down to $$$$.
__________________
Benford's law of controversy - Passion is inversely proportional to the amount of real information available
icerat is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 07:15 AM   #154
blobru
Philosopher
 
blobru's Avatar
 
Join Date: May 2007
Posts: 6,900
Originally Posted by Klimax View Post
Wait, is anybody else missing avatars of other posters?

I'm seeing everybody's.

About the vmstats:
Originally Posted by icerat View Post
System Performance

%vmstat 3
Code:
%vmstat 3
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 7  0    252  86180  55660 7092544    0    0   219    71    7    7 26  3 62  9  0
 4  0    252  80972  55740 7094984    0    0   912  1376 1625 2253 20  1 76  3  0
 3  0    252  57136  55820 7100208    0    0  1621   692 1396 3459 21  1 74  4  0
 3  1    252  57728  55824 7094740    0    0  1211   616 1423 2835 19  1 76  3  0
 4  5    252  47628  55536 7052016    0    0  3113  1072 1399 11270 21  3 65 11  0
 2  5    252  47948  55572 7050052    0    0  3379   479 1529 2090 10  1 79 10  0
- if i'm reading them right, the top line is average values since the last upgrade: so on average over that period, each second there were 7 r (ready, requesting cpu time) processes and 0 b (blocked, waiting for disk input), with 7 in (interrupts: a higher priority process interrupting another) and 7 cs (context switches: recording the state of the interrupted process). In the rest of the lines (the recent values, measured each second[?]), the average for r ready processes is down to 3.2, with 5 b blocked processes in the bottom 2 lines; while interrupts/sec has risen from 7 way up to an average of roughly 1.5K, and cs context switches from 7 to over 4K (note the over 11K context switches once the 5 blocked processes appear, and the jump to over 3k-bytes/sec (bi) disk input into memory).

Which i guess just confirms that even when it's not hanging, the system's a heckuva lot busier than normal. I have no further insight; curious if the numbers suggest anything to anyone with expertise.
__________________
"Say to them, 'I am Nobody!'" -- Ulysses to the Cyclops

"Never mind. I can't read." -- Hokulele to the Easter Bunny

Last edited by blobru; 3rd August 2014 at 07:36 AM.
blobru is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 07:46 AM   #155
idoubtit
Critical Thinker
 
idoubtit's Avatar
 
Join Date: May 2007
Posts: 365
Mr. Adams, on the JREF board, is able to get in and do some technical tweaks to the hardware and software.

One insight he has given me is that right now 40% of the queries are from search engine bots, a good proportion of the traffic is from China. Legitimate traffic?
idoubtit is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 08:43 AM   #156
Sherman Bay
Master Poster
 
Join Date: May 2002
Location: Wisconsin, USA
Posts: 2,334
Originally Posted by idoubtit View Post
Mr. Adams, on the JREF board, is able to get in and do some technical tweaks to the hardware and software.

One insight he has given me is that right now 40% of the queries are from search engine bots, a good proportion of the traffic is from China. Legitimate traffic?
I think not. Your figs roughly match mine, back when I was worrying about such things.

Search engines are good to have, but you are probably lumping "good" engines like Yahoo and Google with "bad" ones who are merely trying to aggregate and repackage data and links for resale to anyone who will buy.

Some of that traffic is the bad guys probing for weakness, harvesters looking for emails, and scammers looking for victims. Unless you have an extremely good filter, it's a fact of life that host providers have to accommodate the bad traffic to some degree; to do otherwise risks filtering out the odd but good guys.
Sherman Bay is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 08:45 AM   #157
Wolfman
Chief Solipsistic
Autosycophant
 
Wolfman's Avatar
 
Join Date: Jan 2007
Location: Vancouver, Canada
Posts: 13,394
Originally Posted by idoubtit View Post
One insight he has given me is that right now 40% of the queries are from search engine bots, a good proportion of the traffic is from China. Legitimate traffic?
Well, I'm in China, as are a few other members...but even with my typically lengthy posts, I don't think we can account for "a good proportion of the traffic". I think we can safely assume that the rest is not the kinda' traffic we want.
__________________
Please check out my business, The Language of Culture
Wolfman is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 09:48 AM   #158
Blue Mountain
Resident Skeptical Hobbit
 
Blue Mountain's Avatar
 
Join Date: Jul 2005
Location: Waging war on woo-woo in Winnipeg
Posts: 6,518
Originally Posted by icerat View Post
yeah, as I note above I don't yet know what the current backup strategy is. Hopefully there is one!

a batch backup of a 32gb database ain't quite so simple ....
Using the Logical Volume Manager makes it a lot easier:
  • Do something to have HTTP stop accepting connections briefly (e.g. shut it down, or block packets on port 80)
  • Stop MySQL, or issue FLUSH TABLES WITH READ LOCK. This gives us a clean and up-to-date copy of the database.
  • Create a snapshot using LVM. This is actually very quick. The snapshot works by setting up a copy-on-write volume: as blocks on the source volume change, a "before" image is written to the CoW volume. Reading from the snapshot will read from either the original volume if the block is unchanged since the snapshot was taken, or from the before image on the CoW volume.
  • Restart MySQL (or issue UNLOCK TABLES) and httpd (unblock port 80). Total downtime: less than 30 seconds.
  • Back up from the snapshot volume at leisure (tar, cp, or even mysqldump if you're feeling masochistic)
  • Delete the snapshot volume when the backup is done, to prevent it from filling up
__________________
The social illusion reigns to-day upon all the heaped-up ruins of the past, and to it belongs the future. The masses have never thirsted after truth. They turn aside from evidence that is not to their taste, preferring to deify error, if error seduce them. Gustav Le Bon, The Crowd, 1895 (from the French)
Canadian or living in Canada? PM me if you want an entry on the list of Canadians on the forum.

Last edited by Blue Mountain; 3rd August 2014 at 10:02 AM. Reason: MySQL: LOCK TABLES
Blue Mountain is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 10:17 AM   #159
Blue Mountain
Resident Skeptical Hobbit
 
Blue Mountain's Avatar
 
Join Date: Jul 2005
Location: Waging war on woo-woo in Winnipeg
Posts: 6,518
For hardware, there are nice systems like the following:

Hwelett-Packard ProLiant DL320e Gen8 v2 Server.

I'm biased toward HP because that's what we run at work. There are similar offerings from IBM and Dell. These computers are built very well and have excellent SAS RAID controllers with battery-backed write caching. Replacement discs can be bought at a premium from the vendor or at a discount from companies in eBay.

With, say, four discs set up in RAID 5 mode, a disc can fail and the system will just carry on. If there's a standby spare, the controller can immediately put the spare into service. Because the SAS discs are hot swappable, the replacement disc can be put into service without bringing down the system.

Setting up one of these systems properly also means setting up the monitoring tools so that the sysadmins receive alerts when something goes wrong. It's of no use having the RAID controller switch over to a spare disc if no-one knows it's happened, because the next disc to fail will destroy the array.
__________________
The social illusion reigns to-day upon all the heaped-up ruins of the past, and to it belongs the future. The masses have never thirsted after truth. They turn aside from evidence that is not to their taste, preferring to deify error, if error seduce them. Gustav Le Bon, The Crowd, 1895 (from the French)
Canadian or living in Canada? PM me if you want an entry on the list of Canadians on the forum.
Blue Mountain is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 11:13 AM   #160
zooterkin
Nitpicking dilettante
Administrator
 
zooterkin's Avatar
 
Join Date: Mar 2007
Location: Berkshire, mostly
Posts: 48,818
Or RAID 6, which needs another disk, but will allow 2 to fail without losing data (but you still have to monitor them, as you say).
__________________
The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts.Bertrand Russell
Zooterkin is correct Darat
Nerd! Hokulele
Join the JREF Folders ! Team 13232
Ezekiel 23:20
zooterkin is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Reply

International Skeptics Forum » Welcome to ISF » Other Skeptical Organizations » JREF

Bookmarks

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -7. The time now is 12:19 AM.
Powered by vBulletin. Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum began as part of the James Randi Education Foundation (JREF). However, the forum now exists as
an independent entity with no affiliation with or endorsement by the JREF, including the section in reference to "JREF" topics.

Disclaimer: Messages posted in the Forum are solely the opinion of their authors.