IS Forum
Forum Index Register Members List Events Search Today's Posts Mark Forums Read Help

Go Back   International Skeptics Forum » Welcome to ISF » Other Skeptical Organizations » JREF
 


Welcome to the International Skeptics Forum, where we discuss skepticism, critical thinking, the paranormal and science in a friendly but lively way. You are currently viewing the forum as a guest, which means you are missing out on discussing matters that are of interest to you. Please consider registering so you can gain full use of the forum features and interact with other Members. Registration is simple, fast and free! Click here to register today.
Tags Future of Forum , ISF , new forum

Reply
Old 3rd August 2014, 12:30 PM   #161
This is The End
 
This is The End's Avatar
 
Join Date: Sep 2007
Posts: 10,924
Originally Posted by icerat View Post
While in principle I agree, the reality is that the forum is broken. There's severe performance issues that need addressing.

You may know something I do not, but the forum seems to work great when it is not having lag issues. So if those "severe performance issues" really aren't just improper hardware or software setup, then I would agree with you.

As for mobile users, that is what we have tapatalk for. As for security updates, are you sure those are even being added to this forum now, let alone when it stops in three years?

If you do take on the task eventually I wish you the best of luck.

Also, thanks for all of the server statistics in post #144!!
__________________
________________________
This is The End is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 12:47 PM   #162
temporalillusion
Technical Admin
 
temporalillusion's Avatar
 
Join Date: Jul 2006
Location: Canada's Texas
Posts: 1,495
Originally Posted by varwoche View Post
In addition to this, there is a custom script for importing big boards onto Xenforo. The built in script works (I've run it a few times as tests), but it by no means fast.

There is a custom import script that might be worth considering: https://marketplace.digitalpoint.com...in-4.1535/item

Or there are even people who specialize in such things: http://xenforo.com/community/threads...ervices.55328/

If donations are part of the equation allocating some to something like that might be worth doing, when that time comes anyway.
__________________
One man's reason that something is not reliable evidence is another man's whine about how others won't buy 3 magic beans with the family cow. - hgc
temporalillusion is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 12:58 PM   #163
temporalillusion
Technical Admin
 
temporalillusion's Avatar
 
Join Date: Jul 2006
Location: Canada's Texas
Posts: 1,495
Originally Posted by icerat View Post
Here's the current basic server configuration. It's a dedicated box installed 5.5 years ago -

Code:
Bare Metal Server Installed 2/19/2009 in Seattle @ Softlayer
CentOS 2.6.18-92.e15
SuperMicro X7QCE Intel Xeon HexCore QuadProc Sata [4Proc]
4x2GB Generic RAM
4x2.13GHz Intel Xeon-Tigerton (7320-Quadcore)
SuperMicro AOC-SIMSO-plus Remote Management Card
2xSuperMicro PWS-1K01-1R Power Supply
SuperMicro BPN-SAS-828TQ Backplane
Adaptec 3405 Drive Controller
Western Digital Raptor 10,000 RPM WD1500ADFD (sdb) for Database
Seagate Cheetah ST373455SS [73GB] (sda) for system
etc, etc
Huh, that's not nearly as bad as I'd thought, it's not a VPS anyway. RAM isn't too bad, though how it's allocated might be an issue (i.e. if MySQL is only getting 2GB). How much is given to Apache might be part of it as well. CPU could be part of it, and disk might be limiting the database as well. My forum runs off an NFS share for the database drives but they are pretty high I/O. An SSD would of course be huge.
__________________
One man's reason that something is not reliable evidence is another man's whine about how others won't buy 3 magic beans with the family cow. - hgc
temporalillusion is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 01:03 PM   #164
temporalillusion
Technical Admin
 
temporalillusion's Avatar
 
Join Date: Jul 2006
Location: Canada's Texas
Posts: 1,495
Originally Posted by icerat View Post
Any preference on a crowd-funding platform to support this?
I mentioned this already to you but I thought I'd bring it up here as well as a more general discussion.

There's two ways to do it, one is to purchase servers and colocate them, the other is to just rent servers. Renting costs more in the long run in general, but you have the not insignificant benefits of being supported by someone else, hardware repairs being someone else's problem, warranty being someone else's problem, and what to do if moving to updated hardware being someone else's problem.. well you get the idea Paying monthly for servers is just so much simpler IMO.

Purchasing servers costs more up front, then has lower cost for a while (assuming the colocation is cheaper than renting the server), then increases again when things start to break or require purchasing more warranty or paying hands and eyes at the colo to do repairs.

I prefer renting, but that kind of goes against the donation style funding.

Not saying donations shouldn't be done, or part of it, or be the whole funding method, just raising the point for consideration.
__________________
One man's reason that something is not reliable evidence is another man's whine about how others won't buy 3 magic beans with the family cow. - hgc
temporalillusion is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 01:08 PM   #165
temporalillusion
Technical Admin
 
temporalillusion's Avatar
 
Join Date: Jul 2006
Location: Canada's Texas
Posts: 1,495
Originally Posted by Donn View Post
Ouch. That takes it out of my range of experience.

I also saw mention of this free backup tool:
http://www.percona.com/software/percona-xtrabackup
.. which looks pretty good.
Just a comment on this, while I LOVE Percona and have used their products in the past, this won't work for the current forum without alterations, as vBulletin 3.x uses MyISAM table types (and that's for InnoDB or XtraDB).

That said I have seen some vBulletin 3.x forums alter their table types to InnoDB, so that might be an option if that was done.
__________________
One man's reason that something is not reliable evidence is another man's whine about how others won't buy 3 magic beans with the family cow. - hgc
temporalillusion is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 01:27 PM   #166
dasmiller
Just the right amount of cowbell
 
dasmiller's Avatar
 
Join Date: Oct 2008
Location: Well past Hither, looking for Yon
Posts: 6,710
Originally Posted by temporalillusion View Post
Huh, that's not nearly as bad as I'd thought, it's not a VPS anyway. RAM isn't too bad, though how it's allocated might be an issue (i.e. if MySQL is only getting 2GB). How much is given to Apache might be part of it as well. CPU could be part of it, and disk might be limiting the database as well. My forum runs off an NFS share for the database drives but they are pretty high I/O. An SSD would of course be huge.
Do we know what else is running on the hardware? I've seen antivirus programs that would use >90% of the system resources for half an hour or more, at random intervals (and that was considered a feature!), rendering an otherwise fast machine completely useless for even things like word processing.

Could we have some rogue mundane process occasionally using all the CPU, or allocating so much RAM that the Forum can't get any and has to resort to disk thrashing?
__________________
"In times of war, we need warriors. But this isn't a war." - Phil Plaitt
dasmiller is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 01:45 PM   #167
temporalillusion
Technical Admin
 
temporalillusion's Avatar
 
Join Date: Jul 2006
Location: Canada's Texas
Posts: 1,495
One last post, I missed a bunch of posts so I apologize for all the replies in order.

So some general my thoughts about the various areas:

Ownership: Be it buying servers via donations, or holding donations to pay for monthly services, the question of who owns what and how they're accountable and such is something to consider. No one wants a situation where someone gets mad and takes their servers and goes home. Not a pleasant thing to think about something that warrants some thought.

Hardware Source: As mentioned you can rent servers or buy and colocate them, so which path will need to be decided.

Reliability: In general (and users will of course disagree) I don't think the forum is a mission critical service requiring five nines of availability. So spending a lot of money on redundancy probably isn't necessary; depending on the service provider's or vendor's warranty and reasonable levels of backups is probably fine. A day of downtime while a server's power supply is replaced or the database drive is replaced and restored from backup is acceptable for the forum IMO.

Hardware: I'd suggest two servers, one for the database and one for the web server. An SSD for the database is a good idea, spindle drive for OS and database logs and SSD for database files. The web server doesn't require as much in the way of disk I/O, so two mirrored drives is sufficient IMO.

OS: CentOS is fine to use, lots of support and info out there. I think CentOS 7 is even available now.

Web Server: Apache works well and I'm not sure nginx is any faster for PHP performance. Static file performance is higher, but I think we should be offloading static resources to a CDN anyway, so Apache is probably fine, but nginx might be fun to use anyway (haven't played with it personally).

Database Server: As mentioned previously, vBulletin 3.x uses MyISAM table types, in which case I'd suggest MariaDB as the database server. It's a drop-in replacement for MySQL but has other things that MySQL either doesn't have, is coming, or is exclusive to the enterprise version. If we converted the table types to InnoDB then we could use Percona which is also a drop in replacement for MySQL. I prefer Percona but maybe that's just because I've been reading the mysqlperformance blog for a long time.

CDN: As mentioned using a CDN like MaxCDN can reduce the load on the HTTP server a lot (most of the requests in a vBulletin page are for static resources). Then the HTTP server can focus on serving up PHP pages. It's relatively cheap and there are a few ways to set it up, someone really skilled with rewrite rules can do it completely via rewrite rules. Personally I just changed the file locations in the vBulletin software itself; seemed easier to me.

Images: Related to that is images in general. These forums allow for images to be uploaded, and icerat mentioned that the forum DB is 32GB which seems quite high to me (my forum is 4.2GB for 5 million posts). If images are stored in the DB then yes that's a significant performance issue, and a storage issue too. In my opinion there are a lot of different free image hosts out there and this is a forum, not an image server. On my forum we simply don't allow image uploads other than avatars, if someone wants to show an image they can upload it to imgur.com themselves. But at bare minimum images should be on the filesystem not in the database, and the CDN could also be used to offload the request for images from the server.

Backups: Technical issues on how to backup the database aside, something like Crashplan ($10/mo business) is inexpensive IMO and gets you offsite backups as well (though Crashplan for Linux is a more complex setup). Some level of backups might also be included with whatever server or colocation plan is obtained.

All I can think of at the moment.
__________________
One man's reason that something is not reliable evidence is another man's whine about how others won't buy 3 magic beans with the family cow. - hgc
temporalillusion is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 01:47 PM   #168
temporalillusion
Technical Admin
 
temporalillusion's Avatar
 
Join Date: Jul 2006
Location: Canada's Texas
Posts: 1,495
Originally Posted by dasmiller View Post
Do we know what else is running on the hardware? I've seen antivirus programs that would use >90% of the system resources for half an hour or more, at random intervals (and that was considered a feature!), rendering an otherwise fast machine completely useless for even things like word processing.

Could we have some rogue mundane process occasionally using all the CPU, or allocating so much RAM that the Forum can't get any and has to resort to disk thrashing?
I don't think anyone has any access to the server as of yet, the stuff icerat posted was forwarded to him if I understand correctly.

So yeah could be something else on the server, heck could be a hacker running a botnet or mining bitcoins!

I doubt the server's been compromised but until one looks one never knows.
__________________
One man's reason that something is not reliable evidence is another man's whine about how others won't buy 3 magic beans with the family cow. - hgc
temporalillusion is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 03:36 PM   #169
Klimax
NWO Cyborg 5960x (subversion VPUNPCKHQDQ)
 
Klimax's Avatar
 
Join Date: Feb 2008
Location: Starship Wanderer - DS9
Posts: 14,282
Originally Posted by temporalillusion View Post
I don't think anyone has any access to the server as of yet, the stuff icerat posted was forwarded to him if I understand correctly.

So yeah could be something else on the server, heck could be a hacker running a botnet or mining bitcoins!

I doubt the server's been compromised but until one looks one never knows.
And it wouldn't be first time... (A tech was caught running bitcoin mining on servers, when they ran too slow.)
__________________
ModBorg

Engine: Ibalgin 400
Klimax is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 3rd August 2014, 07:09 PM   #170
arthwollipot
Observer of Phenomena
Pronouns: he/him
 
arthwollipot's Avatar
 
Join Date: Feb 2005
Location: Ngunnawal Country
Posts: 87,212
I'd just like to say something. A lot of people are posting in this thread, contributing ideas and thoughts. Sharon has asked for people who might be able to help, and that's all very well, but we don't want a huge number of people on the committee unless we want to recreate the forum in the form of a camel.

It seems that everyone feels entitled to input on the discussion. A project like this cannot work that way. A small lean team will be able to get the project completed faster and with fewer complications. This does of course mean that some people are going to feel ignored. I think we as a community need to get over that and provide only the help that is asked for, when it is asked for.

Let's let those who are working work, and not expect them to take every possible opinion into account.
__________________
So take that quantum equation and recalculate the wave by a factor of hoopty doo! The answer is not my problem, it's yours.

Three Word Story Wisdom
arthwollipot is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 05:10 AM   #171
icerat
Philosopher
 
icerat's Avatar
 
Join Date: Apr 2008
Location: sweden
Posts: 5,764
Originally Posted by arthwollipot View Post
I'd just like to say something. A lot of people are posting in this thread, contributing ideas and thoughts. Sharon has asked for people who might be able to help, and that's all very well, but we don't want a huge number of people on the committee unless we want to recreate the forum in the form of a camel.

It seems that everyone feels entitled to input on the discussion. A project like this cannot work that way. A small lean team will be able to get the project completed faster and with fewer complications. This does of course mean that some people are going to feel ignored. I think we as a community need to get over that and provide only the help that is asked for, when it is asked for.

Let's let those who are working work, and not expect them to take every possible opinion into account.
Personally I'm quite happy for people to give whatever their thoughts or ideas are and discuss them here.

Techinical issues are but one area though. From what I understand the general feeling withing JREF is that they would like the forum to be wholly independent of JREF. I don't specifically know why but given the number of lawsuits and threats that have been flying around the skeptic community on various issues of late, I can understand why there might be motivation. Options for doing this while still remaining affiliated (and keeping the name) are being considered.

The question arises as to the organisational form of any future body operating the forum. A new non-profit? Part of an existing non-profit or for-profit entity? Couple of dudes in their mother's basement? None of the above?

Just FYI, the current advertising income does not come close to covering the costs of the forum. While an upgrade to better equipment will, paradoxically, probably result in lower costs, it will likely still not be enough so ongoing funding solutions will need to be found.

Throw in your 2 cents or relevant currency.
__________________
Benford's law of controversy - Passion is inversely proportional to the amount of real information available
icerat is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 05:31 AM   #172
zooterkin
Nitpicking dilettante
Administrator
 
zooterkin's Avatar
 
Join Date: Mar 2007
Location: Berkshire, mostly
Posts: 57,668
I'd rather wait and see what the JREF's position regarding the forum is, before making plans to go solo. As I understand it, TAM would not have happened without the forum. Certainly I would not have heard of TAM otherwise, nor attended half a dozen. The forum provides the volunteers and TAMbassadors that help run TAM, as well as providing scholarships for those unable to afford the cost. Of course, some of could still happen without the forum, but I think participation would be lower.
__________________
The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts.Bertrand Russell
Zooterkin is correct Darat
Nerd! Hokulele
Join the JREF Folders ! Team 13232
Ezekiel 23:20
zooterkin is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 05:43 AM   #173
Spindrift
Time Person of the Year, 2006
 
Spindrift's Avatar
 
Join Date: Apr 2005
Location: Right here!
Posts: 19,246
Originally Posted by arthwollipot View Post
I'd just like to say something. A lot of people are posting in this thread, contributing ideas and thoughts. Sharon has asked for people who might be able to help, and that's all very well, but we don't want a huge number of people on the committee unless we want to recreate the forum in the form of a camel.

It seems that everyone feels entitled to input on the discussion. A project like this cannot work that way. A small lean team will be able to get the project completed faster and with fewer complications. This does of course mean that some people are going to feel ignored. I think we as a community need to get over that and provide only the help that is asked for, when it is asked for.

Let's let those who are working work, and not expect them to take every possible opinion into account.
Should we ask everyone first if they think everyone should be involved?

Why do you hate camels? We are traversing the desert of woo on the camel of skepticism.
__________________
I've always believed that cluelessness evolved as an adaptation to allow the truly appalling to live with themselves. - G. B. Trudeau
A person is smart. People are dumb, panicky dangerous animals and you know it. - Kay, Men in Black.
Enjoy every sandwich. - Warren Zevon
Spindrift is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 05:51 AM   #174
icerat
Philosopher
 
icerat's Avatar
 
Join Date: Apr 2008
Location: sweden
Posts: 5,764
Originally Posted by idoubtit View Post
Mr. Adams, on the JREF board, is able to get in and do some technical tweaks to the hardware and software.
That would be this Mr Adams for those interested. So as you can imagine he has the technical nous but perhaps has other interests he prioritises over this forum

Quote:
One insight he has given me is that right now 40% of the queries are from search engine bots, a good proportion of the traffic is from China. Legitimate traffic?
Quite probably hack attempts. Our servers are regularly attacked by chinese IPs attempting to gain access through various means.

40% of queries from search engine bots sounds a bit high (around 30% is more typical) but certainly within normal range.
__________________
Benford's law of controversy - Passion is inversely proportional to the amount of real information available
icerat is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 06:02 AM   #175
Klimax
NWO Cyborg 5960x (subversion VPUNPCKHQDQ)
 
Klimax's Avatar
 
Join Date: Feb 2008
Location: Starship Wanderer - DS9
Posts: 14,282
Right now, Sucuri proxy is causing problems.
http://www.internationalskeptics.com...1&postcount=11
__________________
ModBorg

Engine: Ibalgin 400
Klimax is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 11:00 AM   #176
BenBurch
Gatekeeper of The Left
 
BenBurch's Avatar
 
Join Date: Sep 2007
Location: The Universe 35.2 ms ahead of this one.
Posts: 37,538
If systems help is needed, consider me a volunteer.
__________________
For what doth it profit a man, to fix one bug, but crash the system?
BenBurch is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 12:18 PM   #177
Soapy Sam
Penultimate Amazing
 
Join Date: Oct 2002
Posts: 28,766
Originally Posted by icerat View Post
Personally I'm quite happy for people to give whatever their thoughts or ideas are and discuss them here.

SNIP

Just FYI, the current advertising income does not come close to covering the costs of the forum. While an upgrade to better equipment will, paradoxically, probably result in lower costs, it will likely still not be enough so ongoing funding solutions will need to be found.

Throw in your 2 cents or relevant currency.
icerat- can you put some numbers on this? (Approximate).
Current
Cost of hardware:-
Cost of software:-
Cost of hosting:-
Cost to maintain:-
Soapy Sam is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 12:43 PM   #178
icerat
Philosopher
 
icerat's Avatar
 
Join Date: Apr 2008
Location: sweden
Posts: 5,764
Originally Posted by Soapy Sam View Post
icerat- can you put some numbers on this? (Approximate).
Current
Cost of hardware:-
Cost of software:-
Cost of hosting:-
Cost to maintain:-
From my current understanding of the system (and I don't have all the facts) I think we could get a way more than good enough dedicated server or VPS for the $250 to $300/mth range.

We'd then want to add some firewalling and backup, maybe another $200.

So we're talking max $500/mth for a better set up than now for hardware/hosting at less cost (which I believe is $500/mth+ not counting firewalling/backup?)

The current vbulletin licence is privately owned, no ongoing costs, but I tend toward upgrading to Xenforo ($250 incl enhanced search) plus a yearly fee for ongoing upgrades/support ($65).

While there's quite a lot of expertise available here I think it's worth paying for some migration software and an experienced tech to handle it. Maybe $500. (ironically some of that is licencing fees to Brian Dunning's partner in crime Shawn Hogan, but that's another story)

So I'd say we're looking at say $1500 for setup and first month, then max $5-600/mth ongoing.

Maintenance is currently all volunteer. Experience to date suggests that might need to change. Personally I'm in favor of pro-moderators/admins on a site this well trafficked. I don't have enough info on the workload to judge costs at this stage.

Right now there is very limited advertising on the site that brings in little revenue. A mix of appropriate advertising plus member donations/subscriptions should be enough to support the site.

My preference - and others I've spoken to seem to feel the same - is that a new non-profit spun off from JREF is probably the way to go. (any legal/accounting folk with time/expertise in setting up and maintaining a non-profit, put your hand up)

I'm also getting the distinct impression there's a lot of backroom history we're dealing with here which has lead to the current somewhat critical situation. That may also make this entire discussion moot and something entirely different happens.

Left hand, meet right hand, pleased to meet you.
__________________
Benford's law of controversy - Passion is inversely proportional to the amount of real information available
icerat is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 01:54 PM   #179
Spindrift
Time Person of the Year, 2006
 
Spindrift's Avatar
 
Join Date: Apr 2005
Location: Right here!
Posts: 19,246
Originally Posted by icerat View Post
From my current understanding of the system (and I don't have all the facts) I think we could get a way more than good enough dedicated server or VPS for the $250 to $300/mth range.

We'd then want to add some firewalling and backup, maybe another $200.

So we're talking max $500/mth for a better set up than now for hardware/hosting at less cost (which I believe is $500/mth+ not counting firewalling/backup?)

The current vbulletin licence is privately owned, no ongoing costs, but I tend toward upgrading to Xenforo ($250 incl enhanced search) plus a yearly fee for ongoing upgrades/support ($65).

While there's quite a lot of expertise available here I think it's worth paying for some migration software and an experienced tech to handle it. Maybe $500. (ironically some of that is licencing fees to Brian Dunning's partner in crime Shawn Hogan, but that's another story)

So I'd say we're looking at say $1500 for setup and first month, then max $5-600/mth ongoing.

Maintenance is currently all volunteer. Experience to date suggests that might need to change. Personally I'm in favor of pro-moderators/admins on a site this well trafficked. I don't have enough info on the workload to judge costs at this stage.

Right now there is very limited advertising on the site that brings in little revenue. A mix of appropriate advertising plus member donations/subscriptions should be enough to support the site.

My preference - and others I've spoken to seem to feel the same - is that a new non-profit spun off from JREF is probably the way to go. (any legal/accounting folk with time/expertise in setting up and maintaining a non-profit, put your hand up)

I'm also getting the distinct impression there's a lot of backroom history we're dealing with here which has lead to the current somewhat critical situation. That may also make this entire discussion moot and something entirely different happens.

Left hand, meet right hand, pleased to meet you.
Somewhat rambling thoughts:

I've worked with a lot of non-profit organizations and have been president of several. While $1500 startup and $6000-$7000 a year does not sound like a lot, it can be to a huge amount to non-profit.

I agree with your point about a pro-admin/mod being needed. Volunteers are great, but their priorities can shift so needed activities don't happen when needed, they get done when they can be done. So you're talking more money. I don't know how much time would be needed for a pro-admin and therefore how much money. I'll throw out a number of $20K per year. I don't know if that's too much or more likely too little.

Will the community here support about $2000+ a month? Advertising can bring in revenue, but how much? I believe we have about 1500 frequent users. Could we get 150 to subscribe at $10-$15 a month? 10% participation in something like that is probably optimistic.

In non-profits it comes down to a matter of whether or not something can be done financially, not whether or not it should be done.

If you're spinning it off to a new non-profit, that can be done for $1000 or less, but would JREF want that?
__________________
I've always believed that cluelessness evolved as an adaptation to allow the truly appalling to live with themselves. - G. B. Trudeau
A person is smart. People are dumb, panicky dangerous animals and you know it. - Kay, Men in Black.
Enjoy every sandwich. - Warren Zevon
Spindrift is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 02:13 PM   #180
rjh01
Gentleman of leisure
Tagger
 
rjh01's Avatar
 
Join Date: May 2005
Location: Flying around in the sky
Posts: 28,092
Making the forum independent of JREF would have huge consequences. Like loss of control of what happens on the forum. Maybe rule 8 could go out the door. Want to sue JREF while still an active member here and boast about it on the forum? Fine. Go ahead.
__________________
This signature is for rent.
rjh01 is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 02:16 PM   #181
George 152
Philosopher
 
Join Date: Jun 2002
Posts: 5,012
The place is running like a dream. No real delays.
Some-one needs a lot of thanks for their work behind the scenes
George 152 is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 02:22 PM   #182
Lisa Simpson
Guest
 
Join Date: Mar 2004
Posts: 21,840
Yes, it's running quicker, but many people still can't log in and I can't access the registration queue, which is growing and growing and growing:

Lisa Simpson is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 02:22 PM   #183
SezMe
post-pre-born
 
SezMe's Avatar
 
Join Date: Dec 2003
Location: Santa Barbara, CA
Posts: 25,183
Originally Posted by icerat View Post
My preference - and others I've spoken to seem to feel the same - is that a new non-profit spun off from JREF is probably the way to go. (any legal/accounting folk with time/expertise in setting up and maintaining a non-profit, put your hand up)

I'm also getting the distinct impression there's a lot of backroom history we're dealing with here which has lead to the current somewhat critical situation. That may also make this entire discussion moot and something entirely different happens.
I would very much like not to see this happen. I know nothing of the backroom but I still am of the opinion that there is a mutually beneficial relationship between JREF and this forum. TAM, as noted, is just one example. I suspect the forum acts as a feed for new JREF members and that that is a two-way street. The spin-off of Skeptical Community is an example to avoid.
SezMe is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 02:24 PM   #184
Spindrift
Time Person of the Year, 2006
 
Spindrift's Avatar
 
Join Date: Apr 2005
Location: Right here!
Posts: 19,246
Originally Posted by SezMe View Post
I would very much like not to see this happen. I know nothing of the backroom but I still am of the opinion that there is a mutually beneficial relationship between JREF and this forum. TAM, as noted, is just one example. I suspect the forum acts as a feed for new JREF members and that that is a two-way street. The spin-off of Skeptical Community is an example to avoid.
I thought the forum provides very few JREF members.
__________________
I've always believed that cluelessness evolved as an adaptation to allow the truly appalling to live with themselves. - G. B. Trudeau
A person is smart. People are dumb, panicky dangerous animals and you know it. - Kay, Men in Black.
Enjoy every sandwich. - Warren Zevon
Spindrift is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 02:36 PM   #185
icerat
Philosopher
 
icerat's Avatar
 
Join Date: Apr 2008
Location: sweden
Posts: 5,764
Originally Posted by Spindrift View Post
Somewhat rambling thoughts:

I've worked with a lot of non-profit organizations and have been president of several. While $1500 startup and $6000-$7000 a year does not sound like a lot, it can be to a huge amount to non-profit.
Yes, been involved in several myself. As I mentioned earlier I'm a fan of developing business systems that allow community organizations to be self-funding and even profitable. The challenge is finding ways to do that. In the US there's actually a formal structure for this now - B-Corporations.

Quote:
I agree with your point about a pro-admin/mod being needed. Volunteers are great, but their priorities can shift so needed activities don't happen when needed, they get done when they can be done. So you're talking more money. I don't know how much time would be needed for a pro-admin and therefore how much money. I'll throw out a number of $20K per year. I don't know if that's too much or more likely too little.
Without having enough info to say for sure, I'm probably willing to have one of my community managers do this (it's part of their role with other communities). As such if the money is there then we willing accept it, if not so be it.

Quote:
Will the community here support about $2000+ a month? Advertising can bring in revenue, but how much? I believe we have about 1500 frequent users. Could we get 150 to subscribe at $10-$15 a month? 10% participation in something like that is probably optimistic.
I agree, however I think there should be an acceptable advertising model that can generate income. What we are doing with other communities is finding community members who have particular products/services that are directly related to that community, and as part of a "pro" membership we co-market to the community.

It has to be done in a way that is acceptable and useful to the community in question. It's also more work, but generates more revenue than things like googleads.

As an example, if this was an independent community it may take a small commission on TAM ticket sales that occur via the site. I'm sure there are other skeptic events around the world that could do with co-marketing through this site.

Quote:
In non-profits it comes down to a matter of whether or not something can be done financially, not whether or not it should be done.

If you're spinning it off to a new non-profit, that can be done for $1000 or less, but would JREF want that?
I think JREF as an organization hasn't yet clearly solidified an answer to that question.
__________________
Benford's law of controversy - Passion is inversely proportional to the amount of real information available
icerat is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 02:44 PM   #186
Childlike Empress
Banned
 
Childlike Empress's Avatar
 
Join Date: Jun 2006
Location: Ivory Tower
Posts: 20,632
Like I suspected and icerat confirmed:

Originally Posted by Childlike Empress View Post
Likely [the problem is current hardware], and not unlikely the contract is a couple of years old given that the forum doesn't seem to have been a JREF priority for the last few years.

Meaning that today you would get a lot more power for the bucks you signed in for, likely enough for the current needs, and I suggest that you lay all of this out to the amazing Randi and let him make the phone call for a new deal.
Originally Posted by icerat View Post
Just FYI, the current advertising income does not come close to covering the costs of the forum. While an upgrade to better equipment will, paradoxically, probably result in lower costs, it will likely still not be enough so ongoing funding solutions will need to be found.

I'd advise to focus on first fixing the immediate problems and pressuring the host a bit (that specs are way above what I expected, there could be simply some defect stuff involved as it was the last time with that router/switch that wasn't fixed for ages) before getting into some pipedreams. I don't doubt that at this stage some serious money could be raised from frustrated forum members, but in the long run it will be difficult to sustain such high running costs including however enthusiastic professional helpers.

Oh, and it seems to be much faster already indeed.
Childlike Empress is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 03:52 PM   #187
Startz
Muse
 
Join Date: Nov 2004
Location: Santa Barbara, CA
Posts: 686
This is a very constructive discussion. I can't contribute anything to the technical side, but a brief comment about money might be helpful.

We have a detailed estimate from icerat that things could be made to run quite well for $500~600 a month. To be way generous, call it $10,000 a year. According to JREF's 2012 990 form, annual JREF revenue is around $1.3 million. Now there are lots of calls on that revenue, but it seems the forum could be kept running for about 1/10th of 1 percent of the JREF budget. This suggests to me the issue is JREF's level of interest, rather than any real money problem.

It would be good to have a frank answer from JREF before worrying too much about funding models.
Startz is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 04:26 PM   #188
wombatwal
Graduate Poster
 
wombatwal's Avatar
 
Join Date: Sep 2006
Location: Van Squad
Posts: 1,484
Originally Posted by Childlike Empress View Post

Oh, and it seems to be much faster already indeed.
That's the power of wanting it to be faster. The Universe responded.
__________________
http://500px.com/Bruce_Wallace

Put him in the van.
wombatwal is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 05:42 PM   #189
jhunter1163
beer-swilling semiliterate
 
jhunter1163's Avatar
 
Join Date: Jul 2006
Location: Connecticut, or King Arthur's Court. Hard to tell sometimes.
Posts: 25,791
Originally Posted by Startz View Post
This is a very constructive discussion. I can't contribute anything to the technical side, but a brief comment about money might be helpful.

We have a detailed estimate from icerat that things could be made to run quite well for $500~600 a month. To be way generous, call it $10,000 a year. According to JREF's 2012 990 form, annual JREF revenue is around $1.3 million. Now there are lots of calls on that revenue, but it seems the forum could be kept running for about 1/10th of 1 percent of the JREF budget. This suggests to me the issue is JREF's level of interest, rather than any real money problem.

It would be good to have a frank answer from JREF before worrying too much about funding models.
I think you're off one decimal place; it would be about 1% of JREF's revenue. Still and all, it's not a huge amount of money, even for a comparatively small non-profit such as JREF.

Could the forum be self-sustaining? Possibly; if the membership put up, say, the hardware cost plus the first year's ongoing expenses, there would be time to develop some sort of model to generate revenue (I hate banner ads like the plague, but I would suffer them to preserve the forum, and I think most here would agree).
jhunter1163 is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 06:01 PM   #190
AdMan
Penultimate Amazing
 
AdMan's Avatar
 
Join Date: Feb 2010
Posts: 10,293
Originally Posted by jhunter1163 View Post
I think you're off one decimal place; it would be about 1% of JREF's revenue. Still and all, it's not a huge amount of money, even for a comparatively small non-profit such as JREF.

Could the forum be self-sustaining? Possibly; if the membership put up, say, the hardware cost plus the first year's ongoing expenses, there would be time to develop some sort of model to generate revenue (I hate banner ads like the plague, but I would suffer them to preserve the forum, and I think most here would agree).

With the number of forum grants to TAM given out each year (20+ this year, for example), I'm quite sure that the forum can probably be self-sustaining just from the income that brings the JREF. I'm hoping they are taking that into account--or maybe they think they can raise more money elsewhere with equal or less effort. I'd be skeptical of that.

That's another reason I agree with those that think it's likely more than just a money issue, but in any case the money issue will need to be dealt with eventually if we do want the forum to continue.

Last edited by AdMan; 4th August 2014 at 06:19 PM.
AdMan is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 07:09 PM   #191
Startz
Muse
 
Join Date: Nov 2004
Location: Santa Barbara, CA
Posts: 686
Originally Posted by jhunter1163 View Post
I think you're off one decimal place; it would be about 1% of JREF's revenue. Still and all, it's not a huge amount of money, even for a comparatively small non-profit such as JREF.
Hey, what's an order of magnitude between friends?
Startz is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 08:49 PM   #192
temporalillusion
Technical Admin
 
temporalillusion's Avatar
 
Join Date: Jul 2006
Location: Canada's Texas
Posts: 1,495
My forum went from a single server to two servers at around the 6-700 concurrent users level (I suspect my forum usage profile has heavier peaks than this one though), so while yeah I was surprised the hardware was more robust than I had thought it was, I still wouldn't be shocked if the slowness was just server scaling.

And even if it is something specific that's eating up resources away from what should get the resources, I still think it's worth moving since you can likely get better servers for cheaper.

But that's still a good point, taking a look at the server and making sure there isn't something else going on should be one of the first things done. vBulletin itself has some cron type jobs it runs on various intervals.

I had an issue once where someone's browser half crashed in a way that was re-submitting the same request to the server many times a second, basically DoSing the server. I figured the user from the IP and emailed them and they rebooted their computer and it stopped. Strange stuff can happen.

But granting volunteers root access to the server is something I can see them being reluctant to do, that's a pretty big amount of trust.
__________________
One man's reason that something is not reliable evidence is another man's whine about how others won't buy 3 magic beans with the family cow. - hgc
temporalillusion is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 10:49 PM   #193
This is The End
 
This is The End's Avatar
 
Join Date: Sep 2007
Posts: 10,924
Originally Posted by icerat View Post
but I tend toward upgrading to Xenforo ($250 incl enhanced search) plus a yearly fee for ongoing upgrades/support ($65).

If the hardware fixes the problems please do not do this. The forum works great (see below post). Do not fix what is not broken.


ETA: And do not overfix what is broken. Like how some computer repair shops actually fix problems, even if it takes some time. Other computer repair shops just instantly reformat and reinstall.
__________________
________________________

Last edited by This is The End; 4th August 2014 at 11:01 PM.
This is The End is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 10:57 PM   #194
This is The End
 
This is The End's Avatar
 
Join Date: Sep 2007
Posts: 10,924
Originally Posted by Digital Dan View Post
Been a developer for client/server database apps and/or web services for the last 25+ years. From the limited info I have seen, I don't think new hardware is required. I think there's a specific root cause, which has not been identified, that is causing it to appear to be a system that has outgrown its hardware.

I have to agree with Dan here.

The forum works absolutely fantastic. Lightning fast page loads. For hours on end. And then randomly it will work horribly slow for hours.

On and off, just like that, for at least 1 year now.

Does not compute.

My first guess would be the large volume of traffic from China and search engines, but it could be some random glitch not yet troubleshot.
__________________
________________________

Last edited by This is The End; 4th August 2014 at 11:03 PM.
This is The End is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 11:07 PM   #195
blobru
Philosopher
 
blobru's Avatar
 
Join Date: May 2007
Posts: 6,900
Yes - the suggestion of a rogue process, sitting in memory and butting in on everything else, that many have made makes sense of the vmstats (the over 200 X as many interrupts as normal, only half as many processes running, free memory shrinking, even before things start to back up midway through the snapshot); certainly worth looking into.
__________________
"Say to them, 'I am Nobody!'" -- Ulysses to the Cyclops

"Never mind. I can't read." -- Hokulele to the Easter Bunny

Last edited by blobru; 4th August 2014 at 11:18 PM.
blobru is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 4th August 2014, 11:55 PM   #196
SezMe
post-pre-born
 
SezMe's Avatar
 
Join Date: Dec 2003
Location: Santa Barbara, CA
Posts: 25,183
Originally Posted by jhunter1163 View Post
Could the forum be self-sustaining? Possibly; if the membership put up, say, the hardware cost plus the first year's ongoing expenses, there would be time to develop some sort of model to generate revenue (I hate banner ads like the plague, but I would suffer them to preserve the forum, and I think most here would agree).
I'd be a bit leery of such an approach. What happens after year one if a revenue model is not found? Is there some way to estimate revenue streams from various business models before starting down one road? Surely, yes, because we can't be the first non-profit to fact this situation.
SezMe is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 5th August 2014, 01:08 AM   #197
Wolfman
Chief Solipsistic
Autosycophant
 
Wolfman's Avatar
 
Join Date: Jan 2007
Location: Vancouver, Canada
Posts: 13,414
I've got a problem...I can view the forum no problem, but every time I try to make a post, I'm taken to the Sucuri Website Firewall, with the following information:

Block details

Your IP: 123.151.32.96
URL: forums.randi.org/newthread.php?do=postthread&f=26
Your Browser: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Firefox/31.0
Block ID: BLACK02
Block reason: Your IP address is listed in our blacklist and blocked from completing this request.
Time: Tue, 05 Aug 2014 02:35:34 -0400
Server ID: cp413

I've had issues with the Sucuri thing before, but never this error message. It seems to arise only when I'm making posts, not when I'm just viewing the forum. Given the previous information that a lot of traffic is coming from China, I'm concerned that perhaps someone changed the security to block Chinese traffic?

I'm currently accessing the forums through a VPN, but that's far from an ideal solution.
__________________
Please check out my business, The Language of Culture
Wolfman is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 5th August 2014, 01:39 AM   #198
Michel H
Banned
 
Join Date: May 2012
Location: Belgium
Posts: 3,398
Originally Posted by idoubtit View Post
Mr. Adams, on the JREF board, is able to get in and do some technical tweaks to the hardware and software.

One insight he has given me is that right now 40% of the queries are from search engine bots, a good proportion of the traffic is from China. Legitimate traffic?
Originally Posted by icerat View Post
...
That would be this Mr Adams for those interested. So as you can imagine he has the technical nous but perhaps has other interests he prioritises over this forum
...
Originally Posted by Lisa Simpson View Post
Yes, it's running quicker, but many people still can't log in and I can't access the registration queue, which is growing and growing and growing:

http://www.internationalskeptics.com...ff92cabfa9.png
Originally Posted by Wolfman View Post
I've got a problem...I can view the forum no problem, but every time I try to make a post, I'm taken to the Sucuri Website Firewall, with the following information:

Block details

Your IP: 123.151.32.96
URL: forums.randi.org/newthread.php?do=postthread&f=26
Your Browser: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Firefox/31.0
Block ID: BLACK02
Block reason: Your IP address is listed in our blacklist and blocked from completing this request.
Time: Tue, 05 Aug 2014 02:35:34 -0400
Server ID: cp413

I've had issues with the Sucuri thing before, but never this error message. It seems to arise only when I'm making posts, not when I'm just viewing the forum. Given the previous information that a lot of traffic is coming from China, I'm concerned that perhaps someone changed the security to block Chinese traffic?

I'm currently accessing the forums through a VPN, but that's far from an ideal solution.
Sharon (idoubtit), I hope Mr. Adams, who is perhaps the person who's been installing the Sucuri website firewall (see here for an explanation) for the forum in recent days, is fully aware of the problems mentioned by Lisa Simpson and Wolfman above. The number of active users is currently rather low (about 325 now, usually about 700 [?]), this may explain why the forum is fast. It would be nice if you could convince this person (or whoever is currently in charge of the hardware) to post in the Forum Help & Member Support sub-forum. In this way, he could explain what he's doing, interact with, and receive advice from the experts of this forum.
Michel H is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 5th August 2014, 02:01 AM   #199
Klimax
NWO Cyborg 5960x (subversion VPUNPCKHQDQ)
 
Klimax's Avatar
 
Join Date: Feb 2008
Location: Starship Wanderer - DS9
Posts: 14,282
Originally Posted by Michel H View Post
Sharon (idoubtit), I hope Mr. Adams, who is perhaps the person who's been installing the Sucuri website firewall (see here for an explanation) for the forum in recent days, is fully aware of the problems mentioned by Lisa Simpson and Wolfman above. The number of active users is currently rather low (about 325 now, usually about 700 [?]), this may explain why the forum is fast. It would be nice if you could convince this person (or whoever is currently in charge of the hardware) to post in the Forum Help & Member Support sub-forum. In this way, he could explain what he's doing, interact with, and receive advice from the experts of this forum.
You missed at least one other problem caused by Sucuri: Missing avatars.
__________________
ModBorg

Engine: Ibalgin 400
Klimax is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 5th August 2014, 02:45 AM   #200
lauwersw
Thinker
 
Join Date: Mar 2008
Posts: 169
Originally Posted by icerat View Post

System Performance

%vmstat 3
Code:
%vmstat 3
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 7  0    252  86180  55660 7092544    0    0   219    71    7    7 26  3 62  9  0
 4  0    252  80972  55740 7094984    0    0   912  1376 1625 2253 20  1 76  3  0
 3  0    252  57136  55820 7100208    0    0  1621   692 1396 3459 21  1 74  4  0
 3  1    252  57728  55824 7094740    0    0  1211   616 1423 2835 19  1 76  3  0
 4  5    252  47628  55536 7052016    0    0  3113  1072 1399 11270 21  3 65 11  0
 2  5    252  47948  55572 7050052    0    0  3379   479 1529 2090 10  1 79 10  0
I'm not an expert on interpreting vmstat but this sample doesn't appear too bad (ie it's not a time when things are lagging badly) but a cpu upgrade would provide some immediate benefits, as I suspect would moving images out of the db, if they're not already.
The last few columns of CPU stats suggest there's plenty of CPU time left. The "wa" column is relatively high, indicating that the OS has to wait for I/O (probably disk, could be network too), but it's nothing problematic. System cpu usage is really low and most work is done in user space, indicating that the application software isn't eating unnecessary kernel resources. To finish there's plenty of CPU time left, so CPU is not the bottleneck for this snapshot. Context switches are fairly high but nothing extraordinary. I have systems that run way higher on all accounts and are still responsive.

It smells indeed like a software scaling problem, which is best solved in the application, but you could improve a lot with better hardware of course. I would concentrate on the disks first, no the CPU. Any chance you can take some "top" snapshots at peak load, or more vmstats, maybe that will show a different picture?

If money allows, go indeed for SSD. Alternatively for disks don't use RAID5: RAID10 or RAID6 are much better for performance and resilience but require more disks. On the other hand RAM is cheap, you could cache most of the DB in memory, only leaving writes loading the disks.

Upgrading from CentOS 5 to 6 might give between 10 to 30% more performance on the same hardware, contributed from different parts of the OS. CentOS 6 supports the ext4 filesystem, CentOS 7 (no personal experience yet) uses XFS by default. Both will give some improvement for MySQL file system performance. The kernel and networking stack will do the rest. Even Upgrading to the latest CentOS version can already help, I've seen about 10% difference in CPU load on a heavily loaded server when upgrading from 5.4 to 5.10.

You should also checkout ktune and tuned, especially on CentOS 6 this might give a free additional benefit.
lauwersw is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Reply

International Skeptics Forum » Welcome to ISF » Other Skeptical Organizations » JREF

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -7. The time now is 09:22 PM.
Powered by vBulletin. Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.

This forum began as part of the James Randi Education Foundation (JREF). However, the forum now exists as
an independent entity with no affiliation with or endorsement by the JREF, including the section in reference to "JREF" topics.

Disclaimer: Messages posted in the Forum are solely the opinion of their authors.