Apologies for the long entry -- tons of stuff to cover, but I've got a cup of green tea and Pandora playing, so let's launch into it:
Why All The Quirky Outages?
<i>* The sound of silence *</i>
<i>* Flowers from the evil seed *</i>
<i>* Of servers crashing *</i>
<i>-- best whistled in trinary</i>
We've been having a fair amount of really quirky behavior on the site recently, and this immediately raises two questions: a) What is going on, and b) why the heck aren't you guys making announcements about it?
The problems in the past week were basically started by one simple thing that led to another and another, like when you go to Starbuck's for a coffee and end up in a chapel with a complete stranger in front of an Elvis impersonator.
<i>Fly, you pools! --ops guy to the memcache servers</i>
But here's what happened: I've mentioned something called memcache before, which is a bunch of computers serving as a fast cache -- basically, before we do any queries to the database, we first see if it's in the cache. Memcache is an amazing bit of programming; over 90% of our database queries are handled by memcache, and if any reason memcache is acting up then the sudden 10x increase in demand makes our database servers go down faster than Fannie Mae stock.
Anyway, we wanted to upgrade to the latest version of memcache, which provided some neat functionality that we could really use. So we switched memcache, but started getting odd reports almost immediately -- nothing reproducible, really, but definitely strange errors.
After much hard work by some devs and ops, we finally discovered that our opcode cache was having problems with the new memcache libraries. (The opcode cache holds the compiled down of our source -- otherwise, PHP has to reinterpret the source every time the page is rendered). So, we swapped the opcode cache (from eAccelerator to APC, if you're interested in the geeky details), and after fixing one or two quirks, it seems to be running pretty smoothly.
Nonetheless, this is just one in a string of various hardware and database failures that we've been dealing with in the past month.
How Can We Keep You Better Informed?
<i>Ignorance is king, and trolls would not prosper by its abdication. --- SF mod</i>
Unfortunately, we don't have many degrees of freedom in terms of announcements. Journal entries aren't really read by many, and it's a bit much to expect users to subscribe to 80 developer journals (although bless the users that read the journals and spread the information -- it helps keep the amount of rampant speculation down). Announcements are sort of a nuclear bomb option that isn't really appropriate for a lot of system messages, and announcements are also tied up with sponsorships, so it's a big deal to make one.
But, for sure we can get better. So the following things are happening:
- in the short term, we're going to start a forum that JUST contains developer updates on different topics. You'll be able to subscribe to any threads that you are particularly interested in.
- we're redoing the main landing page in April, and in the process of doing this we're going to be splitting the announcements into several different types. At this point we're not sure what the exact visuals for system announcements -- we're discussing everything from a tab on a module to a little bouncing icon in the footer -- but at any rate we are going to elevate the visibility of system messages, and make them easier for development to use.
How Come You Never Work On... Rallies and Towns?
<i>Encryption is not necessarily something to be ashamed of — but do it in private and wash your hands afterwards. -- Towns coder</i>
8elly8ean pushed out yesterday another round of improvements for rallies / towns. The lag problems should be substantially better -- please let me know in comments if this isn't the case.
There are even more changes coming for rally -- since 8elly8ean was in the code anyway, we decided to add some things that improve the interface -- much easier and less scary for new users.
How Come You Never Work On... Guilds?
<i>Guilds don't do one thing but bring people together. That's why it's a sin to not work on guilds. -- Guilds coder</i>
Early next week (hopefully Monday) we're pushing out a first set of changes for guilds, that will allow you to assign keywords to your guild. After that we'll be pushing a hugely improved guilds landing page and then looking at some of the guild infrastructure problems.
How Come You Never Work On... Inventory / Customize?
<i>It was the best of ship times, it was the worst of ship times -- poor fieryange1, who has to do an early morning push</i>
Early next week (hopefully in the wee hours of Monday morning), we'll be pushing improved inventory and customize pages (but not inventory arranger, that is still to be looked at). We'll be metering people in, so you might not be able to get into it immediately.
Bugs Fixed This Week
- Entering very long description when creating guild crashes browser
- Guild front pages are having trouble with codes breaking.
- Broken wishlist
- Threads with no titles
- Autosave causes problems when it comes to granting gold
- When an item is added to wishlist for the second time,the message
"already added to wishlist" is missing
- Aquarium doesn't show up as sig when you post journal entry
- Old Arena links takes to white page
- Click on delete button to delete your avatar, the url is pointing avatarsave.gaiaonline instead of www
- When the profile option is set to no and if you're on his friendlist, you can still view the profile
- Drop-down list next to avatar doesn't show up in profile
- Bug fixes for modpay rewrite
- modpay fix
- Previous / Next arrows should always show in Marketplace Inventory
- Gold emotes and granting broken for users in towns.
- Wordbump fixes
Finally
Every italicized quote up there is a mucking of a well known quote from a work of fiction, including science fiction and fantasy. Good luck Googling those!
View User's Journal
Notes From Gaia
Random thoughts from inside Gaia
Panagrammic
Community Member |
User Comments: [38] [add]
|
emorexic-pimp Community Member |
armadillodreamer
Community Member |
|
|
Digital Puppetry Community Member |
iMadame Twinkle
Community Member |
|
|
Eris_Incognito Community Member |
Kamikazi-Cloud
Community Member |
|
|
Ketira Community Member |
Darien S.
Global Moderator |
|
|
Mistress Charlene Community Member |
DawnaV
Community Member |
|
|
Kiyoske Dante Community Member |
Ungoliath
Community Member |
|
|
asylum lane Community Member |
Legend
Community Member |
|
|
Kunai Jones Community Member |
PCLong
Community Member |
|
|
Srintella Tiagara Community Member |
Chocobo Princess
Global Moderator |
|
|
MrsrachaelSnape Community Member |
morozco024
Community Member |
|
|
dreamsk Community Member |
Smaddy
Community Member |
|
|
pH 5 Community Member |
Tevokkia
Community Member |
|
|
Hybrid Community Member |
Naxto
Community Member |
|
|
Asterisk7 Community Member |
Cleocatra
Community Member |
|
|
Akabi Yoru Community Member |
OddBird
Community Member |
|
|
France Hopper Community Member |
SSJ Girl
Community Member |
|
|
DarthDub v2 Community Member |
France Hopper
Community Member |
|
|
Celestial Fireflies Community Member |
ffreddd123
Community Member |
|
|
jollycrazykat11 Community Member |
User Comments: [38] [add]
Community Member