View User's Journal



Development Update 2/9 Site Outage For 15 Minutes We had a site outage for about 15 minutes the other night, which prompted people with "Could not query config information" and other such illuminating errors. Here's what went on, and here's why there wasn't a message about it: We were revamping part of our user infrastructure, the piece of our network that maps particular database servers to particular users. It was a major change, but pretty well tested, so we were reasonably sure that it wouldn't affect anything, but just to be safe we decided to do the change after peak, at 10pm PDT. You should know that we have a bunch of database clusters. One database cluster in particular that is critical is cluster 9. Cluster 9 holds what we call board config -- basically, it's a database of key/value pairs. It allows us to do changes to the site really quickly; for example: - when we roll out a feature, we always have a board config variable that controls it. That way, we can turn off the feature just by setting the value for that feature to 0. - various small strings that we want to be able to change without having to do a code push. For example, whenever the Going Postal entry is updated, we just change board config to contain the new date. (And, by the way, it's been updated, go check it out). We use board configs all over the place. So, what happened? We rolled out the change, and due to a previously unforseen caching error, cluster 9 immediately was inundated with thousands and thousands of requests, like a pretty geek girl at Fry's. This caused cluster 9 to go belly up... ... which means that all of our code went belly up, telling users "Could not query config information." Why couldn't we do that little system message? Well, where do you think we put the system message? In board config, that's where... on the very database server that was down. If we knew the outage was going to be extended, we would have done an emergency code push to alert the users... but, in truth, it was faster to just revert the change and push out the code again, and at that point we would have been messaging that "There was an error 5 minutes ago, but it's fixed," which seemed kind of hard to explain in that little text area. Valentine's Day Event There were some questions in Site Feedback about the Valentine's Day event, and whether there would be one. Derek mentioned it in AtA, and I'm pretty sure that Fleep mentioned it in some thread-that-I-can't-find-now, so rest assured that there will be a V-Day event; it's in QA now. Which reminds me, I once dated a girl that kept having me do all this stuff for these crazy holidays from her culture, and it was only until years later that I discovered she was making up half of them. Overseer Grants This question also came up in AtA. The aquarium guys did mention that they are constantly looking at Overseer grants and will be adjusting them as time goes on. It's not on a set schedule as much as kind of an ongoing thing. Guilds This morning ania1899 fixed the Captain's Log in guilds, so it should be up and running again. Also keep an eye out for the new keyword capability and some UI changes, coming to you as soon as QA says they're okay. Also, in this latest patch DARKNRGY fixed some of the issues regarding guild invites, so please try it out and let us know if they're working better. Bugs Fixed - The Filter quests is not working correctly - Deactivated Journals Still Visible Sitewide - Buy an item from store without logging in, there's no message displays "you should loggin" - Captain's Log is not saving changes made in guild. - Cleaner URLs for reporting - Fixed accepting join requests by Crew - Journal comments formatting is broken

Posted by: Panagrammic Wed Feb 11, 2009 @ 09:02pm

Read my journal!

User Comments: [22] [add]

Eris_Incognito
Community Member

Offline

Commented on: Wed Feb 11, 2009 @ 09:32pm


	Thank you for letting us know what's going on. That the message system was on the down database server is kind of funny... now that it's over.

Commented on: Wed Feb 11, 2009 @ 11:01pm


	Pan, thanks for all the hard work and updates. I really do appreciate it! Any details on the V-day event? wink Something small? Anything? <3 I'll get you candy if you share something!

Lady Kayura
Community Member

Offline

morozco024
Community Member

Offline

Commented on: Wed Feb 11, 2009 @ 11:02pm


	Thanks Pan for the update.

Commented on: Wed Feb 11, 2009 @ 11:09pm


	So thats what happened. xD! I was nearly having a cow because I thought I was getting hacked. Once I relized i was still in gaia I was fine but it was like I was..inbetween the site if you know what I mean. Anyway all of you do a good job.

Srintella Tiagara
Community Member

Offline

Lady Rat
Community Member

Offline

Commented on: Wed Feb 11, 2009 @ 11:10pm


	the dedication of the staff never ceases to amaze me. of course knowing only half the story, i know us gaians can get pretty upset when something we like doesn't work, but know that there are those of us who do appreciate all the work that goes into this site, even in its stormy times.

Commented on: Wed Feb 11, 2009 @ 11:23pm


	Quote: Which reminds me, I once dated a girl that kept having me do all this stuff for these crazy holidays from her culture, and it was only until years later that I discovered she was making up half of them. Oh noe! gonk xd Thanks for the update Pan, it's always much appreciated. heart 0_o; Just a FYI, we I go to preview my message, it shows the Journal entry up top - but it's shows the actual code and it's all one block of text, and not how it should look like with how you formatted it.

pirhan
Community Member

Offline

asylum lane
Community Member

Offline

Commented on: Thu Feb 12, 2009 @ 12:02am


	Thanks for the update! It's always nice to know what's going on 3nodding

Commented on: Thu Feb 12, 2009 @ 12:11am


	Quote: Which reminds me, I once dated a girl that kept having me do all this stuff for these crazy holidays from her culture, and it was only until years later that I discovered she was making up half of them. rofl lmao! that's funny! evil, but funny. twisted

snufflypoo
Community Member

Offline

pinkgalYall
Community Member

Offline

Commented on: Thu Feb 12, 2009 @ 12:33am


	Quote: Which reminds me, I once dated a girl that kept having me do all this stuff for these crazy holidays from her culture, and it was only until years later that I discovered she was making up half of them. xd She be playing games with your mind. xD Thanks for the update Pan! <3

Commented on: Thu Feb 12, 2009 @ 01:58am


	The thread you're looking for is The Sheep of Fleep, and the post where he mentions the event is on this page.

Katerin
Community Member

Offline

iMadame Twinkle
Community Member

Offline

Commented on: Thu Feb 12, 2009 @ 02:06am


	Thank you for explaining. ^^

Commented on: Thu Feb 12, 2009 @ 02:13am


	Thank you for the update. It is much appreciated. smile

Asterisk7
Community Member

Offline

dreamsk
Community Member

Offline

Commented on: Thu Feb 12, 2009 @ 10:46am


	Maybe you should write the error messages. wink I'm sure nobody would be angry with an error message like: "Could not query config information. Which reminds me of one time I dated a girl..." Hooray for V-day event over the 3-day weekend! Will you have to work on Monday??

Commented on: Thu Feb 12, 2009 @ 11:31am


	Sounds like it's a good thing you guys tried rolling out that code at an off-peak time but hey, outages happen. As for a Valentine's event, I honestly was not expecting one so it's pretty cool you're going to do one. Thanks for keeping us updated Pan smile Captcha: Vodka 494 (that's a lot of vodka bottles.)

France Hopper
Community Member

Offline

Nerelda
Community Member

Offline

Commented on: Thu Feb 12, 2009 @ 09:21pm


	Any info on when the Vday event is going to be? I'm not going to be in town this weekend (and without internet) but would like to be able to participate.

Commented on: Fri Feb 13, 2009 @ 06:48am


	Yay, logging is fixed in guilds now. heart

ffreddd123
Community Member

Offline

Cleocatra
Community Member

Offline

Commented on: Fri Feb 13, 2009 @ 02:44pm


	Those holidays seem very amusing a concept XD I'd love to just randomly celebrate holidays :3 Like.. be a holiday celebrating religion thing. Where you celebrate all religion's holidays biggrin and all internet ones :3!

Commented on: Fri Feb 13, 2009 @ 09:44pm


	"some thread-that-I-can't-find-now" His fanthread possibly? x DD

Sgt.PepperHat
Community Member

Offline

DogCow
Community Member

Offline

Commented on: Sat Feb 14, 2009 @ 12:29am


	Yay, board config! A relic of phpBB! razz

Commented on: Mon Feb 16, 2009 @ 04:33am


	I'm pretty satisified with Gaia, you guys are doing a good job. <3

Death the Kid CDO
Community Member

Offline

Bunny Cakes[MB]
Community Member

Offline

Commented on: Wed Feb 18, 2009 @ 07:47am


	i just wanted to make a note again the Week 44 in the original avatar arena did not pay out. i mentioned it a few weeks ago and you PMed me with a few questions but i've heard nothing else and gotten no prize yet so i figured it was worth mentioning. Also thanks for letting us in on the inner workings of gaia, it really helps users to appreciate all your hard work. smile

Commented on: Thu Feb 19, 2009 @ 07:25am


	Just to have it said: I still get an error message in my guild, when I delete or move a post/topic. "Database Error No log message was defined for crew logging."

Hankiebeer
Community Member

Offline

User Comments: [22] [add]

Welcome to Gaia! :: View User's Journal | Gaia Journals

More Information