Printable Version of Topic
Click here to view this topic in its original format
Celtic Radio Community > Technical Support > Hiccup Tonight


Posted by: CelticRadio 11-Jun-2008, 08:32 PM
We had a brief hiccup tonight with our database. Luckily we had a recent copy and we were able to rebuild about 59 lost topics using a script we developed last year for this very purpose.

The only problem is that some of the topic titles were lost. So we updated those threads with what we think were the topic titles! biggrin.gif

I think this was the fastest recovery we ever had from a database crash!

Posted by: TheCarolinaScotsman 11-Jun-2008, 09:12 PM
The music didn't go down; that's the most important thing. wink.gif biggrin.gif thumbs_up.gif note.gif

Posted by: CelticRadio 11-Jun-2008, 10:50 PM
Yes, very important! biggrin.gif

We are working on a behind the scenes admin panel / watch dog so that the lovely Killian can monitor the entire operation during the day and spot troubles right away.

Also we have re-enabled nightly backups of the database which is probably a good idea!




Posted by: Lady of Avalon 12-Jun-2008, 04:25 AM
Yes we lost some of the posts (64) in the historical archives as well because i posted last night in it and there is only 33 posts showing and some are missing also in the celtic heart forum, the poem "The blade" shows 0 replies when in fact there was quite more.

It seems like I see an old page of celtic radio.

LOA

Posted by: CelticRadio 12-Jun-2008, 05:23 PM
Let's give it another try. Looks like the database crashed again, although I can see that we did not completed one little aspect of the database recovery which might have caused this latest problem.

Posted by: Camac 12-Jun-2008, 06:21 PM
QUOTE (Lady of Avalon @ 12-Jun-2008, 05:25 AM)
Yes we lost some of the posts (64) in the historical archives as well because i posted last night in it and there is only 33 posts showing and some are missing also in the celtic heart forum, the poem "The blade" shows 0 replay when in fact there was quite more.

It seems like I see an old page of celtic radio.

LOA

LOA;

What happened? I go get 9 teeth removed and disaster strikes. I guess I shouldn't have screamed so loud.


Camac.

Posted by: CelticRadio 12-Jun-2008, 06:40 PM
I'm not sure if 64 posts where lost. Could you describe the posts, the issue is only effecting the topic table, no posts have been lost so we can recreate which we have done.

We had a bit of weirdness on our server, seems strange it happened at the same time some major security updates happened on our radio server. We also did a few more things today that we should have done yesterday. Please, if this happens send an email to us right away. Killian will be monitoring all day tomorrow.

No worries, we have plenty of backups and are running nightly backups of all data and programs. We also have remote backups too!

Could you let us know when you go to the dentist again? Not sure if that had something to do with it! biggrin.gif

Posted by: A Shrule Egan 12-Jun-2008, 07:51 PM
Lost a few posts in the General Discussion/Ireland thread, that were posted between 6 PM and 7:30 PM. Also, a few that UlsterScotNutt posted during the late morning.

I show them under my profile as making the posts but they do not come up when you click on the thread. It's like there are two seperate threads for the same topic.

Posted by: CelticRadio 12-Jun-2008, 09:10 PM
We updated the title names of the topics as our "rebuilding" script uses all of the posts to rebuild the topic names. I think we counted 59 missing topics last night, but they should have been rebuilt. But can't rule out other missing stuff.

Well, we have a nightly backup of the database scheduled for 2AM every night now, so if it happens again we should be adequetely protected.

Thanks for everyone checking out the forums and letting us know about any problems. We will try to get them fixed ASAP.

Posted by: TheCarolinaScotsman 12-Jun-2008, 09:47 PM
Be aware that since the second hiccup, the count on replies to topics may be off. I've noticed that in some topics (including this one) that the replies before the second outage are not included in the total count even though they are still here.


Posted by: CelticRadio 12-Jun-2008, 09:51 PM
Hmm, might have to call in Bill Gates! biggrin.gif

Thanks for this tid-bit. Did not know that......will be looking at these database tables again.........

Posted by: CelticRadio 12-Jun-2008, 10:03 PM
Also, just so everyone's knows, we rebuilt/repaired the database using a method which we believe will prevent the topic table from crashing again. This seems to be an issue atleast once a year with the topic table.

This repair used a new method we had not used in the past and actually rebuilt the index for the topics. There could be some missing posts / topics / reply count - but if it works it is probably worth the minor inconvenience if we stop these crashes from happening every so often.

Posted by: UlsterScotNutt 13-Jun-2008, 09:43 AM
Wow, little do we realise how much something means to us til its gone!!
Some topics show locked such as TamiMcLeods word of the week and Confederate flag issues, along with days worth of posts missing as A Shrule Egan mentioned in Ireland also, Gas Costs and a few others.

USN

Posted by: UlsterScotNutt 13-Jun-2008, 10:05 AM
Under Celtic Hearts, Title, Description posted by Camac, if you go to reply, it says menu disabled, topic locked.
USN

Posted by: UlsterScotNutt 13-Jun-2008, 10:29 AM
Posted on What are you doing right now... and it accepted my reply but the post and several of the last pages are missing.

Posted by: UlsterScotNutt 13-Jun-2008, 12:45 PM
I am getting subscription notifications of new posted emails but nothing new is showing up in the topic???

Posted by: Lady of Avalon 13-Jun-2008, 04:00 PM
QUOTE (CelticRadio @ 12-Jun-2008, 08:40 PM)
I'm not sure if 64 posts where lost. Could you describe the posts, the issue is only effecting the topic table, no posts have been lost so we can recreate which we have done.

We had a bit of weirdness on our server, seems strange it happened at the same time some major security updates happened on our radio server. We also did a few more things today that we should have done yesterday. Please, if this happens send an email to us right away. Killian will be monitoring all day tomorrow.

No worries, we have plenty of backups and are running nightly backups of all data and programs. We also have remote backups too!

Could you let us know when you go to the dentist again? Not sure if that had something to do with it! biggrin.gif

Hi Paul, I did not say we lost 64 posts in the topic " The war of 1812", I said we lost some we had 64 and now we are down to 33 which is impossible because it seems that all of the posts after june 5th are losts and we did quite a lot of posting after that.
All of UlsterScottNutt are not there and Camac response to them as well as mine.
And there was one from DesertRose as well who said that she enjoy reading about our topic and the last one that I posted myself which was addressed to Ulster.

As for you Camac, I did answer your posts in regards of your Mommy and lollipop and as I see it strangely disapeared as well....and no you got no sympathy from me. smile.gif Man....

Posted by: CelticRadio 13-Jun-2008, 04:54 PM
Thanks for everyones help. Please keep posting problems here and we will evaluate.

I think for the most part, we are not seeing severe issues and that our database rebuild was successful with minimal loss in data.

Going forward we are running nightly DB backups.

We are going to do some testing this weekend to determine if the topic tables has been really fixed this time!

Posted by: stoirmeil 13-Jun-2008, 08:07 PM
A strange little thing is happening that I dont know if its my computer of the site -- the apostrophe is opening a little search window in the lower left of the screen instead of typing an apostrophe.

Posted by: Patch 13-Jun-2008, 10:36 PM
QUOTE (UlsterScotNutt @ 13-Jun-2008, 05:29 AM)
Posted on What are you doing right now... and it accepted my reply but the post and several of the last pages are missing.

Posts that are accepted but not "shown" actually appear on the add reply page under the reply box as last replies. They just do not show up on the topic subject pages. Hope this makes sense.

Slàinte,    

Patch    

Posted by: Patch 13-Jun-2008, 10:38 PM
QUOTE (Patch @ 13-Jun-2008, 05:36 PM)
Posts that are accepted but not "shown" actually appear on the add reply page under the reply box as last replies. They just do not show up on the topic subject pages. Hope this makes sense.

Slàinte,    

Patch    

OK, now it worked.

Slàinte,    

Patch    

Posted by: CelticRadio 14-Jun-2008, 02:49 AM
QUOTE (stoirmeil @ 13-Jun-2008, 10:07 PM)
A strange little thing is happening that I dont know if its my computer of the site -- the apostrophe is opening a little search window in the lower left of the screen instead of typing an apostrophe.

Does not seem like a virus, but perhaps a hijack. I would definitely run some virus scan or perhaps a spyware remover.

MS Defender is a free utility if you have windows XP or vista that might do the job.

Posted by: CelticRadio 14-Jun-2008, 02:50 AM
So far, so good. I detected a little bit of a performance increase in the new build.

Posted by: Lady of Avalon 14-Jun-2008, 05:14 AM
Hi Paul,

I went to the "Historical Archives" and in the topic "The War of 1812" I did what Patch suggested for me to do if I wanted to see the last posts.
I clicked the button "reply" and I saw the last 10 posts that I thought were losts but there was at least 10 previous that I still cannot see.

Which I know they are missing because there are ones from me that I answered to members and they are not there.
So if we are able to see the last 10 by clicking the "reply" button but is there any other way to see what's in between?

Thanks for all your effort Paul. thumbs_up.gif LOA

Posted by: UlsterScotNutt 14-Jun-2008, 08:44 AM
Thanks Patch, didn't think to do that, duh. at least we see the last posts by looking below our current reply. There are still issues with them showing up on the topic forum for all the public to see unless you reply and read them
My Lady, Yes, we are missing lots of posts in between.
I am using my sons custom, King Kong on steroids computer at home with more anti softwares than NASA and still same glitches as on my work computer.

Off I go, he needs to design some CAD parts.

Best regards,
USN

Posted by: Patch 14-Jun-2008, 08:52 AM
I noticed that "what is the price of gasoline" is not adding posts yet

Slàinte,    

Patch    

Posted by: A Shrule Egan 14-Jun-2008, 12:13 PM
It appears that anything that was posted after June 6, up to the time of the crash, has been lost. I haven't checked all the threads but it does seem to be a pattern.

Posted by: Patch 14-Jun-2008, 02:21 PM
QUOTE (UlsterScotNutt @ 14-Jun-2008, 03:44 AM)
Thanks Patch, didn't think to do that, duh. at least we see the last posts by looking below our current reply. There are still issues with them showing up on the topic forum for all the public to see unless you reply and read them
My Lady, Yes, we are missing lots of posts in between.
I am using my sons custom, King Kong on steroids computer at home with more anti softwares than NASA and still same glitches as on my work computer.

Off I go, he needs to design some CAD parts.

Best regards,
USN

If the Chinese can "hack" into the Capitol computers, there isn't much hope for the rest of us. I have some pretty sophisticated protection and still occasionally something slips by. I get a warning and I disable the net. A scan usually turns up a mid level info cookie.

As money gets tight it seems people come up with more scams. Too much idle time creates problems in a few people too.

Slàinte,    

Patch    

Posted by: Patch 14-Jun-2008, 02:29 PM
Also, "what are you doing right now" is not posting replies.

Slàinte,    

Patch    

Posted by: TheCarolinaScotsman 14-Jun-2008, 06:55 PM
I tried to post in recipes, but it showed an error and said the topic was locked. I wanted to answer Shadow's question about wasting server space, so I'll say it here and hope he sees it.

Shadows, even if no one is posting that doesn't mean folks aren't consulting the recipes for that "just right dish". Whether active or as a reference, the food section is an invaluble resource.

Posted by: Rindy 15-Jun-2008, 12:29 PM
I just posted in Word Association page 244. If I click on Word Association from Fun & Games then Word Association Last post-it isn't there. But if I click on my profile most recent posts it's there but not on the right page.

May be it would be easiest to start this thread all over again?

Slainte

Posted by: A Shrule Egan 15-Jun-2008, 01:41 PM
Just replied in the General Discussion/Ireland thread and my post isn't there. I think we still have a hiccup.

Posted by: Aaediwen 15-Jun-2008, 01:45 PM
testing!

Posted by: ctbard 15-Jun-2008, 04:21 PM
Yes, there"s definitely is alot of problems still.

Posted by: CelticRadio 15-Jun-2008, 07:46 PM
Opps, we forgot to run a follow up script once we rebuilt the database.

I was browsing our library of fixes and other types of programs and ran across something called reset topic posts!

Please check out the forum and let me know if this fixed the issues! biggrin.gif

Posted by: UlsterScotNutt 16-Jun-2008, 07:06 AM
Fathers Day, what are you doing... is locked.

All these little I's and O's running around like mad creatures, chaos I tell you.
USN

Posted by: UlsterScotNutt 16-Jun-2008, 07:11 AM
QUOTE (UlsterScotNutt @ 16-Jun-2008, 08:06 AM)
Fathers Day, what are you doing... is locked.

All these little I's and O's running around like mad creatures, chaos I tell you.
USN

Another thing I noticed, I posted at 9:06 and the puter says 9:06 but my reply says 8:06 and the reply before me shows what I think would be a correct time of 8:46. Don't know if that's of any consequence.

Posted by: Camac 16-Jun-2008, 08:06 PM
In Celtic Heart you have Camac listed as the starter of Atrocity Sea it should be John Clements, Also for Camac the poem Would is missing. For a day or two it was listed as Title.
Under Medeval Gate way- The War of 1812 a number of newer postings by me are missing.

Camac.

Posted by: CelticRadio 16-Jun-2008, 09:09 PM
Good! It seems like the problems are limited to only a few posts at this point.

Any closed topics can easily be opened, just contact a super-moderator or post it here.

Thats for everyone reporting the issues!

Posted by: subhuman 16-Jun-2008, 09:30 PM
QUOTE
Hmm, might have to call in Bill Gates!

Oh, please Mac- tell me you're using http://en.wikipedia.org/wiki/MySQL and not MSSQL. That alone could explain your entire problem.
MySQL-
*open source- enough said. I'll take OSS over closed source any day.
*cross-platform. AIX, BSDi, FreeBSD, HP-UX, i5/OS, Linux, Mac OS X, NetBSD, Novell NetWare, OpenBSD, eComStation , OS/2 Warp, QNX, IRIX, Solaris, SunOS, SCO OpenServer, SCO UnixWare, Tru64, Windows 95, Windows 98, Windows ME, Windows NT, Windows 2000, Windows XP, and the 32-bit version of Windows Vista
*less resources required than MSSQL
*more reliable than MSSQL

Community Edition is free, licensed under the GPL. That's all you need- the main advantages of Enterprise Edition are the tech support and also support for more concurrent connections. IPB uses very few connections to the DB, well under the approximately 500 limit in Community Edition.

I also can't stress the importance of a backup SQL server. Either doing live mirroring, or periodic scheduled backups. Both methods can be configured fairly easily. Don't even *think* of mentioning cost, as some of the backup systems I've made in the past were P-333 systems with as little as 128MB/RAM. My current backup SQL server is a Celeron 2.66GHz that I literally made out of spare parts, running eComStation with 256MB/RAM. It outperforms the three quad-core Vista system (all with 1-2 gigs RAM) I've pitted it against.

Moral: for a machine making backups, you don't need blazing speed or all the "bells and whistles." Use older (read: inexpensive) hardware and a matching OS and you'll still get the performance you need.

Posted by: CelticRadio 16-Jun-2008, 09:40 PM
Yes we are using MYsql and backups are running nightly and weekly at a remote location.

The problem was with a table that we have had issues with in the past, but we rebuilt the table index and I think that solved the problem.

We lost about 54 posts, but those were rebuilt using a script I wrote that examines the post and rebuilds the topic.

All and all I think we are in pretty good shape other than some issues which I think we would have anyways if we did not rebuild the topic index.

Thanks very much for your suggestions!

Posted by: UlsterScotNutt 17-Jun-2008, 07:08 AM
It is like listening in on great wizards conversations!!

Thank you for this site.

hiphiphooray, back on line.
USN thumbs_up.gif note.gif beer_mug.gif smile.gif

Posted by: Lady of Avalon 19-Jun-2008, 11:34 AM
Hi all,

Is it possible to restore or have the front page in the historical archives ?

In the topic "thewar of 1812" it shows last post on 30-May by Camac when in reality it is 11-june by Lady of Avalon.

Thanks LOA

Powered by Invision Power Board (https://www.invisionboard.com)
© Invision Power Services (https://www.invisionpower.com)