Hair pulling

from WinCustomize Forums

This is just unbelievable.  I am sure many fo you, like us, have dealt with plenty of machines that run fine 24/7 over years and years.

Today WinCustomize lost another server to total hardware failure. This is the THIRD server in 3 months. This one in some ways was worse and the timing was particularly bad.

The SQL database was on that machine. Luckily, because of our previous experiences, we have backups. But because of an additional glitch, the auto-backup didn't work which means we lost all the database changes since our last manual backup which was just before we left on the holidays. Net result is that we lost any uploads between the time we went away for Christmas (22nd?) and yesterday.

You can imagine how frustrated we are. Oh, we have a pretty good idea the commonolity on this (all 3 servers were quite very new, from the same provider with the same OS). But we won't get into public finger pointing. 

It's just incredible though. We have el-cheapo servers here that have run 24/7 for the past 5 years without any problems.

We'll have the libraries restored with what we have. If authors who have recently submitted stuff could resubmit their stuff we appreciate it and apologize for the inconvenience. Ironically, we have the files (the crash in November wiped out the physical files but the database was fine). I'm just glad we keep backups.

Thanks for your patience and support. Unfortunatley, we are now down a main server again. This one was our fastest/newest (and most expensive) machine unfortunately.

If anyone needs me, I'll be over here pulling my hair out.

18,289 views 47 replies
Reply #1 from WinCustomize Forums Top
I hear ya Froggie... We have had some problems with our servers in our time. You know, those backup/restore procedures always work until there is a problem.
Reply #2 from WinCustomize Forums Top
Hope everything goes smoothly getting this back up & running.
TouchTasks - Access tasks with ease on Windows touchscreens.
Reply #3 from WinCustomize Forums Top
any estimated times till we get library's available again?
Reply #4 from WinCustomize Forums Top
Ouch
Reply #5 from WinCustomize Forums Top
You haven't offended any gypsies, broken mirrors, cursed God or anything recently have you? Sheesh. Wish you could pass some of that luck on to your hardware provider...
Reply #6 from WinCustomize Forums Top
ooooooooooooooooh noooooooo!!!no no no !!!
Reply #7 from WinCustomize Forums Top
I would spare you my hair to pull but sadly can't spare any. We appreciate the good work you're doing!
Reply #8 from WinCustomize Forums Top
....when it rains, it pours

If you need anything from us let us know
Reply #9 from WinCustomize Forums Top
Please save all hair pulled!! I could use some of it. Anyway good luck and thanks for all your doing. It'll be alright!!
Reply #10 from WinCustomize Forums Top
Firstly id like to say this.

man that sucks and know how you feel i format my hard drive at least 6 times a year because of problems.

secondly i would like a real figure of what it would take to not only replace all of your current servers and thier backups, but also a completely secondary main system and its own backups that mirror the main serers in case both the main server and the main backups fail the sercondary server group can take over without missing a step, and since the secondary group wouldnt be doing anything other than basicly recording the main group it wont be at load all the time.

i ask this because i may be able to help, i have contacts at CDW and other manufacturers and can pull some favors.
I offer not because i want to be the hero but because i cant live without Wincustomize

So please E-mail me with an idea of what i need to round up as far as hardware and i'll see what i can do to help Wincustomize be stronger than ever
Reply #11 from WinCustomize Forums Top
Well that kinda sucks. Have you informed the company about this little glitch? Surely they have warranties or something that covers this kind of thing. Servers should not crash. Certainly not three in three months! Thats insane. Oh well.... Good luck with the restore.
Reply #12 from WinCustomize Forums Top
I think we have all been there, we all want things in life to run smoothly but unfortunately that's never the case. I recently was preparing to do a total system backup when I was distracted and when I went to perform the backup a few days later the Hard drive failed beyond recovery. Fortunately Maxtor replaced the drive but the valuable data is lost forever.
Reply #13 from WinCustomize Forums Top
I hope you get the things fixed soon. Das this mean we get another subscription drive, or maby (hopefully) payed skins?
Reply #14 from WinCustomize Forums Top
i'm all about the payed skins thing too makes me wanna work for those promotions double over
Reply #15 from WinCustomize Forums Top
Don't get me wrong but this smilie is in it's right place -
Reply #16 from WinCustomize Forums Top
Don't get me wrong but this smilie is in it's right place -
Reply #18 from WinCustomize Forums Top
I run a small webhosting business and also work for a large one in Florida. If you would like more information on servers and such please go to http://x-gravity.net

I can possibly get you a discount. email: andy @ x-gravity.net
Reply #19 from WinCustomize Forums Top
Same company, same servers, same OS, 3 down in a month??

/me would be looking for someone elese for my server needs.
Reply #21 from WinCustomize Forums Top
"" Same company, same servers, same OS, 3 down in a month??

/me would be looking for someone elese for my server needs.""

this might sound ironic but i would look for alternitives to the current os and hardware..
Reply #22 from WinCustomize Forums Top
Just wondering...

1. Is the servers owned by service provider? if that's the case they should replace with another new one and provide service. If the hardware is owned by Stardock, then that should have 7x24 service for at least 1-3 years, e.g. from Dell.

2. Backup - have no idea how large is the current skin library, if it is talking about 50-100GB, believe it take 2-3 hours to have full backup every day using current DDS, DLT or LTO technology. We have some DB servers that we need to have 2 full backup daily as the data is quite critical to us. Obviously the skin library is critical to both you and your customers.

3. Consider 7x24 support on software support as well. I think you can may have some statistics that how many times the problem happen to the skin library during weekend, midnight, holidays? I understand this increase workload to your team, but I believe everyone would like this system running 7x24.

Reply #23 from WinCustomize Forums Top
ahhhhh so sorry to hear this guys. > Hope everything's back soon good luck.
Reply #25 from WinCustomize Forums Top

Yea, we own the servers. They're all quite new. They're on warranty but the problem is downtime. Being on warranty still means sending them back or having the repair guy come in and take care of the system which takes time.


I'm just beside myself in frustration because we've always had good luck with servers and yet here we are with 3 in just a few months. 


We should be back up by the end of the day today but things will be slow. The server that died was our main server. The one ultra fast machine that was there.  Even before this happened, we were consolidated to fewer machines than our design was for.


With the subscription drive, we will be in better shape soon -- we'll have new servers. It's just a matter of time of getting them all there.