IdeaScale Blog

As many of you are aware we had a day long outage on Friday. I wanted to give everyone a sense of what happened, what we are doing etc.

On Friday (July 3rd) – there was a fire in Fisher Plaza. Fisher Plaza is a “Communication Hub” in the NorthWest – its host to a bunch of datacenters as well as TV and Radio Stations. The fire caused the automatic sprinker system to kick in and essentially shut down power to one of the buildings.

3:00 AM
We learned about this around 3AM Friday morning. All QuestionPro technical staff were online assessing the situation by about 5AM. Since this was a system wide outage (as opposed to a group of servers failing) – we simply had to asses the situation as it developed.

6:00 AM
We get preliminary indication that the root cause of the power failure is the fire and that no-one is allowed to enter/leave the building till Seattle Fire Department does a sweep. At this point we are all online, waiting for Seattle Fire to clear and give the thumbs up. We redirected traffic to QuestionPro.com to a temporary set of servers with a downtime notice. We asked users to check out our twitter feed for updates as well as our status page (status.surveyanalytics.com)

9:00AM
We get indication that Fisher Electrical contractor is trying to get power back online – drying out the equipment to make sure its safe to operate. We start putting updates on the QuestionPro and IdeaScale twiiter accounts (twitter.com/questionpro and twitter.com/ideascale) – both QP and IS are hosted in the same set of cabinets. The entire building is out of power and its a challenge even to get in and out of the building (no elevators, electronic key cards don’t work etc.)

We also make the determination that we should wait till about 5PM to see of the power comes back online before moving all the data to another data-center. We also have space in a backup data-center.

12:00PM
Fisher and Internap communicate that they are bringing in mobile generators in Flat Bed Trucks – the plan is to get the Generators fired up and bypass the electrical room altogether (where the water damage was)

5:00PM
Engineers are still working on bypassing the electrical room. We decide to wait for a couple of more hours. There are a lot of other issues with moving all the data into the backup data-center – re-configuring the systems would take us a longer time and we run the risk of not having enough servers to handle to load. Our backup systems are meant to store backups (not run the entire load and applications.)

10:00PM
Power is restored to HVAC (heating and cooling) equipment. Power is then slowly turned on to all the customer (our) equipment.

3:00AM
Power comes back online – Our servers start humming – All QuestionPro technical staff is online by 5AM – We start working on making sure all services come back online properly. – By about 6:45AM we are all back to normal.

Twitter – Works like a charm:
We tried to keep everyone abreast of issues as they developed on twitter. We were issuing updates and following updates using the following hashtags – #fisherfire and #fisherplazafire. If anyone every doubted the uselfullness of Twitter in an emergency – this has proven (to me at least) first hand that Twitter indeed is amazingly useful to communicate in the face of an emergency.

Through twitter we found out that we were not the only ones affected by this fire – Some of the other sites that went offline are:

• authorize.net
• bing.com/travel (farecast)
• bigfishgames
• Bartell Drugs
• allrecepies.com

Needless to say, this is pretty big disruption of our services. Both Fisher Plaza and Internap have promised us that they’ll come up with a detailed explanation of the issues and steps to prevent such outages in the future. Meanwhile this also exposed a couple of vulenerabilities on our own preparedness. In the spirit of openness I’ll talk about them – and not only will we talk about it, we’ll also do something about it – and keep you posted on progress.

We will be posting a series of blog posts with the hashtag #fisherplazfire to communicate effectively the steps we are taking to make sure this kind of a distuption does not happen in the future. Like with any system, we cannot make things 100% – but we sure as hell can try.

Short Term Issues:

Communication:
One of the shortcomings we noticed was that our Blog (which is our primary medium of communication) was also hosted within our data-center. This has to change — we;ll be moving our blog (blog.questionpro.com) to a hosted WordPress – Rob Hoehn is in charge of that and will oversee that. We’ll also take this opportunity to segment our blogging – we’ll setup three separate blogs (one for QuestionPro, IdeaScale and MicroPoll.)

Automated Phone Message:
We should be able to deliver the same information (like twitter updates) when people call up. We use Angel.com for our hosted PBX system – We’ll setup the system so we can give out updates when users call in in times of emergencies like this.

Pre-Planned Error Page:
We should have a system in place to switch our systems to an error page (when all hell has broken loose) – we had to scramble at the last minute to setup a separate system (in our backup data-center) to host the error page itself.

Long Term Issues:

Real-Time Data-Center Redundancy:
We have full redundancy _within_ the data-center. So if any one of our servers dies (hard drive failure, etc.) – other servers pick up the slack automatically. If one of our database-servers crash, we have replicated servers that will come online automatically within seconds. However, if the entire data-center goes offline, our current plan does not have a solution to move to another data-center within minutes. We have full copies of the data stored offsite – but that is only the data.

What we need to get to, is to _operate_ out of a different data-center in case of a massive emergency like this. This will undoubtedly will double our operating expenses, but given then business we are in, we simply need to do this. Over the next three months, we’ll be figuring out a solution so that we can sustain turning off power to our primary data-center and things move to our backup data-center.

Finally, I want to acknowledge the patience some of you have shown and understanding many of our customers have shown in the face of this emergency. As the CEO and an Owner of this business I do not take this lightly.

If there is something I can do for you, please feel free to ping me directly – vivek[dot]bhaskaran[at]surveyanalytics[dot]com

I started using google back in 1997 – when I was a student at BYU – We were trying to solve a programming assignment and one of my buddies told me about this cool search engine called Google – better than HotBot, Lycos or even Yahoo. We searched on Google – found the answer to our question on the first page – since been addicted to Google.

Yes – there is Bing – but seriously – as they say in the MS campus – the only people who use MSN/Live Search are the people who have not figured out how to change the default Home page on IE.

Today – however Google failed me – I use adium (IM Client) to connect to all the messaging networks (Adium connects to Yahoo, MSN Messenger, Facebook etc.) – For the last two or three days Adium has not been able to connect to the Yahoo servers. At first obviously I thought this was some issue with my network. I wanted to see if others are experiencing this same issue and what the solution could be:

As with everthing, I googled it – Could not find anything interesting! There were a couple of results, but they were referencing blogs/content created months ago!

Screenshot:

I then searched for the same search terms on Twitter:

I downloaded the nightly build of Adium – and bliss….

We added in a quick nifty feature to allow you to keep connected with your ideascale users. An email relay/broadcast solution.

Each of your IdeaScale portal has a unique email address. Connectivity is simple — just send an email to your unique email address, and the system will email everyone of your users. See screenshot below:

Now the issue for you is deliver relevant and updated content to your passionate users.

Over the past few months, we’ve been involved in the launch of about 20 feedback/idea sites for brands as large as ChoiceHotels, Navteq, Boy Scouts of America to smaller startups like OpenSports.com and Buglabs. Our range of implementations include both public facing sites as well as internal employee only sites. Some have succeeded and some have failed and I’ll share with everyone what we learned through this process.

Some of the things we’ve learned:

Progressive Deployment – Launch the idea-portal internally first. Get everyone in your own company involved in brainstorming and providing constructive feedback and suggestions. The tone of any community is dictated by the tribe – the council – the smaller group of users who are avid fans and who take a lot of time and pride in interacting with you. They also help establish the unwritten rules of the road.

Idea Seeding – Never launch an idea portal without any ideas already seeded. It’s like giving someone a blank whiteboard. Your success rate would me much higher if your Idea portal was seeded with a few ideas of your own. Users will then “get it” immediately and start participating.

Establish Credibility – the easiest way to convince users and establish credibility of an idea portal is to unequivocally message the users about the weight of their ideas. Make sure you tell all your users that you will take action based on input that is received. Senior decision makers and stakeholders will be looking to the idea portal for insights and analysis.

Communicate Back – Send a weekly (or monthly) informal update to all the participants of your community regarding the internal status of where you are with the ideas that have been generated. You can use email or your blog as a vehicle to communicate back to the users about progress and demonstrate your willingness to take this seriously

Staff Appropriately – At least for the launch, have someone dedicated to monitor your feedback portal and them respond to all ideas giving your viewpoint and acknowledging good ideas publicly. This also helps set the tone of your community.

Establish Monthly Drawings/Giveaways – Giving out an incentive (Starbucks/Amazon Gift Card) is a sure-fire way of getting users to pay more attention to you. While most users will not be motivated by incentives alone, they do go a long way in convincing users to “give you a shot.”

Cloud Computing

There has been a lot of chatter around “Cloud computing” recently. Microsoft recently jumped into the bandwagon with their launch of Azure Platform and of course Amazon has been doing utility computing via Amazon S3 for almost 2 years now. Google has it’s AppEngine platform and our favorite Salesforce has force.com – Lots of hype and buzz.

OK – What does that mean to your surveys and services that we offer. We’ve always believed in the SaaS model from day 1 – ok – we tried experimenting with deploying QuestionPro on as a server solution, early on when we launched QP, on customer premises – we failed miserably and it cost everyone an arm and a leg. Lessons in entrepreneurship – Fail often and fail fast!

What Next?

We think the next chapter will be about cloud connectors! Data in the cloud is almost a given now. The question now becomes how will the system in the cloud talk to each other, or how effeciently can services and data interoperate with each other.

Salesforce.comWe had taken the first step in this direction – a small but significant one – The QuestionPro -> Salesforce.com integration is an example of a cloud connector. Your CRM Data is in Salesforce.com and Survey Data in QuestionPro.com — Joining the two together makes sense. What is more important, that we changed recently is that we allowed all QuestionPro licenses (even the Web Professional $15/Month) license to have the Salesforce integration enabled — Scott’s post on Dreamforce and new Salesforce.com  pricing.

QuestionPro.com and IdeaScale.com
This is an easy one – Since we own both the services! We’ve now lauched the integration of IdeaScale with QuestionPro – You can deploy surveys to your IdeaScale community directly. See screenshot below:

Upcoming Cloud Connectors:
This is really up to you. If you’d like us to integrate with other web systems, please feel free to post ideas on our QuestionPro ideascale portal (http://questionpro.ideascale.com) — we’re converging all feature and system requests there and will pick from there and move forward.

For now we have the following in mind:

• IdeaScale -> Facebook
• IdeaScale -> MicroPoll
• QuestionPro -> Google Apps (Google Spreadsheets, Google Docs, Google Presentations)
• QuestionPro -> Zoho Apps
• QuestionPro -> SlideShare.com

Conference Update
As some of you probably know, we are at Dreamforce @ San Franscisco – I’ll be down there tomorrow attending DreamForce and Web 2.0 summit for the next couple of days. We’ll be meeting up and going for some quick drinks after the conference tomorrow. Please feel free to twitter me @questionpro – if you are in the conference (or in downtown SF) and we can invite you to come have drinks with us (we’ll pay of course!) Here is Scott (Left) and Rob (Right) @ DreamForce :

Once again, if you’d like to connect up in Dreamforce pls Direct Message me via Twitter @questionpro and we can meet up.

We are excited to be exhibiting at Salesforce’s Annual conference – Dreamforce next week.
On top of that, we will be releasing our salesforce.com integration functionality within all QuestionPro paid licenses!
Starting at $15/month includes:
> UNLIMITED surveys
> UNLIMITED survey questions
> UNLIMITED survey responses
> Mapping of up to 20 questions per survey to salesforce.com.
> Single Sign-On – develop your surveys from within the salesforce.com interface.
> User-friendly mapping wizard for salesforce.com integration.
If you are attending Dreamforce, please stop by Booth #236!

Last week, Puget Sound Business Journal recognized Survey Analytics as one of the 100 fastest-growing private companies in Washington State.

The annual list compiles growth rates of private companies for the previous three years. Survey Analytics was ranked 12th with 309.9% percent revenue growth from 2005-2007.

Survey Analytics was officially founded in 2003. Most customers know us by our flagship product at QuestionPro.com. In addition to QuestionPro.com, we own and operate a series of other websites, most notably MicroPoll.com and IdeaScale.com. Each site offers a decidedly different service, all rallying around a single theme, ‘offering easy to use tools for collecting meaningful online feedback from your customers’

2008 continues to serve as a monumental year for Survey Analytics…with our recent ranking on Inc Magazine’s 500 Fastest Growing Private Companies list…the release of our crowdsourcing feedback product, IdeaScale…and the continuous evolution of features on all QuestionPro license levels. 

We can’t thank our customers enough for all of their support, respect, and patience over the past 5 years.  More and more positive changes to come with of products, features, and service!  Our ongoing goal of providing feature-rich software for research at an affordable price will continue to evolve as we head into 2009.