General

Yesterday’s load issues, Toggl Roadmap

Update (05.01.2012).

After 2 days of instability, we have managed to restore all Toggl functionalities to the norm. Feel free to go back in, try it out and give us feedback yourself! Since we’ve seen the load improving, we can start re-adding some functionalities that we temporarily removed also, including sidebar chart and integrations, so keep an eye out!

So now to explain:

  • The root cause for these events was a toxic cocktail of our database hitting its memory limits due to a fast growth that it experienced, and increased traffic which in a most inconvenient way exposed some parts of unoptimized code.
  • As this indicates, we clearly had scalability issues. We then worked on optimizing the code, and at the same time upgraded our hardware. Sadly there was not an easy or quick enough way out of the situation, but it has now been sorted and we are now better prepared and have contingency plans for the future, should this ever happen again.
  • In general, we recommend using either m.toggl.com, Google Chrome offline extension, or the new Toggl Desktop. These work with much improved offline handling, and can be used for time tracking even in the case of server failure.
  • We’re adding similar offline support to Toggl main page in the upcoming months.

Those were nerve-wrecking hours for you, we are sincerely sorry for all of this, and certainly better prepared for similar issues in the future.

For all Pro users, we will add two days of extra usage of Pro features for free to all of you, to somehwat compensate for the two days that you had to deal with troubled Toggl.

……………………

(04.01.2012)

We had major load issues yesterday, which caused Toggl to behave slow and be occasionally unavailable.

The main cause of the issues was the graph/chart on the home page – there was an error in the request for this graph. Unfortunately the existence of this error only emerged yesterday when we had the largest amount of traffic in Toggl’s lifetime, and parallel to that, a very large number of reports were generated.

For the above reason, that chart has temporarily been removed, but will be returned soon.

Due to a considerable increase in the number of Toggl users recently there are also other requests that need optimizing. Also, I admit that we should have coordinated the notifying of our team and users about the load problems more efficiently.

I apologise again for the problems that occurred yesterday. Availablity is a high priority for us and we will continue working towards improving this long-term.

By On January 4, 2012

  1. Glad you share this with us. There are tons of possible problems related to growth and it’s always best you’re honest about that. They’re the growing pains of any good product, totally normal.

  2. All works well, great job guys. As Suzette I have the same problem that the timer icon on the desktop version is not showing. Still works when you click it. No big deal.

  3. Everything seems to be working except the timer icon is gone on the desktop version. It works to click in that area, but when is the icon coming back? All 4 of my developers are having the same issue with the icon not showing up. Thank you for this awesome application, we love it and are completely dependent on it.

  4. The agony is gone, we are back again. Please read this blogpost, as we just added an update. Thank you all for the patience!

  5. Yeah … very slow here if it works at all … but I know that you are working on it … this is really the best time tracker out there.

  6. Yeah, it was definitely seriously glitchy the last couple days, but seems fine now, and even so, is still pretty much the best thing out there! No other time tracker I’ve found is so easy to use, and probably more importantly for me, so easy to change previous entries or when you forget to stop the timer and it keeps counting all weekend! 🙂 (Loving the idea of it reminding me when I’ve done that, btw- could only be better if you could set your own threshold time!)

    Also, the little pop up notification link thing I clicked to get here… perfect. 😀 Yay Toggl!