Update (05.01.2012).
After 2 days of instability, we have managed to restore all Toggl functionalities to the norm. Feel free to go back in, try it out and give us feedback yourself! Since we’ve seen the load improving, we can start re-adding some functionalities that we temporarily removed also, including sidebar chart and integrations, so keep an eye out!
So now to explain:
- The root cause for these events was a toxic cocktail of our database hitting its memory limits due to a fast growth that it experienced, and increased traffic which in a most inconvenient way exposed some parts of unoptimized code.
- As this indicates, we clearly had scalability issues. We then worked on optimizing the code, and at the same time upgraded our hardware. Sadly there was not an easy or quick enough way out of the situation, but it has now been sorted and we are now better prepared and have contingency plans for the future, should this ever happen again.
- In general, we recommend using either m.toggl.com, Google Chrome offline extension (http://bit.ly/x7y1S1), or the new Toggl Desktop (http://bit.ly/zjAUMz). These work with much improved offline handling, and can be used for time tracking even in the case of server failure.
- We’re adding similar offline support to Toggl main page in the upcoming months.
Those were nerve-wrecking hours for you, we are sincerely sorry for all of this, and certainly better prepared for similar issues in the future.
For all Pro users, we will add two days of extra usage of Pro features for free to all of you, to somehwat compensate for the two days that you had to deal with troubled Toggl.
……………………
(04.01.2012)
We had major load issues yesterday, which caused Toggl to behave slow and be occasionally unavailable.
The main cause of the issues was the graph/chart on the home page – there was an error in the request for this graph. Unfortunately the existence of this error only emerged yesterday when we had the largest amount of traffic in Toggl’s lifetime, and parallel to that, a very large number of reports were generated.
For the above reason, that chart has temporarily been removed, but will be returned soon.
Due to a considerable increase in the number of Toggl users recently there are also other requests that need optimizing. Also, I admit that we should have coordinated the notifying of our team and users about the load problems more efficiently.
I apologise again for the problems that occurred yesterday. Availablity is a high priority for us and we will continue working towards improving this long-term.
Regular updates on this particular issue are and will continue to be posted in the public forum here: http://support.toggl.com/discussions/questions/386-toggl-difficulties
PS. We released a draft version of 2012 Q1 Toggl Development Roadmap to make Toggl development process a bit more transparent.
FYI, I noticed this morning that (for me) yesterday’s afternoon entries were gone, and a timer for the last event of the morning was still running.
FYI, last hour it es happening something like that again.
It seems the system is unstable again this morning.
Agreed – site was slow again loading this morning.
I can confirm issues today
The site still seems with issues today. I just lost a couple of time tracking entries.
I’m still have some troubles to add entries.
No graph today and im still having issues.
I have some entries being displayed in one window and not being show in a report, and now some entries from yesterday disappeared, I hope this all gets straightened out and I did not loose any entries. Toggl has been rock solid for me for a couple of years I hope it returns to its stability quickly.
The same issues as yesterday seem to be acting up again, right now.
Serious issues today aswell. Desktop app unusable.
Is there a way we can log our time spent logging in to toggl??
I confirm that there are still issues (slow, problems saving entries, etc.)
It’s still happening today, Jan. 4. I can barely clock in and out and it’s still rather unreliable. I like the site, though — just wished it worked.
This website is unusable today 1/4/2012
I’m struggling to get the last of my December billing done today. Timeouts and hangs.
Difficult.
But you guys are awesome.
Toggl’s one of my favorite and most used applications.
.
.
.
The online app’s been crashing all day today, as well…
All entries were saved for me yesterday.
In addition to the load issues we have the problem that some old time entries disappeared. In my company we have three employees who lost time entries from december. That’s a serious deal killer.
It’s still slow today.
The desktop app is unusable, it won’t let me login. I have to login into a browser window and track my time from there. Hope it gets resolved soon. Thanks!
Super slow through the browser today.
May not be relevant – is Firefox 9.0.1 not playing well with Toggl? The above thread discusses issue on Toggl’s end, but just wanted to ask to rule out coincidental timing of update of Firefox web browser.
Hitting the Stop button apparently doesn’t register but after a short time, entries show up. So far, manageable. It’s a lovely app,
We will disable reporting for the next 3 hours to bring load back down.
Sorry it took so long. We will continue working with the system for the upcoming hours, in order to resolve the issue for tomorrow.
Like others, I’m seeing problems today too. It keeps “going offline,” and I’m worried it’s not counting all the time entries. Since every lost minute is lost money, I won’t be sticking around if this doesn’t get fixed immediately.
Still slow today – but I figure it’s just like going to the gym during the month of January. All of those new year’s resolutions to get on top of time management naturally brought people to Toggl, and things might be a little more harried. You guys have an amazing app, though. Your big red button has made my time has been so much easier to manager (and bill).
For those who are getting impatient with the new year’s onslaught – relax. That’s what backup systems (or backup gym memberships) are for.
And Toggle will get this right.
Very slow today. I am unable to create time entries.
Please use m.toggl.com for time tracking ATM. It is better suited for offline use.
web interface still very slow here, time updates fail all the time.
The website is still freezing up today 1/14/12.
I have lost entries made today. I keep getting knocked out and having problems logging back in.
I was a Toggl advocate at my company until it dumped my entries, that I can’t recreate, and left me trying to put my day back in.
My company uses Toggl to track PTO and uses it in processing Payroll. My guess is they won’t use Toggl very long.
I am missing the reports option completely now.
We have temporarily removed reports and integrations in the hopes of making the home page tracking more reliable and faster. They will be back in a couple of hours, we’ll let you know. More details here: http://support.toggl.com/discussions/questions/386-toggl-difficulties
Not working for me either (takes forever to log in). And reports is totally gone, which is unfortunate because a client keeps asking me for invoice to close out their december books, and I can’t generate it because I can’t see how many hours I worked for them. Fix ASAP!
Were are the reports???
for me too – the site is still mostly failing for me and i’m back to time tracking by hand. bummer!
@ Yan, Firebus & Craig:
We have temporarily removed access to reports and integrations in the hopes of making the home page tracking more reliable and faster. They will be back in a couple of hours, we’ll let you know. More details here: http://support.toggl.com/discussions/questions/386-toggl-difficulties
Wouldn’t you know it we launched a new customer service department yesterday and are using Toggl to track our service times and volume… what terrible timing
It’s completely unstable for us again today.
Ask your team to use https://m.toggl.com while we’re working on http://www.toggl.com – m.toggl has better offline support and is not affected.
Really slow again, was fast about an hour or so ago, but I come back to it now and very very poor again. Desktop app is failing to connect when asked and causing log corruption because of it.
We are still having issues today. Would appreciate it if this could be fixed. We are considering renewing Toggl for use in the next year. Problems like this make our organization think about trying someone else’s solution for time tracking instead of Toggl.
The reports feature is gone. I forget what they called it, but it was available from a link on the right side of the page. You could go there and see totals for the month, week, and day. You could also generate reports. Is that gone for good? Or has it just been turned off while the bugs are being ironed out?
You guys need a better failover strategy. When you offer SAAS that businesses plugin to stable software (like Basecamp) and upon which billing or timetracking are based, when you have major issues you need to be able to get your customers on something stable quickly.
Geez i thought i was the only one who experience this issue, please fix it ASAP.
Thank you and I’m sorry, the problem has turned out to be much more complicated than we anticipated. We are working on figuring this out and come up with improved contingency plans as we proceed. We will most definitely inform you when the time comes.
Reports are available again, but please be prepared for their requests to take an unusually long time – up to 20 seconds even.
We’re really sorry again – we will keep working on it.
We’re going in for some downtime soon. Server maintenance is planned, which should improve the situation. More details in 10 minutes.
I am still having time entries disappear or stop clocking. Please fix this issue.
Please follow this thread for updates: http://support.toggl.com/discussions/questions/386-toggl-difficulties
Thanks for all the work you are doing. Toggl saves me hours of work. I am sure you do everything to get the service back working as it had in the past.
Thanks for letting us know.
its working !!!
You guys are awesome – keep at it and I’m not going anywhere
Im starting to think to go back to the old version
Still having issues with slow loading today 1/5
Likewise, I am still having major issues today. (1/5)
Hi, I’m still having major issues today as well. May we have an update?
Yeah, it was definitely seriously glitchy the last couple days, but seems fine now, and even so, is still pretty much the best thing out there! No other time tracker I’ve found is so easy to use, and probably more importantly for me, so easy to change previous entries or when you forget to stop the timer and it keeps counting all weekend!
(Loving the idea of it reminding me when I’ve done that, btw- could only be better if you could set your own threshold time!)
Also, the little pop up notification link thing I clicked to get here… perfect.
Yay Toggl!
Come on Toggl Team- love the service, empathize with server load, but let’s
“get’er done”
Yeah … very slow here if it works at all … but I know that you are working on it … this is really the best time tracker out there.
The agony is gone, we are back again. Please read this blogpost, as we just added an update. Thank you all for the patience!
Everything seems to be working except the timer icon is gone on the desktop version. It works to click in that area, but when is the icon coming back? All 4 of my developers are having the same issue with the icon not showing up. Thank you for this awesome application, we love it and are completely dependent on it.
All works well, great job guys. As Suzette I have the same problem that the timer icon on the desktop version is not showing. Still works when you click it. No big deal.
Much, much better now chaps, very happy
Thank you!
Site down again. This is really lousy ;(
@Vjeran, we have not noticed any stability issues lately. Is the issue still persisting?
Glad you share this with us. There are tons of possible problems related to growth and it’s always best you’re honest about that. They’re the growing pains of any good product, totally normal.