Toggl

Information About Desktop App Syncing Issues on October 1st

Starting on October 1st, Toggl Desktop users were experiencing a lot of issues with the apps. These issues resulted from recent updates we rolled out and we are very sorry about that.

Below, you’ll find notes on what caused the issues, how you can avoid the issues, and the steps we took (and are taking) to resolve them and get things back to normal.

How can I fix this now?

On Windows and Mac – Get the latest version as soon as possible (7.4.1023) download Windows 32bit, download Windows 64bit, download Mac 64bit.

The faster we remove the faulty versions from circulation, the faster we can relieve the limit for sync requests and everyone can continue working as normal.

On Linux – There are no current issues with this app; however, users on Linux may see some sync limiting, as we have had to limit syncing abilities across all desktop app platforms.

The whole story

On the 1st of October, our backend monitoring system started sending alerts that the load on our servers was increasing at a fast pace. The backend team started to investigate right away and raised an alarm with all the development teams. After narrowing down the possibilities, we found out that desktop apps had started to make a lot more connections to our servers with the latest stable releases on Mac and Windows. The desktop team called all hands on deck to find the exact point of error, fix it, and release a new version.

Timeline of events

  • On Sep 24th and 26th, we rolled out updates for the Windows (version 7.4.1012) and the Mac (version 7.4.1015) desktop app.
  • Oct 1st, 8:50 am UTC, first issues started happening. It took some time for our users to update their apps, which is why we only detected something was wrong after so many days.
  • Oct 1st, 12 pm UTC, we had to make the choice to kill the desktop API, making the desktop apps usable only in offline mode. The apps were making too many requests to our servers at the time.
  • Oct 1st, 2:45 pm UTC, we pinpointed the issue in our desktop apps and rolled out updates to our users (version 7.4.1022) and re-enabled the desktop API but with a limit on how many requests the app can make in a minute.
  • Oct 2nd, 1:30 pm UTC, we monitored the situation and observed that the newest fixed versions were still causing too many connections to servers. We pinpointed that although the original issue was fixed, a new issue of duplicating entries had appeared. This forced the app to go crazy with syncing.
  • Oct 2nd, 5:30 pm UTC, the desktop team yet again reverted all changes and rolled out new fixed releases for Mac and Windows (version 7.4.1023).
  • Oct 3rd, we’re monitoring the situation and verifying that the latest release works properly before removing the rate-limiting for desktop apps.
  • Oct 4th, We’re working on fixing issues with duplicated time entries. Apps do not create duplicate items anymore but there are still duplicate time entries in our database that were created by previous versions of our apps. We are creating a solution to detect and delete all faulty time entries from our servers. The users are going to be sent an e-mail that their duplicate entries were deleted. In case they detect something extra was deleted, they can write back to us and get their entry restored.

What we learned from this and what are we going to do to prevent it from happening in the future

Audit all network communications.

We will audit all syncing connections in the desktop apps and create an additional layer of automated tests to catch any big changes in network connections.

Add deploy tracking to our desktop apps

We will have a better overview of how each release affects the backend and we will be able to catch and fix potential issues faster.

Improve in-app messaging

Aside from the actual technical issues we also want to address the lack of communication through our desktop apps to avoid confusion in the future. Our apps were suggesting that there is something wrong with a user’s connection instead of informing them about the actual reason why their app was not able to connect to our servers. We will build an in-app information system that will enable us to send important updates to users quickly when needed.

 

These are the main points we are going to work on. If you want to get the whole picture, head over to our Github and check out the full list of issues under our October fix-up project

 

We’d like to apologize again for any inconvenience caused. Thank you for your patience.

The Toggl Desktop team

By On October 3, 2019

  1. Hi there. The Toggle Desktop App is not allowing me to select a Project from the “What are you doing?” drop down menu anymore. As a workaround, I’ve been having to press the Play button, then click Add Project, then type in the Project and select it in that dropdown menu, then close the fly-out window, which is a pain. I’ve downloaded the latest version of the app, but the problem persists. I’m running Toggl on a iMac with Mojave 10.14.6. Any fix for this?

  2. I have installed the 64-bit version on my 64 bit Windows PC, 3 or 4 times. I don’t get any errormessages, but I don’t get any message that the installation is OK. How can I tell if it is?

    • If the installer finish page is shown and you are able to run it. It was a success. If you have issues running the app please contact support with all the details about your system and how you installed the app (download form our website, Microsot Store, Chocolatey).

  3. I’m still having issues with the updated version (Mac OSX 7.4.1023). My autofill glitches when manually adding time entries to my day and additional pop-out windows don’t go away, plus then updating start and stop times the app bumps my cursor past the end time and I have to go back into manually select it every time. Hoping there is a new update coming soon, love the app!

  4. I am very interested in the approach to auditing network communications. I work in open-source electronic health record systems, and we need a scalable way to do this on hospital intranets. Can you share some more detail about your approach?

    • Thank you for showing interest. I’ll gladly share the rough idea of this. We will be doing it inside the desktop app syncing logic. What we’ll do is create automatic tests that will simulate regular user activity and then measure network requests done in a certain period of time after the actions. Let me give a simple example: User starts a time entry -> Now we know that this should trigger sync only once during certain time frame (let’s say 30 seconds). If the request fails for some reason then it’s ok to trigger a new request to sync after 30 seconds. This is what we’ll test. If something is off we get info about it right away and can address it before the new version release.

  5. I really appreciate this update – I love using Toggl and, while I did have tons of issues on October 1, the support team was very responsive and helpful.

  6. I lost all my data from October 1 through to today. A simple email from toggl to me would have saved me that annoyance.

    • The syncing issue should not have caused any loss of data. Please contact our support team (support[at]toggl[dot]com) so that we can look into this for you further.

  7. I have installed the latest version from the above-mentioned link, but still get a message “This version of the app is not supported anymore”

    • Thanks for letting us know! If you send our support team an e-mail (support[at]toggl[dot]com), they’ll be more than happy to walk you through next steps.