MS Teams - Multiple Issues
Incident Report for City University of Seattle
Resolved
Title: Some users may experience multiple issues with their Microsoft Teams

User impact: Users may have experienced multiple issues with their Microsoft Teams.

More info: Affected scenarios included, but weren’t limited to the following:

- Users logging in or unlocking their devices after some time may have had missing messages.
- Users may have been unable to load messages in channels and chats.
- Users were unable to view or download their media (images, videos, audio, call recordings, code snippets).
- Some users were experiencing delays sending and receiving chat messages.
- Anonymous users may have been unable to join meetings.
- Teams connectors for Power Automate/Power Apps were experiencing errors.
-Call Recordings might take longer to appear in user's OneDrive for Business and SharePoint Online.
- Users may have been unable to load previous Copilot history, or new history was not written
- Bots may have been unable to download attachments
- Sending and receiving read receipt notifications may have been delayed
- Users may have experienced issues resuming or rejoining meetings.
- Users within organizations leveraging an Information Barrier may have experienced issues joining meetings
- Users performing a cold boot may have been unable to log into teams and would have seen an "oops" page

Final status: After extended monitoring and various optimizations and mitigation efforts, we’ve confirmed that our Microsoft Teams service and features have been restored or returned to optimal health.

Scope of impact: This issue could have potentially impacted any Microsoft Teams user in the scenarios outlined in the More info section.
Posted Jan 26, 2024 - 21:20 PST
Update
Title: Some users may experience multiple issues with their Microsoft Teams

User impact: Users may experience multiple issues with their Microsoft Teams.

More info: Affected scenarios include, but aren't limited to the following:

Mitigated/improving:

- [Updated] Users logging in or unlocking their devices after some time may have missing messages. We believe that this symptom has been addressed.
- [Updated] Users may have been unable to load messages in channels and chats. We believe that this symptom has been addressed.
- [Updated] Users were unable to view or download their media (images, videos, audio, call recordings, code snippets). We believe that this symptom has been addressed.
- [Updated] Some users were experiencing delays sending and receiving chat messages. We believe this symptom has been largely addressed, though some users may still experience brief delays.
- [Updated] Anonymous users may be unable to join meetings. We’re seeing improvement in this area but some users may still be affected by this scenario.
- [Updated] Teams connectors for Power Automate/Power Apps were experiencing errors. We believe that this symptom has been addressed.
- [Updated] Call Recordings might take longer to appear in user's OneDrive for Business and SharePoint Online. This symptom has been addressed.

Still affected/undergoing further evaluation:

- Users may be unable to load previous Copilot history, or new history is not written
- Bots may be unable to download attachments
- Sending and receiving read receipt notifications may be delayed
- Users may experience issues resuming or rejoining meetings.
- Users within organizations leveraging an Information Barrier may experience issues joining meetings
- Users performing a cold boot may not be able to log into teams and will see an "oops" page

Current status: We’re seeing significant improvements with many of the Teams features affected by this incident as we continue to apply mitigations and optimize traffic. We’re evaluating the remaining impact scenarios and are developing fixes and workstreams to address all lingering impact associated with this event. Please see the “more info” section above for the latest information on each impact scenario.

We understand the impact an issue like this can have on your organization, and we appreciate your partnership and patience as we work to remediate this issue.

Scope of impact: This issue can potentially impact any Microsoft Teams user in the scenarios outlined in the More info section.

Start time: Friday, January 26, 2024 at 8:55 AM CST


Next update by: Friday, January 26, 2024 at 8:30 PM CST
Posted Jan 26, 2024 - 17:29 PST
Update
Current status: We’re continuing to apply mitigations across the affected infrastructure and our telemetry is showing additional improvement in the user experience, though many customers are still affected by this issue. We’re also working to apply fixes to address individual affected Teams features in parallel while our broader remediation strategy is ongoing. We’re evaluating any and all additional workstreams that will allow us to reduce the impact to those customers that are still affected.

We understand the impact an issue like this can have on your organization, and we appreciate your partnership and patience as we work to remediate this issue.

Scope of impact: This issue can potentially impact any Microsoft Teams user in the scenarios outlined in the More info section.

Next update by: Friday, January 26, 2024 at 6:30 PM CST
Posted Jan 26, 2024 - 15:13 PST
Update
From Microsoft:

Current status: Our failover operation did not provide the immediate relief intended for end users in North and South America regions. However, we’re monitoring telemetry closely as we continue to optimize traffic patterns and apply configuration changes intended to reduce customer impact as quickly as possible. We understand the impact an issue like this can have on your organization, and we appreciate your partnership and patience as we work to remediate this issue.

Next update by: Friday, January 26, 2024 at 4:00 PM CST
Posted Jan 26, 2024 - 13:07 PST
Monitoring
Title: Some users may experience multiple issues with their Microsoft Teams

User impact: Users may experience multiple issues with their Microsoft Teams.

More info: Affected scenarios include, but aren't limited to:
- Users performing a cold boot may not able to log into teams and will see an "oops" page
- Users logging in or unlocking their devices after some time may see missing messages
- Users may fail to load messages in channels and chats
- Users are unable to view or download their media (images, videos, audio, call recordings, code snippets)
- Some messages may experience delays being sent
- Call Recordings might take longer to appear in user's OneDrive for Business and SharePoint Online
- Users may be unable to load previous Copilot history, or new history is not written
- Bots may be unable to download attachments
- Sending and receiving read receipt notifications may be delayed
- Anonymous users may be unable to join meetings
- Teams connectors for Power Automate/Power Apps may experiencing errors

Current status: We’ve completed the failover in the Europe, Middle East, and Africa (EMEA) region and telemetry is showing improvement. We’re continuing with the failover processes in the Americas, and are monitoring whilst these processes complete.

Scope of impact: This issue can potentially impact any Microsoft Teams user in the scenarios outlined in the More info section.

Start time: Friday, January 26, 2024 at 8:55 AM CST

Next update by: Friday, January 26, 2024 at 2:00 PM CST
Posted Jan 26, 2024 - 11:35 PST
This incident affected: Office 365 (Teams).