Degraded performance across all APIs
Incident Report for Xero Ltd - API
Postmortem

Xero API Token Refresh Issues:

During the performance issues experienced between Sunday 11 March 23:30 (UTC) & Monday 12 March 00:40 (UTC) Xero API partners using the Partner App experienced issues with access token refresh requests.

Due to the performance issues, in many cases during a access token refresh request, a new token was generated, but not returned. This resulted in many partners receiving the following error responses with HTTP 401 response code.

oauthproblem=tokenrejected&oauthproblemadvice=Token THISISANCCESSTOKEN does not match an expected REQUEST token

oauthproblem=tokenrejected&oauthproblemadvice=The access token is not valid

Resolution:

To resolve this issue for partners, without requiring users to re-authenticate, Xero has implemented a “Token Amnesty”.

The token amnesty began at Monday 12 March 23:30 (UTC). An end date for the amnesty has not yet been set, but is estimated at 3 - 7 days.

During the Token Amnesty period, any access token refresh requests with an invalid access token, but a valid session handle, will respond with a valid access token.

A valid session handle is always required for access token refresh requests, so no changes are required from partners.

If you have Xero Orgs that were disconnected during this period, please retry the access token refresh request in order to receive a valid access token.

After the token amnesty period is complete, any disconnected Xero Orgs will require re-authentication in order to generate a valid access token.

FAQ:

  • Do I need to make any changes to my integration? No.
  • I have many Xero Orgs will invalid access tokens, how can I renew them? Simply make a Access Token Refresh request, with the existing access token & session handle (there are no changes to this request)
  • What is a session handle? The session handle is generated when the Xero Org is initially authenticated, it is required for every API request. Session handles have a 10 year expiry.
  • Does the token amnesty apply to the WFM / XPM API’s? No, the token amnesty only applied to the Partner Apps for the Xero API’s (Accounting, Payroll, Files, Assets). The WFM / XPM OAuth issue is a separately and unrelated issue, please subscribe here for updates.
Still having problems? Reach out to Xero API support via:

Posted Mar 13, 2018 - 15:05 NZDT

Resolved
This incident has been resolved.
Posted Mar 13, 2018 - 00:40 NZDT
Update
A good news update - we’ve made the changes needed to address the performance issue that has impacted many customers today. We’ll continue to monitor very closely but do not hesitate to let us know if you’re experiencing any further performance issues. We realise this has been disruptive to many of our customers. Once we are confident we have fully resolved the issue we will provide a situation report on our blog..
Posted Mar 12, 2018 - 22:43 NZDT
Update
We’re continuing to work on resolving the issue. There is no significant update on progress to give, and unfortunately users are likely to still be experiencing some level of performance impact. Thanks for your patience.
Posted Mar 12, 2018 - 20:00 NZDT
Update
In the past 24 hours, Xero undertook an upgrade to a key database server to improve performance.

This upgrade resulted in an unforeseen performance degradation of the database. The Xero team is working on this as a matter of priority and we will continue to update as we're closer to resolving this.
Posted Mar 12, 2018 - 17:22 NZDT
Update
We're continuing to experience performance degradation issues and our team is working on resolving this as matter of priority.
Posted Mar 12, 2018 - 16:44 NZDT
Update
We’re sorry this is causing you some downtime. We very rarely have extended outages like this. Thank you for your support while we fix this for you.
Posted Mar 12, 2018 - 14:59 NZDT
Update
We're currently working to get affected users back up and running. Thanks for your continued patience.
Posted Mar 12, 2018 - 14:08 NZDT
Update
We are still seeing degraded performance to the APIs and the team are working on it.
Posted Mar 12, 2018 - 13:12 NZDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Mar 12, 2018 - 12:53 NZDT
Identified
Our Product Team have identified the cause, and are working to restore your access to Xero as soon as possible.
Posted Mar 12, 2018 - 12:32 NZDT
Investigating
The team is investigating current degraded performance across the Xero API.
Posted Mar 12, 2018 - 12:28 NZDT