Description of Issue: We have been made aware of an issue for some Customers production and Sandbox Orgs losing access to all SVMX and Salesforce Objects for all users except Administrator.
We are marking this incident as resolved. Salesforce do still indicate on their Trust site that 5 instances remain in a service degradation state, if you are still impacted please keep checking https://status.salesforce.com/incidents/3822 for further updates.
Description of Issue: We have been made aware of an issue for some Customers production and Sandbox Orgs losing access to all SVMX and Salesforce Objects for all users except Administrator.
Action Being Taken: 5 instances remain in a service disruption state. Salesforce continue to work on remediation for all remaining impacted instances. A list of these instances can be found on the Trust site, please continue to check this for any further updates in the meantime: https://status.salesforce.com/incidents/3822
Description of Issue: We have been made aware of an issue for some Customers production Orgs losing access to all SVMX and Salesforce Objects for all users except Administrator.
Action Being Taken: As per the latest update on the Salesforce Trust site, there are now 5 instances that remain in a service degradation state. Salesforce continue to work on remediation for all remaining impacted instances. A list of the impacted instances can be found on the Trust site, please continue to check this for any further updates in the meantime: https://status.salesforce.com/incidents/3822
Description of Issue: We have been made aware of an issue for some Customers production Orgs losing access to all SVMX and Salesforce Objects for all users except Administrator.
Action Being Taken: As per the latest update on the Salesforce Trust site, there are 6 instances that remain in a service degradation state. Salesforce continue to work on remediation for all remaining impacted instances. A list of the impacted instances can be found on the Trust site, please continue to check this for any further updates in the meantime: https://status.salesforce.com/incidents/3822
Description of Issue: We have been made aware of an issue for some Customers production Orgs losing access to all SVMX and Salesforce Objects for all users except Administrator.
Action Being Taken: As per update on the Salesforce Trust page, one more sandbox instance (CS85) has had service restored, leaving 9 still being investigated for remediation for which there is no ETA. . A list of the impacted instances can be found on the Trust site, please continue to check this for any further updates in the meantime: https://status.salesforce.com/incidents/3822
Description of Issue: We have been made aware of an issue for some Customers production Orgs losing access to all SVMX and Salesforce Objects for all users except Administrator.
Action Being Taken: As per update on the Salesforce Trust page, all production orgs affected by the permissions issue are now out of impact. Salesforce are aware of a few customers who still have active support cases, these are being handled on a case by case basis.
44 of the 54 sandbox instances have all had service restored and where required permissions restored to before the incident started. The remaining 10 instances are still being investigated for remediation for which there is no ETA. A list of the impacted instances can be found on the Trust site, please continue to check this for any further updates in the meantime: https://status.salesforce.com/incidents/3822
Description of Issue: We have been made aware of an issue for some Customers production Orgs losing access to all SVMX and Salesforce Objects for all users except Administrator.
Action Being Taken: Salesforce has completed the testing and validation of a fix to restore permissions for users of sandbox instances affected by this incident and as previously noted this has been rolled out to a number of impacted Sandbox Instances. The following Instances as of now have not had the fix applied and Saleforce are continuing to work on remediation plans for these instances: CS51, CS63, CS66, CS26, CS90, CS3, CS69. Please note, this information was provided on the most recent SFDC update call and is not currently reflected on Trust. Salesforce are working to get this updated.
Next Update: When more information becomes available.
Posted May 28, 2019 - 19:14 UTC
Investigating
Description of Issue: We have been made aware of an issue for some Customers production Orgs losing access to all SVMX and Salesforce Objects for all users except Administrator.
Action Being Taken: Salesforce Trust has updated with the following: Salesforce has completed the testing and validation of a fix to restore permissions for users of sandbox instances affected by this incident. We deployed and validated that fix on CS91, CS17, CS47, CS12, CS53, CS25, CS7, CS52, CS59, CS23, CS60, CS61, CS94, CS95, CS77, CS8, CS64, CS96, CS97, CS93, CS15, CS16, CS78, CS11, CS13, CS50, CS30, CS20, CS21, CS68, CS19, CS10, and CS71 and the users’ permissions on those sandboxes have been restored to their pre-incident state. If customers on those sandboxes continue to experience issues, they should log a case with Salesforce Support. This fix will be rolled out in a phased approach as analysis of readiness for each sandbox is performed. We do not have an estimated time on completion of that rollout. The fix will not affect sandbox orgs that were refreshed after the incident began, nor sandbox orgs requested by customers to remain as-is. Our preliminary Incident Summary for this ongoing permissions issue is here: https://sfdc.co/PermSetIncidentSummary.
By Whom: Salesforce
Next Update: When more information becomes available.
Posted May 27, 2019 - 04:44 UTC
Update
Description of Issue: We have been made aware of an issue for some Customers production Orgs losing access to all SVMX and Salesforce Objects for all users except Administrator.
Action Being Taken: As per the latest update on Trust, All production instances remain in performance degradation. They have run an automated process to return all Production Instances to a healthy state and validation checks have completed. Some customers may continue to experience issues that are being investigated. NA59 also remains in performance degradation status, an automated process has been run on that instance and validation checks are being performed. During a performance degradation, end users are able to access the service, however, some functionality within the service may not be available or running at optimal performance. Salesforce is working urgently to resolve the remaining issues and return those instances to a healthy state.
Sandbox instances remain in a Service Disruption state. Salesforce have identified a possible fix and are in the process of testing. The fix is designed to bring the instances back to a healthy state. At this point there is no ETA available for the testing to be completed. As always, further information can be found on the Salesforce Trust site: https://status.salesforce.com/incidents/3822 https://status.salesforce.com/incidents/3815
By Whom: Salesforce Engineering team
Next Update: When more information becomes available.
Posted May 24, 2019 - 09:28 UTC
Update
Description of Issue: We have been made aware of an issue for some Customers production Orgs losing access to all SVMX and Salesforce Objects for all users except Administrator.
Action Being Taken: As noted by Salesforce on Trust all production instances are out of service disruption state and are in a performance degradation state whilst service levels return to normal. In this state some functionality may not be available or running at optimal performance. Some customers may still be experiencing issues. The Known Issue article remains in place and is updated regularly as guidance for Admins to manually restore user permissions. There is currently no further update available for the Sandbox issue that some customers may be experiencing. Daily calls are taking place with Salesforce on this issue as well as updates being requested through the Salesforce Customer Success Manager, but at this time there is no ETA for the Sandbox issue to be resolved. Links to both updates on Salesforce Trust below along with the KI:
Next Update: When more information becomes available.
Posted May 22, 2019 - 13:09 UTC
Update
Description of Issue: We have been made aware of an issue for some Customers production Orgs losing access to all SVMX and Salesforce Objects for all users except Administrator.
Action Being Taken: A further update has been provided by Salesforce through their Trust page to advise that all Production Instances are now out of Service Disruption but some remain in a Service Degradation state. Additionally, any Sandbox instances that were impacted as part of this issue remain in Service Disruption and Salesforce are working to resolve the issue. Links to both updates on Salesforce Trust below:
Next Update: When more information becomes available.
Posted May 21, 2019 - 10:32 UTC
Update
We are continuing to work on a fix for this issue.
Posted May 21, 2019 - 10:09 UTC
Update
Description of Issue: We have been made aware of an issue for some Customers production Orgs losing access to all SVMX and Salesforce Objects for all users except Administrator.
Action Being Taken: A further update has been provided by Salesforce to advise that all impacted Productions have now had the permissions restored to the state prior to the incident occurring. However, there are a couple of instances that may still be experiencing issues. The latest update from Salesforce below: "The automated provisioning to restore permissions has now been executed on all production instances. Following that fix, a subset of users in affected orgs on the NA53, NA57, and NA59 instances had their permission levels reset again, which gave them broader data access than intended. Customers on NA53 and NA59 may have experienced service disruption as we worked through the removal of the inadvertent permissions in the affected customer orgs which has now been completed. For affected orgs on NA53, NA57 and NA59, users who had their permissions broadened have now had their permissions reset to allow login and limited additional functionality. For admins that may need guidance on how to manually restore user permissions, please see the instructions in this Known Issue article http://sfdc.co/PermSetKI. Salesforce is working urgently to resolve the issue. Customers should continue to check Trust for updates."
By Whom: Salesforce Technology team
Next Update: When more information becomes available.
Posted May 20, 2019 - 18:57 UTC
Identified
Description of Issue: We have been made aware of an issue for some Customers production Orgs losing access to all SVMX and Salesforce Objects for all users except Administrator.
Action Being Taken: Further to the last update provided we have been made aware by Salesforce that the issue may still be impacting some customers. The latest update from Salesforce below: "The automated provisioning to restore permissions has now been executed on all production instances. A subset of customers may still be experiencing issues with user permissions and our teams continue to work on this. If you continue to experience issues, please see instructions for admins in the Known Issue article: http://sfdc.co/PermSetKI. We continue to make updates to this article. If your issues persist and cannot be resolved by the information in the Known Issue article, please log a case with Support."
By Whom: Salesforce Technology team
Next Update: When more information becomes available.