A lot has gone down the weekend — and I’m not just talking about Game of Thrones.
It’s been a little crazy in the world of Pardot and Salesforce admins for the last 72 hours. Starting Friday morning, customers started reporting widespread access, performance, and permissioning issues.
The internet promptly dubbed this #Permissiongeddon — and while it’s a cute and catchy name, it’s a pretty serious issue for many of our customers.
Happy Monday, guys! While you pour yourself a cup of joe, here’s a summary of what’s transpired and where you may have a little work to do this morning.
What Happened
Salesforce deployed a database script related to the Pardot integration user that resulted in granting users broader data access than intended.
For a short period of time, in affected orgs ALL users had modify all permissions on all objects. Meaning team members and customers could view and edit things that they weren’t supposed to.
This is massively problematic for obvious reasons — so as a stop gap measure, Salesforce blocked access to all users except admins in affected orgs. And in some cases for admins too. Users couldn’t log in, or if they could, they weren’t able to edit records normally.
Who Was Impacted
This was a pretty widespread issue, and particularly so among Pardot customers. Here’s who was hit:
- Most Pardot-enabled orgs
- Orgs that had Pardot in the past
- Some non-Pardot orgs that share an instance with Pardot orgs
909 people clicked “this issue affects me” on the Known Issue pages, but I haven’t seen a definitive measure of how many environments or customers were impacted by this. But in short, it’s a lot of people.
How Salesforce Has Addressed it Thus Far
Salesforce has been providing frequent updates on their progress remedying the issues. Communication channels have included trust.salesforce.com, email updates, tweets, and customer webinars. This Reddit thread, while unofficial, has also been informative.
The initial guidance from Salesforce was to restore functionality by change setting profiles & permissions from an unaffected Sandbox. If an unaffected Sandbox didn’t exist, the next best course of action was to manually rebuild profiles & permissions. This isn’t a great fix, as it’s a complex and time consuming process for admins — but for orgs where users needed immediate access, it was a better option than sitting around waiting.
On Friday night, Salesforce began deploying a script to restore permissions to their “pre-incident state.” If you or another admin made manual changes in the interim, here’s what to expect:
- If a Profile was deleted, it will not be resurrected.
- If a Profile was created, it will not be deleted.
- If a Profile was UPDATED, that edited profile will be overwritten to its “pre-incident state.”
As of Saturday, Salesforce thought 89% of customers were back in the game, but continued support tickets have indicated that there are more scenarios to address.
What to Do Now
Still with me? On that second cup of coffee yet? Let’s discuss what actions are needed:
1. Check if Your Org is (Still) Impacted
For starters, check is your org is still impacted. Can you log in? If you can log in, can you edit data? If you log in as a Standard User, can you edit data?
If the answer to all three of those is yes, you’re likely in the clear.
2. Option A to Fix Permissions: Wait for Salesforce
If you are still experiencing issues and have the ability to sit tight — give Salesforce a bit more time. They have all hands on deck working on a fix.
3. Option B to Fix Permissions: Update it Yourself
If you’re still experiencing issues and it’s mission critical to get users working in the system, consider updating your profiles and permissions with the instructions previously provided.
The risk of doing this is that Salesforce will eventually finish running its script and any edited profiles will be overwritten.
4. Expect Delays in the Sync Queue & Escalate if Needed
To prevent data loss while working on the fix, Salesforce paused the sync queue with Pardot for most orgs. This is now back up and running for MOST environments with the v1 connector.
The best way to see if you’re still impacted by this is to log into Pardot and go to Admin>Connectors and click the little gear icon by your Salesforce connector. If there’s a massive number next to “sync queue,” then you’re probably frozen and should avoid any massive data updates.
If you’re paused, support CAN get the sync back up and running if you put in a ticket. You may also want to give users a heads up to expect an extra flurry of email alerts when it kicks back into gear.
5. Hang Tight on Sandboxes
The priority is getting Production environments back in ship shape. Sandbox fixes are coming, but expect it to take a bit more time.
Don’t even bother to log a case if a Sandbox is impacted. Seriously, don’t. It will get put on the back burner’s back burner.
6. Make Sure You Have a Pardot-Only Admin User Handy
While admin access to Salesforce was disabled, we were still able to get into our Pardot orgs with non user sync, non-SSO logins. This is a good reminder to keep a least one Pardot-Only Admin account handy for desperate times. (You also need this for external integrations anyway.)
7. Stay Tuned for More Updates
I don’t envy the comms team at Salesforce right now — this is a communication challenge like no other.
One one hand, getting people information FAST is important. But at the same time, the guidance needs to be clear, and correct. Also, communicating there’s an inadvertent exposure of data isn’t a great idea until there’s a fix.
Communication channels have included in app banner messages, trust.salesforce.com, email updates, tweets, and customer webinars. Keep an eye on those for the latest and greatest, and I’ll post updates here as I get them.
Could communication have been a little smoother over the last few days? Sure. But to Salesforce’s credit, it’s clear that they have all engineering and product eyes on this situation and that the highest levels of leadership are engaged in the response. It’s a hard job, they’re human, and they’re doing the best they can.
Questions?
What questions do you have? What else are you hearing? Did I miss anything? Please share it in the comments!
