Exchange Server 2016/2019: Receiving and sending of e-mails disrupted since the new year [solution]
What would the new year be without problems with Microsoft Exchange servers? Already in 2021, sysadmins often have nights with important updates spent, tonight many had to move out after the New Year's Eve party. Many Microsoft mail servers have not delivered and no longer sent e-mails since the New Year.
Update: Microsoft has now provided an official solution to the problem. See further down in this article, under "Solution".
It probably started around 01:00 a.m. our time (or 00:00 a.m. UTC) - some monitoring systems reported that there were problems receiving and delivering emails. This affects most Exchange Server 2016 and 2019. Previous versions such as 2013 or hosted services such as Microsoft 365 or Office 365 are not affected. Troubleshooting this is actually quite simple, although it may have taken some admins hours to figure out.
Jump to section
Microsoft Exchange Server 2016/2019 email outage: An embarrassing bug is the problem
The same problem is said to have existed before my time – in 2000. Die-hard admins are probably experiencing déjà vu today. The problem is actually a really embarrassing one – the date, together with the time, is too long to be parsed correctly as an Int32. Nerds will know that the value range is between -2.147.483.648 and 2.147.483.647, so 2.201.010.001 of course outside of this range. Why do you even save a date together with the time in an integer - well.
How to track down the problem
Microsoft sent the filtering engine a new update at exactly this point in time (01.01.2022/00/01, 02:16 UTC), which is compared with the local virus filter database via this integer. Depending on when the update was automatically downloaded (with us around 400:4.4.7 a.m., see screenshot), the mail flow no longer runs. The anti-malware scanner can no longer process the e-mails and delivery is delayed, also internally - if the malware filtering is not only active for external e-mails. Senders of an e-mail to an Exchange server should see error messages such as "Remote Server returned 'XNUMX XNUMX Message delayed'".
The Event Viewer of an Exchange Server provides all the necessary information. The following error message appears in the Windows log under Application: "The FIP-FS" Microsoft "Scan Engine failed to load. PID: 19104, Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long. "
Solution - This is how emails are delivered again
Automatic correction using a script
Update: Microsoft has now addressed the issue in the Exchange Team Blog approved and provides a working solution.
At the beginning it has to be ResetScanEngineVersion script downloaded directly from Microsoft to reset the version number. The run of the script can currently take some time because the current signatures of the malware scanner have to be downloaded - the download server seems to be very overloaded at the moment. After the download, run the script in the respective folder with an Exchange Management Shell as administrator:
.\Reset-ScanEngineVersion.ps1
Note: For some users, emails still get stuck after the successful execution. This did not happen in our case, but it is advisable to restart the entire server if the sending and receiving of e-mails still does not work.
Manual solution
If there are any problems with the script, as was the case with our reader Rolf (thanks for pointing this out in the Comments), there is also a manual way to fix the error. We will support you as usual.
- First of all, it is checked whether a correction is necessary at all - the version of the engine is checked (open PowerShell as administrator):
Get-EngineUpdateInformation
In some cases this command does not work and an error message is displayed:
Get-EngineUpdateInformation : The term 'Get-EngineUpdateInformation' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
Then the corresponding snap-in must first be added to the PowerShell. Then run the Get command again (with PowerShell as administrator).
Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell Get-EngineUpdateInformation
If the update version starts with "22", a correction is necessary. At "21" the server - luckily - has not yet downloaded any current virus databases and no changes are necessary.
- The next step is to switch to the services and stop the filter management service. To do this, go to the start menu -> services.msc -> Microsoft Filter Management Service (English: Microsoft Filtering Management Service) -> select "Stop". If you are asked whether the Microsoft Exchange Transport Service should also be stopped, confirm this with "Yes". Attention, It will not be possible to send and receive e-mails during this time.
- Start Task Manager and check whether the “updateservice.exe” task is running. If so, end it. Otherwise it goes straight on (it is advisable to back up the following files & folders beforehand, should something go wrong):
- Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft
- Delete all files in this folder (do not delete the metadata folder itself): %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata
- Restart the Microsoft Filtering Management Service under the services
- If the transport service does not start again automatically, start it too
- Open Exchange Management Shell (as administrator) and navigate to %ProgramFiles%\Microsoft\Exchange Server\V15\Scripts. Run the following command there:
Update-MalwareFilteringServer.ps1 SERVERNAME
- If the server name is not known, this can be found out using the following command, see column "Name":
Get-ExchangeServer
- Note: For some users, emails still get stuck after the successful execution. This did not happen in our case, but it is advisable to restart the entire server if the sending and receiving of e-mails still does not work.
- All done! Then continue with checking the solution, see the following part:
Final check of the solution & (re-) activation of the virus scanner
To check the solution, you can now compare the version number - Microsoft has simply extended the year 2021 and set the version number (status: 02.01.2022) to December 33 (2112330001). Open PowerShell as admin:
Get-EngineUpdateInformation
Engine : Microsoft LastChecked : 01.02.2022 05:45:11 +01:00 LastUpdated : 01.02.2022 05:40:18 +01:00 EngineVersion : 1.1.18800.4 SignatureVersion : 1.355.1227.0 SignatureDateTime : 01.01.2022 12:29:06 +01:00 UpdateVersion : 2112330001 UpdateStatus : UpdateAttemptNoUpdate
In some cases this command does not work and an error message is displayed:
Get-EngineUpdateInformation : The term 'Get-EngineUpdateInformation' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
Then the corresponding snap-in must first be added to the PowerShell. Then run the Get command again (with PowerShell as administrator).
Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell Get-EngineUpdateInformation
If the “UpdateVersion” starts with 22, the signature update did not go through and the MalwareFilteringServer should still not be used, otherwise the flow of the e-mails will not work again.
Additional step for workaround users: If the workaround described below has been implemented as an interim solution in the last few days, the MalwareFilteringServer must now be reactivated with the following command:
Set-MalwareFilteringServer -BypassFiltering $False -identity SERVERNAME
If the server name is not known, this can be found out using the following command, see column "Name":
Get-ExchangeServer
The virus scanner is now active again without displaying error messages or disrupting the flow of mail. However, it can still take a few hours before the “UpdateStatus” switches from “NoUpdate” to “Successful”, as the virus databases are usually updated daily around midnight.
Note: For some users, emails still get stuck after the successful execution. This did not happen in our case, but it is advisable to restart the entire server if the sending and receiving of e-mails still does not work.
Original workaround
Note: This workaround should no longer be used because Microsoft has now solved the problem using a different method, see "Solution" above.
At the moment, an interim solution has to be used to get the mail flow up and running again. Incidentally, this should be imported this weekend, as the Exchange server deletes all mails from the queue after 48 hours and rejects the e-mails.
First the MailwareFilteringServer has to be deactivated or bypassed. At the same time, this also means that attachments to e-mails are no longer checked for viruses - all dependent checks, such as checking file extensions, no longer run either. These should also be deactivated (if any have been created), otherwise the delivery of e-mails can be delayed by several minutes.
The following command is executed in the Exchange Management Shell:
Set-MalwareFilteringServer -BypassFiltering $True -identity SERVERNAME
If the server name is not known, this can be found out using the following command, see column "Name":
Get-ExchangeServer
The Microsoft Exchange Transport service should then be restarted to enable delivery again. The warnings displayed can be ignored, even if they appear several times.
Get-Service MSExchangeTransport | Restart-Service
This solves the problem for the time being. The queue of pending emails (incoming and outgoing) should then be processed automatically again. The queue can be examined more closely by simply opening the Exchange Toolbox or executing the following command:
Get-ExchangeServer | Get-Queue
Should a corresponding update come from Microsoft, simply execute the above commands again - but with $ False - again. With this in mind: Happy New Year, dear Exchange fans and everyone who wants to be one this year!
Well done.
Saved my day.
Hi David,
The script is not running for me because the Exchange Server cannot be found under Program Files. The server is installed under Program Files. Can I change the script accordingly?
Thanks in advance.
Hello Rolf,
I have now also added the manual solution variant to the article. In addition, I have now specified when and where which shell (PowerShell as administrator or Exchange Management Shell) has to be executed. Hopefully this will help! LG David
Hi David,
I have a problem: the script is not executed even though I am the administrator. What could be the problem?
Hello Rolf,
do you get any error messages or other indications as to what is going wrong? The script is downloaded in the desired folder and with the Exchange Management Shell (started as admin) are you also in the same directory as the script? LG David
Hi David,
When the script is executed, it is displayed that the path cannot be found or has been moved. When I do the manual solution, the commands say it's not a cmdlet. Unfortunately I cannot send you a screenshot.
Mhmm, tricky. But we'll get there. The error with the cmdlet should only happen when executing "Get-EngineUpdateInformation", we have already shown the solution to the error above or here in a separate article: https://www.techniknews.net/ratgeber/microsoft-powershell-fehler-get-engineupdateinformation-is-not-recognized-as-the-name-of-a-cmdlet/
The other things should actually go through without an error message.
If it still doesn't work, simply upload the screenshot to an image upload service (such as https://picr.eu) and post the link here. Please give me an update if everything is working!
Thank you! helped me!
... And there are people who think there are nanorobots in the vaccine - those from MS don't even get an error-free exchange to run ...
Thank you for the quick publication!
Thank you for the quick publication, it saved a lot of time and searches. Really TOP.
I'm happy to have helped! LG, David from TechnikNews
thank you thank you - we have now managed to do it with us, I've already had the crisis 🙂
It's great that you're back online - have a nice Saturday! LG, David from TechnikNews
Thank you, thank you, thank you for this article !!!! Weekend is saved!
Have a nice weekend! 🙂 LG, David from TechnikNews