Exchange Server 2016/2019: Receiving and sending of e-mails disrupted since the new year [solution]

What wouldn't the New Year be without Proproblems with Microsoft Exchange servers. Already in 2021, sysadmins often have nights with them important updates spent, tonight many had to move out after the New Year's Eve party. Many Microsoft mail servers have not delivered and no longer sent e-mails since the New Year.

Update: Microsoft now has an official fix for this Proproblem provided. See a little further down in this article, under "Resolution".

It should have happened from about 01:00 a.m. our time (or 00:00 a.m. UTC) - some monitoring should have reported that it Proproblems with receiving and delivering e-mails. This affects most Exchange Server 2016 and 2019. Previous versions like 2013 or hosted services like Microsoft 365 or Office 365 are not affected. Troubleshooting this is actually quite easy, although it certainly may have taken some admins hours to figure out.

Microsoft Exchange Server 2016/2019 email failure: It's an embarrassing bug Problem

Even before my time - in the year 2000 - it should be the same Prohave given problems. Die-hard admins should experience their déjà vu today. That ProThe problem is actually a really embarrassing one - the date, together with the time, is too long to be able to correctly parse it as an Int32. Nerds surely know that the range of this is between -2.147.483.648 and 2.147.483.647, so 2.201.010.001 of course outside of this range. Why do you even save a date together with the time in an integer - well.

That's how you get it Proget on the trail

Microsoft sent the filtering engine a new update at exactly this point in time (01.01.2022/00/01, 02:16 UTC), which is compared with the local virus filter database via this integer. Depending on when the update was automatically downloaded (with us around 400:4.4.7 a.m., see screenshot), the mail flow no longer runs. The anti-malware scanner can no longer process the e-mails and delivery is delayed, also internally - if the malware filtering is not only active for external e-mails. Senders of an e-mail to an Exchange server should see error messages such as "Remote Server returned 'XNUMX XNUMX Message delayed'".

The Event Viewer of an Exchange Server provides all the necessary information. The following error message appears in the Windows log under Application: "The FIP-FS" Microsoft "Scan Engine failed to load. PID: 19104, Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long. "

Happy New Year... with an ExchangeProproblem (Picture: TechnikNews/Screenshot)

Solution - This is how emails are delivered again

Automatic correction using a script

Update: Microsoft has that Proproblem meanwhile in the Exchange Team Blog approved and provides a working solution.

At the beginning it has to be ResetScanEngineVersion script downloaded directly from Microsoft to reset the version number. The run of the script can currently take some time because the current signatures of the malware scanner have to be downloaded - the download server seems to be very overloaded at the moment. After the download, run the script in the respective folder with an Exchange Management Shell as administrator:

.\Reset-ScanEngineVersion.ps1

Note: For some users, emails still get stuck after the successful execution. This did not happen in our case, but it is advisable to restart the entire server if the sending and receiving of e-mails still does not work.

Manual solution

Should it work with the script Proproblems, such as our reader Rolf (thanks for the tip in the Comments), there is also a manual way to fix the error. We will support you as usual.

• First of all, it is checked whether a correction is necessary at all - the version of the engine is checked (open PowerShell as administrator):

Get-EngineUpdateInformation

In some cases this command does not work and an error message is displayed:

Get-EngineUpdateInformation : The term 'Get-EngineUpdateInformation' is not recognized as the name of a cmdlet,
function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the
path is correct and try again.

Then the corresponding snap-in must first be added to the PowerShell. Then run the Get command again (with PowerShell as administrator).

• Microsoft PowerShell: Error 'Get-EngineUpdateInformation' is not recognized as the name of a cmdlet

Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell

Get-EngineUpdateInformation

If the update version starts with "22", a correction is necessary. At "21" the server - luckily - has not yet downloaded any current virus databases and no changes are necessary.

• The next step is to switch to the services and stop the filter management service. To do this, go to the start menu -> services.msc -> Microsoft Filter Management Service (English: Microsoft Filtering Management Service) -> select "Stop". If you are asked whether the Microsoft Exchange Transport Service should also be stopped, confirm this with "Yes". Attention, It will not be possible to send and receive e-mails during this time.
• Start Task Manager and check whether the “updateservice.exe” task is running. If so, end it. Otherwise it goes straight on (it is advisable to back up the following files & folders beforehand, should something go wrong):
• Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft
• Delete all files in this folder (do not delete the "metadata" folder itself): %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata
• Restart the Microsoft Filtering Management Service under the services
• If the transport service does not start again automatically, start it too
• Open Exchange Management Shell (as Administrator) and go to %ProNavigate to gramFiles%\Microsoft\Exchange Server\V15\Scripts. Run the following command there:

Update-MalwareFilteringServer.ps1 SERVERNAME

• If the server name is not known, this can be found out using the following command, see column "Name":

Get-ExchangeServer

• Note: For some users, emails still get stuck after the successful execution. This did not happen in our case, but it is advisable to restart the entire server if the sending and receiving of e-mails still does not work.
• All done! Then continue with checking the solution, see the following part:

Final check of the solution & (re-) activation of the virus scanner

To check the solution, you can now compare the version number - Microsoft has simply extended the year 2021 and set the version number (status: 02.01.2022) to December 33 (2112330001). Open PowerShell as admin:

Get-EngineUpdateInformation

Engine : Microsoft
LastChecked : 01.02.2022 05:45:11 +01:00
LastUpdated : 01.02.2022 05:40:18 +01:00
EngineVersion : 1.1.18800.4
SignatureVersion : 1.355.1227.0
SignatureDateTime : 01.01.2022 12:29:06 +01:00
UpdateVersion : 2112330001
UpdateStatus : UpdateAttemptNoUpdate

In some cases this command does not work and an error message is displayed:

Get-EngineUpdateInformation : The term 'Get-EngineUpdateInformation' is not recognized as the name of a cmdlet,
function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the
path is correct and try again.

Then the corresponding snap-in must first be added to the PowerShell. Then run the Get command again (with PowerShell as administrator).

• Microsoft PowerShell: Error 'Get-EngineUpdateInformation' is not recognized as the name of a cmdlet

Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell

Get-EngineUpdateInformation

If the “UpdateVersion” starts with 22, the signature update did not go through and the MalwareFilteringServer should still not be used, otherwise the flow of the e-mails will not work again.

Additional step for workaround users: If the workaround described below has been implemented as an interim solution in the last few days, the MalwareFilteringServer must now be reactivated with the following command:

Set-MalwareFilteringServer -BypassFiltering $False -identity SERVERNAME

If the server name is not known, this can be found out using the following command, see column "Name":

Get-ExchangeServer

The virus scanner is now active again without displaying error messages or disrupting the flow of mail. However, it can still take a few hours before the “UpdateStatus” switches from “NoUpdate” to “Successful”, as the virus databases are usually updated daily around midnight.

Note: For some users, emails still get stuck after the successful execution. This did not happen in our case, but it is advisable to restart the entire server if the sending and receiving of e-mails still does not work.

Original workaround

Note: This workaround should no longer be used because Microsoft ProThe problem is now solved using a different method, see "Solution" above.

At the moment, an interim solution has to be used to get the mail flow up and running again. Incidentally, this should be imported this weekend, as the Exchange server deletes all mails from the queue after 48 hours and rejects the e-mails.

First the MailwareFilteringServer has to be deactivated or bypassed. At the same time, this also means that attachments to e-mails are no longer checked for viruses - all dependent checks, such as checking file extensions, no longer run either. These should also be deactivated (if any have been created), otherwise the delivery of e-mails can be delayed by several minutes.

The following command is executed in the Exchange Management Shell:

Set-MalwareFilteringServer -BypassFiltering $True -identity SERVERNAME

If the server name is not known, this can be found out using the following command, see column "Name":

Get-ExchangeServer

The Microsoft Exchange Transport service should then be restarted to enable delivery again. The warnings displayed can be ignored, even if they appear several times.

Get-Service MSExchangeTransport | Restart-Service

That's it Proproblem fixed for now. The queue of outstanding e-mails (incoming and outgoing) should then be worked through again automatically. The queue can be examined more closely simply by opening the Exchange Toolbox or running the following command:

Get-ExchangeServer | Get-Queue

Should a corresponding update come from Microsoft, simply execute the above commands again - but with $ False - again. With this in mind: Happy New Year, dear Exchange fans and everyone who wants to be one this year!

Recommendations for you

• Microsoft Exchange Server: Security update for February 2023 fixes annoying bug
• Microsoft Exchange Server: January 2023 security updates address five vulnerabilities
• Microsoft Exchange Server: Critical Security Updates for March 2022
• Microsoft Exchange Server: Security updates released for January 2022