Blocking ads removes funding from us!
Researching and writing articles takes a lot of time. Operating our infrastructure costs money.
All of this is funded with advertising revenue.
We don't like advertising either - that's why we avoid annoying banners and pop-ups.
Please give us a chance and deactivate your adblocker!
Alternatively, you can support us here voluntarily.
>> Show us your talent - we are looking for you! <

Language:  Deutsch English (Beta)

Follow us:

Exchange Server 2016/2019: Receiving and sending of e-mails disrupted since the new year [solution]

Microsoft Exchange Server
Image: Microsoft
(Post picture: © 2021 Microsoft)

What wouldn't the new year be without problems with Microsoft Exchange servers. Sysadmins often have nights with us as early as 2021 important updates spent, tonight many had to move out after the New Year's Eve party. Many Microsoft mail servers have not delivered and no longer sent e-mails since the New Year.

Update: Microsoft has now made an official solution to the problem available. See a little further down in this article, under "Solution".

It should have happened from around 01:00 AM UTC (or 00:00 UTC) - some monitors have reported that there are problems with receiving and delivering e-mails. This affects most Exchange Servers 2016 and 2019. Previous versions such as 2013 or hosted services such as Microsoft 365 or Office 365 are not affected. It is actually quite simple to troubleshoot, although it may have taken some admins hours to figure out.

Microsoft Exchange Server 2016/2019 Email Outage: An embarrassing bug is the problem

Even before my time - in 2000 - there should have been the same problem. Die-hard admins should experience their déjà-vu today. The problem is actually a really embarrassing one - the date, together with the time, is too long to be able to parse it correctly as an Int32. Nerds certainly know that the value range of this is between -2.147.483.648 and 2.147.483.647, so it is 2.201.010.001 of course outside of this range. Why do you even save a date together with the time in an integer - well.

This is how you get on the track of the problem

Microsoft sent the filtering engine a new update at exactly this point in time (01.01.2022/00/01, 02:16 UTC), which is compared with the local virus filter database via this integer. Depending on when the update was automatically downloaded (with us around 400:4.4.7 a.m., see screenshot), the mail flow no longer runs. The anti-malware scanner can no longer process the e-mails and delivery is delayed, also internally - if the malware filtering is not only active for external e-mails. Senders of an e-mail to an Exchange server should see error messages such as "Remote Server returned 'XNUMX XNUMX Message delayed'".

The Event Viewer of an Exchange Server provides all the necessary information. The following error message appears in the Windows log under Application: "The FIP-FS" Microsoft "Scan Engine failed to load. PID: 19104, Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long. "

Microsoft Exchange Server Email Bug 2022

Happy New Year ... with an exchange problem. (Image: TechnikNews/Screenshot)

Solution - This is how emails are delivered again

Automatic correction using a script

Update: Microsoft now has the problem on the Exchange team blog approved and provides a working solution.

At the beginning it has to be ResetScanEngineVersion script downloaded directly from Microsoft to reset the version number. The run of the script can currently take some time because the current signatures of the malware scanner have to be downloaded - the download server seems to be very overloaded at the moment. After the download, run the script in the respective folder with an Exchange Management Shell as administrator:

.\Reset-ScanEngineVersion.ps1

Note: For some users, emails still get stuck after the successful execution. This did not happen in our case, but it is advisable to restart the entire server if the sending and receiving of e-mails still does not work.

Manual solution

Should there be problems with the script, such as with our reader Rolf (thanks for the hint in the Comments), there is also a manual way to fix the error. We will support you as usual.

  • First of all, it is checked whether a correction is necessary at all - the version of the engine is checked (open PowerShell as administrator):
Get-EngineUpdateInformation

In some cases this command does not work and an error message is displayed:

Get-EngineUpdateInformation : The term 'Get-EngineUpdateInformation' is not recognized as the name of a cmdlet,
function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the
path is correct and try again.

Then the corresponding snap-in must first be added to the PowerShell. Then run the Get command again (with PowerShell as administrator).

Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell

Get-EngineUpdateInformation

If the update version starts with "22", a correction is necessary. At "21" the server - luckily - has not yet downloaded any current virus databases and no changes are necessary.

  • The next step is to switch to the services and stop the filter management service. To do this, go to the start menu -> services.msc -> Microsoft Filter Management Service (English: Microsoft Filtering Management Service) -> select "Stop". If you are asked whether the Microsoft Exchange Transport Service should also be stopped, confirm this with "Yes". Attention, It will not be possible to send and receive e-mails during this time.
  • Start Task Manager and check whether the “updateservice.exe” task is running. If so, end it. Otherwise it goes straight on (it is advisable to back up the following files & folders beforehand, should something go wrong):
  • Delete the following folder:% ProgramFiles% \ Microsoft \ Exchange Server \ V15 \ FIP-FS \ Data \ Engines \ amd64 \ Microsoft
  • Delete all files in this folder (do not delete the metadata folder itself):% ProgramFiles% \ Microsoft \ Exchange Server \ V15 \ FIP-FS \ Data \ Engines \ metadata
  • Restart the Microsoft Filtering Management Service under the services
  • If the transport service does not start again automatically, start it too
  • Open the Exchange Management Shell (as administrator) and navigate to% ProgramFiles% \ Microsoft \ Exchange Server \ V15 \ Scripts. Execute the following command there:
Update-MalwareFilteringServer.ps1 SERVERNAME
  • If the server name is not known, this can be found out using the following command, see column "Name":
Get-ExchangeServer
  • Note: For some users, emails still get stuck after the successful execution. This did not happen in our case, but it is advisable to restart the entire server if the sending and receiving of e-mails still does not work.
  • All done! Then continue with checking the solution, see the following part:

Final check of the solution & (re-) activation of the virus scanner

To check the solution, you can now compare the version number - Microsoft has simply extended the year 2021 and set the version number (status: 02.01.2022) to December 33 (2112330001). Open PowerShell as admin:

Get-EngineUpdateInformation
Engine : Microsoft
LastChecked : 01.02.2022 05:45:11 +01:00
LastUpdated : 01.02.2022 05:40:18 +01:00
EngineVersion : 1.1.18800.4
SignatureVersion : 1.355.1227.0
SignatureDateTime : 01.01.2022 12:29:06 +01:00
UpdateVersion : 2112330001
UpdateStatus : UpdateAttemptNoUpdate

In some cases this command does not work and an error message is displayed:

Get-EngineUpdateInformation : The term 'Get-EngineUpdateInformation' is not recognized as the name of a cmdlet,
function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the
path is correct and try again.

Then the corresponding snap-in must first be added to the PowerShell. Then run the Get command again (with PowerShell as administrator).

Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell

Get-EngineUpdateInformation

If the “UpdateVersion” starts with 22, the signature update did not go through and the MalwareFilteringServer should still not be used, otherwise the flow of the e-mails will not work again.

Additional step for workaround users: If the workaround described below has been implemented as an interim solution in the last few days, the MalwareFilteringServer must now be reactivated with the following command:

Set-MalwareFilteringServer -BypassFiltering $False -identity SERVERNAME

If the server name is not known, this can be found out using the following command, see column "Name":

Get-ExchangeServer

The virus scanner is now active again without displaying error messages or disrupting the flow of mail. However, it can still take a few hours before the “UpdateStatus” switches from “NoUpdate” to “Successful”, as the virus databases are usually updated daily around midnight.

Note: For some users, emails still get stuck after the successful execution. This did not happen in our case, but it is advisable to restart the entire server if the sending and receiving of e-mails still does not work.

Original workaround

Note: This workaround should no longer be used because Microsoft has now solved the problem using a different method, see "Solution" above.

At the moment, an interim solution has to be used to get the mail flow up and running again. Incidentally, this should be imported this weekend, as the Exchange server deletes all mails from the queue after 48 hours and rejects the e-mails.

First the MailwareFilteringServer has to be deactivated or bypassed. At the same time, this also means that attachments to e-mails are no longer checked for viruses - all dependent checks, such as checking file extensions, no longer run either. These should also be deactivated (if any have been created), otherwise the delivery of e-mails can be delayed by several minutes.

The following command is executed in the Exchange Management Shell:

Set-MalwareFilteringServer -BypassFiltering $True -identity SERVERNAME

If the server name is not known, this can be found out using the following command, see column "Name":

Get-ExchangeServer

The Microsoft Exchange Transport service should then be restarted to enable delivery again. The warnings displayed can be ignored, even if they appear several times.

Get-Service MSExchangeTransport | Restart-Service

This solves the problem for the time being. The queue or queue of the outstanding e-mails (incoming and outgoing) should then be worked through again automatically. The queue can be examined more closely by simply opening the Exchange Toolbox or executing the following command:

Get-ExchangeServer | Get-Queue

Should a corresponding update come from Microsoft, simply execute the above commands again - but with $ False - again. With this in mind: Happy New Year, dear Exchange fans and everyone who wants to be one this year!

Recommendations for you

>> Support us by buying through our partners <

David Wurm

Do that TechnikNews-Ding together with a great team for several years. Works in the background on the server infrastructure and is also responsible for everything editorial. Is fascinated by the latest technology and likes to blog about everything digital. Often found in her spare time doing web development, photography or radio making.

David has already written 842 articles and left 347 comments.

Web | Facebook | Twitter | Insta | YouTube
Mail: david.wurm | at |techniknews.net | please NOT for general inquiries, cooperations! This way: Contact
guest
Your name, which will be shown publicly.
We will not publish your email address.
15 Comments
latest
oldest Best
Inline feedback
View all comments
OKI

Well done.

Saved my day.

Rolf Dau

Hi David,
the script is not carried out for me because the Exchange Server cannot be found under Program Files. The server is installed under Programs. Can you change the script accordingly?
Thanks in advance.

Rolf Dau

Hi David,
I have a problem the script does not run even though I am an administrator. Where is the mistake?

Rolf Dau

Hi David,
When the script is executed, it is displayed that the path cannot be found or has been moved. When I do the manual solution, the commands say it's not a cmdlet. Unfortunately I cannot send you a screenshot.

Jimly

Thank you! helped me!
... And there are people who think there are nanorobots in the vaccine - those from MS don't even get an error-free exchange to run ...

Nils

Thank you for the quick publication!

Stefan

Thank you for the quick publication, it saved a lot of time and searches. Really TOP.

Monika

thank you thank you - we have now managed to do it with us, I've already had the crisis 🙂

Paul Winzinger

Thank you, thank you, thank you for this article !!!! Weekend is saved!