Solved

Update on Winlogin 4005 & Terminal Servers - November 22, 2016

  • 23 November 2016
  • 35 replies
  • 889 views

Userlevel 7
Badge +4
  • Retired Webrooter
  • 288 replies
Important update on new release for customers operating Webroot Secure Anywhere on Terminal Servers in a virtualized environment
 
Issued: 22nd November 2016
 
Overview of known issue:
Webroot has been actively tracking and monitoring an issue that affects some customers operating Terminal Servers in a virtualized environment. During a fault condition, a Winlogin 4005 error occurs where an RDS user is unable to connect to the RDS server, and the event logs display a message indicating that Winlogin has stopped responding.
 
Interim advice issued by Webroot engineering teams recommended setting the WSAB Self-Protection function to ‘Minimum’ on all Server Policies as this significantly reduced the frequency of Winlogin 4005 errors in most cases.
 
Update:
Further to that advice, Webroot engineering teams developed a new, stabilizing agent release to address causes of the 4005 error and made this agent available to a select group of customers to trial from 15th November.
 
This new v9.0.13.75 agent has shown noticeable improvement towards resolution of the 4005 issues with the select group of customers. Given the positive feedback, Webroot made a decision to make this new agent build available to all Webroot customers via ‘Auto updates’ today (Tuesday 22 November). This will eliminate the need for customers to manually install this agent build in in the field and should provide global relief.
 
Webroot engineering teams continue to complete regression and environment testing on a fix. The engineering and QA teams continue to treat this as their highest priority and are predicting that a new build will be targeted for release in December following in depth analysis of the stabilizing agent.
 
Following successful roll out of the fix and a suitable review period, Webroot will remove the recommendation to operate Self-Protection at the minimum setting and restore ‘Maximum’ as the default setting.
 
Webroot Release deployment steps
  • Please log into the Webroot GSM management console and confirm the Webroot agent has been ‘Auto- updated (to v9.0.13.75) and is checking in on all machines.
Next Steps:
  • Webroot will communicate an update to this message on/by Friday 2nd
  • If you face issues with 4005 errors, please contact Webroot Support for assistance at your earliest convenience.
Webroot apologizes for the disruption and inconvenience caused and for the time taken to manage this complex fault.  We also thank you for your continued patience and understanding in this matter.

Kindest regards,
The Webroot Product Management Team
icon

Best answer by JGiffard 1 September 2017, 10:35

View original

35 replies

Userlevel 7
@, please reach out to our Support Team to look into this further for you.
 
Business Technical Support: Call 1-866-254-8400 M-F 7:00 AM – 6:00 PM MT
Open a Support Ticket
Hi is this stil happening? I have a terminal server that has been plagued with this issue for 6 months or more and we use webroot. I never thought to check that it might be webroot.
Userlevel 7
Badge +31
winlogon.exe -  I'd avoid doing that. 
 
You're entirely right in saying that this has been going on for a long time.  No matter what we've tried in house, we can't reproduce this.  Some customers have Terminal Servers that rarely, if ever, see this issue.  It's been a pain in arse for everyone concerned, but particulary our customers. 
 
But there's some good news.  We've been given a full crash dump from a server that was experiencing a 4005 event and that's undergoing analysis at the moment.  I'm expecting that will give us direction as to what the underlying root cause is and then we'll be able to do something about it.  Given the past history, it's going to take a number of releases for us to flush out the solution. 
 
Jonathan
 
 
 
 
 
Userlevel 1
Jonathon,
Thanks for the Policy suggestion.   I have created the policy and applied it to the single server causing me these problems.   I find it kind of unsettling that I have to disable so much of the product just to keep the client's server running.   Seems kind of counter-intuitive - "Please pay for this product, but don't use its security features or your server will crash!"
I have a single 2008 r2 RDS server that has been having these issues for some time - the Event ID 4005 and inability for users to log in.   (WR Agent 9.0.18.34)  Server seems to be completely unresponsive when it happens.  I have to power cycle it to get it back up again.
I found a document somewhere a while ago suggesting an override for C:Windowssystem32winlogon.exe - is this advised, or was it an uninformed suggestion?   (No idea where I saw it...)
I find it amazing that this has been going on for so long without a resolution.    There's a document in this thread somewhere listing patches that are supposed to fix it, but they all seem to be for Server 2012 and r2 - not for 2008.
I am reluctant to do so, but am considering just removing WBSA from this server so I can stop getting urgent SOS calls from the client.    Understand that it is very difficult to recommend a product to my clients that has had an unresolved issue like this for so long.
 
Userlevel 7
Badge +31
Hello,
 
Working closely with our customers, Webroot has identified an issue that manifests itself in the form of existing Terminal Server sessions becoming un-responsive with users no longer being able to log in. The affected Terminal Servers have required restarting in order for normal service to be resumed. For customers who have experienced this, an update to WSA is available now and Webroot support will provide assistance as required. The fix will also be rolled into the next general release of WSA which is forecast to be automatically deployed in October. Forthcoming product bulletins will advise of the exact date.
 
Before applying the agent build we have created to address this issue, please ensure that you have applied the below Microsoft patches. These patches were designed specifically to address 4005 errors/RDS connection issues.
 
http://support.microsoft.com/kb/3172614
http://support.microsoft.com/kb/3179574
http://support.microsoft.com/kb/3197875
http://support.microsoft.com/kb/3197874
 
Before installing the following agent build please ensure that you have removed the agent currently installed and ensure that C:Program DataWRData has been removed (if not please delete this folder:
http://download.webroot.com/9.0.17.32/WRSASME.EXE
Please ensure that you reboot the server after applying the above update. If you experience any further issues, please update your support ticket and the escalation team will get back to you promptly.
 
Thank you for your patience whilst we have investigated and developed the update. It is deeply appreciated by us all at Webroot.
 
Regards
 
Webroot Global Escalation Team”
Userlevel 5
Badge +24
We previously had an RDS server with the 4005 error, which we moved to Silent Audit mode as a result of the problem.  After installing 9.0.17.32, I moved the system back to our normal server policy.  So far no issues.
 
I have since updated all client RDS servers with this version and am monitoring further.
Userlevel 7
Badge +31
The patch addresses a discovered resource leak which was discovered in the current release of WSA v9.17.28
 
Other areas
 
In GSM, create a policy for all of your terminal servers.  Make sure that you copy the settings in the default server policy exactly and then switch off the following:-
 
  • ID Shield
  • Web Threat Shield
  • Firewall
  • System Optimizer
  • Core System Shield
  • Scan Schedule 
 
Once you've done that, assign all of your servers to that policy.
 
This will have the effect of turning down the agent resource usage ( you still have a decent level of protection ).  If there is still an issue, we have a smaller feature area to focus on. 
 
If the everything is stable, start turning each setting back on, leaving a reasonable time gap between each to assess the impact.
 
Jonathan
 
 
 
 
 
Is there something else besides this patch that needs to be done?
Userlevel 7
Badge +31
Please contact our support team immediately so that we can troubleshoot your issue
 
Regards
 
jonathan 
 
This patch does not work we still have servers crashing because of this.
Userlevel 7
Badge +31
Working closely with our customers, Webroot has identified an issue that manifests itself in the form of existing Terminal Server sessions becoming un-responsive with users no longer being able to log in.   The affected Terminal Servers have required restarting in order for normal service to be resumed.   For customers who have experienced this, an update to WSA is available now and Webroot support will provide assistance as required.   The fix will also be rolled into the next general release of WSA which is forecast to be automatically deployed in October.  Forthcoming product bulletins will advise of the exact date.
 
To download and install the update with the fix now, you may run the following executable on any server experiencing these issues:
http://download.webroot.com/9.0.17.32/WRSASME.EXE
If you experience any further issues, please update your support ticket and the escalation team will get back to you promptly.
 
Thank you for your patience whilst we have investigated and developed the update.  It is deeply appreciated by us all at Webroot.
 
If you wish to discuss this matter further or just to talk about the direction of WSA in general, please do contact me directly.
 
 
Regards
 
 
Jonathan.giffard
Senior Product Manager
WSA Business
E: .giffard@webroot.com
 
Userlevel 7
Badge +31
We've identified a resource leak in WSA that can lead to this situation.  There is a patch build being created, with a fix for this, that you will be able to get from support by the end of this week. 
 
The fix will be rolled into the next release of WSA which is expected in the fall of this year. More news on that will be posted neared the time.
 
 
Jonathan.giffard
Senior Product Manager
WSA Business
Userlevel 5
Badge +24
@ wrote:
@ wrote:
@ @ We have a server with 9.0.17.28 that is experiencing the Winlogon 4005 error.  It is a critical error for this client.
 
What version of Windows Server?
@ Server 2012 R2.
@ wrote:
@ @ We have a server with 9.0.17.28 that is experiencing the Winlogon 4005 error.  It is a critical error for this client.
 
What version of Windows Server?
Userlevel 7
Badge +31
We did indeed have several updates in 9.0.17.28 for various scenarios that could result in 4005 issues. We were aware that there could well be others.
 
Our support and engineering teams are working together on your issue and will be contacting you to obtain further details. 
 
Please feel free to contact me at any point
 
 
Jonathan
 
Jonathan
 
Userlevel 7
Badge +35
@ - those fixes were implemented with 9.0.17.24, but there may be something else going on. Our team is looking at the logs and the defect and will be in touch with you shortly. 
Userlevel 5
Badge +24
@ @ We have a server with 9.0.17.28 that is experiencing the Winlogon 4005 error.  It is a critical error for this client.
 
According to the following thread:
https://community.webroot.com/t5/Product-Releases/Product-Update-Bulletin-Update-for-Winlogon-Issues/td-p/295636
 
there was supposed to be a fix in July.  Can you let me know whether that fix has gone live yet?  I am now concerned about other clients of ours that may be experiencing this error, and need to know if I have to disable Webroot on our clients' terminal servers.
 
I have opened Ticket 106309 for reporting the issue and have uploaded WSALOGS.
Userlevel 7
Badge +35
We have an upcoming agent update which will address the winlogin errors on RDS terminal servers. Please see the post and product bulletins here.
 
Click to see post and bulletin
Userlevel 7
Badge +31
The new client does not contain fixes for 4005.  We're about two months away from having a release that contains addtional work to help resolve this issue. 
 
I'm expecting a  private beta before then ; if you are interested in trying that out, please contact me directly.
 
Regards
 
Jonathan.giffard
Senior Product Manager
WSA Business
Any update on this, it is a lot longer than 2 months since JAnuary and a new client is going out this week that has no mention of fixes in it.
Userlevel 1
That's good. Hopefully we can get something that works. Please email me the beta if you want me to run it on a test server I have. More than happy to help. 🙂
Userlevel 7
Badge +31
We're about two months away from having a release that contains addtional work to help resolve this issue. 
 
I'm expecting a  private beta before then ; if you are interested in trying that out, please contact me directly.
 
Regards
 
Jonathan.giffard
Senior Product Manager
WSA Business
 
Userlevel 1
Any updates at all on this? Still running our servers on low protection and so far not had any issues with the latest version but would like to put protection back to full.
 
Our subscription runs out in the next few months and I won't be renewing unless this is fixed. 
Userlevel 1
Thanks for the update. We are quite desperate for a solution now so hopefully it comes quickly. A bit of a worry that the fixes that seemd to help have been removed though - so wishing you all the best to get things done speedily.
Userlevel 7
Badge +31
As part of the changes that were made to quickly deal with the BSOD issue in 9.0.15.43, additional fixes for 4005 were removed  as a precautionary measure.   We're working hard on getting these fixes into a forthcoming, quality, release .   When I have some more details to share with you all, and I know that 4005 is causing pain, I will do. 
 
If there are any people who would like to take part in testing the fixes before the official release, please contact me using message system. 
 
 
Regards,
 
Jon.giffard
Senior PM
WSA - Business
 

Reply