{"id":9146,"date":"2008-07-29T20:22:00","date_gmt":"1999-11-29T20:00:00","guid":{"rendered":"https:\/\/aidanfinn.com\/?p=9146"},"modified":"2008-07-29T20:22:00","modified_gmt":"1999-11-29T20:00:00","slug":"a-particularly-odd-opsmgr-2007-problem-and-solution","status":"publish","type":"post","link":"https:\/\/aidanfinn.com\/?p=9146","title":{"rendered":"A Particularly Odd OpsMgr 2007 Problem (And Solution)"},"content":{"rendered":"<p>The Operations Manager 2007 agent and management server communicate with each other and perform mutual authentication using Kerberos.\u00a0 They&#8217;re in the same forest and hence in the same Kerberos domain.\u00a0 But what happens if you have agents outside the forest?\u00a0 If you read anything from Microsoft (or the OpsMgr book I just bought) you&#8217;d be left under the impression that you <em>must<\/em> install the OpsMgr gateway.\u00a0 You&#8217;d then install a custom X.509 cert (requiring a cert server running on Windows Enterprise Edition) on that machine and on the OpsMgr server.\u00a0 There&#8217;s two problems with this:<\/p>\n<ul>\n<li>What if the un-trusted network is a workgroup, e.g. a DMZ?\u00a0 There&#8217;s no Kerberos domain for the agents on the network to authenticate with the Gateway.<\/li>\n<li>What if you are monitoring many networks with only one or two agents on each network?\u00a0 Are you going to install lots and lots of Gateways?<\/li>\n<\/ul>\n<p>If you are persistent with your searches you will find that:<\/p>\n<ul>\n<li>There is one mention by Microsoft in a downloadable Word document that you can install agents with the X.509 cert so that the agents can communicate directly with the management server.<\/li>\n<li>There is an almost complete <a href=\"http:\/\/www.mcalynn.com\/2007\/08\/certificate-based-agents-are-a-no-brainer-better-think-again\/\">guide<\/a> by Duncan McAlynn on how to install the certs using MOMCERTIMPORT \/SUBJECTNAME (the subject name is the name of the cert in the certificate store).<\/li>\n<\/ul>\n<p>Duncan appears to be the only person to have attempted to document this process so he deserves credit for it.\u00a0 The MS documentation folks have done a poor job with OpsMgr, e.g. failing to cover this subject and failing to document complete management pack authoring.\u00a0 The instructions for setting up the CA are in the <a href=\"http:\/\/www.microsoft.com\/downloads\/details.aspx?FamilyID=49369779-93F7-463F-B2A5-8555B11C5683&amp;displaylang=en\">OpsMgr 2007 Security Guide<\/a> and Duncan walks you through installing the agent.\u00a0 The only missing step is you need to install and import CA and agent certs on the OpsMgr management server(s) so that they have a means for mutual authentication with the agents.<\/p>\n<p>I&#8217;d been doing this successfully on servers and then I hit one server where the agent could not use the cert.\u00a0 I saw the following in the Operations Manager Event Log:<\/p>\n<p><em>Source: OpsMgr Connector<\/em><\/p>\n<p><em>Type: Error<\/em><\/p>\n<p><em>Event ID: 21036<\/em><\/p>\n<p><em>The certificate specified in the registry at HKEY_LOCAL_MACHINESOFTWAREMicrosoftMicrosoft Operations Manager3.0Machine Settings cannot be used for authentication.\u00a0 The error is The credentials supplied to the package were not recognized<br \/>(0x8009030D).<\/em><\/p>\n<p>I reissued that cert, re-imported it, re-installed the agent half a dozen times.\u00a0 I&#8217;d opened a call with MS (thanks to IT Pro Momentum) but the first PSS agent was not the Mae West to deal with.\u00a0 He kept claiming the my CA was at fault but I knew it wasn&#8217;t &#8211; other agents were fine.\u00a0 Finally the ticket got reassigned to Brian who was a pleasure to work with.<\/p>\n<p>He started coming up with some new ideas straight away.\u00a0 The first was maybe the cert store was corrupt.\u00a0 I tried a fix for that (CERTUTIL -F -REPAIRSTORE MY \u201c&lt;thumbprint of agent cert&gt;\u201d) but that didn&#8217;t fix the problem.\u00a0 Brian asked if we could look at the server together using &quot;EasyAssist&quot; &#8230; it&#8217;s MS&#8217;s answer to WebEx or LogMeIn so they can get Remote Assistance over web friendly protocols.\u00a0 We poked around and saw something interesting.<\/p>\n<ul>\n<li>The CA cert in ComputerTrusted Root Authorities was fine.<\/li>\n<li>The agent cert in the ComputerPersonal store was fine.\u00a0 The certification path was fine.<\/li>\n<li>When you run MOMCERTIMPORT it copies the cert into ComputerOperations Manager in the certificate store.\u00a0 I had overlooked this.\u00a0 Here, the certification path was invalid.\u00a0 Weird, because it was fine in the ComputerPersonal store.<\/li>\n<\/ul>\n<p>We manually imported the cert into there and the certification path was still screwed.\u00a0 We re-imported the CA cert but it was still screwed.\u00a0 We re-imported the CA cert and the operations manager copy of the cert.\u00a0 The certification path was fine but the agent didn&#8217;t appear to be using it.\u00a0 We re-ran MOMCERTIMPORT and the certification path was invalid again.\u00a0 OK &#8230; I thought we&#8217;d try this:<\/p>\n<ul>\n<li>Delete all copies of the agent and CA certs from the certificate store.<\/li>\n<li>Brian suggested restarting the cryptography and the OpsMgr Health service.<\/li>\n<li>I went through the process of re-importing: Import the CA cert into ComputerTrusted Root Authorities, import the agent PFX into ComputerPersonal, re-run MOMCERTIMPORT \/SUBJECTNAME and restarted the OpsMgr Health service.<\/li>\n<\/ul>\n<p>Lo and behold &#8230; it worked!\u00a0 In fact, it worked so well that we detected a hardware fault on the server that we hadn&#8217;t known about.\u00a0 Sweet; OpsMgr rules!<\/p>\n<p>A big &quot;Thank You&quot; to Brian for helping out on that one.\u00a0 For the most part, I&#8217;ve always had good dealings with MS PSS agents going back to 2003.\u00a0 It was good to see this one being rescued so professionally.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The Operations Manager 2007 agent and management server communicate with each other and perform mutual authentication using Kerberos.\u00a0 They&#8217;re in the same forest and hence in the same Kerberos domain.\u00a0 But what happens if you have agents outside the forest?\u00a0 If you read anything from Microsoft (or the OpsMgr book I just bought) you&#8217;d be &hellip; <a href=\"https:\/\/aidanfinn.com\/?p=9146\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;A Particularly Odd OpsMgr 2007 Problem (And Solution)&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"footnotes":""},"categories":[35],"tags":[],"class_list":["post-9146","post","type-post","status-publish","format-standard","hentry","category-scom-2007"],"aioseo_notices":[],"jetpack_featured_media_url":"","amp_enabled":true,"_links":{"self":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/9146","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9146"}],"version-history":[{"count":0,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/9146\/revisions"}],"wp:attachment":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9146"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9146"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9146"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}