«0x80070721 A security package specific error occurred» while using WMI between domains

Symptoms:

You have two IBM System x servers with Windows Server 2012 or newer and with «IBM USB Remote NDIS Network Device» network interface enabled. Both of these servers reside in the same AD DS forest but in different domains. You try to setup a WMI session from one to another (using wbemtest, for example). In that case WMI connection fails and you receive an error:

Number: 0x80070721 Facility: Win32 Description: A security package specific error occurred.»
In System log of the server which initiates the connection we have:
Log Name: System
Source: Microsoft-Windows-Security-Kerberos
Date: 10/6/2013 9:13:09 PM
Event ID: 4
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: SRV1.alpha.example.com
Description:
The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server srv1$. The target name used was host/srv2.beta.example.com. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Ensure that the target SPN is only registered on the account used by the server. This error can also happen if the target service account password is different than what is configured on the Kerberos Key Distribution Center for that target service. Ensure that the service on the server and the KDC are both configured to use the same password. If the server name is not fully qualified, and the target domain (ALPHA.EXAMPLE.COM) is different from the client domain (BETA.EXAMPLE.COM), check if there are identically named server accounts in these two domains, or use the fully-qualified name to identify the server.

Why does this happen?

Starting from Windows Server 2012, when one machine connects to another computer’s WMI, Windows asks remote system for IP-addresses of its network interfaces. Then operating system chooses which one of them serves its needs to connect best.
In case of IBM System x, both servers have network interface with the same IP-address – 169.254.95.120. Windows chooses this IP-address as the best and tries to connect to it. But instead of remote system, it connects to itself and you see the error.

When you try to connect with wbemtest, it calls a WMI API. In the background WMI uses DCOM to communicate with other servers. When DCOM establishes a session it uses «ServerAlive2» query to check the other server.
Here is what network capture looks like:

14 14:38:55 10.01.2014 0.0111483 srv2.beta.example.com 135 (0x87) 91.103.70.14 55535 (0xD8EF) DCOM DCOM:IObjectExporter:ServerAlive2 Response {MSRPC:10, TCP:9, IPv4:8}
NetworkAddress: srv1
NetworkAddress: 169.254.95.120
NetworkAddress: 192.0.2.1

Same traffic when we access from another end:

38 14:37:12 10.01.2014 37.1871388 srv1.alpha.example.com 135 (0x87) 10.253.12.2 59278 (0xE78E) DCOM DCOM:IObjectExporter:ServerAlive2 Response {MSRPC:14, TCP:13, IPv4:8}
NetworkAddress: srv2
NetworkAddress: 169.254.95.120
NetworkAddress: 203.0.113.2

The application layer chooses the common IP-address and loops back the real connection. The DCOM gets a Kerberos ticket, and tries to authenticate with it, so this is why we see AP_ERR_MODIFIED errors from Kerberos. So a DCOM based communication (for example WMI) won’t work if the participants has a common IP address.

This problem is known by Microsoft and will not be fixed since it is by design.

How to mitigate it?

  1. Just disable IMM USB Network Interface at one or at both servers. But beware: updating of IMM firmware from Windows or using ASU64 will enable this interface again. If you choose this option, I suggest you to setup monitoring to alert you when that interface leave enabled.
  2. Change IP-address at one if the interfaces to something else. You can use almost any private address but here are recommendations from IBM. You even can use DHCP-server to achieve it (I hope you are using separate VLAN for management interfaces, aren’t you?).

SCDPM: Fail to Modify Disk Allocation after Exchange DAG Switched

Symptoms:

Suppose, you have an Exchange 2010 installation with one or more Database Availability Groups with 2 or more servers in each DAG. You setup backup for one of these DAGs using DPM 2010 UR? or newer (incl. 2012 R2 UR2). Later you change active status for protected copy of mailbox database (for example, you switch active copy to another mailbox server). After that, for database copies which status has changed, you’ll receive following error at Review Disk Allocation page at New/Modify protection group master at DPM console:
"The operation failed because the data source VSS component {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7} is missing.
Check to see that the protected data source is installed properly  and the VSS writer service is running.

ID: 915
Details: The operation completed successfully (0x0)”

If you’ll try to add such DB to secondary DPM server, you’ll receive same error at a disk size calculation step.

This problem is known by Microsoft and will not be fixed.

Why does this happen?

DPM stores information about protected resources into tbl_IM_DataSource and tbl_IM_ProtectedObject tables in DPMDB. If you look into ApplicationPath, LogicalPath or PhysicalPath cells, you find an XML-document describing protected resource. Here is one for Exchange mailbox database in DAG:

DAGNODE2.example.com – DAG-node from which database is protected.
MAILDB01 – name of protected DB
Microsoft Exchange Server\Microsoft Information Store\Replica\DAGNODE2 – path to a copy of protected DB at a DAG node which we are protect. Mind “Replica” element of the path – it means we protect passive (not active) copy of DB. In case of active copy, this part of path won’t exist.

When you change status of mailbox database in DAG, the actual LogicalPath changes, but DPM knows nothing about it and keeps an inconsistent data in DPMDB.

Resolution:

There are two workarounds (choose which suits you best):

At DPM side:

  1. Stop protection of problematic DB with retaining its data.
  2. Add the DB back into an appropriate protection group. DPM will update tbl_IM_DataSource and tbl_IM_ProtectedObject tables.
  3. When consistency check completes, you’ll be able to manage allocated disk space for this DB and setup secondary protection for it.

At Exchange side:

  1. Restore active state for problematic DB as it was when you added it into DPM:
    1. If the DB was in an active state – make it active again.
    2. If the DB was in a passive state – make it passive.

If you require so, you can reinstate DB’s state after modifying disk allocation / setup secondary protection – it doesn’t interfere with synchronization / recovery points creation – it just makes impossible to calculate size of a DB.

SCCM: Device Collection Based On a Local Group Membership

New task came up recently – I need to separate in SCCM self-managed workstations from IT-managed ones. We define following criteria for IT-managed workstations: no other accounts are in local Administrators group except for built-in Administrator, Domain Admins group and a group for Service Desk administrators. All workstations are located in the same OU, so I cannot use OU-based collections.

As you may know, SCCM 2012 doesn’t have built-in tools to get local groups membership. Thanks to Sherry Kissinger who solved this problem for us using Compliance Settings. After you install her package, you’ll get a new Configuration Baseline and Configuration Item in SCCM console named as “WMI Framework For Local Groups with Logging” and “Local Group Members into WMI with Logging”. This package also creates 2 new tables and 1 view into SCCM database: LocalGroupMembers_DATA, LocalGroupMembers_HIST and v_GS_LocalGroupMembers0.

After creating and deploying baseline, you can use v_GS_LocalGroupMembers0 view to create reports based on local groups membership.
Don’t forget: you must not deploy that baseline to domain controllers! For example, you can create a collection which includes all your systems except domain controllers: create new device collection using All Systems as limiting collection and add it with include rule, then add All Domain Controllers collection with exclude rule. You can download MOF-file for such collection here.

Unfortunately, neither LocalGroupMembers_DATA, nor v_GS_LocalGroupMembers0 can be used in WQL-queries when you create a collection.
Am I stuck? Let’s review what do I had for now:

  • I have all data about local groups membership in custom table.
  • I can create any reports using that data.
  • I can create collections using data from standard tables in SCCM DB.
  • But I cannot create collections based on a data from custom SQL-tables.

I need a way to put data from table LocalGroupMembers_DATA into standard SCCM tables and PowerShell is here to save the day.
There are at least two ways to get data from SQL with PowerShell:

  1. Connect to DB directly and use T-SQL queries with SQL cmdlets.
  2. Connect to SQL Server Reporting Services using New-WebServiceProxy cmdlet. Stefan Stranger and Jin Chen wrote an example script to achieve it.

With PowerShell we can do anything with that SQL-data. Our goal is to populate device collections with workstations and here we go again with two different options:

  1. We can add computers into group in AD DS and then create a device collection using this group. For this method to work you need to activate Active Directory Group Discovery discovery method for site and domain where AD group will reside.
  2. Add computers into collection directly using Add-CMDeviceCollectionDirectMembershipRule cmdlet.

Since both group and a report will be useful for me in the future, I’m stick with them.

Now our scenario looks like this:

  1. Activate Active Directory Group Discovery.
  2. Collect local group membership using Compliance Settings.
  3. Create a report with gathered data an any SSRS.
  4. Get names of computers from this report with New-WebServiceProxy cmdlet.
  5. Add these computers into an AD group.
  6. Create a device collection by that AD group.

I build a report where I list all computers don’t comply with conditions discussed earlier.
Here is what first DataSet query looks like:

It can be easily expanded to include another set of groups to ignore.
Mind CompOU parameter: in web-interface you can select multiple OUs where to search computers.
To get a full list of OUs from a forest, you can use another query:

I modified RenderSQLReportFromPosh.v1.000.ps1 so it could populate AD DS group in addition to get data from reports. Here’s its code:

My modified script receives a report from $URL and $ReportPath locations, compares a list from it with members of $GroupName AD DS group and adds/removes computers from that group until it and the report would be the same.
You can find a path for log of actions in $Log variable. Here, script records all computers which were added or removed from the group.
OUs to search are defined into $param1 and $param2 variables. If you need more OUs, create a new parameter variables and do not forget to add them into $parameters.

As last, I created standard device collection based on AD group $GroupName.

You can download report as an RDL-file and a script here. Do not forget to create DataSource in the report to connect to your SSRS instance.

SCCM: All Domain Controllers Collection

There are many ways to create a collection containing all domain controllers. Here are some examples:

By a role of a computer:

By the primary AD DS group:

By AD DS group name:

I personally prefer the first version, by a role of a computer. You can download a MOF-file for this collection here. Just import it as described in How to Create Collections in Configuration Manager article and new “All Domain Controllers” collection will appear in your SCCM console.

Process Monitor 3.10

An important update of Process Monitor was released couple of a days ago:

As you may know, we can use two functions with completely opposite names to open registry key: RegCreateKeyEx and RegOpenKeyEx. When you use RegCreateKeyEx, it creates registry key if it’s non-existed, but just opens it if key exists. RegCreateKeyEx writes which operation (create or open) it’s performed into a separate variable.
RegOpenKeyEx cannot create registry keys and returns error if key doesn’t exist.

Before this release there were no way to determine what operation exactly RegCreateKeyEx perform. “Granted Access” property for execution of that function always contained “Read/Write” value.

From this last update, Process Monitor finally can show you what RegCreateKeyEx does. There is no “Granted Access” property for RegCreateKey operation anymore, it was replaced with new “Disposition” property. “Disposition” may contains following strings:
REG_CREATE_NEW_KEY – if new registry key was created.

REG_OPENED_EXISTING_KEY – if RegCreateKeyEx just opened previously existed key.

“Desired Access” property still contains “Read/Write” value, because we cannot predict which action RegCreateKeyEx will do.

Using this new feature, you can separate RegCreateKeyEx calls: just add new condition into Process Monitor’s filter with following parameters:
Column – “Detail”
Relation – “contains”
Value – “REG_CREATE_NEW_KEY” or “REG_OPENED_EXISTING_KEY”
Action – “Include”

Moved protected server between primary DPM servers – cannot add secondary protection

SYMPTOMS:

You have one secondary DPM-server (DPM3) and 2 primary connected to it (DPM1 and DPM2). You have a server (PS1) protected by DPM1 and DPM3. You decide to move protection of PS1 from DPM1 to DPM2. You stop protection of PS1 at DPM1 and DPM3. You connect PS1 to DPM2. You wait until first replica is created, then you try to add PS1 back to DPM3 but from DPM2 this time.

In that case you cannot see PS1 at DPM3 in the list of data sources protected by DPM2.

CAUSE:

This happens because each protected server has a single record in a table of protected servers at DPM configuration database. One of the attributes of this record is an ID of DPM server protecting that data source. Since secondary DPM knows nothing about you moving data sources between primary servers, it doesn’t update that record with an ID of a new primary DPM server.

RESOLUTION:

You can fix this by manually updating DPMServerId attribute of migrated protected server with an ID of new primary DPM server.

WARNING! This scenario IS NOT supported by Microsoft. Use instructions below at one’s own risk, except and only when you are specifically instructed by Microsoft technical support specialist to do this.

Be sure to backup your DPMDB database before run any T-SQL command!

  1. Connect to DPMDB of your secondary server (You can find more about it here).
  2. Get a list of DPM-servers and their IDs:
    SELECT ServerId, ServerName, DPMServerId FROM [dbo].[tbl_AM_Server] WHERE IsDPM = 1
  3. Write down IDs of DPM1 (<OLD-ID>) and DPM2 (<NEW-ID>).
  4. Determine which primary DPM-server protects PS1 according to the knowledge of DPM3:
    SELECT DPMServerId FROM [dbo].[tbl_AM_Server] WHERE ServerName = '<Protected Server's FQDN>'
  5. Make sure it equals <OLD-ID>, otherwise this instruction is not for you — contact Microsoft Support instead.
  6. Update protected server record with ID of new primary server:
    UPDATE [dbo].[tbl_AM_Server] SET DPMServerId = '<NEW-ID>' WHERE ServerName = '<Protected Server's FQDN>'
  7. Run new/modify protection group master and press “Clear cache” button.
  8. You are now able to protect moved resource at secondary DPM again.
  9. Never ever move protected servers between primary DPMs connected to the same secondary server, since it is NOT supported by Microsoft.

SIMILAR ISSUES:

How to connect to DPM SQL database (DPMDB)

Connection parameters for System Center Data Protection Manager configuration database are located into registry key HKLM\SOFTWARE\Microsoft\Microsoft Data Protection Manager\DB on the computer where DPM server is installed. You can quickly determine essential parameters by running following PowerShell commands:

SQL server name:
(Get-ItemProperty 'HKLM:SOFTWARE\Microsoft\Microsoft Data Protection Manager\DB').SqlServer
>DPMSRV

Database name:
(Get-ItemProperty 'HKLM:SOFTWARE\Microsoft\Microsoft Data Protection Manager\DB').DatabaseName
>DPMDB_DPMSRV

SQL server instance:
(Get-ItemProperty 'HKLM:SOFTWARE\Microsoft\Microsoft Data Protection Manager\DB').InstanceName
>MSSQLSERVER

Use SQL Server Management Studio to connect to DPM database using values from above. If SQL server instance name is “MSSQLSERVER”, you should omit it while connecting to SQL server.
Only local administrators group is alowed to access DPMDB by default. If you run SSMS on a DPM server locally, don’t forget to run it as administrator.

HP Lights-Out Passthrough Service cannot be installed at a terminal server

SYMPTOMS:

When you install HP Lights-Out Passthrough Service at a Terminal Server (which means you have RDP Service enabled for multiple users) you receive following errors:

  • In installer: Service ‘HP Lights-Out Passthrough Service’ (hplopts) failed to start. Verify that you have sufficient privileges to start system services.
  • In Application Log: The Passthrough Service could not be started. You must install Terminal Services or enable Remote Desktop.

SOLUTION:

This happens because of weird prerequisites check.

Set registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\fDenyTSConnections = “0”. Installation should succeed.

Unable to find the callback library jcb.dll (or one of its dependencies)

If you need to defragment your Exchange database at a computer without Exchange components, you will probably use this MS KB: 244525: How to run Eseutil on a computer without Exchange Server. I tried it when I was defragmenting Exchange 2003 database at a Windows Server 2008 R2 machine.

Eseutil was running fine, but after some time (when it used just over 2 GBs of RAM), I received an error described here: 273087: Error With Jcb.dll While Running Eseutil. Unfortunately, none of the methods described there were useful: when I pressed «Cancel» button, I received an «Operation terminated with error -2102 JET_errCallbackNotResolved, A callback function could not be found) after 1168.136 seconds.» error.

Eventually, I grabbed Process Monitor and found out that files described in Microsoft’s KB 244525 and in this thread «Re: JCB.DLL Not Found Error» weren’t enough – you need another one file: ntlsapi.dll. I copied it from a nearest Windows Server 2003 R2 SP2 box to a system where eseutil were working and everything went smoothly.

NetBackup: Client Attributes: The error is ” (1)”

SYMPTOMS:

When you are trying to change or delete client attributes from client list in master server properties, you get an error:
Unable to save data on some hosts
An error occurred on host example.com. The error is " (1)".

RESOLUTION:

  1. Go to NetBackup installation folder (usually it’s %PROGRAMFILES%VeritasNetBackup). Then proceed to dbclient folder.
  2. If there is only one file and it is named “GP_”, go to step 3. If not, I can’t guarantee anything.
  3. Delete dbclient folder.
  4. Restart NBU services.