The IT Crownd

Ok so I am a bit late on this one but like a persistent lover I get there in the end...:) What am I on about?  Powershell and Ops Manager. I just wrote my first PowerShell for Ops Manager and it rocks. Well...not really but I like it and it may save you some time too... How to set the Agent Proxy by PowerShell - would be a better title and probably help people find this! Get-Agent | Where {$_.DisplayName -match "contoso.com"} | foreach{$_.ProxingEnabled = $true;$_.ApplyChanges()} This will enable the Proxy Agent on all machines with contoso.com in the DisplayName. Nice

Well I have been busy! Looking at all the PowerShell I've been pasting together from bits and pieces found around the web (Thanks Google!) it's no wonder that I've been AFK or more correctly AFB for a couple of months. But now back with some PowerShell script-lets under my belt and here's one of them. We have issues *cough* "firewall ports" accessing some of our servers in our remote data centres to push the SCOM agent. We were unable to solve the *cough* "firewall ports" issue so we decided to go round the issue and manually install the agent on each server. I wrote a PowerShell and VBS script to accomplish this.  Here there are.  Nothing fancy and I am sure that things could be done better however this is how we are acheiving bypassing our firewall....errr I mean resolved our issue. The install is silent so you may want to adjust the MSI line and remove /quiet and /qn to help with troubleshooting this script in your environment. Power

I've been busy working on troubleshooting some Exchange 2010 issues, well not really issues with Exchange but issues within our internal processes around Exchange 2010. One of these is how do we work with new starters and how do we process when an employee leaves? This raises a couple of interesting changes in how Exchange 2010 "does things" our old process of dealing with leavers just disabled the AD account, people no longer got email.  This doesn't work in Exchange 2010. I now use an email policy to change the leavers email address to something valid within Exchange however not valid outside, i.e. no MX record point to this domain name. This then leaves all the old emails still valid and assigned to the mailbox.  So I found some nifty (I think) PowerShell to remove all the SMTP addresses from the mailbox except the primary SMTP address. Here you go: # Use this to remove all SMTP email addresses from a mailbox # except for the primary $mailbox = Get-Ma

So it seems that I am not alone in having issues with migrated accounts, even the might Microsoft themselves have issues here!  This is good as it means they go out and write some mean PowerShell which we can all use! Exchange 2010 is stricter in how it deals with delegates, if the user account cannot be found in AD then Exchange will refuse to show any of the delegates this results in Outlook hanging and eventually timing out when a user attempts to look at the delegates configuration tab. Here's the link to clearing out your old delegates! http://blogs.msdn.com/b/emeamsgdev/archive/2012/08/31/powershell-remove-invalid-delegates-from-mailboxes.aspx This is a really nice and simple method of removing invalid delegates - the nice thing is it adds back all the correct delegates too!

Since migrating our users to Exchange 2010 from Exchange 2003 we've had a few issues dotted around the organization where users are unable to edit or even view delegate information on their mailbox. A quick check in powershell shows that there are indeed delegates but these are delegates whose AD account is no longer around...simple fix really. To show all mailbox delegates try this... Get-Mailbox | Where {$_.GrantSendOnBehalfTo -ne $null} | select Name, @{Name='GrantSendOnBehalfTo';Expression={[string]::join(";", ($_.GrantSendOnBehalfTo))}} | Export-CSV C:\SendOnBehalfTo.csv The above will dump its output into a file on C: called SendOnBehalfTo.csv If you are dealing with just one customer you can simplify this to... Get-Mailbox <mailbox-alias>| Where {$_.GrantSendOnBehalfTo -ne $null} | select Name, @{Name='GrantSendOnBehalfTo';Expression={[string]::join(";", ($_.GrantSendOnBehalfTo))}} | ft -wrap In any case to remove a gho

So we've been running SMSPS for a while now, well, easily for two months with no incident.  Until last week. We got some nasty looking notifications, with " Clone Backup DB Error " in the Statistical Result section of the notification. As we have SME too it was already known that there have been issues when a cloned LUN is dismounted after backup validation.  We wondered if this was happening in SMSPS. Opening up SQL Management Studio on the server runnning the DBCC installantly showed the databases mounted included clones used in the DBCC process. Looking in Snap Drive there was the cloned LUN mounted in the folder default location. After detaching the DBs from within SQL and then removing the LUNs from SnapDrive things looked good. Well looked until the backup ran that night :) This time we got " Error During Backup Raw DB " which sounded even more scary. Turns out there was one last DB still mounted in SQL. Cleaning that out sorted things

We had a brand new iPad arrive destined for one of our established users. Nothing out of the ordinary there.  However when the Service Desk team attempted to connect the device to our Exchange Active Sync site the device failed to communicate with Exchange. This was not a case of being unable to sync but a steadfast refusal at the first! No logs and no events generated at all  Great! User was enabled for EAS (as is the default) in Exchange so it wasn't that...hmmmm. We noticed that the Active Directory account wasn't inheriting permissions from the parent.  I couldn't be that.  Could it? Ba-Zinga! - Sorry Sheldon!? Some needed permissions were missing (i've no idea what) and ticking the inhert sorted these out. Why this tick wasn't there in the first place? Go figure.

I've now un-hindered the comments process.  Everyone can comment un moderated.  I will of course prune those comments deemed unsavoury :)

*phew* Finally got everything working with our SnapManager for SharePoint install.  Here's a few gotchas for those brave souls about to embark on a similar voyage of the damned :) Whilst the install is relatively straight forward, almost next, next finish infact the configuration is the opposite. There are quite a number of requests for credentials during the install.  I think we used just two accounts in the end, one for accessing the DFM and one to access SharePoint DBs.  Be aware that there is a "hidden" service running as a user account.  We found that the password was not transferred correctly from the installation wizard to the registry for this account.  Luckily there is a commandline tool to change the password. sdcli dfm_config set -host <hostname> -user <username> -pwd <password> As with all NetApp documentation.  Read it.  Read it and read it again.  Then ignore the majority of it and look elsewhere.  Forums offer much better examp

I've been looking for a way to make our NetScalers load balance the SMTP traffic to our Exchange 2010 environment. Here's a blog posting which shows you how to do it. http://citrix.stefanriek.de/citrix/howto-load-balance-while-preserving-a-clients-source-ip-but-not-using-the-netscaler-as-your-gateway/ Basically you need to make a new service group with ANY as the protocol on which you have enabled USIP. Then in the Virtual Server (again protocol ANY) you select the service group and enable MAC redirection. Here is the kicker. You now need to edit your Exchange server!  Well you need to add a loopback adapter with the IP of the Virtual Server in NetScaler.  Then enable weakhost receive and send on the loopback adapter and enable weakhost receive on the live adapter. netsh interface ipv4 set interface "Your production network adaptor name" weakhostreceive=enabled netsh interface ipv4 set interface "Your loopback network adaptor name" weakhostrecei

I recently found out why our MOSS 2007 FEW which was hosted on a Windows 2008 R2 server was unable to provide MOSS sites. Whilst this was not a service impacting issue, we have plenty of other SharePoint 2007 FEWs so one less didn't matter plus this one was really just doing the Report Service bit in any case. However after a couple of weeks of getting annoyed at this I decided to look for a solution.  A quick trawl of Google later and I found this site which explains things nicely. http://sharepointspot.blogspot.co.uk/2008/12/sharepoint-kerberos-on-windows-2008.html Basically this is due to Windows 2008 R2 doing kerberos in the kernal and as such it ignore any SPN you may have in place and uses the computer object.  In principle this is a great thing, no more SPNs needed!  However poor old MOSS needs SPNs as all the website applicaiton pool MUST run using identities if you are using kerberos. Quick change to the C:\Windows\System32\inetsrv\config\applicationHost.config f

Nothing is every easy. Last week I installed the SCOM agent on our shiny new TMG array.  I followed the instructions from Microsoft and everything went smoothly.  Until I attempted to approve the last member of the array within SCOM Management Console.  An error occured which ditched my console.  I didn't worry too much and I thought nothing of it till the next day when the last server wasn't appearing in SCOM. The console however no longer showed the server as pending - infact the console didn't show the server at all. I found some powershell lying around on the internet which allowed me to approve/reject and list computers in "pending management" there was my server!  I duely approved the pending operation and bingo back in business. Gotta love Powershell.  Just a shame that they don't use powershell to populate to console! To list pending agents...(oh yeah you need to run the Operation Manager Shell - not just Powershell you knew that right?) ge

We've had quite severe performance issues on our Exchange 2010 implementation since roll out.  Basically we ran out of CPU.  Not nice. The replication service on both mailbox servers slowly ramped CPU usage so that the server went up to 100% which then caused all manner of nasty failures. Through-out our troubled times we have Looked at the Exchange 2010 configuration with a fine toothed comb Had 3 "Exchange Experts" come in and check things out Raised PSS call with Microsoft Raise call with VMWare (the servers are all virtual) Increased RAM and CPU to ESX limits (almost - we only have 96GB in our hosts) Tweaked registry settings Moved to use pvSCSI instead of vSAS and back again (pvSCSI has issues in high IO environments!! We need new pvSCSI drivers) Rebuilt a second set of Mailbox servers None of these things has shown us the cause or indeed fixed an unknown cause.  Until we made a configuration mistake. The guys who rebuilt the environment failed to in

Grrr....pulling my hair out on this one. We have SCOM 2007 R2 (which I love) and we are running the excellent management pack for Exchange 2010. If you read an earlier post I finally figured out that the Agent Proxy needs ticking to make discovery work so we are humming along now. Except for one thing. My Public Folder server. We have one mailbox server dedicated to public folders.  The problem here is that this server is never healthy.  KHI: Failed to execute Troubleshoot-DatabaseSpace.ps1 fails. Looking into this script I see that this script uses the Get-MailBoxDatase this will not work on a public folder.  Further investigation leads me to believe that all the scripts within the Exchange 2010 Management Pack are not Public Folder compatible.  Now whether this is because a discovery which isnt running in my environment I don't know...*grrrr*

Bleeding edge?  So you will proboably know what PVSCSI is all about We had two days of worry - our Exchange 2010 Mailbo server was showing events of having a corrupt database.  The event description pointed at the storage.  ESE 482: Thus ensued a witch hunt for the actual culprit.  Storage logs reported all fine...the issue came and went.  Unfortunately when it came the databases failed over to the other node of the DAG. We found a page on VMWare's site detailing that a high IO server could have issues when using PVSCSI and that updates to the driver are required.  The driver updates pre-requiste was of course a patch to ESX 4 too.  Not something we can do on a whim, change management anyone? See here: http://longwhiteclouds.com/2012/03/14/win-2k8-with-pvscsi-critical-issue/#more-801 and here: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2009144 So we had to remove the PVSCSI devices.  This is actually muc

Bit of a fool on this one! Spent a couple of days wondering why our new Exchange 2010 boxes weren't appearing in SCOM only to remember (with a hint from a Google search) that Agent Proxy needs enabling.  Flush Health Service State and Cache. Bingo. d'Oh! Note to self...read the manual!!

From the title you can guess that we have Exchange 2010 (SP1 CU6 - with Hotfix for EVault) and we are using NetScaler VPX to load balance the services.  I used the excellent document from Citrix to configure the NetScalers (we have two in active/active - sort of using VMACs to split the traffic RPC to one, HTTP(S) and SMTP to the other). However I found a document from Microsoft which points out that there is "significant performance penalty" if we configure incorrect persistence for EAS.  The Microsoft document is here: http://technet.microsoft.com/en-us/library/ff625248.aspx This put us in a pickle and no mistake.  This is because in a NetScaler persistence is per vServer.  We have one vServer for HTTPS as we have one IP address for all HTTP traffic.  This vServer would then deal with all HTTPS traffic including OWA,ECP,EWS,RPC over HTTPS and EAS.  OWA requires cookie based persistence, this is also the method the Citrix document recommends.  From the Micros

Recently came up against a brick wall when rolling out 120 or so laptops to our users.  This monster issue came in the guise of the failing 3G cards within our Dell laptops. We saw that after a build of Windows 7 the cards functioned fine and were able to connect to the local provider after configuring. When we came to do our final checks we noticed that some of the laptops had failed 3G connections.  This failure was not laptop model based (we had two models both of which randomly showed the 3G failure) Since the only thing between build and the check was some Windows patches we surmised that one or more patches were the issue.  There were 85 patches.  This took a while.  It turns out that even after removing all the patches the 3G card refused to function correctly. More investigation turned up that the WWAN service was disabled.  Enabling and starting this service sorted out the problem.  Hoorah. Not quite. After a while some laptops showed that the service was disabled ag

As I thought that no one was reading my pages I left most settings to default.  I have however found that some people are reading my pages so have enabled comments for you to tell me how wrong I am! Cheers.

Here's how I got authenticated SMTP working for our environment.  This is pretty useful if you need to send external mails. My Kit: System Center Operations Manager 2007 R2 CU5 Exchange Server 2010 SP1 RU6 Make/choose an AD account with an email address which will send your notifications. In Ops Manager create a Windows Account using the credentials created above. Assign these credentials to the Notifications Account Run As Profile. Set up a channel to use authenticated SMTP using the email address of the account used in step 1. If you want/need create an Exchange Receive Connector with Windows Authentication and Exchange Users permissions group. Test and enjoy. This is pretty simple really but took me a while to figure out as I failed to see that my Notification Account didn't have the email address I set as the Return Address in the Notification Channel wizard.

Now this is something I am quite pleased with however after coming up with the idea and Googling I found plenty of other cleverer people had come up with this before me! Here's one very good example: http://operatingquadrant.com/2009/08/15/scom-automatically-starting-maintenance-mode-when-servers-are-rebooted-for-patching/ Anyways one of the maybe overlooked aspects of SCOM notification is that you can run commands, meaning that an event/alert can then trigger a series of scripted actions.  Really cool. What the blog above explains is how to use this feature to check for an event signalling that the Windows server is going to reboot this is event ID 22 from Windows Update.  I use event ID 1074 as we do not reboot our servers automatically when patching, we do however use Citrix and some of these servers have a reboot scheduled.  Event 1074 is raised by USER32 in the SYSTEM log when a scheduled reboot is attempted.  So you can now safely put your servers into maintenance

Ok a bit late but still happy new year!