Update on NetApp and CDW debacle….

since there has been SOOO much interest in this, I thought I would post an update of what has happened since my last post.

True to their word, a NetApp engineer called at 8:30 AM yesterday and worked with us until about noon.  Ah, progress.  We then broke for luch while some large files were ftp’d up to NetApp.

Upon returning, there was an email from the NetApp engineer explaining he had received some of the files already.  But also he forgot to mention that he had a dentist appointment that afternoon and would not be able to work on the case any more that day.

So this case would be dragged on until day eight.  Eight days without replication to the DR site.  Eight days without resolution.

Not satisfied with this answer, we called the “owner of the case” to see if someone else could look at the data or at least verify everything was being received properly.  No answer.  Left a voicemail.  As of this morning, still no response from her.

Meanwhile, we came into work  this morning and had an email from the engineer…apparently some of the files we were uploading to NetApp’s FTP site “didn’t make it.”

Every day in this little adventure is a fresh hell….

Still no word from my CDW reps on what they are doing to get this expedited.  Apparently I must have really pissed them off as I seem to be abandoned.

Moving at the speed of business…not.

Also in the news…when we had our nine-person conference call that was us essentially asking the same question we had been asking for 4-8 weeks (depending how you count) to a bunch of people, one of the things we asked NetApp for was a white paper or any kind of document or “cookbook” on implementing DR.  What we were particularly interested in was how the NetApp DR solution worked in conjunction with AD and how to avoid any potential pitfall especially when testing.

That was four days ago.  We still haven’t hear back on this and will probably need to ask another twelve times.

I guess I can understand why it would take so long though.  Probably a big document.  If only there was a method of transmitting data and the written word at lightning-like speeds across the country…

UPDATE 09/01/12: Dimitri, I am closing the comments thread.  I am not going to revisit this again as I have spent too much time on this already.  If you want to know the details, I’m sure NetApp has some sort of record of it after 34 phone calls, eight techs, 8 Gig of uploaded data, and 20+ emails.  Suffice to say many of your assumptions in your post are off base and not germane to the issue or applicable to our situation.  It was a well establish system that had a major issue with an update that for all our feet-stamping and complaining to both CDW and NetApp was ignored for almost eight days. No amount of Monday-morning quarterbacking will change that now.

(Update 10-15-2012)

NetApp, CDW, and how not to treat customers

The opinions expressed on this site are the authors personal views and do not reflect those of his employer or his clients in any way.

Back in May of 2011 my company purchased a NetApp solution from CDW that consisted of two production NetApps ans and a passive NetApp solution at a DR site hosted in one of CDWs data centers. The servers were virtualized servers in a VMWare environment hosted on some pretty robust HP hosts.

Almost from the beginning the problems began. CDW sent one of the “best” trainers to work on the implementation and setup and for the knowledge transfer. To say that the whole experience was an abomination would be accurate. The configuration he wanted us to implement based on his “best practices” (three spare drives and two parity drive per twelve disk shelves) was wrong and left little room for growth, the tools he told us to use were out of date and not recommended to be used any more by NetApp, he could figure out how to get drives to mirror, couldn’t create LUNs and map them, etc. In short, it was not a good start.

We worked with CDW to get some remediation and they actually sent someone to get the system up and running for the most part. We still had many things to get running, but the ball was rolling. That’s when the second shoe dropped. We got a project manager that was extremely disorganized, did not follow through on items, and who’s solution to any issue was essentially for use to call the vendor rather than pulling one of the CDW techs (who we had contracts with) to help us. When we did reach out to one of the techs for assistance (per the tech’s own instructions) she was more concerned about chastising us the customer than actually getting us the help we needed…and paid for.

We then, at our insistence, assigned a new project manager. She was very good and got us to where we needed to be. Almost eleven months after we started, she guided us through getting all the little pieces and parts working.

But the process was not smooth and was fraught with setbacks, poor and unknowledgeable employees, threats, tantrums, and hard feelings. But we were done.

And then something happened….

Well, more specifically a few things happened. The first was that we reached the point we needed to start testing the DR solution. We quickly discovered that the best way to do this, and one that provided the best solution for data recovery in a non-disaster scenario was to use FlexClone. This allows an image of a mirror to be brought up without breaking the original mirror which would require a re-synch or possible a re-initialization of data.

Because our DR site was a “passive” site it was not licensed for this or many other items. CDW came up with a quick solution and worked with NetApp to get both of our controllers into active status if we purchased an additional disc shelf. A bit too much like arm-twisting for my liking but I justified it by seeing that with this new space and the active configuration, we could now do some archiving with the NetApp. And most importantly, all sites would now, per CDW, “be the same.” This would come back to haunt us as they were not truly “the same” but were missing a piece of software that would become key in this sorry little tale.

Around June of this year we noticed something at NetApp changed. Prior to June, calls were answered in fairly timely manner. Usually the same day, perhaps the next the call would be returned and the problem resolved. As June wore on, the time frame started slipping. Now it was two and three days. By July it was a solid three days. Now in August it is five to seven days to respond to the call with a solution.

I mention the time frame because it plays into the whole scenario with both NetApp and CDW. You see, CDW has postured themselves as a value-added reseller…that if there is a problem, we can always go back to CDW and get their help in getting the problem resolved.

In the beginning of July we were still working on getting the DR testing solidified and came to a stunning realization: our DR site that was “the same” was not license for SnapDrive. SnapDrive is used to mount RAW LUNs for SQL Database, Exchange Databases, etc. Suddenly we were not sure if we could recover in a disaster. This is almost a year after we started this project and we had no idea if we could recover or not. I was on vacation, so my co-worker asked for pricing on SnapDrive. A week went by with no pricing from CDW, so he asked for it again and was asked by CDW for the serial numbers of the NetApp controllers. Another week when by and I was back from vacation but we still had no pricing. I asked for it again. I was asked for the serial numbers again.

A few days later I called her about it, having still received no pricing. She said she would get it but to hold on because her other customers did not have SnapDrive at their DR sites and she was going to find out what they did. That was on July 27th.

On August 27th, after multiple promises by CDW that they were working on it, I had finally had enough of this nonsense and had to get my CFO involved to help pressure CDW to answer the relatively simple question: do we need SnapDrive at our DR site to recover in a disaster or not?

Almost two months had passed and we did not have an answer to this simple question.

<A brief pause on this story while I add other things stat were going on to fill out the entire picture>

In the meanwhile, we had another issue with our NetApp Operations Manager where is would “lose” the DR site. It wasn’t that the DR site would go down or anything, it would just lose the connection and stop sending data to it, even though it was on line and running. The solution was to increase the timeout period on the NetApp Ops Manager.

It took five days and six people (two from CDW and four from NetApp) to get that answer.

About the time we got that figured out, we discovered that On Tap 8.1.1RC1 had a goofy little glitch that caused it to slowdown the SnapMirror to a crawl, making it impossible to get the data across the network to the DR site. This problem was discovered on August 23. Since then we have heard the all to common refrain from NetApp that “all their engineers are to busy helping someone else with bigger issues” three times on this one case. It is now just a few hours shy of seven days with no resolution to this issue, our DR site is nine days behind in mirroring and our company is at risk.

In short, since June NetApp has been less than prompt in responding to issues that they deem as “non-critical.”

I like to think I am a reasonable guy, but waiting a week for a response to something so critical, or two months for an answer to a simple question is unacceptable. If they are that busy I can only see two possible scenarios: 1) they don’t have enough technicians to fulfill their support obligations or 2) they have some MAJOR software problems.

<Return to the main issue>

At this point, I felt it was necessary to pull my CFO into the problem as our DR site is, to our knowledge, not working. I detailed to him a brief time line of what has occurred, what the response has been thus far, and where we are at, emphasizing that all we are looking for is an answer to the question: do we need SnapDrive or not at the DR site?

He forwarded it on to the folks at CDW to find out what was going on. What he got back was essentially a ten paragraph ad hominem attack on yours truly talking and how they have bent over backwards doing all these wonderful things for our company and how they were just try to save us money…all directly to my CFO and not copying me on any of it. They even tried to contact him via phone bypassing me, the IT manager. Stay Classy, San Diego…

And that, folks, was the breaking point for me. Instead of responding and working to fix the issue and answering the damn question, they felt it was a better idea to attack the person asking the question with personal attacks and half-truths.

That said, the gist of the email to the CFO centered on the idea that the reason we were in the predicament was that we (me and my co-worker) were not properly trained. Additionally they would get with NetApp and arrange a time to get the answer to the question (which is what we were after in the first place and took us three weeks to finally schedule.)

So the conference call began with about ten people on it including at least one NetApp engineer. And so we asked one simple question: do we need SnapDrive or not at the DR site?

That was it.  That was the only point of having all of these highly paid people on a conference call.  It is why we had to pull out executive level company office away from his busy day to ask this question.

The answer from this highly trained and knowledgeable NetApp engineer: “I don’t think so. I’m 99% sure you don’t, but I’ll have to test it.”

Wow. Stunning. Two months we pursued this point and what do we get? A “we’re not sure.”

We all sat there stunned. Surely they discussed this before the call, hadn’t they? Apparently they had not, and our CFO forced them to give us a time to have an answer (which incidentally NetApp missed.)

They finally got back to us the following afternoon and let us know that we did not need SnapDrive do mount a RAW LUN at our DR site.

Just shy of two months later we finally had our answer.

Meanwhile, my CDW reps have now taken to not responding to my emails requesting help with the mirroring issue (not that they were a whole lot of help with the SnapDrive issue) and I have begun purchasing from another vendor.

In short, my recent experience with CDW and NetApp has really sucked. It has become a running gag in our IT Department that the App in NetApp stands for apology, since mostly what they do lately is apologize for their lousy support.

I am really having buyer’s remorse.

(Update 10-15-2012)


An Odd HP Touchpad Charging Problem

While on vacation last month, we took the HP Touchpad with us.  My youngest daughter enjoyed it immensely as she used it to make iCarly-style videos as we made our way across the country. Sadly once we were in California, we were suddenly unable to charge it.

I tried various things to get it to charge, from changing the outlet it was plugged into to leaving it plugged in a whole day to wiggling the mini-usb connector to hopefully get it to make contact with what I imagined was a bad contact. But nothing seemed to work.

When we returned home I kind of wrote it off and let it sit for a few weeks. More than that, I assumed there was an issue internally with the charging port.  Given the current state of support for the HP touchpad that HP offers, that pretty much put my happy little Touchpad in the “junk” category.

But this morning, on a whim, I gave it another try. I noticed my daughter was using the HP Touchpad charger for her cell phone. I unplugged her phone and plugged it into the Touchpad. Nothing happened. I wiggled the connection and tried various things and still nothing happened and the Touchpad continued to get a charge. Again, out of options and assuming the device was dead, I plugged in the phone charge for my daughter’s phone and it started charging! Soon I was able to see the battery charging display. Shortly after that I got a message that I should use the power adapter for the HP Touchpad. I removed the cell phone charger and plugged in the HP charger and it began charging.

It appears that the battery was discharged to a point that the HP charger didn’t have the capacity to charge the battery. Once the battery had some initial charge the HP charger could once again charge the Touchpad properly. The rating on the HP charger is an output of 5.3 volts and 2 Amps, where the cell phone charger was 5 volts and 1 Amp.

Given this, what I suspect happened was that the Touchpad was “on” when the battery was fully discharged. When I tried to charge it with the HP charger, the parasitic drain from the Touchpad being still being “on” was such that the battery was unable to charge but also not enough power to power the Touchpad. But the cell phone charger was insufficient enough to run the Touchpad yet just enough to start slowly charging the battery.  Once the battery had enough charge to run the Touchpad itself the regular charger was able to both charge the battery and run the Touchpad.    In short, my theory here is that the parasitic drain was to great to allow the battery to begin charging, but the cell phone charger had too little power to supply the parasitic drain and thus charged the battery instead sufficiently enough to allow the regular charger to work once again.

That, at least, is my best guess as to what happened with the HP Touchpad. Anyway, it seems to be working again and I am now at 75% charge using the HP Touchpad charger.

09/08/12 Update:  I had a similar issue again, but my own tip did not help this time…well, not completely. This time, however, I noticed that neither charger would maintain positive contact with the connection inside the touchpad.  Through trial and error, I noticed that if it was pressed forwards towards the front of the touchpad it would charge the device.  So I used the weight of the touchpad to maintan the contact by supporting the charger connection a little bit off the table it was resting on.  It doesn’t need to be very far off the vertical surface to allow this do happen.  As you can see from the picture, about 1/2″ or so should do it.


Hope this helps.

Dynamic Distrubution Groups and their Exceptions

One of the most awesome functionalities of Exchange 2007 and 2010 is the introduction and implementation of Dynamic Distribution List.  These list allow a Distribution list to update automatically based on a certain set of criteria.  Very useful if you need to create a Distribution list for an area or by position or department or even everyone that has a mailbox.

Yes, gone are the days of having to manually update distribution groups when employees come and go.  Well, almost gone.  There will always be some tweaking that needs to be done.  Email distrubution list are just like that.  🙂

Which brings me to my point of this post:  While Dynamic Distrubution Groups are awesome, they can be kind of a blunt tool unless used properly. That’s when you have to use the switches in the  Exchange Management Shell to fine tune these DDGs.

In my toying with DDG, I created an “All” group.  It was easy to configure (New>>Dynamic Distribution Group and then follow the prompts) and worked great.  I set mine up to distribute mail to anyone that had a mailbox and then I restricted to to only allow Administrators and Management to send mail to this distribution group.

But soon I ran into a problem.  There were objects on the Exchange server that also had mailboxes and they were also getting mail.  Given the nature of single-instance storage that Exchange 2010 uses (as well as just good housekeeping practices) I need to find a way to eliminate these objects mailboxes from my much-beloved “All” list.

Enter the Exchange Management Shell!  This is done after the initial DDG group is created.

Here is the command syntax for excluding certain mailboxes from a DDG:

Set-DynamicDistributionGroup -Identity DDGName -RecipientFilter {((RecipientType -eq ‘UserMailbox’) -and -not(Name -like ‘Display Name 1‘) -and -not(Name -like ‘Display Name 2‘) -and -not(Name -like ‘Display Name 3‘) -and -not(Name -like ‘Display name etc.’))}

A number of notes here:

  • The Identity is the Alias name of the DDG you want to modify.  In my case, the name was simply “All” (no quotes)
  • The { and } brackets encompass the entire RecipientFilter variable and define it.
  • The Display Name 1, 2, 3, etc. are the display names of the mailboxes you want to exclude
  • I had a string of 15 mailboxes to exclude, so I don’t believe there is any limit.
  • You may be tempted to exclude one mailbox and then run it again to exclude another.  Don’t.  It doesn’t work.  It will re-set the list the the new exclusion only.  If you want to exclude multiple mailboxes you need to chain them together with the -and -not
  • Once this command is run on a DDG, you can no longer modify the recipient filter in the EMC, it must always be don via the shell afterwards.

Once the command is run, you can either check the results in the EMC under Filter>>Preview or, if you are feeling particularly bold, you can run these two commands:

$DDGName = Get-DynamicDistributionGroup “DDGAlias

And then

Get-Recipient -RecipientPreviewFilter $DDGName.RecipientFilter

The $DDgName can be anything you want, it doesn’t matter.  It’s just a variable name.  DDGAlias is the Alias of the Dynamic istrubution Group.

And that’s about all there is to it.



Spammers, Spammers and more Spammers

One thing that will just make me lose my ever-loving mind quicker than anything is spammers.  They just drive me nuts.  This obsession I have with blocking spam sometimes takes on a almost unhealthy level.

I am currently having one of these episodes, and the most infuriating thing about this one is that major ISPs are not only allowing this to occur, the are profiting from it.

The spammer in question is sending emails with the subject lines like:

  • Gartner’s Predictions on VoIP and the Cloud
  • ERP Expert’s Guide to Implementation Success
  • Facebook’s Impact on BI/ERP
  • MS Excel as an ERP/BI Tool – Tricks and Tips
  • 2012 VoIP Systems Buyer’s Guide

Often as not, the familiar name on these is Business Software Evaluations, and the email address is info@techevals.com.

But the absolute worst part is that they all seem to be coming from IP addresses that resolve to servers that host business solutions for Comcast and Verizon.  Specifically comcastbusiness.net and bos.east.verizon.net.  So these two communication giants are not only allowing this (a Google search indicated that both have been made aware of these shinanigans multiple times and have willingly chosen to do nothing about it except collect a check) but are helping them by  allowing them to change static IPs every so often and use throwaway domain names.


So I am left with no choice but to widen the net in which I use to block these spammers.  I was blocking by individual static IP address, but if Comcast and Verizon are going to continue to allow them to change STatic IP addresses to spew out this crap, I’m just going to increase my range.  I’ve got more blocks than they have IP addresses.  🙂

So now I am blocking the following IP addresses:


I’ve also made some conditional filters based on header info, sender domain,  and subject lines that hopefully will block these jerks once and for all.

Hopefully this will help someone else and if anyone knows addtional IP address rangers they use, please let me know and I will add them to my filters and to this list.


Shooting at Sikh Temple in Oak Creek Wisconsin

I just wanted to let the folks at the Sikh Temple in Oak Creek Wisconsin that, although they will probably never see this, my thoughts and prayer are with them.  Through this horrific tragedy, I was encouraged and uplifted by their example at this time of personal suffering and tragedy.   While it may be of little consequence to you now, I hope someday your folks know what a great example you are of people of faith.

And to those, like the shooter, that spread hate, I pray for you too.  The one part you missed in the Master’s teachings was to love one another.  I pray that you mediate on that and let go of the hate.

All you need is love….