expired vv snaps not auto deleting!

[T16]

Anyone come across this error before?

We have a scheduled task in SSMC which creates snaps every weekday for our live vv's. The command SSMC generates is:-

createsv -exp 168h -addtoset xx-vs-prd.Snapset @vvname@-@y@.@m@.@d@.@H@@M@ set:xx-vs-prd

Looks fine to me. However nothing gets autodeleted after the expiry date, the snaps just pile up and up and up. Its pathetic.

What I see is that the snaps which have not expired have a date of expiration, and when I run showvv -expired, there is a big list of vv snaps which have expired, however, it seems that despite running, the system task "remove_expired_vvs" (command: removevv -f -expired -matchbulkobjs) is not removing any of these at all.

I think we will never purchase another 3-par again, has anyone seen this behaviour before? We will have thousands of expired snaps. I see in SSMC the expired ones no longer have the expired bit enabled with a date beside it, but they show up as expired with the showvv -expired command, so really not sure what we do from here!
small example of expired:

xxxpar01 cli% showvv -expired
-Rsvd(MiB)- -(MiB)--
Id Name Prov Compr Dedup Type CopyOf BsId Rd -Detailed_State- Snp Usr VSize
423 xx-prd-3par-20.08.06.1900 snp NA NA vcopy xx-prd-3par 4 RW normal -- -- 1331200
433 xx-prd-3par-20.08.07.1900 snp NA NA vcopy xx-prd-3par 4 RW normal -- -- 1331200
443 xx-prd-3par-20.08.10.1900 snp NA NA vcopy xx-prd-3par 4 RW normal -- -- 1331200
426 xx-prd-ad-20.08.06.1900 snp NA NA vcopy xx-prd-ad 7 RW normal -- -- 614400
436 xx-prd-ad-20.08.07.1900 snp NA NA vcopy xx-prd-ad 7 RW normal -- -- 614400
446 xx-prd-ad-20.08.10.1900 snp NA NA vcopy xx-prd-ad 7 RW normal -- -- 614400
424 xx-prd-hpe-20.08.06.1900 snp NA NA vcopy xx-prd-hpe 5 RW normal -- -- 614400
434 xx-prd-hpe-20.08.07.1900 snp NA NA vcopy xx-prd-hpe 5 RW normal -- -- 614400
444 xx-prd-hpe-20.08.10.1900 snp NA NA vcopy xx-prd-hpe 5 RW normal -- -- 614400
427 xx-prd-sccm-20.08.06.1900 snp NA NA vcopy xx-prd-sccm 8 RW normal -- -- 2252800
437 xx-prd-sccm-20.08.07.1900 snp NA NA vcopy xx-prd-sccm 8 RW normal -- -- 2252800
447 xx-prd-sccm-20.08.10.1900 snp NA NA vcopy xx-prd-sccm 8 RW normal -- -- 2252800
425 xx-prd-vmware-20.08.06.1900 snp NA NA vcopy xx-prd-vmware 6 RW normal -- -- 1331200
435 xx-prd-vmware-20.08.07.1900 snp NA NA vcopy xx-prd-vmware 6 RW normal -- -- 1331200
445 xx-prd-vmware-20.08.10.1900 snp NA NA vcopy xx-prd-vmware 6 RW normal -- -- 1331200


If this wasnt hundreds of thousands of £££££ of kit, I would really like to throw it out of the window and into a skip.

Ahh now I see the scheduled task has been failing for months, nice touch, its green and does not generate an alert, just PERFECT.

System task remove_expired_vvs, Task 10816, has failed.
04-Jul-2020 17:27:01 o'clock BST New
No user
Recommended Action
A system tasks has failed. Contact the HPE Support Center.
General
System wp3par01
Serial number CZ3815XKJ3
Type A system task failed
ID
Message code 0x0500001
Origin System

What a load of excrement.

[T16]

So it appears that out of the box the standard system task of cleaning up expired snaps/vv's is pretty useless and does not work on snaps created from a vvset. Nice touch, super helpful.

The support guys said we should create a new schedule which runs every hour just before the standard system task, and use the -cascade option to remove all expired snaps from vv-sets.

Something like this:-

createsched “removevv -f -expired -cascade" "22 * * * *" removevv_expired_cascade

After checking the cli reference it says "
–cascade
Remove all the descendent volumes as long as none has an active VLUN. It will remove any VLUN
templates as long as there were no active VLUNs. It will remove the volumes from all the volume
sets. If the -expired option is specified, all expired volumes and their descendent volumes will
be removed regardless if they are expired or not."

Im a little dubious this might result in something unexpected, are we OK to go ahead with this custom schedule to remove expired snaps from our vv-sets?! Gulp! This is in prod now, and there cannot be any f-ups!

[Richard Siemers]

createsched “removevv -f -expired -cascade" "22 * * * *" removevv_expired_cascade

After checking the cli reference it says "
–cascade
Remove all the descendent volumes as long as none has an active VLUN. It will remove any VLUN
templates as long as there were no active VLUNs. It will remove the volumes from all the volume
sets. If the -expired option is specified, all expired volumes and their descendent volumes will
be removed regardless if they are expired or not."

Im a little dubious this might result in something unexpected, are we OK to go ahead with this custom schedule to remove expired snaps from our vv-sets?! Gulp! This is in prod now, and there cannot be any f-ups!

Gulp! I feel your pain. You may want to test a small manual batch first. 'removevv -f -expired -cascade SomeDevServer_F_Drive*' and use the showvv command first to test a vv_name wildcard. You know the first time that schedule runs it's going to delete thousands of expired snaps at once. For piece of mind and control, you may want to clean that up in several smaller batches before setting up the new task.

[T16]

phew it seems to have done the trick!

Skip now cancelled.. Hope this helps anyone in a similar position who uses vv-sets mapped to host-sets for ease of automation. The standard system task will not remove snaps in a vv-set, whereas the modified command shown above does the job if it runs ~5mins before the other standard system hourly expired vv task.

A cup of tea to celebrate :

[Richard Siemers]

Glad to hear all is well, and thank you for sharing!


My own personal opinion, and not necessarily the opinion of my employers, current or previous... I am not a FAN using vv sets in export templates because I prefer to have explcit LUN# assignment control. I ran into a situation where hosts needed both some provisioned LUNs for their own OS/swap/local use, and we also a member of a cluster that owned a VV:set. It worked fine, but some hosts had more dedicated LUNs than others... and it triggered our OCD when we saw:

Host1: LUN0-Boot LUN1-Scratch VVSET LUNS 2-10
Host2 LUN0-Boot VVSET LUNS 1-9

So Datastore1 was LUN2 on host1, but LUN1 on host2. Everything worked fine, it was just our own organic gray matter that took issue with it.

*disclaimer* this was been 8+ years on T800 arrays, very possible this has changed.