After upgrading to Spectrum 9.4.4 we have noticed our scheduled AutoDiscoveries fail to model. The modeling process is always locked out due to another process that is running.
With the release of Spectrum 9.4.4 we provided some additional functionality that allows us to place all background discoveries in the "Pending Rediscovery" queue. However, AutoDiscoveries are not consider background discoveries, and therefore they are never added to this queue. The TopologyWrkSpc model keeps track of the AutoDiscovery modeling requests, and will process them once the "Pending Discovery" queue is empty.
If your network is always changing, or you have a flood of traps or reconfigurations, this can cause this queue to never empty, the TopologyWrkSpc will not be able to process any modeling requests because the "Pending Discovery" queue will keep the work space locked.
To determine how deep your "Pending Discovery" queue is backed up:
1) Enable the AutoDiscovery Debug on the VNM model:
a) Open a OneClick client
b) Locate the VNM model in OneClick
c) In the Component Detail pane - Information tab, expand the AutoDiscovery Control subview
d) Expand the Debug Options subview
e) Set Debug AutoDiscovery to On. This will create an ADiscDebug_<date>_<time> file in the $SPECROOT/SS directory, which will contain the discovery activities for the VNM.
2) Open a remote connection to the SpectroSERVER running the discoveries.
3) Open a command line, or “bash -login” shell for Windows.
4) Change directories to $SPECROOT/vnmsh.
5) Open a CLI connection by running ./connect
6) Find the model handle of the "AutoDiscovery" model associated with the "TopologyWrkSpc" Model Type:
a) Run: ./show models | grep -i Topo.
b) This will return 4 columns of data, where:
“Model Handle” “Model Name” “Model Type Handle” “Model Type Name”
Which should look something that like:
0x50002c AutoDiscovery 0x25e0001 TopologyWrkSpc
0x500005 TopOrg 0x102cf Top_Org
c) Make note of the model handle of the AutoDiscovery model.
7) Run the following update action to print out the contents of the “Pending Discovery” queue: ./update action=0x10601 mh=<Model Handle of the AutoDiscovery model>
For example: ./update action=0x10601 mh=0x50002c
For Spectrum 9.4.4 the output of the Pending Discovery queue will be found in the $SPECROOT/SS/VNM.OUT file.
For Spectrum 10.x the output of the Pending Discovery queue will be found in the $SPECROOT/SS/ADiscDebug_<date>_<time> log.
If you find the “Pending Discovery” queue is backed up, you may want to clear the queue by running ./update action=0x10602 mh=<Model handle of the AutoDiscovery model>
Run update action 0x10601 again to check the contents of the queue.
If the queue is empty, you will need to abort the currently running discovery by:
1) Locating the VNM model in OneClick.
2) Select the Information Tab in the Component Detail pane
3) Expand the “AutoDiscovery Control” subview and select the “Abort Discovery” button.
Check the contents of the “Pending Discovery” queue once again to ensure it is still empty.
If you find the "Pending Discovery" queue is filling up faster than you can clear it, and the AutoDiscovery modeling session continues to be locked out, you will want to apply Spectrum_09.04.04.D140 (Spectrum 9.4.4), Spectrum_10.02.00.PTF_10.2.021(Spectrum 10.2), or Spectrum_10.02.01.PTF_10.2.111 (Spectrum 10.2.1) to your SpectroSERVER.
This patch provides an option to disable the "Pending Discovery" queue feature, and will allow the AutoDiscovery modeling to take place once the currently running background discovery completes.
To install the patch:
- Shutdown the SpectroSERVER and processd.
- Place a copy of the Spectrum_09.04.04.D140, Spectrum_10.02.00.PTF_10.2.021, or Spectrum_10.02.01.PTF_10.2.111 media in a directory of your choice. Make sure the directory and path do not haven any directories that contain a space in the name.
- Make sure you are using an account that has root privileges (Solaris/Linux) or an Administrator account (Windows).
- For Solaris/Linux: chmod 755 "Spectrum_09.04.04.D140.<OS>.txe file, Spectrum_10.02.00.PTF_10.2.021.<OS>/txe file, or Spectrum_10.02.01.PTF_10.2.111.<OS>/txe file to give the file executable privileges.
- For Solaris/Linux run: ./Spectrum_09.04.04.D140.<OS>.txe, ./Spectrum_10.02.00.PTF_10.2.021.<OS>.txe, or ./Spectrum_10.02.01.PTF_10.2.111.<OS>.txe
- For Windows simply double click the "Spectrum_09.04.04.D140.Windows.exe", "Spectrum_10.02.00.PTF_10.2.021.Windows.exe file, or "Spectrum_10.02.01.PTF_10.2.111.exe" from. Or from a 'bash -login' shell run ./Spectrum_09.04.04.D140.Windows.exe, ./Spectrum_10.02.00.PTF_10.2.021.Windows.exe, or ./Spectrum_10.02.01.PTF_10.2.111.Windows.exe.
Post Installation Steps:
After installing the patch you will need to add the following line to the $SPECROOT/SS/.vnmrc file to disable the "Pending Discovery" queue:
This change will be read in and take affect once the SpectroSERVER is restarted.
Sustaining Engineering is investigating a more permanent solution, that will provide some prioritization around these requests, that way the queuing will not have to be disabled. The prioritization will be delivered in a future release of Spectrum.