CUCM 8.5(1) Upgrade – Duplicated Missed Calls, Received Calls, Placed Calls Directories

I recently upgraded a clients CUCM 6.1(2) to 6.1(4) and then up to 8.5(1). The upgrade went mostly smoothly however after the upgrade users complained that there directories menu looked like this:

1 – Missed Calls
2 – Received Calls
3 – Placed Calls
4 – Missed Calls
5 – Received Calls
6 – Placed Calls
7 – Personal Directory
8 – Corporate Directory
Weird Huh! This client uses a lot of multiple line presentations on their phones which made me think that maybe it was a new feature and each DN had its own directory menu. A quick test in the lab proved this hunch was wrong.
It turns out that on 6.1(x) upgrades to 8.5(1) it is quite common for the ITL file to get out of whack and cause weird behaviour on the handsets. To rectify this issue I had to re-generate the ITL files.

To do this I followed the following steps on each of the TFTP servers:

 

  • Log into Cisco Unified OS Administration
  • Security > Certificate Management
  • Click Find
  • Click on CallManager.pem
  • Click Regenerate in the top left corner
  • Log into Cisco Unified Servicability
  • Tools > Control Centre – Network Services
  • Select your TFTP Server
  • Restart the Cisco Trust Verification Service
  • Tools > Control Centre – Feature Services
  • Select your TFTP Server
  • Restart the Cisco TFTP Service

Repeat this on each of your TFTP servers and then restart your phones.

SRST w/ VG224 – Using MGCP for fun and profit

We recently had a client with a branch office that they wished to expand with 3x VG224 Analogue gateways. Unfortunately they were only running a 2811 ISR router which is limited to 36 ephones in SRST. Whilst VG224 ports consume no DLU’s in CUCM from a licensing perspective each VG224 ports is a seperate SCCP registration when failed over into SRST. To avoid forcing them to upgrade their existing SRST router we looked at solutions that could provide survivability whilst still being easy to manage maintain.

Option One : Configure the VG224 as H.323
This is a perfectly valid solution and will probably fit most peoples requirements. We chose not to do this for the following reasons:
  • Their existing 2811 voice gateway was already configured up with MGCP and was running perfectly fine. They have consistently deployed this type of gateway across the organisation. Using H.323 for the VG224′s would require converting the voice gateway to H.323 also and break standards.
  • It doesn’t provide the ability to monitor analogue ports from a central configuration database (Cisco Unified Communications Manager)
Option Two : Configure the VG224 as an MGCP gateway with fallback to H.323
This was the option we decided to go for as it provided centralised management, ability to failover without using SRST SCCP registrations and maintained configuration autonomy.
Key configuration points are outlined below:
1. Configure the gateway for MGCP as per usual:
ccm-manager fallback-mgcp
ccm-manager mgcp
no ccm-manager fax protocol cisco
ccm-manager music-on-hold
ccm-manager config server x.x.x.x y.y.y.y
ccm-manager config
2. Configure your VG224 gateway for MGCP in CUCM. You can confirm this is working when you have the CCM successfully configuring the gateway:
%CMAPP-6-CONFIG_DONE: Configuration by CCM is doneVG224#  sh run | i mgcp
ccm-manager fallback-mgcp
ccm-manager mgcp
mgcp
mgcp call-agent y.y.y.y 2427 service-type mgcp version 0.1
mgcp dtmf-relay voip codec all mode out-of-band
mgcp rtp unreachable timeout 1000 action notify
mgcp modem passthrough voip mode nse
mgcp package-capability rtp-package
mgcp package-capability sst-package
no mgcp package-capability res-package
no mgcp package-capability fxr-package
no mgcp timer receive-rtcp
mgcp sdp simple
mgcp fax t38 inhibit
mgcp rtp payload-type g726r16 static
mgcp profile default
You can can also confirm it is working with the following :
VG224#sh ccm-manager
MGCP Domain Name: VG224
Priority        Status                   Host
============================================================
Primary         Registered              y.y.y.y
First Backup    Backup Ready             x.x.x.x
Second Backup   NoneCurrent active Call Manager:    y.y.y.y
Backhaul/Redundant link port:   2428
Failover Interval:              30 seconds
Keepalive Interval:             15 seconds
Last keepalive sent:            16:07:55 WAST Jul 11 2011 (elapsed time: 00:00:12)
Last MGCP traffic time:         16:07:55 WAST Jul 11 2011 (elapsed time: 00:00:12)
Last failover time:             14:54:19 WAST Jul 11 2011 from (10.24.0.241)
Last switchback time:           14:54:49 WAST Jul 11 2011 from (10.24.0.240)
Switchback mode:                Graceful
MGCP Fallback mode:             Enabled/OFF
Last MGCP Fallback start time:  14:54:52 WAST Jul 11 2011
Last MGCP Fallback end time:    14:55:10 WAST Jul 11 2011
MGCP Download Tones:            Disabled
TFTP retry count to shut Ports: 2Configuration Auto-Download Information
=======================================
Current version-id: 1306736658-af125081-aa59-4451-b342-2c7091b18cc9
Last config-downloaded:00:00:00
Current state: Waiting for commands
Configuration Download statistics:
Download Attempted             : 31
Download Successful          : 31
Download Failed              : 0
Configuration Attempted        : 3
Configuration Successful     : 3
Configuration Failed(Parsing): 1
Configuration Failed(config) : 0
Last config download command: New Registration
FAX mode: disable
Configuration Error History:
3. Configure the VG224 for Fallback Mode as per usual:
application
global
service alternate Default
!
ccm-manager fallback-mgcp
!
4. Configure your VG224 ports in CUCM and on the VG224:
!
voice-port 2/0
cptone AU
timeouts ringing infinity
timing digit 50
timing inter-digit 50
caller-id enable
!
dial-peer voice 500 pots
service mgcpapp
destination-pattern 12345
port 2/0
!
5. Configure your dial-peers as required pointing them at the SRST gateway:
dial-peer voice 900 voip
destination-pattern .T
session target ipv4:1.1.1.1
6. Now on your SRST gateway you must configure appropriate H.323 dial-peers for routing the extension ranges to the VG224′s for when it is in mgcp-fallback:
dial-peer voice 123 voip
description Send to VG224
destination-pattern 2219X
session target ipv4:2.2.2.2
This solution will provide MGCP registration back to your CUCM for both your SRST Voice Gateway and VG224 when in service but allow for fallback to H.323 in a WAN failure scenarios. All IP handsets can still register SRST via SCCP to your gateway in WAN failure.
Caveats & Points of Interest

  • Supplementary Services are only available for the VG224 when they are registered via SCCP. Things like Audible Message Waiting Indicators, Call Transfer, Call Waiting, Distincitve Ring, Feature Access Codes and Speed Dial will not work when the ports are MGCP registered.
  • Because we are using this to provide fallback as well any changes to DN on the CUCM would still require updating of the destination-pattern under the pots dial-peer for that port.

CUCM – Partition and Calling Search Spaces

Overview

One of the concepts that many junior voice engineers struggle with is the class of control features of Partitions and Calling Search Spaces. I remember feeling like I was banging my head against a wall repeatedly until I hit that Ah-hah moment and it all made sense.

Now there are many great explanations out there on the inter webs that are all technically correct and great. Some use the Lock + Key analogy like the CIPT1 Study guide and some use a White Pages/Directory Analogy. In my opinion these can leave people without a whole lot of voice experience more confused than when they started.

Purpose – What are Partitions and Calling Search Spaces used for anyway?

In my opinion Partitions and Calling Search Spaces are primarily used as a way to find “things” in the phone system. They are used so endpoints (Phones, Gateways, Trunks) can dial resources (Directory Numbers, On-net Route Patterns, Off-net Route Patterns, Translation Patterns). From an end-users perspective these resources are generally represented by a number. Because you can use these Dial Plan constructs to limit what resources an endpoint can find, it allows you the ability to implement Class of Service.

Sure, you could just put everything in the none partition and be done with it all together but Tom Hollingsworth has already told us why this is a bad idea over at his blog networkingnerd.net

Yeah, Ok. But you still haven’t told me what Partitions and Calling Search Spaces do!

Hold your horses. This is the part you’ve all been waiting for….< Insert Drum Roll > Here it is:

  • Partitions contain “things”
  • Calling Search Spaces find “things”

That’s it. I really believe this is the simplest way to think of how these two constructs work and behave.

Partitions

So we know that partitions can contain “things” but what exactly can they contain? A partition will only contain one or more of the following things:

  • Directory Numbers – These Directory Numbers represent other numbers in the Phone System that can be associated to Devices.
  • Route Patterns – Route Patterns represent off-net and on-net destinations and control how a call is routed to a destination. Route Patterns use Route Lists and Route Groups to send calls over a trunk or a gateway.
  • Translation Patterns -  I consider Translations Patterns an intermediate step to either matching a Directory Number or a Route Pattern. They are used to Transform the Calling or Called Party Information before using a Calling Search Space to find a suitable match. It is also worth noting that a Translation Patten can also be set to Block instead of route which can be useful when using the Line/Device Approach for implementing Class of Service
  • Transformation Patterns – Transformation Patterns are similar to Translation Patterns except they are not part of the CUCM’s routing construct. They are used for modifying Calling or Called Party Information for purposes of presentation but do not affect call routing. When you are using Transformation Patterns you should use dedicated Partitions and Calling Search Spaces for this function.

Calling Search Spaces

So if partitions contain “things” and Calling Search Spaces find “things”, what do they look in too find them? Well they look in partitions of course.

The following constructs use Calling Search Spaces to find things:

  • Devices – Phones use a CSS to find available patterns. This is generally referred to as the Device Calling Search Space
  • Directory Numbers – Directory Numbers also use a CSS to find available patterns. This is referred to as the Line Calling Search Space.
  • Trunks
  • Gateways
  •  Translation Patterns

This list was by no means exhaustive and there is a vast number of features that require Calling Search Spaces to be defined to control access to DN’s/Patterns for that feature. Things like Time of Day Routing, Presence, Call Forward Settings etc

When a Calling Search Space is looking through a series of ordered partitions it doesn’t work like an ACL in Cisco IOS that searches sequentially through the available partitions top-down until it finds a suitable match. No, It processes all listed partitions and matches on the best or longest match. The only time in which the order of the partitions matter is when you have equally specific patterns, in which the pattern in the partition closer to the top of the CSS list will win as a tie-breaker.

OK – So I think I get it, but I’ve noticed my Phone and my Directory Number both have a CSS. What’s up with that?

Well CUCM is flexible in that you could choose to set a CSS on just one of these elements and leave the other blank.  When you have both a Device and a Line CSS the CUCM combines the two into a single CSS with Line CSS above the Device CSS.

There is a good reason to use both. In order to effectively use features like extension mobility but enforce local dialling habits at remote offices it is recommended to set the Device CSS as too allow full access to the Dial Plan and then use the Line CSS to restrict unauthorised patterns from being dialled. The Partitions in your Line CSS should contain Translation Patterns that are set to block the restricted patterns and also have the “Urgent Priority” flag checked (It is on by default).

If you would like more information about the Line/Device approach please check the SRND.

ASA – Test AAA-SERVER

On an ASA you can test AAA Authentication from the command-line with the following syntax:

test aaa-server [authentication|authorization] <aaa_server_group> [host <name>|<host_ip>] username <user> password <pass>

For example:

test aaa-server authentication AAAGroup username adminuser password ThisIsAWeakPassword


Configure Terminal Revert – We Aint Reloading in 10 Anymore!

As of IOS 12.4(23)T you no longer need to rely on a “reload in” command to save you from potentially locking yourself out of a remotely managed router.

You can enter configure mode with “config terminal revert time x” and IOS will save the running configuration to a backup file on flash to revert to after x minutes.

When your changes are successful and you want to cancel the revert you simply exit configuration mode and feed the router “config confirm” to commit your changes persistently. Though obviously you will still need to write your running configuration to NVRAM as per usual!

What is the advantage of this? Well for one you don’t have to wait  for the router to reload to get access again and secondly any other services this router is providing eg Voice gateway, transcoding etc will be unaffected.

 

Lets take a look at how this behaves on the CLI:

 

Enable Configuration archiving

 

Router(config)#archive
Router(config-archive)#path flash:backups
Router(config-archive)#maximum 14
Router(config-archive)#exit
Router(config)#exit

 

Enter Configure mode with a reversion timer

Router#conf terminal revert timer 2

Rollback Confirmed Change: Backing up current running config to flash:backups-0

Enter configuration commands, one per line.  End with CNTL/Z.
Router(config)#
*Apr 11 02:38:30.571: %ARCHIVE_DIFF-5-ROLLBK_CNFMD_CHG_BACKUP: Backing up current running config to flash:backups-0

*Apr 11 02:38:30.5711: %ARCHIVE_DIFF-5-ROLLBK_CNFMD_CHG_START_ABSTIMER: User: console: Scheduled to rollback to config flash:backups-0 in 2 minutes

 

Make a configuration change

Router(config)#host
Router(config)#hostname RevertMe
RevertMe(config)#

 

Take note of reversion warnings

*Apr 11 02:39:30.571: %ARCHIVE_DIFF-5-ROLLBK_CNFMD_CHG_WARNING_ABSTIMER: System will rollback to config flash:backups-0 in one minute. Enter “configure confirm” if you wish to keep what you’ve configured

 

Reversion takes place

Rollback Confirmed Change: rolling to:flash:backups-0

Total number of passes: 1
Rollback Done

*Apr 11 02:40:30.571: %ARCHIVE_DIFF-5-ROLLBK_CNFMD_CHG_ROLLBACK_START: Start rolling to: flash:backups-0
*Apr 11 02:40:30.575: Rollback:Acquired Configuration lock.
*Apr 11 02:40:33.091: %PARSER-6-EXPOSEDLOCKRELEASED: Exclusive configuration lock released from terminal ’0′ -Process= “Policy Manager”, ipl= 0, pid= 21

Router(config)#

 

 

Alternatively Commit your changes

Router(config)#hostname RevertMe
RevertMe(config)#exit
*Apr 11 02:46:08.615: %SYS-5-CONFIG_I: Configured from console by consoleonf
RevertMe#configure confirm
*Apr 11 02:46:10.899: %ARCHIVE_DIFF-5-ROLLBK_CNFMD_CHG_CONFIRM: User: console: Confirm the configuration change
RevertMe#copy running-config startup-config
Building configuration…
[OK]
RevertMe#

Catalyst 3750 – Are they really that bad?

Over the past few years I have seen many comments in the blogosphere questioning the reliability of the Catalyst 3750 platform. I believe Ethan Banks (@ecbanks) has posted on this in the past and more recently Aaron Conaway (@aconaway). I thought I’d post my experiences over the past 4-5 years managing ~40x 3750 switch stacks composed of 2-5 switches without a single stack failure in production operation (we did have one stack member die due to lightning strike and the rest of the stack continued to function).

I will qualify that all of these stacks are in a relatively static physical installations but many of them have been subjected to temperature ranges of 22C to 30C (72f – 86f for the American folk). They aren’t configured with any advanced features, only routing/switching/access-lists and weren’t running bleeding edge IOS (mostly c3750-i5-mz.122-20.SE). Most of these stacks were at the edge delivering Ethernet services in a residential environment and not in the core or data center.

I will also add that I think both Ethan and Aaron are awesome engineers that I respect greatly and I’m and avid reader of their blogs, my experience with the 3750 just hasn’t been the same as they have described. Perhaps some of my anally retentive practices in deploying these switches has minimised the occurrence of failure.

PUT THE STACK CABLES IN PROPERLY

The stack cable is heavy with quite a relatively shallow connector; you should install this in the same way you would change a tire. I seat the connector and hand screw the cable in until it is firm and then use a flat head screwdriver a few turns on each side at a time until the cable is fully screwed in.

When provisioning a new stack I always screw all the stack cables in first and then power on the switches.

SYNCHRONISE YOUR IOS

Install the same IOS on a new stack member as the rest of the stack before installing using the archive command.

Switch#archive download-sw tftp://x.x.x.x/c3750-i5-mz.122-20.SE.tar

PROVISION NEW SWITCHES BEFORE HAND

Provision new switches on the stack before installing. I also configure the new switch with the appropriate switch number.

Switch(conf)#switch x renumber y

RIG THE ELECTION

Use the switch x priority command to rig the election on all switches. Highest priority wins the stack election.

I usually want switch 1 to be the master so give it a priority of 15 descending that number with each subsequent switch

Switch(conf)#switch 1 priority 15
SYNCHRONISE CONFIG

Manually drop the config of the switch onto new stack members before adding them too the stack. I know you don’t technically have to do this I just like removing all chances of error.

FINAL VERDICT

I think 3750′s are a great switch which in my experience have been rock solid performers. I would recommend them on the edge of anyone’s network, the ability to grow capacity as needed and distributed fault tolerance are big pluses. Would I use them in a data center? Probably not. I would also suggest that if you required a stack size of 4-5 switches off the bat then a 4500 would probably be a better option for cost vs features/performance.

CUCM 7.X – Rewrite Outbound CLI Presentation with Calling Party Transformation Masks

Problem :

You want your MGCP controlled E1/T1 to present a single outbound CLI for a group of DN’s that is different from their individual DID’s. You also want all other DN’s to present as normal.

  • You could hardset the Caller ID DN field on the E1 under “Call Routing Information – Outbound Calls” – This will overwrite all DN’s not just the ones you want
  • You could adjust the “External Phone Number Mask” on the DN. – This works but it will also change the phone display and can have AAR Implications
  • You could duplicate your dialplan in a seperate partition with new CSS’s and Route lists. Overwrite the CLI at the Route List – This is just yuck and doesn’t scale

Solution:
Transformation Patterns to the Rescue! A Transformation Pattern will allow you to adjust the Calling Party number when it hits the gateway conditionally based on the original internal calling party number.

  • Create a Partition called CLIRewrite
  • Create a CSS called CLIRewrite that looks at Partition CLIRewrite
  • Configure your MGCP Gateway:
  1. Device > Gateway
  2. Select your Gateway
  3. Select your E1/T1 Interface
  4. Under “Call Routing Information – Outbound Calls Uncheck “Use Device Pool Calling Party Transformation CSS”
  5. Choose “CLIRewrite” in Calling Party Transformation CSS Combo Box
  • Create your Transformation Mask:
  1. Call Routing > Transformation Pattern > Calling Party Transformation Pattern
  2. Click Add New
  3. Under Pattern enter the internal DN you wish to rewrite (Pattern matches like 28XX also work for matching multiple DN’s)
  4. Under Calling Party Transformation Mask enter the full number you wish to present out your E1
  • Reset your MGCP gateway under Devices>Gateways
  • I also had to do a “no mgcp” and then “mgcp” on the gateway itself
  • Make a test call to your mobile phone to see the re-written number