We recently began using what we call an SMS Push application.  SMS Push allows us to send Text Messages to our subscribers in bulk.  Typically these messages are some Marketing promotion or a Customer Service announcement.   The message text and a list of MDNs is provided to the application and it sends them serially, one at a time.  We wanted to speed that up.

The SMS Push application was provided by Quantum System Integrators in Costa Mesa, CA.  We’ve worked with them on several projects before and they’ve always high quality software and services with excellent customer support.   The application runs on Sun Solaris and is started from the shell command line.  It sends a predetermined messages to each MDN in a list provided as a command-line argument.  The application spawns a single instance of itself and sends the messages .  Through-put has been about 1 message per second, using an SMPP link to our Primal Technologies SMS-C. (more…)

Share This

All voice network operators are continually looking for ways to cut the cost of delivering outgoing long distance calls. The best way to do this is to use 2 or more Inter-Exchange Carriers (IEC, in FCC-speak) or what you and I might call “long distance service providers,” and cherry pick the cheaper provider for each dialed destination. This approach is called “Least Cost Routing.” Seems simple, right? Well, it’s not, really.

First there is the tyranny of numbers. A Least Cost Routing application typically will use the Local Exchange Routing Guide (LERG) to determine all possible dialed destinations. The LERG defines roughly 450,000 destinations. For each destination you might have a cost proposal from several IEC. We tried evaluating 8 IEC.

(more…)

Share This

Starting late last year our long-distance network has been overwhelmed with calls to various for-free services. These services often appear to be the proverbial free lunch, with no apparent business model because the service is 100% free to the end user. Typical offerings include free conference calls or voice chat rooms. These services can completely destroy a flat-rate all-you-can-eat telephony service provider.

(more…)

Share This

A good part of the last couple days has been spent looking for missing call waiting tones. Call waiting tones are the soft background beeps that usually are heard when you are talking on the phone and someone else tries to call your phone. The sound you’ll typically hear is a double beep of around 440 Hertz for 300 milliseconds. Well, we lost ours.

We’re not entirely certain when we lost it. Looking back, it appears that the first subscriber report may have been received in mid-September. Only recently have the volume of complaints risen to a level that caused anyone to intervene. (There’s probably a process improvement or two lurking within that statement.)
Because call waiting tones are provided by a component of the Nortel switch, we immediately opened a dialog with their Technical Assistance Service (TAS.) Starting yesterday morning several engineers at Nortel TAS with on a conference bridge with several of our Engineers trying to isolate the problem. (more…)

Share This

During last night’s maintenance window our Packet-Voice Gateways (PVG) were upgraded as part of our preparation to upgrade the entire cellular switch to the latest version of software, MTX14. Preparations have been underway for several weeks, leading up to what Nortel calls the One Night Process (ONP.) The preparations include component-by-component changes to hardware, software and firmware. Mostly this has been pretty uneventful, save for the issues caused by us being one of the Verification offices (VO) for Nortel’s then-new Packet MSC. Nortel has dealt with issues as they arose, and the upgrade has only slipped a day or two.

PVG preparation work had been completed according to schedule during the maintenance window by our Nortel installer and a member of the Network Operations staff. The summary of the night’s activities was already sent, network traffic was starting to pick-up, the day shift was sipping the day’s first latté as the night-shift left to go home. (more…)

Share This

Putty is an ssh client for Microsoft Windows which can automate secure logins to remote computers. This tutorial allows you to automate these logins using your public/private key pair.

Putty can be downloaded from here: http://www.chiark.greenend.org.uk/~sgtatham/putty/. Download and install the entire putty distribution.

During this procedure the following high-level steps are taken:

• First, create a public-private key pair using puttygen.
• Second, create a session for logging into a particular host. This session will include the IP address, and can include the username and location of your private key
• Lastly, add your public key to the host you’ll be logging into.

Each of these steps is detailed below. (more…)

Share This

This became a surprisingly full day. An earthquake near Taiwan around 1 PM local time disrupted communications lines to our out-sourced call center in Manila, the Philippines. This was certainly a surprise to our Customer Care group as they were under the, apparently mistaken, impression that diverse redundant voice routes were provided by the out-source company.

Regardless, it caused the Engineering department some serious headache. As the assumption had been that redundancy was built-in, a business recovery plan had not been created. A scramble was required to cobble together a work-around to reduce the impact on our customers, whose calls in the meantime were being dumped on the floor.

(more…)

Share This

We’ve started a forum on Teletips Network called Cable Conundrums for sharing pictures of telecoms racks. Our first post doesn’t actually look too bad; though the camera was uncharacteristically kind. In person the racks seem far messier. The pictures show these racks from the front. From behind the cabling looks far worse. Dangerously so. There have been several times when someone working in the rack accidentally bumped a cable and caused a hiccup to one of the applications running in that rack.

This has caused some discussion with my colleagues about what is the appropriate level of effort to expend dressing cables and in general attending to the cosmetic aspects of rack ownership. Given the relative rarity of such service affecting event, we generally agree that the time we saved by doing so little of this was probably worthwhile. To have spent more time neatening things as we installed them would’ve slowed us at a critical time and, in fact, has not seemed to have hurt us since. Maybe it is simply too soon to tell.

Something we have not agreed on is whether Ethernet switches should face the front or back of the rack. In the pictures they can be seen facing the front. But this has contributed to the cabling mess: cables exit the switch from the front of the rack and all of them must curl around to the server NICs at the back. If the Ethernet switch faced the rear of the rack then shorter CAT5 cables could’ve been used. This would’ve cluttered the rack interior less, and would’ve allowed for more direct cabling runs. We’ll try this with the next rack we install.

Share This

Often it happens that an interesting or important detail crosses my desk for which there is no file or folder, or some other ready place to note the knowledge. These snippets come as emails, faxes, documents, meeting notes, conference calls or from my own notebook. It is far too easy to set aside or lose pieces of paper, and the volume of email is so huge even Google struggles to find things. The black hole of email, especially Microsoft Outlook email, is worthy of its very own blog. The flow of paper is a continual reminder how important it is to save the data somewhere and worry about the taxonomy later.

For each piece of equipment I manage I’ve created a separate text document or spreadsheet. In it I record the interesting and essential details for managing that equipment, ways to reduce event impacts, tips and tricks for getting reports and generally making the equipment do more of what I want it to do. I’ve heard this documentation called “paper brains.”

This unstructured input can become large and unwieldy. Eventually some of it may begin to lend itself to a classification scheme. If so, I like to structure it so it can be more easily accessed when it is needed. Either way, having it is better than not having because my memory is so horribly unreliable.

When an equipment outage occurs I try to document the problem and note the details of the repair. You’ll see many of those appearing on this site. This is how I communicate with my colleagues and add to the “corporate memory” for that equipment. Same when troubleshooting an individual subscriber complaint. If the front line customer care folks cannot resolve an issue and it gets escalatated to me, I document the results and share them among my peer group. The document goes into the the online repository for others to access. It is my hope that if I were to get hit by a bus someone else could take up my job behind me without impacting the business too much.

Share This