BlackHat 2008 DC

Last week I was in DC for BlackHat 2008. I had a great time and seen some interesting talks on security, Hacking, Pen-testing, Networking, and much more. I have to say my favorite talk was from Felix "FX" Lindner titled "Developments in Cisco IOS Forensics. I would highly recommend you to visit his site and read the white paper he released (found here) on his companies site "Recurity Labs". Slides and presentations from BlackHat should be on there site soon for download. In other news I have heard of routers getting hijacked due to poor ACL's and SNMP traffic being sent over public networks in plain-text. It is important to keep your router locked down and protected. If your router got accessed and changed by an unauthorized person the first thing they might do is to lock you out. I have heard of reports where this is happened to a large multi-site company and they where blackmailed for money to get access back to there routers. With networks expanding over many miles, cities,  and countries it's important to keep you network safe. In the case of this reported company,  the cost of sending people out to password recover the routers was a lot more than the blackmailer's offer so the company paid them and then locked down the devices after they regained access. This could of been avoided and the skills needed to lock down a router is not CCIE level stuff! just using ACL's and a understanding of how the network is designed can prevent this kind of attack. Other issues with "unauthorized access" is even if you can regain access it's best to reload the IOS and review you config's. I say this since I have learned from Felix's presentation at BlackHat that some attackers load non-Cisco patches to the IOS. If an unauthorized IOS patch was made to your devices it is very difficult to identify the malicious code. With infected IOS code your routers you risk them becoming members of bot-nets, reset unexpectedly, or relay/hide unwanted traffic or tunnels. My recommendation is to only trust IOS code you get directly from Cisco. In the end of the day it does pay to keep you Cisco contracts up to date so when you need that clean IOS fix your CCO login can save the day.

References in this post:

http://www.blackhat.com/html/bh-dc-08/bh-dc-08-speakers.html#FX

www.recurity-labs.com

www.cisco.com 

Motorola DOCSIS 3.0 Ultra-Broadband Site Online

http://business.motorola.com/ultrabroadbandsolutions/home.html

Click on the pictures to enlarge them

It’s good to see Motorola releasing technical information to the web without the forced login. It looks like they still have plans for both I-CMTS & M-CMTS to support the MSO’s with there DOCSIS 3.0 rollout. It would be nice to see more whitepapers listed and maybe some CLI guides too. One of the issues I have had in the past with Motorola’s Broadband Products is that there is no real public documentation available (just marketing stuff). Where Cisco has way too much available and can cause an informational overload or confuse an engineer because features in one version of IOS might not work in another. Anyway check out Moto’s site and let me know what you think. DOCSIS 3.0 is coming and the big boys are getting ready to test the waters (some already are now) are you ready to jump in?

When the 10K meets an older Acterna DSAM….

+

Recently during an ISO upgrade I found (well a fellow engineer I work with found) that the older Acterna (now JDSU) DSAM meters failed BPI registration. It was interesting because all the modems on the CMTS worked fine and other meters did not have this issue. Well after a lot of trouble shooting from myself and other engineers it was found that the newer JDSU meters did not have the same issue. In the end it was that the older meters did not have a valid self signed Certificate and they had to be upgraded via JDSU TAC. Now the prior IOS was 12.3(17b)BC and we upgraded to 12.3(21a)BC and noticed the issue. So just as a warning to other Engineers you may want to test your older Acterna meters if you upgrade your IOS. If they do not pass BPI/BPI+ just call JDSU and have them add valid certs to the meters. This can be done via hooking the DSAM up to an Ethernet connect with a public IP for the JDSU TAC to access the meter. Basically your meter is fine and even if you have the BPI fail issue your meter can still work fine with all your normal RF testing and you could just use a modem to test DOCSIS with till you upgrade the meter.

February 7 2008 (9:00AM)

Cisco 10012uBR CMTS: Wiring the Beast…

Installing a new Cisco 10k can be a pain in it’s own but with the micro (MCX) RF cabling and the DIY cable kits your frustration can peak out. I wanted to make a post with links and info on wiring the Cisco 10K for those that may of not had the fun of this special experience.

What cable set did I order or do I need?

Here are some of the choices:

• Dual-shielded cables
• Quad-shielded cables

Now when you order your cable you will hopefully get the 10 color kit but some time you will end up with the 5 color kit which is harder to get use too.

This is a picture from Cisco’s site of the 10 color cable

The back of the 10K with line cards looks like this:

Here is a picture of how the cable connects to the 10K

Now for the recommended wiring of the cable kits:

10 color cable kits

5 color cable kits

All information in this post is from Cisco’s website and the full document can be found here: http://www.cisco.com/univercd/cc/td/doc/product/cable/ubr10k/ubr10012/frus/ubrmc520.htm

This post in PDF Here

(Note this article was written 3 months ago and since there has been rumor that some vendors have a sub $100[in bulk] DOCSIS 3.0 modem)

Is DOCSIS 3.0 Really Here?

Author: Brian Wilson

CISSP, CCNA, CCSE, CCAI, MCP, JNCIA, Network+, Security+

Slimjim100@slimjim100.com

Co-Author: Owen Parsons

CCNA, CCCS, A+, Network+, NCTI Senior Master Technician

docsisdude@gmail.com

So you’re an MSO with a DOCSIS network and want to know when you can start moving to DOCSIS 3.0 to gain all the new bells and whistles to include bandwidth, IPv6, & advanced security. DOCSIS 3.0 has the ability to give you over 100+Mbps to the customer, new security features, and support for IPv6 so you can save the internet’s IP resources. A rather important question remains, are there any vendors already selling DOCSIS 3.0 networks and devices? The answer is not the quick “yes” a vendor’s PowerPoint presentation may lead you to believe.

The most profound issue with DOCSIS 3.0 revolves around the modems themselves. There are no true DOCSIS 3.0 modems on the market at this time. All of the vendors have a 3.0(D)ownsteam only modems. This just gives you the downstream channel bonding, but does not have the upstream channel bonding IPv6, or the security features that makes DOCSIS 3.0 so enticing. The other issue that arises is “do the modems they’re selling today, have the ability to be upgraded to full DOCSIS 3.0”? Well in a short the answer is “no” they will not.   The reason for this lack of upgrade ability is the Broadcom chipset supporting the 256-bit AES encryption and the additional upstream tuners are not available today. This chipset is needed to implement the security functions required in the DOCSIS 3.0 specification.   At this point the chips are not 100% ready or at least not in mass production. So no matter how bad you want to get your network to DOCSIS 3.0 you are faced with the lack of true DOCSIS 3.0 modems. If you do decide on using Pre-DOCSIS 3.0 downstream only modems you need to make sure the modems you buy are not proprietary and bound to a specific brand of CMTS. If that is the case you would be in a predicament if you ever choose to switch CMTS vendors. Not only would this cause a headache for your customers, but it would create an unnecessary capital investment as you would have to forklift all the proprietary modems and replace them with newer 100% DOCSIS 3.0 modems.

With these new DOCSIS 3.0  modems slated to cost multiple hundreds of dollars each, this would be an unwelcome PO in your accounting department. So choose your modem carefully and make sure they can be upgraded or you may be regretting your decision to arrive early at the DOCSIS 3.0 party. Another large obstacle will be the price of the modem. Currently you can buy DOCSIS 2.0 modems in bulk for roughly $40.00USD. These newer DOCSIS 3.0 modems are rumored to initially cost anywhere from $100-$250 each. With a DOCSIS 3.0 modem costing that much it is prohibitively expensive to put one in every home. It’s very likely that these modems won’t make it to the residential customer anytime soon. The DOCSIS model is built around standards so nothing is going to stop a power user from going to their local WalMart or BestBuy and paying $250.00USD for a new DOCSIS 3.0 modem. On the other hand, not many users have that kind of money to spend on a modem and there is little justification for stores to even carry them. Why as a consumer would you pay hundreds of dollars more for a modem when the old modem works and is basically free in comparison.

So the question is, how do you transition from your current DOCSIS 1.x/DOCSIS 2.0 network to a full 3.0 network? I don’t see the move to DOCSIS 3.0 happening nearly as fast as the industry is buzzing and it will most likely start with business customer first. These business customers have a more attractive ROI and can justify the capital being spent on them.  Once the efficiency of manufacturing gets in place these modems will cost less, but the raw cost of multiple tuners and brand new chips will always make them more expensive than a DOCSIS 2.0 modem. The true cost breakthroughs will come when the raw materials come down in cost. Single chips that can replace multiple tuners, more chips being produced thus further lowering the initial cost to the manufacturer. This is years away but once it happens the cost per modem will drop, also an MSO’s ability to negotiate pricing and buy in bulk will further expedite this process.

I think once the modems are around $60.00 wholesale you will see the MSO’s stocking up on them and installing them in residential “power user” homes. The cable industry is in a period of growth with many new technologies providing never before seen opportunities. If they want to party it’s going to cost them billions to get to the next level, but when they do get there the customer experience will be amazing. Hopefully we will catch up with many of the Asian MSO’s and be able to make a 100+Mbps just a simple mouse click away.

The 3 Major Players

DOCSIS 3.0

Pros:

• IPv6

• Bandwidth (Downstream 100mbps+ & Upstream 50+mbps)

• 256 bit AES encryption

• SNMP v3

• Channel Bonding (Upstream & Downstream)

• IPDR

• Support IGMPv3

• Multicast QoS

• Improved ability to monitor DOCSIS devices

Cons:

• Availability

• Complexity

• Cost

• Number of vendors

• Having to replace parts of network

• RF bandwidth needed

• RF plant conditions to support higher QAMs

• 2-4 DS carriers have to be adjacent to each other

• Only one of the bonded channels has the MAC/scheduling info inside it

• VoIP Protection currently only on one downstream (not in the edge QAM)

References:

Many vendor presentations (Cisco, Motorola, Bigband, Arris)

Cablelabs listed public specs (www.cablelabs.com)

Google (www.google.com)

Link to this Article in PDF Here

Review: uCertify Network+ PrepKit

By Brian Wilson

CISSP, CCAI, CCNA, CCSE, JNCIA, Security+, Network+, MCP

Slimjim100@slimjim100.com

       This is a review on uCertify’s Network+ Prepkit available over at www.ucertify.com. The uCertify Prepkit is a quick download from their site. Once you install it on your computer, you have access to the demo version which gives you some practice questions and limited use of the Prepkit. Upon buying the full Prepkit, you will be sent a license key that will unlock all the questions and features. Now you can get started learning. Some of the major advantages with the Network+ Prepkit is the fact that it is more than just a simple study guide.

Inside the Prepkit you will find:

• Diagnostic test

• 7 large Practice tests

• Final exam, an Adaptive test

• Ability to create custom tests

• Interactive quiz with 154 questions

• Study notes

• Flash cards

• Articles

• Ability to track your Progress

I recently reviewed the Security+ PrepKit from uCertify and was asked to review the Network+ Guide also. I decided this time I would put it to the test by getting 2 free copies of the PrepKit and having some associates try their hands at the actual CompTIA Network+ Exam. I figured the only real way to test the quality of the PrepKit was to put it to use with 2 people that I knew wanted to study for the CompTIA exam. I recruited the 2 subjects and asked that they only use the uCertify PrepKit to study for there exams. Now I already felt impressed about uCertify’s guides (based on my recent review of the Security+ guide), but it was now time to see how it would fair in a live test.

The 2 subjects sat for the exam and both passed with decent scores. I do want to add that both of the test subjects had over 3 years of networking experience. With their experience and the uCertify Network+ PrepKit, they were able to pass the exam and attain the CompTIA Network+ certifications. I would also like to note that this was the first IT Certification that either of the two candidates had ever attempted. With the proof on the table, I have to endorse the uCertify Network+ PrepKit as it has proven itself to be the right study guide to pass the Network+ Exam.

This Review uCertify Network+ in PDF

BTW if you would like to buy any of the Prepkits from uCertify use this discount code "BRISON" for 10% off! Thanks for reading my review and look forward as I plan on reviewing uCertify's Network+ PrepKit very soon.

January 3 2008 (10:00PM)   

Update (4 January 2008)

Time to stop the attach of the MAC Clones

First… Happy New Year!!! I have been busy lately chatting with other DOCSIS engineers and assisting/brainstorming with them on newer ways to ID and prevent modem cloning (theft of service). I am sure all DOCSIS Engineers out there know about the different cable modem hacking sites and have there own little ways of minimize the impact of these criminal services. Now not to get on a soap box since I think Hacking in it’s real form is a good thing but using advanced knowledge to assist others to break the law and steel in not cool at all.  Anyway to the point While talking with one Engineer friend in particular I found his method to work around flaws in the CMTS’s he has to deal with a great idea. Now if your in a Cisco, Motorola, or an Arris CMTS world you are good to go because they actually enforce BPI+ but some of the other bastard CMTS’s (no longer made or supported models) might not implicitly apply DOCSIS 1.1 standards and this can lead to crackers abusing flaws in DOCSIS 1.0’s BPI. I will explain in a later post the neat trick my friend did to reduce cloning and theft but I would like to cover some of the basics to reduce theft of service.

DOCSIS 1.0

• Configure network to only allow TFTP from Authorized server to avoid rouge config files.
• Set modem filter to only allow HFC interface to pull TFTP from your servers.
• Set your SNMP access to only respond to your management network from source IP’s on the HFC interface of the modem (not the CPE address space).
• Monitor your devices via SNMP and make sure you track the config file names to the correct MAC addresses.
• Test all DOCSIS devices to make sure they respond to SNMP (if they fail to respond block the MAC via an ACL)

DOCSIS 1.1

• Do all of the above steps listed.
• If possible and all devices are DOCSIS 1.1 or above (no DOCSIS 1.0 modems) use the CMTS’s vendor command to “Enforce BPI+” and “TFTP Source Verify” (this will not let hacked firmware force the modem to DOCSIS 1.0 BPI).
• Make sure to upgrade all modem firmware to a ECN RFI 02030 load and maintain few version load to make rouge modem identification easier.
• Enable and setup “Cable Shared Secret” on your DOCSIS interfaces of the CMTS (change your shared secret often if not monthly).
• If using a Cisco CMTS enable “Dynamic Shared Secret” so that a dynamic secret key is established at the time the config file is requested.

There are many other methods of preventing hack, cracked, modified, & cloned modems from steeling service on your network. It is important to try to force BPI+ (DOCSIS 1.1) if possible on your Network. With BPI+ the modems certificates and keys are linked to it’s MAC address so a clone can not match the key value. When the keys fail you will see the cloned modems in a state of Reject(pk), Reject(kek) or Reject(tek) keep in mind that there is other reasons for a failed BPI+ modem to not come online and if you have a large number of modems in Reject(pk) first check to make sure the CA root-cert is installed (Cisco the cert should be 996 sized cert if the root-cert is 958 you have a corrupted or incorrect root-cert) and a working NTP server is configure as the encryption for BPI+ like any encryption is time sensitive. Other benefits to BPI+ is the fact that the data transmitted from the modems is encrypted so RF sniffing will be unable to recompile your customers data and assist to protect there privacy and reduce you liability for there privacy getting breached.

Last but not least you should have scripts available to detect cloned modems and ACL’s to block devices not running BPI+. This will eliminate most if not all theft of service on your network and also improve your paying customer experience.

Other non MSO direct ways to prevent theft of service is to push the vendors to remove all diagnostic ports and access from the modems internal motherboards and to sign the boot code of the mode to a chipset SN number so if the boot code was changed the modem would no longer work. This is a very good idea and with the cost of DOCSIS 2.0 modems so cheap it would be worth the modem costing a few dollars more is it prevented the chances of hacked modems on the plant.

I would say the very last step is to go down hard on cable theft of service and make sure to prosecute as this will make an example and be a deterrent for others not to try to modify there DOCSIS devices to steel service.

If you have any other idea on how to prevent and stop theft of service please feel free to e-mail also feel free to contact me for questions and comments you may have.

slimjim100(at)slimjim100.com

Update (4 January 2008) 

Cisco IOS Release 12.3(21)BC introduces a DOCSIS 1.1-compliant and above security enhancement that helps to eliminate denial-of-service (DOS) attacks that are caused by cloned cable modems. 

commands:

Router# cable privacy bpi-plus-enforce

More info linked below:

http://www.cisco.com/univercd/cc/td/doc/product/cable/cab_rout/cmtsfg/ufg_ccmd.htm

Ok so I decided to try a new banner. Since I am not good with Photoshop or any graphical software I will seek help from any of my readers that would like to give it a shot. I am looking for a more professional looking banner. I also want to hear from you! E-mail me to let me know what topics you want covered here and also send me a quick note on what CMTS's you are running and where your cable plant is. I have networked with many DOCSIS Engineers from all over the world and I hope to start posting there lessons learned in hopes this site might be able to assist even more users. So e-mail me @ slimjim100(at)slimjim100.com and tell me a little about yourself.

First I would like to say Happy New Year and I hope the best to everyone! I have been very busy here on my time off from work with all of the  holiday stuff but I wanted to add a few comments here. One thing I have noticed is that the headhunters like to recruit during the holidays as I got bothered a few times via my phone and e-mail. I had stated in a past post I was thinking about leaving the DOCSIS world for a different gig... Well so far nothing panned out with that so I might be around for a bit longer. You also may of noticed the "Need Help Call me" thing at the top of this page... Well lets just say I am trying it out to see if anyone will call. If you need help and can't afford to pay feel free to e-mail me and I will try to help. I also decided to revamp my Modem status guide (info in guide is from Cisco's documentation). I still have a new review coming for uCertfiy's Network+ Exam guide. After I finished my Review of the Security+ Guide they asked if I could review there Network+ guide and I agreed too on the terms they let me pick a few people to just use there Prep Guide to study for the test and write my review based on a true exam takers point of view. Well so far one the guys that I asked to try the network+ with uCertify's guides has done very well but I will save the rest for my up and coming review. Also to let everyone that read the blog now I did not take any favors or money from uCeritfy to review there guides I am an advocate to people gaining knowledge and bettering them selves and certification is one way of doing this and after looking at there price and quality I decided to review there products free.

Review: uCertify Security+ PrepKit

By Brian Wilson

CISSP, CCAI, CCNA, CCSE, JNCIA, Security+, Network+, MCP

Slimjim100@slimjim100.com

      I recently had a chance to try out uCertify’s Security+ PrepKit. I was asked to try it out and see what I thought of it. Seeing as I took the self study route for the Security+ Certification last year and passed it with a very high score I figured I could make a fair and honest assessment of this test preparation kit. In the past I have used the different vendor’s books and brain dumps and found some are well written While most are just not worth your time or money. In fact instead of helping with your study efforts they can often frustrate you with poor organization and usability. In this case not only is the preparation kit well written and easy to use I was also pleasantly surprised to see the simple layout of the uCertfiy’s Test Prep.

Features worth mentioning:

• Diagnostic tests
• Many practice tests
• Adaptive tests
• Ability to make custom test
• Flash Cards
• Notes & Objectives for review
• Progress tracker
• Online Prepkit updater
• Ability to bookmark test questions
• Very easy to navigate GUI

I feel compelled to inform you that this is not a brain dump. You get a full study guide and a nice progress chat to help you gauge where you are in your study process. I enjoyed the Objectives and Notes that fully examined the content and allowed you to fully understand the objectives of the Security+ exam. I also found the flash cards and ability to bookmark questions in the practice tests helpful. uCertify has been around since 1997 and there pass rate for exam takers using there content is around 97% (according to their marketing info). I have to believe that if you followed this Prep-Guide you would have a very good chance of passing the exam and truly understanding the content. In the end I was very satisfied and would recommend it to anyone wanting to take CompTIA’s Security+ exam.

CompTIA will most likely be seeing an increase in people taking the Network+ & Security+ certification because of DoD directive 8570.1. Which will soon require many personnel that work for the Department of Defense to have networking and security related certifications if they hold the role of Information Assurance Technician or Information Assurance Manager. With this new directive requiring certification I feel it further adds industry value to the CompTIA certs among other required certifications per the new directive. If you are a contractor or employee for the US government now you may want to start taking the certifications as it’s only a matter of time be for it might be a requirement for many other positions in the government. With that said I would highly recommend the uCertify Prepkits for your exam perpetration needs.

More information on DoD Directive 8570.1

This Review uCertify Security+ in PDF

Link to uCertify

BTW if you would like to buy any of the Prepkits from uCertify use this discount code "BRISON" for 10% off! Thanks for reading my review and look forward as I plan on reviewing uCertify's Network+ PrepKit very soon.

Questions, questions, questions.... I have seen a lot of hits to this BLOG with interesting search terms and I would like to extend out my knowledge to other DOCSIS Engineers out there that might have a question about DOCSIS, CMTS setup, or lessons learned. I chat with many other DOCSIS Engineers throughout the week and always see new and interesting bugs and issues out in the plant so feel free to fire your questions this way. I can't say that I will always have the answer but I can take a shoot at it.

Jobs... Yes I am looking at a few places to advance my career and at this time I have not made a 100% commitment ether way. But I can say I have had a lot of Job offers sent my way that did not fit what I was looking for or where I wanted to live. With that said I would like to extend the job offers I get to any other DOCSIS Engineers out there. If you are a DOCSIS Engineer or a Technical MSO Engineer and would like to be in the loop on current job openings with many MSO's and other companies serving the Cable industry let me know as I can add you resume to my resume page on this site and I can assure you the recruiters and companies will e-mail you. I get around 5-10 e-mails per week with job offers. I wish some of the job offers where in Georgia :p.

How much is a DOCSIS Engineer worth on the job Market??? From the offers I have seen this is a wide and open range depending on years experience and how much you like to travel.

DOCSIS Engineer 2-4 years Exp (30% travel) worth about: $50,000.00 - $70,000.00 (relo package)

DOCSIS Engineer 4-6 Years Exp (20-50% travel) worth about $75,000 - $95,000.00 (relo & sign-on bonus)

DOCSIS Engineer 6 - 10 Years Exp (10-50% travel) worth about $85,000 - $120,000.00 ( Full relo, sign-on bonus, other perks)

DOCSIS Software Engineer 2-8 Years Exp (0 - 20% travel) worth about $45,000.00 - $90,000.00 (relo)

DOCSIS Sales Engineer 3-10 Years Exp (50 - 90% travel) worth about $80,000.00 - $130,000.00 (relo, sign bonus, sales bonus)

The above rates are just from recent job offers and are expecting the engineers to fully understand DOCSIS 1.x -3.0, VoIP, Provisioning, IPv6, PacketCable, OCAP, & DSG. If you feel your under paid it might be because you are not marketing yourself or you are trying to stay in an area where there is a lower cost of living or less of a demand for DOCSIS engineers. I have notice there are about 4 hot cities where DOCSIS Engineers can get paid well and the rest of the country is about 10 - 30% below the numbers quoted above. Again this is just what I have pulled from job postings and offers I have received.

Happy Thanksgiving!!!

Finely I have gotten some of the answers to questions that have bothered me... Ok I was told by a contact at Motorola that the SB6120 is gong to get Cablelabs certification and will be a true DOCSIS 3.0 modem. I was also told it is possible that it will be a sub $100 CPE device so if this is a true DOCSIS 3.0 (up & down stream bonding, 256bit AES, & IP6) we have a real chance at seeing DOCSIS 3.0 soon. I was also informed that it might make it to the market mid 2008 (please do not quote me on the price or dates as this could just be rumors). The other burning question I have had to the vendors is what will the MSO's do for more bandwidth n the return spectrum (upstream QAM's and bandwidth). I have received many  interesting responses.  I have been told a few things like going to 64QAM on a 6.4Mhz channel width to get more bandwidth (DOCSIS 2.0 & 3.0 only). So that still leaves little spectrum left if you wanted to run 2 6.4 mhz upstreams. Now the idea of going from QPSK,16QAM, 32QAM to 64QAM  can help but it would have to be only DOCSIS 2.0 & 3.0. I guess this will force the MSO's to replace all DOCSIS 1.0 &  1.1 modems to get the full benefits of the DOCSIS 2.0 & 3.0 modems. In the end you will still end up supporting your older DOCSIS 1.X devices till you can get the resources to replace the out dated CPE. Other issues are any QPSK TDM VoIP stuff you have out there and the good old return from your DAC.  Anyway if you have any ideas or comments feel free to e-mail me at slimjim100(at)slimjim100.com and share your thoughts.

Looks like it's possible I might be leaving the MSO industry to a position at a Carrier Class Provider. Since I do not state where I work in any of my entries (but if you looked around you could figure it out) I feel I can say I really like the company I work for I just need to be able to advance. I might be leaving so I can grow my carrier and move up a little in pay. Don't get me wrong and think I am chasing the dollars I love the company I am with now but I feel I have hit a glass ceiling and I need to keep expanding my skills and career. So with the chance of me leaving DOCSIS I am not sure where this Blog will go and I might just continue to post to it but in a non-DOCSIS engineer capacity. I do have contacts in the DOCSIS world still and I have also been thinking of having some of my fellow engineers from other MSO's post here too. My goal of this site when I started it was to network with other DOCSIS Engineers and to vent on the stress of the job. I always hated not finding any documentation that was technical and only finding papers that where marketing fluff. With DOCSIS 3.0, DSG, TLS, Open Cable, and so much more coming on the market there is a real need for engineering notes on how real deployments end up.

Where's the Upstream?????

Another thought on my mind... I have been wonder lately "what's the deal with Upstream Spectrum"! What I mean is everyone is focusing on getting more downstreams and even more upstreams but where is the spectrum for more upstream bandwidth? This is likely to end up biting the industry in the butt if they do not start working on a way to get more out of the 5-42Mhz spectrum. With all the current devices using some of that spectrum it will be hard to keep allotting more and more of it when there is nothing left. Even if they find new spectrum or find a better way to use what they have now the older equipment that needs the return path will most likely not be compatible with newer ways to use the 5-42Mhz (or beyond 42Mhz). Time will tell what the solution will be for this little issue. And the problem will only grow as everything is moving to DOCSIS. All DOCSIS CPE need a way to get back to the Headend like DSG and the next generation of VoIP. devices.

Looks like BigBand has decided not to continue in the CMTS market. I received a news releases and a few e-mails from fellow engineers that BigBand has laid off a lot of there CMTS staff and plans to drop the Cuda CMTS. So I wonder what this means to MSO's that have the Bigband Cuda's deployed in there networks now. I expect with the contacts and agreements Bigband had with many of the MOS's that they will keep a small staff to support the CMTS but for how long. Looks like there DOCSIS 3.0 plans are now stopped and your choices for CMTS's is pretty much limited to Cisco, Motorola, & Arris.  Bigband is best known for there video technology and according to the press release that's where they plan to focus there energy. It kind of feels like a flash back to Terayon and how they dropped out of the CMTS and modem business to focus on there video side and then got bought out by Motorola. So I figure it's a matter of time before Bigband gets bought out and one company I could see buying them would be Cisco. I am thinking Cisco since Cisco's last video company purchase was Scientific Atlanta. Cisco bought SA to compete with Motorola on Set-top boxes. But since Motorola bought out Terayon and has there Video technology I think Cisco will look at buying Bigband to keep the competition up with Motorola. Now all this is just wild guesses I am making but time will tell if I am right.

Other thoughts... I have not heard to much lately about the joint venture with Motorola and Juniper Networks on DOCSIS 3.0 I am wonder if it died out. I first heard about it in 2005 and it was later discussed in early 2007 but there is just not a lot of news about it. I think with Junipers background in core routing and Motorola's experience in the RF and DOCSIS area it could be an awesome match and really put pressure on Cisco. But with the lack of press releases I figure it's ether died or is on a back burner some where. In other news Motorola should be releasing the TX-32 card for the BSR soon and it will be interesting to see how it works in a live plant with there software. I have to say I like there hardware designs but my experience with the BSR software is less than perfect. While I think Cisco's hardware design on the CMTS RF redundancy is.... well crap. Cisco has very stable software (IOS) most of the time. In a perfect world you would have the BSR design with Cisco IOS software running on it. That would be one stable and well made CMTS but we know that is not going to happen. With DOCSIS 3.0 getting ramped up it's time to figure out if your buying into the I-CMTS or M-CMTS idea or maybe your looking at a hybrid network. Ether way your network is going to get very complex and is going to need some killer bandwidth to support DOCSIS 3.0. Speaking of DOCSIS 3.0 are you ready for IPv6 or do you plan to keep to IPv4 and some how support all the devices coming your way. With everything going DOCSIS (Modems, eMTA's, & DSG) IPv4 will not be able to handle the address and security needs soon. Time will tell on when and how the MSO industry gets to IPv6 and DOCSIS 3.0.

Useful Cisco Show commands for uBR CMTS's. I have been playing with a Cisco uBR7246VXR on my desk and have found some commands I have not used before but look to be very useful while trouble shooting. This is some newer commands not listed in my document of Common Troubleshooting Commands for Cisco CMTS.

• show cable modem calls

docsis_toy#show cable modem calls

Cable Modem Call Status Flags:
H: Active high priority calls
R: Recent high priority calls
V: Active voice calls (including high priority)

MAC Address IP Address I/F Prim CMCallStatus LatestHiPriCall
Sid (min:sec)

docsis_toy#

• show cable calls

docsis_toy#show cable calls

Interface ActiveHiPriCalls ActiveAllCalls PostHiPriCallCMs RecentHiPriCMs
Cable3/0 0 0 0 0
Cable4/0 0 0 0 0
Cable5/0 0 0 0 0
Cable6/0 0 0 0 0

Total 0 0 0 0

• show cable tech-support

------------------ show cable modem Cable3/0 ------------------

MAC Address IP Address I/F MAC Prim RxPwr Timing Num BPI
State Sid (dBmv) Offset CPE Enb
0015.2fc6.d59e 172.1.1.3 C3/0/U0 online(pt) 5 0.00 1127 0 Y
0000.cac1.d295 172.1.1.2 C3/0/U0 online(pt) 6 *1.75 1222 0 Y
0018.c0dc.9fb2 172.1.1.4 C3/0/U0 online(pt) 7 *1.75 1131 0 Y
00e0.6f89.4cb4 172.1.1.5 C3/0/U0 online(pt) 8 0.00 1419 0 Y


------------------ show cable modem Cable3/0 connectivity ------------------

Prim 1st time Times %online Online time Offline time
Sid online Online min avg max min avg max
5 Oct 22 2007 3 99.89 10:56 7h57m 23h28m 00:06 00:30 00:43
6 Oct 22 2007 3 99.80 10:27 7h57m 23h28m 00:32 00:55 01:11
7 Oct 22 2007 3 99.86 10:40 7h57m 23h28m 00:06 00:39 00:57
8 Oct 22 2007 7 99.78 00:16 3h24m 23h28m 00:01 00:26 01:04


------------------ show interface Cable3/0 sid ------------------

Sid Prim MAC Address IP Address Type Age Admin Sched Sfid
State Type
5 0015.2fc6.d59e 172.1.1.3 stat 23h54m enable BE 11
6 0000.cac1.d295 172.1.1.2 stat 23h54m enable BE 13
7 0018.c0dc.9fb2 172.1.1.4 stat 23h54m enable BE 15
8 00e0.6f89.4cb4 172.1.1.5 stat 23h54m enable BE 17


------------------ show interface Cable3/0 sid counter ------------------

Sid Req-polls BW-reqs Grants Packets Frag Concatpkts
issued received issued received complete received
5 0 29 29 24 0 6
6 0 39 39 30 0 0
7 0 97 32 24 0 0
8 0 70 70 58 0 2


------------------ show interface Cable3/0 sid association ------------------

Sid Prim Online IP Address MAC Address Interface VRF Name
5 online(pt) 172.1.1.3 0015.2fc6.d59e Bu1
6 online(pt) 172.1.1.2 0000.cac1.d295 Bu1
7 online(pt) 172.1.1.4 0018.c0dc.9fb2 Bu1
8 online(pt) 172.1.1.5 00e0.6f89.4cb4 Bu1


------------------ show interface Cable3/0 modem 0 ------------------


SID Priv bits Type State IP address method MAC address
5 10 modem online(pt) 172.1.1.3 dhcp 0015.2fc6.d59e
6 10 modem online(pt) 172.1.1.2 dhcp 0000.cac1.d295
7 10 modem online(pt) 172.1.1.4 dhcp 0018.c0dc.9fb2
8 10 modem online(pt) 172.1.1.5 dhcp 00e0.6f89.4cb4

• show controllers cable

docsis_toy#show controllers cable 3/0
Interface Cable3/0
Hardware is MC16C
BCM3210 revision=0x56B1

Cable3/0 Downstream is up
Frequency 699.0000 MHz, Channel Width 6 MHz, 256-QAM, Symbol Rate 5.360537 Msps
FEC ITU-T J.83 Annex B, R/S Interleave I=32, J=4
Downstream channel ID: 0
Dynamic Services Stats:
DSA: 0 REQs 0 RSPs 0 ACKs
0 Successful DSAs 0 DSA Failures
DSC: 0 REQs 0 RSPs 0 ACKs
0 Successful DSCs 0 DSC Failures
DSD: 0 REQs 0 RSPs
0 Successful DSDs 0 DSD Failures
DCC: 0 REQs 0 RSPs 0 ACKs
0 Successful DCCs 0 DCC Failures
DCC end of transaction counts:
DCC unknown cause(0) offline(0) if down(0) no cm(0)
DCC no resource(0) no retries(0) reject(0) unknown state (0)
DCC rebuild err (0) T15 timeout(0) reinit MAC (0) dcc succeeds(0)
DCC wcm(0)
Cable3/0 Upstream 0 is up
Frequency 19.984 MHz, Channel Width 3.200 MHz, 16-QAM Symbol Rate 2.560 Msps
Spectrum Group is overridden
US phy MER(SNR)_estimate for good packets - 25.7240 dB
Nominal Input Power Level 0 dBmV, Tx Timing Offset 1419
Ranging Backoff automatic (Start 0, End 3)
Ranging Insertion Interval automatic (60 ms)
US throttling off
Tx Backoff Start 3, Tx Backoff End 5
Modulation Profile Group 2
Concatenation is enabled
Fragmentation is enabled
part_id=0x3137, rev_id=0x03, rev2_id=0xFF
nb_agc_thr=0x0000, nb_agc_nom=0x0000
Range Load Reg Size=0x2C
Request Load Reg Size=0x07
Minislot Size in number of Timebase Ticks is = 2
Minislot Size in Symbols = 32
Bandwidth Requests = 0xD1
Piggyback Requests = 0x12
Invalid BW Requests= 0x63
Minislots Requested= 0x144E
Minislots Granted = 0xE3
Minislot Size in Bytes = 16
Map Advance (Dynamic) : 2018 usecs
UCD Count = 42168

Many more fun uBR Show commands found here on Cisco's Site

 

Wow my article is number 1# or at least on the front page of Network World! Last week I was talking with Brad Reese and we decided it would be cool for me to write an article about some of the technology I deal with and maintain. After talking about the Cisco 10K and how much the list price is($980,000.00). Brad had the idea to write a small article about a 1 million dollar router. I liked the sound of it so I threw together a quick little paper talking about the Cisco 10012 CMTS and sent it over to him for editing. Well it hit Network Worlds site yesterday very early in the morning and by 8:00am it was #1 in Google news using the search phase "Cisco Router". I thought that was too cool to be number one in Google news for any key search phase but then this morning I got an e-mail from Brad Reese saying my article was on the front page of Network World. I guess this is my 5 minutes of internet fame... Anyway I was thinking of writing a small article on DOCSIS 3.0 kind of a little paper explaining where the mass deployment in the industry is at this time and the pros and cons of DOCSIS 3.0. I do not expect this next paper to get too much attention since it is very industry narrow but hopefully if finds it's way to some of the smaller MSO's that might be thinking about DOCSIS 3.0. I will post here once it gets posted online and if for some reason it is not a good fit for Network World I will publish it here.

Bugs, bugs, bugs.... Well it looks like there is a little Cisco bug (CSCsj26808) I have ran into recently and boy does it cause SNMP issues. While running the Cisco IOS 12.3(17b)BC4 code on a Cisco 10012Ubr router I have seen an issue with the "total active devices" on downstreams. Now this issue sound very simple and not too troublesome since it's just a modem count of total devices and is not really an outage condition but the real pain comes when you are trying to monitor the CMTS with SNMP tools and the device count is a negative number so it crashes or forces the MIB walker to skip the interface. In the end you get the wrong numbers in your tools and this causes all kinds of possible issues for other monitoring tools. The only workaround known is to reset the affected line card. Just doing a line card fail over will not fix this issue you have to power down the affected card and them re-power it to clear the condition. I have a quick and easy non-outage way to do this but it does take about 10 minutes per affected card. Here is the procedure I came up with below.

Note: Hot swapping or Power cycling a card may cause all the IOS configuration for that particular line card to be lost therefore the configuration should be saved before proceeding with the swap out.

The commands  here are  explaining the process  assuming that you are using HCCP bitmaps (will also work with HCCP global).

- Do a "show running-config" and copy/paste the configuration for the cable line card that needs to be replaced onto a text editor.

- Do a "show cable modem summary total" this will give you a baseline of the number of DOCSIS devices before you begin.

- Do a "show cable call" this will let you see if any active 911 calls are in progress (you should not perform any kind of maintenance when 911 calls are active).

- hccp (group) switch (member) (i.e. router# hccp 1 switch 50) This will force the active card to fail over to the protect card so you do not interrupt service. you will repeat this command till all groups and members on the card to be replaced are failed over to protect.

- Do a "cable power off <slot/port>" to power off the line card that needs to be replaced or reset.

- Remove the old line card from the uBR10K chassis (Only do this step if replacing a line card)

- Insert the new line card into the uBR10K chassis (Only do this step if replacing a line card)

- Do a "cable power on <slot/port>" to power on the newly inserted line card.

- Do "show running configuration" to make sure the config on the replaced/reset line card is present.

-  (This step is if any config is lost) Paste the saved IOS configuration for the line card from the text editor

- Do a "show hccp brief" to verify the groups and member to fail back to the replaced line card.

- hccp (group) switch (member) (i.e. router# hccp 1 switch 50) This will fail the from the protect card back to the normal working card. Repeat this command till all the protected groups & members are back to there normal working card.

- Do a "show cable call" to make sure you PacketCable calls or SIP traffic is back up and running (you may want to make test calls).

- Do a "show cable modem summary total" and compare it to the one you ran before the card swap. Now you might have less devices online now but as long as the difference is less than 5% you should be fine since some of the device might take longer to come back online.

- Make sure all cable downstream / upstream interfaces are up and operation. You might have to do a "no shutdown" on some interfaces.

- Do a "wr mem"  or "copy running configuration start-up configuration" to save the IOS configuration

This so far has corrected the bug I have found but the next test is to see how long this work-around keeps the bug away. The fix to this bug is in 12.3(21a)BC3 but keep in mind that with every upgrade you are risking your exposure to new kinds of bugs so do your due diligence to lab test any new IOS before a live deployment. Also from what I have heard this issue affects both the Cisco uBR10012 & uBR7246.

I am back form Chicago now and I have to say I enjoyed Chicagocon. I was very happy with the presentation I gave on Cain & Able and it got a lot of good comments. After I finished my talk on Cain I was approach by the CEH instructor and he wanted me to go over Cain in his class the next day. Some other good talks at Chicagocon where Chris Gates (MetaSpoilt), Lance Spitzer (The Honeynet Project), John Dvorak (The Next Decade in Desktop Computing), and many others. I also was taking my CISSP review while I was there and I have to say that was one of the hardest test I have ever done. In the end I passed my CISSP exam so it was worth all the stress and studying. Anyway now I am back to my normal job and I am getting back into the grove. I did have a few friends ask where I plan on working next since everyone they know that has passed there CISSP left the company they where working for to find a new higher paying job. I think I will just stay where I am unless some one comes looking for me. I also have been getting a lot of hits from headhunters. I think the headhunters are picking up there pace to find DOCSIS 3.0 contracts to respond to the industry. I still do not see all the MSO’s jumping to 3.0 any time too soon since there is still no mass produced DOCSIS 3.0 modems and the cost of the modems will be way to high. There is also the debate with I-CMTS and M-CMTS still in the air but as of last time I have heard from the vendors they are now offering both options for CMTS’s. Right now I think the MSO’s will focus on DOCSIS 3.0D and just do the downstream channel bonding that is available today.

DSG . . . , Digital Set-top Gateway so what does this mean? Well it looks like DOCSIS will be the highway to the next generation on cable boxes for your TV (video over IP then over DOCSIS). It started with data and then there was voice & now video. What does this mean to your network? I think it's time to buy more Set-Top boxes, CMTS's, and now we really will have to make sure we have QoS setup right. Voice has to be the highest priority, then video and lat data. The days of your non MSO VoIP service could be limited if the QoS on data is set and there is congestion. Next is how do we as DOCSIS engineers trouble shoot all this and what kinda of tools will the field guys have to look at the QAM, Signal level, VoIP Quality, and possible multicast video steams? The future is looking good for DOCSIS engineers as you will have lots of job security but will you have the skills and tools to really work with the whole network in your hands? The only thing left will be cellular and maybe soon that will be backhauled with T-1 over DOCSIS. T-1 over DOCSIS is already deployed in some MSO's so we just need to make sure the plant stays very clean cause will will need very high QAM's to support the data needed for all the services running over DOCSIS now and in the future.