Managing CE eBGP details & common/accepted CE-facing BGP practices

[justin at justinshore.com (Justin Shore)]

Evening, Justin.  Thanks for the reply.

Justin M. Streiner wrote:
> You could certainly store all of the relevant config details in a 
> database of some sort, and it certainly can't hurt to do so.  Same goes 
> for backing up your device configurations - always a good idea.  As far 
> as storing things like ASNs, allowed prefixes, etc, you may want to look 
> at storing that information in an RPSL format.  Many providers require 
> their customers to register route/AS/policy objects either in one of the 
> Internet Routing Registries (RADB, AltDB, etc...), or in a similar 
> system that's operated by the provider.  They then use this information 
> for pushing out configuraiton changes such as the list of prefixes that 
> a customer is allowed to announce, etc.

RPSL could definitely be useful when we starting reaching that class of 
customer.  It's probably too grand for our users at this point 
(especially having them register anything on their own).  I'll 
definitely do more research on this though.

> It's good that you're thinking about this now, particularly the "is this 
> customer legitimately allowed to announce prefix ABC, or source their 
> traffic from AS XYZ"?  Most larger providers put at least some 
> limitations on what they will allow a customer to announce, though the 
> level of investigation done to attempt to establich legitimacy for those 
> announcements varies.  Some providers require customers to provide some 
> sort of Letter of Authorization/Agency for the prefixes they want to 
> announce.

The process hasn't been established yet for validating a request to 
permit a prefix announcement.  I expect it will be a manual verification 
process involving WHOIS lookups on the prefix, route-view queries to see 
if it's currently being advertised and probably a historical check on 
that prefix to see if it was previously advertised and from where.  The 
best way to avoid network abuse issues is to not let them happen to 
begin with.  I think we will also require some sort of written legal 
agreement stating that the prefix in question belongs to the customer 
and that we're authorized to permit it's advertisement across our 
network.  If anyone has any sample documents for use in this process I 
would be interested in seeing them.

> Yes.  As long as the organizations that are using the private AS aren't 
> 1. trying to advertise the same space, 2. possibly connect directly to 
> each other directly, or 3. expecting to be able to connect to multiple 
> upstream providers, then you should be OK.  VZB (former UUNET) did 
> something similar, using AS7046 (not a private AS, but the principle is 
> the same), and I believe other carriers have had similar arrangements 
> for customers.

Ah, yes connecting to each other could be a problem.  I would think that 
it would only be an issue if I carried the private ASN across my iBGP 
infrastructure, each customer received full routes from me and I let 
them see the private ASNs as well.  I could mitigate that problem with 
remove-private-as, I believe.  I'd need to think on that some more.  If 
the customer wants to be multi-homed and expect reachability then they 
should get an ASN.  Otherwise both SPs are advertising their prefixes 
and the customer won't have much or possibly any control over which 
inbound path was preferred.

> MD5 is good, but most providers I've seen make this an opt-in feature - 
> they don't force their customers to use it.  Setting a reasonable 
> max-prefix limit and adjusting it as the number of prefixes a customer 
> announces is always a good idea, and I'd consider it to be a best 
> practice.  Prefix lists and route maps can do different things, or 
> accomplish the same task in different ways.  It also depends on what 
> functionality you want to offer your customers.  Do you plan to publish 
> and support a consistent set of customer-settable BGP communities for 
> doing things like selective AS prepends?  Do you plan to tag incoming 
> advertisements with communities to identify them as customer routes, and 
> pass those communities to your customers and peers?  Some providers use 
> AS path ACLs, and others avoid them at all costs.

I think I'll make MD5 part of the default config and let the customer 
ask for it to be removed if they choose to not have it.  Same for GTSM. 
  I'm a fan of max-prefixes.  I think double the routes I expect to 
receive, 75% warning and a restart interval of 5m would be a good place 
to start.  That would let me catch things happening before they got out 
of hand (in normal circumstances) and give me a fail-safe in case they 
decide to get crazy.

I do plan on implementing a BGP community solution but for now I'm going 
to keep it simple.  I have bigger fish to fry at the moment but I'll try 
to get it done before we get asked for it by a customer.  I will tag 
transit and customer routes.  The ISP Essentials book had some good 
insight on that if memory serves me correctly.  How fancy it gets will 
depend on my time and customer demand.  I've seen some extremely complex 
setups that I could not replicate if my life depended on it.

The AS-Path filtering should only come into effect when we get a 
customer with their own ASN in which cases we'll actually pass on their 
advertisements (whereas with private ASNs we're only carrying them 
internally and summarizing on the upstream edges).  That way I can 
ensure 1) that they claim to source the prefixes from their ASN and not 
someone else's and 2) that they don't insert BS ASNs into the path.

> If the customers need to connect to multiple upstream providers, it's 
> much less hassle to have them originate their prefixes from their own 
> AS, and then you just need to propagate them.  If they are single-homed, 
> you can announce the prefix(es) for them and then just statically route 
> them to the customer.  Other things to consider are sound ingress/egress 
> filtering policies, loose/strict RPF etc.  Implementing a Netflow based 
> monitoring solution, and applying flow caching to PE-CE interfaces is 
> also agood idea.

I have a good set of residential and tiny SMB customer ACLs.  I'm trying 
to decide what I should use for these larger SMBs.  I'm sure some will 
be custom but I need to come up with a sane default set of ACLs.  Some 
things I refuse to not block.  The customer can find a different 
provider if they want certain things to not be blocked.

Until we get a multi-homed customer that wants inbound reachability 
we'll go with strict uRPF; that's already in my interface templates.

I have NFSen set up but all it's really doing right now is filling my 
hard drives.  I haven't had much time to do anything else with it I'm 
afraid.  I'll point NF to MARS box when it gets here though.  I forget, 
does route-cache flow alter the packet switching capabilities of an 
interface these days or is that just to turn on NF on an interface? 
Seems like since CEF it only turns on NF.

> Again, it's great that you're being proactive about this.  Setting up 
> good policies now will make things much easier to maintain and provision 
> in the future as your network grows.  Don't forget to document those 
> policies thoroughly, particularly if other people will be tasked with 
> implementing it on a dat yo day basis, i.e. handling new customer 
> turn-ups, making BGP prefix list modifications, etc.

That's my goal.  I prefer to be organized from the get go whenever 
possible.  I don't like surprises in my work (unless it's a big, 
unexpected raise).

> It seems like you have a pretty good handle on what you need to do.  I 
> don't know that there will be much of the holy war angle to the 
> responses. Different people adopt different customer routing policies 
> for different reasons - some technical, some business, some political.  
> While there are different ways of accomplishing this task, most of the 
> more scalable ways have already been a part of most large providers' 
> policies for awhile.

I have an idea of what all needs to be implemented but I'm short on time 
to hone my skills by trial and error.

> One-offs are bad, in my opinion, so the more you do to avoid them now, 
> the fewer headaches you will have down the road.

Definitely.  One-offs are like using duct tape to repair a Porsche. 
It's just not right.

Thanks for the input.  It's much appreciated.
  Justin