Creating a routing and billing platform with OpenSIPS

Creating a routing and billing platform
with OpenSIPS
How hard could it be?
by
James Pearson
ViciDial Group
www.vicidial.com
A brief history
The core business is hosted ViciDial call centers
at vicihost.com. ViciDial offers full blended
operation and is very popular with clients needing
more then just basic inbound and outbound calls.
What we discovered is:
● Customers did not know what they needed
● Carriers did not know what they were doing
● The carriers that did won't return your call
● Instant Gratification was not there
● A good opportunity for more revenue
● Hopefully we do know what we are doing
The growing pains
Turns out we didn't quite know what we were
doing at first. But try we did...
●
Having sub accounts with carriers
●
Having a referral/residual account with carriers
●
Asterisk as a CDR gateway (which worked?)
●
FreeSWITCH as a CDR gateway
●
Hosted SBC
●
Potentially SBC hardware like Sonus or Cisco
Nothing beats home brew
So what do we really need to do?
●
Handle up to 1,000 calls per second
●
Blend multiple carriers and endpoints
●
Enforceable call control and billing
●
Accurate CDRs per client
●
Integrate with a database back end
●
Stable and scalable of course
●
Use the in-house talent pool (and know a guy)
●
Open source with an active community
The First Generation
●
●
●
●
●
●
●
●
●
●
OpenSIPS v.1.5 was the latest at the time
'opensips' database is replicated from
master through MySQL
Separate local database for acc and
dialog
Use MySQL socket for good performance
SQL Queries written as single table
selects for good querycache performance
in MySQL
PERL Process to scan the local acc table
and generate CDRs
Dynamic client set-up and control
provided through mysql calls
Autonomous should any one part of the
platform go down
Dynamic routing used for termination
Dispatcher for origination to multiple
endpoints
●
●
●
●
●
●
●
Using the local mysql socket with query cache gives near memcache
performance, in my testing it was single digit percentage difference
OpenSIPS is fully autonomous and does not have any external
service requirements to work other then power and internet
Local in-memory modules used to provide routing and initial access
control for endpoints to protect against overloading mysql
Caller ID enforced so that only allowed caller IDs for that client are
used to prevent spoofing and 000-000-0000 as a caller ID
Media Gateways are used to give audio feedback to the client and
handle carriers with special requirements
Asterisk dialplan written to fail-over on chanunavail from server 1 to
server 2 providing redundancy and ability to do maintenance
In 4 years of production since going live have only experienced 6
issues that cause an outage to clients, most involved shared memory
exhaustion in the opensips process and lasted less then 5 minutes
●
Has handled over a billion call attempts in 4 years
●
Currently averages around 120K calls per hour
Successful failure!?
It worked and worked well. Unfortunately the
needs were more complex then anticipated.
●
Exceptions. There are always exceptions...
●
CDR generation process was nightmarish
●
Somewhat troublesome dialplan code
●
Inefficient use of resources and coding
●
The scale of the business grew
●
So did the things we needed to do
The second version
OpenSIPS Forwarding Servers
●
●
●
●
●
●
●
Very small resource requirement
No active database calls
Uses Dynamic Routing for LCR
Uses Dispatcher for inbound
High CPS and concurrent calls
Easy for carriers to setup
Allows control servers to be slower
OpenSIPS Control Servers
●
Tenants with multiple customers
●
Routing generated from live database calls
●
Controls client capabilities and usage
●
Generates billable CDRs
●
Handles media through RTP Proxy
●
Automatic NAT detection and handling
●
Audio feedback provided through gateways
●
Support special services like Sangoma CPA
General routing overview
Client routing block
RTP Proxy auto detection block
RTP Proxy reply block
Some compromises and notes
●
No support for registration, it's all IP ACL
●
Tons of band-aids to fix external carrier issues
●
Getting coherent CDRs was a challenge
●
Load Balance doesn't skip the last destination
●
4 endpoints: Carrier, Customer, Proxy, GW/Proc
●
3 to 8 single-table DB calls per attempt, avg 5
●
400 lines in version 1 to 1000 lines in version 2
●
There is no good way to put it on a flowchart
General database layout
●
Two databases on control server, one replicated
and one local
●
PERL program scans local DB for CDRs
●
Processed CDRs are moved to an archive table
●
●
●
Another PERL program scans for control
changes to reload modules in OpenSIPS
OpenSIPS does not directly interact with billing,
only with determining the client for the CDR
Allows for autonomous operation
A visual DB representation
Database tweaks and tricks
●
3 separate databases: CDR, Control, Billing
●
CDR and Control needed for OpenSIPS
●
Coherent CDRs allowed for greater capacity
●
MariaDB works better then MySQL
●
MyISAM better then InnoDB on most tables
●
OpenSIPS schema had issue with indexes
●
CDRs rolled to archive table to prevent locks
●
Smaller query cache gives speed improvement
So what's it look like?
Old Hardware
New Hardware
New Hardware
Old Hardware
●
1U full-depth
●
3U 8-Node hot-swap chassis
●
Intel Quad-Core 2.4 GHZ
●
Intel Quad-Core 3.1 GHZ
●
8GB RAM
●
16GB RAM
●
RAID-10 73GB 10K-rpm
●
Linux RAID-1 240GB SSD
●
Serial Console
●
IPMI w/ SOL on separate LAN
●
Redundant PSU on 1-ph
●
Redundant PSU on 2-ph
●
Power hungry system
●
Bonded NIC with VLAN
●
Single network per NIC
●
PXE boot installation
●
~375 watts at full load
●
~115 watts at full load
Random rants
●
●
Carriers are always right, even when wrong
No matter how convoluted the billing, someone
can think of a way to make it 'better'
●
And that better way is always 'simple'
●
Web interfaces are horrible torturous things
●
●
You will be asked to prove that 300K CDRs are
correct like there is a secret way to do that
Any actual bugs/issues found will be blown out
of proportion and never forgotten about
The front side of the problem
●
Billing can be and is generally crazy
●
Written in PHP with JavaScript
●
A comprehensive solution becomes complex
●
If it looks right it must be right, regardless
●
●
●
Someone will want the background in
cornflower blue, not navy blue
Always takes longer to make the GUI then
everything else, at least for me
Never underestimate the average user
See, pretty.
In conclusion
●
●
●
●
●
OpenSIPS has proven to be versatile and
robust with active development and support
There is no simple way to build a billing
platform, despite what you have been told
While being a precise fit, OpenSIPS is a very
involved option
Always plan to over-build for tomorrow's needs
There are great resources available from the
community