Saturday, February 29, 2020

SIP Monitoring based on SIP3 and Heplify

Few words upfronts.

 - What is SIP3?

Follow the link :) If you don't know why you need it (or similar system) - you don't.

 - Why SIP3 and not Homer
Homer 3 was really good. It actually covers all my needs in SIP traffic monitoring. Than Homer 5 with new GUI was introduced. And things slowly start getting worse. Mostly cause of really strange changes in UI, which can stop working at any moment. No, it will respond, show something, but sometimes not the real picture. The usual answer - log out and back in. With Homer 7 things went even worse. Recommendation from developers is using the latest alpha version. If set up the system with a recommended setup (set of docker containers), it's not a "set and forget" solution. No. It will stop processing traffic at some time without any warning. For monitoring system which I need not every day (actually I need mostly post-mortem traces), it always ends up with log into the system and finds out it stops working like a week ago. Not the best option.
But it's really my experience. Maybe I'm not that lucky at all.

 - Why heplify and not captain (which is a part of SIP3)?
The answer is quite simple. Heplify - single binary, captain (at the moment) - a full zoo of tools, like docker, ansible, python, etc. Yes, I'm aware, that captain also can provide RTCP stats, but overhead is too big.

So, getting to the good part.

Installation and setting up

Central server

I'm gonna use CentOS 8 (minimal). Not like I'm a big fan of it, but developers like it. So, most tested one

1. Install docker

# yum install -y yum-utils   device-mapper-persistent-data  lvm2
# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
# containerdSource=https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.6-3.3.el7.x86_64.rpm
# dnf install $containerdSource -y
# dnf install docker-ce docker-ce-cli -y
# systemctl disable firewalld
# systemctl start docker
# systemctl enable --now docker

 

2. Install ansible, pip, git


# yum install -y epel-release
# yum install -y ansible git python3-pip
# pip3 install docker-py


3. Install SIP3 backend


# cd /usr/src/
# git clone https://github.com/sip3io/sip3-ansible.git


Note on taking metrics from public network services.

/usr/src/sip3-ansible/roles/sip3-salto/templates/application.yml.j2
#! Server
server:
  uri: udp://0.0.0.0:15060


#! Management socket
management:
  uri: udp://0.0.0.0:15090

#! Metrics
metrics:
  logging:
    step: 1000
  influxdb:
    uri: http://sip3-influxdb:8086
    db: sip3
    step: 1000

#! MongoDB
mongo:
  uri: mongodb://sip3-mongodb:27017
  db: sip3
  bulk-size: 1
  collections:
    - prefix: attributes
      indexes:
        ascending: [name]
      max-collections: 7
    - prefix: sip_register_index
      indexes:
        ascending: [created_at, terminated_at, src_addr, src_host, dst_addr, dst_host, caller, callee, state]
        hashed: [call_id, x_call_id]
      max-collections: 7
    - prefix: sip_register_raw
      indexes:
        ascending: [created_at]
        hashed: [call_id]
      max-collections: 7
    - prefix: sip_call_index
      indexes:
        ascending: [created_at, terminated_at, src_addr, src_host, dst_addr, dst_host, caller, callee, state, error_code, error_type, duration, setup_time, establish_time]
        hashed: [call_id, x_call_id]
      max-collections: 7
    - prefix: sip_call_raw
      indexes:
        ascending: [created_at]
        hashed: [call_id]
      max-collections: 7
    - prefix: sip_message_index
      indexes:
        ascending: [created_at, terminated_at, src_addr, src_host, dst_addr, dst_host, caller, callee, state]
        hashed: [call_id, x_call_id]
      max-collections: 7
    - prefix: sip_message_raw
      indexes:
        ascending: [created_at]
        hashed: [call_id]
      max-collections: 7
    - prefix: sip_options_index
      indexes:
        ascending: [created_at, terminated_at, src_addr, src_host, dst_addr, dst_host, caller, callee, state]
        hashed: [call_id, x_call_id]
      max-collections: 7
    - prefix: sip_options_raw
      indexes:
        ascending: [created_at]
        hashed: [call_id]
      max-collections: 7

#! Application
sip:
  message:
    x-correlation-header: X-Call-ID
    exclusions: [MESSAGE, SUBSCRIBE, NOTIFY, OPTIONS, REGISTER]
  calls:
    aggregation-timeout: 120000

attributes:
  record-ip-addresses: false
  record-call-users: false


It's not to index in the database all IP addresses and usernames are seen on the system, cause the production system usually resides on public networks, where are tons of spam/attack calls and packets. Also retention period for database is set to 7 days, which is ok for me.

# ansible-playbook /usr/src/sip3-ansible/playbooks/trial/sip3-backend.yml

Relax and wait for system installs.

4. Add autostart


 /etc/systemd/system/sip3-backend.service

[Unit]
Description=Start Ansible Playbook SIP3
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/ansible-playbook /usr/src/sip3-ansible/playbooks/trial/sip3-backend.yml
TimeoutStartSec=10

[Install]
WantedBy=default.target




# systemctl daemon-reload
# systemctl enable sip3-backend


5. (Optional )Add hosts aliases to db

To get more friendly trances to read. Point, no spaces in names are allowed.


# docker exec -it sip3-mongodb mongo
# use sip3
# db.hosts.insert({"name" : "my_softswitch", "sip": ["X.X.X.X", "10.10.0.1"]})

Installing heplify (capture agent)

Log in to the server you want to collect SIP traffic from

# cd /usr/local/bin/
# wget https://github.com/sipcapture/heplify/releases/download/1.56/heplify
# chmod +x heplify


/etc/systemd/system/heplify.service
[Unit]
Description=Captures packets from wire and sends them to SIP3
After=network.target

[Service]
WorkingDirectory=/usr/local/bin
ExecStart=/usr/local/bin/heplify -i any -m SIP -hs SIP3_SERVER_ADDRESS:15060
ExecStop=/bin/kill ${MAINPID}
Restart=on-failure
RestartSec=10s
Type=simple

[Install]
WantedBy=multi-user.target




# systemctl daemon-reload
# systemctl enable heplify
# systemctl start heplify


System using

Log into your newly installed system on
http://SIP3_SEVER_ADDRESS with admin:admin credentials.

Sometimes, just after login, you will need to refresh your page. Actually, make sure Go! button is active
It's your search string. Simple, but yet powerful. It really looks like Wireshark one.
You can add search critera there based on sip.
or ip.
criteria
ip.src_host and ip.dst_host are hosts we added at step 5 installing Central Server paragraph.

Changing password is quite tricky, cause SIP3 relies on Grafana auth (Grafana is shipped as well) and to change password, head to

 http://SIP3_SEVER_ADDRESS/grafana

and use it interface to change password. As a bonus, you have Grafana dashboard with some useful metrics. And for sure you can add one you like. How - read Grafana manual :)

Adding batteries

One is a key feature of SIP3 is the ability to glue call legs in a 1 call flow. Usual algorithm is based on a strict match From and To numbers, but it's really rare case (at least, in my scenarios). There is a possibility to write user-defined functions, but I'll leave it to official manual. Another way to match call is to use custom SIP headers to match call legs. The default SIP3 header is X-Call-ID. So, if one leg holds Call-ID standart header matching other call X-Call-ID header, they would consider as one call and glue together.

In my case, I'm using FusionPBX, the solution is really simple. Add dialplan entry:
It will add X-Call-ID header to call equal to current Call-ID, if it's not present. Also, FreeSwitch preserves X-Headers over call legs. which is perfect for us.

So, we will have a picture like this

Upgrading

# cd /usr/src
# git pull
# ansible-playbook /usr/src/sip3-ansible/playbooks/trial/sip3-backend.yml


Happy using!