BGP Ranking

For an Internet Service Provider, AS numbers are a logical representation of the other ISP peering or communicating with its autonomous system. ISP customers are using the capacity of the Internet Service Provider to reach Internet services over other AS. Some of those communications can be malicious (e.g. due to malware activities on an end-user equipments) and hosted at specific AS location.

In order to provide an improved security view on those AS numbers, a trust ranking scheme is implemented based on existing dataset of compromised systems, malware C&C IP and existing datasets. BGP Ranking provides a way to collect such malicious activities, aggregate the information per ASN and provide a ranking model to rank the ASN from the most malicious to the less malicious ASN.

The official website of the project is: https://github.com/D4-project/bgp-ranking/

There is a public BGP Ranking at http://bgpranking.circl.lu/

BGP Ranking is free software licensed under the GNU Affero General Public License

BGP Ranking is a software to rank AS numbers based on their malicious activities.

Python client

```bash $ pip install git+https://github.com/D4-project/BGP-Ranking.git/#egg=pybgpranking\&subdirectory=client $ bgpranking --help usage: bgpranking [-h] [--url URL] (--asn ASN | --ip IP)

Run a query against BGP Ranking

optional arguments: -h, --help show this help message and exit --url URL URL of the instance. --asn ASN ASN to lookup --ip IP IP to lookup ```

History

• The first version of BGP Ranking was done in 2010 by Raphael Vinot with the support of Alexandre Dulaunoy. CIRCL supported the project from the early beginning and setup an online version to share information about the malicious ranking of ISPs.

• In late 2018 within the scope of the D4 Project (a CIRCL project co-funded by INEA under the CEF Telecom program), a new version of BGP Ranking was completed rewritten in python3.6+ with an ARDB back-end.

• In January 2022, BGP Ranking version 2.0 was released including a new backend on kvrocks and many improvements.

Online service

BGP Ranking service is available online http://bgpranking.circl.lu/.

A Python library and client software is available using the default API available from bgpranking.circl.lu.

CURL Example

Get the ASN from an IP or a prefix

bash curl https://bgpranking-ng.circl.lu/ipasn_history/?ip=143.255.153.0/24

Response

json { "meta": { "address_family": "v4", "ip": "143.255.153.0/24", "source": "caida" }, "response": { "2019-05-19T12:00:00": { "asn": "264643", "prefix": "143.255.153.0/24" } } }

Get the ranking of the AS

curl -X POST -d '{"asn": "5577", "date": "2019-05-19"}' https://bgpranking-ng.circl.lu/json/asn

Note: date isn't required.

Response

json { "meta": { "asn": "5577" }, "response": { "asn_description": "ROOT, LU", "ranking": { "rank": 0.0004720052083333333, "position": 7084, "total_known_asns": 15375 } } }

Get historical information for an ASN

curl -X POST -d '{"asn": "5577", "period": 5}' https://bgpranking-ng.circl.lu/json/asn_history

Response

json { "meta": { "asn": "5577", "period": 5 }, "response": { "asn_history": [ [ "2019-11-10", 0.00036458333333333335 ], [ "2019-11-11", 0.00036168981481481485 ], [ "2019-11-12", 0.0003761574074074074 ], [ "2019-11-13", 0.0003530092592592593 ], [ "2019-11-14", 0.0003559027777777778 ] ] } }

Server Installation (if you want to run your own)

IMPORTANT: Use pipenv

NOTE: Yes, it requires python3.6+. No, it will never support anything older.

Install redis

bash git clone https://github.com/antirez/redis.git cd redis git checkout 5.0 make make test cd ..

Install ardb

bash git clone https://github.com/yinqiwen/ardb.git cd ardb DISABLE_WARNING_AS_ERROR=1 make # ardb (more precisely rocksdb) doesn't compile on ubuntu 18.04 unless you disable warning as error cd ..

Install & run BGP Ranking

``bash git clone https://github.com/D4-project/BGP-Ranking.git cd BGP-Ranking pipenv install echo BGPRANKING_HOME="'pwd`'" > .env pipenv shell

Starts all the backend

start.py

Start the web interface

start_website.py ```

Shutdown BGP Ranking

bash stop.py

Directory structure

Config files: bgpranking / config / *.json

Per-module parsers: bgpraking / parsers

Libraries : brpranking / libs

Raw dataset directory structure

Files to import

Note: The default location of <storage_directory> is the root directory of the repo.

<storage_directory> / <vendor> / <listname>

Last modified date (if possible) and lock file

<storage_directory> / <vendor> / <listname> / meta

Imported files less than 2 months old

<storage_directory> / <vendor> / <listname> / archive

Imported files more than 2 months old

<storage_directory> / <vendor> / <listname> / archive / deep

Databases

Intake (redis, port 6579)

Usage: All the modules push their entries in this database.

Creates the following hashes:

python UUID = {'ip': <ip>, 'source': <source>, 'datetime': <datetime>}

Creates a set intake for further processing containing all the UUIDs.

Pre-Insert (redis, port 6580)

Usage: Make sure th IPs are global, validate input from the intake module.

Pop UUIDs from intake, get the hashes with that key

Creates the following hashes:

python UUID = {'ip': <ip>, 'source': <source>, 'datetime': <datetime>, 'date': <date>}

Creates a set to_insert for further processing containing all the UUIDs.

Creates a set for_ris_lookup to lookup on the RIS database. Contains all the IPs.

Routing Information Service cache (redis, port 6581)

Usage: Lookup IPs against the RIPE's RIS database

Pop IPs from for_ris_lookup.

Creates the following hashes:

python IP = {'asn': <asn>, 'prefix': <prefix>, 'description': <description>}

Ranking Information cache (redis, port 6582)

Usage: Store the current list of known ASNs at RIPE, and the prefixes originating from them.

Creates the following sets:

python asns = set([<asn>, ...]) <asn>|v4 = set([<ipv4_prefix>, ...]) <asn>|v6 = set([<ipv6_prefix>, ...])

And the following keys:

python <asn>|v4|ipcount = <Total amount of IP v4 addresses originating this AS> <asn>|v6|ipcount = <Total amount of IP v6 addresses originating this AS>

Long term storage (ardb, port 16579)

Usage: Stores the IPs with the required meta informations required for ranking.

Pop UUIDs from to_insert, get the hashes with that key

Use the IP from that hash to get the RIS informations.

Creates the following sets:

```python

All the sources, by day

|sources = set([, ...])

All the ASNs by source, by day

| -> set([, ...])

All the prefixes, by ASN, by source, by day

|| -> set([, ...])

All the tuples (ip, datetime), by prefixes, by ASN, by source, by day

||| -> set([|, ...]) ```

Issues

argparse code probably outdated

opened on 2021-10-10 07:32:44 by SteveClement

After installing all the required deps it fails on a namespace error.

(venv) [email protected]:~/code/BGP-Ranking$ bgpranking Traceback (most recent call last): File "/home/misp/code/BGP-Ranking/venv/bin/bgpranking", line 7, in <module> exec(compile(f.read(), __file__, 'exec')) File "/home/misp/code/BGP-Ranking/client/bin/bgpranking", line 43, in <module> if args.which == 'simple': AttributeError: 'Namespace' object has no attribute 'which'

Missing dependency when installing as guided in the README.md

opened on 2021-10-10 07:08:34 by SteveClement

(venv) [email protected]:~/code/mailoney$ pip install git+https://github.com/D4-project/BGP-Ranking.git/#egg=pybgpranking\&subdirectory=client Collecting pybgpranking Cloning https://github.com/D4-project/BGP-Ranking.git/ to /tmp/pip-install-pnqwv90c/pybgpranking_777f9736e6d849e8bb2af02b4252c0fe Running command git clone -q https://github.com/D4-project/BGP-Ranking.git/ /tmp/pip-install-pnqwv90c/pybgpranking_777f9736e6d849e8bb2af02b4252c0fe Resolved https://github.com/D4-project/BGP-Ranking.git/ to commit 36688e85c17442652b0b893605b50741c01bd62a Collecting requests Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB) Requirement already satisfied: charset-normalizer~=2.0.0 in ./venv/lib/python3.8/site-packages (from requests->pybgpranking) (2.0.6) Collecting urllib3<1.27,>=1.21.1 Using cached urllib3-1.26.7-py2.py3-none-any.whl (138 kB) Collecting idna<4,>=2.5 Using cached idna-3.2-py3-none-any.whl (59 kB) Collecting certifi>=2017.4.17 Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB) Using legacy 'setup.py install' for pybgpranking, since package 'wheel' is not installed. Installing collected packages: urllib3, idna, certifi, requests, pybgpranking Running setup.py install for pybgpranking ... done Successfully installed certifi-2021.10.8 idna-3.2 pybgpranking-0.1 requests-2.26.0 urllib3-1.26.7 (venv) [email protected]:~/code/mailoney$ bgpranking Traceback (most recent call last): File "/home/misp/code/mailoney/venv/bin/bgpranking", line 12, in <module> from pyipasnhistory import IPASNHistory ModuleNotFoundError: No module named 'pyipasnhistory'

Import error - ModuleNotFoundError: No module named 'api'

opened on 2021-10-10 07:06:04 by SteveClement

Trying to import bgpranking results in error. See below.

``` (venv) [email protected]:~/code/mailoney$ pip install bgpranking-web Collecting bgpranking-web Using cached bgpranking-web-1.2.tar.gz (2.1 kB) Collecting requests Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB) Collecting urllib3<1.27,>=1.21.1 Using cached urllib3-1.26.7-py2.py3-none-any.whl (138 kB) Collecting idna<4,>=2.5 Using cached idna-3.2-py3-none-any.whl (59 kB) Collecting certifi>=2017.4.17 Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB) Requirement already satisfied: charset-normalizer~=2.0.0 in ./venv/lib/python3.8/site-packages (from requests->bgpranking-web) (2.0.6) Using legacy 'setup.py install' for bgpranking-web, since package 'wheel' is not installed. Installing collected packages: urllib3, idna, certifi, requests, bgpranking-web Running setup.py install for bgpranking-web ... done Successfully installed bgpranking-web-1.2 certifi-2021.10.8 idna-3.2 requests-2.26.0 urllib3-1.26.7 (venv) [email protected]:~/code/mailoney$ ipython Python 3.8.10 (default, Sep 28 2021, 16:10:42) Type 'copyright', 'credits' or 'license' for more information IPython 7.28.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import bgpranking_web

ModuleNotFoundError Traceback (most recent call last) in ----> 1 import bgpranking_web

~/code/mailoney/venv/lib/python3.8/site-packages/bgpranking_web/init.py in ----> 1 from api import *

ModuleNotFoundError: No module named 'api' ```

Old historical data

opened on 2019-11-27 05:09:55 by DuhaAlasiri

We would like to thank you for providing historical BGP Ranking data, but can we go back with the data before 2018-04-10? We could do that in the old BGP Ranking and wonder if this one does the same.

Thank you for your efforts.

Lookup per invalid IP address gives Internal Server Error

opened on 2019-05-29 18:37:42 by btlogy

Likely introduced by #1, query sent with invalid or empty IP gives a 500: ``` $ curl -vvv 'https://bgpranking.circl.lu/ipasn' -d 'ip=' ...

POST /ipasn HTTP/1.1 Host: bgpranking.circl.lu User-Agent: curl/x.x.x Accept: / Content-Length: 3 Content-Type: application/x-www-form-urlencoded

• upload completely sent off: 3 out of 3 bytes < HTTP/1.1 500 INTERNAL SERVER ERROR < Date: Wed, 29 May 2019 18:30:38 GMT < Server: gunicorn/x.x.x < Strict-Transport-Security: max-age=15768000 < Content-Type: text/html < Content-Length: 290 < Connection: close <

Internal Server Error

The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

... ```

Vizualize new entries / new entries doing bad things

opened on 2019-05-22 12:29:34 by gallypette

It would be interesting to be able to see when new AS arrive, and when/if they start to misbehave.

Releases

BGP Ranking version 2.1 released 2022-04-27 08:16:20

BGP Ranking version 2.1 released including updated dependencies.

BGP Ranking version 2.0 release - major upgrade 2022-01-18 13:54:43

BGP Ranking version 2.0 release - major upgrade 2.0 (2022-01-18)

BGP Ranking version 2.0 is released including an improvement back-end relying on kvrocks and many improvements including source import, additional sources and many bugs fixed.

New

• Add missing types in ssfetcher. [Raphaël Vinot]

• Abuse.ch lists - SSLBlacklist, ThreatFox. [Raphaël Vinot]

Changes

• Bump deps, add pybgpranking2. [Raphaël Vinot]

• Use best source from ipasnhistory if possible. [Raphaël Vinot]

• Improve shadow server import, support network in sanitizer. [Raphaël Vinot]

• Improve logging when something is broken when caching. [Raphaël Vinot]

• Improve logging for parser on exception. [Raphaël Vinot]

• Bump deps. [Raphaël Vinot]

• Slight changes in migrate script. [Raphaël Vinot]

• Remove old file. [Raphaël Vinot]

• Bump deps. [Raphaël Vinot]

• Improve logging. [Raphaël Vinot]

• Move API to restx. [Raphaël Vinot]

• Migrate to new framework. [Raphaël Vinot]

• Sync code with prod. [Raphaël Vinot]

• Remove bambenekconsulting feeds (not free anymore) [Raphaël Vinot]

Fix

• Incorrect redirect in asn. [Raphaël Vinot]

• Avoid exception if the source is created after we try to get the modules. [Raphaël Vinot]

• Do not cache load_all_modules_configs, it is dynamic. [Raphaël Vinot]

• Properly forward data to ipasnhistory. [Raphaël Vinot]

• Incorrect regex, again. [Raphaël Vinot]

• Incorrect regex match for list update. [Raphaël Vinot]

• Missing variable in threatfox parser. [Raphaël Vinot]

• Properly name abuse.ch SSL blacklist. [Raphaël Vinot]

• Avoid exception on GET request for ipasnhistory proxy. [Raphaël Vinot]

• POST for ipasnhistory via bgpranking works again. [Raphaël Vinot]

• Also start the website. [Raphaël Vinot]

Other

• Merge pull request #13 from D4-project/dependabot/pip/jinja2-2.11.3. [Steve Clement]

• Build(deps): bump jinja2 from 2.10.3 to 2.11.3. [dependabot[bot]]

Bumps jinja2 from 2.10.3 to 2.11.3. - Release notes - Changelog - Commits