Open-source framework to detect outliers in Elasticsearch events

NVISOsecurity, updated 🕥 2022-12-08 00:46:25

ee-outliers

Framework to easily detect outliers in Elasticsearch events.

Developed in Python and fully dockerized!

version badge tests badge

Documentation

Introduction

Using ee-outliers

Misc.

What is ee-outliers?

ee-outliers is a framework to detect statistical outliers in events stored in an Elasticsearch cluster. It uses easy to write user-defined configuration files to decide which & how events should be analysed for outliers.

The framework was developed for the purpose of detecting anomalies in security events, however it could just as well be used for the detection of outliers in other data.

The only thing you need is Docker and an Elasticsearch cluster and you are ready to start your hunt for outlier events!

Why ee-outliers?

Although we love Elasticsearch, its search language is still lacking support for complex queries that allow for advanced analysis and detection of outliers - features we came to love while using other tools such as Splunk.

This framework tries to solve these limitations by allowing the user to write simple use cases that can help in spotting outliers in your data using statistical and models. Machine learning models are under development.

How it works

The framework makes use of statistical models that are easily defined by the user in a configuration file. In case the models detect an outlier, the relevant Elasticsearch events are enriched with additional outlier fields. These fields can then be dashboarded and visualized using the tools of your choice (Kibana or Grafana for example).

The possibilities of the type of anomalies you can spot using ee-outliers is virtually limitless. A few examples of types of outliers we have detected ourselves using ee-outliers during threat hunting activities include:

  • Detect beaconing (DNS, TLS, HTTP, etc.)
  • Detect geographical improbable activity
  • Detect obfuscated & suspicious command execution
  • Detect fileless malware execution
  • Detect malicious authentication events
  • Detect processes with suspicious outbound connectivity
  • Detect malicious persistence mechanisms (scheduled tasks, auto-runs, etc.)
  • …

Visit the page Getting started to get started with outlier detection in Elasticsearch yourself!

Contact

ee-outliers is developed & maintained by NVISO Labs.

You can reach out to the developers of ee-outliers by creating an issue in github.
For any other communication, you can reach out by sending us an e-mail at [email protected].

We write about our research on our blog: https://blog.nviso.eu
You can follow us on twitter: https://twitter.com/NVISO_Labs

Thank you for using ee-outliers and we look forward to your feedback! 🐀

License

ee-outliers is released under the GNU GENERAL PUBLIC LICENSE v3 (GPL-3). LICENSE

Acknowledgements

We are grateful for the support received by INNOVIRIS and the Brussels region in funding our Research & Development activities.

Getting started →

NVISO Labs logo

Issues

Bump certifi from 2017.7.27.1 to 2022.12.7

opened on 2022-12-08 00:46:25 by dependabot[bot]

Bumps certifi from 2017.7.27.1 to 2022.12.7.

Commits


Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/NVISOsecurity/ee-outliers/network/alerts).

Bump pillow from 9.0.1 to 9.3.0

opened on 2022-11-22 10:24:36 by dependabot[bot]

Bumps pillow from 9.0.1 to 9.3.0.

Release notes

Sourced from pillow's releases.

9.3.0

https://pillow.readthedocs.io/en/stable/releasenotes/9.3.0.html

Changes

... (truncated)

Changelog

Sourced from pillow's changelog.

9.3.0 (2022-10-29)

  • Limit SAMPLESPERPIXEL to avoid runtime DOS #6700 [wiredfool]

  • Initialize libtiff buffer when saving #6699 [radarhere]

  • Inline fname2char to fix memory leak #6329 [nulano]

  • Fix memory leaks related to text features #6330 [nulano]

  • Use double quotes for version check on old CPython on Windows #6695 [hugovk]

  • Remove backup implementation of Round for Windows platforms #6693 [cgohlke]

  • Fixed set_variation_by_name offset #6445 [radarhere]

  • Fix malloc in _imagingft.c:font_setvaraxes #6690 [cgohlke]

  • Release Python GIL when converting images using matrix operations #6418 [hmaarrfk]

  • Added ExifTags enums #6630 [radarhere]

  • Do not modify previous frame when calculating delta in PNG #6683 [radarhere]

  • Added support for reading BMP images with RLE4 compression #6674 [npjg, radarhere]

  • Decode JPEG compressed BLP1 data in original mode #6678 [radarhere]

  • Added GPS TIFF tag info #6661 [radarhere]

  • Added conversion between RGB/RGBA/RGBX and LAB #6647 [radarhere]

  • Do not attempt normalization if mode is already normal #6644 [radarhere]

... (truncated)

Commits


Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/NVISOsecurity/ee-outliers/network/alerts).

Wrong count of number of outliers and whitelisted outliers

opened on 2020-08-17 13:57:33 by maximilienroberti

If you observe the function process_outlier() in analyzer.py you can see that self.total_outliers is incremented event if the outlier is whitelisted. https://github.com/NVISO-BE/ee-outliers/blob/58021dc20f6cbbe411c0a6337ea39a82fc139a9d/app/helpers/analyzer.py#L220-L235 Also, I observed that the number of whitelisted outliers represented by the variable 'self.nr_whitelisted_elements' is never incremented in that function. As a result, it does not count the number of whitelisted outliers in simplequery, word2vec and sudden_appearance models.

Improvement idea sudden_appearance model

opened on 2020-08-17 10:57:49 by maximilienroberti

Add parameter min_aggregator_bucket with default value around 1000. It would classify an event as outlier only if the aggregator bucket has a size bigger than the parameter min_aggregator_bucket. Why? By observing the outliers on production, most of the FP( near to the total ) have small number of events in their aggregation bucket. To illustrate the idea, let's observe the following example use-case: ```

SUDDEN APPEARANCE - RENAMED PROCESS

[sudden_appearance_winlog_renamed_process] es_query_filter=exists:winlog.event_id AND winlog.event_id: 1

aggregator=winlog.event_data.Description.keyword target=process.name

history_window_days=7 history_window_hours=0

Size of the sliding window defined in DDD:HH:MM

Therefore, 20:13:20 will correspond to 20 days 13 hours and 20 minutes

sliding_window_size=03:00:00

sliding_window_step_size=00:01:00

outlier_type=first observation outlier_reason=sudden appearance of a renamed process outlier_summary=sudden appearance of a process renamed to {process.name} with description {winlog.event_data.Description}

run_model=1 test_model=0 `` It is worth noting thatwinlog.event_data.Description, which is selected asaggregator parameter, corresponds to the process description which stay constant even if you change the name of the process. Therefore, the goal of this use-case is to catch events with process name that suddenly change (ATT&CK T1218 or T1036) while the description stay the same. Example:powershell.exewith descriptionWindows PowerShellthat is suddenly renamed tocatchme.exe`. If an event is caught as outlier with a aggregator bucket of size near to 1, it will simply means that a new unseen process is suddenly running but not that a process has been suddenly renamed. At the opposite, if the bucket size is big it means that a lot of event has been observed with a certain description and a certain name and that the name has been suddenly changed.

This improvement is easy to implement and for my point of view, essential to make sudden_appearance work efficiently. @daanraman @michielmeersmans What do you think?

whitelist information should not be added to events

opened on 2020-07-22 12:19:38 by daanraman

Screenshot 2020-07-22 at 14 18 39

Notifier TheHive

opened on 2020-06-15 08:58:33 by V1D1AN

Hello, ee-outliers seems like a good project, do you plan to add "notifier" like "TheHive" or other ?? SMTP is only the possibility for the moment.

NVISO
GitHub Repository Homepage

outliers netsec threat-hunting statistics security-monitoring anomaly-detection outlier-detection siem cirt security-operations ee-outliers ml machine-learning statistical-analysis