In General
Summary
(Source: Official Web Page)
Nowadays, incident handling has become more and more challenging due to increasing amounts of collected data used in the process and also due to lack of automation. Since scalability plays a key role in making efficient incident handling, ENISA supports good community driven initiatives to improve incident handling. The main goal of this project is to automate and improve incident handling process for CERTs by providing easy to set up and deploy solutions for Incident Response process.
Contributors: ENISA, CNCS (National Cyber Security Centre – Portugal), CERT.AT, CERT-EU, CERT.BE
IntelMQ is a solution for CERTs for collecting and processing security feeds, pastebins, tweets using a message queue protocol. It’s a community driven initiative called IHAP (Incident Handling Automation Project) which was conceptually designed by European CERTs during several InfoSec events. Its main goal is to give to incident responders an easy way to collect and process threat intelligence thus improving the incident handling processes of CERTs. IntelMQ’s design was influenced by AbuseHelper, however it was re-written from scratch and aims at:
- Reduce the complexity of system administration
- Reduce the complexity of writing new bots for new data feeds
- Reduce the probability of events lost in all process with persistence functionality (even system crash)
- Use and improve the existing Data Harmonization Ontology
- Use JSON format for all messages
- Integration of the existing tools (AbuseHelper, CIF)
- Provide easy way to store data into Log Collectors like ElasticSearch, Splunk
- Provide easy way to create your own black-lists
- Provide easy communication with other systems via HTTP RESTFUL API
IntelMQ Manager is a graphical interface to manage configurations for the IntelMQ framework. A IntelMQ configuration is a set of config files which describe which bots and processing steps should be run in which order. It is similar to describing the dataflow in dataflow oriented languages. IntelMQ Manager is therefore an intuitive tool to allow non-programmers to specify the data flow in IntelMQ.
Resources
https://www.enisa.europa.eu/activities/cert/support/incident-handling-automation
https://github.com/certtools/intelmq/blob/master/docs/User-Guide.md
https://github.com/certtools/intelmq
https://github.com/certtools/intelmq-manager
Backend
Installation
The installation guide is based on Ubuntu 14.04 and uses Python 3.4. It has been tested in a VirtualBox VM. It mostly follows the original installation guide, which should obviously kept under surveillance for updates :)
# Switch to root level
justme@UBUNTU14-VM:~$ sudo -s
root@UBUNTU14-VM:~$ apt-get update
root@UBUNTU14-VM:~$ apt-get upgrade
# Python 3, curl and headers
#
root@UBUNTU14-VM:~$ apt-get install python3 python3-pip
root@UBUNTU14-VM:~$ apt-get install git build-essential libcurl4-gnutls-dev libffi-dev
# 'psutil' python module
# https://github.com/giampaolo/psutil/blob/master/INSTALL.rst
# (Note: this is not in the original documentation)
#
root@UBUNTU14-VM:~$ cd /tmp
root@UBUNTU14-VM:~$ apt-get install gcc python-dev
root@UBUNTU14-VM:~$ wget https://bootstrap.pypa.io/get-pip.py
root@UBUNTU14-VM:~$ python get-pip.py
root@UBUNTU14-VM:~$ pip install psutil
# REDIS database
#
root@UBUNTU14-VM:~$ apt-get install redis-server
# INTELmq from sources
#
root@UBUNTU14-VM:~$ git clone https://github.com/certtools/intelmq.git /tmp/intelmq
root@UBUNTU14-VM:~$ cd /tmp/intelmq
root@UBUNTU14-VM:/tmp/intelmq$ pip3 install -r REQUIREMENTS
root@UBUNTU14-VM:/tmp/intelmq$ python3.4 setup.py install
root@UBUNTU14-VM:/tmp/intelmq$ # Create a dedicated user account:
root@UBUNTU14-VM:/tmp/intelmq$ useradd -d /opt/intelmq -U -s /bin/bash intelmq
root@UBUNTU14-VM:/tmp/intelmq$ echo 'export PATH="$PATH:$HOME/bin"' > /opt/intelmq/.profile
root@UBUNTU14-VM:/tmp/intelmq$ chmod -R 0770 /opt/intelmq
root@UBUNTU14-VM:/tmp/intelmq$ chown -R intelmq.intelmq /opt/intelmq
root@UBUNTU14-VM:/tmp/intelmq$ echo 'export INTELMQ_PYTHON=/usr/bin/python3.4' >> /opt/intelmq/.profile
# Test it...
#
root@UBUNTU14-VM:/tmp/intelmq# # Switch to intelmq user account:
root@UBUNTU14-VM:/tmp/intelmq# su - intelmq
$intelmq@UBUNTU14-VM:~$ intelmqctl --h
usage:
intelmqctl --bot [start|stop|restart|status] --id=cymru-expert
intelmqctl --botnet [start|stop|restart|status]
intelmqctl --list [bots|queues]
:
:
Execution
root@UBUNTU14-VM:/opt# su - intelmq intelmq@UBUNTU14-VM:~$ pwd /opt/intelmq intelmq@UBUNTU14-VM:~$ intelmqctl -n start intelmqctl: Starting Botnet... intelmqctl: Starting file-output... intelmqctl: file-output is running. intelmqctl: Starting malware-domain-list-collector... intelmqctl: malware-domain-list-collector is running. intelmqctl: Starting malware-domain-list-parser... intelmqctl: malware-domain-list-parser is running. intelmqctl: Botnet is running. intelmq@UBUNTU14-VM:~$
Frontend: IntelMQ Manager
Introduction
IntelMQ Manager is a graphical interface to manage configurations for the IntelMQ framework. A IntelMQ configuration is a set of config files which describe which bots and processing steps should be run in which order. It is similar to describing the dataflow in dataflow oriented languages. IntelMQ Manager is therefore an intuitive tool to allow non-programmers to specify the data flow in IntelMQ.
https://github.com/certtools/intelmq-manager
Installation
The installation of the web application is described here: https://github.com/certtools/intelmq-manager/blob/master/docs/INSTALL.md
Hereafter, a quick recap:
# (L)A<del>M</del>P # root@UBUNTU14-VM:/tmp$ apt-get install git apache2 php5 libapache2-mod-php5 # INELmq Manager root@UBUNTU14-VM:/tmp$ git clone https://github.com/certtools/intelmq-manager.git /tmp/intelmq-manager root@UBUNTU14-VM:/tmp$ cp -R /tmp/intelmq-manager/intelmq-manager/* /var/www/html/ root@UBUNTU14-VM:/tmp$ chown -R www-data.www-data /var/www/html/ root@UBUNTU14-VM:/tmp$ usermod -a -G intelmq www-data root@UBUNTU14-VM:/tmp$ vi /etc/sudoers --> add the line: --> www-data ALL=(intelmq) NOPASSWD: /opt/intelmq/bin/intelmqctl root@UBUNTU14-VM:/tmp$ vi /var/www/html/php/config.php --> add if not present: --> $CONTROLLER = "sudo -u intelmq /opt/intelmq/bin/intelmqctl %s"; root@UBUNTU14-VM:/tmp$ /etc/init.d/apache2 restart
Then, o joy: http://localhost