Intelmq

---------------------------------

Configuration
In 1.1.0 the default value for the parameter `error_dump_message` was set to `false`. The recommended value, used in previous and future release is `true` to not loose any data in case of errors. Users are advised to check the values configured in their `defaults.conf` file.

Postgres databases
The following statements optionally update existing data.
Please check if you did use these feed names and eventually adapt them for your setup!
SQL
UPDATE events
SET "classification.taxonomy" = 'abusive content', "classification.type" = 'spam', "classification.identifier" = 'spam', "malware.name" = NULL, "source.fqdn" = "source.reverse_dns", "source.reverse_dns" = NULL, "source.url" = "destination.url", "destination.url" = NULL
WHERE "malware.name" = 'spam' AND "feed.name" = 'Drone';


In the section for 1.1.0 there was this command:

UPDATE events
SET "classification.identifier" = 'open-portmapper',
"protocol.application" = 'portmap'
WHERE "classification.identifier" = 'openportmapper' AND "feed.name" = 'Open-Portmapper' AND "protocol.application" = 'portmapper';

`protocol.application` was incorrect. To fix it you can use:

UPDATE events
SET "protocol.application" = 'portmapper'
WHERE "classification.identifier" = 'open-portmapper' AND "feed.name" = 'Open-Portmapper' AND "protocol.application" = 'portmap';


MongoDB databases
In previous version the MongoDB Output Bot saved the fields `time.observation` and `time.source` as strings in ISO format. But MongoDB does support saving datetime objects directly which are converted to its native date format, enabling certain optimizations and features. The MongoDB Output Bot now saves these values as datetime objects.

----------------------------------
Requirements
- Python 3.4 or newer is required.

Tools
- `intelmqctl start` prints bot's error messages in stderr if it failed to start.
- `intelmqctl check` checks if all keys in the packaged defaults.conf are present in the current configuration.

Contrib / Modify Expert
The malware name rules of the modify expert have been migrated to the [Malware Name Mapping repository](https://github.com/certtools/malware_name_mapping).
See `contrib/malware_name_mapping/` for download and conversion scripts as well as documentation.

Shadowserver Parser
The classification type for malware has been changed from "botnet drone" to the more generic "infected system".
The classification identifiers have been harmonized too:

| old identifier | new identifier |
|-|-|
| openmdns | open-mdns |
| openchargen | open-chargen |
| opentftp | open-tftp |
| openredis | open-redis |
| openportmapper | open-portmapper |
| openipmi | open-ipmi |
| openqotd | open-qotd |
| openssdp | open-ssdp |
| opensnmp | open-snmp |
| openmssql | open-mssql |
| openmongodb | open-mongodb |
| opennetbios | open-netbios-nameservice |
| openelasticsearch | open-elasticsearch |
| opendns | dns-open-resolver |
| openntp | ntp-monitor |
| SSL-FREAK | ssl-freak |
| SSL-Poodle | ssl-poodle |
| openmemcached | open-memcached |
| openxdmcp | open-xdmcp |
| opennatpmp | open-natpmp |
| opennetis | open-netis |
| openntpversion | ntp-version |
| sandboxurl | sandbox-url |
| spamurl | spam-url |
| openike | open-ike |
| openrdp | open-rdp |
| opensmb | open-smb |
| openldap | open-ldap |
| blacklisted | blacklisted-ip |
| opentelnet | open-telnet |
| opencwmp | open-cwmp |
| accessiblevnc | open-vnc |

In the section Postgres databases you can find SQL statements for these changes.

Some feed names have changed, see the comment below in the section Configuration.

Harmonization
You may want to update your harmonization configuration
- Newly added fields:
- `destination.urlpath` and `source.urlpath`.
- `destination.domain_suffix` and `source.domain_suffix`.
- `tlp` with a new type TLP.
- Changed fields:
- ASN fields now have a new type `ASN`.
- Classification:
- New value for `classification.type`: `vulnerable client` with taxonomy `vulnerable`.
- New value for `classification.type`: `infected system` with taxonomy `malicious code` as replacement for `botnet drone`.
- Renamed `JSON` to `JSONDict` and added a new type `JSON`. `JSONDict` saves data internally as JSON, but acts like a dictionary. `JSON` accepts any valid JSON.

Some bots depend on the three new harmonization fields.

Configuration
A new harmonization type `JSONDict` has been added specifically for the `extra` field. It is highly recommended to change the type of this field. The change is backwards compatible and the change is not yet necessary, IntelMQ 1.x.x works with the old configuration too.

The feed names in the shadowserver parser have been adapted to the current subjects. Old subjects will still work in IntelMQ 1.x.x. Change your configuration accordingly:
* `Botnet-Drone-Hadoop` to `Drone`
* `DNS-open-resolvers` to `DNS-Open-Resolvers`
* `Open-NetBIOS` to `Open-NetBIOS-Nameservice`
* `Ssl-Freak-Scan` to `SSL-FREAK-Vulnerable-Servers`
* `Ssl-Scan` to `SSL-POODLE-Vulnerable-Servers`

The Maxmind GeoIP expert did previously always overwrite existing data. A new parameter `overwrite` has been added,
which is by default set to `false` to be consistent with other bots.

The bot `bots.collectors.n6.collector_stomp` has been renamed to the new module `bots.collectors.stomp.collector`. Adapt your `runtime.conf` accordingly.

The parameter `feed` for collectors has been renamed to `name`, as it results in `feed.name`. Backwards compatibility is ensured until 2.0.

Postgres databases
The following statements optionally update existing data.
Please check if you did use these feed names and eventually adapt them for your setup!
SQL
ALTER TABLE events
ADD COLUMN "destination.urlpath" text,
ADD COLUMN "source.urlpath" text;
ALTER TABLE events
ADD COLUMN "destination.domain_suffix" text,
ADD COLUMN "source.domain_suffix" text;
ALTER TABLE events
ADD COLUMN "tlp" text;
UPDATE events
SET "classification.type" = 'infected system'
WHERE "classification.type" = 'botnet drone';
UPDATE events
SET "classification.identifier" = 'open-mdns'
WHERE "classification.identifier" = 'openmdns' AND "feed.name" = 'Open-mDNS';
UPDATE events
SET "classification.identifier" = 'open-chargen'
WHERE "classification.identifier" = 'openchargen' AND "feed.name" = 'Open-Chargen';
UPDATE events
SET "classification.identifier" = 'open-tftp'
WHERE "classification.identifier" = 'opentftp' AND "feed.name" = 'Open-TFTP';
UPDATE events
SET "classification.identifier" = 'open-redis'
WHERE "classification.identifier" = 'openredis' AND "feed.name" = 'Open-Redis';
UPDATE events
SET "classification.identifier" = 'open-ipmi'
WHERE "classification.identifier" = 'openipmi' AND "feed.name" = 'Open-IPMI';
UPDATE events
SET "classification.identifier" = 'open-qotd'
WHERE "classification.identifier" = 'openqotd' AND "feed.name" = 'Open-QOTD';
UPDATE events
SET "classification.identifier" = 'open-snmp'
WHERE "classification.identifier" = 'opensnmp' AND "feed.name" = 'Open-SNMP';
UPDATE events
SET "classification.identifier" = 'open-mssql'
WHERE "classification.identifier" = 'openmssql' AND "feed.name" = 'Open-MSSQL';
UPDATE events
SET "classification.identifier" = 'open-mongodb'
WHERE "classification.identifier" = 'openmongodb' AND "feed.name" = 'Open-MongoDB';
UPDATE events
SET "classification.identifier" = 'open-netbios-nameservice', "feed.name" = 'Open-NetBIOS-Nameservice'
WHERE "classification.identifier" = 'opennetbios' AND "feed.name" = 'Open-NetBIOS';
UPDATE events
SET "classification.identifier" = 'open-elasticsearch'
WHERE "classification.identifier" = 'openelasticsearch' AND "feed.name" = 'Open-Elasticsearch';
UPDATE events
SET "classification.identifier" = 'dns-open-resolver', "feed.name" = 'DNS-Open-Resolvers'
WHERE "classification.identifier" = 'opendns' AND "feed.name" = 'DNS-open-resolvers';
UPDATE events
SET "classification.identifier" = 'ntp-monitor'
WHERE "classification.identifier" = 'openntp' AND "feed.name" = 'NTP-Monitor';
UPDATE events
SET "classification.identifier" = 'ssl-poodle', "feed.name" = 'SSL-POODLE-Vulnerable-Servers'
WHERE "classification.identifier" = 'SSL-Poodle' AND "feed.name" = 'Ssl-Scan';
UPDATE events
SET "classification.identifier" = 'ssl-freak', "feed.name" = 'SSL-FREAK-Vulnerable-Servers'
WHERE "classification.identifier" = 'SSL-FREAK' AND "feed.name" = 'Ssl-Freak-Scan';
UPDATE events
SET "classification.identifier" = 'open-memcached'
WHERE "classification.identifier" = 'openmemcached' AND "feed.name" = 'Open-Memcached';
UPDATE events
SET "classification.identifier" = 'open-xdmcp'
WHERE "classification.identifier" = 'openxdmcp' AND "feed.name" = 'Open-XDMCP';
UPDATE events
SET "classification.identifier" = 'open-natpmp', "protocol.application" = 'natpmp'
WHERE "classification.identifier" = 'opennatpmp' AND "feed.name" = 'Open-NATPMP' AND "protocol.application" = 'nat-pmp';
UPDATE events
SET "classification.identifier" = 'open-netis'
WHERE "classification.identifier" = 'opennetis' AND "feed.name" = 'Open-Netis';
UPDATE events
SET "classification.identifier" = 'ntp-version'
WHERE "classification.identifier" = 'openntpversion' AND "feed.name" = 'NTP-Version';
UPDATE events
SET "classification.identifier" = 'sandbox-url'
WHERE "classification.identifier" = 'sandboxurl' AND "feed.name" = 'Sandbox-URL';
UPDATE events
SET "classification.identifier" = 'spam-url'
WHERE "classification.identifier" = 'spamurl' AND "feed.name" = 'Spam-URL';
UPDATE events
SET "classification.identifier" = 'open-ike'
WHERE "classification.identifier" = 'openike' AND "feed.name" = 'Vulnerable-ISAKMP';
UPDATE events
SET "classification.identifier" = 'open-rdp'
WHERE "classification.identifier" = 'openrdp' AND "feed.name" = 'Accessible-RDP';
UPDATE events
SET "classification.identifier" = 'open-smb'
WHERE "classification.identifier" = 'opensmb' AND "feed.name" = 'Accessible-SMB';
UPDATE events
SET "classification.identifier" = 'open-ldap'
WHERE "classification.identifier" = 'openldap' AND "feed.name" = 'Open-LDAP';
UPDATE events
SET "classification.identifier" = 'blacklisted-ip'
WHERE "classification.identifier" = 'blacklisted' AND "feed.name" = 'Blacklisted-IP';
UPDATE events
SET "classification.identifier" = 'open-telnet'
WHERE "classification.identifier" = 'opentelnet' AND "feed.name" = 'Accessible-Telnet';
UPDATE events
SET "classification.identifier" = 'open-cwmp'
WHERE "classification.identifier" = 'opencwmp' AND "feed.name" = 'Accessbile-CWMP';
UPDATE events
SET "classification.identifier" = 'open-vnc'
WHERE "classification.identifier" = 'accessiblevnc' AND "feed.name" = 'Accessible-VNC';

---------------------------------

Libraries
- Some optional dependencies do not support Python 3.3 anymore. If your are still using this unsupported version consider upgrading. IntelMQ 1.0.x itself is compatible with Python 3.3.

Postgres databases
Use the following statement carefully to upgrade your database.
Adapt your feedname in the query to the one used in your setup.
SQL
UPDATE events
SET "classification.taxonomy" = 'abusive content', "classification.type" = 'spam', "classification.identifier" = 'spamlink', "malware.name" = NULL, "event_description.text" = 'The URL appeared in a spam email sent by extra.spam_ip.', "source.url" = "destination.ip", "destination.ip" = NULL
WHERE "malware.name" = 'l_spamlink' AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "classification.taxonomy" = 'other', "classification.type" = 'other', "classification.identifier" = 'proxyget', "malware.name" = NULL, "event_description.text" = 'The malicious client used a honeypot as proxy.'
WHERE "malware.name" = 'proxyget' AND "feed.name" = 'Spamhaus CERT';

---------------------------------
Postgres databases
Use the following statement carefully to upgrade your database.
Adapt your feedname in the query to the one used in your setup.
SQL
UPDATE events
SET "extra" = json_build_object('source.local_port', "extra"->'destination.local_port')
WHERE "feed.name" = 'Spamhaus CERT' AND "classification.type" = 'brute-force' AND "classification.identifier" = 'telnet';

---------------------------------

Postgres databases
Use the following statement carefully to upgrade your database.
Adapt your feedname in the query to the one used in your setup.
SQL
UPDATE events
SET "classification.taxonomy" = 'intrusion attempts', "classification.type" = 'brute-force', "classification.identifier" = 'rdp', "protocol.application" = 'rdp', "malware.name" = NULL
WHERE "malware.name" = 'iotrdp' AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "classification.taxonomy" = 'vulnerable', "classification.type" = 'vulnerable service', "classification.identifier" = 'openrelay', "protocol.application" = 'smtp', "malware.name" = NULL
WHERE "malware.name" = 'openrelay' AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "protocol.application" = 'portmapper'
WHERE "classification.identifier" = 'openportmapper' AND "feed.name" = 'Open-Portmapper';
UPDATE events
SET "protocol.application" = 'netbios-nameservice'
WHERE "classification.identifier" = 'opennetbios' AND "feed.name" = 'Open-NetBIOS-Nameservice';
UPDATE events
SET "protocol.application" = 'ipsec'
WHERE "classification.identifier" = 'openike' AND "feed.name" = 'Vulnerable-ISAKMP';
UPDATE events
SET "classification.taxonomy" = 'intrusion attempts', "classification.type" = 'brute-force', "classification.identifier" = 'ssh', "malware.name" = NULL, "protocol.application" = 'ssh'
WHERE "malware.name" = 'sshauth' AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "classification.taxonomy" = 'intrusion attempts', "classification.type" = 'brute-force', "classification.identifier" = 'telnet', "malware.name" = NULL, "protocol.application" = 'telnet'
WHERE ("malware.name" = 'telnetauth' OR "malware.name" = 'iotcmd' OR "malware.name" = 'iotuser') AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "classification.taxonomy" = 'information gathering', "classification.type" = 'scanner', "classification.identifier" = 'wordpress-vulnerabilities', "malware.name" = NULL, "event_description.text" = 'scanning for wordpress vulnerabilities', "protocol.application" = 'http'
WHERE "malware.name" = 'wpscanner' AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "classification.taxonomy" = 'information gathering', "classification.type" = 'scanner', "classification.identifier" = 'wordpress-login', "malware.name" = NULL, "event_description.text" = 'scanning for wordpress login pages', "protocol.application" = 'http'
WHERE "malware.name" = 'w_wplogin' AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "classification.taxonomy" = 'intrusion attempts', "classification.type" = 'scanner', "classification.identifier" = 'scanner-generic', "malware.name" = NULL, "event_description.text" = 'infected IoT device scanning for other vulnerable IoT devices'
WHERE "malware.name" = 'iotscan' AND "feed.name" = 'Spamhaus CERT';

---------------------------------

Configuration
- `bots.parsers.cleanmx` removed CSV format support and now only supports XML format. Therefore, CleanMX collectors must define the `http_url` parameter with the feed url which points to XML format. See Feeds.md file on documentation section to get the correct URLs. Also, downloading the data from CleanMX feed can take a while, therefore, CleanMX collectors must overwrite the `http_timeout_sec` parameter with the value `120`.
- The classification mappings for the n6 parser have been corrected:

| n6 classification | Previous classification | | | Current classification | | | Notes |
|-|-|-|-|-|-|-|-|
| | taxonomy | type | identifier | taxonomy | type | identifier |
| dns-query | other | other | ignore me | other | other | dns-query |
| proxy | vulnerable | proxy | open proxy | other | proxy | openproxy |
| sandbox-url | ignore | ignore | ignore me | malicious code | malware | sandboxurl | As this previous taxonomy did not exist, these events have been rejected |
| other | vulnerable | unknown| unknown | other | other | other |

Postgres databases
Use the following statement carefully to upgrade your database.
Adapt your feedname in the query to the one used in your setup.
SQL
UPDATE events
SET "classification.identifier" = 'dns-query'
WHERE "feed.name" = 'n6' AND "classification.taxonomy" = 'other' AND "classification.type" = 'other' AND "classification.identifier" = 'ignore me';
UPDATE events
SET "classification.taxonomy" = 'malicious code' AND "classification.type" = 'malware' AND "classification.identifier" = 'sandboxurl'
WHERE "feed.name" = 'n6' AND "classification.taxonomy" = 'vulnerable' AND "classification.type" = 'ignore' AND "classification.identifier" = 'ignore me';
UPDATE events
SET "classification.taxonomy" = 'other' AND "classification.type" = 'other' AND "classification.identifier" = 'other'
WHERE "feed.name" = 'n6' AND "classification.taxonomy" = 'vulnerable' AND "classification.type" = 'unknow' AND "classification.identifier" = 'unknow';