--------------------------------
Requirements
IntelMQ now uses YAML for the runtime configuration and therefore needs the `ruamel.yaml` library.
Configuration
The `defaults.conf` file was removed. Settings that should effect all the bots are not part of the runtime.conf file and are configured in the `global` section in that file.
The `intelmqctl upgrade-config` command migrates the existing values from the `defaults.conf` file to the `runtime.conf` file under the `global` section and then deletes the `defaults.conf` file.
The `pipeline.conf` file was removed. The source- and destination-queues of the bots are now configured in the bot configuration itself, thus in the `runtime.conf` file.
The `intelmqctl upgrade-config` command migrates the existing configuration from the `pipeline.conf` file to the individual bot configurations in the `runtime.conf` configuration file.
The `runtime.conf` file was replaced by a `runtime.yaml` file. IntelMQ moves the file for you if it does not find a runtime.conf but a runtime.yaml file. When IntelMQ changes the file, it now writes YAML syntax.
When using the official deb/rpm-packages or the official Docker image
Unfortunately, the automatic upgrade procedures has a flaw.
The packages provide a default runtime configuration, but only for new installations if there is no previously existing installation.
But as the runtime configuration was renamed from `/etc/intelmq/runtime.conf` to `/etc/intelmq/runtime.yaml`, this check comes to nothing, and the `/etc/intelmq/runtime.yaml` get installed.
But only the new filename is considered by IntelMQ itself, so the configuration *appears* to be lost.
To fix this:
- remove the newly provided `runtime.yaml`
- make sure that the `runtime.conf` is the correct file with your correct configuration
- IntelMQ will rename and convert the configuration automatically, but we need to trigger the migration of the `pipeline.conf` and `defaults.conf`:
sudo -u intelmq intelmqctl upgrade-config -f -u v300_pipeline_file_removal
sudo -u intelmq intelmqctl upgrade-config -f -u v300_defaults_file_removal
sudo -u intelmq intelmqctl upgrade-config -f -u v301_deprecations
Tools
intelmqdump
The command `e` for deleting single entries by given IDs has been merged into the command `d` ("delete"), which can now delete either entries by ID or the whole file.
The command `v` for editing entries has been renamed to `e` ("edit").
Cronjobs
The deprecated shell scripts
- `update-asn-data`
- `update-geoip-data`
- `update-tor-nodes`
- `update-rfiprisk-data`
have been removed in favor of the built-in update-mechanisms (see the bots' documentation). A crontab file for calling all new update command can be found in `contrib/cron-jobs/intelmq-update-database`.
Bots
Both the XMPP collector bot and the XMPP output bot were removed. This [was evaluated on the mailinglist](https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html)
and the XMPP bots were deprecated in 391d625.
Sieve expert
The Sieve expert bot has had major updates to its syntax. Breaking new changes:
* the removal of the `:notcontains` operator, which can be replaced using the newly added
expression negation, e.g `! foo :contains ['.mx', '.zz']` rather than `foo :notcontains ['.mx', '.zz']`.
* changed operators for comparisons against lists of values, e.g `source.ip :in ['127.0.0.5', '192.168.1.2']` rather than `source.ip == ['127.0.0.5', '192.168.1.2']`
The "old" syntax with `==` on lists is no longer valid and raises an error.
New features:
* arbitrary nesting of if clauses + mixed conditionals and actions in the same level of nesting
* new matches on fields containing list values and boolean values
* new list-based actions
* negation of arbitrary expressions and expression groups separated by brackets through a prepended `!`, e.g `! src.port :in [80, 443]`
* non-string values accepted by `add`/`add!`/`update`
The [sieve bot documentation](https://docs.intelmq.org/latest/user/bots/#sieve) has been updated to reflect on these new changes.
Data format
The classification scheme has been updated to better match the [Reference Security Incident Taxonomy (RSIT)](https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/). The following labels were renamed, deleted or merged into others:
| old taxonomy name | old type name | new taxonomy name | new type name |
|-|-|-|-|
| abusive content | | abusive-content | |
| information content security | | information-content-security | |
| information content security | leak | information-content-security | data-leak |
| information content security | dropzone | other | other (identifier: ``dropzone``) |
| intrusion attempts | | intrusion-attempts | |
| information gathering | | information-gathering | |
| intrusions | backdoor | intrusions | system-compromise |
| intrusions | compromised | intrusions | system-compromise |
| intrusions | defacement | information-content-security | unauthorised-information-modification |
| intrusions | unauthorized-login | intrusions | system-compromise |
| intrusions | unauthorized-command | intrusions | system-compromise |
| malicious code | | malicious-code | |
| malicious code | c2server | malicious-code | c2-server |
| malicious code | malware | malicious-code | infected-system / malware-distribution |
| malicious code | dga domain | other | dga-domain |
| malicious code | malware | other | malware |
| malicious code | ransomware | malicious-code | infected-system |
| vulnerable | vulnerable client | vulnerable | vulnerable-system |
| vulnerable | vulnerable service | vulnerable | vulnerable-system |
| other | unknown | other | undetermined |
- For the taxonomy 'availability', the type `misconfiguration` is new.
- For the taxonomy 'intrusions', the type `system-compromise` is new.
- For the taxonomy 'other', the types `malware` and `undetermined` are new.
The old `classification.type` names can still be used in code, and they are automatically converted to the new names.
Existing data in databases and alike are *not* changed automatically.
See the section "Postgres databases" below for instructions to update existing data in databases.
"Malware"
The previously existing classification type "malware" under the taxonomy "malicious code" was removed, as this type does not exist in the RSIT.
Most of the usages were wrong anyway, and should have been infected-device, malware-distribution or something else.
There is only one usage in IntelMQ, which can not be changed.
And that one is really about malware itself (or: the hashes of samples). For this purpose, the new type "malware" under the taxonomy "other" was created, *slightly* deviating from the RSIT in this respect, but the "other" taxonomy can be freely extended.
Removal of deprecated bots and behaviour
- The bot `intelmq.bots.experts.ripencc_abuse_contact.expert` has been removed. It was replaced by `intelmq.bots.experts.ripe.expert` and marked as deprecated in 2.0.0.beta1.
- Modify expert: Compatibility with the deprecated configuration format (before 1.0.0.dev7) was removed.
- RT collector: compatibility with the deprecated parameter `unzip_attachment` (removed in 2.1.0) was removed.
Postgres databases
The following statements optionally update existing data for the harmonization classification changes:
SQL
UPDATE events
SET "classification.taxonomy" = 'abusive-content'
WHERE "classification.taxonomy" = 'abusive content';
UPDATE events
SET "classification.taxonomy" = 'information-content-security'
WHERE "classification.taxonomy" = 'information content security';
UPDATE events
SET "classification.type" = 'data-leak'
WHERE "classification.type" = 'leak' AND "classification.taxonomy" = 'information-content-security';
UPDATE events
SET "classification.taxonomy" = 'intrusion-attempts'
WHERE "classification.taxonomy" = 'intrusion attempts';
UPDATE events
SET "classification.taxonomy" = 'information-gathering'
WHERE "classification.taxonomy" = 'information gathering';
UPDATE events
SET "classification.type" = 'system-compromise'
WHERE "classification.type" IN ('backdoor', 'compromised', 'unauthorized-login', 'unauthorized-command');
UPDATE events
SET "classification.taxonomy" = 'information-content-security', "classification.type" = 'unauthorised-information-modification'
WHERE "classification.taxonomy" = 'intrusions', "classification.type" = 'defacement'
UPDATE events
SET "classification.taxonomy" = 'malicious-code'
WHERE "classification.taxonomy" = 'malicious code';
UPDATE events
SET "classification.type" = 'c2-server'
WHERE "classification.taxonomy" = 'malicious-code' AND "classification.type" = 'c2server';
UPDATE events
SET "classification.taxonomy" = 'other', "classification.type" = 'dga-domain'
WHERE "classification.taxonomy" = 'malicious-code' AND "classification.type" = 'dga domain';
UPDATE events
SET "classification.type" = 'vulnerable-system'
WHERE "classification.taxonomy" = 'vulnerable' AND ("classification.type" = 'vulnerable service' OR "classification.type" = 'vulnerable client');
UPDATE events
SET "classification.type" = 'undetermined'
WHERE "classification.taxonomy" = 'other' AND "classification.type" = 'unknown';
Depending on the data (e.g. feed), the correct statement for the `malware` type deprecation may be either this:
sql
UPDATE events
SET "classification.type" = 'infected-system'
WHERE "classification.taxonomy" = 'malicious-code' AND ("classification.type" = 'malware' OR "classification.type" = 'ransomware');
or this:
sql
UPDATE events
SET "classification.type" = 'malware-distribution'
WHERE "classification.taxonomy" = 'malicious-code' AND ("classification.type" = 'malware' OR "classification.type" = 'ransomware');
or this:
sql
UPDATE events
SET "classification.taxonomy" = 'other'
WHERE "classification.type" = 'malware';