Intelmq

---------------------------------

Bots
Cymru Whois Lookup
The cache key calculation has been fixed. It previously led to duplicate keys for different IP addresses and therefore wrong results in rare cases. The cache key calculation is intentionally not backwards-compatible. Therefore, this bot may take longer processing events than usual after applying this update.
More details can be found in [issue 1592](https://github.com/certtools/intelmq/issues/1592).

Harmonization
Shadowserver Feed/Parser
The feed "Blacklisted-IP" has been renamed by ShadowServer to "Blocklist". In IntelMQ, the old name can still be used in IntelMQ until version 3.0.

----------------------------------

Requirements
MaxMind GeoIP Expert Bot
The current python library versions of geoip (version 4) and maxminddb (version 2) no longer support Python 3.5. Keep older versions of these libraries if you are using this Python version.

Configuration
Abuse.ch URLHaus

The current documented value for the `column` parameter was:
json
['time.source', 'source.url', 'status', 'extra.urlhaus.threat_type', 'source.fqdn', 'source.ip', 'source.asn', 'source.geolocation.cc']

Better is:
json
['time.source', 'source.url', 'status', 'classification.type|__IGNORE__', 'source.fqdn|__IGNORE__', 'source.ip', 'source.asn', 'source.geolocation.cc']

----------------------------------

Requirements
- IntelMQ no longer supports Python 3.4, Python `>=` 3.5 is required.
CentOS 7 (with EPEL) provides both Python 3.4 and Python 3.6. If IntelMQ was installed with Python 3.4, the code needs to be re-installed with Python 3.6 and removed for Python 3.4. Application data is compatible. To install needed packages: `yum install python36 python36-devel python36-requests`.
- The *AMQP collector* requires the `pika` library minimum version 1.0.

Configuration

ElasticSearch Output Bot
The ElasticSearch Output bot does no longer support (only) ElasticSearch version 5, but only version 7 (1513).

Microsoft Azure Collector Bot
The Bot has been majorly changed to support the current Azure Python library `azure-storage-blob>=12.0.0`.
This also changes the required configuration parameters. The new required parameters are:

* `redis_cache_db`: 5
* `redis_cache_host`: `"127.0.0.1"`
* `redis_cache_password`: null, depending on your Redis server configuration
* `redis_cache_port`: 6379, depending on your Redis server configuration
* `redis_cache_ttl`: 864000 (10 days), depending on how old fast the data you are fetching is removed from the storage
* `connection_string`: connection string as given by Microsoft, includes endpoint and authentication information
* `container_name`: name of the container to connect to

The previous parameters `account_name`, `account_key` and `delete` are not supported anymore.

URLVir Feeds and Parser
All URLVir feeds have been discontinued. The URLVir Parser has been removed.
The `intelmqctl upgrade-config` command warns if you have these feed and the bot in use.

---------------------------------

Requirements
The python library `requests` is required by the core.

Harmonization
The regular expression of the field `protocol.transport` has been updated to accommodate the value `nvp-ii`.
`intelmqctl upgrade-config` handles the change to automatically upgrade your configuration.

Taichung feed
The Taichung feed "Netflow (Recent 30)" with URL `https://www.tc.edu.tw/net/netflow/lkout/recent/30` is no longer available and gives an error 500.
As a drop-in replacement the Parser as well as the Feed documentation are now adapted to the full feed available at `https://www.tc.edu.tw/net/netflow/lkout/recent/`.
The `intelmqctl upgrade-config` command takes care of this change.

Abuse.ch Zeus Tracker Feed
The Abuse.ch Zeus Tracker has been discontinued on 2019-07-08. The `intelmqctl upgrade-config` command warns if you have this feed in use.

Abuse.ch Ransomware Tracker Feed
The Abuse.ch Ransomware Tracker has been discontinued on 2019-12-08. The `intelmqctl upgrade-config` command warns if you have this feed in use.

Bitcash.cz Feed
The Bitcash.cz Banned IPs / Blocklist feed previously available under `https://bitcash.cz/misc/log/blacklist` is no longer available. The `intelmqctl upgrade-config` command warns if you have this feed in use.

Fraunhofer DDoS Attack Feed
The Fraunhofer DDoS Attack feed previously available under `https://feed.caad.fkie.fraunhofer.de/ddosattackfeed/` is no longer available. The `intelmqctl upgrade-config` command warns if you have this feed in use.

Bambenek Feeds
Many Bambenek feeds require a license now and URLs have changed. See https://osint.bambenekconsulting.com/feeds/ for more information. The `intelmqctl upgrade-config` command also warns if you have previously documents feeds in use and migrates the URL for the DGA domain feed.

Nothink Feeds and Parser
All Nothink Honeypot feeds have been discontinued and current the data available covers the time until 2019. The Nothink Parser has been removed.
The `intelmqctl upgrade-config` command warns if you have these feed and the bot in use.

---------------------------------

MaxMind GeoIP
MaxMind requires a registration before being able to download the GeoLite2 database starting with 2019-12-30: https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/
If the provided `update-geoip-data` script is used, the license key can now be set second parameter.

Libraries
When using MISP integrations, make sure your currently installed PyMISP version supports the installed Python version. Any PyMISP version newer than 2.4.119.1 requires Python 3.6 or newer.

---------------------------------

Tools
`intelmqctl check` and `intelmqctl upgrade-config` checks/applies a generic harmonization upgrade, checking for it's completeness.

Harmonization
In version 2.1.0 a new field `extra` has been added to *Reports*. You need to add this to the *report* section in your harmonization configuration, or run `intelmqctl upgrade-config`:
json
"extra": {
"description": "All anecdotal information of the report, which cannot be parsed into the data harmonization elements. E.g. subject of mails, etc. This is data is not automatically propagated to the events.",
"type": "JSONDict"
},


Configuration
Defaults
For AMQP brokers, the port 15671 was used as default for connecting to the management interface. But RabbitMQ's default is 15672. This was corrected in this version. If you changed the port in RabbitMQ or IntelMQ (`intelmqctl_rabbitmq_monitoring_url`), the settings needs to be adapted.