Msticpy

New Features

We've updated the Kql/Azure Sentinel data provider so that it no longer depends on
IPython/Jupyter functionality to work. This prevented using the provider from
Python scripts and applications. The new implementation uses Kqlmagic as
a standard import and no longer requires IPython magics to work.

08c7cf1Ianhelle/kql as function 2021 08 04 (190) Kqlmagic operations called as func vs. IPython magics
Now using Kqlmagic.kql() function in kql_driver, allowing it to be used from python code.

Fixes

The main fix here was to adapt to changes in the Mordor attack data repository
Mordor is now known as [OTRF Security Datasets](https://github.com/OTRF/Security-Datasets)
(Although it will continue to be known as Mordor in MSTICPy for the time being).

- 17f9704Ianhelle/modor url refix 2021 08 04 (189)
- fabde53fix img src for broken preview images (191)
- 17b9227Pebryan/7 16 21 sent api update (187)* added incident collection
- Updated pandas version to pandas>=1.1.5
- Updated docstrings for Azure and Azure Sentinel APIs

Potentially breaking changes
- MSTICPY now requires pandas 1.1.5 as a minimum

New Features

Timeline duration visualization 176

This new variant of the timeline control allows you to group by one or more columns and view timelines as start/end of activity related to those columns. The graphic below shows durations of communication between different source and destination IPs using a given protocol.

Like other timeline controls it's accessible as a pandas extension.

python
az_net_flows_df.mp_timeline.plot_duration(
group_by=["SrcIP", "DestIP", "L7Protocol"]
)



You can also import and use the plot function directly:

python
from msticpy.nbtools.timeline_duration import display_timeline_duration

display_timeline_duration(
host_logons,
group_by="Account",
ref_events=host_logons.sample(3),
ref_col="TargetUserName",
);


Read about this in our [Event Timeline documentation](https://msticpy.readthedocs.io/en/latest/visualization/EventTimeline.html#timeline-durations)

Fixes

184 MDEFix Update MDE connectors

The Microsoft Defender for Endpoint provider and queries have had a major update, correcting many schema issues (we were still using old column names in a number of cases

185 from microsoft/ianhelle/init_notebook_changes-2021-07-14

- Removed nb_check.py update from azure_ml_tools.py
- Updated text about how to fix missing PyGObject
- Fixed MpConfigEdit widget to show checkbox for making backups prior to updating msticpyconfig
- Fix to remove casefold of OptionButtons value

8707868Fixing Mordor and azure_ml_tools tests
- Fixed Mordor URL for mordor data provider

1ec1a00bypassing GeoIP notebook test because of IPStack intermittent problems (181)

Fixes

788b4ee
- Added new context manager in MsticpyUserError to stop standard display of msticpy user exceptions.
Note this doesn't suppress the exception, just stops exception being output to notebook even if exception is caught.<br>
Example:
python
from msticpy.common.exceptions import MsticpyUserError
with MsticpyUserError.no_display_exceptions():
do stuff that may cause exceptions to be display.



- Fixed typo in path for msticpyconfig.yaml was causing searches to always fail.
- Updated test_nbinit.py - fix one test. Using temp_path instead of tmpdir fixture.

A few hotfixes for bugs

33e9fd4Fixes to data_providers, data view, pivots and nbinit
- Replace list.pop with list.remove in data_providers.py - caused an error when using provider internal query_time control to set query start and end params.
- Add exception catching around Pivot instantiations in pivot_register_reader.py to prevent errors when loading Pivots with some
providers missing configuration settings.
- Prevented push_notebook from being called before displaying control - this was causing an error when first displaying the DataViewer control
- Removed pandas option to return html schema in nbinit.py - Azure ML notebook dataviewer throws a react exception when trying to display DataFrame columns with complex types (e.g. dict or list vs. string/int/etc). Turning this off prevents the AML viewer being used by default.

Highlights

The highlights of this release (which is really 1.2.0 with some last-minute fixes) are: two new data providers for Azure Resource Graph and Sumologic and a DataViewer control for pandas dataframes.

Azure Resource Graph provider

The [Azure Resource graph](https://azure.microsoft.com/en-us/features/resource-graph/)
provider lets you query Azure resources using KQL queries. This works much like our other
query providers and gives you a lot more flexibility in navigating around Azure resources
than the current AzureData provider.

Explore more in the [Resource Graph Driver notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/ResourceGraphDriver.ipynb) and the [online docs](https://msticpy.readthedocs.io/en/latest/data_acquisition/ResourceGraphDriver.html)

Many thanks to Ryan Cobb rcobb-scwx for creating and contributing this.

Sumologic provider

[Sumo Logic Inc. ](https://www.sumologic.com), is a cloud-based machine data analytics service
focusing on security, operations and BI use cases. This provider allows you to connect to
and query your data from MSTICPy via their [Search API](https://help.sumologic.com/APIs/Search-Job-API/About-the-Search-Job-API)
Explore more in the [Sumologic Data Connector notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/Sumologic-DataConnector.ipynb)

Many thanks to Julien (juju4) for building and contributing this.

DataViewer

The data viewer uses the [Bokeh DataTable control[(https://docs.bokeh.org/en/latest/docs/user_guide/interaction/widgets.html#datatable)
to display and browse through data in a pandas DataFrame. It lets you sort by column, choose which columns to display and filter by multiple columns. It keeps a synchronized copy of the DataFrame with column choice and filtering applied, so that you can always access the data as it appears in the control.


Explore more in the [DataViewer notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/DataViewer.ipynb)
and the [Dataview online docs](https://msticpy.readthedocs.io/en/latest/visualization/DataViewer.html)

New Features and Updates

ab6eb73 - (164) DataViewer control

- DataViewer control - Bokeh based data table control
- Documentation in DataViewer.rst
- Notebook in DataViewer.ipynb

3ab31ad- Azure Resource Graph Data Provider (167)

- Co-authored-by: Ryan Cobb <rcobbsecureworks.com>
- Co-authored-by: Pete Bryan <peter.bryanmicrosoft.com>

2a33d61- Sumologic Data Provider (165)

- Co-authored-by: juju4 <juju4users.noreply.github.com>
- Co-authored-by: Ian Hellen <ianhellemicrosoft.com>

4c4b8ca- Ianhelle/getting started fixes 2021 06 04 (170)

- Updates to init_notebook in nb_init.py
- Moved most of the logic of the Azure Sentinel notebooks nb_check script into msticpy - azure_ml_tools
- Will now create msticpyconfig.yaml if it doesn't exist
- Output message from init_notebook when msticpyconfig.yaml is created in nbinit.py

8214c90 - Minor fixes to VT TI Provider and TIBrowser (173)

- Adding QueryTime instance to Dataproviders in data-data_providers.py. Let's you specify time range (or use default 1 day range) for all queries from provider
- Added Browshot to mpconfig_defaults.yaml so that it can be edited in MP settings editor

ab6eb73 (164)

- added recent blog articles and video to a renamed resources page for RTD - blob_articles.

Fixes

8214c90 - Minor fixes to VT TI Provider and TIBrowser (173)

- Updating VT TI Provider for better severity accuracy with additional "detected_" types.
- Changing ti_browser to display "information" results if no "warning" or "high" items in dataset.
- Updated compound_ctrls.py and mp_config_file.py to work when azure-keyvault-secrets not installed

- Update to QueryTime to fix default max time when "day" is used for units.

e6ea232- Pebryan/2021 6 7 grey noise updates (171)

- Update GreyNoise naming and added custom UA

4c4b8ca- Ianhelle/getting started fixes 2021 06 04 (170)

- Fixing a query error in kql_sent_azuresentinel.yaml
- Fixing an issue with contain naming when creating pivot functions
- Fixing link in TIProviders.rst
- Fixing mypy error in azure_auth_core
- Adding check for KV settings before trying to initialize secrets client in provider_settings.py
- Adding function to search for file in utility.py
- mp_config_edit.py:
- Renaming "Save File" button to "Save Settings"
- Add config_filepath parameter to specify file to save when settings not imported from a file
- Changing item update button to "Update" rather than Save in comp_edit.py
- Strip extraneous quotes from string "default" value in mp_config_control.py
- Corrected open params to "w" rather than "w+" (write append)
- Added "browse" as alias for browse_queries in data_providers.py.
- Fixed misreporting of DataFrame column name in exception when column parameter does not match a column in the input data. in pivot_register.py
- Adding azure_ml_tools.py (from nb_check.py script)
- Added "severities='all'" parameter option. Added friendly warning (rather than exception) when no data is present in the input (either empty DF or nothing above the default severities - in ti_browser.py
- Fixing additional areas breaking in timeline plot module if you don't supply "source_columns" parameter.
- Fixing mypy warning in domain_utils.py
- Fixed allowing geolite to create a database folder even if parent folders don't exist.
- Adding "col" and "column" aliases for lookup_iocs in tilookup.py
- Changed score logic for IBM XForce - prevent misreporting of score==1 as warning.
- Updating version in _version.py
- Added test_azure_ml_tools.py
- unit_test_lib updates:
- Added change_directory context manager (with lock) to allow individual test cases to change directory
- Fixed TEST_DATA_PATH to always return absolute path
- Fixing pylint warnings about the lack of use of context managers in mordor_driver.py, base64unpack.py, morph_charts.py
- Pylint warnings suppression about lowercase enums in tilookup.py
- Fixing version number format problem in azure_ml_tools
- Fixing warning that wasn't processing an f-string in tilookup.py

This minor release includes some major updates to the pivot functions.
It also includes the addition of the [Greynoise Community API](https://greynoise.io/)
to our set of Threat Intel providers. Also

**Warning** - this includes some potentially breaking changes since we have
renamed a number of pivot functions to have shorter, friendlier names.
In particular a number of Azure Sentinel query functions have shortened names.

Updates

**Pivot phase 3** with better joins, more pd accessor functions and short/friendly aliases for
pivot functions (especially queries) 140
- Added short names for many queries
- Adding shortcut functions to entities - these allow you to create links from pivot functions in containers
e.g. IpAddress.whois() -> IpAddress.util.whois()
The shortcut methods have the advantage of being easier to see and type in. They
have the further benefit of working as instance methods. So, given an an instance you
can do this:
python
my_ip = IpAddress(Address="157.23.49.1")
my_ip.whois()
previously you'd have to do IpAddress.util.whois(my_ip) - this still works

- Standard pivot functions now also support joins (previously these were implemented only for Data query and TI functions)
- joins now support `join_ignore_case` parameter to [join case-insensitively](https://msticpy.readthedocs.io/en/latest/data_analysis/PivotFunctions.html#joining-input-to-output-data)
- Added [PD accessors](https://msticpy.readthedocs.io/en/latest/data_analysis/PivotFunctions.html#a-pandas-processing-pipeline-with-pivot-functions) for:
- `filter_cols` - column filtering with patterns or lists of columns
- `filter` - string or regex will match against all columns (can also match on partial numbers with the `numeric_col=True` option)
- `sort` - sort on one or more columns or a pattern to match columns
- `list_to_rows` - expand embedded lists in one or more cols to rows
- `parse_json` - turn JSON strings in column into Python objects
- Updates to Pivot docs and notebooks for new pivot behavior
- Added remove_pivot_funcs method to pivot.pivot.py + unit test in test_pivot

Implemented **Greynoise API** as TI Provider
- [Greynoise Community API](https://greynoise.io/)
- [TI Docs](https://msticpy.readthedocs.io/en/latest/data_acquisition/TIProviders.html)

Entities
- Added additional entities from schema: iot_device, mail_cluster, mail_message, mailbox, submission_mail
- Adding str and repr to Edge in entity_graph.py
- Adding (not yet used) graph_property.py to handle automatic creation of edges

Other updates
- Added tilookup functions to allow enabling/disabling individual providers
- Add better timestamp conversion in vtlookup3.py.
- Added check for attribute name similarity in query_container - if trying to access an attribute that doesn't exist it will return likely matches in the exception
- Extracted dup code from pivot_data_queries to pivot_register.
- Updating [blog articles](https://msticpy.readthedocs.io/en/latest/blog_articles.html) links in docs with recent publications
- Capturing output from load_user_defaults in nbinit.py to display as HTML
- Adding find_anomaly_periods function and shorter alias for timeseries_anomalies_stl in timeseries.py


Fixes

- Fixed a problem with logon failure query parameters.
- Minor corrections to queries in kql_sent_az_network and kql_sent_azure
- Fixing path handling in file.py entity
- Fixing test in test_pivot_pd_accessor for renamed parameter.
- Updating notebooks for some pivot function renaming.
- Fix spelling errors in SettingsEditor.rst
- Remove unnecessary list comprehension in test_security_alert.py
- Add tests for list_to_rows and parse_json in pivot_pd_accessor.py - test_pivot_pd_accessor.py
- Fixed some errors in entities with description_str and identity fields
- Add missing FullName property to Host
- Fixed exception in IpAddress
- Fixing some test errors in test_mordor_driver (due to test file race conditions)
- Removing duplicate mp_config file
- Correcting problem with passing namespace argument to notebooklets in user_config.py
- Fixes to analyze_imports.py and import_analyzer.py that prevented use from commandline and use with other packages.
- Fix to list_to_rows pd accessor in pivot_pd_accessor.
- Changed timeline.py so that it doesn't error when no source_columns are supplied
- Fixing bug in calculating period in timespan.py
- Cosmetic fixes in eventcluster.py
- Fixing an issue in ti_browser if passed an empty dataframe.
- Updating requirements files for updated versions of lxml and aiohttp for security issues.