Msticpy

New Features

A notebook containing some of the features of MSTICPy 2.0
is available at [What's new in MSTICPy 2.0](https://github.com/microsoft/msticpy/blob/main/docs/notebooks/What's%20New%20in%20MSTICPy%202.0.ipynb)

Dropping Python 3.6 Support

As of this release we only officially support Python 3.8 and above.
We will try to support Python 3.6 if the fixes required are small
and contained but make no guarantees of it working completely on
Python prior to 3.8

DataFrame to Graph/Network

You can convert a pandas DataFrame into a NetworkX graph or
plot directly as a graph using Bokeh interactive plotting.

You pass the functions the column names for the source and target nodes to build a basic graph. You can also name other columns to be node or edge attributes. When displayed these attributes are visible as popup details courtesy of Bokeh’s Hover tool.

python
proc_df.head(100).mp_plot.network(
source_col="SubjectUserName",
target_col="Process",
source_attrs=["SubjectDomainName", "SubjectLogonId"],
target_attrs=["NewProcessName", "ParentProcessName", "CommandLine"],
edge_attrs=["TimeGenerated"],
)


Pivots without initialization/dynamic data query import

The pivot functionality has been overhauled - it is now initialized
automatically in `init_notebook`.

Previously queries from
data providers were added at initialization - meaning that you had
to create your query providers before starting pivot or re-initialize
pivot. Data providers now dynamically add relevant queries as pivot
functions when you authenticate. Also for some providers, such
as Azure Sentinel, that support multiple instances, pivot now
supports separate instance naming so that each Workspace has a
separate instance of a given pivot query.

The naming of the Threat Intelligence pivot functions has been
simplified considerably.

VirusTotal and RiskIQ relationships should now be available as
pivot functions (you need the VT 3 and PassiveTotal packages installed
respectively to enable this functionality).

Simplify imports in msticpy

The root module in msticpy now has several modules and
classes that can be directly accessed from it (rather than
having to import them)

python
import msticpy as mp
mp.init_notebook()
qry_prov = mp.QueryProvider("MDE")
ti = mp.TILookup()


Also a number of commonly-used classes are imported by default
by `init_notebook`, notable all of the entity classes.
This makes it easier to use pivot functions without any initialization
or import steps.

- entities
python
import msticpy as mp
mp.init_notebook()

IpAddress.whois("123.45.6.78")


Consolidation of Pandas accessors

Pandas accessors are extensions to DataFrames allowing you to
call custom functionality as a DataFrame method.

Almost all of the core MSTICPy functions previously available in
various accessors (plus a few new ones) are accessible in:

- df.mp - analysis and transform functions
- df.mp_plot - visualization functions

python3

df.mp.ioc_extract(...)
df.mp.to_graph(...)
df.mp.mask(...)

df.mp_plot.timeline(...)
df.mp_plot.timeline_values(...)
df.mp_plot.process_tree(...)
df.mp_plot.network(...)
df.mp_plot.folium_map(...)


MS Defender Queries available to MS Sentinel Query Provider

Since Sentinel now has the ability to import Microsoft data, we've
made the Defender queries usable from the MS Sentinel provider.

Many of these queries are now available as Pivot functions.

ContiLeaks notebook added to MSTICPy Repo

We are privileged to host Thomas's awesome ContiLeaks notebook.
Thanks fr0gger

New Queries added

Several new Sentinel and MS Defender queries have been added.

See the new [built-in query list](https://msticpy.readthedocs.io/data_acquisition/DataQueries.html)

Documentation Additions and Updates

The documentation for V2.0 is now live and available at https://msticpy.readthedocs.io
(Previous versions are still online and can be accessed through
the ReadTheDocs interface).

- New [MSTICPy Quickstart Guide](https://msticpy.readthedocs.io/getting_started/QuickStart.html)
- Updated [Installing guide](https://msticpy.readthedocs.io/getting_started/Installing.html)
- Updated [Threat Intel Lookup documentation](https://msticpy.readthedocs.io/data_acquisition/TIProviders.html)
- Updated [Time Series analysis documentation](https://msticpy.readthedocs.io/visualization/TimeSeriesAnomalies.html)
- New [Plot Network Graph from DataFrame](https://msticpy.readthedocs.io/visualization/NetworkGraph.html)
- Updated [Plotting Folium maps](https://msticpy.readthedocs.io/visualization/FoliumMap.html)
- Updated [Pivot functions](https://msticpy.readthedocs.io/data_analysis/PivotFunctions.html)
- Updated [Jupyter and Sentinel](https://msticpy.readthedocs.io/getting_started/JupyterAndAzureSentinel.html)

The API documentation has been split into separate modules to
make it easier to navigate. The API docs also now support "InterSphinx".
This means that MSTICPy references to objects in other packages (e.g. Python
standard library, pandas, Bokeh) have active links that will take you
to the native documentation for that item.

Also, the sample notebooks for most of these features have been updated
along the same lines. See [MSTICPy Sample notebooks](https://github.com/microsoft/msticpy/tree/main/docs/notebooks)

Miscellaneous Improvements

- The MS Sentinel provider now support a timeout parameter allowing you
lengthen and shorten the default.
- MSTICPy network requests use a custom User Agent header so that you
can identify or track requests from MSTICPy/Notebooks.

Plus a lot more that I can't recall at the moment.

What's Changed - The gory detail of the PRs

* Sync changes to main into v2 branch by ianhelle in https://github.com/microsoft/msticpy/pull/330
* Ianhelle/msticpy v2.0.0 merge updates 2022 03 14 by ianhelle in https://github.com/microsoft/msticpy/pull/338
* Ianhelle/implement isort 2022 02 15 by ianhelle in https://github.com/microsoft/msticpy/pull/327
* Ianhelle/implement isort branch post-fixes 2022 03 21 by ianhelle in https://github.com/microsoft/msticpy/pull/346
* Ianhelle/pivot dataprov selfload 2022 03 15 by ianhelle in https://github.com/microsoft/msticpy/pull/343
* Ianhelle/main mergeback 2022 04 05 by ianhelle in https://github.com/microsoft/msticpy/pull/355
* Merging changes from main for geoip.py, config editor and kusto_driver by ianhelle in https://github.com/microsoft/msticpy/pull/359
* Pebryan/2022 4 14 auth merge by petebryan in https://github.com/microsoft/msticpy/pull/368
* Fixed minor issues by petebryan in https://github.com/microsoft/msticpy/pull/372
* Ianhelle/v2 reorg directories 2 2022 04 12 by ianhelle in https://github.com/microsoft/msticpy/pull/377
* Ianhelle/mpconfigedit fix from main 2022 05 22 by ianhelle in https://github.com/microsoft/msticpy/pull/396
* Added pd accessor for time series functions. by ianhelle in https://github.com/microsoft/msticpy/pull/381
* Added new Sentinel Search Features - merge from main by ianhelle in https://github.com/microsoft/msticpy/pull/380
* Ianhelle/ti async lookup 2022 04 27 by ianhelle in https://github.com/microsoft/msticpy/pull/383
* Ianhelle/folium accessor 2022 04 30 by ianhelle in https://github.com/microsoft/msticpy/pull/384
* Updated tweet action to include more details by petebryan in https://github.com/microsoft/msticpy/pull/406
* Add Device Code fallback option for when interactive auth isn't avaliable. by petebryan in https://github.com/microsoft/msticpy/pull/401
* Adding OData Delegated Auth Support into 2.0 by petebryan in https://github.com/microsoft/msticpy/pull/410
* Removed plaintext token cache from MSAL auth and replaced it with fall back to in memory caching by petebryan in https://github.com/microsoft/msticpy/pull/414
* Ianhelle/kql nbinit fixes merge2.0 2022 05 18 by ianhelle in https://github.com/microsoft/msticpy/pull/412
* Ianhelle/geoip init fix 2022 05 27 by ianhelle in https://github.com/microsoft/msticpy/pull/421
* Ianhelle/geoip init fix 2022 05 27 by ianhelle in https://github.com/microsoft/msticpy/pull/422
* Ianhelle/geoip init fix 2022 05 27 by ianhelle in https://github.com/microsoft/msticpy/pull/423
* Ianhelle/read the docs fixes 2022 05 29 by ianhelle in https://github.com/microsoft/msticpy/pull/424
* Ianhelle/sentinel workspace lookup 2022 05 19 by ianhelle in https://github.com/microsoft/msticpy/pull/419
* Fix for list_hunting_queries function by pensivepaddle in https://github.com/microsoft/msticpy/pull/417
* Update calls to credential.modern.get_token by FlorianBracq in https://github.com/microsoft/msticpy/pull/429
* Adding ContiLeaks Analysis by fr0gger in https://github.com/microsoft/msticpy/pull/428
* Networkx graphs from dataframe by ianhelle in https://github.com/microsoft/msticpy/pull/427
* Ianhelle/msticpy __init__ imports and Quickstart doc by ianhelle in https://github.com/microsoft/msticpy/pull/435
* Ianhelle/main updates to msticpy v2.0.0 2022 06 14 by ianhelle in https://github.com/microsoft/msticpy/pull/444
* [fix] Revert to Py 3.7 build with typing-extensions by ianhelle in https://github.com/microsoft/msticpy/pull/448
* [fix] if AuthKey or ApiID is None by ianhelle in https://github.com/microsoft/msticpy/pull/449
* Ianhelle/query pivot naming 2022 06 06 by ianhelle in https://github.com/microsoft/msticpy/pull/437
* Ianhelle/folium update docs 2022 05 29 by ianhelle in https://github.com/microsoft/msticpy/pull/438
* Ianhelle/timeline updates 2022 06 14 by ianhelle in https://github.com/microsoft/msticpy/pull/441
* Ianhelle/merge2.0 to main 2022 06 14 by ianhelle in https://github.com/microsoft/msticpy/pull/443
* [fix] Fixing renamed y/value_col parameter by ianhelle in https://github.com/microsoft/msticpy/pull/452
* [fix] Incorrect handling of kwargs in timeline_values.py by ianhelle in https://github.com/microsoft/msticpy/pull/455
* Ianhelle/ti and graph doc update 2022 06 15 by ianhelle in https://github.com/microsoft/msticpy/pull/453
* Ianhelle/mp user agent 2022 06 16 by ianhelle in https://github.com/microsoft/msticpy/pull/454
* Added features to support new notebooklet development. by petebryan in https://github.com/microsoft/msticpy/pull/456

New Contributors
* pensivepaddle made their first contribution in https://github.com/microsoft/msticpy/pull/417
* fr0gger made their first contribution in https://github.com/microsoft/msticpy/pull/428

**Full Changelog**: https://github.com/microsoft/msticpy/compare/v1.8.2...v2.0.0.rc3

New Features

There are several new features in V 2.0.0 of MSTICPy. The major
items include:

- Folium map update - plot a map using multiple layers, custom
icons, colors and tooltips from a single function call.
- Time Series - calculate and display a Time Series anomalies
plot from a single function call.
- Threat Intelligence lookups - individual providers run asynchronously
(simultaneously) making it many times faster to perform lookups
across providers. Lookup progress is also displayed with a progress
bar

Pre-release documentation for v2.0.0 is on [ReadtheDocs](https://msticpy.readthedocs.io/release-msticpy-v2.0.0)
Note: API documentation should be up-to-date but user-guides for new features
are still TBD.

Folium map update

The Folium module in MSTICPy has always been a bit complex to use
since it normally required that you convert IP addresses to MSTICPy
IpAddress entities before adding them to the map. You can now
plot maps with a single function call from a DataFrame containing
IP addresses or location coordinates. You can group the data
into folium layers, specify columns to populate popups and tooltips
and to customize the icons and coloring.


plot_map

A new `plot_map` function (in the msticpy.vis.foliummap module) that
lets you plot mapping points directly from a DataFrame. You can
specify either an `ip_column` or coordinates columns (`lat_column` and
`long_column`). In the former case, the geo location of the IP address
is looked up using the MaxMind GeoLiteLookup data.

You can also control the icons used for each marker with the
`icon_column` parameters. If you happen to have a column in your
data that contains names of FontAwesome or GlyphIcons icons
you can use that column directly.
More typically you would combine the `icon_column` with the
`icon_map` parameter. You can specify either a dictionary or a
function. For a dictionary, the value of the row in `icon_column`
is used as a key - the value is a dictionary of icon parameters
passed to the Folium.Icon class. For a function, the `icon_column`
value is passed to the function as a single parameter and the return value
should be a dictionary of valid parameters for the `Icon` class.
You can read the documentation for this function in the
[docs](https://msticpy.readthedocs.io/release-msticpy-v2.0.0/api/msticpy.vis.foliummap.html)

plot_map pandas accessor

Plot maps from the comfort of your own DataFrame!
Using the msticpy `mp_plot` accessor you can plot maps directly
from a DataFrame containing IP or location information.
The `folium_map` function has the same syntax as `plot_map`
except that you omit the `data` parameter.

python

df.mp_plot.folium_map(ip_column="ip", layer_column="CountryName")





Layering, Tooltips and Clustering support

In `plot_map` and `.mp_plot.folium_map` you can specify
a `layer_column` parameter. This will group the data
by the values in that column and create an
individually selectable/displayable layer in Folium. For performance
and sanity reasons this should be a column with a relatively
small number of discrete values.

Clustering of markers in the same layer is also implemented by
default - this will collapse multiple closely located markers
into a cluster that you can expand by clicking or zooming.

You can also populate tooltips and popups with values
from one or more column names.

Classic interface
The original FoliumMap class is still there for more manual
control. This has also been
enhanced to support direct plotting from IP, coordiates or GeoHash
in addition to the existing IpAddress and GeoLocation entities.
It also supports layering and clustering.

Threat Intelligence Providers - Async support

When you have configured more than one TI provider, MSTICPy will
execute requests to each of them asynchronously. This will bring big
performance benefits when querying IoCs from multiple providers.
Note: requests to individual providers are still executed synchronously
since we want to avoid swamping provider services with multiple
simultaneous requests.

We've also implemented progress bar tracking for TILookups, giving a visual
indication of progress when querying multiple IoCs.

Combining the progress tracking with asynchronous operation means
that not only is performing lookups for lots of observables faster
but ou will also less likely to be left guessing whether or not your kernel
has hung.

TI Providers are now also loaded on demand - i.e. only when you have
a configuration entry in your msticpyconfig.yaml for that provider.
This prevents loading of code (and possibly import errors) due to providers
which you are not intending to use.

Finally, we've added functions to enable and disable providers
after loading TILookup:
- [ti_lookup.enable_provider](https://msticpy.readthedocs.io/release-msticpy-v2.0.0/api/msticpy.context.tilookup.html#msticpy.context.tilookup.TILookup.enable_provider)
- [ti_lookup.disable_provider](https://msticpy.readthedocs.io/release-msticpy-v2.0.0/api/msticpy.context.tilookup.html#msticpy.context.tilookup.TILookup.disable_provider)
- [ti_lookup.set_provider_state](https://msticpy.readthedocs.io/release-msticpy-v2.0.0/api/msticpy.context.tilookup.html#msticpy.context.tilookup.TILookup.set_provider_state)

python

from msticpy.context import TILookup
ti_lookup = TILookup()

iocs = ['162.244.80.235', '185.141.63.120', '82.118.21.1', '85.93.88.165']
ti_lookup.lookup_iocs(iocs, providers=["OTX", "RiskIQ"])





Time Series pandas accessor

Although the Time Series functionality was relatively simple to
use, it previously required several disconnected steps to compute
the time series, plot the data, extract the anomaly periods. Each of
these needed a separate function import. Now you can do all of these
from a DataFrame via pandas accessors.
(currently there is a separate accessor `df.mp_timeseries` but we are
still working on consolidating our pandas accessors so this may change
before the final release.)

Because you typically still need these separate outputs, the accessor
has multiple methods:

- `df.mp_timeseries.analyze` - takes a time-summarized DataFrame
and returns the results of a time-series decomposition
- `df.mp_timeseries.plot` - takes a decomposed time-series and
plots the anomalies
- `df.mp_timeseries.anomaly_periods` - extracts anomaly periods
as a list of time ranges
- `df.mp_timeseries.anomaly_periods` - extracts anomaly periods
as a list of KQL query clauses
- `df.mp_timeseries.apply_threshold` - applies a new anomaly
threshold score and returns the results.

[See documentation](https://msticpy.readthedocs.io/en/release-msticpy-v2.0.0/api/msticpy.analysis.timeseries.html#msticpy.analysis.timeseries.MsticpyTimeSeriesAccessor)

Analyze data to produce time series.

python

df = qry_prov.get_networkbytes_per_hour(...)
ts_data = df.mp_timeseries.analyze()


Analyze and plot time series anomalies

python

df = qry_prov.get_networkbytes_per_hour(...)
ts_data = df.mp_timeseries.analyze().mp_timeseries.plot()



Analyze and retrieve anomaly time ranges

python

df = qry_prov.get_networkbytes_per_hour(...)
ts_data = df.mp_timeseries.analyze().mp_timeseries.anomaly_periods()



In next pre-release

Plot networks (graphs) directly from a DataFrame

One frequently-requested feature is the ability to easily plot
networks from data. For example you may want to view the interactions
between account names and IP addresses. This feature use
[Networkx](https://networkx.org) to build the graph and
[Bokeh](https://docs.bokeh.org/) to plot the graph.

> Note: The graph has the usual Bokeh interactivity - zoomin, panning, selecting,
> hover-over tooltips. It does not allow you to move individual
> nodes and interactively recalculate the layout. For the
> latter, you can use this functionality to build a networkx graph
> and plot using something like GraphViz or PyViz.

The network plot will give you two functions:

- `df.mp.to_graph` to convert a DataFrame to a networkx graph
- `df.mp_plot.network` create and plot the graph in a single step.

(There is also a separate function `msticpy.vis.network_plot.plot_nx_graph`
that will just do the NX -> plot operation)

You can specify the columns to use as source and target. An edge
is created between source and target when the two occur on
in the same row (or more than one row). You can also
specify columns to use as node and edge attributes.

To Do items

We intend to add the following before release:

- allow you to specify the networkx layout algorithm to use
(currently it uses the default `spring_layout`)
- assign edge `weight` attribute based on number of rows contributing
to an edge

MS Sentinel Workspaces API

Lets you query and resolve details for Sentinel workspaces.
This is integrated into the MpConfigEdit and MpConfigFile utilities
to let you lookup workspace details when you are editing your
settings:

- paste in a URL from the Sentinel Azure portal to populate workspace settings
- or resolve full details from partial workspace such as the workspace ID.

Other important fixes

The API details for most of the MSTICPy functions were not being
generated - this should now be fixed.


What's Changed (GitHub PR Summary)
* Added pd accessor for time series functions. by ianhelle in https://github.com/microsoft/msticpy/pull/381
* Added new Sentinel Search Features - merge from main by ianhelle in https://github.com/microsoft/msticpy/pull/380
* Ianhelle/ti async lookup 2022 04 27 by ianhelle in https://github.com/microsoft/msticpy/pull/383
* Ianhelle/folium accessor 2022 04 30 by ianhelle in https://github.com/microsoft/msticpy/pull/384
* Updated tweet action to include more details by petebryan in https://github.com/microsoft/msticpy/pull/406
* Add Device Code fallback option for when interactive auth isn't available. by petebryan in https://github.com/microsoft/msticpy/pull/401
* Adding OData Delegated Auth Support into 2.0 by petebryan in https://github.com/microsoft/msticpy/pull/410
* Removed plaintext token cache from MSAL auth and replaced it with fall back to in memory caching by petebryan in https://github.com/microsoft/msticpy/pull/414
* Ianhelle/kql nbinit fixes merge2.0 2022 05 18 by ianhelle in https://github.com/microsoft/msticpy/pull/412
* Ianhelle/geoip init fix 2022 05 27 by ianhelle in https://github.com/microsoft/msticpy/pull/421
* Ianhelle/geoip init fix 2022 05 27 by ianhelle in https://github.com/microsoft/msticpy/pull/422
* Ianhelle/geoip init fix 2022 05 27 by ianhelle in https://github.com/microsoft/msticpy/pull/423


**Full Changelog**: https://github.com/microsoft/msticpy/compare/v2.0.0.rc1...v2.0.0.rc2

One of our main goals for V2.0.0 was to re-organize MSTICPy to be more logical and easier to
use and maintain. Several years of organic growth had seen modules created in places that
seemed like a good idea at the time but did not age well.

The discussion about the V2 structure can be found here 320.

This first pre-release of V2.0 has a small number of new features but is mostly
about testing compatibility. We'll be adding features to future pre-releases in
the next week or two.

**Due to the re-organization, many features are no longer in places where they used
to be imported from! **

We have tried to maintain compatibility with old locations by adding "glue" modules.
These allow import of many modules from their previous locations but will issue a Deprecation warning if loaded from here.
The warning will contain the new location of the module - so you should update your code to point to this new location.

This table gives a quick overview of the V2.0 structure

| folder | description |
|-----------|----------------------------------------------------------------------------------|
| analysis | data analysis functions - timeseries, anomalies, clustering |
| auth | authentication and secrets management |
| common | common used utilities and definitions (e.g. exceptions) |
| config | configuration and settings UI |
| context | enrichment modules geoip, ip_utils, domaintools, tiproviders, vtlookup |
| data | data acquisition/queries/storage/uploaders |
| datamodel | entities, soc objects |
| init | package loading and initialization - nbinit, pivot modules |
| nbwidgets | nb widgets modules |
| transform | simple data processing - decoding, reformatting, schema change, process tree |
| vis | visualization modules including browsers |


Notable things that have moved:

* most things from the sectools folder have migrated to context, transform or analysis
* most things from the nbtools folder have migrated to:
* init - (not to be confused with __init__) - package initialization
* vis - visualization modules
* pivot functionality has moved to init

If you are having trouble finding a module, we have added a simple search function

python
import msticpy
msticpy.search("riskiq")


Any matches will be returned in a table with links to the module documentation

---
<h4>Modules matching 'riskiq'</h4>
<table class='table_mod'>
<tr class='cell_mod'><th>Module</th><th>Help</th></tr>
<tr class='cell_mod'><td>msticpy.context.tiproviders.riskiq</td><td>
<a href={https://msticpy.readthedocs.io/en/latest/api/msticpy.sectools.tiproviders.html#msticpy-sectools-tiproviders-riskiq-module} target='_blank'>msticpy.context.tiproviders.riskiq</a></td>
</tr>
</table>

---

The Help link is to the current release version of the documentation, so will not
map to the correct path on ReleaseTheDocs for the release documentation. You should be able to
map this to the pre-release doc version by adding v2.0.0.rc1 into the URL.

https://msticpy.readthedocs.io/en/v2.0.0rc1/api/msticpy.sectools.tiproviders.html

Other Features/Changes

Pandas accessors have been consolidated into two main ones:

* df.mp.*func* - contains all of the transformation functions like base64 decoding, ioc searching, etc.
* df.mp_plot.*func* - contains all of the visualization accessors (timeline, process tree, etc.)

We've also done a lot of work to clean up warnings, move some remaining Python unittest tests to pytest
format and random other things that you do when you're cleaning house. A fuller but
not especially readable list of changes follows.

Please let us know if you hit any issues with the re-jigged structure
or any other bugs. File them as issues or reach out to msticpymicrosoft.com

List of changes


Co-authored-by: Ian Hellen <ianhellemicrosoft.com>
Co-authored-by: Pete Bryan <pebryanmicrosoft.com>

* 857835dIanhelle/mpconfigedit fix from main 2022 05 22 (396)* Fix for MpConfigEdit ValueError
* Updating Dockerfile source to mcr anaconda
* bd96d40Ianhelle/v2 reorg directories 2 2022 04 12 (377)* Merging changes from main for geoip.py, mp_config_edit, mp_config_file, pkg_config and kusto_driver
* Some fixes to Kusto common_imports
* now works with Kusto config entry without instance suffix
* can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query
* added explicit "database" key in query files - can be used instead of the more opaque "data_family.database" encoding
in the data_families key.
* Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration
* Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py.
* Fixed bug in mp_config_edit.py and mp_config_file.py where empty/new msticpyconfig.yaml didn't save any settings.
* Reorganized logic for handling parameters and failing on invalid file path in config module.
* Moved analysis.data to transform folder
* Moved data.context to context
* Moved auth and secrets modules to auth folder
* Moved nbwidgets to new folder
* Updated deprecation warning in glue modules to v2.2
* Fix to URLs in README.md
* Adding init folder - moved:
* nbinit.py, user_config.py, pivot.py, azure_ml_tools.py to here
* also moved vt_pivot.py and pivot_ti_provider.py to init/pivot_init
* Renaming datamodel/pivots to datamodel/pivot
* Moved azure_blob_storage.py to data/storage folder
* Refactored query_container to data_types.py - to be separate types for queries (query_container.py) and pivots (pivot_container.py)
* Moved browsers to vis folder
* Updating API docs for moves
* Moved all pivot functions to init folder.
* Added functionality to pkg_config to delete and translate settings (for AzureSentinel->MSSentinel switch TBD)
* Added automatic acquisition of globals() in nbinit.py
* Fixed a couple of bugs in pivot_pipeline.py
* Removed direct import of pivot into datamodel/pivot and added code to add them dynamically after init.pivot initiialization.
* Added trap to timeline when supplied with no data.
* Fixed incorrect escaping in regex in kql_driver.py
* Notebook updates for errors and invalid links.
* Added script to run all notebooks for testing
* Changing the pattern for httpx timeout to default to Timeout(None). This can be overridden in settings and in the case of drivers and TILookup in runtime parameter (timeout=x). Other components use the default.
* Updated typing rigor for pkg_config::get_http_timeout
* Fixed bugs in test test_pkg_config.py and test_code_view.py
* Fixed re-auth on query issue in KQL driver
* Fixed kql_driver tests
* Kusto provider
* now works with Kusto config entry without instance suffix
* can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query
* added explicit "database" key in query files - can be used instead of the more opaque "data_family.database" encoding
in the data_famiies key.
* Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration
* Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py.
* Fixed bug in mp_config_edit.py and mp_config_file.py where empty/new msticpyconfig.yaml didn't save any settings.
* Reorganized logic for handling parameters and failing on invalid file path in config module.
* e2e743aIanhelle/main mergeback 2022 04 05 (355)* Updated nbwidgets - GetText, QueryTime, GetEnvironmentKey to work with notebook parameters.
* Fixed query_time widget so that you can reset time range from parameter
* Added additional unit test for QueryTime setter
* Simplified SelectAlert (in select_alert and nbdisplay) to remove title line. this was not updating so every alert selected would add another titlel line.
* Changed structure and formatting of alert item display - removing CompromisedEntity and adding ProductName.
* Updated TimeSpan class so it has more flexible constructor
* Added account_id as a parameter for list_aad_signins_by_account query
* Fix for kql_driver - reconnecting for each query loses original kwargs (including mp_az_auth) setting, so reverts
to defaults. This can cause errors if the defaults are different to user-specified parameters.
There is also a problem in azure_auth.py - if a user has AzureCLI settings, these override everything. I've removed this since we don't really want people configuring auth methods from these settings.

* aa14784moved list_sentinel_workspaces to AzureData
* f3e82d9New MSAL delegated auth methods added and support for this added to Graph providers. Added ability to pass tenant ID to KQL provider fixing issue 333. Minor fixes added incl merge from 352.
* ba33c38Ianhelle/pivot dataprov selfload 2022 03 15 (343)* Query providers load pivots dynamically when created.
* Renamed query paths and changed data_providers so that only env-specific queries are loaded.
* Moving ensure_df_datetimes to common/data_utils.py to avoid circular imports
* Consolidated data-related pandas accessors into single module.
* Fixing circular dependency in iocextract
* Fixing linting errors in data_providers.py, azure_resource.py, host.py, process.py, pivot_data_queries.py
* Adding default timeout values to httpx calls.
* Changing tor_exit_nodes.py Tor provider to defer download of tor list until first lookup
* Fixing test for trigger Tor node download before running test.
* Fixing case where MpConfigEdit loads with no current msticpyconfig.yaml. (from main branch)
* Now loads with empty settings rather than throw exception.
* Added unit test case
* Aligning splunk_uploader params with base class
* 243f196Ianhelle/implement isort branch post-fixes 2022 03 21 (346)* move query files
* Stub files and restructure fixes
* Merging in updates to Azure auth
* Making tests more resilient for multiple environments
* restructure cyberreason and splunk queries
* Fixed incorrect vtlookup
* Updating missed Conda version for respx
* Updated cybereason folder names
* Fixed Sentinel APIs
* Updated test mocked data to match new Sentinel APIs
* Renamed data.context_providers to context
* Renamed data.common to data.core
* Added docstrings to redirection files so that they link to right location in read-the-docs
* Updated docs with new paths
* Updated notebooks with new module paths
* Changed RTD to generate an API page for each module.
* Add text to deprecation warning that we'll remove in v2.0.0
* Adding isort to pre-commit.yaml
* Isorting all of the imports in msticpy, tests and tools
* Renaming query folders
* Removing some automatic imports from msticpy.__init__.py
* Adding search function to find modules to utility.py
* Fixing old paths in test_timeline.py
* Adding triggers for release branches
* Adding isort to requirements-dev and conda-reqs-dev.txt
* Bandit FPs in anomaly sequence modules
* Fixed failing clustering notebook
* Errors in notebook and keyvault tests
* Fixing test issues in MicrosoftDefender.ipynb and EventClustering.ipynb
* Bug using wrong httpx code syntax in odata_driver.py
* Removing auto-load of VTLookup in nbinit
* Forcing notebook tests to use test msticpyconfig-test.yaml
* Added missing __init__.py to tests/data/browsers
* Removing vtlookup import from sectools init because of circular import error
* Moving IPStack check for API key to first call (rather than __init__) to avoid error on load.
* prospector config changed produces deprecation warning and non-zero exit code.
* Fixing misconfigured prospector.yaml
* Getting rid of warning from test_nbinit
* Adding McCabe suppression to ip_utils.py
* removing version restriction for prospector in Github actions python-package.yaml
* Powershell viewer PR
* Elastic driver skeleton and changes to allow driver-specific param substitution
* Cleaning up some import redirections to point to new locations
* Fixing circular import in vtfile_behavior
* Adding placeholder class to allow imports to sort of work even if sub-modules fail to import
* Refactored _value_or_default in query_source to reduce complexity
* Fixed import errors in elastic_driver.py and splunk_driver.py
* Fixed import from old location in nbinit
* Fixing warning in code_view.py
* Fixed test failure in test_code_view
* Changing test_timeline.py to use new mp_plot accessor in place of deprecated one.
* Fixing test failure in test_timeline.py
* 3f3140aSync changes to main into v2 branch (330)* adding devcontainer files (321)
* Syncing recent IPython-related changes (to skip magic creation if not in IPython)
* f7923e5Minor quality of life improvements to AzureData & MicrosoftSentinel (331)* Updated names in AzureData to match MicrosoftSentinel
* Added connection checks to Sentinel features

PRs
* Sync changes to main into v2 branch by ianhelle in https://github.com/microsoft/msticpy/pull/330
* Ianhelle/msticpy v2.0.0 merge updates 2022 03 14 by ianhelle in https://github.com/microsoft/msticpy/pull/338
* Ianhelle/implement isort 2022 02 15 by ianhelle in https://github.com/microsoft/msticpy/pull/327
* Ianhelle/implement isort branch post-fixes 2022 03 21 by ianhelle in https://github.com/microsoft/msticpy/pull/346
* Ianhelle/pivot dataprov selfload 2022 03 15 by ianhelle in https://github.com/microsoft/msticpy/pull/343
* Ianhelle/main mergeback 2022 04 05 by ianhelle in https://github.com/microsoft/msticpy/pull/355
* Merging changes from main for geoip.py, config editor and kusto_driver by ianhelle in https://github.com/microsoft/msticpy/pull/359
* Pebryan/2022 4 14 auth merge by petebryan in https://github.com/microsoft/msticpy/pull/368
* Fixed minor issues by petebryan in https://github.com/microsoft/msticpy/pull/372
* Ianhelle/v2 reorg directories 2 2022 04 12 by ianhelle in https://github.com/microsoft/msticpy/pull/377
* Ianhelle/mpconfigedit fix from main 2022 05 22 by ianhelle in https://github.com/microsoft/msticpy/pull/396


**Full Changelog**: https://github.com/microsoft/msticpy/compare/v.1.5.2...v2.0.0.rc1

Minor release fixing a few usability issues.

What's Changed
* Adding full Delegated Auth support to all OData Drivers by petebryan in https://github.com/microsoft/msticpy/pull/409
This allows MDE and Graph users to use User-delegated authentication rather than app ID/secrets
* Fixes for usability bugs in kql_driver, nbinit, user_config - added typing-extensions requirement by ianhelle in https://github.com/microsoft/msticpy/pull/411
- Kql driver will revert to Kqlmagic-based device authentication if Azure Authentication fails
- Kql driver suppresses "missing PyGObject message" - a dependency that isn't required in this scenario
- init_notebook produced spurious error message about Virus Total libraries not being available even if they were not used.
- User config throws error if user has partial auto-load configuration in msticpy
* Replace MSAL auth plaintext file cache with memory cache by petebryan in https://github.com/microsoft/msticpy/pull/413
- removed ability to use plaintext token cache because of security concerns
* Update API version for list_alert_rules by FlorianBracq in https://github.com/microsoft/msticpy/pull/399
* Updating Dockerfile source to mcr anaconda by ianhelle in https://github.com/microsoft/msticpy/pull/397
Docker source switched to trusted anaconda source for supply chain security
* Updated Tweet bot to include more context in the tweets by petebryan in https://github.com/microsoft/msticpy/pull/403
* Updated tweet action to include more detail in the tweets by petebryan in https://github.com/microsoft/msticpy/pull/405
* Adding Microsoft SECURITY.MD by microsoft-github-policy-service in https://github.com/microsoft/msticpy/pull/407
* Bump readthedocs-sphinx-ext from 2.1.5 to 2.1.6 by dependabot in https://github.com/microsoft/msticpy/pull/400


**Full Changelog**: https://github.com/microsoft/msticpy/compare/v1.8.1...v1.8.2

What's Changed
* MpConfigEdit throws exception when passed file name
* MpConfigEdit throws error with invalid file path. by ianhelle in https://github.com/microsoft/msticpy/pull/395


**Full Changelog**: https://github.com/microsoft/msticpy/compare/v1.8.0...v1.8.1