Overview
We've finally decided that we're ready to put a 1.0.0 release number
on MSTICPy and move out of the Beta world.
There are several new features in this release and some significant updates to other features.
New:
- Flexible dependencies with pip extras
- Settings management and auto-load of components
- An experimental SQL to KQL translator
Update:
- Significant updates to the pivoting library.
- Updates to notebook initialization include Azure CLI single sign-on and
Azure ML data explorer.
Semantic versioning
Although we've been a little "flexible" with our use of version numbers
while in beta, we plan to adopt standard practice for [semantic versioning](https://semver.org/)
moving forward:
- Updates to major release number will indicate significant breaking changes
- Minor release number increments will indicate significant new or changed features
- Point/patch updates will indicate fixes and minor feature improvements
E.g. version *Major.Minor.Patch*
New Features
MSTICPy Dependencies - Implementing Extras 128
MSTICPy relies on a number of Python packages such as pandas, numpy
and bokeh. Previously, many rarely-used dependencies were installed regardless of whether
they were going to be used or not. We've now split MSTICPy dependencies into "extras".
The base install of MSTICPy only installs a core set of dependencies - others can be installed
on demand by specifying the name of the extra feature that you want to use.
This is coupled with a refactoring of the code so that, should you try
to use functionality that does not have a required dependency installed, an informative
exception message is displayed telling you which extra you need to install.
In this release we are using
[KqlmagicCustom](https://pypi.org/project/KqlmagicCustom/) (still in pre-release). This
also uses "extras" and has a significantly smaller dependency list for its base
install. The switch to KqlmagicCustom alone has halved the MSTICPy dependency set for a core installation.
The install time should be reduced from several minutes to a few seconds in
typical usage.
Feature details
- Implemented pip "extras" for msticpy install - drastically reduces
install time for core msticpy.
- Refactored many modules to emit informative warning if user tries to load functionality
that requires a different "extra"
- Refactored unit tests to work with missing extras.
- Added extras documentation to [Installing documentation](https://msticpy.readthedocs.io/en/latest/getting_started/Installing.html)
- Added requirements-all.txt - that will always install *all* dependencies.
- Added pre-commit hook to generate requirements-all.txt
[Documenation](https://msticpy.readthedocs.io/en/latest/getting_started/Installing.html#selective-installation-using-extras)
MSTICPY configuration settings management 136
The MSTICPy configuration file has grown to be quite complex. We've tried
to address this by creating some interactive tools to let you create and edit
settings using a simple GUI. We've also created a notebook that walks you through
creating your settings file for the first time.
The Getting Started and ConfiguringYourEnvironment notebooks in the
[Azure Sentinel Notebooks GitHub repo](https://github.com/Azure/Azure-Sentinel-Notebooks)
have also been updated to use these tools.
MSTICPy itself, has a number of initialization/loading steps that you need to carry out
before starting to use it in a notebook. The other part of this feature
is the ability to specify, in your settings, what components to load at initialization.
Components that can be auto-loaded include: DataProviders, TI Providers,
[Notebooklets](https://msticnb.readthedocs.io/en/latest/) and Pivot functions.
These are specified in the msticpyconfig.yaml
(you can edit these settings with the Settings editor) and are auto-loaded
when you run `init_notebook()` at the start of your notebook.
Feature details
- Flexible UI for configuring MSTICPy settings
- User environment configuration for notebooks - lets you specify (in
settings), which providers/modules, etc. that should be loaded automatically.
- Added HTML output from nbinit to show imported modules
- Added `check_version` in `init_notebook` function to indicate if
a new version of MSTICPy has been released.
- Added a function to retrieve and show current KV secrets
- Updated [MSTICPy Configuration documentation](https://msticpy.readthedocs.io/en/latest/getting_started/msticpyconfig.html)
- Added [MSTICPy Settings Editor documenation](https://msticpy.readthedocs.io/en/latest/getting_started/SettingsEditor.html)
- Added [MSTICPy Settings notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/MPSettingsEditor.ipynb)
- Added [documentation diagram](https://github.com/microsoft/msticpy/blob/master/docs/diagrams/MPSettingsConfig.png)
Config editor, auto-load, dependencies and Sql2Kql translator
[Documentation](https://msticpy.readthedocs.io/en/latest/getting_started/SettingsEditor.html)
SQL to KQL Translator 132
This is a simple (and somewhat experimental) feature to convert ANSI SQL to KQL queries.
- Core feature implemented as `msticpy.data.sql_to_kql` using `moz_sql_parser`
- Support for SELECT, WHERE, JOIN, UNION, GROUP BY and others.
- Support for limited SparkSQL extensions
- [SQL TO KQL Conversion Documentation](https://msticpy.readthedocs.io/en/latest/data_acquisition/SqlToKql.html)
- [SQLToKql Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/SqlToKql.ipynb)
[Documentation](https://msticpy.readthedocs.io/en/latest/data_acquisition/SqlToKql.html)
Updated Features
Pivot Functions 131
The pivot functionality has been updated with several usability and
documentation improvements. Notable features include:
- Pivot browser, letting you browse/search for pivot functions.
- Persistent Pipelines - you can save pipelined pivot operations in a
YAML file and run them on input DataFrames
> **Note**: Shortly after the MSTICPy v1.0.0 release, we will be releasing
> another update to Pivot Functions that will (among other things)
> shorten pivot function names to be friendlier and easy to type.
> If you are starting
> to use Pivot functions, please bear this in mind when testing/coding.
Full list of changes
- Made AzureSentinel and MDE the preferred names for LogAnalytics and MDE drivers.
- Added pivot_browser UI - pivot_browser.py
- Added ability to read pipeline definitions from yaml files - pivot_pipeline.py
- Adding pivot.tee_exec pipeline function - in pivot_pd_accessor.py
- Add ability to add arbitrary/ad hoc functions as pivots - in pivot.py
- Exposing get_timespan function in Pivot class as public function - in pivot.py.
- Added DNS entity to several pivot functions - mp_pivot_reg.yaml
- Fixed some queries for more consistency.
- Pivot data query functions now prefixed with table name.
- Added ability for pivot functions to return raw output.
- Add joins for pivot data queries in pivot_data_queries.py
- Add "print" query debug parameter in data_providers.py
- Add find_entity function in entities __init__.py
- Add "pivots" attribute (an alias for get_pivot_list) in entity.py
- Add ability to set timespan more flexibly. Calling set_timespan() no longer resets the timespan.
- Add PivotBrowser method to Pivot class - in pivot.py
- Switched engine to "Python" for pd.read_csv in pivot_magic_core.py to handle more formatting types.
- Add positional parameters to pipeline step and cleaned up code in pivot_pipeline.py
- Updated [PivotFunctions documentation](https://msticpy.readthedocs.io/en/latest/data_analysis/PivotFunctions.html)
- Updated [PivotFunctions Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/PivotFunctions.ipynb)
- Added [PivotFunctions-Introduction notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/PivotFunctions-Introduction.ipynb)
[Documentation](https://msticpy.readthedocs.io/en/latest/data_analysis/PivotFunctions.html)
Notebook Initialization
MSTICPy has a module `nbinit` and we use the main function in that module
(`init_notebook`) to streamline a lot of the setup needed to get a
notebook up and running. It does things like: import modules, install required
packages, check configuration.
We've made a few significant updates to this module including:
- Defaulting to using the native data browser in Azure ML notebooks
- Tidying up and minimizing output so that warnings are clearer and
help URLs are displayed clearly
- Enabling Single sign-on for notebooks using Azure CLI (all notebooks
in a session will use the Azure CLI authenticated session to obtain tokens
for your Azure Sentinel workspaces)
Miscellaneous Updates and Additions
- 9a8d648 (149)
- Adding time unit control to QueryTime widget to allow interactive setting of day/week/month.
Also increased the max range for these units - in nbwidgets.py
- Docs: Updated notebooksamples.rst with latest notebooks. https://msticpy.readthedocs.io/en/latest/notebooksamples.html
- Docs: Added Releases.rst - pointer to GitHub release page. https://msticpy.readthedocs.io/en/latest/Releases.html
- Docs: Generated new DataQueries.rst doc https://msticpy.readthedocs.io/en/latest/data_acquisition/DataQueries.html
- Docs: Added links to medium articles in ReadtheDocs https://msticpy.readthedocs.io/en/latest/blog_articles.html
- Docs: Added [SplunkProvider documentation](https://msticpy.readthedocs.io/en/latest/data_acquisition/SplunkProvider.html)
- Docs: Updated [README](https://github.com/microsoft/msticpy/blob/master/README.md) and [package summary](https://msticpy.readthedocs.io/en/latest/getting_started/PackageSummary.html) section of MSTICPy to align with current features
- Reordered parameters in wsconfig.py WorkspaceConfig so that you can supply the workspace name as single positional parameter
python
ws_config = WorkspaceConfig("MyWorkspace")
- Change kql_driver.py so that you can supply a WorkspaceConfig instance as the "connection_str"
python
qry_prov.connect(WorkspaceConfig("MyWorkspace"))
- Change to pivot_register to return single "raw" result if it is a list of one item
- Add Pivot and entities as auto-imported items
- Add `create` static method to entity to instantiate entity from dict or pd.Series
to allow easier programmatic creation of entities.
python
acct = Entity.create({"Name": "ian", "Type": "Account"})
- Adding more Azure Sentinel queries for VMComputer and DNSEvents tables.
- `nbinit.init_notebook()`:
- Added resource URLs to warnings
- added KQLMAGIC_CONFIGURATION to enable trying AzureCLI SSO by default.
- added pandas config to return schema with dataframe html to render using native nteract data browser.
- cbe918e
- geoip.py - Adding more descriptive to text on how to fix things in exception if no API keyvault_settings
- tilookup.py - added raise MsticpyUserConfig exception with help URIs if no providers are enabled - in lookup_ioc, lookup_iocs
- 4c32ab9
- Changed URL formatting so that URL help links in MSTICPy exceptions use about="_blank" to open in new tab - in exceptions.py.
- Switched to using a list for output instead of concatenated string in `MsticpyUserError._repr_html_()`
- Tidied up formatting of text and updated exception URL in wsconfig.py
- nbinit.py:
- Add about="_blank" to URLs in warnings,
- Tidied up some text
- Changed red error to orange warning
- Trapped seaborn import error if not installed
- Print out list of imported packages by default
- Changed Development Status classifier to 'Production' and adding some extra keywords in setup.py
- Replaced matplotlib `draw_entity_alert_graph` with Bokeh version in nbdisplay.py
- f386ccd (136)
- DataMasking:
- Adding hash_account as separate item type to data_obfus.py
- Making hash_ip more flexible - ignoring things like localhost
- Updating documentation, tests and mapping file.
- 1cebe8c (131)
- Made 'AzureSentinel' and 'MDE' the preferred names for LogAnalytics and MDE drivers:
`ws_config = WorkspaceConfig("AzureSentinel")`
- Update MordorData.rst doc with better intro section
Fixes
- 0b7d1dc Updating KqlmagicCustom version
- 48f02b5
- Handle cases where empty data set is passed to timeline functions - timeline.py
- Fixing occasional race condition in testing with KeyringClient.is_keyring_available
- 5915b9d
- Adding MC0001 McCabe suppression to nbinit.py
- Fixing typo in DF name in syslog_utils.py
- f01d094 Updating version to 1.0.0
- Fixing test_security_alert.py unit test
- Adding small test for KeyringClient - test_provider_secrets.py
- c4c1fa2
- Linux bug - Error loading secret_settings in environment that doesn't have a supported
Keyring backend. Added test to KeyringClient before trying to load.
Also fixed logic to properly honor settings (subject to above check) - previously the
`use_keyring` param defaulted to True so would try to load Keyring even
if settings were = False.
- bd99b11
- If running in IPython environment check_and_install_missing_packages() will use `%pip`
IPython magic function rather than `subprocess` in utility.py. This ensures that
the pip install will end up in the same Python environment that the notebook kernel
is running in.
- Suppressing warnings in Kqlmagic load in kql_driver.py
- File entity "FullPath" generation now handles None values for directory and separator more gracefully in file.py
- Changed color attribute extraction for `draw_alert_entity_graph` and `plot_entity_graph`
so that it defaults to a color even if the node has no color attribute - in nbdisplay.py
- Handle inter-entity references in child entities in security_alert.py (e.g. if entity1[$id=1]
has entity2[$id=2]) nested inside it and other entities have references ($ref=2) to
entity2
- Handle cases where no "Name" attribute is defined in account entities.
- Added additional test cases for nested entity references in test_security_alert.py
- bee0f26, 69256bf, cbe918e
- If any warning from WorkspaceConfig, print this out - nbinit.py
- Added additional extra for KqlmagicCustom to install `pyperclip`
- Changed az_connect defaults to all auth_methods if None supplied in params or config - azure_auth.py
- azure_auth_core.py - added `default_auth_methods()` function
- keyvault_settings.py - default to using all auth_methods
- provider_settings.py - do not instantiate SecretSettings unless KeyVault config has settings
- wsconfig.py:
- Changed wording of warnings/error messages
- Does not now raise an exception if no suitable config found
- Does not raise a Python warning if we rely on searching to find a config file.
- Changing default entity graph plot size in `draw_entity_alert_graph`/`plot_entity_graph` shows the plot by default unless hide=True parameter - nbdisplay.py
- nbinit.py
- Changed all output so that it is HTML text vs. print
- Captured output from called functions that print output
- Minor rewording and changing header size of titles.
- security_alert_graph.py - check if File entity has FullPath attribute before accessing it.
- Added moz_sql_parser back to mypy.ini
- Changed test_pkg_config.py so that it gives clearer message when test fails
- added "KqlmagicCustom[jupyter-extended]" to conda exceptions
- Updated test_pkg_config.py to account for WorkspaceConfig no longer produces and error.
- 9a8d648 (149)
- Updated msticpy notebooks
- Updated setup.py, requirements.txt and conda-reqs with new/non-conflicting version
- Fix to test_nbinit.py and import_analyzer.py
- Added test script for dependencies/extras
- Updating version to pre3/rc3
- Fix to README
- Removed random unicode char in PivotFunctions.rst
- Fixing test error in test_nbinit.py
- Some fixes to notebooks (mainly getting rid of unneeded Seaborn refs)
- Doc string error - prospector - in test_mp_extras.py
- Fixed QueryProviderDocumenter.ipynb
- Added KqlmagicCustom[jupyter-basic] to core components and Updating docs for this in Installing.rst. Updated test_pkg_imports.py to account for this.
- Setting KQLMAGIC_EXTRAS_REQUIRE env var in package __init__.py to prevent warnings in Kqlmagic
- Minor clarification in MPSettingsEditor.ipynb
- Fix bug in ip_utils.py convert to entities
- Fix to provider name handling in user_config.py
- Fixing wording in `SelectAlert` widget
- Fixing potential None value error in GetText and GetEnvironmentText
- Fixing a few bugs and tidying in `convert_to_ip_entities` in ip_utils.py
- Fixing issue 146 - Error is thrown when AzureSentinel config is not present in msticpyconfig.yaml file
- Added extra tests for nbinit.py in test_nbinit.py
- Fixing Issue 148 infinite recursion issue in process_tree_utils.py
- Adding file lock to unit_test_lib::custom_mp_config - because settings are global, multiple test processes can change the settings on each other.
- Minor updates to test_nbinit.py
- Fix in pivot_register.py - `iterate_func` did not pass through **kwargs to function.
Also added a few comments to explain what's going on.
- Bug in security_alert_graph - if NTDomain attribute is None
- Added local function cache to better handle repeated IPs
- Missing update to `all_ips` perf optimization in `convert_to_ip_entities`
- Adding filelock to dev requirements-dev.txt
- Corrected pip extras syntax in Installing.rst
- Invalid return type in pkg_config.py:: validate_config
- Added `prompt_for_ws` function to wsconfig.py. Also added more detail to warnings on how to fix things.
- Adding `markdown` and `beautifulsoup4` to requirements-dev.txt and conda-reqs-dev.txt
- Fixing test error - seems like subtle change or bug in pandas groupby behavior - in sessionize.py
- Adding more verbose output to try to catch spurious errors in test_nbinit.py
- Workarounds for some test errors in test_nbinit.py and test_user_config.py
- Updating requirements-dev.txt with pip-compatible versions.
- Adding same updates to conda-reqs-dev.txt and conda-reqs-dev-pip.txt
- Error fixed in initializing notebooklets and pivots in user_config
- Making test more flexible in test_ip_utils.py to allow for IP address moving ASNs.
- 1cebe8c (131)
- Remove unneeded code from keyvault_client.py
- Fixed pivot_register_reader to skip classes that cannot be instantiated (e.g. IPStack if user doesn't have API key)
- f386ccd (136)
- Correction to FoliumMap.ipynb, removing dropna from read_csv in FoliumMap.ipynb
- fix to config2kv.py to correct some problems, Also added a function to retrieve and show current KV secrets
- fix for ipwidgets warning about deprecated `on_submit()` method
- multiple fixes for typos and duplicate section names in: DataProviders.rst, UploadData.rst, PivotFunctions.rst
- added SplunkProvider.rst doc for Splunk provider
- fixed issue in nbinit.py where extra_imports were being lost.
- fix for QueryTime in nbwidgets.py - exception if user types invalid value into date field.
- fixed several issues in test_mp_release.cmd with messed up folders/current folder.
- efccf19 (137)
- Couple of minor fixes to txt2df magic.