Overview
We've finally decided that we're ready to put a 1.0.0 release number
on MSTICPy and move out of the Beta world.
There are three new features in this release and one major update.
- Flexible dependencies with pip extras
- Settings management and auto-load of components
- An experimental SQL to KQL translator
- Significant updates to the pivoting library.
New Features
MSTICPY Dependencies - Implementing Extras 128
We've split MSTICPy dependencies into "extras". This means that the
base install of MSTICPy does not install every dependency. Previously,
many rarely-used dependencies were installed regardless of whether
they were going to be used or not.
This is couple with a refactoring of the code so that, should you try
to use functionality that does not have a dependency installed, an informative
exception message is displayed telling you which extra you need to install.
- Implemented pip "extras" for msticpy install - drastically reduces
install time for core msticpy.
- Refactored many modules to emit informative warning if user tries to load functionality
that requires a different "extra"
- Refactored unit tests to work with missing extras.
- Added pyperclip to pkg dependencies exceptions.
- Added extras documentation to [Installing documentation](https://msticpy.readthedocs.io/en/latest/getting_started/Installing.html)
- Added requirements-all.txt - that will always install *all* dependencies.
- Added pre-commit hook to generate requirements-all.txt
MSTICPY config settings management 136
The MSTICPy configuration file has grown to be quite complex. We've tried
to address this by creating some interactive tools to let you create and edit
settings using a simple GUI and creating a notebook that walks you through
creating your settings file for the first time.
MSTICPy itself has a number of initialization/loading steps that you need to carry out
before starting to use it in a notebook. The other part of this feature
is the ability to specify, in settings, what to load at initialization.
Components that can be auto-loaded include: DataProviders, TI Providers,
Notebooklets and Pivot functions. These are specified in the msticpyconfig.yaml
(you can edit these settings with the Settings editor) and auto-loaded
when you run `init_notebook()` at the start of your notebook.
- Flexible UI for configuring MSTICPy settings
- User environment configuration for notebooks - lets you specify (in
settings), which providers/modules, etc. that should be loaded automatically.
- Added minimal output from nbinit to show imported modules
- Added check_version in init_notebook function to indicate if
a new version of MSTICPy has been released.
- Added a function to retrieve and show current KV secrets
- Updated [MSTICPy Configuration documentation](https://msticpy.readthedocs.io/en/latest/getting_started/msticpyconfig.html)
- Added [MSTICPy Settings Editor documenation](https://msticpy.readthedocs.io/en/latest/getting_started/SettingsEditor.html)
- Added [MSTICPy Settings notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/MPSettingsEditor.ipynb)
- Added [documentation diagram](https://github.com/microsoft/msticpy/blob/master/docs/diagrams/MPSettingsConfig.png)
Config editor, auto-load, dependencies and Sql2Kql translator
SQL to KQL Translator 132
- Core feature implemented as msticpy.data.sql_to_kql using moz_sql_parser
- Support for limited SparkSQL extensions
- [SQL TO KQL Conversion Documentation](https://msticpy.readthedocs.io/en/latest/data_acquisition/SqlToKql.html)
- [SQLToKql Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/SqlToKql.ipynb)
Updated Features
Pivot Functions 131
The pivot functionality has been updated with several usability and
documentation improvements. Notable features include:
- Pivot browser, letting you browse/search for pivot functions.
- Persistent Pipelines - you can save pipelined pivot operations in a
YAML file and run them on input DataFrames
Full list of changes
- Made AzureSentinel and MDE the preferred names for LogAnalytics and MDE drivers.
- Added pivot_browser UI - pivot_browser.py
- Added ability to read pipeline definitions from yaml files - pivot_pipeline.py
- Adding pivot.tee_exec pipeline function - in pivot_pd_accessor.py
- Add ability to add arbitrary/ad hoc functions as pivots - in pivot.py
- Exposing get_timespan function in Pivot class as public function - in pivot.py.
- Added DNS entity to several pivot functions - mp_pivot_reg.yaml
- Fixed some queries for more consistency.
- Pivot data query functions now prefixed with table name.
- Added ability for pivot functions to return raw output.
- Add joins for pivot data queries in pivot_data_queries.py
- Add "print" query debug parameter in data_providers.py
- Add find_entity function in entities __init__.py
- Add "pivots" attribute (an alias for get_pivot_list) in entity.py
- Add ability to set timespan more flexibly. Calling set_timespan() no longer resets the timespan.
- Add PivotBrowser method to Pivot class - in pivot.py
- Switched engine to "Python" for pd.read_csv in pivot_magic_core.py to handle more formatting types.
- Add positional parameters to pipeline step and cleaned up code in pivot_pipeline.py
- Updated [PivotFunctions documentation](https://msticpy.readthedocs.io/en/latest/data_analysis/PivotFunctions.html)
- Updated [PivotFunctions Notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/PivotFunctions.ipynb)
- Added [PivotFunctions-Introduction notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/PivotFunctions-Introduction.ipynb)
Miscellaneous Updates and Additions
- added [SplunkProvider documentation](https://msticpy.readthedocs.io/en/latest/data_acquisition/SplunkProvider.html)
- Updated [README](https://github.com/microsoft/msticpy/blob/master/README.md) and [package summary](https://msticpy.readthedocs.io/en/latest/getting_started/PackageSummary.html) section of MSTICPy to align with current features
Fixes
- Updated formatting for new black version
- Remove unneeded code from keyvault_client.py
- Fixed pivot_register_reader to skip classes that cannot be instantiated (e.g. IPStack if user doesn't have API key)
- Additions/corrections to Installing.rst
- Correction to FoliumMap.ipynb - removing `dropna` from read_csv in FoliumMap notebook
- Adding vt, vt_graph to Sphinx mock list
- Fixed some problems and renamed module locations in notebooks and RST docs.
- Some corrections to documentation in AzureSentinel and DataAcquisition docs.
- some fixes to tests for test_pkg_imports and import_analyzer.py
- fix to config2kv.py to correct some problems
- fix for ipwidgets warning about deprecated on_submit() method
- multiple fixes for typos and duplicate section names in: DataProviders.rst, UploadData.rst, PivotFunctions.rst
- fixed issue in nbinit.py where extra_imports were being lost.
- fix for QueryTime in nbwidgets.py - exception if user types invalid value into date field.
- fixed several issues in test_mp_release.cmd with messed up folders/current folder.
- Bandit warning on use of random.randint()
- Removing test "secret" from MPSettingsEditor.ipynb triggering credscan warning