Msticpy

Sentinel updates
WorkspaceConfig and Sentinel QueryProvider (azure_monito_driver) have had a few updates:
- handle both old (Kqlmagic) and standard connection string formats in WorkspaceConfig
- removing a lot of legacy code from WorkspaceConfig
- Allow additional connection parameters to be used with MSSentinel QueryProvider for
authentication parameters (e.g. you can now supply authentication parameters like "client_id", "client_secret" to `query_provider.connect()`)
- `msticpyconfig.yaml` now supports using an "MSSentinel" key in place of "AzureSentinel"
- Workspace entries in msticpyconfig.yaml support an `Args` subkey, where you can add authentication parameters - these will be supplied to the `connect()` method if not overridden on the command line. Like Args sections for other providers, the values here can be text or references to environment variables or Azure Key Vault secrets.
- Fix to MSSentinel API update_incident to add full properties

Splunk Updates
- Added jwt authentication token expiry check.

Other fixes
Fix for vtlookup3.py
- Fixed problematic way of using nestasyncio - this was causing failures when run from a langchain agent.
Fix for lookup/tilookup
- If the progress parameter was not passed it would still try to cancel a non-existent progress task and cause an exception.
QueryProviders
- Fix split query time-ranges calculation - thanks to pjain90 for spotting this.

What's Changed
* Set up CI with 1ES Azure Pipelines by ianhelle in https://github.com/microsoft/msticpy/pull/763
* Update ws_config to handle kqlmagic connection strings by ianhelle in https://github.com/microsoft/msticpy/pull/767
* Fix split query time-ranges calculation by ianhelle in https://github.com/microsoft/msticpy/pull/762
* Add support for ruff and u/p devcontainer by ianhelle in https://github.com/microsoft/msticpy/pull/765
* Add jwt auth token expire check and modify some messages when connecting Splunk by Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/770
* WSConfig updates by ianhelle in https://github.com/microsoft/msticpy/pull/771
* Pass `true` for props into `_build_sent_data` when calling `update_incident` by kylelol in https://github.com/microsoft/msticpy/pull/774
* Changing cert thumbprint from Sha1 to Sha256 in Az Kusto driver by ianhelle in https://github.com/microsoft/msticpy/pull/775

New Contributors
* kylelol made their first contribution in https://github.com/microsoft/msticpy/pull/774
* pjain90 made their first contribution in https://github.com/microsoft/msticpy/pull/762

**Full Changelog**: https://github.com/microsoft/msticpy/compare/v2.11.0...v2.12.0

This is a minor release mainly to add a warning for Kusto/Sentinel queries that return partial results.
A close friend of MSTICPy (thx Cyb3r-Monk) had spotted that MSTICPy does not report partial results when doing split queries so it's possible to lose data from the query range silently.

Due to an unfortunate admin error, the fix for this was committed direct to main, so no PR for this is available. :-(

If you want the query to fail (throw an exception) rather than just warn you can supply a new parameter `fail_if_partial`.
This only affects the Sentinel query provider and works for standard as well as split queries.

NOTE: the documentation has a typo and calls this `fail_on_commit` - we'll fix that in the next release to support both `fail_if_partial` and `fail_on_partial`

Example
python3
qry_prov.exec_query(query_string, fail_if_partial=True)



What's Changed
* Missing PR for partial query warning and fixes for pandas deprecation warnings [See the diff for changes](https://github.com/microsoft/msticpy/compare/d1c0912b1bf6b2388fdb92193e8ae5a500549172...90bcd24a2a0230e8a189e6d607e3fb46f2226d66)
* Fixing group.apply for pandas < 2.2.1 by ianhelle in https://github.com/microsoft/msticpy/pull/759
* Added missing quotation in code block by ryan-aus in https://github.com/microsoft/msticpy/pull/753
* Bump httpx from 0.25.2 to 0.27.0 by dependabot in https://github.com/microsoft/msticpy/pull/754
* Bump readthedocs-sphinx-ext from 2.2.3 to 2.2.5 by dependabot in https://github.com/microsoft/msticpy/pull/743
* Updated conda reqs files for new packages by ianhelle in https://github.com/microsoft/msticpy/pull/758
* Build break fix for splunk SDK by ianhelle in https://github.com/microsoft/msticpy/pull/760

New Contributors
* ryan-aus made their first contribution in https://github.com/microsoft/msticpy/pull/753

**Full Changelog**: https://github.com/microsoft/msticpy/compare/v2.10.0...v2.11.0

What's Changed
* Add nest_asyncio to run threaded queries by FlorianBracq in https://github.com/microsoft/msticpy/pull/737
* Bump sphinx-rtd-theme from 1.3.0 to 2.0.0 by dependabot in https://github.com/microsoft/msticpy/pull/738
* Bump httpx from 0.25.0 to 0.25.2 by dependabot in https://github.com/microsoft/msticpy/pull/736
* Adding Virus Total Search Capabilities by secops-account in https://github.com/microsoft/msticpy/pull/739
* Add security token auth and credential loading from msticpyconfig.yaml to SplunkUploader by Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/731
* fix: updated _get_query_status in the azure monitor driver by aka0 in https://github.com/microsoft/msticpy/pull/745
* Added M365DGraph to the supported environments for existing queries by d3vzer0 in https://github.com/microsoft/msticpy/pull/748
* Small Typo correction in SentinelWatchlists.rst by Korving-F in https://github.com/microsoft/msticpy/pull/746
* Fix ibm_xforce TI provider for domain names and URLs by pcoccoli in https://github.com/microsoft/msticpy/pull/749
* Update python-package.yml by ianhelle in https://github.com/microsoft/msticpy/pull/750
* Ianhelle/aml updates 2024 01 31 by ianhelle in https://github.com/microsoft/msticpy/pull/751
* Ianhelle/warning fixes 2024 02 11 by ianhelle in https://github.com/microsoft/msticpy/pull/752

New Contributors
* secops-account made their first contribution in https://github.com/microsoft/msticpy/pull/739
* aka0 made their first contribution in https://github.com/microsoft/msticpy/pull/745
* Korving-F made their first contribution in https://github.com/microsoft/msticpy/pull/746
* pcoccoli made their first contribution in https://github.com/microsoft/msticpy/pull/749

**Full Changelog**: https://github.com/microsoft/msticpy/compare/v2.9.0...v2.10.0

Some of the highlights of this release:

IPQualityScore
New TI provider submitted by petebryan - provides a lot of interesting stats on IPs.

Defender Advanced Hunting API
Thanks to d3vzer0 our MS Defender client is now able to use the support Graph-based API rather than the legacy
APIs. To use this, for the moment use the DataEnvironment name `M365DGraph` when you create
query provider. In the next 0.x release we will switch the other aliases for M365D, MDE, MDATP to use this
new interface and deprecate the existing ones.

Startup errors when running in unexpected environments.
`init_notebook` made some (incorrect) assumptions about when it would be running in a Synapse environment.
Azure Machine Learning have recently changed their default compute to be a Synapse environment.
Fixes here will correct failures due to faulty detection of environment type.

Startup fixes and perf improvements
We've optimized some of the imports done within the package at startup so msticpy should be quicker to
load.

Azure env credentials fix
Although we previously supported the Azure EnvironmentCredential credential type, our implementation allowed
you to use only with ClientID + ClientSecret. The changes allow it to be used with other supported
credential formats - notably username + password and certificate authentication using a certificate file.

Improvements to Entities
Although these are not visible to most people, we try to keep our Entity definitions in sync with the official
Microsoft "V3" entity definitions. We've added a few entity types and updated some of the attributes
to bring this in line, while still allowing backwards compatible attributes to be used.

What's Changed
* Ianhelle/entity updates 2023 09 01 by ianhelle in https://github.com/microsoft/msticpy/pull/718
* Ianhelle/lazy-import-init-2023-09-26 by ianhelle in https://github.com/microsoft/msticpy/pull/717
* Fix Azure env credential authentication by ianhelle in https://github.com/microsoft/msticpy/pull/722
* Update documentation for installing in isolated env by ccianelli22 in https://github.com/microsoft/msticpy/pull/724
* Bump isort to 5.12.0 in pre-commit config by 2xyo in https://github.com/microsoft/msticpy/pull/723
* Remove stack trace from logging by FlorianBracq in https://github.com/microsoft/msticpy/pull/729
* fix: init_notebook and entities by ianhelle in https://github.com/microsoft/msticpy/pull/730
* Fix time span values by FlorianBracq in https://github.com/microsoft/msticpy/pull/728
* Added additional DataProvider for Advanced Hunting via Graph by d3vzer0 in https://github.com/microsoft/msticpy/pull/725
* Allow POST HTTP method by 2xyo in https://github.com/microsoft/msticpy/pull/726
* Bump readthedocs-sphinx-ext from 2.2.2 to 2.2.3 by dependabot in https://github.com/microsoft/msticpy/pull/716
* Added new TI Provider - IPQualityScore by petebryan in https://github.com/microsoft/msticpy/pull/733

New Contributors
* 2xyo made their first contribution in https://github.com/microsoft/msticpy/pull/723

**Full Changelog**: https://github.com/microsoft/msticpy/compare/v2.8.0...v2.9.0

A few bugs had crept in over the last couple of releases: some due to buggy coding, some due the world moving forward. So, many items in this release are to address these.

Among the feature improvements are the following:
* Documentation and scripts from ccianelli22 for creating a MSTICPy install for use in isolated (no Internet) environments. This is super useful for customers operating in sovereign clouds or other air-gapped high-security environments.
* Added Splunk authentication method using security token rather than username/password - thanks Tatsuya-hasegawa
* Query yaml file validation by FlorianBracq
* Paging for large CyberReason queries by FlorianBracq
* Modern method to obtain cloud-specific URL endpoints for Azure services. Previously, we were relying on msrestazure, which is now deprecated for this purpose. Many thanks to ccianelli22 for the work to do this.
* Fix (by me) for a bug I'd introduced with the switch to using Azure-monitor-query library for MS Sentinel. When using a connection string with this new driver, the logic failed to parse and extract details from this correctly. Many thanks to cindraw for reporting this bug.


What's Changed
* Update mde_proc_pub.pkl by FlorianBracq in https://github.com/microsoft/msticpy/pull/709
* Update Introduction.rst by praveenjutur in https://github.com/microsoft/msticpy/pull/700
* Update methodology of getting endpoints for cloud environment by ccianelli22 in https://github.com/microsoft/msticpy/pull/704
* Validation of the YAML structure of query files by FlorianBracq in https://github.com/microsoft/msticpy/pull/660
* Intsights api update by FlorianBracq in https://github.com/microsoft/msticpy/pull/710
* Fix m365d/mde hunting query options by Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/702
* Cybereason pagination support + multi-threading by FlorianBracq in https://github.com/microsoft/msticpy/pull/707
* Add bearer token auth to splunk driver by Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/708
* fix wl bug when creating a new wl when wl count is 0 by ccianelli22 in https://github.com/microsoft/msticpy/pull/719
* Update installation docs to include installation for isolated envs by ccianelli22 in https://github.com/microsoft/msticpy/pull/715
* Fixing regular expression error for connection string in WorkspaceConfig by ianhelle in https://github.com/microsoft/msticpy/pull/706
* Fix documentation formatting, update steps for downloading msticpy by ccianelli22 in https://github.com/microsoft/msticpy/pull/720

New Contributors
* praveenjutur made their first contribution in https://github.com/microsoft/msticpy/pull/700
* ccianelli22 made their first contribution in https://github.com/microsoft/msticpy/pull/704

**Full Changelog**: https://github.com/microsoft/msticpy/compare/v2.7.0...v2.8.0

Updated method to dynamically fetch Azure endpoints (rather than relying on deprecated msrestazure).
Updated version of Insight data provider