Msticpy

This is a minor release mainly to add a warning for Kusto/Sentinel queries that return partial results.
A close friend of MSTICPy (thx Cyb3r-Monk) had spotted that MSTICPy does not report partial results when doing split queries so it's possible to lose data from the query range silently.

Due to an unfortunate admin error, the fix for this was committed direct to main, so no PR for this is available. :-(

If you want the query to fail (throw an exception) rather than just warn you can supply a new parameter `fail_if_partial`.
This only affects the Sentinel query provider and works for standard as well as split queries.

NOTE: the documentation has a typo and calls this `fail_on_commit` - we'll fix that in the next release to support both `fail_if_partial` and `fail_on_partial`

Example
python3
qry_prov.exec_query(query_string, fail_if_partial=True)



What's Changed
* Missing PR for partial query warning and fixes for pandas deprecation warnings [See the diff for changes](https://github.com/microsoft/msticpy/compare/d1c0912b1bf6b2388fdb92193e8ae5a500549172...90bcd24a2a0230e8a189e6d607e3fb46f2226d66)
* Fixing group.apply for pandas < 2.2.1 by ianhelle in https://github.com/microsoft/msticpy/pull/759
* Added missing quotation in code block by ryan-aus in https://github.com/microsoft/msticpy/pull/753
* Bump httpx from 0.25.2 to 0.27.0 by dependabot in https://github.com/microsoft/msticpy/pull/754
* Bump readthedocs-sphinx-ext from 2.2.3 to 2.2.5 by dependabot in https://github.com/microsoft/msticpy/pull/743
* Updated conda reqs files for new packages by ianhelle in https://github.com/microsoft/msticpy/pull/758
* Build break fix for splunk SDK by ianhelle in https://github.com/microsoft/msticpy/pull/760

New Contributors
* ryan-aus made their first contribution in https://github.com/microsoft/msticpy/pull/753

**Full Changelog**: https://github.com/microsoft/msticpy/compare/v2.10.0...v2.11.0

What's Changed
* Add nest_asyncio to run threaded queries by FlorianBracq in https://github.com/microsoft/msticpy/pull/737
* Bump sphinx-rtd-theme from 1.3.0 to 2.0.0 by dependabot in https://github.com/microsoft/msticpy/pull/738
* Bump httpx from 0.25.0 to 0.25.2 by dependabot in https://github.com/microsoft/msticpy/pull/736
* Adding Virus Total Search Capabilities by secops-account in https://github.com/microsoft/msticpy/pull/739
* Add security token auth and credential loading from msticpyconfig.yaml to SplunkUploader by Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/731
* fix: updated _get_query_status in the azure monitor driver by aka0 in https://github.com/microsoft/msticpy/pull/745
* Added M365DGraph to the supported environments for existing queries by d3vzer0 in https://github.com/microsoft/msticpy/pull/748
* Small Typo correction in SentinelWatchlists.rst by Korving-F in https://github.com/microsoft/msticpy/pull/746
* Fix ibm_xforce TI provider for domain names and URLs by pcoccoli in https://github.com/microsoft/msticpy/pull/749
* Update python-package.yml by ianhelle in https://github.com/microsoft/msticpy/pull/750
* Ianhelle/aml updates 2024 01 31 by ianhelle in https://github.com/microsoft/msticpy/pull/751
* Ianhelle/warning fixes 2024 02 11 by ianhelle in https://github.com/microsoft/msticpy/pull/752

New Contributors
* secops-account made their first contribution in https://github.com/microsoft/msticpy/pull/739
* aka0 made their first contribution in https://github.com/microsoft/msticpy/pull/745
* Korving-F made their first contribution in https://github.com/microsoft/msticpy/pull/746
* pcoccoli made their first contribution in https://github.com/microsoft/msticpy/pull/749

**Full Changelog**: https://github.com/microsoft/msticpy/compare/v2.9.0...v2.10.0

Some of the highlights of this release:

IPQualityScore
New TI provider submitted by petebryan - provides a lot of interesting stats on IPs.

Defender Advanced Hunting API
Thanks to d3vzer0 our MS Defender client is now able to use the support Graph-based API rather than the legacy
APIs. To use this, for the moment use the DataEnvironment name `M365DGraph` when you create
query provider. In the next 0.x release we will switch the other aliases for M365D, MDE, MDATP to use this
new interface and deprecate the existing ones.

Startup errors when running in unexpected environments.
`init_notebook` made some (incorrect) assumptions about when it would be running in a Synapse environment.
Azure Machine Learning have recently changed their default compute to be a Synapse environment.
Fixes here will correct failures due to faulty detection of environment type.

Startup fixes and perf improvements
We've optimized some of the imports done within the package at startup so msticpy should be quicker to
load.

Azure env credentials fix
Although we previously supported the Azure EnvironmentCredential credential type, our implementation allowed
you to use only with ClientID + ClientSecret. The changes allow it to be used with other supported
credential formats - notably username + password and certificate authentication using a certificate file.

Improvements to Entities
Although these are not visible to most people, we try to keep our Entity definitions in sync with the official
Microsoft "V3" entity definitions. We've added a few entity types and updated some of the attributes
to bring this in line, while still allowing backwards compatible attributes to be used.

What's Changed
* Ianhelle/entity updates 2023 09 01 by ianhelle in https://github.com/microsoft/msticpy/pull/718
* Ianhelle/lazy-import-init-2023-09-26 by ianhelle in https://github.com/microsoft/msticpy/pull/717
* Fix Azure env credential authentication by ianhelle in https://github.com/microsoft/msticpy/pull/722
* Update documentation for installing in isolated env by ccianelli22 in https://github.com/microsoft/msticpy/pull/724
* Bump isort to 5.12.0 in pre-commit config by 2xyo in https://github.com/microsoft/msticpy/pull/723
* Remove stack trace from logging by FlorianBracq in https://github.com/microsoft/msticpy/pull/729
* fix: init_notebook and entities by ianhelle in https://github.com/microsoft/msticpy/pull/730
* Fix time span values by FlorianBracq in https://github.com/microsoft/msticpy/pull/728
* Added additional DataProvider for Advanced Hunting via Graph by d3vzer0 in https://github.com/microsoft/msticpy/pull/725
* Allow POST HTTP method by 2xyo in https://github.com/microsoft/msticpy/pull/726
* Bump readthedocs-sphinx-ext from 2.2.2 to 2.2.3 by dependabot in https://github.com/microsoft/msticpy/pull/716
* Added new TI Provider - IPQualityScore by petebryan in https://github.com/microsoft/msticpy/pull/733

New Contributors
* 2xyo made their first contribution in https://github.com/microsoft/msticpy/pull/723

**Full Changelog**: https://github.com/microsoft/msticpy/compare/v2.8.0...v2.9.0

A few bugs had crept in over the last couple of releases: some due to buggy coding, some due the world moving forward. So, many items in this release are to address these.

Among the feature improvements are the following:
* Documentation and scripts from ccianelli22 for creating a MSTICPy install for use in isolated (no Internet) environments. This is super useful for customers operating in sovereign clouds or other air-gapped high-security environments.
* Added Splunk authentication method using security token rather than username/password - thanks Tatsuya-hasegawa
* Query yaml file validation by FlorianBracq
* Paging for large CyberReason queries by FlorianBracq
* Modern method to obtain cloud-specific URL endpoints for Azure services. Previously, we were relying on msrestazure, which is now deprecated for this purpose. Many thanks to ccianelli22 for the work to do this.
* Fix (by me) for a bug I'd introduced with the switch to using Azure-monitor-query library for MS Sentinel. When using a connection string with this new driver, the logic failed to parse and extract details from this correctly. Many thanks to cindraw for reporting this bug.


What's Changed
* Update mde_proc_pub.pkl by FlorianBracq in https://github.com/microsoft/msticpy/pull/709
* Update Introduction.rst by praveenjutur in https://github.com/microsoft/msticpy/pull/700
* Update methodology of getting endpoints for cloud environment by ccianelli22 in https://github.com/microsoft/msticpy/pull/704
* Validation of the YAML structure of query files by FlorianBracq in https://github.com/microsoft/msticpy/pull/660
* Intsights api update by FlorianBracq in https://github.com/microsoft/msticpy/pull/710
* Fix m365d/mde hunting query options by Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/702
* Cybereason pagination support + multi-threading by FlorianBracq in https://github.com/microsoft/msticpy/pull/707
* Add bearer token auth to splunk driver by Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/708
* fix wl bug when creating a new wl when wl count is 0 by ccianelli22 in https://github.com/microsoft/msticpy/pull/719
* Update installation docs to include installation for isolated envs by ccianelli22 in https://github.com/microsoft/msticpy/pull/715
* Fixing regular expression error for connection string in WorkspaceConfig by ianhelle in https://github.com/microsoft/msticpy/pull/706
* Fix documentation formatting, update steps for downloading msticpy by ccianelli22 in https://github.com/microsoft/msticpy/pull/720

New Contributors
* praveenjutur made their first contribution in https://github.com/microsoft/msticpy/pull/700
* ccianelli22 made their first contribution in https://github.com/microsoft/msticpy/pull/704

**Full Changelog**: https://github.com/microsoft/msticpy/compare/v2.7.0...v2.8.0

Updated method to dynamically fetch Azure endpoints (rather than relying on deprecated msrestazure).
Updated version of Insight data provider

More detailed release notes in the full release.

Main Changes

Two new TI Providers:
* CrowdSec (thanks to sbs2001)
* AbuseIPDB (thanks to rrevuelta)

Updated Data providers for Sentinel/Azure Monitor/Log Analytics and Kusto/Azure Data Explorer
These were introduced in v2.5.0 but are now the default drivers for these providers.

Query Editor
ipywidgets based query template editor - this is somewhat provisional so please be sure to test and
report bugs.

Updates to Authentication - esp for the AzureData and MicrosoftSentinel API modules
- You can now authenticate by supplying an AzureCredential as a `credential` parameter
- The connect methods for these support `cloud` parameter to specify different sovreign clouds
- The init and connect methods are instrumented with logging to help debug issues:

python3
import msticpy as mp
from msticpy.context.azure.sentinel_core import MicrosoftSentinel

mp.set_logging_level("INFO")
mssentinel = MicrosoftSentinel()
mssentinel.connect()


Other items
- MS Sentinel delete watchlist API added by mbabinski
- Splunk fixes added by Tatsuya-hasegawa

What's Changed
* Add CrowdSec TIProvider by sbs2001 in https://github.com/microsoft/msticpy/pull/673
* Added delete_watchlist_item method by mbabinski in https://github.com/microsoft/msticpy/pull/682
* Update pandas requirement from <2.0.0,>=1.4.0 to >=1.4.0,<3.0.0 by dependabot in https://github.com/microsoft/msticpy/pull/653
* Bump sphinx from 6.1.3 to 7.1.0 by dependabot in https://github.com/microsoft/msticpy/pull/686
* Add AbuseIPDB TIProvider by rrevuelta in https://github.com/microsoft/msticpy/pull/687
* Typo corrections in queries by ianhelle in https://github.com/microsoft/msticpy/pull/684
* Ianhelle/query editor 2023 04 21 by ianhelle in https://github.com/microsoft/msticpy/pull/685
* Few fix splunk driver by Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/688
* Ianhelle/mssentinel auth 2023 08 01 by ianhelle in https://github.com/microsoft/msticpy/pull/690
* Updating timeline docs to prioritize pd accessors by ianhelle in https://github.com/microsoft/msticpy/pull/691
* Fix splunk uploader create index option by Tatsuya-hasegawa in https://github.com/microsoft/msticpy/pull/692
* v2.7.0 - changing new kql/sentinel drivers to be defaults by ianhelle in https://github.com/microsoft/msticpy/pull/696

New Contributors
* sbs2001 made their first contribution in https://github.com/microsoft/msticpy/pull/673
* mbabinski made their first contribution in https://github.com/microsoft/msticpy/pull/682

**Full Changelog**: https://github.com/microsoft/msticpy/compare/v2.6.0...v2.7.0.pre1