Some new data-related features in this release.
- Support for the new (still in preview at time of writing) Dynamic Summaries feature of MS Sentinel
- Added ability to create and use "ad-hoc" parameterized queries for data providers
- Simple search mechanism for finding queries
- Support for JSON queries for CyberReason
Support for Microsoft Sentinel Dynamic Summaries
Dynamic Summaries are a Sentinel feature that allow you to persist results of
query jobs in a summarized/serialized form. This might be useful for keeping
results of daily watch jobs, for example. We will be using it in MSTICPy notebooks
to publish more complex result sets from automated notebook runs.
MSTICPy operations available include:
- Retrieve list of current dynamic Summaries
- Retrieve a full dynamic summary
- Create a dynamic summary
- Delete a dynamic summary
- Update an existing dynamic summary
Examples:
python
list dynamic summaries
sentinel.list_dynamic_summaries()
create a dynamic summary in Sentinel
sentinel.connect()
sentinel.create_dynamic_summary(
name="My_XYZ_Summary",
description="Summarizing the running of the XYZ job.",
data=summary_df,
tactics=["discovery", "exploitation"],
techniques=["T1064", "T1286"],
search_key="host.domain.dom",
)
The MSTICPy support also includes a `DynamicSummary` class that lets you
manipulate dynamic summary objects more easily
python
can also import the class directly
from msticpy.context.azure.sentinel_dynamic import DynamicSummary
dyn_summary = DynamicSummary(....)
This example shows using the "factory" method - new_dynamic_summary
dyn_summary = sentinel.new_dynamic_summary(
summary_name="My new summary",
summary_description="Description of summary",
source_info={"TI Records": "misc"},
summary_items=ti_summary_df,
)
Add the local summary object to add to the Sentinel dynamic summaries.
sentinel.create_dynamic_summary(dyn_summary)
Retrieve a dynamic summary from Sentinel
dyn_summary = sentinel.get_dynamic_summary(
summary_id="cea27320-829c-4654-bbf0-b14367483418"
)
the return value is a DynamicSummary object
dyn_summary
DynamicSummary(id=cea27320-829c-4654-bbf0-b14367483418, name=test2, items=0)
By default `get_dynamic_summary` returns the header data for the summary.
The next example shows how you can also fetch full data for the dynamic
summary (by adding `summary_items=True`). From the returned object,
you can convert the summary items to a pandas DataFrame.
> Note: fetching summary items is done via the Sentinel QueryProvider
> since the APIs do not support retrieving these.
python
dyn_summary = sentinel.get_dynamic_summary(
summary_id="cea27320-829c-4654-bbf0-b14367483418",
summary_items=True
)
dyn_summary.to_df()
<html><body>
<!--StartFragment-->
index | Ioc | IocType | QuerySubtype | Provider | Result | Severity | Details | TimeGenerated
-- | -- | -- | -- | -- | -- | -- | -- | --
OTX | hXXp://38[.]75[.]37[.]1/static/encrypt.min.js | url | | OTX | True | 2 | {‘pulse_count’: 3, ‘names’: [‘Underminer EK’ | 2022-12-15 01:55:15.135136+00:00
VirusTotal | hXXp://38[.]75[.]37[.]1/static/encrypt.min.js | url | | VirusTotal | False | 0 | Request forbidden. Allowed query rate may ha | 2022-12-15 01:55:15.135136+00:00
XForce | hXXp://38[.]75[.]37[.]1/static/encrypt.min.js | url | | XForce
<!--EndFragment-->
</body>
</html>
You can also create dynamic summaries from a DataFrame and append
DataFrame records to an existing dynamic summary.
Read the full documentation in [MSTICPy Sentinel Dynamic Summaries doc](https://msticpy.readthedocs.io/en/latest/data_acquisition/SentinelDynamicSummaries.html)
New QueryProvider API to dynamically add a parameterized query.
MSTICPy has always supported the ability to run ad hoc text queries for different providers
and return the results as a DataFrame. Using a static query string like this is quick and easy
if you only want to run a query once but what if you want to re-run with different time
range or host name? A lot of tedious editing or string search/replace!
Adding a full query template to MSTICPy, on the other hand, is overkill for this kind of thing.
Dynamic parameterized queries are especially suited for notebooks - you can create an
in-line parameterized query and have it update with the new parameters every time
you run the notebook.
To use dynamic queries - define the query with parameter placeholders (delimited
with curly braces "{" and "}"), then create parameter objects (these handle any special
formatting for datetimes, lists, etc.).
You add the list of parameter objects along with the replaceable parameter values
when you run the query, as shown below.
python
intialize a query provider
qry_prov = mp.QueryProvider("MSSentinel")
define a query
query = """
SecurityEvent
| where EventID == {event_id}
| where TimeGenerated between (datetime({start}) .. datetime({end}))
| where Computer has "{host_name}"
"""
define the query parameters
qp_host = qry_prov.Param("host_name", "str", "Name of Host")
qp_start = qry_prov.Param("start", "datetime")
qp_end = qry_prov.Param("end", "datetime")
qp_evt = qry_prov.Param("event_id", "int", None, 4688)
add the query
qry_prov.add_custom_query(
name="get_host_events",
query=query,
family="Custom",
parameters=[qp_host, qp_start, qp_end, qp_evt]
)
query is now available as
qry_prov.Custom.get_host_events(host_name="MyPC"....)
[See Dynamically Adding Queries in MSTICPy Docs](https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProviders.html#dynamically-adding-new-queries)
QueryProvider - Query Search
As the number of queries for some providers grows, it has become more difficult to quickly
find the right query. We've implemented a simple search capability that lets you search
over the names or properties of queries. It takes four parameters:
- `search` - search terms to look for in the
query name, description, parameter names, table and query text.
- `table` - search terms to match on the target table of the query.
(note: not all queries have the table parameter defined in their metadata)
- `param` - search terms to match on a parameter name
- `case` - boolean to force case-sensitive matching (default is case-sensitive).
The first three parameters can be a simple string or an iterable (e.g. list, tuple)
of search terms. The search terms are treated as regular expressions. This
means that a the search terms are treated as substrings (if no other
regular expression syntax is included).
Find all queries that have the term "syslog" in their properties
python
qry_prov.search("syslog")
equivalent to qry_prov.search(search="syslog")
['LinuxSyslog.all_syslog',
'LinuxSyslog.cron_activity',
'LinuxSyslog.list_account_logon_failures',
...
[See Search queries in MSTICPY Docs](https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProviders.html#searching-for-a-query)
Support for JSON queries in Data Providers
FlorianBracq has updated the CyberReason data provider so that it supports JSON queries. The
mechanism that we used for KQL and SQL queries breaks JSON since it is a simple string substitution.
Other data sources that use JSON queries include Elastic - we are planning to leverage the same
mechanism to support parameterized Elastic queries in a future release.
Thanks FlorianBracq!
What Else has Changed?
* Kql query formatting by FlorianBracq in https://github.com/microsoft/msticpy/pull/595
* Fix minor linting issues in main by petebryan in https://github.com/microsoft/msticpy/pull/604
* Updated M365D and MDE data connectors with correct scopes when using delegated auth. by petebryan in https://github.com/microsoft/msticpy/pull/580
* Ianhelle/remove extranous nb 2022 11 28 by ianhelle in https://github.com/microsoft/msticpy/pull/588
* Enable native JSON support for Data Providers + move Cybereason driver to native JSON by FlorianBracq in https://github.com/microsoft/msticpy/pull/584
* Adding query search to data_providers.py by ianhelle in https://github.com/microsoft/msticpy/pull/587
* Fix typo by FlorianBracq in https://github.com/microsoft/msticpy/pull/606
* Ianhelle/mypy cache 2023 01 17 by ianhelle in https://github.com/microsoft/msticpy/pull/608
* Added API to QueryProvider to add a custom query at runtime by ianhelle in https://github.com/microsoft/msticpy/pull/586
* Bump sphinx from 5.3.0 to 6.1.3 by dependabot in https://github.com/microsoft/msticpy/pull/605
* Bump httpx from 0.23.0 to 0.23.3 by dependabot in https://github.com/microsoft/msticpy/pull/607
* Dynamic Summaries Sentinel API and DynamicSummary class. by ianhelle in https://github.com/microsoft/msticpy/pull/593
* Update sentinel_analytics.py list_alert_rules API version. by pensivepaddle in https://github.com/microsoft/msticpy/pull/592
**Full Changelog**: https://github.com/microsoft/msticpy/compare/v2.2.0...v2.3.0