Tm1py

Highlights

1. Support for TM1 v12

TM1py now works with TM1 v12.

python
with TM1Service(
address="us-east-2.aws.planninganalytics.ibm.com",
api_key="AB4VfG7T8wPM-912uFKeYG5PGh0XbS80MVBAt7SEG6xn",
iam_url="https://iam.cloud.ibm.com/identity/token",
tenant="YA9A2T8BS2ZU",
database="Database") as tm1:
print(tm1.server.get_product_Version())

by rclapp in https://github.com/cubewise-code/tm1py/pull/1000

2. CRUD operations to manage servers on v12

python

with ManageService(domain=domain, root_client=root_client, root_secret=root_secret) as manager:
manager.create_database(instance_name="instance name",
database_name="database_name",
product_version="12.0.0",
number_replicas=1,
cpu_requests="1000m",
cpu_limits="2000m",
memory_limits="2G",
memory_requests="1G",
storage_size="20Gi")

manager.scale_database(instance_name="instance name", database_name="database name", replicas=2)

manager.create_database_backup(instance_name="instance name",
database_name="database name",
backup_set_name="my backup")


by rclapp

3. Asynchronous `execute_mdx` functions

Speed up your MDX executions by assigning more than one worker-thread in TM1.

python
mdx = """
SELECT
{TM1SubsetAll([Big Dimension])} ON ROWS,
{TM1SubsetAll([Small Dimension])} ON COLUMNS
FROM [Big Cube]
"""

with TM1Service(**tm1params) as tm1:
cells = tm1.cells.execute_mdx(
mdx=mdx,
leverage 4 worker threads in TM1 for the extraction
max_workers=4,
parallelization on rows axis
async_axis=1)


by vmitsenko in https://github.com/cubewise-code/tm1py/pull/935 and https://github.com/cubewise-code/tm1py/pull/1030


4. Hierarchy updates from data frames

Create and update TM1 hierarchies directly from pandas dataframes.

Stores | ElementType | Alias:a | City:s | Square Footage:n | level001 | level000 | level001_weight | level000_weight |
|:---------|:--------------|:-----------------|:--------------|-------------------:|:------------|:-----------|------------------:|------------------:|
S151 | Numeric | Boardwalk Games | New York City | 120 | USA | World | 1 | 1 |
S143 | Numeric | Strategy Vault | Zurich | 250 | Switzerland | World | 1 | 1 |
S811 | Numeric | Cardboard Castle | Sydney | 80 | Sydney | World | 1 | 1 |

python
columns = ["Stores", "ElementType", "Alias:a", "City:s", "Square Footage:n", "level001",
"level000", "level001_weight", "level000_weight"]
data = [
['S151', "Numeric", "Boardwalk Games", "New York City", 120, "USA", "World", 1, 1],
['S143', 'Numeric', "Strategy Vault", "Zurich", 250, "Switzerland", "World", 1, 1],
['S811', 'Numeric', "Cardboard Castle", "Sydney", 80, "Sydney", "World", 1, 1],
]

with TM1Service(**tm1params) as tm1:
tm1.hierarchies.update_or_create_hierarchy_from_dataframe(
dimension_name="Stores",
hierarchy_name="Stores",
df=DataFrame(data=data, columns=columns)
)



<img width="552" alt="image" src="https://github.com/cubewise-code/tm1py/assets/15182131/2b098efd-a375-47f1-94f9-4a50a0b0ea49">


by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/944 and https://github.com/cubewise-code/tm1py/pull/1011



New Features, Improvements, and Bugfixes


* Improve `get_elements_dataframe`with optional `use_blob` by MariusWirtz in 908 and https://github.com/cubewise-code/tm1py/pull/1040
* Allow spreading on `write` with `use_blob` & `use_ti` by Kevin-Dekker in https://github.com/cubewise-code/tm1py/pull/909
* Add `proxies` arg to `TM1Service` by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/911
* Show parameters on decorated functions in PyCharm by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/913
* Cater to breaking change in urllib3 2.0 release by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/918
* Fix wrong position brackets in `extract_cellset_raw` by vmitsenko in https://github.com/cubewise-code/tm1py/pull/923
* Add `keep_existing_attributes` to dimension update function to fix 922 by hermie64 in https://github.com/cubewise-code/tm1py/pull/924
* Enforce non-async request on project PUT to fix 872 by nicolasbisurgi in https://github.com/cubewise-code/tm1py/pull/925
* Add `get_edges_under_consolidation` Function by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/927, https://github.com/cubewise-code/tm1py/pull/930
* Add method to delete elements by Kevin-Dekker in https://github.com/cubewise-code/tm1py/pull/939
* Avoid failure on attempt to write empty cellset by rclapp in https://github.com/cubewise-code/tm1py/pull/947
* Delete multiple elements using TI by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/940
* Make JSON parsing for Chore more robust by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/950
* Make JSON parsing for NativeView more robust by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/955
* Remove temp view in `_execute_view_csv_use_blob` by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/959
* Added clear view argument to write_through_blob by cubewise-gng in https://github.com/cubewise-code/tm1py/pull/964
* Add `arranged_axes` arg to `execute_view_csv` functions by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/961
* Allow optional weight columns on `get_elements_dataframe` function result by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/975
* Add new methods to expose cell partitioning by rclapp in https://github.com/cubewise-code/tm1py/pull/976
* Allow usage of `get_attribute_of_elements` function with numpy arrays by MBonotto in https://github.com/cubewise-code/tm1py/pull/987
* Add search function `search_error_log_filenames` for Process Error Log Filenames by adscheevel in https://github.com/cubewise-code/tm1py/pull/1004
* Include `clear_view` arg in `write_dataframe` by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/1012
* Add `replace_element` function to Hierarchy class by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/1014
* Add `update_or_create` function in ElementService.py by yyzz1010 in https://github.com/cubewise-code/tm1py/pull/1018
* Add `allow_empty_alias` argument to `get_elements_dataframe` function by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/1013
* Add `execution_path` property to `Chore` class by wimgielis in https://github.com/cubewise-code/tm1py/pull/1021
* Redesign legacy `require_admin` decorators to more granular `require_data_admin` etc. checks by adscheevel in https://github.com/cubewise-code/tm1py/pull/1016
* New `extract_cellset_axes_raw_async`, `extract_cellset_cell_async` and `extract_cellset_axes_cardinality` by vmitsenko in https://github.com/cubewise-code/tm1py/pull/957
* Add function `clear_with_dataframe` to clear cubes with dataframe https://github.com/cubewise-code/tm1py/issues/1006 by 151N3 in https://github.com/cubewise-code/tm1py/pull/1026, https://github.com/cubewise-code/tm1py/pull/1032
* Add function `get_values` by raeldor in https://github.com/cubewise-code/tm1py/pull/1020
* Fix issue on `restore_from_file` function by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/1039
* Redesign TM1py services architecture by rclapp and adscheevel in https://github.com/cubewise-code/tm1py/pull/1042
* Substite nan attribute values with sensible values in `execute_mdx_dataframe` when `include_attributes` is `True` by Kevin-Dekker in https://github.com/cubewise-code/tm1py/pull/1045
* Add `search_string_in_name` function to search for files in TM1 by adscheevel in https://github.com/cubewise-code/tm1py/pull/1049

Stats
- Contributors: 15
- Commits: 178
- Changed Files: 72
- Added Lines: 7,205
- Deleted Lines: 1,643

How to upgrade TM1py


pip install tm1py --upgrade


New Contributors
* vmitsenko made their first contribution in https://github.com/cubewise-code/tm1py/pull/923
* hermie64 made their first contribution in https://github.com/cubewise-code/tm1py/pull/924
* MBonotto made their first contribution in https://github.com/cubewise-code/tm1py/pull/987
* yyzz1010 made their first contribution in https://github.com/cubewise-code/tm1py/pull/1018
* wimgielis made their first contribution in https://github.com/cubewise-code/tm1py/pull/1021
* 151N3 made their first contribution in https://github.com/cubewise-code/tm1py/pull/1026


**Full Changelog**: https://github.com/cubewise-code/tm1py/compare/1.11.1...2.0

Fixes

* Add `proxies` arg to `TM1Service` 911
* Show parameters on decorated functions in PyCharm e.g. `execute_mdx_dataframe` 913
* Cater to breaking in change in urllib3 2.0 release 918

What's Changed
* Fix encoding issue in `execute_mdx_dataframe` with `use_blob` by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/904
* Issue/tcp keepalive by macsir in https://github.com/cubewise-code/tm1py/pull/902


**Full Changelog**: https://github.com/cubewise-code/tm1py/compare/1.11...1.11.1

Highlights

1. Improved performance https://github.com/cubewise-code/tm1py/pull/882, https://github.com/cubewise-code/tm1py/pull/885

With this release TM1py makes better use of existing capabilities in the TM1 server to handle large read and write operations.

> The `write` function

Pass `use_blob=True` to the `write` or `write_dataframe` function to use the new optimized write mode.

python
with TM1Service(**params) as tm1:
cells = {
('Actual', 'Germany', 'T Series 4.0 L Sedan', 'Units', 'Jan'): 1500,
('Actual', 'Germany', 'T Series 4.0 L Sedan', 'Units', 'Feb'): 2100,
('Actual', 'Germany', 'T Series 4.0 L Sedan', 'Units', 'Mar'): 1100
}
tm1.cells.write("SalesCube", cells, use_blob=True)


On a sample of 1 million cell updates, this performs on par with Turbo Integrator and up to 6x faster than the previous `write` function. https://github.com/cubewise-code/tm1py/pull/882#issuecomment-1487440628

-----

> The `write_async` function

The `write_async` function now makes use of `use_blob` by default. The number of parallel worker threads in TM1 can be controlled through the `max_workers` argument. A reasonable value for `slice_size` needs to be provided.

E.g. if you are expecting 1 million cell updates, a reasonable choice would be `10` for `max_workers` and `100000` for `slice_size`.


python
with TM1Service(**params) as tm1:
tm1.cells.write_async(
cube_name="SalesCube",
cells=cells,
slice_size=100_000,
max_workers=10)


On a sample of 1 million cell updates, this performs ~5 times better than plain Turbo Integrator and ~5 times better than the previous `write_async` function. https://github.com/cubewise-code/tm1py/pull/882#issuecomment-1487440628

-----

> The `execute_mdx_dataframe`, `execute_view_dataframe` functions

Pass `use_blob=True` to the `execute_mdx_dataframe`, and `execute_view_dataframe` functions to use the new optimized read mode.

python
with TM1Service(**sdata_params) as tm1:
df = tm1.cells.execute_view_dataframe(cube_name="Sales Cube", view_name="Default", private=False, use_blob=True)


On large data sets this performs 20% to 40% better. It also reduces the memory footprint by ~70%. https://github.com/cubewise-code/tm1py/pull/885#issuecomment-1494983649



2. New `shaped` argument in the `execute_view_dataframe` function https://github.com/cubewise-code/tm1py/pull/893

Pass `shaped=True` to the `execute_mdx_dataframe`, and `execute_view_dataframe` functions to retrieve the data frame in the original shape of the cube view.

<img width="859" alt="image" src="https://user-images.githubusercontent.com/15182131/231799551-e6ce281e-0b4f-4d37-aa04-dfb9d0f53ffb.png">


python
with TM1Service(**sdata_params) as tm1:
df = tm1.cells.execute_view_dataframe(
cube_name="Sales Cube",
view_name="Default",
shaped=True,
use_blob=True)


| | account1 | Jan | Feb | Mar | Apr | May | Jun | Jul | Aug | Sep | Oct | Nov | Dec |
|---:|:---------------|-----------:|-----------:|-----------:|-----------:|-----------:|-----------:|-----------:|-----------:|-----------:|-----------:|-----------:|-----------:|
| 0 | Gross Margin% | 57.5155 | 56.4074 | 59.1279 | 59.5385 | 57.7353 | 57.1263 | 58.4302 | 59.7887 | 56.9305 | 56.9046 | 59.8059 | 56.2537 |
| 1 | Price | 19360 | 19532 | 19218.5 | 19631 | 19941 | 19870 | 19365 | 19216 | 19241 | 19099.5 | 19883.5 | 19432.5 |
| 2 | Units | 1 | 2 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 1 | 1 |
| 3 | Sales | 19.36 | 39.064 | 38.437 | 19.631 | 19.941 | 19.87 | 19.365 | 19.216 | 19.241 | 38.199 | 19.8835 | 19.4325 |
| 4 | Variable Costs | 8.225 | 17.029 | 15.71 | 7.943 | 8.428 | 8.519 | 8.05 | 7.727 | 8.287 | 16.462 | 7.992 | 8.501 |



_Please note the months on the columns in the table above_


3. Re-authenticate after session timeout https://github.com/cubewise-code/tm1py/pull/856

Now TM1py will re-authenticate and retry the operation once if the operation failed due to a session timeout.

This is helpful in situations where a TM1py script has been idle for a while and once it attempts to restart interaction with TM1 it fails due to a session timeout.

New Features, Improvements, and Bugfixes

* Fully support TM1 sandbox functionality by adscheevel in https://github.com/cubewise-code/tm1py/pull/844
* Support use of alternate hierarchies in `get_elements_dataframe` function by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/876
* Allow `dtype` to be inferred by pandas with argument `infer_dtype` in `execute_mdx_dataframe_shaped` by Kevin-Dekker in https://github.com/cubewise-code/tm1py/pull/879

Acknowledgments
Big thanks to cwffonseca, Kevin-Dekker, and adscheevel for contributing code to this release, and many others for reporting bugs and requesting new features.

How to upgrade TM1py


pip install tm1py --upgrade


**Full Changelog**: https://github.com/cubewise-code/tm1py/compare/1.10.2...1.11

What's Changed
* Implement `get_elements_dataframe` function by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/818 and https://github.com/cubewise-code/tm1py/pull/829
* Fix error in `execute_mdx_elements_value_dict` by meyersrl in https://github.com/cubewise-code/tm1py/pull/823
* add pyproject.toml / update gitignore by scrambldchannel in https://github.com/cubewise-code/tm1py/pull/826
* adding lamda expression to filter tuple by ldelberg in https://github.com/cubewise-code/tm1py/pull/831
* Create element attributes when creating hierarchy by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/833
* Add `uses_hierarchies` function to `DimensionService` by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/821
* fix to tm1projecttask by nicolasbisurgi in https://github.com/cubewise-code/tm1py/pull/835
* Fix handling of `]` in element names in `execute_mdx` by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/842

New Contributors
* meyersrl made their first contribution in https://github.com/cubewise-code/tm1py/pull/823
* ldelberg made their first contribution in https://github.com/cubewise-code/tm1py/pull/831

**Full Changelog**: https://github.com/cubewise-code/tm1py/compare/1.10.1...1.10.2

Fixes issue 819 that was introduced with release 1.10

What's Changed
* Add argument skip_sandbox_dimension to execute_mdx by cubewise-gng in https://github.com/cubewise-code/tm1py/pull/812
* Add option to pass element selection as iterator in get_value function by tobiaskapser in https://github.com/cubewise-code/tm1py/pull/814
* Fix `verify_version` function by MariusWirtz in https://github.com/cubewise-code/tm1py/pull/820

New Contributors
* cubewise-gng made their first contribution in https://github.com/cubewise-code/tm1py/pull/812

**Full Changelog**: https://github.com/cubewise-code/tm1py/compare/1.10.0...1.10.1