Acceleration library for CosmoTech cloud-based solution development.

Introduction

The CosmoTech Acceleration Library (CoAL) provides a comprehensive set of tools and utilities to accelerate the development of solutions based on the CosmoTech platform. It offers a unified interface for interacting with CosmoTech APIs, managing data, and integrating with various cloud services.

Main Components

csm-data

csm-data is a powerful CLI tool designed to help CosmoTech solution modelers and integrators interact with multiple systems. It provides ready-to-use commands to send and retrieve data from various systems where a CosmoTech API could be integrated.

# Get help on available commands
csm-data --help
# Get help on specific command groups
csm-data api --help

datastore

The datastore provides a way to maintain local data during simulations and comes with csm-data commands to easily send those data to target systems. It offers:

• Format flexibility (Python dictionaries, CSV files, Pandas DataFrames, PyArrow Tables)
• Persistent storage in SQLite
• SQL query capabilities
• Simplified data pipeline management

from cosmotech.coal.store.store import Store
from cosmotech.coal.store.native_python import store_pylist
# Initialize and reset the data store
my_datastore = Store(reset=True)
# Create and store data
my_data = [{"foo": "bar"}, {"foo": "barbar"}, {"foo": "world"}, {"foo": "bar"}]
store_pylist("my_data", my_data)
# Query the data
results = my_datastore.execute_query("SELECT foo, count(*) as line_count FROM my_data GROUP BY foo").to_pylist()
print(results)
# > [{'foo': 'bar', 'line_count': 2}, {'foo': 'barbar', 'line_count': 1}, {'foo': 'world', 'line_count': 1}]

CosmoTech API Integration

CoAL provides comprehensive tools for interacting with the CosmoTech API, allowing you to:

• Authenticate with different identity providers (API Key, Azure Entra, Keycloak)
• Manage workspaces and files
• Work with the Twin Data Layer for graph data
• Handle runners and runs
• Process and transform data
• Build end-to-end workflows

import os
from cosmotech.coal.cosmotech_api.connection import get_api_client
# Set up environment variables for authentication
os.environ["CSM_API_URL"] = "https://api.cosmotech.com"  # Replace with your API URL
os.environ["CSM_API_KEY"] = "your-api-key"  # Replace with your actual API key
# Get the API client
api_client, connection_type = get_api_client()
print(f"Connected using: {connection_type}")
# Use the client with various API instances
from cosmotech_api.api.organization_api import OrganizationApi
org_api = OrganizationApi(api_client)
# List organizations
organizations = org_api.find_all_organizations()
for org in organizations:
    print(f"Organization: {org.name} (ID: {org.id})")
# Don't forget to close the client when done
api_client.close()

Other Components

• coal: Core library with modules for API interaction, data management, etc.
• csm_data: CLI tool for data management and integration with various systems
• orchestrator_plugins: Plugins that integrate with external orchestration systems
• translation: Internationalization support for multiple languages

Getting Started

Installation

pip install cosmotech-acceleration-library

Basic Usage

Check out the tutorials directory for comprehensive examples of how to use the library:

• CosmoTech API Integration
• Data Store Usage
• csm-data CLI

Key Features

Cloud Service Integration

CoAL provides built-in support for various cloud services:

• Azure: Azure Data Explorer (ADX), Azure Storage, Azure Functions
• AWS: S3 buckets, and more
• Database Systems: PostgreSQL, SingleStore, and others

Data Management

• Load and transform data from various sources
• Store and query data locally
• Export data to different formats and destinations
• Manage datasets in the CosmoTech platform

Orchestration Integration

• Provides plugins that integrate with external orchestration systems
• Supports data transfer between orchestration steps
• Offers utilities for handling parameters and configurations
• Enables seamless integration with the CosmoTech platform during orchestrated workflows

Documentation and Tutorials

Comprehensive documentation is available at https://cosmo-tech.github.io/CosmoTech-Acceleration-Library/

Tutorials

• CosmoTech API: Learn how to interact with the CosmoTech API directly: authentication, workspaces, Twin Data Layer, and more.
• Data Store: The datastore is your friend to keep data between orchestration steps. It comes with multiple ways to interact with it.
• csm-data: Make full use of csm-data commands to connect to services during your orchestration runs.

Testing and Code Coverage

The CosmoTech Acceleration Library maintains a comprehensive test suite to ensure reliability and stability. We use pytest for testing and pytest-cov for coverage reporting.

Running Tests

To run the test suite:

# Install test dependencies
pip install -e ".[test]"
# Run tests with coverage reporting
pytest tests/unit/coal/ --cov=cosmotech.coal --cov-report=term-missing --cov-report=html

Coverage Reports

After running tests with coverage, you can view detailed HTML reports:

# Open the HTML coverage report
open coverage_html_report/index.html

We maintain high test coverage to ensure code quality and reliability. All pull requests are expected to maintain or improve the current coverage levels.

Test Generation Tools

To help maintain test coverage, we provide tools to identify untested functions and generate test files:

# Find functions without tests
python find_untested_functions.py
# Generate test files for a specific module
python generate_test_files.py --module cosmotech/coal/module/file.py
# Generate test files for all untested functions
python generate_test_files.py --all

These tools help ensure that every function has at least one test, which is a requirement for contributions to the project.

Contact

For support, feature requests, or contributions, please use the GitHub repository.