Using Embedded Python with InterSystems IRIS

Since InterSystems IRIS 2022.1, Python has been fully integrated into the InterSystems IRIS kernel and can be used with similar performance to ObjectScript. This 'Embedded Python' modality means Python runs locally to the data, reducing the latency cost of transferring the data.

Embedded Python differs from other connection methods ( Connecting to InterSystems IRIS from Client-Side Python Applications page for more detail.) because the Python kernel is running on the same machine as the data.

To maximize learning from this guide, it is recommended to test the code provided on your own InterSystems IRIS instance. The easiest way to run an InterSystems IRIS sandbox is to use InterSystems IRIS Community Edition in a docker container, which can be run with the following command. For more information see running IRIS see Get InterSystems IRIS Community with Docker or Get InterSystems IRIS Community Edition with an Install Kit.

docker run --name my-iris --publish 1972:1972 --publish 52773:52773 -d intersystems/iris-community:latest-em

Throughout this guide, there will be references for IRIS classes, many of which you may wish to import onto your IRIS instance. For information on how to do this, see Intro To InterSystems IRIS Classes or Setting Up your Development Environment in VS Code.

A major benefit of using Python with InterSystems IRIS is allowing access to the wide ecosystem of Python Libraries available to be installed from PyPI. Packages can be installed with pip, however, they need to be installed to a specific target directory. Standard pip commands will result in an error message: error: externally-managed-environment ...

To install Python libraries for use in InterSystems IRIS classes, specify the install target with the --target flag. The target is the {IRIS-INSTALL-DIR}/mgr/python.

The following shows the install command for the package numpy with the standard install location.

pip install --target /usr/irissys/mgr/python numpy 

If you are calling into IRIS from Python Files, as described in the Calling IRIS from Python Files section below, you can also create and use virtual environments in the standard way, the details for this are included in that section.

There are several modalities for using Embedded Python with InterSystems IRIS, the choice of which will depend on the use case. Python methods can be called directly from ObjectScript. Alternatively, methods and class methods within Python classes can be defined in Python, or Python can be accessed in standalone Python files and connect directly to InterSystems IRIS. You can also write custom SQL functions in Python (this won't be covered in this guide here, for more information see the relevant section of documentation).

For Python-first development, it is recommended to use standalone Python files as this is the only Embedded Python modality that allows a native Python developer environment, including using linters, debuggers and complete error messages. However, other modalities allow easy integration of Python methods into existing workflows, including directly into ObjectScript code.

Python can be used from ObjectScript, including importing specific Python modules. This modality is best if you have an ObjectScript method, but require functionality which is easier to implement in Python. This uses the class %SYS.Python to import the Python module. The following example shows the standard Python Library base64 being used to decode an encoded string.

// Import base64 Python Library
set base64 =  ##class(%SYS.Python).Import("base64")

set encodedString = "SGVsbG8sIFdvcmxkIQ=="

// Decode the Base64-encoded string using the imported Python module
set decodedString = base64.b64decode(encodedString)

// Write the decoded string
write decodedString

In this example, a Python module base64 is imported and then the functions from this module can be used as expected.

To use built-in Python classes and functions, including datatypes, you can import builtins, the Python module of built in classes.

// Import the Python built-ins
set builtins = ##class(%SYS.Python).Import(builtins)

// Create a python list
set mylist = builtins.list()

do mylist.append(1)
do mylist.append(2)
do mylist.append("foo")
do mylist.append("bar")

// Use Python Print
do builtins.print(mylist)

// Outputs: [1, 2, 'foo', 'bar']

To use custom Python files, you need to add the path to the file to the Python `sys` path, which is where Python searches for modules. This is shown in detail in the Calling IRIS from Python Files section below.

Internal class methods and methods can be written in Python by adding a keyword tag [language=python] after a method definition. This method is useful for short methods or integrating methods into existing workflows.

Class packagename.PythonClass Extends %RegisteredObject
{

Property Name As %String;

Parameter CONSTANTNAME = 2;

ClassMethod PythonClassMethod(a As %Integer, b As %Integer) [ Language = python ]
{
 
	import iris
	
	## Access the constant
	constant = iris.cls(__name__)._GetParameter('CONSTANTNAME')
	
	value = (a+b) * constant
	
	s = f"The result of ' ({a} + {b}) X {constant} ' is {value}"
	print(s)
}

Method PythonMethod(pGreeting As %String) [ Language = python ]
{
    # Access Object properties
    print(pGreeting +" "+ self.Name)
}

}

You could use this class from the ObjectScript terminal like normal:

do ##class(packagename.PythonClass).PythonClassMethod(1, 2) 
// prints "The result of ' (1 + 2) X 2 ' is 6"
set obj = ##class(packagename.PythonClass).%New()

set obj.Name = "John"
do obj.PythonMethod("Hello")
// prints "Hello John"

Embedded Python can be used in standalone Python files, which can be called from within InterSystems IRIS classes or executed from the terminal in the same way as normal Python files. This method is best for full Python-first development, as it allows standard Python development tools (e.g. linters and debuggers) to be used.

Calling into IRIS from Embedded Python requires some environmental set-up to allow access to IRIS from Python. This section will show how this can be done from a bash terminal on the machine running InterSystems IRIS.

If you wish to skip the set-up steps, there is an Embedded Python Template available on the developer community which allows you to immediately build and run a docker container with the environment pre-configured, skipping the set up below.

If you are running InterSystems IRIS in a docker container as recommended, use the following command on your standard terminal (or Powershell if you are running Windows) to start a bash terminal within your docker container.

docker exec -it my-iris bash

Before using Embedded Python, certain environmental variables need to be set which may not be set by default. Run the following commands in the terminal to ensure the correct Python kernel is being accessed. This assumes InterSystems IRIS is installed in the default location, if this is incorrect, please change the IRISINSTALLDIR variable before running.

# Set the IRIS install location. This is the default for docker containers. 
export IRISINSTALLDIR=/usr/irissys

# Add the binaries to the PATH
export PATH=$PATH:$IRISINSTALLDIR/bin

# Set the default Python to the IRISPYTHON kernel
export PYTHONPATH=$IRISINSTALLDIR/lib/python

At this point, you can use Python, you need to use irispython as the command. If you try to import the iris module, the connection will likely be rejected, because the service is not enabled, and your credentials are not set.

Along with the install locations, to connect to IRIS from Python, we also need to set the credentials to log in to IRIS. If you are using a Docker container sandbox, you can use the following default Namespace and passwords as follows, otherwise change these commands to your user password. You can disable authentication on the embedded python, as detailed in the section below, this is not recommended for production.

# The Namespace to connect to 
export IRISNAMESPACE=USER

# Username
export IRISUSERNAME=SuperUser

# Password
export IRISPASSWORD=SYS

These will need to be set in every new terminal instance or login. To make these accessible by default, these can be added to .bashrc, which is loaded every time a new terminal is opened. A command to add them to your .bashrc is shown below. Alternatively, you can add these environmental variables to a Dockerfile or Docker run command (not shown).

cat >> ~/.bashrc << 'EOF'
export IRISINSTALLDIR=/usr/irissys
export PATH="$PATH:$IRISINSTALLDIR/bin"
export PYTHONPATH="$IRISINSTALLDIR/lib/python"
export IRISNAMESPACE=USER
export IRISUSERNAME=SuperUser
export IRISPASSWORD=SYS
EOF

# Load the .bashrc 
source ~/.bashrc

Finally, the service by which Python files call into InterSystems IRIS is called %Service_CallIn. This is, by default, disabled in a new InterSystems IRIS instance. Before connecting to InterSystems IRIS from Python (with Embedded Python), this service needs to be enabled. This can either be done from the terminal or the Management Portal.

To do this from the terminal, first open an IRIS terminal:

iris session iris

Then run:

// Change namespace to %SYS
set $NAMESPACE = "%SYS"

// Save the current settings to a new array called `prop`
do ##class(Security.Services).Get("%Service_CallIn",.prop)

// Enable the service
set prop("Enabled")=1

// Uncomment to Allow unauthenticated access (removes the need for environmental credentials)
// set prop("AutheEnabled")=48

// Modify the settings using our edited settings array
do ##class(Security.Services).Modify("%Service_CallIn",.prop)

From the Management Portal, you can go to System Administration -> Security -> Services -> Go, then select %Service_CallIn from the list, click the Service Enabled checkbox and press Save.

As mentioned in the prerequisites section above, global package installs with pip need to include a specific target. When running Python files which call into IRIS, you can create an activate a virtual Python environment, which can be used normally. Virtual environments are effective for separating requirements used for specific projects.

When you've activated a virtual environment (using the correct Python kernel), you can install packages into the environment with standard pip installs. This process is shown below:

# Create a virtual environment called .venv in the current directory
irispython -m venv .venv

# Activate the virtual environment
source .venv/bin/activate

# Install packages as normal
pip install numpy

After the set-up above, Python can be used as irispython, including creating a Python shell. For example, if we had the following function defined in a file at /home/irisowner/python_files/hello_world.py:

import iris
def main(name):

    res = iris.sql.exec("SELECT 'Hello From IRIS-SQL'")
    print([x for x in res])

    print("Hello World from Python")
    return "Hello World from "+name

if __name__=="__main__":
    print(main("Bash Terminal"))

We could run this with:

irispython /home/irisowner/python_files/hello_world.py

# Outputs: 
# [['Hello From IRIS-SQL']]
# Hello World from Python
# Hello World from Bash Terminal

You can also run Python files from IRIS classes. To do this, you need to use the Python module sys to add the path to the file to the module search path. The following IRIS class shows this using both an ObjectScript class method and a Python class method:

Class sample.RunPythonFile Extends %RegisteredObject
{

ClassMethod RunFromPython() As %Status [ Language = python ]
{
    # Import the Python sys module
    import sys
    
    # Add the path to the file to the module search path  
    sys.path.append("/home/irisowner/python_files")
    
    # Import the file
    import hello_world 

    # Call the main function
    result = hello_world.main("IRIS ClassMethod in Python")
    
    # Print Result
    print(result)
}

ClassMethod RunFromObjectScript() As %Status
{
    // Import the Python sys module
    set sys = ##class(%SYS.Python).Import("sys")
    
    // Add the path to the file to the module search path 
    do sys.path.append("/home/irisowner/python_files")

    // Import the file
    set helloworld = ##class(%SYS.Python).Import("hello_world")

    // Call the main function
    set result = helloworld.main("IRIS ClassMethod in ObjectScript")
    
    // Print Result
    write result,!
    quit $$$OK
}

}

We can then run these class methods from the IRIS terminal with:

do ##class(sample.RunPythonFile).RunFromObjectScript()
// Output: 
// [['Hello From IRIS-SQL']]
// Hello World from Python
// Hello World IRIS ClassMethod in ObjectScript

do ##class(sample.RunPythonFile).RunFromPython()
// Output: 
// [['Hello From IRIS-SQL']]
// Hello World from Python
// Hello World from IRIS ClassMethod in Python

The sections above show the different ways you can run Python code in InterSystems IRIS with Embedded Python. This section shows how the iris module in Embedded Python can be used to perform tasks in IRIS. The code in this section can be used irrespective of which of the above modalities you are using, assuming they are set up correctly.

This is meant to be a quick guide to the most common usage for the module, demonstrating how InterSystems IRIS can be controlled directly from Python code. For a complete reference to the module, see the InterSystems IRIS Python Module Reference Documentation.

The Embedded Python IRIS module is imported with import iris, without requiring installation. Please note  import iris is different in Server-side (embedded) Python and client-side Python through the DB-API or IRIS-native API.

To access internal classes, you simply refer to it using with iris.package.class, for example:

import iris

# Run a class method with:
# iris.PackageName.ClassName.ClassMethodName()

iris.packagename.PythonClass.PythonClassMethod() # The class defined above

print(iris._SYS.System.GetInstanceName()) # Prints "IRIS" or the current instance name

The main exception to this simple access syntax is that special characters in a class or method call, e.g. % and $ are replaced by an underscore, for example .%New() in ObjectScript becomes _New() in Python.

We can see this when instantiating class objects. If we had a persistent class sample.Person (run the code in the Using SQL section below to create this class) we could access it as follows:

import iris

new_person = iris.sample.Person._New()
new_person = "Jane Doe"
new_person.Age = 28

person._Save()

You can run ObjectScript code directly with iris.execute:

import iris

iris.execute("write 'Hello World from Objectscript'")
iris.execute("set ^demoGlobal(1) = 'foo'")
iris.execute("zwrite ^demoGlobal")

Like in ObjectScript, SQL can be run in "Embedded" style, where the SQL command is executed directly, or Dynamically, where the statement is prepared with placeholders for values, which can be added at run-time.

Embedded:

import iris

# Drop table (so file can be re-run)
iris.sql.exec("DROP TABLE IF EXISTS sample.Person")

# Create table
iris.sql.exec("CREATE TABLE sample.Person (Name VARCHAR(250), Age INTEGER)")

# Insert Value
iris.sql.exec("INSERT INTO sample.Person (Name, Age) VALUES ('Jane', 26)")

# Query
rs = iris.sql.exec("SELECT Name, AGE FROM sample.Person") 

for row in rs:
    print(row)

# Prints:
# ['Jane', 26]

Dynamic:

import iris

# Use ? as a placeholder
query = 'INSERT INTO sample.Person (Name, Age) VALUES (?, ?)'

# Prepare Query
stmt = iris.sql.prepare(query)

# Define data
people = [("John", 34), ("Amy", 27), ("Peter", 57)]

# Iterate over data
for person in people:

    # Execute statement with data as parameters for the ? placeholders
    stmt.execute(person[0], person[1])    

The SQL result set class also allows you to parse the results directly into a pandas DataFrame. This requires installing pandas first:

pip install --target /usr/irissys/mgr/python pandas

Then, you can output a DataFrame from a query results object with ResultSet.dataframe():

import iris

# Prepare statement using ? as a placeholder
stmt = iris.sql.prepare("SELECT Name, Age FROM sample.Person WHERE Age > ?" )

# pass in argument(s) for placeholder
rs = stmt.execute(age)

# Use the resultset.dataframe() function 
df = rs.dataframe()
print(df.head())

# Prints:
#     name  age
# 0   John   34
# 1  Peter   57