Calicarpa

AI Security

Protect your business against supply-chain attacks distributional shift poisoning attacks model stealing prompt injection detection evasion statistical biases model backdooring data leaks ransomwares data biases AI misuse

Production line with people and machines scaning

Artificial intelligence creates fantastic opportunities that can drastically increase the added value of your business.

But rushed integrations of poorly secured AI systems will expose your business to much greater risks.

Cyber risks must not be underestimated. Recall that the cost of cybercrime is evaluated at $11 trillion in 2023 alone.

Calicarpa can offer its unique, world-renowned AI security expertise to accompany your management and technical teams.

Our AI security solutions have been consistently published at the most prestigious scientific venues (NeurIPS, ICML, ICLR, etc).

Calicarpa's co-founders co-authored 5 out of the 9 papers on state-of-the-art model poisoning mitigations reported in the 2024 Trustworthy and Responsible AI report from the US National Institute of Standards and Technology.

Which code

does your applications run?

Above 70% of the code in the codebase of most industries' applications is open-source software. Unless your company is like no other, your applications are mostly composed of open-source code.

In spite of being praised for its higher security and software quality, open-source is from year to year increasingly more used as a vector of supply chain attacks.

And this is leaving aside outdated or unmaintained open-source dependencies, an issue affecting 90% of the applications' codebases, which like legacy code definitely harms application security.

Which infrastructures

run your applications?

Supply chain attacks may impact your developers first. This is a serious risk, that begins with the mere installation of a dependency, and goes on throughout the software development cycle, with each compilation and each testing execution. Such supply chain attacks may easily steal your intellectual property and tamper with your infrastructures.

And of course, if you distribute software to run on-premises, the supply chain risks your applications suffered also carry on to your customers, in addition to potential vulnerabilities introduced by e.g. legacy code. Reputational and legal costs could ensue...

Limit the attack surface exposed to vulnerable or malicious code

with in-application component isolation and privilege reduction.

Complex software almost inevitably include vulnerabilities; even established and actively-maintained open-source projects. Legacy code and dependencies may carry invisible, lingering vulnerabilities throughout your infrastructures.

Malicious actors have leveraged, and will keep leveraging vulnerabilities and undue trust in the supply chain for the purpose of extorsion, IP theft and overall destabilization.

Internalizing each of your dependencies, to fully and systematically review and rewrite legacy and unmaintained code, would be too costly for most organizations, and ultimately insufficient.

Our pragmatic approach is rather to contain the risks and threats, by limiting the privileges of each software component to its bare minimum.

from calicarpa import sandbox

sandbox.load("/path/to/configuration/file")

import foo

# foo runs with a restricted view of the system:

# - limited/rearranged view of the file system,

# - limited/translated/deactivated networking,

# - bounded resource consumptions (CPU, RAM, etc),

# - restricted syscalls (limiting kernel attack surface),

# - and isolation from other processes.

# foo may have limited access to other loaded Python modules,

# and these other modules may access foo's functions and data

# as specified by the sandboxing configuration.

foo.bar().baz() # running within foo's isolated system view

Leveraging Python ubiquity and standardized interfaces

Our solution comes as a single library file, designed to be easily added to (and removed from) existing Python packages, be it your own code or dependencies.

This is possible thanks to the versatile and reflective Python data model, along with a well-defined, extensive and extensible-by-design standard library, covering e.g. data serialization and core import mechanisms.

In-application isolation without code change

Common container solutions do not isolate individual components. A single compromised dependency thus compromises the entire application. Our solution runs different components isolated in different sandboxes, transparently re-interconnecting these components from across their respective sandboxes.

Beyond isolating parts of your own software, our solution can also isolate internal components of large dependencies, e.g. machine learning frameworks.

Native Linux sandboxing made easy

Building upon the security primitives and administration interfaces of the Linux kernel (namespaces, secure computing mode, control groups, etc), our library offers native sandboxing capabilities with a straightforward Python API.

This means not only Python scripts, but any library and application can be sandboxed. This capability opens up many use cases; see below some examples.

Vulnerable package example

Let's consider a generic, network-facing HTTP service connected to a database. This service logs every HTTP request but, reminiscent of CVE-2021-44228, the logger is vulnerable to a format string attack ultimately offering remote, arbitrary code execution.

An attack payload would execute in the logger. Without protection, an attacker would then inherit all of the application privileges. In this example: database tampering, network exfiltration, etc.

Such an attack would be thwarted if the logger only had minimum privileges instead, e.g. append-only to an already open file/socket.

logger.py

database.py

network.py

def log(request):

# Vulnerable code here,

# e.g. format string attack

# reminiscent of Log4Shell.

def is_allowed(request):

# Check write permission for the request

def update(request):

# Discard unauthorized requests

if not is_allowed(request):

raise PermissionError

# Application-dependent processing

import database

import logger

# Called upon HTTP POST request

def process_http_post(request):

logger.log(request)

database.update(request)

Malicious dependency example

Unlike vulnerable dependencies, malicious ones execute malware as soon as they are loaded, without requiring a subsequent trigger. The main implication is that malicious code will also be executed in development environments.

While supply chain attacks affect every (software) industry, machine learning and data analysis may be among the most exposed ones. Factors include:

documentations and resources often hardly raise security awareness,
model, dataset and code sharing are widespread and appear harmless.

Did you know machine learning datasets are not "just data" in practice?

dataset.py

model.py

training.py

# Malicious code

# Anything goes here

class MyModel:

# Model implementation

# Not relevant for this example

import model

import dataset

# Malicious code already ran

# at this point!

def main():

# Model training loop

# skipped for brevity

Calicarpa has been founded by experts in information security and science communication. Over the last 7 years, we initiated the field of robust distributed machine learning and developed practical algorithms and software systems.

We have been advancing the state-of-the-art consistently, publishing and presenting our research on machine learning vulnerabilities and defenses at the most prestigious conferences.

Please find asidebelow a selected list of our publications.