​
Sign up to the platform

The first step is to create an account via our console. This is just a one time requirement, in order to create your unique user ID and API key. The entire console is replicated in the on-prem offering, all deployed locally.

​
Get access to the docker images

All images for on-prem deployment are available on Docker Hub. These are private images, with access granted to users on our Enterprise Plan. If you would like to get access for on-prem deployment, then get in touch with us!

​
Pull the images

In order to set up on-prem, you need to pull the following 3 images:

  • unifyai/orchestra-on-prem:latest: The backend app
  • unifyai/dataset-evaluation-on-prem:latest: The dataset evaluation service
  • unifyai/unify-interfaces-on-prem:latest: The console app

Given that these images are private, you first need to log into docker in your terminal before pulling the images. We will give your docker account read-only access to the images, as part of the enterprise onboarding process.

​
Local folder structure

We will also share a folder with you called on-prem, which contains six files, as explained below.

Firstly, the following three files store environment variables for the three different applications:

  • .env-console contains the necessary environment variables to set up the console app
  • .env-db contains the necessary environment variables to set up the database app
  • .env-orchestra contains the necessary environment variables to set up the backend app

There are also two .json files, as explained below:

  • sessionInfo.json contains the session information for the console app, it’s mainly used for displaying data on your profile page. The file should contain the following fields when we share the folder with you:

    • name: first name
    • email: email id
    • image: a link to the profile image

  • userInfo.json contains more information about the user currently using the app. This is something we need to do so to bypass the login process. The file should contain the following fields when we share the folder with you:

    • id: user id
    • name: first name
    • lastName: last name
    • image: a link to the profile image
    • email: the email id
    • createdAt: the date when the account was created
    • apiKey: the details about your api key
    • UserDetails: your organization and jobTitle (Optional)

Finally, there is also a Docker Compose file, which stores the configuration for orchestrating everything via Docker Compose, which spins up the containers:

  • docker-compose.yaml: contains the configuration for starting the Docker Compose service

​
Setting up

In order to set up, you would need to populate the .env_orchestra file with the credentials of the providers you would like to use:

  • For providers like OpenAI, Anthropic, etc. (which require a single API key), you would need to add your key corresponding to the ORCHESTRA_OPENAI_KEY variable and so on.
  • For using AWS Bedrock, you would need to add,
    • AWS_SECRET_KEY_ID
    • AWS_SECRET_ACCESS_KEY
    • AWS_REGION (the location can also be instead passed as the region field in /chat/completions)
  • For using VertexAI, you would need to add,
    • GOOGLE_APPLICATION_CREDENTIALS
    • ORCHESTRA_VERTEXAI_PROJECT
    • ORCHESTRA_VERTEXAI_LOCATION (again, the location can also be instead passed as the region field in /chat/completions)
  • For using Azure, you would need to add,
    • for Azure OpenAI endpoints (e.g. gpt-4o and gpt-4o-mini models),
      • AZURE_PROJECT
      • AZURE_OPENAI_API_KEY
    • for Azure AI endpoints (e.g. llama-3.1 models),
      • AZURE_REGION (you can instead pass this as the region field in /chat/completions)
      • AZURE_AI_LLAMA_3_1_8B_CHAT_API_KEY
      • AZURE_AI_LLAMA_3_1_70B_CHAT_API_KEY
      • AZURE_AI_LLAMA_3_1_405B_CHAT_API_KEY

Along with the above, you would also need to uncomment a line in the docker-compose.yaml that mounts the service account .json onto the container. For e.g. if application_default_credentials.json is the name of the file, then GOOGLE_APPLICATION_CREDENTIALS would be set as /app/src/application_default_credentials.json.

Once you’ve set the environment variables, you can now spin up the containers for the images by running docker compose up in the same folder as the docker-compose.yaml file. There’s a total of 5 containers that should be running:

  • console
  • orchestra
  • dataset_evaluation
  • db
  • redis

The containers should start running after that, you should wait until you see the log entry of on-prem-orchestra saying “Uvicorn running on http://0.0.0.0:8000” (as orchestra takes a few seconds to be up and running). After that is displayed, you’re set to start using the console!

​
Usage

The console should be usable at http://localhost:3000, you should be able to access features like uploading datasets, running evaluations, adding custom api keys, adding custom endpoints etc. You can also use all the endpoints in the API Reference by making calls to http://localhost:8000 instead of https://api.unify.ai

​
Python Client

In order to use the python client with the on-prem deployment, you need to set the UNIFY_BASE_URL environment variable as the address to the orchestra service. For e.g. if you’re using the python client in the same host as the orchestra service, then you should set the UNIFY_BASE_URL to http://localhost:8000.

​
Updates

The Docker images will continue to be updated regularly as the platform continues to evolve, with new features released on a weekly basis. In order to update your local environment based on the latest versions, you would need to:

  • Pull the latest images: same as above
  • Update the service to use the latest image: docker compose up -d --no-deps --build <service_name>

<service_name> can be one of console, orchestra or dataset_evaluation. The goal here is to make sure that your data remains intact, so we would advice against deleting the db service.

​
Vendor-Specific Instructions

Most of the major cloud providers include vendor-specific products and services, to streamline development on their respective cloud. Our on-prem offering can easily be extended to support and integrate with these inbuilt services. If you’re on an Enterprise Plan, just let us know your needs and we’ll be able to accommodate!

​
Google Cloud Platform (GCP)

Our Google Cloud Platform (GCP) specific features are outlined below, with instructions for how to enable them.

​
Pub/Sub

By default, the services deployed on-prem communicate using Redis. But you have the option to instead use GCP’s Pub/Sub service if it integrates better with your deployment environment.

In order to use GCP’s Pub/Sub messaging instead:

  • Set the PUBSUB_MESSAGING_TOPIC environment variable in the .env_orchestra as the name of the topic you’d like to set.
  • Set the PUBSUB_PROJECT_NAME environment variable in the same file as the name of the project you’re using.
  • <topic_name> will be set as the publisher topic in orchestra and the <topic_name>-sub will be set as the subscriber to that topic in the dataset-evaluation service.
  • You would also need to copy over your service account file to both the services for the same, this can be done by setting the volumes: of the docker-compose.yaml for both the services.

​
Support

Our Enterprise Plan includes full hands-on support and services, so of course feel free to ping us any time, and we’ll dive in and make sure everything is running smoothly on your end! 🧑‍💻

Deploy Your RouterEndpoint Status