Tables

As you might expect, tables are 2D grids of your data, making it easy to view, manipulate and organize whatever you’ve logged.

The rows each represent a different log, and the columns represent the different fields (parameters and entries) of the log.

import unify
unify.log(x=0, y={"a": [1, 2, 3]}, msg="hello", score=0.123)
unify.log(x=1, y={"b": [4, 5, 6]}, msg="goodbye", score=0.456)

This will be shown as:

Note that the most recent log is always shown at the top of the table, with row numbers in an ascending manner from top to bottom. Columns are all strongly typed, and you can see the type by hovering over the column header.

​

Column Manipulation

​

Hiding Columns

Columns can easily be hidden by clicking the dropdown and pressing “hide column”. They can then be re-added by either:

• hovering between columns and pressing the ”+” button
• clicking the column visibility dropdown and toggling the column

GIF

​

Moving Columns

You can move columns by simply dragging them to where you want them.

​

Pinning Columns

You can pin columns to the left by dragging the left-most edge of the column towards the right (think Excel). This column will then be locked in place when you scroll left and right within the table.

​

Grouping

When it comes to data manipulation, grouping is probably the most useful and versatile feature. Consider the following set of (dummy) LLM evaluations:

questions = [
    "What is the weather in Paris?",
    "What is the weather in Tokyo?",
    "What is the weather in London?",
]

sys_msgs = [
    "You are a helpful assistant",
    "You are a helpful weather assistant",
    "You are a helpful assistant that can use tools",
]


for with_tool in [True, False]:
    for sys_msg in sys_msgs:
        with unify.Experiment(), unify.Params(
            with_tool=with_tool,
            sys_msg=sys_msg,
        ):
            for i, question in enumerate(questions):
                unify.log(
                    question=question,
                    score=i*0.25 +random.random()/2
                )

By default, the data is just shown flat, in the order it was logged (most recent at the top).

However, if we want to see the results of each unique experiment, we can simply group by the experiment column, and unfold the data we want to see. The reduction metrics are automatically calculated for each column.

Conversely, if we want to see which question was the most difficult, we can group by question instead.

We can see that the LLM seems to find London the easiest, then Tokyo, then Paris (again, a contrived example 😁).

Similarly, if we want to what the impact of a system message is, we can group by this column instead, and see the statics reduced across all experiments.

Combined, this makes it very easily to quickly probe the data across all of your experiments, enabling you to easily transcending the boundaries of rigid “experiments” flexibly.

​

Sorting

As you’d expect, sorting can also be applied to any column. Simply clicking “sort” will toggle the state between ascending, descending, and off. Rows are sorted in the order of the column sorts. For example, sorting by column A and then column B will prioritize sorting based on A, and only defer to B when the values in Column A are equal.

​

With Grouping [🚧]

When grouping is applied, the contents of each group are sorted as usual. However, the groups themselves are also sorted based on the reduction metric that is set in the bottom left. This is the reduction metric that is shown in each cell during grouping.

This makes it easy to quickly show the highest performing experiments at the top of the table (for example).

​

Filtering

So far, everything we’ve considered has been about viewing data. Filtering enables you to select a subset of the data that you want to view. Each column type has a different set of filter options.

Let’s consider the following data for self-proclaimed “10x” engineers on the team:

num_employees = 20
ages = [
    random.randint(20, 50)
    for _ in range(num_employees)
]
catchphrases = random.choices(
    [
        "hello... friend",
        "hell no",
        "did you ask o3?",
        "will do it tomorrow",
        "ask the intern",
    ],
    k=num_employees,
)
last_logins = [
    (datetime.now() - timedelta(
        days=random.randint(0, 5)
    )).isoformat()
    for _ in range(num_employees)
]
open_task_progress = [
    {
        task: random.random()
        for task in random.sample(
            [
                "add NextJS loader",
                "SQL migration",
                "refactor life",
            ],
            random.randint(0, 3),
        )
    }
    for _ in range(num_employees)
]

for age, catchphrase, last_login, otp in zip(
    ages, catchphrases, last_logins, open_task_progress
):
    unify.log(
        age=age,
        how_10x=random.random()*10,
        catchphrase=catchphrase,
        last_login=last_login,
        open_task_progress=otp,
        will_to_live=random.choice([True, False]),
    )

​

Base

Firstly, all columns support filtering the following:

• Exists: whether or not the field was actually included in the log
• Is None: if the field was included, but a value of None was written

Additionally, all columns support arbitrarily chaining filter expressions together using AND and OR operators. We will also soon support parentheses for nested filter expressions. Everything else is column specific.

​

Strings

• Includes: whether or not the string includes the substring
• Excludes: whether or not the string excludes the substring

​

Numbers

• ==, !=, >, >=, <, <=: standard comparison operators

​

DateTime

• >, <: standard comparison operators
• Relative Mode: select relative to the current time
• Absolute Mode: select an absolute point in time

​

Booleans

• True: is true
• False: is false

GIF

​

Lists and Dicts

• Includes: whether or not the list includes the item / dict includes the key
• Excludes: whether or not the list excludes the item / dict excludes the key

GIF

​

Groups [🚧]

Groups can also be directly filtered. This is useful if you want to select a specific subset of experiments, for example. The group filtering is applied independently to the standard column-wise filtering. If you want to propagate the group filters to the columns filters, you can simply press “propagate” in the group filter dropdown.

​

Global [🚧]

Filters at the column level apply the filter directly to that column, and each column filter is effectivrely stitched together with an AND operator. However, you can write your own expressive filter expressions by the global filter, using basic Python syntax, using the column names as variable names.

unify.log(x=-1, y=-1)
unify.log(x=-1, y=1)
unify.log(x=1, y=-1)
unify.log(x=1, y=1)

Let’s imagine we want to filter to find the logs where x*y > 0. We couldn’t achieve this with column filters, but we can with the global filter, using an expression such as x*y>0 or (x>0 and y>0) or (x<0 and y<0).

We can also switch the global filter to “search” mode, which simply searches all cells for a matching substring, after casting all contents to string form (including dicts, lists, etc).

​

Derived Columns

Derived columns make it possible to create new columns based on existing ones. Again, all general Python syntax is supported.

For example, we could derive a new column that compute the length of the vector [x, y].

for _ in range(20):
    unify.log(x=random.random(), y=random.random())

We just need to provide the expression (x**2 + y**2)**0.5:

Filtering, grouping, and sorting all work on derived columns, just like any other. However, derived columns also support updating their equation.

GIF

​

Joins [🚧]

We will soon support creating new tables by joining existing ones 🔗

For example, this will make it easy to compare multiple experiments side by side, even in cases where the example ids in the test sets have changed across experiments (preventing the ability to do simple row-wise correspondence).

Stay tuned 📡 👀