Our previous post detailed how our backend was structured to support lightweight Temporal workflows as thin coordination layers (following the Saga pattern). Business logic doesn’t happen inside of workflows, but rather inside RPC endpoints that the workflows interact with.

AI Agents are workflows where the orchestrator happens to be a language model rather than a static script. A backend designed to support versatile lightweight workflows is already a backend designed to support AI agents. Whether those workflows are deterministic or stochastic in nature shouldn’t matter from your backend’s perspective.

Agents as Workflow Orchestrators

If your backend supports Temporal well, it means you have a system that can answer a large number of business questions with well-scoped endpoints that handle the gathering of data end-to-end, and you support chaining those questions together to handle complex business and analytics needs. Doesn’t that sound exactly like an agent workflow?

What does make an AI different than Temporal is trust. Your workflows were written by a human - or at least code reviewed by one. AI are not (yet) accountable or predictable like a human is, so they can’t have the same access scope that your workflows have. Agents represent a novel security threat; a highly capable and potentially hazardous actor that you deploy within your own infrastructure. Your agent will leak everything and break things. Build first with the concept that AI agents are maximally hostile, and add trust from there.

Agents are operating on your developer’s machine, potentially with a developer’s internet access and permissions. You don’t want it firing off an email containing customer’s PII, and you don’t want it deleting production customer records because it thinks they’re causing a bug. Your backend access layer needs to make this structurally impossible. Prompt engineering is not enough.

This is why giving your agent unrestricted access to a tool like https://github.com/fullstorydev/grpcurl is a mistake. Our solution was to deploy an MCP proxy, https://github.com/Basic-Capital/grpcmcp, which we stood up with kubernetes. Many MCPs can be hosted locally on a developer’s machine, but our MCP server is acting both as a standard tool use helper and as access control, so it needs to be hosted in an environment that an agent can’t modify. This proxy reflects a subset of our endpoints that we’ve demarcated as ai-accessible using just a single protobuf field and one protobuf annotation:

enum AiAccessLevel {
  AI_ACCESS_LEVEL_BLOCKED = 0;
  AI_ACCESS_LEVEL_ALLOWED = 1;        // both internal and customer-facing
  AI_ACCESS_LEVEL_INTERNAL_ONLY = 2;  // internal tools only
}

extend google.protobuf.MethodOptions {
  optional AiAccessLevel ai_access_level = 50003;
}

rpc GetDataAboutSomething(...) returns (...) {
  option (bcoptions.ai_access_level) = AI_ACCESS_LEVEL_ALLOWED;
}

This is the access control surface for AI in our codebase. Any method is BLOCKED by default. This makes it very easy to adjust and interpret access control in-code.

grpcmcp picks up the annotation through --require-method-option, a flag that filters which gRPC methods are exposed as MCP tools by proto method-option tag and value.

Enforcement lives in a single gRPC interceptor, which looks to the X509 certificate used against our MTLS encrypted RPC endpoints to determine the actor type and then enforces the access level as follows:

class AiAccessLevelInterceptor : ServerInterceptor {
    override fun <ReqT, RespT> interceptCall(...): Listener<ReqT> {
        val level = call.methodDescriptor.schemaDescriptor
            ?.let { it as? ProtoMethodDescriptorSupplier }
            ?.methodDescriptor?.options
            ?.getExtension(CustomOptions.aiAccessLevel)
            ?: AiAccessLevel.AI_ACCESS_LEVEL_BLOCKED

        val isExternalAgent = requestContext?.actorType ==
            ActorType.EXTERNAL_AI_AGENT

        val allowed = when (level) {
            AI_ACCESS_LEVEL_ALLOWED       -> true
            AI_ACCESS_LEVEL_INTERNAL_ONLY -> !isExternalAgent
            else                          -> false
        }

        if (!allowed) {
            call.close(Status.PERMISSION_DENIED, Metadata())
            return NoOpListener()
        }

        return next.startCall(piiRedactingCall, headers)
    }
}

Every PII field in our protos carries a debug_redact annotation — a built-in protobuf field option that we dual-purposed as the source of truth for "this field should never leave the wire to an agent". These annotations allow our RPC interceptor to trivially enforce redacting by reading the annotation and stripping the field before the response leaves the interceptor.

message Human {
  string uuid = 1;
  string given_name = 2 [debug_redact = true];
  optional string middle_name = 14 [debug_redact = true];
  string surname = 3 [debug_redact = true];
  Date date_of_birth = 4 [debug_redact = true];
  string email = 5 [debug_redact = true];
  optional string phone_number = 6 [debug_redact = true];
}

This gave us jarvis, the in-house name for our internal AI assistant. Rather than laboriously chaining together RPC calls by hand to solve production issues, Claude Code (or any other agentic harness) can talk to the jarvis MCP and one-shot diagnose and fix a lot of production problems using natural language.

Agents surface issues when they talk to your backend, and descriptive errors will let the agent know why a call failed or what an alternative endpoint to use would be. Well-named methods are easier for the agent to retrieve from context. All of these principles are table stakes in traditional software engineering, but in our new world they also happen to reduce query time and hallucinations. Agents shouldn’t be an excuse to fire off the slop cannons but an additive reward for good design.

The next step was rolling out a customer-facing version of this tool. A key insight is that there’s not a large distinction between a power user and a debugging developer; both want to chain together business logic to answer novel questions.

This gave us the idea for Vision, the customer-facing version of jarvis. Rather than living in a standalone proxy, Vision lives in our application code, and effectively defines an alternative backend interface to the user. Rather than the traditional approach of building out extensive, bespoke, feature-creeped visualizations for power users, an agent allows our clever customers to answer novel questions that we didn’t predict.

Agents as Frontends

In the same way that jarvis is a stochastic analogue to Temporal, Vision is a stochastic analogue to a user dashboard. It reuses the user’s auth to provide multi-tenancy, structurally preventing data leaks between users.

Vision runs as an in-process gRPC server inside the application backend, so the model’s tool calls flow through the same interceptor chain as production traffic from our user-facing dashboards; auth, PII redaction, request-context propagation, tracing, etc:

private val server: Server =
    InProcessServerBuilder.forName(serverName)
        .intercept(AiAccessLevelInterceptor())
        .intercept(AssistantRequestContextInterceptor(auditDao, env, jwtManager))
        .addAuthInterceptors(systemSecurity)
        .intercept(GrpcServerExceptionInterceptor(env, includeDebugInfo = true))
        .apply { services.forEach { addService(it) } }
        .build()

Every tool call the model issues is wrapped in call credentials that carry the customer’s real auth token, with the request context tagged as agent-driven.

internal class BearerTokenCredentials(
    private val token: String,
    private val identity: Identity,
    private val userUuid: String,
    private val parentContext: RequestContext? = null,
) : CallCredentials() {
    override fun applyRequestMetadata(...) {
        val requestContext = RequestContext(
            actorType  = ActorType.EXTERNAL_AI_AGENT,
            actor      = "Vision",
            actingFor  = userUuid,
            parent     = parentContext,
        )
        val metadata = Metadata()
        metadata.put(GrpcMetadataKeys.Authorization, "Bearer $token")
        metadata.put(GrpcMetadataKeys.IdentityType,  identity.type.name)
        metadata.put(GrpcMetadataKeys.IdentityUuid,  identity.uuid)
        applier.apply(metadata)
    }
}

Vision was powerful out-of-the-box because it was dogfooding flows that our engineering and operations teams had already been using. Developers are heavily incentivized to fix their own pain, so extending these developer tools to users gives us naturally battle-tested products that are agent-ready.

Most server applications start simple: a request comes in, the server does some work on a thread, the server sends back a response. Eventually, one inevitably runs into a class of engineering problem that doesn’t fit this model: a process that spans minutes, hours, or days. A payment that needs to wait for external settlement. An onboarding flow that blocks on identity verification. A multi-step order that can fail at any point and needs to pick up exactly where it left off.

The instinct is to reach for queues, cron jobs, and state machines stitched together with database flags. We’ve all built these systems. They work, right up until they devolve into piecing together state from databases and queues as you joylessly debug why something happened twice, or why a customer got stuck in a half-completed state that your recovery logic didn’t account for. A whole class of engineer was created in the 2010’s to reinvent the solution to this problem at every company.

At Basic Capital, “quality of life” for our technical staff is an important metric for success. Building at Basic Capital should inspire joy, it should feel like playing Factorio, and not be a death-by-thousand-chores slog.

Temporal offers a different model: write your long-running process as a straightforward function, and let the infrastructure handle durability. Your code reads like a simple procedural program, but it survives process restarts, network failures, and deployments without losing progress.

Our inaugural blog-post walks through how we set up Temporal in a Kotlin backend, the architectural insight that made it click for us, how it has yielded compounding returns for our engineering and ops teams, and how it’s laid the foundation for agentic AI products.

The Mental Model

The key insight powering Basic Capital’s foundational systems: a Temporal workflow doesn’t need to contain business logic. It can simply coordinate it.

In our architecture, the application backend is a set of gRPC services that owns all business logic, database access, and domain rules. Temporal workflows sit alongside them as orchestrators, calling into those services, waiting for external events, and ensuring that multi-step processes complete reliably.

This means:

• Application services remain the single source of truth for business logic

• Workflows are lightweight coordination layers, easy to read and reason about

• You get durability without polluting data models with orchestration artifacts

Setting it up with an example

Suppose you wanted to build a simple order fulfillment workflow at Basic Capital: an order is created, waits for payment confirmation from an external provider, and then gets fulfilled. If the payment succeeds, we ship it, otherwise we cancel the order.

Step 1: Define the Workflow Interface

Workflow interfaces live in a shared library so both the application server (which starts workflows) and the worker (which executes them) can reference them.

@WorkflowInterface
interface OrderFulfillmentWorkflow : RunnableWorkflow {
    @WorkflowMethod
    fun run(orderUuid: String)

    @SignalMethod
    fun paymentCompleted(success: Boolean)
}

A few things to note:

• @WorkflowInterface and @WorkflowMethod are Temporal annotations that define the entry point

• @SignalMethod defines a method that external systems can call to push data into a running workflow

Signals are what make Temporal workflows genuinely reactive. Instead of polling a database for state changes, your application backend pushes events directly into the running workflow.

Step 2: Define the Activities

Activities are where side effects happen. In our architecture, they’re thin wrappers around gRPC calls:

@ActivityInterface
interface OrderFulfillmentSteps : BasicCapitalActivity {
    fun getOrder(orderUuid: String): Order
    fun reserveInventory(orderUuid: String)
    fun fulfillOrder(orderUuid: String)
    fun cancelOrder(orderUuid: String, reason: String)
}

class OrderFulfillmentStepsImpl(
    private val orderClient: OrderServiceCoroutineStub,
) : OrderFulfillmentSteps {
    override fun getOrder(orderUuid: String): Order {
        return runBlocking {
            orderClient.getOrder(
                GetOrderRequest(orderUuid = orderUuid)
	).order
        }
    }
	
    override fun reserveInventory(orderUuid: String) {
        runBlocking {
            orderClient.reserveInventory(
                ReserveInventoryRequest(orderUuid = orderUuid)
            )
        }
    }
	
    override fun fulfillOrder(orderUuid: String) {
        runBlocking {
            orderClient.fulfillOrder(
                FulfillOrderRequest(orderUuid = orderUuid)
            )
        }
    }	
	
    override fun cancelOrder(orderUuid: String, reason: String) {
        runBlocking {
            orderClient.cancelOrder(
                CancelOrderRequest(orderUuid = orderUuid, reason = reason)
            )
        }
    }
}

Notice what’s not here: no database queries, no business rules, no validation. All of that lives in the gRPC service implementation. The activity is just a remote procedure call.

Step 3: Implement the Workflow

Now the satisfying part, the workflow implementation reads like pseudocode:

class OrderFulfillmentWorkflowImpl : OrderFulfillmentWorkflow {
    private val steps = getActivityStub(OrderFulfillmentSteps::class)
    private var paymentSucceeded: Boolean? = null

    override fun run(orderUuid: String) {
        // Reserve inventory immediately
        steps.reserveInventory(orderUuid)
		
        // Wait for payment provider callback — could be seconds or hours
        Workflow.await { paymentSucceeded != null }
		
        if (paymentSucceeded == true) {
            steps.fulfillOrder(orderUuid)
        } else {
            steps.cancelOrder(orderUuid, "Payment failed")
        }
    }
	
    override fun paymentCompleted(success: Boolean) {
        this.paymentSucceeded = success
    }
}

Workflow.await { paymentSucceeded != null } is the magic line. The workflow suspends here, not by burning a thread, but by persisting its state and shutting down. When a signal arrives via paymentCompleted(...), Temporal replays the workflow history, restores the state, evaluates the condition, and resumes execution.

If the worker crashes while waiting, nothing is lost. If you deploy a new version, it picks up where it left off. The workflow could wait for five seconds, five days, or even five weeks — the outcome is the same.

Step 4: Configure Task Queues

Activities and workflows are grouped into task queues, which map to pools of workers. We created our own abstraction on top of this to make it especially straightforward for engineers (or Claude Code).

You can configure multiple task queues, each with their own configured parallelism and rate limits.

Step 5: Wire it all up

On the application side, starting a workflow is a one-liner wrapped in a thin client we created for convenience:

class BasicCapitalTemporal(private val workflowClient: WorkflowClient) {

    // Idempotent workflow creation
    fun <T : RunnableWorkflow> startNewWorkflow(
        workflowId: String,
        workflowProcessor: KClass<T>,
        workflowInvocation: (T) -> Unit,
    ): WorkflowMetadata {
        val workflow = workflowClient.newWorkflowStub(
            workflowProcessor.java,
            WorkflowOptions.newBuilder()
                .setTaskQueue(workflowProcessor.getTaskQueue())
                .setWorkflowId(workflowId)
                .setWorkflowIdConflictPolicy(USE_EXISTING)
                .build(),
            )
	
        val execution = WorkflowClient.start { workflowInvocation(workflow) }
            return WorkflowMetadata(execution.workflowId, execution.runId)
        }

    // Signaling existing workflows
    fun <T : RunnableWorkflow> withExistingWorkflow(
        workflowId: String,
        workflowProcessor: KClass<T>,
        fn: (T) -> Unit,
    ) {
        val workflow =
            temporalClient.newWorkflowStub(
            workflowProcessor.java,
            workflowId,
        )
            
        fn(workflow)
    }
}

The USE_EXISTING conflict policy makes workflow creation naturally idempotent. Additionally, as a convention, we use our business objects’ unique identifiers as their workflowId’s.

From a gRPC endpoint, kicking off the workflow looks like:

// Inside your gRPC service handler
val order = orderDao.createOrder(request)

temporal.startNewWorkflow(
    workflowId = order.uuid,
    workflowProcessor = OrderFulfillmentWorkflow::class,
) {
    it.run(order.uuid)
}

// Inside the event handler for an external payment completion webhook event
temporal.withExistingWorkflow(
    workflowId = orderUuid,
    workflowProcessor = OrderFulfillmentWorkflow::class,
) {
    it.paymentCompleted(success = true)
}

That’s it. The webhook handler sends a signal, the workflow wakes up and continues.

Configuring Retry and Timeout Behavior

The Temporal SDK allows you to configure retry policies and timeouts:

private val retryOptions = RetryOptions.newBuilder()
    .setInitialInterval(Duration.ofSeconds(5))
    .setMaximumInterval(Duration.ofDays(1))
    .setBackoffCoefficient(2.0)
    .setMaximumAttempts(18)
    .build()
	
private val defaultActivityOptions = ActivityOptions.newBuilder()
    .setRetryOptions(retryOptions)
    .setStartToCloseTimeout(Duration.ofMinutes(5))
    .build()

The appropriate configuration is situationally dependent, but the arrangement that works for us today is an exponential backoff from 5 seconds to 1 day and 18 max attempts. A transient failure gets retried aggressively at first and gracefully backs off, covering roughly 5 days of retry window.

Why This Architecture Works

The pattern of “Temporal as orchestrator, gRPC services as executors” has a few properties that compound over time:

• Workflows stay simple - Because they only coordinate calls and manage state transitions, most of our workflows are under 50 lines. They’re easy to review, easy to test, and easy to reason about when something goes wrong.

• Business logic doesn’t scatter - Every domain operation lives in exactly one place: the gRPC service that owns it. Temporal never needs to know how inventory reservation works—it just knows to call something like reserveInventory(), and handle success or failure.

• The system is observable - Temporal’s UI shows every running workflow, its current state, its full event history, and every signal it’s received. When a workflow is stuck, you can see exactly which activity failed and how many times it’s been retried.

• Signals replace polling - Instead of workflows polling a database every 30 seconds to check if payment completed, our webhook handlers can simply send a signal directly to the workflow.

The Bigger Picture

The value of Temporal isn’t that it does something you can’t do with queues and state machines. It’s that it lets you express complex, long-running processes in code that looks boring: sequential, readable, and obviously correct. In production systems that handle real money, boring is exactly what you want.

We run 70+ workflow types in production, covering everything from account onboarding to trade settlement to SPV incorporation. Each one follows the same pattern: a workflow interface with signals, an activity interface that wraps gRPC calls, and an implementation that reads like a step-by-step runbook.

Think of it like automating a human sitting and stringing together API calls to run your business. This mental model is particularly powerful, as it has laid the foundation for us to build Agentic operations, allowing us to use natural language to orchestrate complex research and processes on top of our foundational platform. We’re excited to share more about this very soon.

At Basic Capital, we’re building a vertically integrated 401(k) and IRA product, a native record keeping system, brokerage & trading systems, credit origination systems, the world’s only Retirement Mortgage, and agentic plan administration. Our engineering challenges span financial systems, UI/UX, 3rd party integrations and partner API’s, data engineering, and AI infrastructure. Our bedrock makes building all of these things feel downright joyful, and we’re just getting started. If you’re interested in joining, we’re hiring.