Proposal: Introducing 'Spaces' - A Unifying Model for Concurrency, Isolation, and Performance

[axx83]

Summary

This proposal introduces a new, high-level architectural primitive for C# called a "Space". The core philosophy is to unify two historically conflicting worlds of programming into a single, cohesive platform: the productivity and safety of high-level managed languages (like C#) and the raw performance and control of low-level systems languages (like C++/Rust).

The "Space" model aims to solve fundamental architectural challenges, most notably the classic "Monolith vs. Microservices" debate. It would allow developers to design and reason about a system with the simplicity of a monolith while enabling it to be deployed with the scalability and independence of microservices.

---

I. The Core Philosophy & Problem Statement

Modern systems demand both complex business logic and high-performance data processing. Today, this often requires building polyglot systems, leading to increased complexity in communication (gRPC, REST), deployment, and developer skill sets. C# is a phenomenal language for productivity, but when absolute performance is needed (e.g., in a game engine's physics loop, a financial high-frequency trading model, or a data-intensive AI pipeline), developers often have to resort to C++ interop or separate services written in another language.

The "C# Plus" vision aims to eliminate this trade-off. It proposes evolving C# to gracefully span the entire spectrum from high-level, managed business logic to low-level, unmanaged, performance-critical computations, all within the same language and project.

---

II. The Foundational Concept: The "Space" Model

A Space is a completely isolated unit of computation. Think of it as its own "universe" with well-defined rules and a strict communication protocol for interacting with other universes.

The "Harmony" Space: The Central Orchestrator
At the heart of every application built with this model is a special, implicit Space known as Harmony.

• The Entry Point: The Harmony space serves as the application's entry point, analogous to the Main method in a traditional C# application. It's the root from which all other operations and Space interactions originate.
• The Main Syncer: More importantly, Harmony acts as the primary orchestrator or "main syncer". While individual Spaces can communicate with each other to perform complex, multi-stage tasks (potentially with their own local coordinators), all of these interactions are ultimately in service of a higher-level goal initiated and managed by Harmony. It is the conductor 🎼 that directs the symphony of independent Spaces.

Core Characteristics (Applicable to All Spaces)

• True Isolation: Each Space, including Harmony, possesses its own dedicated memory heap and its own thread(s). This is a stronger guarantee than a simple thread. By design, it eliminates entire classes of concurrency problems like race conditions and deadlocks between Spaces.
• Asynchronous Message Passing: Spaces communicate exclusively through asynchronous message passing. This mechanism is abstracted away from the developer behind intuitive new language keywords, making inter-Space communication feel natural and declarative.

---

III. The "Logic Spectrum": Fine-Grained Space Configuration

Not all tasks are created equal. The power of the Space model comes from the developer's ability to configure the execution characteristics of each Space declaratively, likely within the .csproj file. This creates a "spectrum of logic" where you can dial in the perfect trade-offs for each part of your application.

Configuration Dimensions:

• Mode (Execution Environment):
  • Managed: The standard C# experience with the Garbage Collector (GC) and full .NET ecosystem access.
  • Unmanaged: For maximum performance, operating without a GC, offering C++/Rust-like performance.
• Compilation (Code Generation):
  • AOT (Ahead-of-Time): Compiles directly to native machine code during the build process.
  • JIT (Just-in-Time): Uses the standard JIT compiler at runtime.
• Runtime (Process Model):
  • ThreadGroup: The Space runs as a group of threads within the same host process.
  • Isolate: The Space runs as a completely separate, isolated OS process, enabling microservices.
• Transport (Communication Protocol):
  • Defines the underlying mechanism used for communication between Spaces. This allows for significant performance tuning based on how the Spaces are deployed.
  • SharedMemory: Ultra-low latency communication for Spaces running within the same process (ThreadGroup).
  • gRPC: Robust, network-based communication for Spaces running in separate processes (Isolate).
  • NamedPipes: Efficient inter-process communication on the same machine.

Example Extended .csproj configuration:

<Project Sdk="Microsoft.NET.Sdk">

  <Space Include="BusinessLogic">
    <Mode>Managed</Mode>
    <Compilation>JIT</Compilation>
    <Runtime>ThreadGroup</Runtime>
  </Space>

  <Space Include="PhysicsEngine">
    <Mode>Unmanaged</Mode>
    <Compilation>AOT</Compilation>
    <Runtime>ThreadGroup</Runtime>
  </Space>

  <Space Include="ReportingService">
    <Mode>Managed</Mode>
    <Compilation>AOT</Compilation>
    <Runtime>Isolate</Runtime>
  </Space>

  <DefaultSpaceCom Transport="gRPC" />

  <SpaceCom From="BusinessLogic" To="PhysicsEngine" Transport="SharedMemory" />
</Project>

---

IV. Syntax and Developer Interaction

The developer experience is paramount. The interaction with Spaces should feel like a natural extension of C#.

Defining Spaces

A Space can be defined for an entire file or for a specific block of code.

// File-scoped definition
space PhysicsEngine;

public class Vector3 { ... }
public static class CollisionDetector { ... }

// Block-scoped definition
space BusinessLogic {
    public class OrderProcessor { ... }
}

"Direct" or "Lambda" Spaces

For small, one-off, performance-critical operations, a direct block can be used to create a temporary, inline Space with a specific configuration.

// In a managed Space...
public double[] ProcessData(double[] rawData)
{
    // Offload this heavy computation to a temporary, unmanaged, AOT-compiled Space
    // for maximum performance, without the overhead of a formal named Space.
    var result = direct(new SpaceConfig { Mode = Unmanaged, Compilation = AOT }) {
        // This code block is now in an unmanaged context.
        // It can't use managed types that require a GC.
        // It would operate on 'rawData' after it's been marshalled.
        // ... heavy processing ...
        return processedData;
    };
    return result;
}

Inter-Space Communication: swait and srun

To maintain the isolation model, new keywords are proposed for communication.

• swait (Space Wait): Asynchronously calls into another Space and waits for the result. This is intentionally distinct from await, which is used for I/O or async work within the same Space. swait signifies crossing an isolation boundary.

// From BusinessLogic Space
async Task<bool> IsValidPlacement(GameObject item)
{
    // 'swait' into the PhysicsEngine space, executing the code in that context.
    // The 'item' object would be serialized, sent, and deserialized.
    bool hasCollision = swait PhysicsEngine {
        DetectCollision(item.Position, item.Bounds);
    };
    return !hasCollision;
}

• Composition: swait calls can be composed logically.

// Wait for both to complete (like Task.WhenAll)
var (resultA, resultB) = swait SpaceA { Op1() } and swait SpaceB { Op2() };

// Wait for the first to complete (like Task.WhenAny)
var firstResult = swait SpaceA { LongOp() } or swait SpaceB { FastOp() };

• Custom Transports: The underlying communication mechanism (e.g., gRPC for Isolate Spaces, Shared Memory for ThreadGroup Spaces) can be overridden for specific calls.

// Force a high-performance shared memory transport for this specific call
var data = swait DataProcessingSpace { ProcessBigData() } 
           with { Transport = Transport.SharedMemory };

• srun (Space Run): A "fire-and-forget" mechanism to invoke a task in another Space without waiting for a result. This makes the developer's intent clearer than a discarded swait.

// Log this event in the background without blocking the current flow.
srun LoggingSpace {
    Log.Info($"User {userId} performed an action.");
};

Contextual Polymorphism: overspace

The same conceptual type may need vastly different implementations depending on its host Space's rules (e.g., managed vs. unmanaged). overspace is the mechanism that enables this, allowing a single type name to have unique data structures and behaviors in different contexts.

The primary way to achieve this is by using partial class/struct spread across multiple files, where each file declares a specific space context. The overspace keyword is mandatory on members to explicitly declare them as part of that Space's specific implementation, preventing accidental name collisions.

Example 1: Using partial class Across Multiple Files

// File: DataBuffer.Managed.cs
space BusinessLogic;

public partial class DataBuffer
{
    // The 'overspace' keyword is required here for this Space's implementation.
    private overspace List<byte> _buffer = new List<byte>();

    public overspace void Add(byte b) => _buffer.Add(b);
    public overspace int Count => _buffer.Count;
}

// File: DataBuffer.Unmanaged.cs
space PhysicsEngine;

public unsafe partial class DataBuffer
{
    // A completely different, unmanaged implementation for the same type.
    private overspace byte* _buffer;
    private overspace int _count;

    public overspace void Add(byte b) { /* ... pointer manipulation ... */ }
    public overspace int Count => _count;
}

Advanced Co-location with overspace <SpaceName>

For greater flexibility and to keep related logic together, the overspace keyword can be combined with a Space's name. This allows you to define a member for a different, specific Space from within any file, regardless of the file's primary space context.

This is extremely powerful for co-locating a type's default implementation alongside its highly specialized counterparts.

Example 2: Using Explicit Targeting in a Single File

// File: DataManager.cs
space BusinessLogic; // The default context for this file is "BusinessLogic"

public partial class DataManager
{
    // --- Implementation for BusinessLogic (managed, using implicit 'overspace') ---

    // The implicit 'overspace' keyword here applies to "BusinessLogic"
    private overspace List<byte> _data;
    public overspace int GetCount() => _data.Count;

    // --- Implementation for PhysicsEngine (unmanaged, using explicit 'overspace <SpaceName>') ---
    
    // Here, we explicitly target the "PhysicsEngine" Space for these members,
    // defining them right alongside the BusinessLogic implementation.
    private overspace PhysicsEngine unsafe byte* _data;
    private overspace PhysicsEngine int _count;
    public overspace PhysicsEngine int GetCount() => _count;
}

Regardless of the syntax used, the compiler and runtime treat BusinessLogic.DataBuffer and PhysicsEngine.DataBuffer as entirely distinct types that just happen to share a name. They do not need to have matching method signatures or fields.

The Cross-Space Marshalling Challenge

While the default mechanism is a safe, by-value copy (serialization), the model is designed to be extensible. Advanced scenarios could leverage specialized transports to enable zero-copy transfers, such as passing ownership of a memory block or using shared memory, for performance-critical paths.

Furthermore, the contextual polymorphism introduced by overspace presents a unique data marshalling challenge. Because the same type (e.g., DataBuffer) can have completely different underlying data structures in different Spaces (e.g., a List<byte> in BusinessLogic vs. a byte* and int in PhysicsEngine), standard serialization is insufficient. A direct memory copy is impossible, and a simple field-by-field serializer would fail as the fields themselves don't match.

To address this, the model must include a mechanism for developers to define explicit Space-to-Space conversion routines. This logic would be co-located with the type definitions, ensuring that a type is responsible for its own representation across boundaries. This could be achieved through a new convention, such as a specially recognized constructor that takes the type's representation from another Space as its source.

Example: Defining a Cross-Space Converter

Let's expand the DataBuffer example. To allow it to be passed from the managed BusinessLogic Space to the unmanaged PhysicsEngine Space, we would define a special constructor within the PhysicsEngine's partial class definition:

// File: DataBuffer.Managed.cs
space BusinessLogic;

public partial class DataBuffer
{
    // The 'overspace' keyword is required here for this Space's implementation.
    private overspace List<byte> _buffer = new List<byte>();

    public overspace void Add(byte b) => _buffer.Add(b);
    public overspace int Count => _buffer.Count;
    
    // For the runtime to access the data during conversion
    internal overspace IReadOnlyList<byte> GetBuffer() => _buffer; 
}

// File: DataBuffer.Unmanaged.cs
space PhysicsEngine;

public unsafe partial class DataBuffer
{
    // A completely different, unmanaged implementation.
    private overspace byte* _buffer;
    private overspace int _count;

    public overspace void Add(byte b) { /* ... pointer manipulation ... */ }
    public overspace int Count => _count;

    // --- The explicit Space Converter ---
    // This constructor teaches the PhysicsEngine's DataBuffer how to build 
    // itself from the BusinessLogic's version.
    public overspace DataBuffer(overspace BusinessLogic DataBuffer source)
    {
        _count = source.Count;
        // Allocate unmanaged memory
        _buffer = (byte*)System.Runtime.InteropServices.Marshal.AllocHGlobal(_count);
        // Copy the data from the managed list to the unmanaged buffer
        for (int i = 0; i < _count; i++)
        {
            _buffer[i] = source.GetBuffer()[i];
        }
    }
}

The compiler and runtime would then use this converter automatically during an swait or srun call. When an object is passed from BusinessLogic to PhysicsEngine, the runtime would detect the existence of this specific constructor (public overspace DataBuffer(overspace BusinessLogic DataBuffer source)) and invoke it on the destination Space to reconstitute the object's state in the new, appropriate format.

This approach offers several key advantages:

• Explicit Control: Developers have full, unambiguous control over the conversion logic.
• Performance: The conversion can be highly optimized (e.g., using Span<T> and direct memory copies) for maximum performance.
• Type Safety: The entire process is type-safe and verified at compile time.
• Co-location: The conversion logic lives directly with the type's destination implementation, making the code easy to understand and maintain.

---

V. The Safety Model: "The Airlock"

To guarantee absolute isolation, communication between Spaces follows a strict "airlock" model, where everything passing through is controlled and reconstructed from scratch. This model is built on three fundamental rules:

• Strict Space Scoping
  Code is strictly scoped to its own Space and can only invoke functions defined within that context. While a class is treated as a universal concept that exists across all Spaces, an object's concrete manifestation. Its available methods and behaviors are unique to the Space it currently occupies.
• swait/srun is the Only Gateway
  All interaction must happen through the swait and srun keywords. They are the secure doors to the airlock, ensuring that all communication adheres to the system's rules and the safety model.
• Pure Data Transfer with Contextual Behavior
  This is the core rule of the safety model. When an object is passed across a Space boundary, its state (data) is copied, but its behavior (methods) is reconstituted entirely at the destination based on the local context.

To better illustrate the interaction with overspace, consider this example:

Class Definitions:

// File: MyClass.SpaceA.cs
space A;
public partial class MyClass 
{
    public void RunA() { /* Original logic, specific to Space A */ }
}

// File: MyClass.SpaceB.cs
space B;
public partial class MyClass 
{
    public void RunB() { /* New logic, specific to Space B */ }

    // Explicitly override RunA's behavior for Space B
    public overspace void RunA() { /* Specialized logic for RunA inside Space B */ }
}

Usage:

// Executing inside Space A:
var obj = new MyClass();
obj.RunA(); // Valid, calls Space A's original version.
// obj.RunB(); // Compile error! RunB() does not exist in Space A's context.

// Now, let's pass the 'obj' object to Space B:
srun B {
    // Inside this block, 'obj' has been reconstituted from the original's data.
    // It now adheres to the definition of MyClass within Space B.

    obj.RunB(); // Perfectly valid.
    obj.RunA(); // Also valid, but this now calls the 'overspace' version from Space B.
};

This example clearly demonstrates how an object's data is preserved across the boundary, but its behavior completely conforms to the rules of the Space in which it is currently executing.

---

VI. Proposed Implementation Path into Modern C#

A feature of this magnitude demands a clear, pragmatic, and non-disruptive integration strategy. This proposal envisions the "Space" model not as a rewrite of the C# language or the .NET runtime, but as a series of targeted, modular extensions that build upon the existing, mature infrastructure. The core principle is additive evolution, ensuring that the feature is a powerful, opt-in superset that respects the stability of the entire ecosystem.

---

The Default Space: Ensuring 100% Backward Compatibility

The cornerstone of this design is ensuring complete backward compatibility. To achieve this, all existing C# applications and libraries can be considered to operate exclusively within a single, implicit Default Space.

This Default Space is, for all practical purposes, the traditional .NET execution environment in use today. It runs on the main application thread(s), utilizes the standard Garbage Collector, and has access to the entire Base Class Library. From the compiler and runtime's perspective, a legacy C# application is simply a single-space system where none of the new cross-boundary rules or features apply.

Conceptually, this single-space execution can be thought of as a state of "harmony," where all code shares the same memory and rules without the complexity of isolation boundaries. Formalizing the existing environment as the Default Space guarantees that no changes are required for the millions of existing .NET codebases. They will continue to compile and run exactly as they do today, with no performance overhead or behavioral changes.

---

An Additive Approach to Compiler and Runtime Features

Integrating Spaces would require targeted additions to the compiler (Roslyn) and the runtime (CLR), designed as self-contained components that augment, rather than replace, core functionality.

Runtime (CLR) Enhancements

This approach learns from the lessons of past isolation mechanisms like AppDomains, but provides a far more lightweight, language-integrated, and performant model designed for modern concurrency patterns rather than legacy code sandboxing.

The existing CLR components for managed execution (GC, JIT, type loader) would remain largely untouched. The new functionality would be introduced via a new, parallel runtime layer:

• Space Lifecycle Manager: A new runtime service would be responsible for the entire lifecycle of a Space. This includes allocating its isolated memory heap, creating and managing its dedicated thread(s), and handling its destruction. For Isolate runtime mode, this manager would be responsible for spawning and communicating with the new OS process.
• Inter-Space Communication (ISC) Layer: This would be the robust engine powering swait and srun. When a cross-space call is initiated, this layer would:
  1. Select the appropriate transport mechanism based on .csproj configuration or inline with specifiers (e.g., SharedMemory for co-located Spaces, gRPC for Isolate Spaces).
  2. Invoke the developer-defined cross-space converter to marshall the data from the source Space's representation to the destination's.
  3. Manage the asynchronous state machine for swait calls, ensuring the calling thread is efficiently suspended and resumed when the result is available.
• Unmanaged Space Environment: To support the Unmanaged mode, a lightweight, GC-less memory manager would be added. This component would run alongside the main GC but would be responsible exclusively for the heaps of unmanaged Spaces, offering explicit memory allocation and deallocation patterns. I believe inspiration could be drawn from languages like Rust for this part.

Compiler (Roslyn) Extensions

The compiler's role is to understand the new syntax and enforce the Space safety model at compile time.

• Syntax Analysis (Parsing): The C# grammar would be extended to recognize the new keywords (space, overspace, swait, srun, direct) and associated syntax, producing a corresponding syntax tree.
• Semantic Analysis & Binding: This is the most significant area of change. The semantic analysis phase would need to be augmented with a new, Space-aware analysis module:
  1. Context Tracking: The compiler must track which Space a given piece of code belongs to.
  2. Boundary Enforcement: It must validate all swait/srun calls, ensuring they are the only mechanism used to cross a Space boundary and flagging illegal direct calls.
  3. Polymorphism Resolution: When analyzing code, the compiler will resolve method calls and field access based on the current Space context. It will correctly bind data.GetCount() to the List<byte>based implementation when in the BusinessLogic Space, and to the byte*based implementation when in the PhysicsEngine Space.
  4. Converter Binding: The compiler will discover the explicit cross-space conversion constructors and validate their signatures, ensuring a valid marshalling path exists for objects passed between Spaces.
• IL Generation: The compiler will emit specialized Intermediate Language (IL) for cross-space operations. Instead of a standard call or callvirt instruction, a swait block would be translated into IL that invokes the new Inter-Space Communication (ISC) Layer in the runtime, passing along the target Space and the data payload.

---

The Opt-In Superset Model

The "Spaces" feature set is designed as a superset of C#. A project would explicitly opt-in to using this functionality, for instance, through a property like <EnableSpaces>true</EnableSpaces> in its .csproj file. If this flag is not present, the new keywords would be disabled, and the compiler would behave exactly as it does today.

This approach ensures that the core C# language remains stable, predictable, and approachable for all developers. The Spaces model provides a powerful new architectural dimension for those building complex, high-performance systems, without imposing any cognitive or runtime overhead on the rest of the ecosystem.

---

VII. Tooling & IDE Integration

This feature would require deep IDE integration to be successful:

• Active Space Context: The IDE (e.g., Rider) would have a dropdown in the editor to select the "Active Space Context" you are currently working in.
• Context-Aware Tooling:
  • Code Filtering: When "PhysicsEngine" is the active context, the IDE would gray out or hide the partial class members belonging to the "BusinessLogic" Space.
  • IntelliSense & Error Analysis: The analyzer would understand the rules of the active Space. IntelliSense would be context-aware, only suggesting members and methods available in the active Space.
  • Debugging: The debugger would be Space-aware, showing you which version of a function is being executed and allowing you to step across swait boundaries seamlessly.

---

VIII. Open Questions & Areas for Discussion

This is a foundational concept, and many areas require further exploration. This proposal's goal is to start that conversation.

1. Testing Model: How do we effectively write unit and integration tests for systems built with Spaces?
2. State & Data Management: What are the best practices for managing shared state or database access across different Spaces?
3. Ecosystem & Interoperability: How would existing libraries, especially those heavy with statics or environmental assumptions, work with Spaces?
4. Security Model: How do we define trust boundaries and permissions between Spaces, especially when some are running as separate processes (Isolate)?
5. Developer Culture & Methodology: How does this model change the way we architect and think about software design in the .NET ecosystem?
6. Unmanaged Memory Model: What would the precise semantics of memory management (ownership, lifetimes) be in an unmanaged Space? Could we draw inspiration from Rust?
7. Dynamic Space Lifecycle: Beyond temporary direct blocks, what would a robust API for creating, managing, and destroying long-lived Spaces at runtime look like?
8. Configuration Precedence: What is the order of precedence if an inline transport configuration (e.g., ...with { Transport = ... }) conflicts with a static configuration in the .csproj file?
9. Serialization Control & Performance: How can developers control the serialization process across swait boundaries to manage performance? Could the model support zero-copy, move semantics, or ownership transfer for high-performance scenarios?
10. Interaction with Generics: How would overspace and contextual polymorphism behave with complex generic types and constraints?

---

IX. Conclusion

The introduction of the Spaces model represents a pivotal evolution for C#, positioning it to become a truly universal language. It would empower developers to use a single, cohesive platform to build everything from conventional enterprise applications to the most demanding high-performance systems, effectively ending the need to switch languages for performance-critical tasks.

Crucially, this expanded capability does not come at the cost of approachability. The core of C# remains a familiar, productive, object-oriented language. The entire Spaces feature set is designed as an advanced, opt-in layer. Newcomers can learn and be productive in C# without ever touching a Space, preserving its gentle learning curve.

While the "Spaces" paradigm is vast enough to form the basis of a new programming language, it is proposed here as an evolution of C# for a critical reason: its potential can only be fully realized when built upon a world-class foundation. C# has already perfected the pillars of productivity, type safety, and tooling that a concept like Spaces desperately needs to succeed.

Therefore, this proposal stems not only from a belief in the model itself, but from a firm conviction that C#'s existing excellence is the key to unlocking that potential. The ambition here is to build upon the language's greatest strengths, ensuring it remains the definitive leader for the software challenges of today and the decades to come.

For advanced developers, the core C# OOP principles become the foundation for organizing objects and files, while the execution flow is seen as a series of isolated data streams built upon that foundation. The Spaces model does not introduce new complexity into the ecosystem; it formalizes, simplifies, and standardizes the immense knowledge already required for building microservices or high-performance systems. Any developer who needs this feature is already grappling with distributed system architecture, serialization, and concurrency. This proposal simply brings those concepts into the language as first-class citizens, making them explicit and manageable.

Ultimately, this proposal opens the door to a new paradigm of "unconscious multitasking" - a way to write concurrent code that feels natural and declarative. It's a model built for the reality of modern multi-core hardware, ensuring that C# is not only relevant but is a leading language for the challenges of the next decade and beyond.

Thank you for considering this proposal. I look forward to the community's feedback.

[HaloFour]

This feels like a runtime request (and an absolutely massive one) long before any actual language syntax should be considered.

[axx83]

Thank you, that's a very valid point.

The ultimate goal of 'Spaces' is to solve a major architectural problem: how to develop a codebase with the simplicity of a monolith, but deploy it with the flexibility of microservices. For that to work, the developer experience is everything. If it isn't easy and safe to use, nobody will adopt it, and that larger goal will fail.

And that's why I completely agree with you that we need a phased approach. This is a very long-term plan.

The first step absolutely has to be at the runtime level. We could start by building the core concepts (like isolation and high-performance communication) as a library. This would let the community experiment with and prove out the mechanics without any language changes.

Then, once that library is mature and its usage patterns are clear, we could integrate the language syntax (space, swait, etc.) as a high-level abstraction on top of it.

The reason I presented the full syntax vision upfront was to use it as a "North Star." It helps ensure that the initial runtime library is designed in the right direction, to serve that final goal of a natural and safe developer experience.

So, in short: You're right, the runtime is the essential proving ground. Let's start there with a library. But we should be clear: that library's success is measured by how quickly it enables us to begin language integration. The runtime provides the engine, but the language is the vehicle that actually gets developers to their destination.

[CyrusNajmabadi]

Ok. Circle back once the core runtime, libraries, and other data/messaging primitives are hammered out. We can then see what might make sense at a language level at that point.

[TahirAhmadov]

I would be interested in a real world example of a program written in C++ and C# using all modern (Span) and classic features (unsafe), and the performance difference, and where that difference is coming from.

[axx83]

Thank you, @CyrusNajmabadi for the insightful feedback. I completely agree that the foundational runtime, library, and messaging primitives are the right place to focus the effort first.

To facilitate this deeper, long-term discussion and to properly document the CScaf/Spaces vision, I have created a dedicated GitHub repository.

CScafLang

I've migrated the full proposal to the repository's Wiki for easier navigation. To address the significant design challenges ahead in a structured manner, I have also opened the Discussions tab. Each major topic will be addressed sequentially, and I've started with a thread dedicated to the first and most critical area: the core runtime and library primitives you mentioned.

I would be grateful if we could continue the detailed architectural discussion there. I look forward to hearing more of your thoughts.

[huoyaoyuan]

The whole article is based on invalid assumptions and deprecated concepts.

I. The Core Philosophy & Problem Statement

C# is a phenomenal language for productivity, but when absolute performance is needed (e.g., in a game engine's physics loop, a financial high-frequency trading model, or a data-intensive AI pipeline), developers often have to resort to C++ interop or separate services written in another language.

This is the first invalid precondition. C# is high performance and sometimes outperforms C++, since it gets access to raw SIMD instructions.
The performance problem of C# is mainly GC latency. In extremely sensitive areas there may be a need for non-GC language, but for usual areas it's not the case. The .NET runtime is also reducing the latency gradually.

Marshalling data between two systems harms performance much more if both systems are high performance ready.

II. The Foundational Concept: The "Space" Model

• Each Space, including Harmony, possesses its own dedicated memory heap and its own thread(s). This is a stronger guarantee than a simple thread.

This is the deprecated concepts of AppDomain and .NET Remoting. They fail to provide a good safety or reliability even with old-school C#, and are suffering from severe security threats.
In practice, any in-process isolation (AppDomain) is unreliable because nothing truely prevents a part to access another. Any inter-process isolation (.NET Remoting) must be explicitly verified because the counterpart can always be malicious.

• Asynchronous Message Passing: Spaces communicate exclusively through asynchronous message passing. This mechanism is abstracted away from the developer behind intuitive new language keywords, making inter-Space communication feel natural and declarative.

This is exactly what .NET Remoting and binary serialization did. It's been proven as fundamentally vulnerable.

IV. Syntax and Developer Interaction

"Direct" or "Lambda" Spaces

For small, one-off, performance-critical operations, a direct block can be used to create a temporary, inline Space with a specific configuration.

Data marshalling will be the most performance-harmful operation. In the example code, how to pass the result array is very unclear.

• swait (Space Wait): Asynchronously calls into another Space and waits for the result. This is intentionally distinct from await, which is used for I/O or async work within the same Space. swait signifies crossing an isolation boundary.

Synchronization is very heavy and requires a lot of participation from the OS. Asynchoros operations should be used for I/O where is naturally a long latency, not high performance operations.

Example 1: Using partial class Across Multiple Files

// File: DataBuffer.Managed.cs
space BusinessLogic;

public partial class DataBuffer
{
    // The 'overspace' keyword is required here for this Space's implementation.
    private overspace List<byte> _buffer = new List<byte>();

    public overspace void Add(byte b) => _buffer.Add(b);
    public overspace int Count => _buffer.Count;
}

// File: DataBuffer.Unmanaged.cs
space PhysicsEngine;

public unsafe partial class DataBuffer
{
    // A completely different, unmanaged implementation for the same type.
    private overspace byte* _buffer;
    private overspace int _count;

    public overspace void Add(byte b) { /* ... pointer manipulation ... */ }
    public overspace int Count => _count;
}

Not mentioning span making it non-sense for discussing performance.

The Cross-Space Marshalling Challenge

While the default mechanism is a safe, by-value copy (serialization), the model is designed to be extensible.

Again, serialization is not safe by default. It must deal carefully with the data it can handle.

Because the same type (e.g., DataBuffer) can have completely different underlying data structures in different Spaces

To address this, the model must include a mechanism for developers to define explicit Space-to-Space conversion routines.

This is exactly the problem for P/Invoke and already handled by custom marshalers.

    // --- The explicit Space Converter ---
    // This constructor teaches the PhysicsEngine's DataBuffer how to build 
    // itself from the BusinessLogic's version.
    public overspace DataBuffer(overspace BusinessLogic DataBuffer source)
    {
        _count = source.Count;
        // Allocate unmanaged memory
        _buffer = (byte*)System.Runtime.InteropServices.Marshal.AllocHGlobal(_count);
        // Copy the data from the managed list to the unmanaged buffer
        for (int i = 0; i < _count; i++)
        {
            _buffer[i] = source.GetBuffer()[i];
        }
    }

A very bad example. Using AllocHGlobal not NativeMemory or stackalloc, copying data in loop instead of using Span.Copy etc. It's insulated from any modern C#.

Moreover, the best for efficiency is to not marshal the data. P/Invoke tend to pin instead of copy data when appropriate.

V. The Safety Model: "The Airlock"

• Code is strictly scoped to its own Space and can only invoke functions defined within that context.

This is what AppDomain provides. It didn't provide real safety but just made the CLR unnecessarily complex.

• All interaction must happen through the swait and srun keywords. They are the secure doors to the airlock, ensuring that all communication adheres to the system's rules and the safety model.

It's not secure by default. Any little oversight can break the entire safety. That's why coreclr abandons security concepts totally and relies on operating system.

• This is the core rule of the safety model. When an object is passed across a Space boundary, its state (data) is copied, but its behavior (methods) is reconstituted entirely at the destination based on the local context.

This is not universal nor safe. Many objects can't be copied because it holds references to resources and others, and that's why MarshalByRefObject existed in .NET Remoting. Copying state of objects is unsafe because the validity of content is hard to guarantee.

VI. Proposed Implementation Path into Modern C#

The entire discussion is based on ancient C#. Modern C# is going to provide more safety on already high-performance routines (#9705).

This approach learns from the lessons of past isolation mechanisms like AppDomains, but provides a far more lightweight, language-integrated, and performant model designed for modern concurrency patterns rather than legacy code sandboxing.

AppDomain has been shown as a wrong direction of approach. The proposal is not lightweight comparing to RPC-wrappers.

• Unmanaged Space Environment: To support the Unmanaged mode, a lightweight, GC-less memory manager would be added. This component would run alongside the main GC but would be responsible exclusively for the heaps of unmanaged Spaces, offering explicit memory allocation and deallocation patterns. I believe inspiration could be drawn from languages like Rust for this part.

You can already achieve this today in C#, by writing everything in struct, using NativeMemory.Alloc and Span<T> to manage them.

IX. Conclusion

The entire proposal is dealing a non-existent problem with deprecated concepts and unnecessary complexity.

First, managed code isn't slow. SIMD instructions are used aggressively in the BCL, from the basic components like string. The JIT can switch aggressively for instruction set and do dynamic optimization with dynamic data, making it faster than unmanaged code. The work to rewrite BCL string/number routines from unmanaged to managed improved performance a lot. Even for cases when managed code isn't optimized enough, the marshalling overhead can easily be more than the performance difference. Marshalling gap between systems will also eliminate a lot of chance of optimization.
Sometimes GC latency is a problem, but there are already solutions in modern C#. The array pool is built-in and heavily used by BCL and third-party libraries. Spans eliminate the need of allocation in the first place. The native memory allocators provide an option to not use managed heap at all. If these are not sufficient, then the process shouldn't enter CLR at all - a CLR in the process will already pay the latency.
Handling RPC implicitly is a complex task and proven wrong. Unintentional marshalling does easily introduce security holes. The built-in binary serialization has been removed entirely from CLR. Protobuf/gPRC and other high-performance serialization approaches are the correct direction.
Integrating such concepts to the language is especially non-extensible. Any design oversights or behavior changes can't be handled at all. P/Invoke and COM marshalling are excellent examples. CLR is deprecating the built-in implicit marshalling system, preferring source generators to auto-generate the logic explicitly. This simplifies the toolchain from complex run-time CLR code to compile-time generators, and removes the compatibility concern to existing programs - their behavior retains with their build time.
Such "formalization" is we want to explicitly avoid because it caused many problems. Instead, it should be a toolchain that assembles individual components.

[CyrusNajmabadi]

@axx83 Could you restart that discussion over at @csca. Thanks :)

[huoyaoyuan]

The Core Problem: Taming Complexity at Scale

The true endgame is to provide a sane way to build and maintain million-LOC, Google-scale systems without the exponential growth in operational and cognitive complexity we see today.

A million LOC is not that large. .NET BCL itself is more than that. When I worked at Microsoft I've dealed with even larger codebase in monolithic repository. In that case, other engineering complexity is more challenging than the language itself.

Imagining the Million-LOC Project: The True Endgame of SOP

The World Today (The Microservice Archipelago):

• The codebase is a sprawling archipelago of hundreds of separate repositories.
• To fix a single cross-cutting bug, a developer must become a cognitive archeologist, digging through wikis to understand service dependencies, cloning multiple repos, and trying to run a fraction of the system locally.

Nothing prevents you to use monolithic repository today. A lot of aforementioned problems are caused by repository separation, not really by today's programming model.

• The IDE becomes a simple text editor. "Go to Definition" stops at the network boundary (HttpClient.PostAsync). Static analysis is confined to the tiny island of a single service.

Yes, this is a totally valid point. However, it can still be addressed by external tools like more "linking" support in IDE.

The World with CScaf / Structural-Oriented Programming (The Unified Codebase):

• Refactoring a core model like User is now a fearless, compile-time-safe operation. You can rename a property, and the compiler will instantly show you all 500 points of impact across all 100 "services".

This doesn't require any built-in RPC support. The 100 services can simply refer the same shared definition file of the data model. This is already the practice we are widely employing.

it makes the language and the compiler aware of the entire system's architecture, providing a level of safety and tooling integration that even Piper cannot achieve on its own.

This is unnecessarily complex with very niche return. The layers of components should be orthogonal as possible. The language should not know anything about cross-thread, cross-process or network. It just need to know the common concept of suspending to wait for asynchronous result. Building abstractions around asynchronous concepts should be done by library, not language.

1. On AppDomain vs. Spaces: It's About Codebase Architecture, Not Sandboxing

This seamless transition from a "code-as-monolith" to a "deploy-as-microservices" model is the core value that AppDomain never aimed to provide.

While it can be useful for building and deployment, it's very difficult to implement. Removing AppDomain simplifies the implementation of CLR dramatically because state tracking is much simpler. To behave the same, each AppDomain or Space needs to have it's own seperated static variables etc. CLR has abandoned the idea to maintain multiple set of this within the same instance.
Again, this should be a toolchain feature. There can be a way to gather multiple ASP.NET applications and run them within a same process, or an easy launcher to run multiple CLRs in the same process (this is currently unsupported).

2. On Performance: Enabling Non-GC, Not Just "Beating C++"

A non-GC Space is proposed to allow developers to optimize critical hot-paths with advanced, manual memory management—tasks currently reserved for C++/Rust—without making the entire application unsafe or leaving the C# ecosystem.

This is still based on the wrong point. First, manual memory management is performing better in most cases. This has been discussed by a lot of discussions. Then, you are having access to full manual mode today in C#. It would just prevent you from using the vast majority of BCL, resulting in a C-without-macro ecosystem.

It’s about completing the high-performance story that NativeAOT has started.

NativeAOT did NOT start a high-performance story. It does improve startup performance by its nature, but not steady state performance. In text intensive applications it usually performs worse because of lower level of SIMD instruction set. It's the largest misunderstanding that AOT is faster.
Moreover, NativeAOT is also orthogonal to C# language. The language level knows exactly nothing about the compiling model.

3. On Marshalling, swait, and Inter-Space Objects: Removing the Human Factor

Any cross-process call, whether in a traditional microservice or a Space, incurs similar costs and risks. However, Spaces can fundamentally improve this. Because the compiler understands both the caller and the callee within the same codebase, we are no longer limited to human-defined contracts like Protobuf.

The human factor is already removed in practice, by using shared source file for interface definition. Protocols like HTTP can also be easily generated by existing tooling. There are also automations to generate C# definition from other RPC protocol and vice versa.

An "object" passed between Spaces should not be thought of as a traditional, state-filled object. Instead, an object reference from another Space acts merely as an opaque handle, a pointer, or an anchor to that remote Space. It contains no data itself.

Data is not passed implicitly. Data is only transferred when a developer explicitly passes it as an argument in a swait block.

It's even worse to pass references instead of data implicitly. Passing references around different execution systems requires proper lifetime tracking and failure handling. COM interop is an example. It's also being separated from the CLR and becoming an external tooling to generate the boilerplates.

The "Spaces" concept is born from the modern needs of application growth and the trend of decentralization. Applications have become too large for a single monolithic deployment, but that does not mean the codebase must also be fragmented. Structural-Oriented Programming, as realized by "Spaces", finally gives us the tools to manage this new reality, allowing us to build systems whose architecture can evolve as seamlessly as their code.

While some of the problems do exist in practice, the proposal to integrate everything into the language is not the correct direction. The early days of CLR showed the problem caused by excess coupling. Some abstractions are abandoned, some are outdated but inflexible to change because of compatibility, some lack critical dimension of abstraction.
Building automation and abstraction for individual layers is the current practice. While unrelated to this proposal, the Interop type universion is an interesting example of building a flexible and lightweight abstraction focusing on one layer only.

[TahirAhmadov]

If our community were to spend time discussing pie in the sky ideas like a monumental change to the runtime and/ or C#, I think it would be better to concentrate on the long list of "warts" which people generally agree on. For example, array variance. It may sound small, but there are dozens of such things. And it's still a very open question whether or when such a project (restarting .NET) is actually undertaken.