CLEAR

CLEAR is a memory safe language, with a declarative concurrency model, that runs on a Go-like Green Fiber runtime.

It's designed to be:

1. Correct
2. Safe
3. Understandable
4. Scalable
5. BLAZING fast

WHAT DOES CLEAR LOOK LIKE

The SMOOTH operator

bill = users AS @u
  s> UNNEST _.orders
  s> SELECT _.price * @u.discount
  s> COCURRENT REDUCE(0, (acc, x) -> acc + x );

-- Automatically generates efficient, parallel partial reduction.

Combine with in-line error handling

FN myFunc(id: Int64, name: String) ->
  val = fetchData(id, name) OR RAISE
   s> parseHeader OR EXIT "Invalid Header"
   s> parseBody OR EXIT "Invalid Body"
   s> fetchUser
      s> RECOVER(defaultUser())
   s> saveToDb(id, name, _);

CATCH Input WITH(ParseError)
  IF __error.context == "Invalid Header" -> logInvalidHeader(__error.snapshot.header());
  RETURN defaultPage();
DEFAULT
  RAISE __error;
END

WHY CLEAR?

• SQL solved the problem of writing code once, and it constantly improving as the engine improves.
• Go proved the engine/run-time being added into the language can be fantastic.
• Rust proved that affine types can manage memory without a garbage collector simply (it's borrow checker is what gives it a bad reputation for being complicated, not Affine types).

CLEAR attempts to merge the best of Rust, Go, and SQL to build the language of the future: one that can constantly leverage new and better architectures and run your code as fast as possible without you having to tell it HOW to do that exactly - like SQL code.

Declarative Concurrency

Concurrency is not hard. SQL is the most commonly used programming language in the world, and it executes parallel queries as effiently as possible.

Concurrency is only hard when you have to tell the computer exactly how to achieve it efficiently. CLEAR does the hard part for you like a SQL engine.

In CLEAR, you describe the strategy you want to employ, and the compiler generates the how. When it's mature, you'll be able to trust that it leverages its runtime as efficiently as possible (as Go does currently).

-- `notify()` users in parallel, with back pressure
users
  s> CONCURRENT(workers: 8, parallel: TRUE) EACH notify

-- Spawn a short-lived green fiber, do all allocations in an Arena for speed:
BG { @micro:arena -> foo() }

Profile Guided Optimization

In CLEAR, the compiler can tell when you're probably employing a bad strategy, and changing it is typically just a one-line fix, rather than a full-app rearchitecture.

For the v0.1-pre release, CLEAR comes with a Control Plane and it can detect when you've employed a bad strategy. It works with the profiler to help you pick better strategies. For some, it can self-correct (like if you picked a bad stack-size for a reentrant fiber). In the near future, it will correct from contention issues. In the far future, the Control Plane aims to be able to protect you from even heavily skewed workloads (when you picked the wrong strategy, like shared-nothing).

CLEAR is designed such that you can override default compiler behviors if you know what you're doing, but you don't have the tools to shoot yourself in the foot.

Full Access to the Entire C Library

CLEAR lowers to Zig, which has native access to the entire C library.

In addition, Zig supports compiling to any target from any machine. I.e. you can compile for a Mac architecture from your Linux workstation.

BUILDING & TESTING

If you want to contribute, see CONTRIBUTING.md.

Prerequisites

• Ruby 3.x (for the compiler)
• Bundler (gem install bundler)
• Zig 0.16.0 (for runtime compilation)
• Go 1.21+ (for benchmark baselines, optional)
• Rust/Cargo (for benchmark baselines, optional)

Quick Start

bundle install                       # Install Ruby dependencies (one time)

./clear build hello.cht              # Compile a CLEAR program
./clear run hello.cht                # Build + execute
./clear test hello.cht               # Test with leak detection

The clear CLI

# Build
./clear build foo.cht                # Produces ./foo binary
./clear build foo.cht -o bin/app     # Custom output path
./clear build foo.cht --safe         # With bounds/overflow checks (-O ReleaseSafe)

# Run
./clear run foo.cht                  # Build + execute
./clear run foo.cht -- --port 8080   # Pass arguments to the program
CLEAR_THREADS=0 ./clear run app.cht  # Multi-threaded fiber runtime

# Test
./clear test foo.cht                 # Test with GPA leak detection + scheduler

FFI modules (.zig files referenced via EXTERN ... FROM) are auto-detected and linked.

Test Suites

# Ruby compiler specs (parallel)
bundle exec prspec spec/

# Transpile integration tests - two ways:
./clear test transpile-tests/                    # Run all at once (139 tests)
./clear test transpile-tests/58_bg.cht           # One at a time

# Package integration
cd transpile-tests/module-integration && zig build test

# FFI integration
cd transpile-tests/ffi-integration && zig build test

Benchmarks

ruby benchmarks/runner.rb --smoke benchmarks/24_json_api/   # CLEAR only, fast (~5s)
ruby benchmarks/runner.rb --fast benchmarks/05_hashmap/     # All langs, quick (~30s)
ruby benchmarks/runner.rb benchmarks/05_hashmap/            # Normal (5 runs)
ruby benchmarks/runner.rb --release benchmarks/05_hashmap/  # Exhaustive (5x load)
ruby benchmarks/runner.rb --all                             # All benchmarks
ruby benchmarks/runner.rb --smoke --all                     # Smoke test everything
ruby benchmarks/runner.rb --cores=4 benchmarks/17_kvstore/  # Control core count

Performance

Single-Core (HashMap):

• Numeric i64 keys outperform hand-optimized C by ~2x due to Zig's AutoHashMap and frame-arena allocation.
• Sring workloads and other workloads typically remain within 0-30% of hand-optimzied C.

Multi-Core, Non-Adversarial:

• Achieves throughput parity (within 5%) with Rust/Tokio and Go.
• CLEAR uses similar peak memory to Rust, about ~50% of Go due to the absence of a garbage collector.

Multi-Core, Adversarial (Pathological Workloads):

• Demonstrates high throughput and competitive p99.9 latency under skewed/adversarial loads using cooperative scheduling.
  • Though Go's preemptive scheduler maintains a slight edge in p99 consistency.

Reality: Multi-Core KV Store (vs Dragonfly):

• Matches Dragonfly's raw I/O and sharded HashMap throughput (within 5-10%) and offers better p99 latency.
  • Explicit caveat that this compares a minimal parser to a feature-complete production database.
  • Though this is achieved in ~50 lines of code.

See docs/benchmarks.md for table comparisons.

DISCLAIMER

CLEAR is currently in v0.1-pre release. It is an architectural preview and is NOT production-ready.

• Benchmarks: There's an extensive set of benchmarks, but it is hard to make "apples to apples" comparisons with mature ecosystems (Go, Rust, C). The current state of benchmarks should serve as evidence CLEAR is currently competitive for most cases. Take it with a grain of salt. It should not be considered proof of anything.
• Tail Latency: While throughput is high in the benchmarks, CLEAR's p99.9 latency is NOT expected to be competitive with Go's preemptive scheduler across all adversarial workloads in this release (and probably not until the v0.4 release).
• Standard Library: The current "standard library" is a barebones shim used for bootstrapping and internal testing. It is highly likely that none of the current internal APIs will survive into the v0.3 release.
• Stability: Although CLEAR has an extensive test suite, several significant bugs were identified in the final week before the demo-release. The v0.1-pre release required a major refactor to reach a reasonable level of assurance use-after-free, double-free, and memory leaks won't compile for most of the supported language. The v0.1 release will still be an unstable preview and is not representative of the stability goals for v0.2.
• Safety: The examples, benchmarks, and transpile-tests cover a wide range of the language and there are no use-after-free, double-free, or memory leaks. Do not expect that everything you can compile will be as safe as Rust for the v0.1 release (and certainly not before).
• Linux Only: CLEAR is not currently cross platform. It will only support x86 Linux until v0.3+.

KNOWN SCALING ISSUES

• Zig's std.Thread.Mutex scales poorly vs Rust's parking_lot at high core counts (2x gap at 32 cores) - see benchmarks/17_kvstore/README.md.
  • Zig will likely fix these issues before a CLEAR v0.3 release.
  • CLEAR's RwLock (@shared:writeLocked) bypasses Zig's stdlib, using pthread_rwlock_t with writer-preferring attributes directly.

VISION

The goal of CLEAR is:

1. To be able to run a REPL / VM, to live-debug like you can in Ruby, to write working code faster than you can even in scripting languages.
   • CLEAR aims to achieve a level of fearless concurrency above Rust, Pony, or Elixir.
   • Rust and actor models guarantee memory safety. But you can still have higher-level logic races / non-deterministic state transitions that are very painful to debug.
   • CLEAR aspires to make this as easy as debugging sequential code in a scripting language with an world-class debugger by providing deterministic replay of concurrent events (in the VM).
   • In other languages, Stateless Model Checking is a separate, difficult tool you have to opt into. In CLEAR, it's just how ./clear test works.
2. For ./clear doctor to be able to walk you ~95% of the way from that to HFT-Standards of C speed, and ADA-level safety.
3. For code, even at the highest-level of optimization, to be easily understandable.
4. A Control Plane so reliable that even if your most heavily optimized application experiences wildly unpredictible workloads, it can glide through it gracefully.
5. To be able to distribute loads across multiple machines effortlessly like BEAM, but with native speeds.