Complete Tauq Workflows

TQN for humans, TBF for transport. Write TQN on both sides.

TBF is the compact wire format that reduces 87 KB to 14 KB while staying transparent to the user

The Three Components

TQN

Tauq Notation - readable text format

  • • 54% fewer tokens than JSON
  • • Human-readable and editable
  • • Best for LLMs and config

TBF

Tauq Binary Format - compact binary

  • • 84% smaller than JSON
  • • Columnar encoding
  • • Best for storage/transport

TQQ

Tauq Query - data transformations

  • • Filter and transform data
  • • Flexible pipeline language
  • • Compose queries

🚀 TBF: The Transport Layer

Why TBF for Transport?

  • ✓ 84% smaller - 87 KB → 14 KB for 1000 records
  • ✓ Transparent - Write TQN on both sides, TBF handles transport
  • ✓ Fast parsing - Zero-copy deserialization in Rust
  • ✓ Wire format - Perfect for APIs, network, databases

The Pattern

Write TQN
   ↓
Convert to TBF (compact)
   ↓
Send/Store TBF (84% smaller)
   ↓
Convert back to TQN
   ↓
Read TQN (user never sees binary)

Five Additional Workflows

🚀

PRIMARY: TQN ↔ TBF Transport

Write/read TQN on both ends. TBF is the compact transport layer.

1
Write in TQN
config.tqn (human-readable, 54% fewer tokens)
2
Convert to TBF
tauq build config.tqn --format tbf
3
Transport via network/disk
14 KB instead of 100+ KB (84% smaller)
4
Convert back to TQN
tauq format data.tbf --output tqn
5
Read/edit TQN
Human-readable on the other end

Benefits:

✓ Readable everywhere ✓ Compact transport ✓ 84% smaller over wire ✓ No format mismatch
🔄

Query → Multi-Format Output

Process with TQQ, output as both TQN and TBF

1
Input data
users.tqn (TQN format)
2
Transform
tauq exec pipeline.tqq < users.tqn
3
Output TQN
For human review (readable)
4
Output TBF
For storage/network (compact)
5
Store both
TQN in git, TBF in database

Benefits:

✓ Flexible pipeline ✓ Dual output ✓ Version control ✓ Efficient storage
🌐

API Gateway: TBF Transport

Receive TQN, store as TBF, send as TBF

1
Request as TQN
POST /api/users (TQN body)
2
Parse TQN
tauq::compile_tauq() → JSON
3
Validate
serde::from_value() checks schema
4
Store as TBF
database.insert(tbf::to_bytes())
5
Return TBF
Response in compact binary format

Benefits:

✓ Token savings (LLM) ✓ Compact storage ✓ Efficient network ✓ Type-safe
🏭

Data Lake Integration

Multi-source → TBF → Apache Iceberg

1
Read sources
CSV, JSON, TQN, Parquet
2
Normalize schema
All → Rust structs
3
Combine records
Merge from all sources
4
Write TBF
TbfFileWriter → columnar format
5
Store in Iceberg
Ready for SQL analytics

Benefits:

✓ Multi-source support ✓ Columnar storage ✓ 84% compression ✓ Analytics ready
⚙️

Configuration Management

Define in TQN, deploy as TBF

1
Define schema
config.tqn (readable)
2
Validate
tauq validate config.tqn
3
Build TBF
tauq build config.tqn --format tbf
4
Deploy
kubectl create configmap --from-file
5
Load at runtime
tbf::from_bytes(&config)

Benefits:

✓ Version controllable ✓ Validated config ✓ Compact deployment ✓ Fast loading
🤖

LLM Integration

Lower token costs with TQN format

1
Format as TQN
54% fewer tokens than JSON
2
Send to LLM
More context in same budget
3
LLM returns TQN
Same token efficiency
4
Validate output
Parse & type-check
5
Store as TBF
Compact persistence

Benefits:

✓ Lower costs ✓ More context ✓ Faster inference ✓ Token efficient

Format Selection Matrix

Use Case Format Why
Write data TQN 54% fewer tokens, human-readable
Display data TQN Easy to read and edit
Transform data TQQ Flexible query language
LLM context TQN 54% fewer tokens = lower cost
Store in DB TBF 84% smaller, indexed efficiently
Network transfer TBF 84% smaller, faster parse
Config files TQN Easy to version control
Data lakes TBF Native columnar format

Code Examples

CLI: Convert TQN to TBF

                # Write data in TQN
cat > users.tqn << 'EOF'
!def User id name age
1 Alice 30
2 Bob 28
EOF

# Convert to TBF (54→14 KB for 1000 records)
tauq build users.tqn --format tbf -o users.tbf

# Verify size
ls -lh users.*
Language: bash

Rust: TQN → JSON → TBF

                use tauq::tbf;

// 1. Parse TQN text
let tqn_text = r#"!def User id name age
1 Alice 30
2 Bob 28"#;

let json = tauq::compile_tauq(tqn_text)?;

// 2. Convert to Rust types
#[derive(serde::Serialize, serde::Deserialize)]
struct User { id: u32, name: String, age: u32 }
let users: Vec<User> = serde_json::from_value(json)?;

// 3. Serialize as TBF (compact binary)
let tbf_bytes = tbf::to_bytes(&users)?;

// 4. Transport/store as binary
database.insert("users", tbf_bytes);
Language: rust

TypeScript: Receive & Parse TBF

                import * as tauq from 'tauq';

// 1. Fetch TBF from API
const response = await fetch('/api/users');
const tbfBuffer = await response.arrayBuffer();

// 2. Parse TBF to JavaScript
const users = tauq.tbf.decode(
  new Uint8Array(tbfBuffer)
);

// 3. Use immediately (84% smaller than JSON)
users.forEach(user => {
  console.log(`${user.name}: ${user.age}`);
});
Language: typescript

Rust: API with Format Negotiation

                use actix_web::{web, HttpResponse};
use tauq::tbf;

#[post("/api/users")]
async fn create_users(
  body: web::Bytes,
  query: web::Query<std::collections::HashMap<String, String>>
) -> HttpResponse {
  // Parse input (TQN → JSON)
  let tqn = String::from_utf8(body.to_vec()).unwrap();
  let json = tauq::compile_tauq(&tqn).unwrap();
  let users: Vec<User> = serde_json::from_value(json).unwrap();

  // Store as TBF (84% smaller)
  for user in &users {
    db.insert(user.id, tbf::to_bytes(user).unwrap());
  }

  // Return format based on query param
  let response = match query.get("format").map(|s| s.as_str()) {
    Some("tqn") => tauq::format_to_tauq(&serde_json::to_value(&users).unwrap()),
    _ => String::from_utf8(tbf::to_bytes(&users).unwrap()).unwrap(),
  };

  HttpResponse::Ok().body(response)
}
Language: rust

Common Integration Patterns

Microservices

Flow: Service A → TBF (compact) → Network → TBF → Service B

Benefit: 84% smaller messages, faster serialization

Example: Microservices communicating over gRPC with TBF

LLM Integration

Flow: Data → TQN (54% tokens) → LLM → TQN → Parse

Benefit: Lower API costs, more context in same budget

Example: Fit 2x data in LLM context, half the cost

Time-Series Analytics

Flow: Sensors → TQN → TQQ (aggregate) → TBF → Iceberg

Benefit: Log compactly, analyze efficiently, query with SQL

Example: IoT sensors logging metrics to data lake

Configuration Management

Flow: Write TQN → Validate → Deploy TBF → Load → Parse

Benefit: Human-readable source, efficient deployment

Example: Kubernetes ConfigMap with TBF payload

Performance Summary (1000 Records)

Tokens

JSON
24,005
TQN
11,012 (-54%)

Binary Size

JSON
87 KB
TBF (Schema)
14 KB (-84%)

Parse Time

JSON
2.5 ms
TBF (Zero-copy)
<0.1 ms

Ready to Build with Tauq?

Choose your entry point:

Start with TQN Use TBF Format Learn TQQ Full Guide