“

Series: S8 Technology Deep Dives · Article 11 | Level: Advanced | Reading Time: 20 min

Redis's 5 Roles: Full-Stack Mastery from Cache to Session

#TL;DR

Redis is far more than a cache. In a modern PaaS platform, Redis simultaneously plays five critical roles: caching layer, rate limiter, deduplication engine, hot terms ranking, and session store. This article dissects the data structure choices, deployment patterns, failure handling strategies, and concrete applications within an Ontology-driven intelligent decision platform for each role. We start from underlying data structures and incrementally build a comprehensive multi-role Redis architecture.

#1. Introduction: Why Redis Can Wear Many Hats

Redis's ability to serve multiple roles within a single system stems from its unique architectural philosophy. It is not a simple key-value store but an in-memory data structure server. Each data structure is carefully designed with O(1) or O(log N) time complexity, enabling Redis to complete complex operations at sub-millisecond latency.

In traditional architectures, developers often introduce separate middleware for each need: Memcached for caching, token buckets for rate limiting, Bloom filters for deduplication, Elasticsearch for trending analysis, and databases for session storage. This approach results in exponential growth in operational complexity. Redis's multi-data-structure capabilities allow us to solve multiple classes of problems with a single piece of infrastructure.

#1.1 Core Advantages of Redis

Redis's single-threaded event loop model guarantees operation atomicity, which is crucial in distributed systems. When implementing "check-and-set" operations, Redis's MULTI/EXEC transactions or Lua scripts naturally avoid race conditions.

The memory-first storage strategy keeps Redis latency stable at the microsecond level. According to official Redis benchmarks, on commodity hardware, Redis can handle over 100,000 SET/GET operations per second. This performance characteristic makes Redis suitable as a critical component on hot paths.

Redis 6.0's multi-threaded I/O model further improved network throughput while maintaining single-threaded command execution. This means we can achieve higher throughput without changing the programming model.

#1.2 Redis's Position in the Ontology Platform

In an Ontology-driven intelligent decision platform, Redis sits on the hot data access path. The Control Layer caches metadata query results through Redis, the Data Layer uses Redis for real-time data deduplication, and the Intelligence Layer relies on Redis for fast retrieval of reasoning results. This cross-Layer unified usage pattern makes Redis an indispensable infrastructure component.

#2. Role One: Caching Layer

#2.1 Cache Strategy Selection

The core questions of caching strategy are "when to write" and "when to invalidate." In the Ontology platform, we select different strategies based on data characteristics.

Cache-Aside is the most commonly used pattern. The application first queries Redis; if the cache misses, it queries the database and then writes the result to Redis. The advantage is simplicity; the drawback is that first requests always miss.

async def get_object_type(type_rid: str) -> ObjectType:
    cache_key = f"ontology:object_type:{type_rid}"
    cached = await redis.get(cache_key)
    if cached:
        return ObjectType.model_validate_json(cached)
    obj_type = await db.query_object_type(type_rid)
    await redis.setex(cache_key, 3600, obj_type.model_dump_json())
    return obj_type

Write-Through synchronously updates the cache on data writes. This pattern ensures cache data consistency but adds write latency. It is suitable for write-rare-read-heavy scenarios such as metadata registration.

Write-Behind writes data to the cache first and then asynchronously batch-writes to the database. This pattern improves write performance but risks data loss. In the platform's metrics collection scenario, we use this pattern to buffer high-frequency metric data.

#2.2 Cache Key Design

Good key design is the foundation of an efficient caching system. We adopt a hierarchical namespace design:

{Layer}:{entity}:{identifier}:{version}

For example: control:object_type:ri.onto.main.object-type.Employee:v3

This design supports prefix-based batch operations. For example, when an Object Type's schema changes, we can use the SCAN command with a prefix pattern to batch-invalidate related caches.

#2.3 Cache Penetration and Avalanche Protection

Cache penetration occurs when queries for non-existent data bypass the cache and hit the database directly. We use Bloom filters (Redis's BF.EXISTS command) for pre-filtering. For Object Type queries in the Ontology, we maintain a Bloom filter containing all valid RIDs. Non-existent RIDs are intercepted at the Bloom filter layer.

async def get_with_bloom_filter(type_rid: str) -> Optional[ObjectType]:
    if not await redis.execute_command("BF.EXISTS", "ontology:types:bloom", type_rid):
        return None  # Bloom filter determines non-existence
    return await get_object_type(type_rid)

Cache avalanche occurs when massive caches expire simultaneously, causing database pressure spikes. We avoid this by adding random jitter to TTLs:

base_ttl = 3600
jitter = random.randint(0, 600)
await redis.setex(key, base_ttl + jitter, value)

#2.4 Multi-Level Cache Architecture

In high-availability scenarios, we implement a two-level cache architecture with local cache plus Redis. The local cache (such as Caffeine or Python's cachetools) handles ultra-hot data, while Redis serves as the distributed shared cache. The local cache TTL is set shorter (typically 30 seconds to 1 minute) to balance consistency and performance.

When data changes, Redis Pub/Sub notifies all nodes to invalidate their local caches. This pattern is widely used in the Control Layer's Schema Registry, ensuring schema changes propagate to all nodes within seconds.

#3. Role Two: Rate Limiter

#3.1 The Necessity of Rate Limiting

In a PaaS platform, rate limiting is a critical mechanism for protecting system stability. Unrestricted API calls can overload backend services, affecting service quality for all tenants. Redis's atomic operation characteristics make it an ideal choice for implementing distributed rate limiters.

#3.2 Fixed Window Counter

The simplest rate limiting implementation is the fixed window counter, using Redis's INCR and EXPIRE commands:

async def fixed_window_rate_limit(
    client_id: str, limit: int, window_seconds: int
) -> bool:
    key = f"ratelimit:fixed:{client_id}:{int(time.time()) // window_seconds}"
    count = await redis.incr(key)
    if count == 1:
        await redis.expire(key, window_seconds)
    return count <= limit

The drawback of this approach is the window boundary problem: at the moment of window transition, a client can send twice the limit in requests.

#3.3 Sliding Window Log

The sliding window log uses Redis Sorted Sets to store timestamps of each request. This approach is precise but memory-intensive:

async def sliding_window_log(
    client_id: str, limit: int, window_seconds: int
) -> bool:
    key = f"ratelimit:sliding:{client_id}"
    now = time.time()
    pipeline = redis.pipeline()
    pipeline.zremrangebyscore(key, 0, now - window_seconds)
    pipeline.zadd(key, {str(now): now})
    pipeline.zcard(key)
    pipeline.expire(key, window_seconds)
    results = await pipeline.execute()
    return results[2] <= limit

#3.4 Token Bucket Algorithm

The token bucket is the most flexible rate limiting algorithm, supporting burst traffic. We implement it atomically in Redis using a Lua script:

local key = KEYS[1]
local rate = tonumber(ARGV[1])
local capacity = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local requested = tonumber(ARGV[4])

local last_time = tonumber(redis.call('hget', key, 'last_time') or now)
local tokens = tonumber(redis.call('hget', key, 'tokens') or capacity)

local elapsed = now - last_time
tokens = math.min(capacity, tokens + elapsed * rate)

if tokens >= requested then
    tokens = tokens - requested
    redis.call('hset', key, 'last_time', now)
    redis.call('hset', key, 'tokens', tokens)
    redis.call('expire', key, math.ceil(capacity / rate) * 2)
    return 1
else
    redis.call('hset', key, 'last_time', now)
    redis.call('hset', key, 'tokens', tokens)
    return 0
end

#3.5 Multi-Dimensional Rate Limiting

In the Ontology platform, we implement multi-dimensional rate limiting strategies:

• Tenant level: Maximum 1,000 API calls per second per tenant
• User level: Maximum 100 API calls per second per user
• Endpoint level: Specific expensive operations (e.g., full schema export) limited to 10 per minute
• Global level: System-wide maximum of 50,000 requests per second

These limits are implemented through composite keys and can be dynamically adjusted via the configuration center.

#4. Role Three: Deduplication Engine

#4.1 Business Scenarios for Deduplication

In data pipelines, exactly-once message delivery is a classic challenge. Network retries, consumer restarts, and other situations can cause messages to be processed multiple times. Redis provides multiple data structures to achieve efficient deduplication.

#4.2 Exact Deduplication with SET

For scenarios requiring exact deduplication, we use Redis SET to store processed message IDs:

async def is_duplicate(message_id: str, ttl: int = 86400) -> bool:
    key = f"dedup:messages:{message_id}"
    result = await redis.set(key, "1", nx=True, ex=ttl)
    return result is None  # If SET NX returns None, the key already exists

SET NX (Set if Not Exists) is an atomic operation that naturally avoids race conditions in concurrent scenarios. TTL ensures expired dedup records are automatically cleaned up, preventing unbounded memory growth.

#4.3 Probabilistic Deduplication with Bloom Filters

When message volumes are extremely large, the memory overhead of exact deduplication may be unacceptable. Redis's Bloom filter module provides a probabilistic deduplication solution:

async def probabilistic_dedup(message_id: str) -> bool:
    exists = await redis.execute_command("BF.EXISTS", "dedup:bloom", message_id)
    if exists:
        return True  # Possibly duplicate (has false positive rate)
    await redis.execute_command("BF.ADD", "dedup:bloom", message_id)
    return False

The Bloom filter's false positive rate is configurable. For 100 million messages, a 0.1% false positive rate requires only about 120MB of memory, whereas exact deduplication would need over 3GB.

#4.4 Cardinality Deduplication with HyperLogLog

HyperLogLog is suited for "counting unique elements" scenarios, such as calculating daily active users. It achieves cardinality estimation with less than 1% error using extremely low memory (only 12KB per HyperLogLog):

async def count_unique_users(date: str, user_id: str) -> int:
    key = f"stats:unique_users:{date}"
    await redis.pfadd(key, user_id)
    return await redis.pfcount(key)

#4.5 Deduplication in Practice for Data Pipelines

In the Ontology platform's data pipeline, we employ a layered deduplication strategy. The first layer uses Bloom filters to quickly filter obvious duplicate messages (approximately 99% of duplicates are intercepted at this layer). The second layer uses exact SET deduplication for secondary confirmation of messages that the Bloom filter flagged as "possibly duplicate."

This layered architecture reduces memory usage by over 90% while maintaining zero-miss deduplication accuracy. The Bloom filter's false positives only result in a small number of additional SET queries, with negligible impact on overall performance.

#5. Role Four: Hot Terms Ranking

#5.1 The Challenge of Real-Time Ranking

Search term trending, popular entity ranking, and similar features require real-time updates and queries. Traditional relational databases struggle to support real-time sorting under high-frequency update scenarios. Redis's Sorted Set, with a skip list as its underlying data structure, provides O(log N) insertion and O(log N + M) range queries (where M is the number of returned elements).

#5.2 Basic Ranking Implementation

async def record_search_term(term: str) -> None:
    key = f"hotterms:{datetime.now().strftime('%Y%m%d%H')}"
    await redis.zincrby(key, 1, term)
    await redis.expire(key, 86400)  # Retain for 24 hours

async def get_top_terms(n: int = 10) -> list[tuple[str, float]]:
    key = f"hotterms:{datetime.now().strftime('%Y%m%d%H')}"
    return await redis.zrevrange(key, 0, n - 1, withscores=True)

#5.3 Multi-Time-Dimension Aggregation

In practice, we need to support rankings across multiple time dimensions (hourly, daily, weekly). The ZUNIONSTORE command efficiently aggregates data across multiple time periods:

async def get_daily_top_terms(date: str, n: int = 10) -> list[tuple[str, float]]:
    hour_keys = [f"hotterms:{date}{h:02d}" for h in range(24)]
    dest_key = f"hotterms:daily:{date}"
    await redis.zunionstore(dest_key, hour_keys)
    await redis.expire(dest_key, 172800)
    return await redis.zrevrange(dest_key, 0, n - 1, withscores=True)

#5.4 Decay Ranking Algorithm

Simple cumulative counting fails to reflect the temporal nature of "hotness." We implement an exponential decay ranking algorithm where recent searches contribute more to the ranking than older ones:

async def record_with_decay(term: str, half_life_hours: float = 6.0) -> None:
    now = time.time()
    score = math.pow(2, now / (half_life_hours * 3600))
    key = "hotterms:decayed"
    await redis.zadd(key, {term: score}, gt=True)

By selecting an appropriate half-life parameter, you can control the "freshness" of the leaderboard. A half-life of 6 hours means searches from 6 hours ago contribute half as much, and searches from 24 hours ago contribute only 1/16th.

#5.5 Ontology Entity Heat Tracking

In the Ontology platform, we track access heat for Object Types and Link Types to optimize cache strategies and recommend related entities. Every time a user queries an Object Type, we increment its score in a Sorted Set:

async def track_entity_access(entity_rid: str, entity_type: str) -> None:
    key = f"entity_heat:{entity_type}:{datetime.now().strftime('%Y%m%d')}"
    await redis.zincrby(key, 1, entity_rid)

Heat data drives multiple optimizations: high-heat entities get automatically extended cache TTLs, low-heat entities are preferentially evicted from cache, and the recommendation engine prioritizes high-heat related entities.

#6. Role Five: Session Store

#6.1 Why Choose Redis for Session Storage

Traditional cookie-based or database-based session management faces numerous challenges in distributed environments. Cookie solutions are limited by capacity and security concerns. Database solutions have latency unsuitable for frequently accessed session data. Redis, with its sub-millisecond latency and flexible data expiration mechanisms, is the preferred solution for session storage.

#6.2 Session Data Structure Design

We use Redis Hash to store session data, with each field corresponding to a session attribute:

async def create_session(user_id: str, metadata: dict) -> str:
    session_id = str(uuid.uuid4())
    key = f"session:{session_id}"
    session_data = {
        "user_id": user_id,
        "created_at": str(time.time()),
        "last_active": str(time.time()),
        "ip_address": metadata.get("ip", ""),
        "user_agent": metadata.get("user_agent", ""),
        "tenant_id": metadata.get("tenant_id", ""),
    }
    await redis.hset(key, mapping=session_data)
    await redis.expire(key, 7200)  # 2-hour expiry
    # Maintain user session index
    await redis.sadd(f"user_sessions:{user_id}", session_id)
    return session_id

async def refresh_session(session_id: str) -> bool:
    key = f"session:{session_id}"
    if not await redis.exists(key):
        return False
    await redis.hset(key, "last_active", str(time.time()))
    await redis.expire(key, 7200)  # Renew
    return True

#6.3 Session Security Mechanisms

Session storage must account for security. We implement the following security measures:

Session fixation attack prevention: After successful user authentication, destroy the old session and create a new one.

Concurrent session limiting: Through user session indices (user_sessions:{user_id}), we can limit the maximum concurrent sessions per user.

Session hijacking detection: Store the IP and User-Agent at session creation time. If these change in subsequent requests, trigger re-authentication.

async def validate_session(session_id: str, request_ip: str) -> bool:
    key = f"session:{session_id}"
    session = await redis.hgetall(key)
    if not session:
        return False
    if session.get("ip_address") != request_ip:
        await redis.delete(key)  # Suspicious activity, destroy session
        return False
    await refresh_session(session_id)
    return True

#6.4 Distributed Session Management

In multi-datacenter deployments, session data needs cross-region synchronization. We use Redis replication and Redis Cluster for session high availability:

• Write operations are routed to the primary node
• Read operations can be served from the nearest replica
• Eventual consistency of session data is guaranteed through asynchronous replication

For scenarios requiring strong consistency (such as payment operations), we embed a version number in the session and use Redis's WATCH/MULTI/EXEC transactions to guarantee atomicity.

#6.5 Session Data Lifecycle Management

Redis's key expiration mechanism (TTL) naturally supports automatic session expiry. But in some scenarios, we need more granular lifecycle management:

Sliding expiration: Automatically renew on each request, ensuring active sessions do not expire.

Absolute expiration: Regardless of activity, sessions exceeding the maximum lifetime (e.g., 24 hours) are forcefully expired.

Graceful logout: When a user actively logs out, immediately delete session data and clean up all associated indices.

#7. Unified Architecture: Five-in-One Deployment

#7.1 Redis Instance Planning

In production environments, we do not recommend deploying all five roles on a single Redis instance. Based on data characteristics and SLA requirements, we divide Redis into three groups:

• Cache group: Cache + hot terms ranking. Use maxmemory-policy allkeys-lru, allowing data eviction when memory is insufficient.
• Persistence group: Sessions + deduplication. Enable AOF persistence to ensure data durability.
• Rate limiting group: Rate limiting. Deploy independently to avoid latency impacts from large keys in other roles.

#7.2 Monitoring and Alerting

Each role has its critical monitoring metrics:

#7.3 Failure Recovery Strategies

For the cache role, when Redis fails, we can degrade directly to database queries — proper database capacity planning is essential. For the session role, Redis failure means all users need to re-login, so Redis Sentinel or Cluster must be used for high availability. For the rate limiting role, when Redis fails, the strategy should be "allow" rather than "deny," preventing the rate limiter failure from making the entire system unavailable.

#8. Performance Optimization Practices

#8.1 Pipeline Batch Operations

Redis network round-trip time (RTT) is often the performance bottleneck. Using Pipeline packs multiple commands into a single send, reducing network round trips:

async def batch_cache_check(keys: list[str]) -> dict[str, Optional[str]]:
    pipeline = redis.pipeline()
    for key in keys:
        pipeline.get(key)
    results = await pipeline.execute()
    return dict(zip(keys, results))

In the Ontology platform, when loading an Object Type and all its Property Types, we use Pipeline to fetch all cached data at once, reducing latency from N * RTT to 1 * RTT.

#8.2 Lua Script Optimization

For multi-step operations requiring atomicity, Lua scripts not only guarantee atomicity but also reduce network round trips. Redis caches compiled Lua scripts (via EVALSHA); subsequent calls only need the script's SHA1 hash and parameters.

#8.3 Memory Optimization

Redis memory usage can be optimized in several ways:

• Use Hash instead of multiple Strings: When storing multiple attributes of the same entity, Hash is more memory-efficient than multiple independent String keys.
• Integer encoding optimization: Redis uses shared objects for small integers (0-9999), incurring no additional memory allocation.
• Compressed lists: When Hash, List, or Set element counts are small, Redis uses ziplist encoding, which is extremely memory-efficient.

#8.4 Connection Pool Management

In high-concurrency scenarios, Redis connection creation and destruction overhead cannot be ignored. We use connection pools to reuse connections:

redis_pool = redis.ConnectionPool(
    host="redis-cluster.internal",
    port=6379,
    max_connections=50,
    retry_on_timeout=True,
    socket_timeout=1.0,
    socket_connect_timeout=1.0,
)

Connection pool size should balance concurrency capability and resource consumption. A general recommendation is 1.5x the expected concurrency.

#9. High Availability and Disaster Recovery

#9.1 Redis Sentinel

Redis Sentinel provides automatic failover capability. When the primary node fails, Sentinel automatically elects a new primary and notifies all clients. In the Ontology platform, we deploy at least 3 Sentinel instances per Redis group, distributed across different availability zones.

#9.2 Redis Cluster

For scenarios where data volume exceeds single-node memory, Redis Cluster provides automatic sharding and high availability. Data is automatically distributed across 16,384 slots by CRC16 hash, with each primary node responsible for a subset of slots.

Note that Redis Cluster does not support cross-slot transactions or Lua scripts. When designing cache keys, Hash Tags (e.g., {tenant:123}:cache:key) can ensure related keys are assigned to the same slot.

#9.3 Data Persistence Strategies

• RDB snapshots: Periodically generate binary snapshots of data, suitable for disaster recovery.
• AOF log: Records every write operation, supporting finer-grained recovery.
• Hybrid persistence: Introduced in Redis 4.0, hybrid persistence combines RDB's fast loading with AOF's data completeness.

For session and deduplication roles, we recommend AOF persistence configured with appendfsync everysec, balancing performance and data safety.

#10. Summary and Future Outlook

#10.1 Role Selection Decision Tree

When designing a multi-role Redis architecture, follow this decision tree:

1. Is exact data required? If some error is acceptable, consider probabilistic data structures (Bloom filters, HyperLogLog).
2. Is data loss acceptable? If data can be lost, use LRU eviction policies; otherwise enable persistence.
3. Is atomicity required? If multi-step atomic operations are needed, use Lua scripts.
4. Latency requirements? If sub-millisecond latency is required, ensure Redis is deployed in the same availability zone.

#10.2 Future Evolution

Redis 7.0's Function mechanism will replace EVAL/EVALSHA, providing better script management capabilities. Redis Stack integrates Search, JSON, TimeSeries, and other modules, further expanding Redis's application scenarios. In subsequent versions of the Ontology platform, we plan to use Redis Search module to replace some Elasticsearch functionality, further simplifying the architecture.

#Key Takeaways

1. Redis is more than a cache — it is a multi-functional in-memory data structure server that can simultaneously serve as cache, rate limiter, deduplication engine, ranking system, and session store.
2. Data structures determine efficiency — choosing the right data structure (String, Hash, Set, Sorted Set, Stream) is key to Redis application optimization.
3. Group deployments — different roles have different SLA and data persistence requirements and should be deployed in groups rather than mixed on a single instance.
4. Lua scripts are powerful tools — for multi-step operations requiring atomicity, Lua scripts guarantee correctness while improving performance.
5. Monitoring drives operations — each role has specific monitoring metrics and alert thresholds; Redis without monitoring is a ticking time bomb.

#Next Article

The next article, S8-12: PostgreSQL Metadata Storage, will deep dive into PostgreSQL's role as the metadata store in the Ontology platform, including schema design, indexing strategies, flexible JSONB usage, and coordination with the Redis caching layer.

tags: [redis, cache, rate-limiting, deduplication, session, sorted-set, bloom-filter, ontology-paas, S8]