swift-server/redistack

Non-blocking, event-driven Swift client for Redis.

redistack's Issues

Clarify type and module names

https://forums.swift.org/t/discussion-swiftnio-redis-client/19325/5?u=mordil

Add Type-Safety to `RedisPipeline.enqueue`

From the proposal discussion thread: https://forums.swift.org/t/discussion-nioredis-nio-based-redis-driver/22455/15

Is there are reason we shouldn't also have a declarative way for the command themselves? Something like

public enum RESPCommand {
    case get(RESPValueConvertible)
    case increment(RESPValueConvertible)
    case set(RESPValueConvertible, RESPValueConvertible)
    // ...
}

That way, batching of requests can become more natural and we don't need

public func enqueue<T>(operation: (RedisClient) -> EventLoopFuture<T>) -> RedisPipeline

where we have this weird operation: (RedisClient) -> EventLoopFuture<T> which I commented about before. Correct me if I'm wrong but I believe the T is just there because we need support all of the singular operations like $0.get(...) which already return an EventLoopFuture but of different types... If we created an enum RESPCommand we could have

connection.sendBatch([.get("foo"), .set("bar", 1), .increment("buz")])

Use `ByteBuffer` for `RESPValue` data storage

From the proposal discussion thread: https://forums.swift.org/t/discussion-nioredis-nio-based-redis-driver/22455/11

Change

public enum RESPValue {
    case .simpleString(String)
    case .bulkString([UInt8])
}

to

public enum RESPValue {
    case .simpleString(ByteBuffer)
    case .bulkString(ByteBuffer?)
}```

Is the library compatible with Redis sentinel?

Quoting the documentation:
You need Sentinel support in your clients. Popular client libraries have Sentinel support, but not all.

Add methods for all commands

• Cluster
• Connection
• Geo
• Hashes
• Keys
• List
• Set
• Sorted Sets

async / await command APIs

Are there plans to add async/await versions of the existing EventLoopFuture commands?

I'd be happy to add them if it's something folks would be interested in.

i.e.

extension RedisClient {
    public func echo(_ message: String) async throws -> String {
        echo(message).get()
    }
}

Replace `Data` in RESPValue with `ByteBuffer`

Right now the RESPValue.bulkString stores a reference to Data, but it might be possible / easy to replace it with a ByteBuffer

Provide a "First Class" API for Lua Scripting

I'm opening this up for discussion of what people may want to do with Lua scripting integration through the Swift client against Redis.

The list of supported commands: https://redis.io/commands#scripting

Support SSWG Logging

Now that swift-log is publicly available - NIORedis should pull it in as a dependency and provide logs.

Does redis support domain socket connection to redis server ?

I tried to connect to the redis server using RediStack. And the client and the server are both on the same server. When I tried to connect to the redis-server using unix socket (unix:///tmp/mm.sock) , I got the folllowing error :
RediStack.RedisConnection.Configuration.ValidationError.(unknown context at $100654b1c).Kind.invalidURLScheme)

Expose `NIORedis` objects for composeability

Remove all NIO umbrella imports

We should not depend on the NIO umbrella module.

Instead we should explicitly

• import NIOCore
• import NIOPosix
• import NIOConcurrencyHelpers
• import NIOEmbedded
• ...

where needed.

We should also add explicit dependencies in the Package.swift

Acceptance criteria:

This code in scripts/soundness.sh can be commented in again and CI passes:

# This checks for the umbrella NIO module.
printf "=> Checking for imports of umbrella NIO module... "
if git grep --color=never -i "^[ \t]*import \+NIO[ \t]*$" > /dev/null; then
    printf "\033[0;31mUmbrella imports found.\033[0m\n"
    git grep -i "^[ \t]*import \+NIO[ \t]*$"
    exit 1
fi
printf "\033[0;32mokay.\033[0m\n"

Redis Queue Driver disable Continuous debug log in console

I have configured RedisQueue Driver for my vapor project. While running the app the console becomes flooded with continuous debug log from the driver connection. I can barely see the other logs. Is there any way to disable the RedisQueue driver connection log? Screenshot of my console is attached. (Notes: I am running the service with docker compose)

Unexpected Results with Concurrent RedisPipeline and RedisConnection Usage

When using a RedisPipeline created from a RedisConnection, both have access to the same Channel instance.

The former will write then flush in two separate steps, while the latter writeAndFlush in a single step.

When using both concurrently - say on separate threads - or even if you for some reason need to break up the pipeline with other non-pipeline calls, the results are unexpected as the connection's command is written in between the pipeline's.

Failing Unit Test

    func test_concurrentChannelUsage() throws {
        let pipeline = connection.makePipeline()
            .enqueue { $0.get("key") }
            .enqueue { $0.increment("key") }
            .enqueue { $0.increment("key", by: 30) }

        _ = try connection.increment("key").wait()

        pipeline.enqueue { $0.decrement("key") }

        let results = try pipeline.execute().wait()

        XCTAssertEqual(results.count, 4)
        XCTAssertTrue(results[0].isNull)
        XCTAssertEqual(results[1].int, 1)
        XCTAssertEqual(results[2].int, 31)
        XCTAssertEqual(results[3].int, 30)
    }

Redis retry configuration is potentially confusing

The Redis retry configuration is potentially confusing, and downright problematic for Vapor.

RedisConnectionPool.Configuration.init() is implemented like this. Note that it specifies a default timeout in the signature of 60 seconds, but if nil is passed in, it uses an internal timeout of 10 ms.

        public init(
            initialServerConnectionAddresses: [SocketAddress],
            maximumConnectionCount: RedisConnectionPoolSize,
            connectionFactoryConfiguration: ConnectionFactoryConfiguration,
            minimumConnectionCount: Int = 1,
            connectionBackoffFactor: Float32 = 2,
            initialConnectionBackoffDelay: TimeAmount = .milliseconds(100),
            connectionRetryTimeout: TimeAmount? = .seconds(60),
            onUnexpectedConnectionClose: ((RedisConnection) -> Void)? = nil,
            poolDefaultLogger: Logger? = nil
        ) {
            self.initialConnectionAddresses = initialServerConnectionAddresses
            self.maximumConnectionCount = maximumConnectionCount
            self.factoryConfiguration = connectionFactoryConfiguration
            self.minimumConnectionCount = minimumConnectionCount
            self.connectionRetryConfiguration = (
                (initialConnectionBackoffDelay, connectionBackoffFactor),
                connectionRetryTimeout ?? .milliseconds(10) // always default to a baseline 10ms
            )
            self.onUnexpectedConnectionClose = onUnexpectedConnectionClose
            self.poolDefaultLogger = poolDefaultLogger ?? .redisBaseConnectionPoolLogger
        }

The Vapor wrapper around this passes nil down through its call stack.

It's not clear what the intent is in RediStack. Is a default timeout 60 seconds or 10 milliseconds? Certainly the public documentation implies it's 60 seconds, and only by a careful reading of the actual code can you determine that it ends up as 10 ms if you pass nil.

about the dependency url

Today's afternoon, I tried to integrate the Redis to vapor 4 (my swift version is 5.4)

according to this doc , I copied the url ( .package(url: "https://gitlab.com/mordil/RediStack.git", .branch("master"))) to my Package.swift

but the complier reported an unwanted problem, it indicated :" https://gitlab.com/Mordil/RediStack: An unknown error occurred. unexpected return value from ssl handshake -9806 (-1)"

Meanwhile, I found another reference on vapor's doc (https://docs.vapor.codes/4.0/redis/overview/). Unfortunately, it was also unable to be fixed.

Finally, I attempted to substitute the "gitlab" by "github", then the dependency package can be install on my app

To be honesty, I had spent more to an hour to address this issue. so , would you mind to change the url into the right way ?

SwiftMetrics Integration

Counters

• Success / failed requests
• Total count of connections made
• Total count of requests made

Gauge

• Number of active connections

Timer

• Time to request completion

[RFC] Do you use RediStack and/or Interested in contributing?

Two of the graduation paths for the SSWG Incubation process out of sandbox are:

• Document that it is being used successfully in production by at least three independent end users which, in the SSWG judgement, are of adequate quality and scope
• Have a healthy number of committers

Eventually, it would be great to be promoted by the SSWG to a higher maturity level, but I need the communities help.

⚠️ I have no intention to abandon or transition ownership of the project, this is simply a call to action raise this project's maturity and stability._

There are a few ways you can help, check out the full RFC on the source GitLab repo.

Improve Error Handling

Rename `ExecutionModel.eventLoopGroup` cases

enum ExecutionModel {
    case managed(threadCount: Int)
    case unmanaged(EventLoopGroup)
}```

Rework Scan unit tests

The unit tests for all the scan command iterations are failing non-deterministically.

They should be re-written so that they do fail reliably.

Add an source index to RESP3ParsingError

We should add an index to the RESP3ParsingError, so that we can track where in the buffer parsing failed. This can be immensely helpful when trying to debug RESP3 values with a more complex structure.

Original Issue

          Could we add an index here? That way we can correlate the error back to a specific location in the received ByteBuffer. Definitely helps debugging should this ever trigger

Originally posted by @Joannis in #71 (comment)

Add Pub/Sub Capabilities

Make `RedisConnection.close()` return a future.

error when setting a key to json string

ERR Protocol error: expected '$', got '+'

I'm setting door_uuid to {"locked":true,"open":false}
(where uuid is a uuid)

SwiftPolyfill.swift fails on swift 5.8.1

compilation fails with:

/.build/checkouts/RediStack/Sources/RediStack/Cluster/SwiftPolyfill.swift:43:55: error: 'AsyncStream' is only available in macOS 10.15 or newer
    ) -> (stream: AsyncStream<Element>, continuation: AsyncStream<Element>.Continuation) {
                                                      ^
/.build/checkouts/RediStack/Sources/RediStack/Cluster/SwiftPolyfill.swift:40:17: note: add @available attribute to enclosing static method
    static func makeStream(
                ^
/.build/checkouts/RediStack/Sources/RediStack/Cluster/SwiftPolyfill.swift:39:1: note: add @available attribute to enclosing extension
extension AsyncStream {
^

Add support for Streams

Look into adding convenience APIs for Redis 5's Streams

https://redis.io/topics/streams-intro

How do I connect using tls?

I was ready to use TLS to connect, but I kept getting a clue. I've noticed that version 1.5 of release even adds nio-ssl as the default dependency, but so far I've been trying for hours and don't understand how to use TLS connections.
Maybe this is a very basic question, but I can't find a relevant post and can't find a relevant answer, can you prompt me, thank you!

Support for Cluster Mode?

Hi there,

When connected to our Redis Cluster, we're getting a lot of errors of the form:

MOVED 12858 <internal-ip>:6379

From looking at the post here: https://forums.swift.org/t/redistack-future-plans/65703
and the fact that there isn't a RedisClusterClient yet, my understanding is that this client doesn't yet support cluster mode, and my best option is to reduce it to a single shard/replica for now. Is that correct?

ThreadSanitizer failures due to metrics

RediStack 1.4.0 triggers multiple Thread Sanitizer failures due to lack of thread-safety in metrics when multiple connections are in use simultaneously, as in the unit tests for Vapor's RediStack integration provider. swift-demangled transcript of the TSan output is attached. Full test run output is available here.

Improve RedisPipeline Results Ergonomics

Right now - the RedisPipeline.execute() method resolves [RESPValue] - but this creates an unwieldy API compared to how RedisClient in general behaves.

In enqueue(operation:) the eventual resolves value is just ignored and discarded - but it would be nice to be able to capture the type information to somehow instead return [RESPValueEncodable] or have a way to grab in a reliable, non-optional, type-safe way the actual return values from the RedisClient interface so that it's consistent.

Right now, commands like zpopmax(from:max:) have a relatively complex response structure that users of RedisPipeline are left to figure out themselves, lest we just expose our helper method _mapSortedSetResponse(_:scoreIsFirst:)

Feature Request: Support for Command Pipelining

I am using the RediStack library in my Swift project with Vapor, and I need to send multiple commands to Redis together as a batch to improve performance. However, I couldn't find a direct way to send multiple commands together using the existing API.

Current Behavior

The send(command: String...) function appears to set connection.sendCommandsImmediately = true under the hood, which sends each command immediately and cannot be configured otherwise. Even if I construct RedisCommand myself, I don't see any way to send it manually, only through send(command: String...). Please correct me if I'm wrong.
I found several old discussions about this topic, but all of them seem irrelevant.

Refactor RESP Encoder/Decoder to use ByteBuffer directly

Is it possible to make `RedisConnectionPool` Sendable?