quickbirdeng/combinerxswiftperformance

A test suite comparing the performance of Combine and RxSwift

Combine vs. RxSwift Performance Benchmark Test Suite 📊

This project contains a benchmarking test suite for comparing the performance of the most commonly used components and operators in RxSwift and Combine. For a detailed comparison of RxSwift with Combine have a look at our blog post.

The RxSwift performance benchmark tests are the original ones used in the RxSwift project. We removed the two tests from RxCocoa testing Drivers, since there is no equivalent in Combine. The Combine tests are 1:1 translated tests from the Rx test-suite and should, therefore, be easily comparable.

Important update: As mentioned correctly the old numbers were created with XCTests running in DEBUG mode. The differences seem not so critical in Release builds. We have updated all the numbers and graphs to use release builds.

As a summary Combine was faster in every test and on average 41% more performant than RxSwift. These statistics show every test-method and its results. Lower is better.

Test Results Summary

Test	RxSwift (ms)	Combine (ms)	Factor
PublishSubjectPumping	227	135	168%
PublishSubjectPumpingTwoSubscriptions	400	246	163%
PublishSubjectCreating	295	250	118%
MapFilterPumping	123	132	93%
MapFilterCreating	168	114	147%
FlatMapsPumping	646	367	176%
FlatMapsCreating	214	121	177%
FlatMapLatestPumping	810	696	116%
FlatMapLatestCreating	263	180	146%
CombineLatestPumping	298	282	106%
CombineLatestCreating	644	467	138%

Testing Details

Machine: MacBook Pro 2018, 2,7 GHz Intel Core i7, 16 GB IDE: Xcode 11.0 beta 5 (11M382q) Testing Device: iPhone XR Simulator

Performance Test Example: PublishSubject Pumping

For every test we replace the RxSwift component with the corresponding Combine component. In this case PublishSubject with PassthroughSubject.

RxSwift

func testPublishSubjectPumping() {
    measure {
        var sum = 0
        let subject = PublishSubject<Int>()

        let subscription = subject
            .subscribe(onNext: { x in
                sum += x
            })

        for _ in 0 ..< iterations * 100 {
            subject.on(.next(1))
        }

        subscription.dispose()

        XCTAssertEqual(sum, iterations * 100)
    }
}

Combine

func testPublishSubjectPumping() {
    measure {
        var sum = 0
        let subject = PassthroughSubject<Int, Never>()

        let subscription = subject
            .sink(receiveValue: { x in
                sum += x
            })

        for _ in 0 ..< iterations * 100 {
            subject.send(1)
        }
        
        subscription.cancel()
        
        XCTAssertEqual(sum, iterations * 100)
    }
}

Detailed Performance Test Results: RxSwift vs. Combine

PublishSubjectPumping

PublishSubjectPumpingTwoSubscriptions

PublishSubjectCreating

MapFilterPumping

MapFilterCreating

FlatMapsPumping

FlatMapsCreating

FlatMapLatestPumping

FlatMapLatestCreating

CombineLatestPumping

CombineLatestCreating

combinerxswiftperformance's Issues

Add on-device test results?

I just built this project and its tests in Release mode, and ran it on an iPhone Xs Max running iOS 13 beta 5 (build 17A5547d). Here's my results, if you'd like to add them to the project/post.

2019-08-07 11:13:01.663002-0700 SpeedTest[686:332338] Launching with XCTest injected. Preparing to run tests.
2019-08-07 11:13:01.678508-0700 SpeedTest[686:332338] Waiting to run tests until the app finishes launching.
Test Suite 'All tests' started at 2019-08-07 11:13:01.831
Test Suite 'SpeedTestTests.xctest' started at 2019-08-07 11:13:01.831
Test Suite 'CombineTests' started at 2019-08-07 11:13:01.831
Test Case '-[SpeedTestTests.CombineTests testCombineLatestCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testCombineLatestCreating]' measured [Time, seconds] average: 0.317, relative standard deviation: 6.035%, values: [0.373770, 0.315254, 0.311101, 0.313436, 0.308780, 0.308293, 0.310991, 0.309422, 0.308666, 0.308311], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testCombineLatestCreating]' passed (3.484 seconds).
Test Case '-[SpeedTestTests.CombineTests testCombineLatestPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testCombineLatestPumping]' measured [Time, seconds] average: 0.254, relative standard deviation: 6.278%, values: [0.301132, 0.246959, 0.246933, 0.246930, 0.247169, 0.256698, 0.248910, 0.247402, 0.247251, 0.251685], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testCombineLatestPumping]' passed (2.793 seconds).
Test Case '-[SpeedTestTests.CombineTests testFlatMapLatestCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testFlatMapLatestCreating]' measured [Time, seconds] average: 0.174, relative standard deviation: 9.541%, values: [0.223468, 0.168516, 0.168904, 0.168670, 0.168296, 0.168805, 0.168205, 0.167899, 0.166943, 0.167873], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testFlatMapLatestCreating]' passed (1.990 seconds).
Test Case '-[SpeedTestTests.CombineTests testFlatMapLatestPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testFlatMapLatestPumping]' measured [Time, seconds] average: 0.666, relative standard deviation: 2.373%, values: [0.713103, 0.658805, 0.659432, 0.661265, 0.660234, 0.660054, 0.661817, 0.661364, 0.660532, 0.661145], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testFlatMapLatestPumping]' passed (6.909 seconds).
Test Case '-[SpeedTestTests.CombineTests testFlatMapsCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testFlatMapsCreating]' measured [Time, seconds] average: 0.119, relative standard deviation: 13.664%, values: [0.167399, 0.113666, 0.113355, 0.113330, 0.113366, 0.113154, 0.113349, 0.113165, 0.113210, 0.113298], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testFlatMapsCreating]' passed (1.439 seconds).
Test Case '-[SpeedTestTests.CombineTests testFlatMapsPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testFlatMapsPumping]' measured [Time, seconds] average: 0.368, relative standard deviation: 4.310%, values: [0.415549, 0.362704, 0.363176, 0.361723, 0.362332, 0.363070, 0.363443, 0.363041, 0.362221, 0.362598], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testFlatMapsPumping]' passed (3.932 seconds).
Test Case '-[SpeedTestTests.CombineTests testMapFilterCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testMapFilterCreating]' measured [Time, seconds] average: 0.114, relative standard deviation: 15.577%, values: [0.166588, 0.107760, 0.107645, 0.107777, 0.107944, 0.107716, 0.107834, 0.107345, 0.107410, 0.107330], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testMapFilterCreating]' passed (1.387 seconds).
Test Case '-[SpeedTestTests.CombineTests testMapFilterPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testMapFilterPumping]' measured [Time, seconds] average: 0.146, relative standard deviation: 11.349%, values: [0.195804, 0.140595, 0.140445, 0.140486, 0.140395, 0.140572, 0.140685, 0.140616, 0.140502, 0.140615], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testMapFilterPumping]' passed (1.713 seconds).
Test Case '-[SpeedTestTests.CombineTests testPublishSubjectCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testPublishSubjectCreating]' measured [Time, seconds] average: 0.217, relative standard deviation: 8.163%, values: [0.269519, 0.210303, 0.210658, 0.210114, 0.210629, 0.210665, 0.210536, 0.210768, 0.210886, 0.210952], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testPublishSubjectCreating]' passed (2.417 seconds).
Test Case '-[SpeedTestTests.CombineTests testPublishSubjectPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testPublishSubjectPumping]' measured [Time, seconds] average: 0.126, relative standard deviation: 13.366%, values: [0.176685, 0.120584, 0.120393, 0.120398, 0.120567, 0.120582, 0.120625, 0.120456, 0.120588, 0.120272], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testPublishSubjectPumping]' passed (1.513 seconds).
Test Case '-[SpeedTestTests.CombineTests testPublishSubjectPumpingTwoSubscriptions]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testPublishSubjectPumpingTwoSubscriptions]' measured [Time, seconds] average: 0.213, relative standard deviation: 8.093%, values: [0.264186, 0.206561, 0.206730, 0.206495, 0.206359, 0.206669, 0.206638, 0.206718, 0.206959, 0.208698], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testPublishSubjectPumpingTwoSubscriptions]' passed (2.378 seconds).
Test Suite 'CombineTests' passed at 2019-08-07 11:13:31.788.
	 Executed 11 tests, with 0 failures (0 unexpected) in 29.955 (29.956) seconds
Test Suite 'RxSwiftTests' started at 2019-08-07 11:13:31.788
Test Case '-[SpeedTestTests.RxSwiftTests testCombineLatestCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testCombineLatestCreating]' measured [Time, seconds] average: 0.424, relative standard deviation: 4.463%, values: [0.480355, 0.417623, 0.417502, 0.417144, 0.417412, 0.417046, 0.417352, 0.417095, 0.417495, 0.417291], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testCombineLatestCreating]' passed (4.489 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testCombineLatestPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testCombineLatestPumping]' measured [Time, seconds] average: 0.244, relative standard deviation: 7.176%, values: [0.297019, 0.238397, 0.238588, 0.238113, 0.239896, 0.240161, 0.238805, 0.236376, 0.238680, 0.238684], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testCombineLatestPumping]' passed (2.697 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapLatestCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapLatestCreating]' measured [Time, seconds] average: 0.166, relative standard deviation: 11.201%, values: [0.221616, 0.159861, 0.159359, 0.159367, 0.159202, 0.160056, 0.159847, 0.159774, 0.159760, 0.159958], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapLatestCreating]' passed (1.911 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapLatestPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapLatestPumping]' measured [Time, seconds] average: 0.569, relative standard deviation: 2.964%, values: [0.619199, 0.562502, 0.563592, 0.561449, 0.563635, 0.560609, 0.563452, 0.563670, 0.563945, 0.565810], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapLatestPumping]' passed (5.940 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapsCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapsCreating]' measured [Time, seconds] average: 0.158, relative standard deviation: 10.053%, values: [0.205639, 0.153024, 0.152946, 0.152780, 0.152942, 0.152866, 0.152381, 0.152329, 0.152604, 0.152438], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapsCreating]' passed (1.832 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapsPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapsPumping]' measured [Time, seconds] average: 0.522, relative standard deviation: 3.159%, values: [0.570750, 0.517117, 0.514583, 0.515876, 0.517330, 0.514596, 0.515429, 0.516117, 0.519115, 0.514222], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapsPumping]' passed (5.467 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testMapFilterCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testMapFilterCreating]' measured [Time, seconds] average: 0.130, relative standard deviation: 12.899%, values: [0.179818, 0.124204, 0.124040, 0.124229, 0.124294, 0.123647, 0.124333, 0.124075, 0.123292, 0.124630], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testMapFilterCreating]' passed (1.549 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testMapFilterPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testMapFilterPumping]' measured [Time, seconds] average: 0.129, relative standard deviation: 13.754%, values: [0.182531, 0.123416, 0.123589, 0.123505, 0.123656, 0.123497, 0.123388, 0.121803, 0.123525, 0.123406], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testMapFilterPumping]' passed (1.544 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectCreating]' measured [Time, seconds] average: 0.219, relative standard deviation: 8.262%, values: [0.273149, 0.212711, 0.212802, 0.212842, 0.212670, 0.212838, 0.212886, 0.213015, 0.213080, 0.212935], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectCreating]' passed (2.441 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectPumping]' measured [Time, seconds] average: 0.203, relative standard deviation: 8.249%, values: [0.252791, 0.197036, 0.196841, 0.197075, 0.196869, 0.197041, 0.196938, 0.197119, 0.197845, 0.196916], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectPumping]' passed (2.278 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectPumpingTwoSubscriptions]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectPumpingTwoSubscriptions]' measured [Time, seconds] average: 0.361, relative standard deviation: 4.768%, values: [0.412536, 0.357086, 0.356588, 0.355424, 0.354889, 0.354509, 0.355141, 0.353818, 0.354854, 0.354913], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectPumpingTwoSubscriptions]' passed (3.862 seconds).
Test Suite 'RxSwiftTests' passed at 2019-08-07 11:14:05.797.
	 Executed 11 tests, with 0 failures (0 unexpected) in 34.008 (34.009) seconds
Test Suite 'SpeedTestTests.xctest' passed at 2019-08-07 11:14:05.797.
	 Executed 22 tests, with 0 failures (0 unexpected) in 63.963 (63.965) seconds
Test Suite 'All tests' passed at 2019-08-07 11:14:05.797.
	 Executed 22 tests, with 0 failures (0 unexpected) in 63.963 (63.966) seconds

The timings appear to be generated in Debug

In Release builds, the Combine timings are about the same (since Combine is shipped as a Release build from Apple) but the RxSwift timings are roughly 3-5 times faster.