[C++] `ReplayMerge` with multicast live destination doesn't merge.

[yeetfield]

[C++] `ReplayMerge` with multicast live destination doesn't merge.

[mikeb01]

ReplayMerge can be tricky to get correct, the best approach would be to submit a failing test case that we can look at to see if there is an underlying issue.

[yeetfield]

Thanks! I'll work towards building a simple repro. We're going to first retry this test with multiple hosts, rather than do this on one.

[yeetfield]

It turns out that there were some more fundamental issues in how I'd set up the multicast connectivity. I was using a multicast group that wasn't actually within the range reserved for us by our network engineers, and there was no connectivity at all, so this isn't an Aeron problem.

[mikeb01]

Looking quickly at your logging, I suspect there may be an issue with the replay_destination. Try setting the replay_destination to have the same address as the live_destination, but a different port.

[yeetfield]

Hi Mike,

Indeed, looking back on this, the replay destination based on localhost makes no sense.

My company has been in correspondence with Adaptive recently towards formalizing developer support, but in the interim I thought I'd post my query here.

I'm currently experiencing the same phenomenon with a hopefully more correct configuration involving Onload. In this scenario, I have three hosts, one for each of: a producer, an archive, and a consumer. The producer publishes to a multicast endpoint, 239.192.11.91:20123. The archive records every publication created by the producer. The consumer can either start in replay-merge mode and bootstrap from the archive or start a regular subscription and listen immediately to the live feed on 239.192.11.91:20123.

I'm observing that the replay-merge scenario doesn't quite work as expected; when my replay-merge catches up, I see that it adds the live destination (based on application logging and checking the ReplayMerge), but I see no evidence of this in the output of AeronStat on the subscriber host. Consequently, the ReplayMerge never detects that there are multiple active transports for this image, and so never considers itself "merged". I can see a few entries for the unicast replay from the Archive, but no evidence of a multicast live destination. This is in contrast to the entries I see when I start a regular subscription (not originating from a replay-merge) on the same host, where I can see the multicast group.

Apologies for the deluge of information, but here's the output of the publication host's AeronStat with no subscriptions other than the one created by the Archive:

2024-04-24 21:37:23.264+0000 - Aeron Stat (CnC vaeron version=1.43.0 commit=940b6ccf43), pid 143400, client liveness 10,000,000,000 ns
===========================
  0 :       25,522,540,032 - Bytes sent
  1 :               73,160 - Bytes received
  2 :                    0 - Failed offers to ReceiverProxy
  3 :                    0 - Failed offers to SenderProxy
  4 :                    0 - Failed offers to DriverConductorProxy
  5 :                    0 - NAKs sent
  6 :                   62 - NAKs received
  7 :                   45 - Status Messages sent
  8 :            1,219,331 - Status Messages received
  9 :               12,165 - Heartbeats sent
 10 :                   12 - Heartbeats received
 11 :                   21 - Retransmits sent
 12 :                    0 - Flow control under runs
 13 :                    0 - Flow control over runs
 14 :                    0 - Invalid packets
 15 :                    0 - Errors: version=1.43.0 commit=940b6ccf43
 16 :                   14 - Short sends
 17 :                    0 - Failed attempts to free log buffers
 18 :                    1 - Sender flow control limits, i.e. back-pressure events
 19 :                    0 - Unblocked Publications
 20 :                    0 - Unblocked Control Commands
 21 :                    0 - Possible TTL Asymmetry
 22 :                    0 - ControllableIdleStrategy status
 23 :                    0 - Loss gap fills
 24 :                    2 - Client liveness timeouts
 25 :                    0 - Resolution changes: driverName= hostname=[producer host]
 26 :           33,544,129 - Conductor max cycle time doing its work in ns: DEDICATED
 27 :                    0 - Conductor work cycle exceeded threshold count: threshold=1000000000ns DEDICATED
 28 :           15,346,333 - Sender max cycle time doing its work in ns: DEDICATED
 29 :                    0 - Sender work cycle exceeded threshold count: threshold=1000000000ns DEDICATED
 30 :           11,251,323 - Receiver max cycle time doing its work in ns: DEDICATED
 31 :                    0 - Receiver work cycle exceeded threshold count: threshold=1000000000ns DEDICATED
 32 :               15,021 - NameResolver max time in ns
 33 :                    0 - NameResolver exceeded threshold count
 34 :               76,544 - Aeron software: version=1.43.0 commit=940b6ccf43
 35 :          162,455,552 - Bytes currently mapped
 36 :                    0 - backpressures
 39 :                    0 - admin actions
 40 :    1,713,994,642,784 - client-heartbeat: 47 version=1.43.0 commit=940b6ccf43
 41 :                    0 - snd-bpe: 50 -1658328154 0 aeron:udp?endpoint=239.192.11.91:20123|channel-snd-ts-offset=20|interface=172.20.12.6/25|media-rcv-ts-offset=28|fc=min|ssc=true|channel-rcv-ts-offset=36
 42 :                    1 - snd-channel: aeron:udp?endpoint=239.192.11.91:20123|channel-snd-ts-offset=20|interface=172.20.12.6/25|media-rcv-ts-offset=28|fc=min|ssc=true|channel-rcv-ts-offset=36 0.0.0.0:20123
 43 :                    1 - snd-local-sockaddr: 42 0.0.0.0:20123
 44 :                    1 - fc-receivers: 50 -1658328154 0 aeron:udp?endpoint=239.192.11.91:20123|channel-snd-ts-offset=20|interface=172.20.12.6/25|media-rcv-ts-offset=28|fc=min|ssc=true|channel-rcv-ts-offset=36
 45 :                7,840 - pub-pos (sampled): 50 -1658328154 0 aeron:udp?endpoint=239.192.11.91:20123|channel-snd-ts-offset=20|interface=172.20.12.6/25|media-rcv-ts-offset=28|fc=min|ssc=true|channel-rcv-ts-offset=36
 46 :              131,072 - snd-lmt: 50 -1658328154 0 aeron:udp?endpoint=239.192.11.91:20123|channel-snd-ts-offset=20|interface=172.20.12.6/25|media-rcv-ts-offset=28|fc=min|ssc=true|channel-rcv-ts-offset=36
 47 :            8,396,448 - pub-lmt: 50 -1658328154 0 aeron:udp?endpoint=239.192.11.91:20123|channel-snd-ts-offset=20|interface=172.20.12.6/25|media-rcv-ts-offset=28|fc=min|ssc=true|channel-rcv-ts-offset=36
 48 :                7,840 - snd-pos: 50 -1658328154 0 aeron:udp?endpoint=239.192.11.91:20123|channel-snd-ts-offset=20|interface=172.20.12.6/25|media-rcv-ts-offset=28|fc=min|ssc=true|channel-rcv-ts-offset=36

and here it is once the replay merge has caught up, based on wall-time, but isn't considered "merged"

2024-04-24 21:40:48.663+0000 - Aeron Stat (CnC vaeron version=1.43.0 commit=940b6ccf43), pid 143400, client liveness 10,000,000,000 ns
===========================
  0 :       25,522,638,560 - Bytes sent
  1 :               73,160 - Bytes received
  2 :                    0 - Failed offers to ReceiverProxy
  3 :                    0 - Failed offers to SenderProxy
  4 :                    0 - Failed offers to DriverConductorProxy
  5 :                    0 - NAKs sent
  6 :                   62 - NAKs received
  7 :                   45 - Status Messages sent
  8 :            1,221,249 - Status Messages received
  9 :               14,219 - Heartbeats sent
 10 :                   12 - Heartbeats received
 11 :                   21 - Retransmits sent
 12 :                    0 - Flow control under runs
 13 :                    0 - Flow control over runs
 14 :                    0 - Invalid packets
 15 :                    0 - Errors: version=1.43.0 commit=940b6ccf43
 16 :                   14 - Short sends
 17 :                    0 - Failed attempts to free log buffers
 18 :                    1 - Sender flow control limits, i.e. back-pressure events
 19 :                    0 - Unblocked Publications
 20 :                    0 - Unblocked Control Commands
 21 :                    0 - Possible TTL Asymmetry
 22 :                    0 - ControllableIdleStrategy status
 23 :                    0 - Loss gap fills
 24 :                    2 - Client liveness timeouts
 25 :                    0 - Resolution changes: driverName= hostname=[producer host]
 26 :           33,544,129 - Conductor max cycle time doing its work in ns: DEDICATED
 27 :                    0 - Conductor work cycle exceeded threshold count: threshold=1000000000ns DEDICATED
 28 :           15,346,333 - Sender max cycle time doing its work in ns: DEDICATED
 29 :                    0 - Sender work cycle exceeded threshold count: threshold=1000000000ns DEDICATED
 30 :           11,251,323 - Receiver max cycle time doing its work in ns: DEDICATED
 31 :                    0 - Receiver work cycle exceeded threshold count: threshold=1000000000ns DEDICATED
 32 :               15,021 - NameResolver max time in ns
 33 :                    0 - NameResolver exceeded threshold count
 34 :               76,544 - Aeron software: version=1.43.0 commit=940b6ccf43
 35 :          162,455,552 - Bytes currently mapped
 36 :                    0 - backpressures
 39 :                    0 - admin actions
 40 :    1,713,994,848,287 - client-heartbeat: 47 version=1.43.0 commit=940b6ccf43
 41 :                    0 - snd-bpe: 50 -1658328154 0 aeron:udp?endpoint=239.192.11.91:20123|channel-snd-ts-offset=20|interface=172.20.12.6/25|media-rcv-ts-offset=28|fc=min|ssc=true|channel-rcv-ts-offset=36
 42 :                    1 - snd-channel: aeron:udp?endpoint=239.192.11.91:20123|channel-snd-ts-offset=20|interface=172.20.12.6/25|media-rcv-ts-offset=28|fc=min|ssc=true|channel-rcv-ts-offset=36 0.0.0.0:20123
 43 :                    1 - snd-local-sockaddr: 42 0.0.0.0:20123
 44 :                    2 - fc-receivers: 50 -1658328154 0 aeron:udp?endpoint=239.192.11.91:20123|channel-snd-ts-offset=20|interface=172.20.12.6/25|media-rcv-ts-offset=28|fc=min|ssc=true|channel-rcv-ts-offset=36
 45 :               40,640 - pub-pos (sampled): 50 -1658328154 0 aeron:udp?endpoint=239.192.11.91:20123|channel-snd-ts-offset=20|interface=172.20.12.6/25|media-rcv-ts-offset=28|fc=min|ssc=true|channel-rcv-ts-offset=36
 46 :              144,192 - snd-lmt: 50 -1658328154 0 aeron:udp?endpoint=239.192.11.91:20123|channel-snd-ts-offset=20|interface=172.20.12.6/25|media-rcv-ts-offset=28|fc=min|ssc=true|channel-rcv-ts-offset=36
 47 :            8,429,248 - pub-lmt: 50 -1658328154 0 aeron:udp?endpoint=239.192.11.91:20123|channel-snd-ts-offset=20|interface=172.20.12.6/25|media-rcv-ts-offset=28|fc=min|ssc=true|channel-rcv-ts-offset=36
 48 :               40,640 - snd-pos: 50 -1658328154 0 aeron:udp?endpoint=239.192.11.91:20123|channel-snd-ts-offset=20|interface=172.20.12.6/25|media-rcv-ts-offset=28|fc=min|ssc=true|channel-rcv-ts-offset=36

The only evidence, apparent to me anyway, that the publisher's even aware of the new subscriber is this fc-receivers, which seems to be "flow control receivers", i.e. the downstream parties participating in flow control. The fact that fc-receivers has changed suggests to me that the control channel (239.192.11.92:20123) associated with the multicast publication is working correctly, but I may be mistaken. I tried to listen in with some basic multicast utilities, but maybe Onload is getting in the way here; I'm not sure.

Going to the consumer host now, this is the output of its AeronStat while it's got one subscription in the "caught up, but not merged" state:

2024-04-24 21:43:17.033+0000 - Aeron Stat (CnC vaeron version=1.43.0 commit=940b6ccf43), pid 113766, client liveness 10,000,000,000 ns
===========================
  0 :        3,936,279,456 - Bytes sent
  1 :       53,867,449,625 - Bytes received
  2 :                    0 - Failed offers to ReceiverProxy
  3 :                    0 - Failed offers to SenderProxy
  4 :                    0 - Failed offers to DriverConductorProxy
  5 :                   75 - NAKs sent
  6 :                    0 - NAKs received
  7 :            2,206,639 - Status Messages sent
  8 :              478,641 - Status Messages received
  9 :                3,746 - Heartbeats sent
 10 :               48,761 - Heartbeats received
 11 :                    0 - Retransmits sent
 12 :                   13 - Flow control under runs
 13 :                    0 - Flow control over runs
 14 :                    1 - Invalid packets
 15 :                    2 - Errors: version=1.43.0 commit=940b6ccf43
 16 :                    0 - Short sends
 17 :                    0 - Failed attempts to free log buffers
 18 :                    0 - Sender flow control limits, i.e. back-pressure events
 19 :                    0 - Unblocked Publications
 20 :                    0 - Unblocked Control Commands
 21 :                    0 - Possible TTL Asymmetry
 22 :                    0 - ControllableIdleStrategy status
 23 :                    0 - Loss gap fills
 24 :                    0 - Client liveness timeouts
 25 :                    0 - Resolution changes: driverName= hostname=[consumer host]
 26 :          115,306,540 - Conductor max cycle time doing its work in ns: DEDICATED
 27 :                    0 - Conductor work cycle exceeded threshold count: threshold=1000000000ns DEDICATED
 28 :           16,002,666 - Sender max cycle time doing its work in ns: DEDICATED
 29 :                    0 - Sender work cycle exceeded threshold count: threshold=1000000000ns DEDICATED
 30 :           15,932,194 - Receiver max cycle time doing its work in ns: DEDICATED
 31 :                    0 - Receiver work cycle exceeded threshold count: threshold=1000000000ns DEDICATED
 32 :               99,805 - NameResolver max time in ns
 33 :                    0 - NameResolver exceeded threshold count
 34 :               76,544 - Aeron software: version=1.43.0 commit=940b6ccf43
 35 :        1,676,541,952 - Bytes currently mapped
 37 :        1,007,543,008 - rcv-pos: 51017991 515649580 20 aeron:udp?mtu=1408|sparse=true|term-length=65536|endpoint=172.20.12.70:0
 38 :                    1 - snd-channel: aeron:udp?mtu=1408|sparse=true|term-length=65536|endpoint=172.20.175.116:8010 172.20.175.112:47397
 39 :        1,007,543,008 - sub-pos: 51017988 515649580 20 aeron:udp?mtu=1408|sparse=true|term-length=65536|endpoint=172.20.12.70:0 @0
 42 :               64,480 - sub-pos: 51017993 515649581 0 aeron:udp?control-mode=manual @0
 45 :                    1 - rcv-local-sockaddr: 58 172.20.12.70:49617
 46 :               64,480 - rcv-hwm: 51051342 -1658328154 0 aeron:udp?control-mode=manual
 47 :        1,007,543,104 - rcv-hwm: 51017991 515649580 20 aeron:udp?mtu=1408|sparse=true|term-length=65536|endpoint=172.20.12.70:0
 49 :               64,480 - rcv-pos: 51051342 -1658328154 0 aeron:udp?control-mode=manual
 51 :                    1 - rcv-local-sockaddr: 74 172.20.175.112:40808
 52 :               64,480 - rcv-hwm: 51017997 515649581 0 aeron:udp?control-mode=manual
 53 :               64,480 - rcv-pos: 51017997 515649581 0 aeron:udp?control-mode=manual
 55 :          671,018,048 - snd-pos: 51017989 1283955948 10 aeron:udp?mtu=1408|sparse=true|term-length=65536|endpoint=172.20.175.116:8010
 57 :          671,050,816 - pub-lmt: 51017989 1283955948 10 aeron:udp?mtu=1408|sparse=true|term-length=65536|endpoint=172.20.175.116:8010
 58 :                    1 - rcv-channel: aeron:udp?mtu=1408|sparse=true|term-length=65536|endpoint=172.20.12.70:0 172.20.12.70:49617
 62 :                    1 - snd-local-sockaddr: 38 172.20.175.112:47397
 63 :          671,047,360 - snd-lmt: 51017989 1283955948 10 aeron:udp?mtu=1408|sparse=true|term-length=65536|endpoint=172.20.175.116:8010
 64 :    1,713,994,996,579 - client-heartbeat: 51017987 version=1.43.0 commit=940b6ccf43
 68 :                    0 - snd-bpe: 51017989 1283955948 10 aeron:udp?mtu=1408|sparse=true|term-length=65536|endpoint=172.20.175.116:8010
 69 :          671,018,112 - pub-pos (sampled): 51017989 1283955948 10 aeron:udp?mtu=1408|sparse=true|term-length=65536|endpoint=172.20.175.116:8010
 74 :                    1 - rcv-channel: aeron:udp?control-mode=manual
 75 :               64,480 - sub-pos: 51017993 -1658328154 0 aeron:udp?control-mode=manual @13120
 77 :                    1 - rcv-local-sockaddr: 74 0.0.0.0:20123

What surprises me is that I see no evidence of the live endpoint, 239.192.11.91:20123, here, despite the application's logs suggesting that ReplayMerge::isLiveAdded() is true.

I'll start another program now that creates a regular subscription to 239.192.11.91:20123 to show the difference:

2024-04-24 21:45:51.591+0000 - Aeron Stat (CnC vaeron version=1.43.0 commit=940b6ccf43), pid 113766, client liveness 10,000,000,000 ns
===========================
  0 :        4,259,196,704 - Bytes sent
  1 :       54,352,217,737 - Bytes received
  2 :                    0 - Failed offers to ReceiverProxy
  3 :                    0 - Failed offers to SenderProxy
  4 :                    0 - Failed offers to DriverConductorProxy
  5 :                   75 - NAKs sent
  6 :                    0 - NAKs received
  7 :            2,267,038 - Status Messages sent
  8 :              517,754 - Status Messages received
  9 :                3,746 - Heartbeats sent
 10 :               51,979 - Heartbeats received
 11 :                    0 - Retransmits sent
 12 :                   13 - Flow control under runs
 13 :                    0 - Flow control over runs
 14 :                    1 - Invalid packets
 15 :                    2 - Errors: version=1.43.0 commit=940b6ccf43
 16 :                    0 - Short sends
 17 :                    0 - Failed attempts to free log buffers
 18 :                    0 - Sender flow control limits, i.e. back-pressure events
 19 :                    0 - Unblocked Publications
 20 :                    0 - Unblocked Control Commands
 21 :                    0 - Possible TTL Asymmetry
 22 :                    0 - ControllableIdleStrategy status
 23 :                    0 - Loss gap fills
 24 :                    0 - Client liveness timeouts
 25 :                    0 - Resolution changes: driverName= hostname=[consumer host]
 26 :          115,306,540 - Conductor max cycle time doing its work in ns: DEDICATED
 27 :                    0 - Conductor work cycle exceeded threshold count: threshold=1000000000ns DEDICATED
 28 :           16,002,666 - Sender max cycle time doing its work in ns: DEDICATED
 29 :                    0 - Sender work cycle exceeded threshold count: threshold=1000000000ns DEDICATED
 30 :           15,932,194 - Receiver max cycle time doing its work in ns: DEDICATED
 31 :                    0 - Receiver work cycle exceeded threshold count: threshold=1000000000ns DEDICATED
 32 :               99,805 - NameResolver max time in ns
 33 :                    0 - NameResolver exceeded threshold count
 34 :               76,544 - Aeron software: version=1.43.0 commit=940b6ccf43
 35 :        1,726,877,696 - Bytes currently mapped
 36 :                    1 - rcv-channel: aeron:udp?channel-rcv-ts-offset=36|fc=min|media-rcv-ts-offset=28|interface=172.20.12.70/25|channel-snd-ts-offset=20|endpoint=239.192.11.91:20123 0.0.0.0:20123
 37 :        1,492,392,128 - rcv-pos: 51017991 515649580 20 aeron:udp?mtu=1408|sparse=true|term-length=65536|endpoint=172.20.12.70:0
 38 :                    1 - snd-channel: aeron:udp?mtu=1408|sparse=true|term-length=65536|endpoint=172.20.175.116:8010 172.20.175.112:47397
 39 :        1,492,392,032 - sub-pos: 51017988 515649580 20 aeron:udp?mtu=1408|sparse=true|term-length=65536|endpoint=172.20.12.70:0 @0
 41 :                    1 - rcv-local-sockaddr: 36 0.0.0.0:20123
 42 :               89,120 - sub-pos: 51017993 515649581 0 aeron:udp?control-mode=manual @0
 45 :                    1 - rcv-local-sockaddr: 58 172.20.12.70:49617
 46 :               89,120 - rcv-hwm: 51051342 -1658328154 0 aeron:udp?control-mode=manual
 47 :        1,492,392,128 - rcv-hwm: 51017991 515649580 20 aeron:udp?mtu=1408|sparse=true|term-length=65536|endpoint=172.20.12.70:0
 49 :               89,120 - rcv-pos: 51051342 -1658328154 0 aeron:udp?control-mode=manual
 50 :               89,120 - rcv-pos: 65888193 -1658328154 0 aeron:udp?channel-rcv-ts-offset=36|fc=min|media-rcv-ts-offset=28|interface=172.20.12.70/25|channel-snd-ts-offset=20|endpoint=239.192.11.91:20123
 51 :                    1 - rcv-local-sockaddr: 74 172.20.175.112:40808
 52 :               89,120 - rcv-hwm: 51017997 515649581 0 aeron:udp?control-mode=manual
 53 :               89,120 - rcv-pos: 51017997 515649581 0 aeron:udp?control-mode=manual
 54 :               89,120 - sub-pos: 65885094 -1658328154 0 aeron:udp?channel-rcv-ts-offset=36|fc=min|media-rcv-ts-offset=28|interface=172.20.12.70/25|channel-snd-ts-offset=20|endpoint=239.192.11.91:20123 @85920
 55 :          993,934,976 - snd-pos: 51017989 1283955948 10 aeron:udp?mtu=1408|sparse=true|term-length=65536|endpoint=172.20.175.116:8010
 57 :          993,967,744 - pub-lmt: 51017989 1283955948 10 aeron:udp?mtu=1408|sparse=true|term-length=65536|endpoint=172.20.175.116:8010
 58 :                    1 - rcv-channel: aeron:udp?mtu=1408|sparse=true|term-length=65536|endpoint=172.20.12.70:0 172.20.12.70:49617
 61 :               89,120 - rcv-hwm: 65888193 -1658328154 0 aeron:udp?channel-rcv-ts-offset=36|fc=min|media-rcv-ts-offset=28|interface=172.20.12.70/25|channel-snd-ts-offset=20|endpoint=239.192.11.91:20123
 62 :                    1 - snd-local-sockaddr: 38 172.20.175.112:47397
 63 :          993,964,288 - snd-lmt: 51017989 1283955948 10 aeron:udp?mtu=1408|sparse=true|term-length=65536|endpoint=172.20.175.116:8010
 64 :    1,713,995,151,344 - client-heartbeat: 51017987 version=1.43.0 commit=940b6ccf43
 66 :    1,713,995,151,237 - client-heartbeat: 65885009 version=1.43.0 commit=940b6ccf43
 68 :                    0 - snd-bpe: 51017989 1283955948 10 aeron:udp?mtu=1408|sparse=true|term-length=65536|endpoint=172.20.175.116:8010
 69 :          993,934,976 - pub-pos (sampled): 51017989 1283955948 10 aeron:udp?mtu=1408|sparse=true|term-length=65536|endpoint=172.20.175.116:8010
 74 :                    1 - rcv-channel: aeron:udp?control-mode=manual
 75 :               89,120 - sub-pos: 51017993 -1658328154 0 aeron:udp?control-mode=manual @13120
 77 :                    1 - rcv-local-sockaddr: 74 0.0.0.0:20123

(What's also surprising to me about this is that both snapshots of AeronStat have 77 rows, even though I'd expect there to be more after I start the second program.)

Does anything here stand out to you as far as explaining why the replay-merge can't seem to latch onto the live feed?

Thank you!

[mikeb01]

It is still quite tricky to see exactly what is going on solely from the AeronStat. Example code could make it a bit easier for us figure out what is going on. A couple of questions:

• Does your test work when you are not using multicast?
• Does your test work without OnLoad?

[yeetfield]

I've made a repro available here: https://github.com/yeetfield/aq_aeron_repro. I'll have to look into how recording unicast publications work (does the archive have to be on the same host as the subscription?). I'll also try disabling onload here. Thank you!

[yeetfield]

Disabling Onload via USE_ONLOAD=false did not have any effect on my repro.

[mikeb01]

The key issue is that you need to set the session-id on the replay channel so that the live destination will merge into the same image as the replay destination. We need this in order to get the image's active image count to reach 2. I've attached a patch to this ticket that shows the code changes required to get the merge to occur.

patch.txt

[yeetfield]

Thank you! That did the trick.